When an OpenStackDataPlaneNodeSet is deployed with TLS Enabled, communications between dataplane services and with control plane services can be encrypted using TLS connections.
Functionality has been added to the openstack-operator to generate the needed certificates for all compute nodes in the nodeset. The details on how to enable this functionality and how dataplane services (including custom services) can take advantage of this functionality is provided here.
In addtion, an attribute has been added to the OpenStackDataplaneService spec to allow a CACert TLS bundle to be provided.
Certificates for dataplane services are generated by certmanager issuers, which are referenced in the OpenStackDataplaneService attributes below. These issuers must be created beforehand.
In addition, OpenStackDataplaneService contains an attribute that allows the deployer to specify a secret containing a TLS CA bundle. This secret should also be created beforehand.
-
Create the issuers and cacert bundle secrets as described in the pre-requisites above. These were likely created as part of the control plane deployment. (TODO - link to issuer/cacert docs when available)
-
Enable TLS on the OpenStackDataPlaneNodeSet and create the nodeset. Ensure that the install-certs or similar service (described below) is in the list of services before any services that require certs.
-
Deploy the OpenStackDataPlane. Certs should be created and copied to the compute nodes in the nodeset.
The OpenstackDataPlaneNodeSet has an attribute tlsEnabled, which defaults to false. The certficate generation code will be executed only if this attribute is set to true.
Certificate generation is controlled by several attributes in the OpenstackDataplaneService specification. An example is provided below.
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
name: libvirt
spec:
label: libvirt
playbook: osp.edpm.libvirt
tlsCerts:
default:
contents:
- dnsnames
- ips
networks:
- CtlPlane
keyUsages:
- digital signature
- key encipherment
- server auth
- client auth
issuer: osp-rootca-issuer-internal
caCerts: combined-ca-bundle
A more minimal configuration is provided below:
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
name: service1
spec:
label: service1
playbook: osp.edpm.service1
tlsCerts:
default:
contents:
- dnsnames
caCerts: combined-ca-bundle
This optional attribute is a string pointing to the secret containing the TLS CA certificate bundle to be mounted for the dataplane service. This secret is expected to be created in the same namespace (default: openstack) beforehand.
Not all dataplane services will require TLS certificates. For example, dataplane services that install the OS or download caches do not need TLS certificates.
tlsCerts is a map of certificates to be generated for the service. By convention, the default pre-defined services use "default" as the hash key if only one service is needed, but this is not required. Ultimately, the hash key will be part of the path where the cert is located.
Most services will only require one certificate. Some though, like libvirt, may require more than one certificate.
If tlsCerts is defined (and tlsEnabled is set on the nodeset), certs will be generated as prescribed by the following attributes:
This attribute describes what information is included in the subject alternative names (SAN) in the certificate. At this time, only two values are possible ("dnsnames" and "ips"). In the libvirt example, both attributes are added. This attribute is required.
This attribute describes which networks will be added to the SAN. For instance, in the libvirt example configuration, the DNSName and IPAdress for the node on the Ctlplane will be added to the SAN. If networks is not defined, the relevant contents for all networks will be added to the SAN. So, in the configuration for service1 above, dns names for all networks on the node are added to the SAN.
This attribute corresponds to the label for the certmanager issuer that is used to issue the certificate.
The label can be different from the name of the issuer. There can be only one issuer with the specified label.
If more than one issuer has the label, an error is generated. If the issuers attribute is not set, as in the
configuration for service1, the certificates are issued with the default root CA for internal TLS as defined
in lib-common, which is set to the label "osp-rootca-issuer-internal" for the rootca-internal issuer.
This attribute is a list of key uages to be included as key usage extensions in the certificate. The strings that correspond to valid usages are provided by the certmanage api. If this attribute is not provided, the default set of key usages as defined in lib-common. will be used. These are "key encipherment", "digital signature" and "server auth". In the above examples, we see that libvirt defines this attribute because the "client auth" key usage is also needed.
This attribute specifies whether or not to mount the certificates and keys generated for all dataplane services (for all nodes in the nodeset) in the ansible EE pod for the service. This attribute defaults to false.
The current design has a special dataplane service "install-certs" that is expected to run before any services that need certificates, and which has this attribute set to true. The purpose of this dataplane service is to copy the certs to the correct place on the compute nodes. This dataplane service is described in more detail below.
When tlsEnabled is set to True on the nodeset, and tlsCerts is defined for the dataplane service, certificates will be requested from the certmanager issuer designated in the issuer attribute (or a default) as described above.
The contents of the certificate (subject name, subject alternative names, etc.) are defined using the contents and issuer attributes as described above.
The certficates are generated when an OpenstackDataplaneDeployment is created, but before any ansible EE pods are created.
When the certificates are created, certmanager stores the certificates in secrets which are named "cert-<service_name>-<hash_key>-<node_name>-#". The # symbol represents the secret number, beginning with 0. Kubernetes distributions, such as Red Hat Openshift Platform, have a maximum secret size of 1 MiB. If the size of the created certificates and keys is larger than the maximum size of a single secret, then multiple secrets are created. Each secret receives its number and contains the certificate, key and cacert.
The certificates for all the nodes in the node set for a given service are collected in secrets named
"<nodeset>-<service_name>-<hash_key>-certs-", where the symbol represents the generated secret
number that starts at 0. These secrets are mounted in the ansibleEE when addCertMounts is enabled.
A dataplane service ("install-certs") has been added to added to copy over the certificates to the compute nodes. As noted above, this service has the addCertMounts attribute set to True. It is expected that this service will be executed before any other services that require TLS certs.
The service:
-
Mounts the
<nodeset>-<service_name>-<hash_key>-certs-#secrets for all services that have tlsCertsEnabled` set to "true". -
For each node, calls the
osp.edpm.install_certsrole which copies all the certificates and keys for that node to/var/lib/openstack/certs/<service_name>/<hash_key>. The cacert bundles are copied to/var/lib/openstack/cacerts/<service_name>.
Code should then be added to each service’s ansible role to use the certs as needed. For example, in libvirt’s role, we move the certs and keys to standard locations on the compute host. Other roles may mount the certs and keys into their containers using kolla or otherwise. The certs and keys for all the services are available as needed for all services.
The secrets that store the certificates and keys that are generated by certmanager (which are named cert-<service_name>-<hash_key>_<node_name>) are owned by certmanager. When they are created, they are labelled using "osdp-service", "osdp-service-cert-key" and "osdpns" to indicate the dataplane service, hash key and nodeset accordingly.
At the end of the deployment, these secrets are hashed and the values are stored in the secretHashes status field of the nodeset and deployment. In this way, these cert secrets are treated in exactly the same way as any other dataplane service related secrets.
Certmanager will automatically renew certificates prior to their expiration, which will result in modifications to the secrets.
The deployer can periodically review the hashes for these secrets to determine if any of them have changed - this is currently expected to be a manual process - and then may choose to invoke a new deployment to update the certificates and keys.
Based on the above description, the steps are pretty straightforward.
-
Add a tlsCerts attribute to your dataplane service. Set the contents, networks and issuer according to your needs. The service1 configuration is a minimal specification and will provide a cert with dnsNames for all the interfaces of the compute node in the SAN, issued by the internal TLS CA. This is probably sufficient for most use cases.
-
Add a specification for a CACertBundle. This attribute can be added to mount a CACert bundle even if no cert generation is needed.
-
The "install-certs: service should run before your service. It will copy the certs and cacerts to a standard location. See the section above.
-
Modify your role to do something with the generated certs.