Integrating Workload Identity Manager with OpenShift Service Mesh¶

These instruction show to use an CyberArk Workload Identity Manager (formerly known as Firefly) Sub CA to sign the mTLS certificates of OpenShift Service Mesh. The advantages of this setup are:

• Your PKI team will be able to see how many certificates have been created for the mesh.
• Your PKI team will be able enforce policies for the mesh certificates.
• Your platform will be using the PKI of your organization, rather than shadow PKI which OpenShift Service Mesh uses by default.

Prerequisites¶

A Red Hat OpenShift v4 cluster.

Step 1: Create projects (namespaces)¶

First, create the two projects (namespaces) used in this tutorial.

oc create namespace venafi
oc create namespace istio-system

Step 2: Install OperatorHub Operators¶

Install the following operators from the OperatorHub dashboard in the OpenShift web console:

1. Red Hat OpenShift Service Mesh: This operator will allow you to install a service mesh later in this tutorial. It installs the CyberArk Certificate Manager Operator and CRDs required to create a service mesh. Multiple service meshes can be run within a single OpenShift cluster. Read the Red Hat OpenShift Service Mesh 2.x installation documentation to learn more.

2. CyberArk Certificate Manager Operator for Red Hat OpenShift: Install this operator in the venafi project (namespace). Choose the option to create a venafi project (namespace) if it does not already exist.

This operator will allow you to install cert-manager, Istio CSR, and CyberArk Workload Identity Manager components of CyberArk Certificate Manager for Kubernetes. Read the CyberArk Certificate Manager Operator installation page to learn more.

Step 3: Prepare the Workload Identity Manager service account, policy, and configuration¶

1. Create an Workload Identity Manager service account in Certificate Manager - SaaS. Save the private key and the client ID as you will need both later.
2. Create a Workload Identity Manager Policy to use when issuing certificates in Certificate Manager - SaaS.
3. Create a Workload Identity Manager configuration to map the policy to the service account in Certificate Manager - SaaS.

Step 4: Create a service account private key secret¶

Before Workload Identity Manager can be installed, a Secret resource containing the Certificate Manager - SaaS service account private key must be created:

export FIREFLY_PRIVATE_KEY_FILE=path/to/firefly-private.key
oc create secret generic venafi-credentials \
  --namespace venafi \
  --from-file=svc-acct.key=${FIREFLY_PRIVATE_KEY_FILE}

Step 5: Create an image pull secret¶

1. Next, Create a new CyberArk Registry service account. This is required to pull Helm charts and Docker images from the CyberArk private OCI registries.

2. Apply the image pull secret to the venafi namespace (copy and paste the exact command from the Add a New Service Account (Settings > Service Accounts > New) page of Certificate Manager - SaaS):

   oc create namespace venafi
   
   oc apply --namespace=venafi -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: venafi-image-pull-secret
   data:
     .dockerconfigjson: YOUR_BASE64_ENCODED_PULL_SECRET
   type: kubernetes.io/dockerconfigjson
   EOF
   

Step 6: Publish the Workload Identity Manager root CA certificate to your cluster¶

Istio requires a special ConfigMap resource in each namespace, containing the root CA that signed the istiod serving certificate. The Istio sidecars use this to verify the serving certificate when they connect to istiod to request workload certificates. Istio CSR will take care of creating these ConfigMap resources for you, but you must tell it which CA certificate to use. In this tutorial, Istio CSR will be using a gRPC serving certificate that has been signed by Workload Identity Manager, so we need to download the Workload Identity Manager root certificate, put it in a Secret in the cluster, and then tell Istio CSR to mount that Secret and use the CA certificate contained within it.

To download the Workload Identity Manager root certificate from Certificate Manager - SaaS:

1. Click Configurations > Issuer Configurations.
2. Click the link in the Sub CA Provider column of the "Workload Identity Manager Configurations" page.
3. Click the link in the CA Account column of the "Workload Identity Manager Sub CA Providers" page.
4. Click Download chain > Root certificate first.

Copy the root CA certificate into a Secret resource in the venafi project (namespace) of your OpenShift cluster:

oc create secret generic firefly-root-ca \
    --namespace=venafi \
    --from-file=ca.crt=firefly-root-ca.pem

Step 7: Install cert-manager, Istio CSR and Workload Identity Manager¶

1. Create a manifest venafi-components.yaml to configure the installation of cert-manager, Istio CSR, and Workload Identity Manager. You can use one of the samples below:

   CyberArk RegistryOwn Registry

   venafi-components.yaml

   apiVersion: installer.venafi.com/v1alpha1
   kind: VenafiInstall
   metadata:
     name: venafi-components
   spec:
     globals:
       enableDefaultApprover: true # (1)!
       imagePullSecretNames: [venafi-image-pull-secret]
       namespace: venafi
       vcpRegion: US
       region: US
     certManager:
       install: true
     firefly:
       install: true
       acceptTOS: true
       clientID: ${FIREFLY_SERVICE_ACCOUNT_CLIENT_ID} # (2)!
       values:
         deployment:
           config:
             controller:
               enabled: true
               certManager:
                 caRootChainPopulation: true
                 checkApproval: true
     venafiConnection:
       install: true
     certManagerIstioCSR:
       install: true
       trustDomain: cluster.local # (3)!
       runtimeConfigMapName: istio-csr-ca # (4)!
       values:
         app:
           certmanager:
             preserveCertificateRequests: true
             additionalAnnotations:
             - name: firefly.venafi.com/policy-name
               value: ${FIREFLY_POLICY_NAME}
   
           controller:
             configmapNamespaceSelector: "maistra.io/member-of=istio-system"
   
           istio:
             namespace: istio-system
             revisions: ["basic"]
   
           tls:
             rootCAFile: /firefly-root-ca/ca.crt
   
         volumeMounts:
           - name: firefly-root-ca
             mountPath: /firefly-root-ca
   
         volumes:
           - name: firefly-root-ca
             secret:
               secretName: firefly-root-ca
   

   1. Istio CSR works best with the cert-manager default approver.
   2. Replace with the client ID of the Workload Identity Manager service account.
   3. Provide the name of the Istio trust domain.
   4. This is the default name of the ConfigMap from which Istio CSR loads runtime configuration. You will create this ConfigMap in the next steps, in the venafi namespace, where Istio CSR is deployed.

   Regional registries

   The example above uses the CyberArk US registry parameters. If you want to use a different CyberArk registry replace vcpRegion: US and region: US with the relevant regional repository value. vcpRegion EU and vcpRegion UK users should use region: EU. vcpRegion US, vcpRegion CA, vcpRegion AU, and vcpRegion SG user should use region US. See the examples below:

   • EU registry

     venafi-components.yaml

     ...
     spec:
       globals:
         ...
         vcpRegion: EU
         region: EU
         ...
     

   • UK registry

     venafi-components.yaml

     ...
     spec:
       globals:
         ...
         vcpRegion: UK
         region: EU
         ...
     

   • Australia registry

     venafi-components.yaml

     ...
     spec:
       globals:
         ...
         vcpRegion: AU
         region: US
         ...
     

   • Canada registry

     venafi-components.yaml

     ...
     spec:
       globals:
         ...
         vcpRegion: CA
         region: US
         ...
     

   • Singapore registry

     venafi-components.yaml

     ...
     spec:
       globals:
         ...
         vcpRegion: SG
         region: US
         ...
     

   For more information on CyberArk OCI registries, see Configuring access to a CyberArk OCI Registry.

   venafi-components.yaml

   apiVersion: installer.venafi.com/v1alpha1
   kind: VenafiInstall
   metadata:
     name: venafi-components
   spec:
     globals:
       enableDefaultApprover: true # (1)!
       imagePullSecretNames: [venafi-image-pull-secret]
       namespace: venafi
     certManager:
       install: true
     firefly:
       install: true
       acceptTOS: true
       clientID: ${FIREFLY_SERVICE_ACCOUNT_CLIENT_ID} # (2)!
       values:
         deployment:
           config:
             controller:
               enabled: true
               certManager:
                 caRootChainPopulation: true
                 checkApproval: true
     venafiConnection:
       install: true
     certManagerIstioCSR:
       install: true
       trustDomain: cluster.local # (3)!
       runtimeConfigMapName: istio-csr-ca # (4)!
       values:
         app:
           certmanager:
             preserveCertificateRequests: true
             additionalAnnotations:
             - name: firefly.venafi.com/policy-name
               value: ${FIREFLY_POLICY_NAME}
   
           controller:
             configmapNamespaceSelector: "maistra.io/member-of=istio-system"
   
           istio:
             namespace: istio-system
             revisions: ["basic"]
   
           tls:
             rootCAFile: /firefly-root-ca/ca.crt
   
         volumeMounts:
           - name: firefly-root-ca
             mountPath: /firefly-root-ca
   
         volumes:
           - name: firefly-root-ca
             secret:
               secretName: firefly-root-ca
   

   1. Istio CSR works best with the cert-manager default approver.
   2. Replace with the client ID of the Workload Identity Manager service account.
   3. Provide the name of the Istio trust domain.
   4. This is the default name of the ConfigMap from which Istio CSR loads runtime configuration. You will create this ConfigMap in the next steps, in the venafi namespace, where Istio CSR is deployed.

   Notes

   Use the trustDomain field to specify the Istio trust domain.

   Tip

   For a complete list of CyberArk Certificate Manager Operator for Red Hat OpenShift configuration parameters, see the CyberArk Certificate Manager Operator API reference page.

2. Apply the manifest:

   export FIREFLY_SERVICE_ACCOUNT_CLIENT_ID=<REPLACE_ME>
   export FIREFLY_POLICY_NAME=<REPLACE_ME>
   oc apply -f <(envsubst < venafi-components.yaml)
   

   Info

   We use envsubst to substitute some Workload Identity Manager configuration in the YAML template. Alternatively, you could edit the YAML directly and then apply it.

3. Monitor the progress of the installation.

   You can monitor the actions that are being performed by the CyberArk Certificate Manager Operator, as follows:

   oc describe VenafiInstall -namespace venafi
   

   oc logs deploy/vcp-operator --follow --namespace venafi
   

   oc get events --watch --namespace venafi
   

Step 8: Verify the installation¶

1. Verify whether cert-manager, Istio CSR and Workload Identity Manager are successfully installed by running the following command:

   oc get venafiinstall,pods --namespace venafi
   

   Sample output:

   NAME                                                   STATUS   LAST SYNC
   venafiinstall.installer.venafi.com/venafi-components   Synced   103s
   
   NAME                                           READY   STATUS    RESTARTS   AGE
   pod/cert-manager-bf4fdfdbb-cvnqb               1/1     Running   0          4m13s
   pod/cert-manager-cainjector-57b7b6f978-mtw5n   1/1     Running   0          4m13s
   pod/cert-manager-istio-csr-b446698f7-xtl59     1/1     Running   0          118s
   pod/cert-manager-webhook-857699b5bc-9fmdj      1/1     Running   0          4m13s
   pod/firefly-588458dcbd-pb7r6                   1/1     Running   0          119s
   pod/firefly-588458dcbd-xxw4n                   1/1     Running   0          119s
   

Step 9: Create an Istio CSR "runtime configuration"¶

Istio CSR need some more "runtime configuration" to know what issuerRef to use for the Certificate and CertificateRequests that it generates..

1. Create the following ConfigMap:

   istio-ca.yaml

   # The runtime configuration ConfigMap that tells Istio CSR which issuer to use.
   # This is installed in the **venafi** namespace.
   # The name must match the name given in the VenafiInstall resource earlier.
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: istio-csr-ca
     namespace: venafi
   data:
     issuer-name: firefly
     issuer-kind: Issuer
     issuer-group: firefly.venafi.com
   

2. Apply the manifest by running the following command:

   oc apply -f istio-ca.yaml
   

   After you apply the ConfigMap, Istio CSR will create a cert-manager Certificate referencing the CyberArk Workload Identity Manager issuer, which cert-manager will reconcile and place the resulting leaf certificate in a Secret called istiod-tls. istiod in the istio-system namespace will use that signed certificate for its gRPC server.

3. Monitor the Istio CSR controller:

   If you want to understand what actions are being performed by Istio CSR, you can monitor its logs, as follows:

   oc logs deploy/cert-manager-istio-csr --follow --namespace venafi
   

4. Verify that the CA certificate has been created:

   oc -n istio-system get cert-manager,configmap,secret
   

   Sample output:

   NAME                                                  APPROVED   DENIED   READY   ISSUER    REQUESTER                                   AGE
   certificaterequest.cert-manager.io/istiod-dynamic-1   True                True    firefly   system:serviceaccount:venafi:cert-manager   3m20s
   
   NAME                                         READY   SECRET       AGE
   certificate.cert-manager.io/istiod-dynamic   True    istiod-tls   3m21s
   
   NAME                                 DATA   AGE
   configmap/kube-root-ca.crt           1      15m
   configmap/openshift-service-ca.crt   1      15m
   
   NAME                              TYPE                      DATA   AGE
   secret/builder-dockercfg-cgfvn    kubernetes.io/dockercfg   1      15m
   secret/default-dockercfg-h5x89    kubernetes.io/dockercfg   1      15m
   secret/deployer-dockercfg-r9t4q   kubernetes.io/dockercfg   1      15m
   secret/istiod-tls                 kubernetes.io/tls         2      3m20s
   

5. Examine the resulting istiod-tls certificate.

   The resulting istiod serving certificate should have been signed by your Workload Identity Manager Sub CA. You can check this using the following command:

   oc get secret istiod-tls -n istio-system -o jsonpath='{.data.tls\.crt}' \
   | base64 -d  \
   | openssl x509 -noout -text
   

   Certificate:
       Data:
           Version: 3 (0x2)
           Serial Number:
               33:fb:0b:06:8c:c0:c5:48:70:08:ac:40:35:90:2a:bb
           Signature Algorithm: sha256WithRSAEncryption
           Issuer: C = GB, ST = England, L = Bristol, O = Venafi, OU = Team cert-manager, CN = firefly-1-20241217171011 firefly-richardw-1
           Validity
               Not Before: Dec 17 17:18:49 2024 GMT
               Not After : Mar 17 17:18:49 2025 GMT
           Subject: CN = istiod.istio-system.svc
   ...
           X509v3 Subject Alternative Name:
               DNS:istiod-basic.istio-system.svc, URI:spiffe://cluster.local/ns/istio-system/sa/istiod-service-account
   ...
   

   The certificate should also have a DNS name for an Istio revision named basic. The revision name basic represents the name of the ServiceMeshControlPlane (SCMP) which you will deploy in the next step.

   Important

   If you have previously deployed Istio and later deleted it, some resources may have been left behind in the istio-system namespace, which may interfere with Istio CSR. In particular, you should delete the following Secret resources which may contain stale CA certificates:

   • cacerts
   • istio-ca-secret

   You can see the priority of loading of CA certificates in the istio-operator webhook CA source code.

   Info

   Read 2.3.4. Upgrading the control plane in the OpenShift documentation which has this to say about the use of revision for supporting multiple service meshes in a cluster,rather than for supporting canary upgrades:

   Although you can deploy multiple versions of the control plane in the same cluster, Red Hat OpenShift Service Mesh does not support canary upgrades of the service mesh. That is, you can have different SCMP resources with different values for spec.version, but they cannot be managing the same mesh.

Step 10: Deploy the Service Mesh Control Plane¶

The ServiceMeshControlPlane object creates an istiod control plane, you use this to configure your service-mesh to use cert-manager Istio CSR to sign certificates.

1. Create a ServiceMeshControlPlane

   istio-installation.yaml

   apiVersion: maistra.io/v2
   kind: ServiceMeshControlPlane
   metadata:
     name: basic
     namespace: istio-system
   spec:
     version: v2.6
     addons:
       grafana:
         enabled: false
       prometheus:
         enabled: false
       kiali:
         enabled: false
     gateways:
       enabled: false
     security:
       certificateAuthority:
         cert-manager:
           address: cert-manager-istio-csr.venafi.svc:443
         type: cert-manager
       dataPlane:
         mtls: true
       identity:
         type: ThirdParty
     tracing:
       type: None
   ---
   apiVersion: maistra.io/v1
   kind: ServiceMeshMemberRoll
   metadata:
     name: default
     namespace: istio-system
   spec:
     members:
     - test-project-1
   

   oc apply -f istio-installation.yaml
   

   Important

   If you have previously deployed Istio and later deleted it, some resources may have been left behind in the istio-system namespace, which may interfere with Istio CSR. In particular, you should delete the following Secret resources which may contain stale CA certificates:

   • cacerts
   • istio-ca-secret

   You can see the priority of loading of CA certificates in the istio-operator webhook CA source code.

   Info

   We disabled all addons and gateways, to reduce the resource requirements for the Istio control plane, which allows us to verify this tutorial on a single node CRC (Code Ready Containers) cluster. In production, you may want to enable the add-ons and gateways.

   Info

   The ServiceMeshMemberRoll contains a reference to the test-project-1 project (namespace) which we will create in the next step. This ensures that the maistra.io/member-of: istio-system label will be added to the namespace when we create it.

Step 11: Verify mTLS¶

1. First, create a new project to deploy some sample applications:

   oc new-project test-project-1
   

2. Verify that the namespace has been labelled.

   The ServiceMeshMemberRoll which we created in the previous step, contains a reference to the test-project-1 project (namespace) which tells the OpenShift Service Mesh Operator to add the maistra.io/member-of: istio-system label to the namespace. You can see this by using the following command:

   oc get ns test-project-1 -o yaml
   

3. Verify that the Istio CA has been added to the namespace.

   The presence of the maistra.io/member-of: istio-system label on the namespace, tells Istio CSR to create an istio-ca-root-cert ConfigMap in the namespace. You can see this as follows:

   oc get cm istio-ca-root-cert -o yaml --namespace test-project-1
   

   If you're following the Istio CSR logs, you'll also see a message like this:

   I1219 14:18:44.090159       1 configmap.go:188] "creating configmap with root CA data" logger="controller.configmap" namespace="test-project-1" configmap="istio-ca-root-cert"
   

4. Deploy a sample server and a sample client:

   oc apply -n test-project-1 -f https://raw.githubusercontent.com/maistra/istio/maistra-2.4/samples/httpbin/httpbin.yaml
   

   oc apply -n test-project-1 -f https://raw.githubusercontent.com/maistra/istio/maistra-2.4/samples/sleep/sleep.yaml
   

5. Verify that the server is using an Workload Identity Manager issued certificate.

   In these examples, we exec into the Istio side-car container of the sleep Pod, and run the openssl tool to examine the TLS connection to the httpbin server:

   oc exec deploy/sleep -n test-project-1 -c istio-proxy \
   -- openssl s_client -CAfile /var/run/secrets/istio/root-cert.pem httpbin:8000
   

   oc exec deploy/sleep -n test-project-1 -c istio-proxy \
   -- openssl s_client -CAfile /var/run/secrets/istio/root-cert.pem  httpbin:8000 \
   | openssl x509 -noout -text
   

   You should see the details of the serving certificate returned by the istio-proxy of the httpbin server, and you should see that it has been signed by our Workload Identity Manager Sub CA. Here's an example of the output:

   CONNECTED(00000003)
   ---
   Certificate chain
    0 s:O =
      i:C = GB, ST = England, L = Bristol, O = Venafi, OU = Team cert-manager, CN = firefly-1-20241219114107 firefly-richardw-1
      a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
      v:NotBefore: Dec 19 15:40:34 2024 GMT; NotAfter: Mar 19 15:40:34 2025 GMT
    1 s:C = GB, ST = England, L = Bristol, O = Venafi, OU = Team cert-manager, CN = firefly-1-20241219114107 firefly-richardw-1
      i:C = US, O = "Venafi, Inc.", OU = Built-in, CN = Dedicated - Venafi Cloud Built-In Intermediate CA - G1
      a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
      v:NotBefore: Dec 19 11:40:46 2024 GMT; NotAfter: Mar 19 11:41:16 2025 GMT
    2 s:C = US, O = "Venafi, Inc.", OU = Built-in, CN = Dedicated - Venafi Cloud Built-In Intermediate CA - G1
      i:C = US, O = "Venafi, Inc.", OU = Built-in, CN = Dedicated - Venafi Cloud Built-In CA
      a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
      v:NotBefore: Jul 21 21:34:37 2023 GMT; NotAfter: Jul 19 21:35:07 2028 GMT
    3 s:C = US, O = "Venafi, Inc.", OU = Built-in, CN = Dedicated - Venafi Cloud Built-In CA
      i:C = US, O = "Venafi, Inc.", OU = Built-in, CN = Dedicated - Venafi Cloud Built-In CA
      a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
      v:NotBefore: Jul 21 21:34:36 2023 GMT; NotAfter: Jul 18 21:35:06 2033 GMT
    4 s:C = US, O = "Venafi, Inc.", OU = Built-in, CN = Dedicated - Venafi Cloud Built-In Intermediate CA - G1
      i:C = US, O = "Venafi, Inc.", OU = Built-in, CN = Dedicated - Venafi Cloud Built-In CA
      a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
      v:NotBefore: Jul 21 21:34:37 2023 GMT; NotAfter: Jul 19 21:35:07 2028 GMT