Venafi Kubernetes components deployment best practices¶

Learn how to configure the Venafi Kubernetes components for deployment in production environments.

Introduction¶

You will learn how to use Helm chart values to override settings such as the number of replicas in each Deployment and the resource requests for CPU and memory.

Discover methods to ensure the high availability of the Venafi Kubernetes components, whilst avoiding inefficient over-allocation of your cluster resources.

Isolate Venafi Kubernetes components on a dedicated Node pool¶

Venafi Kubernetes components should be treated as part of your platform's control plane. Some Venafi Kubernetes components, such as cert-manager, create and modify Kubernetes Secret resources and most of the components cache TLS Secret resources in memory.

If an untrusted or malicious workload runs on the same Node as the Venafi Kubernetes components, and gains root access to the underlying node, it may be able to read the Secrets that the controller has cached in memory. You can mitigate this risk by running Venafi Kubernetes components on nodes that are reserved for trusted platform operators.

The Helm charts for Venafi Kubernetes components have parameters to configure the Pod tolerations and Node affinity (or nodeSelector) for each component. The exact values of these parameters will depend on your particular cluster.

Example¶

This example demonstrates how to use:

• taints to repel non-platform Pods from Nodes which you have reserved for your platform's control-plane.
• tolerations to allow cert-manager Pods to run on those Nodes.
• nodeSelector to place the cert-manager Pods on those Nodes.

To isolate Venafi Kubernetes components on a dedicated Node pool:

1. Label the Nodes:

   kubectl label node ... node-restriction.kubernetes.io/reserved-for=platform
   

2. Taint the Nodes:

   kubectl taint node ... node-restriction.kubernetes.io/reserved-for=platform:NoExecute
   

3. Then install cert-manager using the following Helm chart values:

   nodeSelector:
     kubernetes.io/os: linux
     node-restriction.kubernetes.io/reserved-for: platform
   tolerations:
   - key: node-restriction.kubernetes.io/reserved-for
     operator: Equal
     value: platform
   
   webhook:
     nodeSelector:
       kubernetes.io/os: linux
       node-restriction.kubernetes.io/reserved-for: platform
     tolerations:
     - key: node-restriction.kubernetes.io/reserved-for
       operator: Equal
       value: platform
   
   cainjector:
     nodeSelector:
       kubernetes.io/os: linux
       node-restriction.kubernetes.io/reserved-for: platform
     tolerations:
     - key: node-restriction.kubernetes.io/reserved-for
       operator: Equal
       value: platform
   
   startupapicheck:
     nodeSelector:
       kubernetes.io/os: linux
       node-restriction.kubernetes.io/reserved-for: platform
     tolerations:
     - key: node-restriction.kubernetes.io/reserved-for
       operator: Equal
       value: platform
   

High Availability¶

This section is about increasing the Venafi Kubernetes resilience to voluntary and involuntary disruptions.

For example, as a platform administrator, you need to periodically drain and decommission a Kubernetes node without causing any downtime to the Venafi Kubernetes services or the applications relying on the Venafi Kubernetes services.

As a platform administrator, you need to be confident that the Venafi Kubernetes components can be deployed in such a way that they are resilient to a Kubernetes node failure.

Replicas¶

Use two or more replicas to achieve high availability.

Each of the Venafi Kubernetes components has one or more Kubernetes Deployment resources. By default each Deployment has one replica. The advantage of using one replica by default, is that you can easily evaluate the components on a single node cluster; on a kind or a minikube cluster, for example. The disadvantage is that one replica does not provide high availability. The Helm charts for Venafi Kubernetes components have parameters to configure the replicas for each controller, and in production you should use two or more replicas to achieve high availability.

Most of the Venafi Kubernetes components use leader election to ensure that only one replica is active. This prevents conflicts that would arise if multiple controllers were reconciling the same API resources. So in general multiple replicas can be used for high availability but not load balancing.

You don't need to use more than two replicas of a controller Pod. Using two replicas for each Deployment ensures that there is a standby Pod scheduled to a Node and ready to take leadership, if the current leader encounters a disruption. For example, a voluntary disruption, such as the draining of the Node on which the leader Pod is running, or an involuntary disruption, such as the leader Pod encountering an unexpected deadlock. Further replicas may add a degree of resilience if you have the luxury of sufficient Nodes with sufficient CPU and memory to accommodate standby replicas.

PodDisruptionBudget¶

For high availability you should also deploy a PodDisruptionBudget resource, with minAvailable=1 or with maxUnavailable=1. This ensures that a voluntary disruption, such as the draining of a Node, can not proceed until at least one other replica has been successfully scheduled and started on another Node. The Helm charts for Venafi Kubernetes components have parameters to enable and configure a PodDisruptionBudget.

Topology Spread Constraints¶

Consider using Topology Spread Constraints, to ensure that a disruption of a node or data center does not degrade the operation of cert-manager.

For high availability, you don't want the replica Pods to be scheduled on the same Node, because if that node fails, both the active and standby Pods will exit. There will be no further reconciliation of the resources by that controller until there is another Node with enough free resources to run a new Pod, and until that Pod has become Ready.

It is also desirable for the replica Pods to run in separate data centers (availability zones), if the cluster has nodes distributed between zones. Then, in the event of a failure at the data center hosting the active Pod, the standby Pod will immediately be available to take leadership.

Fortunately, you may not need to do anything to achieve these goals because Kubernetes > 1.24 has Built-in default constraints which should mean that the high availability scheduling described above will happen implicitly. But in case your cluster does not have those defaults, each of the Venafi Kubernetes component Helm charts have parameters to explicitly add Topology Spread Constraints to the Pods.

Liveness Probes¶

Some Venafi Kubernetes components have liveness probes. Those components that have liveness probes have sensible defaults for the following options:

• initialDelaySeconds
• periodSeconds
• timeoutSeconds
• successThreshold
• failureThreshold

If necessary, you can override the defaults by supplying Helm chart values.

Venafi may add liveness probes to the other components in the future, but only where there is a clear benefit. This cautious approach to liveness probes is guided by the following paragraph in the Kubernetes documentation Configure Liveness, Readiness and Startup Probes:

Liveness probes can be a powerful way to recover from application failures, but they should be used with caution. Liveness probes must be configured carefully to ensure that they truly indicate unrecoverable application failure, for example a deadlock

Priority Class¶

The reason for setting a priority class is summarized as follows in the Kubernetes blog Protect Your Mission-Critical Pods From Eviction With PriorityClass:

Pod priority and preemption help to make sure that mission-critical pods are up in the event of a resource crunch by deciding order of scheduling and eviction.

If the Venafi Kubernetes components are mission-critical to your platform, then set a priorityClassName on the Pods to protect them from preemption, in situations where a Kubernetes node becomes starved of resources. Without a priorityClassName the Pods of the Venafi Kubernetes components may be evicted to free up resources for other workloads, and this may cause disruption to any applications that rely on them.

Most Kubernetes clusters will come with two built-in priority class names: system-cluster-critical and system-node-critical, which are used for Kubernetes core components. These can also be used for critical add-ons, such as the Venafi Kubernetes components.

Here are the Helm chart values you can use for cert-manager, for example:

global:
  priorityClassName: system-cluster-critical

The Venafi Kubernetes components allow you to set the priorityClassName using Helm chart values, and the Venafi Kubernetes Manifest utility makes it easy to set the values for multiple components.

On some clusters the ResourceQuota admission controller may be configured to limit the use of certain priority classes to certain namespaces. For example, Google Kubernetes Engine (GKE) will only allow priorityClassName: system-cluster-critical for Pods in the kube-system namespace, by default.

In such cases you will need to create a ResourceQuota in the venafi namespace:

# venafi-resourcequota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: venafi-critical-pods
  namespace: venafi
spec:
  hard:
    pods: 1G
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
      - system-node-critical
      - system-cluster-critical

kubectl apply -f venafi-resourcequota.yaml

If you are using CSI driver or CSI driver for SPIFFE it is essential to set priorityClassName: system-node-critical. These components are deployed using DaemonSet resources and must run on the same nodes as the general applications on your platform (as opposed to components like cert-manager which should run on isolated nodes). By assigning priorityClassName: system-node-critical to these CSI driver pods, you can ensure that they have a higher scheduling priority compared to other general application pods.

Setting default values for HA deployments using the Venafi CLI tool¶

To generate HA values for any of the supported components (see the example that follows), and include them in a Venafi manifest, use the --ha-values-dir flag for venctl components kubernetes manifest generate:

venctl components kubernetes manifest generate \
   --namespace venafi \
   --cert-manager \
   --approver-policy-enterprise \
   --csi-driver \
   --trust-manager \
   --venafi-connection \
   --venafi-enhanced-issuer \
   --venafi-kubernetes-agent \
   --ha-values-dir venafi-values > venafi-components.yaml

Using the --ha-values-dir flag will overwrite any previously generated values. If you want to change any of the Venafi HA values, you can pass your custom values as with previous versions of the Venafi CLI tool - your customizations will take precedence.

For example, if you wish to increase the memory request for the cert-manager controller:

cat cert-manager-controller-memory.values.yaml
resources:
  requests:
    memory: 1Gi

venctl components kubernetes manifest generate \
   --namespace venafi \
   --cert-manager \
   --cert-manager-values-files cert-manager-controller-memory.values.yaml \
   --approver-policy-enterprise \
   --csi-driver \
   --trust-manager \
   --venafi-connection \
   --venafi-enhanced-issuer \
   --venafi-kubernetes-agent \
   --ha-values-dir venafi-values > venafi-components.yaml

Scaling¶

This section explains how to accurately allocate CPU and memory resources to match the actual usage needs of the Venafi Kubernetes applications running in your Kubernetes environment. This is sometimes referred to as right sizing.

This is important for the following reasons:

• Efficiency: Matching the resource requests with the actual usage helps prevent the wastage of resources.
• Performance: Using appropriate resource requests will improve the performance of the components.
• Reliability: Right sizing improves reliability by avoiding crashes caused by memory exhaustion, for example.

However, right sizing is a continuous process that requires ongoing monitoring, adjustment, and optimization.

The Helm charts for the Venafi Kubernetes components do not all include resource requests and limits, so you should supply resource requests and limits that are appropriate for your cluster.

What are appropriate resource requests and limits? This depends on factors such as the size of your cluster and the number and nature of the workloads that run on the cluster.

Memory¶

Use vertical scaling to assign sufficient memory resources to Venafi Kubernetes components. The memory requirements will be higher on clusters with very many API resources or with large API resources. This is because each of the components reconciles one or more Kubernetes API resources, and each component will cache the metadata and sometimes the entire resource in memory, to reduce the load on your Kubernetes API server.

For example, if the cluster contains very many CertificateRequest resources, you will need to increase the memory limit of the Venafi Enhanced Issuer Pod. The default memory limit of Venafi Enhanced Issuer is 128MiB, which is sufficient to cache ~3000 CertificateRequest resources.

You can use tools like Kubernetes Resource Recommender, to measure the typical memory usage of the Venafi Kubernetes components in your cluster and to help you choose appropriate memory limits.

CPU¶

Use vertical scaling to assign sufficient CPU resources to the Venafi Kubernetes components. The CPU requirements will be higher on clusters where there are very frequent updates to the resources which are reconciled by the components. Whenever a resource changes, it will be queued to be re-reconciled by the component. Higher CPU resources allow the component to process the queue faster.

You can use tools like Kubernetes Resource Recommender, to measure the typical CPU usage of the Venafi Kubernetes components in your cluster and to help you choose appropriate CPU requests.

Next Steps¶

• Helm-based installation methods explains how to install and configure the Helm charts of the Venafi Kubernetes components.
• Configuring Venafi Kubernetes Manifest tool explains how to configure Venafi Kubernetes components using the Venafi Kubernetes Manifest utility.
• Approver Policy supported Helmfile values reference
• Approver Policy Enterprise supported Helmfile values reference
• cert-manager supported Helmfile values reference
• CSI Driver supported Helmfile values reference
• Trust Manager supported Helmfile values reference
• Venafi Enhanced Issuer supported Helmfile values reference