Venafi Kubernetes components deployment best practices¶

Learn how to configure the Venafi Kubernetes components for deployment in production environments.

Introduction¶

You will learn how to use Helm chart values to override settings such as the number of replicas in each Deployment and the resource requests for CPU and memory.

Discover methods to ensure the high availability of the Venafi Kubernetes components, whilst avoiding inefficient over-allocation of your cluster resources.

Isolate Venafi Kubernetes components on a dedicated Node pool¶

Venafi Kubernetes components should be treated as part of your platform's control plane. Some Venafi Kubernetes components, such as cert-manager, create and modify Kubernetes Secret resources and most of the components cache TLS Secret resources in memory.

If an untrusted or malicious workload runs on the same Node as the Venafi Kubernetes components, and gains root access to the underlying node, it may be able to read the Secrets that the controller has cached in memory. You can mitigate this risk by running Venafi Kubernetes components on nodes that are reserved for trusted platform operators.

The Helm charts for Venafi Kubernetes components have parameters to configure the Pod tolerations and Node affinity (or nodeSelector) for each component. The exact values of these parameters will depend on your particular cluster.

Example¶

This example demonstrates how to use:

• taints to repel non-platform Pods from Nodes which you have reserved for your platform's control-plane.
• tolerations to allow cert-manager Pods to run on those Nodes.
• nodeSelector to place the cert-manager Pods on those Nodes.

To isolate Venafi Kubernetes components on a dedicated Node pool:

1. Label the Nodes:

   kubectl label node ... node-restriction.kubernetes.io/reserved-for=platform
   

2. Taint the Nodes:

   kubectl taint node ... node-restriction.kubernetes.io/reserved-for=platform:NoExecute
   

3. Then install cert-manager using the following Helm chart values:

   nodeSelector:
     kubernetes.io/os: linux
     node-restriction.kubernetes.io/reserved-for: platform
   tolerations:
   - key: node-restriction.kubernetes.io/reserved-for
     operator: Equal
     value: platform
   
   webhook:
     nodeSelector:
       kubernetes.io/os: linux
       node-restriction.kubernetes.io/reserved-for: platform
     tolerations:
     - key: node-restriction.kubernetes.io/reserved-for
       operator: Equal
       value: platform
   
   cainjector:
     nodeSelector:
       kubernetes.io/os: linux
       node-restriction.kubernetes.io/reserved-for: platform
     tolerations:
     - key: node-restriction.kubernetes.io/reserved-for
       operator: Equal
       value: platform
   
   startupapicheck:
     nodeSelector:
       kubernetes.io/os: linux
       node-restriction.kubernetes.io/reserved-for: platform
     tolerations:
     - key: node-restriction.kubernetes.io/reserved-for
       operator: Equal
       value: platform
   

High Availability¶

This section is about increasing the Venafi Kubernetes resilience to voluntary and involuntary disruptions.

For example, as a platform administrator, you need to periodically drain and decommission a Kubernetes node without causing any downtime to the Venafi Kubernetes services or the applications relying on the Venafi Kubernetes services.

As a platform administrator, you need to be confident that the Venafi Kubernetes components can be deployed in such a way that they are resilient to a Kubernetes node failure.

Replicas¶

Use two or more replicas to achieve high availability.

Each of the Venafi Kubernetes components has one or more Kubernetes Deployment resources. By default each Deployment has one replica. The advantage of using one replica by default, is that you can easily evaluate the components on a single node cluster; on a kind or a minikube cluster, for example. The disadvantage is that one replica does not provide high availability. The Helm charts for Venafi Kubernetes components have parameters to configure the replicas for each controller, and in production you should use two or more replicas to achieve high availability.

Most of the Venafi Kubernetes components use leader election to ensure that only one replica is active. This prevents conflicts that would arise if multiple controllers were reconciling the same API resources. So in general multiple replicas can be used for high availability but not load balancing.

You don't need to use more than two replicas of a controller Pod. Using two replicas for each Deployment ensures that there is a standby Pod scheduled to a Node and ready to take leadership, if the current leader encounters a disruption. For example, a voluntary disruption, such as the draining of the Node on which the leader Pod is running, or an involuntary disruption, such as the leader Pod encountering an unexpected deadlock. Further replicas may add a degree of resilience if you have the luxury of sufficient Nodes with sufficient CPU and memory to accommodate standby replicas.

PodDisruptionBudget¶

For high availability you should also deploy a PodDisruptionBudget resource, with minAvailable=1 or with maxUnavailable=1. This ensures that a voluntary disruption, such as the draining of a Node, can not proceed until at least one other replica has been successfully scheduled and started on another Node. The Helm charts for Venafi Kubernetes components have parameters to enable and configure a PodDisruptionBudget.

Topology Spread Constraints¶

Consider using Topology Spread Constraints, to ensure that a disruption of a node or data center does not degrade the operation of cert-manager.

For high availability, you don't want the replica Pods to be scheduled on the same Node, because if that node fails, both the active and standby Pods will exit. There will be no further reconciliation of the resources by that controller until there is another Node with enough free resources to run a new Pod, and until that Pod has become Ready.

It is also desirable for the replica Pods to run in separate data centers (availability zones), if the cluster has nodes distributed between zones. Then, in the event of a failure at the data center hosting the active Pod, the standby Pod will immediately be available to take leadership.

Fortunately, you may not need to do anything to achieve these goals because Kubernetes > 1.24 has Built-in default constraints which should mean that the high availability scheduling described above will happen implicitly. But in case your cluster does not have those defaults, each of the Venafi Kubernetes component Helm charts have parameters to explicitly add Topology Spread Constraints to the Pods.

Liveness Probes¶

Some Venafi Kubernetes components have liveness probes. Those components that have liveness probes have sensible defaults for the following options:

• initialDelaySeconds
• periodSeconds
• timeoutSeconds
• successThreshold
• failureThreshold

If necessary, you can override the defaults by supplying Helm chart values.

Venafi may add liveness probes to the other components in the future, but only where there is a clear benefit. This cautious approach to liveness probes is guided by the following paragraph in the Kubernetes documentation Configure Liveness, Readiness and Startup Probes:

Liveness probes can be a powerful way to recover from application failures, but they should be used with caution. Liveness probes must be configured carefully to ensure that they truly indicate unrecoverable application failure, for example a deadlock

Priority Class¶

The reason for setting a priority class is summarized as follows in the Kubernetes blog Protect Your Mission-Critical Pods From Eviction With PriorityClass:

Pod priority and preemption help to make sure that mission-critical pods are up in the event of a resource crunch by deciding order of scheduling and eviction.

If the Venafi Kubernetes components are mission-critical to your platform, then set a priorityClassName on the Pods to protect them from preemption, in situations where a Kubernetes node becomes starved of resources. Without a priorityClassName the Pods of the Venafi Kubernetes components may be evicted to free up resources for other workloads, and this may cause disruption to any applications that rely on them.

Most Kubernetes clusters will come with two built-in priority class names: system-cluster-critical and system-node-critical, which are used for Kubernetes core components. These can also be used for critical add-ons, such as the Venafi Kubernetes components.

Here are the Helm chart values you can use for cert-manager, for example:

global:
  priorityClassName: system-cluster-critical

The Venafi Kubernetes components allow you to set the priorityClassName using Helm chart values, and the Venafi Kubernetes Manifest utility makes it easy to set the values for multiple components.

On some clusters the ResourceQuota admission controller may be configured to limit the use of certain priority classes to certain namespaces. For example, Google Kubernetes Engine (GKE) will only allow priorityClassName: system-cluster-critical for Pods in the kube-system namespace, by default.

In such cases you will need to create a ResourceQuota in the venafi namespace:

# venafi-resourcequota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: venafi-critical-pods
  namespace: venafi
spec:
  hard:
    pods: 1G
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
      - system-node-critical
      - system-cluster-critical

kubectl apply -f venafi-resourcequota.yaml

If you are using CSI driver or CSI driver for SPIFFE it is essential to set priorityClassName: system-node-critical. These components are deployed using DaemonSet resources and must run on the same nodes as the general applications on your platform (as opposed to components like cert-manager which should run on isolated nodes). By assigning priorityClassName: system-node-critical to these CSI driver pods, you can ensure that they have a higher scheduling priority compared to other general application pods.

Setting default values for HA deployments using the Venafi CLI tool¶

To generate HA values for any of the supported components (see the example that follows), and include them in a Venafi manifest, use the --ha-values-dir flag for venctl components kubernetes manifest generate:

venctl components kubernetes manifest generate \
   --namespace venafi \
   --cert-manager \
   --approver-policy-enterprise \
   --csi-driver \
   --trust-manager \
   --venafi-connection \
   --venafi-enhanced-issuer \
   --venafi-kubernetes-agent \
   --ha-values-dir venafi-values > venafi-components.yaml

Using the --ha-values-dir flag will overwrite any previously generated values. If you want to change any of the Venafi HA values, you can pass your custom values as with previous versions of the Venafi CLI tool - your customizations will take precedence.

For example, if you wish to increase the memory request for the cert-manager controller:

cat cert-manager-controller-memory.values.yaml
resources:
  requests:
    memory: 1Gi

venctl components kubernetes manifest generate \
   --namespace venafi \
   --cert-manager \
   --cert-manager-values-files cert-manager-controller-memory.values.yaml \
   --approver-policy-enterprise \
   --csi-driver \
   --trust-manager \
   --venafi-connection \
   --venafi-enhanced-issuer \
   --venafi-kubernetes-agent \
   --ha-values-dir venafi-values > venafi-components.yaml

Scaling¶

This section explains how to accurately allocate CPU and memory resources to match the actual usage needs of the Venafi Kubernetes applications running in your Kubernetes environment. This is sometimes referred to as right sizing.

This is important for the following reasons:

• Efficiency: Matching the resource requests with the actual usage helps prevent the wastage of resources.
• Performance: Using appropriate resource requests will improve the performance of the components.
• Reliability: Right sizing improves reliability by avoiding crashes caused by memory exhaustion, for example.

However, right sizing is a continuous process that requires ongoing monitoring, adjustment, and optimization.

The Helm charts for the Venafi Kubernetes components do not all include resource requests and limits, so you should supply resource requests and limits that are appropriate for your cluster.

What are appropriate resource requests and limits? This depends on factors such as the size of your cluster and the number and nature of the workloads that run on the cluster.

Memory¶

Use vertical scaling to assign sufficient memory resources to Venafi Kubernetes components. The memory requirements will be higher on clusters with very many API resources or with large API resources. This is because each of the components reconciles one or more Kubernetes API resources, and each component will cache the metadata and sometimes the entire resource in memory, to reduce the load on your Kubernetes API server.

For example, if the cluster contains very many CertificateRequest resources, you will need to increase the memory limit of the Venafi Enhanced Issuer Pod. The default memory limit of Venafi Enhanced Issuer is 128MiB, which is sufficient to cache ~3000 CertificateRequest resources.

You can use tools like Kubernetes Resource Recommender, to measure the typical memory usage of the Venafi Kubernetes components in your cluster and to help you choose appropriate memory limits.

Benchmarks¶

The following charts and tables show the maximum memory usage of the Venafi Kubernetes components with: 1000, 5000, and 10000 Certificate resources. The private key size affects the size of the Secret resources, and larger Secret resources cause higher memory use in some components, due to in-memory caching. The measurements were collected using a Venafi benchmarking tool called tlspk-bench. The components were installed in a Kind cluster using the Venafi CLI tool, with default Helm chart values. The unit of measurement is mebibyte (MiB). You can use these measurements to choose initial memory limits for each component.

RSA 2048¶

ECDSA 256¶

CPU¶

Use vertical scaling to assign sufficient CPU resources to the Venafi Kubernetes components. The CPU requirements will be higher on clusters where there are very frequent updates to the resources which are reconciled by the components. Whenever a resource changes, it will be queued to be re-reconciled by the component. Higher CPU resources allow the component to process the queue faster.

You can use tools like Kubernetes Resource Recommender, to measure the typical CPU usage of the Venafi Kubernetes components in your cluster and to help you choose appropriate CPU requests.

Component Notes¶

cert-manager-controller¶

Cache Size¶

When Certificate resources are the dominant use-case, as for example when workloads need to mount the TLS Secret or when gateway-shim is used, the memory consumption of the cert-manager controller will be roughly proportional to the total size of those Secret resources that contain the TLS key pairs. This is because the cert-manager controller caches the entire content of these Secret resources in memory. If large TLS keys are used (e.g. RSA 4096) the memory use will be higher than if smaller TLS keys are used (e.g. ECDSA).

The other Secrets in the cluster, such as those used for Helm chart configurations or for other workloads, will not significantly increase the memory consumption, because cert-manager will only cache the metadata of these Secrets.

When CertificateRequest resources are the dominant use-case, as for example with CSI driver or with Istio CSR, the memory consumption of the cert-manager controller will be much lower, because there will be fewer TLS Secrets and fewer resources to be cached.

Disable client-side rate limiting for Kubernetes API requests¶

By default cert-manager throttles the rate of requests to the Kubernetes API server to 20 queries per second. Historically this was intended to prevent cert-manager from overwhelming the Kubernetes API server, but modern versions of Kubernetes implement API Priority and Fairness, which obviates the need for client side throttling. You can increase the threshold of the client-side rate limiter using the following helm values:

# helm-values.yaml
config:
  apiVersion: controller.config.cert-manager.io/v1alpha1
  kind: ControllerConfiguration
  kubernetesAPIQPS: 10000
  kubernetesAPIBurst: 10000

Restrict the use of large RSA keys¶

Certificates with large RSA keys cause cert-manager to use more CPU resources. When there are insufficient CPU resources, the reconcile queue length grows. This delays the reconciliation of all Certificates. A user who has permission to create a large number of RSA 4096 certificates might accidentally or maliciously cause a denial of service for other users on the cluster.

Set revisionHistoryLimit: 1 on all Certificate resources¶

By default, cert-manager will keep all the CertificateRequest resources that it creates (revisionHistoryLimit):

The maximum number of CertificateRequest revisions that are maintained in the Certificate's history. Each revision represents a single CertificateRequest created by this Certificate, either when it was created, renewed, or Spec was changed. Revisions will be removed by oldest first if the number of revisions exceeds this number. If set, revisionHistoryLimit must be a value of 1 or greater. If unset (nil), revisions will not be garbage collected. The default value is nil.

On a busy cluster these will eventually overwhelm your Kubernetes API server because of the memory and CPU required to cache them all and the storage required to save them.

Use a tool like Kyverno to override the Certificate.spec.revisionHistoryLimit for all namespaces.

Enable Server-Side Apply¶

By default, cert-manager uses Update requests to create and modify resources like CertificateRequest and Secret, but on a busy cluster there will be frequent conflicts as the control loops in cert-manager each try to update the status of various resources.

You will see errors, like this one, in the logs:

I0419 14:11:51.325377       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-trigger" key="team-864-p6ts6/app-7" error="Operation cannot be fulfilled on certificates.cert-manager.io \"app-7\": the object has been modified; please apply your changes to the latest version and try again"

This error is relatively harmless because the update attempt is retried, but it slows down the reconciliation because each error triggers an exponential back off mechanism, which causes increasing delays between retries.

The solution is to turn on the Server-Side Apply Feature, which causes cert-manager to use HTTP PATCH using Server-Side Apply when ever it needs to modify an API resource. This avoids all conflicts because each cert-manager controller sets only the fields that it owns.

You can enable the server-side apply feature gate with the following Helm chart values:

# helm-values.yaml
config:
  apiVersion: controller.config.cert-manager.io/v1alpha1
  kind: ControllerConfiguration
  featureGates:
    ServerSideApply: true

cert-manager cainjector¶

You can reduce the memory consumption of cainjector by configuring it to only watch resources in the venafi namespace, and by configuring it to not watch Certificate resources. Here's how to configure the cainjector using Helm chart values:

cainjector:
  extraArgs:
  - --namespace=venafi
  - --enable-certificates-data-source=false

Next Steps¶

• Helm-based installation methods explains how to install and configure the Helm charts of the Venafi Kubernetes components.
• Configuring Venafi Kubernetes Manifest tool explains how to configure Venafi Kubernetes components using the Venafi Kubernetes Manifest utility.
• Approver Policy supported Helmfile values reference
• Approver Policy Enterprise supported Helmfile values reference
• cert-manager supported Helmfile values reference
• CSI Driver supported Helmfile values reference
• Trust Manager supported Helmfile values reference
• Venafi Enhanced Issuer supported Helmfile values reference