Backing Up¶

To configure the desired state of a Kubernetes cluster, resources are created on that cluster. These resources can be created manually or through an external process based on an external source of truth (think GitOps).

Always use an external source of truth for storing your desired state.

Based on this desired state, other resources are automatically generated. It's sometimes difficult to figure out whether a resource was part of the original definition of the desired state or was automatically generated (for example, a certificate resource can be created manually, but it can also be generated based on annotations on an Ingress resource).

The minimum requirements for your cert-manager backup are:

1. Make sure you can restore the "desired state"

   In practice: Have a persistent external source-of-truth for your desired state (see "Ideal scenario: External system acts as source-of-truth for desired state").

   If you don't have this, create a backup of the "desired state" defined in your cluster (see "Alternative scenario: Cluster acts as source-of-truth for desired state.").

2. Backup dynamically generated resources that are costly to regenerate

   In practice: Back up cert-manager secrets that contain TLS certificates.

   If the secret resource already exists for a certificate resource and is up to date, the certificate is marked as Ready automatically, without reissuance. For more information, see the cert-manager documentation.

For more information about backing up and restoring the cert-manager component, see the cert-manager documentation.

Ideal scenario¶

An external system acts as the source-of-truth for the desired state¶

Storing the desired state in a Git repository is a commonly-used method to do this. This makes it easy to revert to an earlier version if a resource was changed that is causing an issue.

Ideally, a GitOps setup automatically applies the resources stored in Git to the cluster. This prevents drift between the desired state in Git and on the cluster.

However, a backup of all the resources in the cluster can also be useful. In case a cluster must be re-generated, the backup can be used to restore TLS secrets which might be costly to regenerate. You should restore these before the certificate resources are created to prevent the cert-manager controller from starting issuance of the certificate.

Alternative scenario¶

The cluster acts as the source-of-truth for the desired state¶

This setup is suboptimal compared with the previous scenario.

In this scenario, you don't have an external system that acts as the source-of-truth for the desired state. Instead, the Kubernetes cluster state is used as the source-of-truth.

In this situation, it's hard to back up only the desired state. The Kubernetes cluster state contains a mix of resources that belong to the desired state, and resources that were generated by controllers.

The easiest backup strategy is to back up all Kubernetes resources (including all cert-manager's custom resources, RBAC and service accounts created for Venafi Connection, generated Certificates, and so forth). Velero is a useful tool for this. There are some challenges that must be considered when working with Velero, most of them relate to backing-up the cert-manager resources. For more information on this, see the cert-manager documentation.

An example tutorial on how to use Velero to back up a full Kubernetes cluster, including a cert-manager installation, can be found on the cert-manager documentation site.