Troubleshooting Venafi Enhanced Issuer¶

To troubleshoot Venafi Enhanced Issuer problems, start by using the kubectl describe command to view detailed information about your resources and recent events. If this doesn't resolve the problem, consider temporarily increasing the logging level. However, be mindful that excessive logging can impact performance, so only use this as a last resort.

Common troubleshooting procedures¶

Inspecting the Issuer¶

Use the following command to inspect the Issuer:

kubectl describe venafiissuers -n venafi cert-issuer

Note

Adjust the namespace and the issuer name to correspond with your specific configuration.

Output:

Name:         cert-issuer
Namespace:    venafi
Labels:       <none>
Annotations:  <none>
API Version:  jetstack.io/v1alpha1
Kind:         VenafiIssuer
Metadata:
  Creation Timestamp:  2024-10-08T09:00:28Z
  Generation:          1
  Resource Version:    40981
  UID:                 8ba4f625-a8d5-4174-aeda-39a6213205e5
Spec:
  Certificate Name Expression:  optional.none().or(request.?commonName).or(request.dnsNames[?0]).or(request.uris[?0]).or(request.emailAddresses[?0]).or(request.ipAddresses[?0]).value()
  Venafi Connection Name:       my-venafi-connection
  Zone:                         my-application\corp-web
Status:
  Conditions:
    Last Transition Time:  2024-10-08T16:00:41Z
    Message:               Not ready yet: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post "https://api.venafi.cloud/v1/oauth2/v2.0/10f2c691-ab9b-112d-bfed-b3b2b59a7a20/token": http 400: invalid_client: Not found
    Observed Generation:   1
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type     Reason          Age                     From                                Message
  ----     ------          ----                    ----                                -------
  Warning  RetryableError  2m41s (x16 over 5m27s)  venafi-enhanced-issuer.jetstack.io  Not ready yet: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post "https://api.venafi.cloud/v1/oauth2/v2.0/10f2c691-ab9b-112d-bfed-b3b2b59a7a20/token": http 400: invalid_client: Not found

Inspecting a Venafi Connection¶

Use the following command to inspect a Venafi Connection:

kubectl describe venaficonnections -n venafi my-venafi-connection

Note

Adjust the namespace and the Venafi Connection name to correspond with your specific configuration.

Below is an example of an authentication failure when the Venafi Enhanced Issuer is connected to the Venafi Control Plane:

Name:         my-venafi-connection
Namespace:    venafi
Labels:       <none>
Annotations:  <none>
API Version:  jetstack.io/v1alpha1
Kind:         VenafiConnection
Metadata:
  Creation Timestamp:  2024-10-08T09:00:28Z
  Generation:          1
  Resource Version:    41021
  UID:                 93685c02-d367-46ed-a24e-7d3c9f790b11
Spec:
   Vcp:
    Access Token:
    Service Account Token:
      Audiences:
        api.venafi.cloud
      Name:  sa-vei
    Vcp O Auth:
      Tenant ID:  10f2c691-ab9b-112d-bfed-b3b2b59a7a20
Status:
  Conditions:
    Last Transition Time:  2024-10-08T16:00:41Z
    Last Update Time:      2024-10-08T16:00:53Z
    Message:               connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post "https://api.venafi.cloud/v1/oauth2/v2.0/10f2c691-ab9b-112d-bfed-b3b2b59a7a20/token": http 400: invalid_client: Not found
    Observed Generation:   1
    Reason:                Pending
    Status:                False
    Type:                  VenafiEnhancedIssuerReady
Events:                    <none>

Troubleshooting a failed certificate request¶

Troubleshooting a failed certificate request has three steps:

Checking the Certificate resource.
Checking the CertificateRequest.
Check the issuer state.

Step 1: Checking the Certificate resource¶

First, check if you have a Certificate resource created in your namespace. You can get these using kubectl get certificate:

kubectl get certificate

Output:

NAME                READY   AGE
example-com-tls     False   1h

If none is present and you plan to use the ingress-shim: check the ingress annotations more about that is in the ingress troubleshooting guide. If you are not using the ingress-shim: check the output of the command you used to create the certificate.

If you see one with ready status False you can get more info using kubectl describe certificate. If the status is True that means that cert-manager has successfully issued a certificate.

 kubectl describe certificate <certificate-name>

Output:

[...]
Status:
  Conditions:
    Last Transition Time:        2020-05-15T21:45:22Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
  Next Private Key Secret Name:  example-tls-wtlww
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    105s  cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  105s  cert-manager  Stored new private key in temporary Secret resource "example-tls-wtlww"
  Normal  Requested  104s  cert-manager  Created new CertificateRequest resource "example-tls-bw5t9"

Here you will find more info about the current certificate status under Status as well as detailed information about what happened to it under Events. Both will help you determine the current state of the certificate. The last status is Created new CertificateRequest resource, it is worth taking a look at in which state CertificateRequest is to get more info on why your Certificate isn't getting issued.

Step 2: Checking the `CertificateRequest`¶

The CertificateRequest resource represents a CSR in cert-manager and passes this CSR on onto the issuer. You can find the name of the CertificateRequest in the Certificate event log or using kubectl get certificaterequest.

To get more info, you again run kubectl describe:

kubectl describe certificaterequest <CertificateRequest name>

Output:

API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Spec:
  Request: [...]
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letencrypt-production
Status:
  Conditions:
    Last Transition Time:  2020-05-15T21:45:36Z
    Message:               Waiting on certificate issuance from order example-tls-fqtfg-1165244518: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age    From          Message
  ----    ------        ----   ----          -------
  Normal  OrderCreated  8m20s  cert-manager  Created Order resource example-tls-fqtfg-1165244518

Here you will see any issues regarding the Issuer configuration as well as Issuer responses.

Step 3: Check the issuer state¶

If in the above steps you saw an "issuer not ready" error, you can do the same steps again for (cluster) issuer resources:

kubectl describe venafiissuer <Issuer name>
kubectl describe venaficlusterissuer <ClusterIssuer name>

Adjusting the log level¶

Venafi Enhanced Issuer's logging level can be adjusted to control the amount of detailed information it generates. Changing the log level is primarily recommended for troubleshooting purposes. By default, the log level is set to info and the output is in JSON format.

To increase verbosity, directly modify the Venafi Enhanced Issue deployment's configuration to alter the logging verbosity.

Deployment modification¶

Edit the Venafi Enhanced Issuer's deployment by running the following command:
```
kubectl edit deployment -n venafi venafi-enhanced-issuer
```

Set the log level value by adding the --zap-log-level flag to the spec.template.spec.containers.args section:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: venafi-enhanced-issuer
  namespace: venafi
spec:
  template:
    spec:
      containers:
      - args:
        - --zap-log-level=debug # (1)!
        ...
        - --health-probe-bind-address=:8081
        - --metrics-bind-address=0.0.0.0:8080

The --zap-log-level flag has the following values:
- debug
- info
- error
Any integer value greater between 0 and 8 (corresponds to custom debug levels).

Save your changes and quit the text editor to apply your changes.

Custom debug levels¶

To troubleshoot issues related to HTTP requests, you can use logging levels from 6 to 8. However, be aware that sensitive information, such as authentication tokens, may be revealed.

Note

Excessive logging can impact performance. Use this setting wisely and revert to the default level once troubleshooting is complete.

Examples¶

Default output:

{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting EventSource","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","source":"kind source: *v1alpha1.VenafiIssuer"}
venafi-enhanced-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting EventSource","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","source":"kind source: *v1alpha1.VenafiClusterIssuer"}
venafi-enhanced-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting Controller","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest"}
venafi-enhanced-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting workers","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","worker count":1}

Output overriding the log level to debug:

{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting EventSource","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","source":"kind source: *v1alpha1.VenafiIssuer"}
venafi-enhanced-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting EventSource","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","source":"kind source: *v1alpha1.VenafiClusterIssuer"}
venafi-enhanced-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting Controller","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest"}
venafi-enhanced-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting workers","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","worker count":1}
venafi-enhanced-issuer
{"level":"debug","ts":"2024-10-08T09:14:53Z","logger":"VEI.Reconcile","msg":"Request is Ready. Ignoring.","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"my-demo-cert-1","namespace":"app1"},"namespace":"app1","name":"my-demo-cert-1","reconcileID":"ff0bb87d-d2df-4adc-a4ba-1469d28704a8"}

Troubleshooting Venafi Enhanced Issuer¶

Common troubleshooting procedures¶

Inspecting the Issuer¶

Inspecting a Venafi Connection¶

Troubleshooting a failed certificate request¶

Step 1: Checking the Certificate resource¶

Step 2: Checking the CertificateRequest¶

Step 3: Check the issuer state¶

Adjusting the log level¶

Deployment modification¶

Custom debug levels¶

Examples¶

Step 2: Checking the `CertificateRequest`¶