Troubleshooting Enterprise Issuer for Next-Gen Trust Security¶

To troubleshoot Enterprise Issuer, start by running kubectl describe on your resources to check their status and recent events. If that doesn't reveal the problem, you can temporarily increase the logging level as described in Adjusting the log level.

Common troubleshooting procedures¶

Use these procedures to inspect resources and diagnose common problems with Enterprise Issuer.

Inspecting the Issuer¶

Run kubectl describe on your issuer, replacing the namespace and issuer name with your own:

kubectl describe venafiissuers -n venafi cert-issuer

Example output:

Name:         cert-issuer
Namespace:    venafi
Labels:       <none>
Annotations:  <none>
API Version:  jetstack.io/v1alpha1
Kind:         VenafiIssuer
Metadata:
  Creation Timestamp:  2024-10-08T09:00:28Z
  Generation:          1
  Resource Version:    40981
  UID:                 8ba4f625-a8d5-4174-aeda-39a6213205e5
Spec:
  Certificate Name Expression:  optional.none().or(request.?commonName).or(request.dnsNames[?0]).or(request.uris[?0]).or(request.emailAddresses[?0]).or(request.ipAddresses[?0]).value()
  Venafi Connection Name:       my-venafi-connection
  Zone:                         corp-web
Status:
  Conditions:
    Last Transition Time:  2024-10-08T16:00:41Z
    Message:               Not ready yet: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post "https://api.venafi.cloud/v1/oauth2/v2.0/10f2c691-ab9b-112d-bfed-b3b2b59a7a20/token": http 400: invalid_client: Not found
    Observed Generation:   1
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type     Reason          Age                     From                                Message
  ----     ------          ----                    ----                                -------
  Warning  RetryableError  2m41s (x16 over 5m27s)  venafi-enhanced-issuer.jetstack.io  Not ready yet: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post "https://api.venafi.cloud/v1/oauth2/v2.0/10f2c691-ab9b-112d-bfed-b3b2b59a7a20/token": http 400: invalid_client: Not found

Inspecting a Connection resource¶

Run kubectl describe on your Connection resource, replacing the namespace and resource name with your own:

kubectl describe venaficonnections -n venafi my-venafi-connection

Below is an example of an authentication failure when Enterprise Issuer is connected to Next-Gen Trust Security:

Name:         my-venafi-connection
Namespace:    venafi
Labels:       <none>
Annotations:  <none>
API Version:  jetstack.io/v1alpha1
Kind:         VenafiConnection
Metadata:
  Creation Timestamp:  2024-10-08T09:00:28Z
  Generation:          1
  Resource Version:    41021
  UID:                 93685c02-d367-46ed-a24e-7d3c9f790b11
Spec:
   Vcp:
    Access Token:
    Service Account Token:
      Audiences:
        api.venafi.cloud
      Name:  sa-vei
    Vcp O Auth:
      Tenant ID:  10f2c691-ab9b-112d-bfed-b3b2b59a7a20
Status:
  Conditions:
    Last Transition Time:  2024-10-08T16:00:41Z
    Last Update Time:      2024-10-08T16:00:53Z
    Message:               connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post "https://api.venafi.cloud/v1/oauth2/v2.0/10f2c691-ab9b-112d-bfed-b3b2b59a7a20/token": http 400: invalid_client: Not found
    Observed Generation:   1
    Reason:                Pending
    Status:                False
    Type:                  VenafiEnhancedIssuerReady
Events:                    <none>

Troubleshooting a failed certificate request¶

To troubleshoot a failed certificate request, check the Certificate resource, then the CertificateRequest, and finally the issuer state.

Step 1: Check the Certificate resource¶

First, check if you have a Certificate resource in your namespace using kubectl get certificate:

kubectl get certificate

Example output:

NAME                READY   AGE
example-com-tls     False   1h

If no Certificate resource exists and you are using the ingress-shim, check your ingress annotations. See the ingress troubleshooting guide for details. If you are not using the ingress-shim, check the output of the command you used to create the certificate.

If you see a Certificate with a False ready status, run kubectl describe certificate for more detail. A True status means cert-manager has successfully issued the certificate.

kubectl describe certificate <certificate-name>

Example output:

[...]
Status:
  Conditions:
    Last Transition Time:        2020-05-15T21:45:22Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
  Next Private Key Secret Name:  example-tls-wtlww
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    105s  cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  105s  cert-manager  Stored new private key in temporary Secret resource "example-tls-wtlww"
  Normal  Requested  104s  cert-manager  Created new CertificateRequest resource "example-tls-bw5t9"

The Status section shows the current certificate state, and Events shows what has happened. If the last event is Created new CertificateRequest resource, check the CertificateRequest to find out why the certificate is not being issued.

Step 2: Check the CertificateRequest¶

The CertificateRequest resource represents a certificate signing request (CSR) in cert-manager and passes it to the issuer. You can find the name of the CertificateRequest in the Certificate event log or by running kubectl get certificaterequest.

Run kubectl describe on the CertificateRequest:

kubectl describe certificaterequest <CertificateRequest name>

Example output:

API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Spec:
  Request: [...]
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letencrypt-production
Status:
  Conditions:
    Last Transition Time:  2020-05-15T21:45:36Z
    Message:               Waiting on certificate issuance from order example-tls-fqtfg-1165244518: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age    From          Message
  ----    ------        ----   ----          -------
  Normal  OrderCreated  8m20s  cert-manager  Created Order resource example-tls-fqtfg-1165244518

The output shows any issues with the issuer configuration and its responses.

Step 3: Check the issuer state¶

If either of the previous steps showed an "issuer not ready" error, inspect the issuer resources directly:

kubectl describe venafiissuer <Issuer name>
kubectl describe venaficlusterissuer <ClusterIssuer name>

Adjusting the log level¶

You can adjust Enterprise Issuer's logging level to control its detail level. Changing the log level is primarily recommended for troubleshooting purposes. By default, the log level is set to info and the output is in JSON format.

To increase verbosity, directly modify the Enterprise Issuer deployment's configuration to alter the logging verbosity.

Deployment modification¶

Edit Enterprise Issuer's deployment by running the following command:
```
kubectl edit deployment -n venafi enterprise-issuer
```

Set the log level value by adding the --zap-log-level flag to the spec.template.spec.containers.args section:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: enterprise-issuer
  namespace: venafi
spec:
  template:
    spec:
      containers:
      - args:
        - --zap-log-level=debug # (1)!
        ...
        - --health-probe-bind-address=:8081
        - --metrics-bind-address=0.0.0.0:8080

The --zap-log-level flag has the following values:
- debug
- info
- error
Any integer value greater between 0 and 8 (corresponds to custom debug levels).

Save your changes and quit the text editor to apply your changes.

Custom debug levels¶

To troubleshoot issues related to HTTP requests, you can use logging levels from 6 to 8. However, be aware that sensitive information, such as authentication tokens, may be revealed.

Note

Excessive logging can impact performance. Use this setting wisely and revert to the default level once troubleshooting is complete.

Examples¶

Example output at the default info level:

{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting EventSource","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","source":"kind source: *v1alpha1.VenafiIssuer"}
enterprise-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting EventSource","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","source":"kind source: *v1alpha1.VenafiClusterIssuer"}
enterprise-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting Controller","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest"}
enterprise-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting workers","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","worker count":1}

Example output at the debug level:

{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting EventSource","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","source":"kind source: *v1alpha1.VenafiIssuer"}
enterprise-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting EventSource","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","source":"kind source: *v1alpha1.VenafiClusterIssuer"}
enterprise-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting Controller","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest"}
enterprise-issuer
{"level":"info","ts":"2024-10-08T09:14:53Z","logger":"VEI","msg":"Starting workers","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","worker count":1}
enterprise-issuer
{"level":"debug","ts":"2024-10-08T09:14:53Z","logger":"VEI.Reconcile","msg":"Request is Ready. Ignoring.","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"my-demo-cert-1","namespace":"app1"},"namespace":"app1","name":"my-demo-cert-1","reconcileID":"ff0bb87d-d2df-4adc-a4ba-1469d28704a8"}

Troubleshooting Enterprise Issuer for Next-Gen Trust Security¶

Common troubleshooting procedures¶

Inspecting the Issuer¶

Inspecting a Connection resource¶

Troubleshooting a failed certificate request¶

Step 1: Check the Certificate resource¶

Step 2: Check the CertificateRequest¶

Step 3: Check the issuer state¶

Adjusting the log level¶

Deployment modification¶

Custom debug levels¶

Examples¶

Related links¶