🏡 Frank Zhao

Ingress certificates with private CA

5 min read

Summary

Notes on how to set up a private CA using step-ca, integrate with cert-manager and use to automatically issue certificates for ingresses. We will use a JWK provisioner in a cluster-wide issuer to issue certificates.

Install step-ca

In this example, I install step-ca on a RaspberryPi and configure it as a systemd service.

Install step-ca from Debian packages:

wget https://dl.smallstep.com/cli/docs-ca-install/latest/step-cli_arm64.deb
wget https://dl.smallstep.com/certificates/docs-ca-install/latest/step-ca_arm64.deb
sudo dpkg -i step-cli_arm64.deb
sudo dpkg -i step-ca_arm64.deb

Create a user for step-ca and set the home directory.

sudo useradd --system --home /etc/step-ca --shell /bin/false step
sudo mkdir /etc/step-ca
sudo chown step:step /etc/step-ca

Set up the step user for convenience:

sudo su - step
echo 'export STEPPATH=/etc/step-ca' > /etc/step-ca/.bash_profile
source /etc/step-ca/.bash_profile

Go through the init process. In this case, since I already have a different service running on port 443, I chose to use port 8444.

step ca init

The root and intermediate certificates will be created in /etc/step-ca/certs. Add the new private CA certificate to the system trust store. Note this file will also be at /usr/local/share/ca-certificates/Smallstep_Root_CA_xx.crt.

step certificate install $(step path)/certs/root_ca.crt

To add a certificate into the system trust store on Debian, CA certificates can be added into /usr/local/share/ca-certificates/ in PEM format with the .crt extension. Then refresh the system trust store using sudo update-ca-certificates.

Set up the systemd service. Note you may need to create password.txt using the password from the prior step.

$ cat /etc/systemd/system/step-ca.service 
[Unit]
Description=step-ca service
Documentation=https://smallstep.com/docs/step-ca
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=30
StartLimitBurst=3
ConditionFileNotEmpty=/etc/step-ca/config/ca.json
ConditionFileNotEmpty=/etc/step-ca/password.txt
 
[Service]
Type=simple
User=step
Group=step
Environment=STEPPATH=/etc/step-ca
Environment=STEPDEBUG=1
WorkingDirectory=/etc/step-ca
ExecStart=/usr/bin/step-ca config/ca.json --password-file password.txt
ExecReload=/bin/kill --signal HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=30
StartLimitBurst=3
 
; Process capabilities & privileges
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
SecureBits=keep-caps
NoNewPrivileges=yes
 
; Sandboxing
; This sandboxing works with YubiKey PIV (via pcscd HTTP API), but it is likely
; too restrictive for PKCS#11 HSMs.
;
; NOTE: Comment out the rest of this section for troubleshooting.
ProtectSystem=full
ProtectHome=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
PrivateTmp=true
ProtectClock=true
ProtectControlGroups=true
ProtectKernelTunables=true
ProtectKernelLogs=true
ProtectKernelModules=true
LockPersonality=true
RestrictSUIDSGID=true
RemoveIPC=true
RestrictRealtime=true
PrivateDevices=true
SystemCallFilter=@system-service
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
ReadWriteDirectories=/etc/step-ca/db
 
[Install]
WantedBy=multi-user.target

Enable the service and start it.

sudo systemctl daemon-reload

Check logs for errors

journalctl -u step-ca.service -f

Step Issuer and Cert Manager

These can be installed with their respective Helm charts.

helm repo add smallstep  https://smallstep.github.io/helm-charts
helm repo add jetstack https://charts.jetstack.io
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.16.3 \
  --set crds.enabled=true
helm install \
  step-issuer smallstep/step-issuer \
  --namespace step-issuer \
  --create-namespace

Create new provisioner for step-issuer

On the step-ca host, set up new credentials for the new provisioner that the step issuer will use in the Kubernetes cluster.

step ca provisioner add "step-issuer" --type JWK --create # Make nots of the password.
sudo systemctl restart step-ca.service
step ca provisioner list | grep step-issuer -A 9 # Make note of the 'kid'.
step ca root | step base64 # Make note of this base64 encoded certificate.

Create the issuer

In this example, I want to issue certificates to Grafana. Note that there are three seperate namespaces to consider here:

  1. Grafana
  2. cert-manager
  3. step-issuer

By the end of this configuration, generated certificates will be placed in a secret in the grafana namespace, using an issuer in the step-issuer namespace, with cert-manager parsing annotations on the ingress. Since we are dealing with cross-namespace resources, we will use a StepClusterIssuer rather than a StepIssuer.

Create the provisioner secret with the password from before:

kubectl -n step-issuer create secret \
		-n step-issuer generic step-issuer-provisioner-password \
		--from-literal=password=<password from before>

Create the StepClusterIssuer:

apiVersion: certmanager.step.sm/v1beta1
kind: StepClusterIssuer
metadata:
 name: step-cluster-issuer
spec:
 # The CA URL.
 url: https://ca.juju.net:8444
 # The base64 encoded version of the CA root certificate in PEM format.
 caBundle: <base64 encoded cert from above>
 # The provisioner name, kid, and a reference to the provisioner password secret.
 provisioner:
   name: step-issuer
   kid: <kid from before>
   passwordRef:
     name: step-issuer-provisioner-password
     key: password
     namespace: step-issuer

Check the status of the issuer:

kubectl get stepclusterissuers.certmanager.step.sm -o yaml
...
  status:
    conditions:
    - lastTransitionTime: "2025-01-26T08:41:53Z"
      message: StepClusterIssuer verified and ready to sign certificates
      reason: Verified
      status: "True"
      type: Ready

Ingress configuration

Update the ingress to specify a certificate should be issued. In this example I use a Traefik ingress, but it will be similar for nginx.

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/issuer: step-cluster-issuer # reference the issuer name
      cert-manager.io/issuer-group: certmanager.step.sm # use cert-manager
      cert-manager.io/issuer-kind: StepClusterIssuer # use the cluster issuer
      meta.helm.sh/release-name: grafana
      meta.helm.sh/release-namespace: grafana
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/service.serversscheme: https
    name: grafana
    namespace: grafana
  spec:
    ingressClassName: traefik
    rules:
    - host: grafana.kube
      http:
        paths:
        - backend:
            service:
              name: grafana
              port:
                number: 80
          path: /
          pathType: Prefix
    tls:
    - hosts:
      - grafana.kube # FQDN for this ingress
      secretName: grafana-tls # Secret to be created to store the certificate

Once changes to the ingress are applied, you can view the status of the certificate.

kubectl -n grafana get certificaterequests.cert-manager.io

It may also be useful to check the events in the namespace when debugging any issues.

Graph View

Backlinks