Local Argo CD in k3d with Trusted HTTPS Routes

Published on May 31, 2024 by Ian Homer

The non-trusted certificate warnings when I spun up Argo CD locally in a k3d cluster were bugging me. Let's fix them and get Argo CD spun up locally without these warnings, with trusted certificates and end-to-end encrypted flows.

TL;DR

Spin up k3d cluster:

k3d cluster create my-cluster -p "443:443@loadbalancer"

Install Argo CD with helm:

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd --namespace argocd --create-namespace

Apply traefik route:

kubectl apply -f - << EOF
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: argocd-route
  namespace: argocd
spec:
  entryPoints:
    - websecure
  routes:
    - match: HostSNI('argocd.local')
      services:
        - name: argocd-server
          port: 443
  tls:
    passthrough: true
EOF

echo "127.0.0.1 argocd.local" | sudo tee -a /etc/hosts

Set up a local certificate authority. First of all, create local certificates for the CA.

mkdir -p ~/local/certs/
openssl genrsa -out ~/local/certs/local-ca.key 2048
openssl req -new -x509 -subj "/CN=Local CA" \
  -key ~/local/certs/local-ca.key \
  -out ~/local/certs/local-ca.crt

Trust local CA

sudo security add-trusted-cert -d -r trustRoot \
  -k "/Library/KeyChains/System.keychain" \
  ~/local/certs/local-ca.crt

Create certificate for Argo CD

openssl req -subj '/CN=argocd.local' \
  -new -newkey rsa:2048 -sha256 -noenc -x509 \
  -addext "subjectAltName = DNS:argocd.local" \
  -CA ~/local/certs/local-ca.crt \
  -CAkey ~/local/certs/local-ca.key \
  -keyout ~/local/certs/argocd-local.key \
  -out ~/local/certs/argocd-local.crt

Set as Argo CD certificate

kubectl create -n argocd secret tls argocd-server-tls \
  --cert=$HOME/local/certs/argocd-local.crt \
  --key=$HOME/local/certs/argocd-local.key

Enjoy https://argocd.local with no certificate warnings. If you want to go into a little more depth into any of those steps, read on.

Spin up Argo CD in a local k3d cluster

Let's go into those steps in a little more detail. Create our k3d cluster with HTTPS port enabled.

k3d cluster create my-cluster -p "443:443@loadbalancer"

And install Argo CD

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd --namespace argocd --create-namespace

Now we're ready to experiment with certificate and route configuration to get a trusted end-to-end encrypted flow to the Argo CD dashboard.

Install local Argo CD route

Create our traefik argocd-route.yaml to give us a route to connect to Argo CD. Note that this uses a IngressRouteTCP which allows us to configure TLS and in particular allow traefik, with passthrough: true, to indicate that TLS should not terminate at traefik and instead rely on termination on the backend service.

# argocd-route.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: argocd-route
  namespace: argocd
spec:
  entryPoints:
    - websecure
  routes:
    - match: HostSNI(`argocd.local`)
      services:
        - name: argocd-server
          port: 443
  tls:
    passthrough: true

Apply the resource.

kubectl apply -f argocd-route.yaml

We'll also register the local domain name for argocd.local in our /etc/hosts, so we can use that domain name to access Argo CD in the cluster. The virtual hosting configuration in the HostSNI in the IngressRouteTCP resource above, picks up on this host we have come in on and routes through to Argo CD.

127.0.0.1 argocd.local

When we access the Argo CD dashboard on https://argocd.local we get the expected certificate warnings that we are going to fix.

Argo CD certificate warnings

We do also see a warning when we login with the CLI

❯ argocd login argocd.local --grpc-web
WARNING: server certificate had error: tls: failed to verify
certificate: x509: certificate signed by unknown authority.
Proceed insecurely (y/n)?

And digging below the surface, we also see this warning signal with curl

❯ curl -v https://argocd.local

* Host argocd.local:443 was resolved.
* IPv6: (none)
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:443...
* Connected to argocd.local (127.0.0.1) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

We can see that the certificate is the one that Argo CD has deployed by default and hence not surprising that we are getting those warnings.

echo | openssl s_client -showcerts -connect argocd.local:443 2>/dev/null

CONNECTED(00000003)
---
Certificate chain
 0 s:O=Argo CD
   i:O=Argo CD
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 31 13:27:14 2024 GMT; NotAfter: May 31 13:27:14 2025 GMT
-----BEGIN CERTIFICATE-----

Argo CD documentation on TLS Configuration guides us on options, and we'll essentially update the cert and key as described in the section on "Inbound TLS options for argocd-server".

Create local certificate authority

We could create a self-signed certificate and set up trust for that self-signed certificate, but I anticipate creating a few certificates to help with other routes beyond Argo CD. Instead I'm going to create a local certificate authority (CA) that I can set up trust for, and then any certificates I create with that CA will also be trusted.

Note also that public certificate authorities will not generate certificates for internal names, like .local addresses. If we used a CA like Let's Encrypt we'd need to change our local domain naming scheme. That's a pretty good option too, although it does create an external dependency which I don't need at this point.

I'll create these in a local ~/local/certs folder where I keep, and reuse, local test certificates.

mkdir -p ~/local/certs
openssl genrsa -out ~/local/certs/local-ca.key 2048
openssl req -new -x509 -subj "/CN=Local CA" \
  -key ~/local/certs/local-ca.key \
  -out ~/local/certs/local-ca.crt

On macOS, we can trust this local CA with the security command, but you can do this in other ways if you prefer.

sudo security add-trusted-cert -d -r trustRoot \
  -k "/Library/KeyChains/System.keychain" \
  ~/local/certs/local-ca.crt

Create certificate signed by our local CA

Now we are ready to create a self-signed CA. I'll do this with a one-liner, but if you are likely to renew the certificate at some point in the future, then the 3 step process may help - 1) create private key, 2) create certificate signing request (CSR), 3) sign certificate. Since we're just working on temporary stack, the one-liner is good enough.

openssl req -subj '/CN=argocd.local' \
  -new -newkey rsa:2048 -sha256 -noenc -x509 \
  -addext "subjectAltName = DNS:argocd.local" \
  -CA ~/local/certs/local-ca.crt \
  -CAkey ~/local/certs/local-ca.key \
  -keyout ~/local/certs/argocd-local.key \
  -out ~/local/certs/argocd-local.crt

Now, update the TLS secret argocd-server-tls that drives the Argo CD certification used for TLS termination. Argo CD reloads this once this secret changes.

kubectl create -n argocd secret tls argocd-server-tls \
  --cert=$HOME/local/certs/argocd-local.crt \
  --key=$HOME/local/certs/argocd-local.key

The certificate is now trusted when we access the dashboard and we not longer have the certificate warnings in the browser.

Argo CD certificate warnings .

We can confirm the appropriate certificate that we created for argocd.local is being used.

❯ echo | openssl s_client -showcerts -connect argocd.local:443 2>/dev/null

CONNECTED(00000003)
---
Certificate chain
 0 s:CN=argocd.local
   i:CN=Local CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 31 13:46:12 2024 GMT; NotAfter: Jun 30 13:46:12 2024 GMT

Why do we need to set a subjectAltName?

If we create certificate without the subjectAltName (SAN), e.g. with:

openssl req -subj '/CN=argocd.local'   \
  -new -newkey rsa:2048 -sha256 -noenc -x509 \
  -CA ~/local/certs/local-ca.crt        \
  -CAkey ~/local/certs/local-ca.key     \
  -out ~/local/certs/argocd-local.crt   \
  -keyout ~/local/certs/argocd-local.key

And the update the Argo CD TLS certificate.

kubectl create secret tls argocd-server-tls \
  --save-config --dry-run=client             \
  --key=$HOME/local/certs/argocd-local.key \
  --cert=$HOME/local/certs/argocd-local.crt \
  -o yaml | \
kubectl apply -n argocd -f -

Then the certificate warning will show again in the browser. Access to the endpoint with curl will not give a warning, although the argocd command line does give us a useful message indicating that the SAN is missing.

❯ argocd login argocd.local --grpc-web
WARNING: server certificate had error: tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead. Proceed insecurely (y/n)?

A Stack Exchange post gives a good description on why SAN is needed on local domains nowadays.

To fix the certificate, regenerate the certificate with the SAN and update the argocd-server-tls secret and certificate. Access to the Argo CD dashboard should be all valid again.

No more certificate warnings

Although Argo CD was functional without fixing these certificate warnings, it is less of distraction when we fixed it right. An alternative approach would be to disable TLS termination on Argo CD by setting server.insecure to true.

helm install argocd argo/argo-cd --namespace argocd --create-namespace
  --set configs.params."server\.insecure"=true

We could then set up TLS termination in traefik, along with the appropriate certificate, with traefik TLS configuration. Having unencrypted traffic inside the cluster is OK, but feels less than good, especially since Argo CD has TLS termination on, by default.

We could also leave the flow unencrypted outside of the cluster, and come on HTTP over port 80. However, we would never want to leave the flow unencrypted in a non-local deployment, and doing so locally just doesn't feel right.

All-in-all, unencrypted flows on purely local stacks are not a major concern, however local stacks are always a safe place to practice this and be ready for production like stacks.

Clean up the cluster

Delete the cluster once done.

k3d cluster delete my-cluster

If you are not using the CA certificate again, you may want to remove that trust, since it is no longer needed. On macOS the Keychain Access tool can delete the certificate from your local machine.

Delete CA certificate trust .