Automate Let's Encrypt Certificates with the Caddy Web Server

I was setting up a small internal network with a few hosted services and I wanted these services to route through virtual hosts so I could start configuring so good semantic host names for each of these services. Naturally, these services also needed to be deliver over https with trusted certificates.

This felt like a good opportunity to explore using Caddy server and trying out its claim "By default, Caddy automatically obtains and renews TLS certificates for all your sites.". It proved to work really well and gives a low effort way to generate trusted certificates and keep them updated, so let me take you through how I set it up.

DNS Hosting and ACME challenge

This exercise is for an internal network, but it's still good practice to have fully qualified domain names (FQDN) for your services, e.g. myservice.mydomain.com as opposed to myservice.local. Read Why using .local as your domain name extension is a BAD idea for a multitude of reasons. For today's use case, having a domain name that I control is going to allow me to generate certificates with Let's Encrypt. If you want to try this out, you'll need a domain name to hand that you can configure.

Let's Encrypt supports both HTTP and DNS challenge types, that allow Let's Encrypt to verify that we have control of the domain we're requesting a certificate for. This is all part of the ACME standard. The HTTP challenge type involves dropping a file on a web server at the location http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>, which Let's Encrypt needs to be able to read. I'm working with an internal network with no public access, so the HTTP challenge is not going to work for me.

The DNS challenge, on the other hand, needs a TXT entry to be added to the DNS settings for the domain. This allows Let's Encrypt to verify the domain ownership without needing access to the internal network.

I host the domain name I am going to be using for this internal network on namecheap. I do like namecheap as a registrar and DNS hosting services, unfortunately though when it comes to API access to updating TXT entries for a domain the API access is not fine grained. I can get just one API key and the key has a little too much control for my liking. To give me more control I directed the nameservers to Cloudflare:

kirk.ns.cloudflare.com
tess.ns.cloudflare.com

Cloudflare provides DNS hosting and API access as part of their free tier. Furthermore the API token access control configuration allowed me to generate an API token for 1 specific domain that could only update DNS. I could also lock this API access to a single IP address.

Caddy build and configuration with Ansible

To customise the modules enabled with Caddy, we need to build a custom binary. For my use case, I need Caddy with the dns.providers.cloudflare module.

xcaddy handles this build process. I'm using Ansible to orchestrate this system set up, and we can configure this Ansible as follows.

First set up the Cloudsmith apt repositories which hosts the Caddy packages.

- name: Add Cloudsmith apt key
  apt_key:
    url: https://dl.cloudsmith.io/public/caddy/stable/gpg.key
    state: present
    id: 65760C51EDEA2017CEA2CA15155B6D79CA56EA34

- name: Add xcaddy apt repository
  apt_repository:
    repo: deb https://dl.cloudsmith.io/public/caddy/xcaddy/deb/debian any-version main
    state: present
    filename: xcaddy-stable

Then install caddy and xcaddy. xcaddy builds the binary and the caddy install brings in some other configuration files, in particular the systemd set up.

- name: Install caddy
  apt:
    update_cache: yes
    name: caddy
    state: present

- name: Install xcaddy
  apt:
    update_cache: yes
    name: xcaddy
    state: present

Then we can build our Caddy custom build.

- name: Build Caddy
  command: |
    xcaddy build --with github.com/caddy-dns/cloudflare --output /tmp/caddy
  environment:
    PATH: "/usr/local/go/bin:{ { ansible_env.PATH } }"
  args:
    creates: /tmp/caddy

Next we do the Caddy binary shuffle, and bring our custom binary into place following the Caddy build from source docs.

- name: Divert default caddy to make way for custom one
  command: dpkg-divert --divert /usr/bin/caddy.default --rename /usr/bin/caddy
  args:
    creates: /usr/bin/caddy.default

- name: Copy Caddy binary into place
  copy:
    src: /home/admin/tmp/caddy
    remote_src: true
    dest: /usr/bin/caddy.custom
    force: false
    mode: a+x

- name: Update service to custom caddy
  command: |
    update-alternatives --install /usr/bin/caddy caddy \
      /usr/bin/caddy.custom 50
  args:
    creates: /usr/bin/caddy

Finally bring in the templated Caddyfile and enable the service.

- name: Copy Caddyfile into place
  template:
    src: Caddyfile.j2
    dest: /etc/caddy/Caddyfile
    mode: 0644
  notify: reload Caddy

- name: Enable Caddy service
  service:
    name: caddy
    enabled: yes
    state: started

Caddyfile and ACME DNS challenge

The Caddyfile is where we can configure Caddy. This sets up the auto certificates with Cloudflare as DNS provider along with our virtual hosting.

{
  acme_dns cloudflare { { cloudflare_api_key } }
}

https://foo.test.com {
  reverse_proxy 127.0.0.1:8080
}

We need this Cloudflare API key secret injected in. See Handling secrets in your Ansible playbooks on a few ways you can do this.

With all this in place, when Caddy starts up, the certificate are auto generated and the virtual host https://foo.test.com has trusted certificates all set up. We can see the certificates generated in /var/lib/caddy/.local/share/caddy.

openssl x509 -noout -text -in \
  /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/foo.test.com/foo.test.com.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:27:48:6d:4e:cf:d4:9c:1a:14:14:a5:0d:45:b3:f8:b9:f0
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = US, O = Let's Encrypt, CN = E5
        Validity
            Not Before: Jun 16 19:21:20 2024 GMT
            Not After : Sep 14 19:21:19 2024 GMT
        Subject: CN = foo.test.com

We can also get some visibility on the certificate generation process in the logs, with systemctl status caddy.service where we see Caddy initiating the certificate request process with Let's encrypt, then validating the DNS challenge before issuing a certificate.

waiting on internal rate limiter
done waiting on internal rate limiter
using ACME account
authorization finalized
validations succeeded; finalizing order
got renewal info
successfully downloaded available certificate chains
certificate obtained successfully
releasing lock

Auto certificate renewal

Let's Encrypt certificates are only valid for 90 days. This is deliberately short to enforce re-validation of the domain. With the process automated, re-validation shouldn't be a headache. I didn't get a chance to test the renewal today, but I expect Caddy to take care of the renewal of certificates when they approach their expiry date.

For cloud services, such as AWS ACM, we get this certificate renewal taken care of for us. This is one of the advantages of cloud services where don't need to be concerned about this kind of complexity.

It's good to see that Caddy helps make this straightforward outside of a cloud environment. Historically I've set up environments like certbot and nginx and although analogous to the Caddy set up, it does take a little more to set up and maintain.

I've now got myself a low maintenance set up for a collection of services on a local network. I can continue with spinning up those services and mapping them into the front door that Caddy provides. All the traffic to these services are over https with trusted certificates, so I don't need to set up any other explicit certificate trusts as I had to do when I created self signed certificates for the ArgoCD over https blog. Fingers crossed if Caddy does its thing Cardd with automatically renew the certificates for me. I can choose to do TLS termination at Caddy, keeping many of my certificate concerns in one place.

Doing this has also helped me refresh my understanding of the certificate generation and renewal process. Having leaned on cloud services recently that take care a lot of this certificate generation for us, it has been some time since I've used Let's Encrypt (and other similar services, like ZeroSSL) directly.