Mutual TLS

The Things Stack supports mutual TLS authentication via a supported proxy (Envoy and Traefik).

Mechanism

The Things Stack does not terminate mTLS connections: this is done by the proxy. The proxy is expected to forward the presented TLS client certificate via an HTTP header to The Things Stack.

The Things Stack verifies this header with a pre-configured list of CAs.

The Things Stack supports mTLS authentication for LoRa Basics™ Station CUPS connections.

TLS termination

In the standard deployment model, AWS Network Load Balancer (NLB) terminates TLS, and then forwards unencrypted packets to The Things Stack via AWS internal network.

Since the NLB does not support verifying client TLS certificates, the proxy performs TLS termination, only on the relevant ports. This includes 8987 (LoRa Basics™ Station CUPS mTLS) port.

Certificate Requirements

The CN (Common Name) field of the certificate must be the 16 digit gateway EUI. Example: 0123456789ABCDEF

Update Instructions

The SupportProxyTLS and LBSCUPSmTLSEnabled options are available in multiple templates. Make sure to set it to true in all the templates if the proxy should terminate LoRa Basics™ Station CUPS mTLS Connections.

If not, it should be set to false in all the templates (this is the default). Setting it true in one and false in another will lead to to errors.

Preparation

• If you are deploying TTS for the first time, complete the steps with SupportProxyTLS and LBSCUPSmTLSEnabled set to false (this is the default).
• If you are updating to a version with mTLS support from a previous version, first deploy the following templates in this exact order with SupportProxyTLS and LBSCUPSmTLSEnabled set to false.
  • 3-1-security-group-rules: The template will be deployed with no changes.
  • 3-2-load-balancer-rules: The template will be deployed with no changes.
  • 4-1-secrets: The template will be deployed with no changes.
  • 4-2a-configuration: CloudFormation will mark GSConfiguration for a change but this is only on the template. No resources will be changed.
  • 5-4-ecs-services: There will be changes to the GS and GCS Task roles. The GS/GCS services/tasks will not be updated.
  • 5-7-ecs-proxy: The template will be deployed with no changes.
  • 5-8a-certs-le: The template will be deployed with no changes.

Enabling mTLS Support

Once the preparation step is complete, run the following steps in the exact same order as described here.

Step 1: Configuration of the CA Certificate Store

First, deploy template 2-4c-mtls-s3. This generates an S3 bucket to store CA certificates.

After this template has been deployed, navigate to S3 and find the bucket that was created. You can check the Resources tab in Cloud Formation to find the bucket.

Create an index.yml file that contains the CA store configuration and upload it to the root of the bucket.

The contents of this file are as below.

common:
  - common.pem

Create a common folder at the root of this repository. This folder contains the common trusted root certificates that are used to verify the client certificate. In the future, there may be specific groups of trusted root certificates for specific purposes.

Step 2: Fetch and Store Server TLS Credentials

The Things Stack AWS ECS deployment uses Certbot to obtain a TLS server certificate from Let’s Encrypt and stores it in AWS Certificate Manager.

As certificates stored in ACM cannot be retrieved, a copy of the TLS server certificate is stored in AWS Secrets Manager. The proxy loads the TLS server certificate from AWS Secrets Manager.

Update 4-1-secrets with the following setting:

• Set SupportProxyTLS set to true.

This creates a ProxyTLSSecret for the TLS server certificate secrets and also a GCS token hash key secret, which is used by the Gateway Configuration Server and the Identity Server.

Next, update 5-8a-certs-le with the following settings:

• Set SupportProxyTLS to true. If ExistingCertARN is set, clear this before running the update.
• Fetch new certificates by manually requesting it using the instructions.
• When fetching the certificates, make note of the CertificateArn field from the logs where the new certificates are written.
• Once the certificates are successfully queried, rerun the same 5-8a-certs-le template but this time set, the ExistingCertARN with the CertificateArn value. This ensures that Certbot renews these certificates and doesn’t replace them.

Step 3: Update the Load Balancer and the Proxy

We now forward traffic from 8987 on the AWS NLB to 8987 on Envoy.

First update 3-1-security-group-rules. Set LBSCUPSmTLSEnabled to true. This will allow traffic from 8987 to flow into the cluster.

Next Update 3-2-load-balancer-rules.

• Set SupportProxyTLS to true.
• Set TLSCertificateARN with the ARN from the previous step and update.
• Set LBSCUPSmTLSEnabled to true.

The new certificate is picked up by all TLS listeners and new listener and target group for LoRa Basics™ Station CUPS mTLS is created.

Now, update 5-7-ecs-proxy. Set SupportProxyTLS and LBSCUPSmTLSEnabled to true.

Make sure that the image of the proxy is minimum v3.30.0. Images from earlier releases do not contain changes necessary for mTLS support.

You can check if the connection is successful by querying the certificates presented by the 8987 listener.

$ openssl s_client -connect <domain>:8987

Step 4: Update The Things Stack Components

Since mTLS is supported for LoRa Basics™ Station CUPS , The Things Stack Identity Server and Gateway Configuration Server components require additional configuration.

In 4-2a-configuration, set SupportProxyTLS to true and deploy it.

Once that is completed, deploy 5-4-ecs-services with SupportProxyTLS set to true.

Step 5: Testing

Now that the above setup is complete, we can now test if everything worked. For testing instructions check the mTLS Authentication page.

Additional Operator Notes

Reloading Certificates

Envoy (proxy) supports reloading TLS certificates without dropping active connections.

This is done by using the static SDS configuration.

The 5-8b-ecs-certbot-scheduled-task runs a periodic task to fetch new server TLS credentials.

If there are new credentials, they will also be stored in Secrets Manager.

The -aws-proxy images used for the proxy contain a cronjob that queries Secrets Manager (daily by default) and updates the proxy configuration.