Skip to content
The Things Stack documentation home
Getting Started Concepts Cloud Enterprise Integrations API Hardware
Getting Started Concepts
Cloud Enterprise
Integrations API Hardware
Get The Things Stack

    Recent searches

      Popular pages

      • Getting Started
      • Adding Gateways
      • Adding Devices
      • Payload Formatters
      • Webhooks
      • Authentication

      No results for “”.

      • Check the spelling, or try fewer or more general words.
      • Ask the community on The Things Network Forum
      • Contact support
      ↑↓ to navigate ↵ to open Esc to close

      The Things Stack Enterprise

      • Overview
      • AWS
        • AWS Marketplace AMI
        • AWS ECS
          • Architecture
          • Prerequisites
          • Deployment
          • Database Operations
          • Monitoring
          • Updating
          • Best Practices
          • Interoperability
          • Mutual TLS
          • Troubleshooting AWS ECS Deployment
          • Template Changelog
      • Kubernetes
      • Docker
      • The Things Join Server
      • Manage The Things Stack
      New in 3.30.0

      Mutual TLS

      Updated 20 January 2025 Edit on GitHub
      Copy page
      • View as MarkdownThis page as plain text
      • Open in ChatGPTAsk questions about this page
      • Open in ClaudeAsk questions about this page

      The Things Stack supports mutual TLS authentication via a supported proxy (Envoy and Traefik).

      Mechanism

      The Things Stack does not terminate mTLS connections: this is done by the proxy. The proxy is expected to forward the presented TLS client certificate via an HTTP header to The Things Stack.

      The Things Stack verifies this header with a pre-configured list of CAs.

      The Things Stack supports mTLS authentication for LoRa Basics™ Station CUPS connections.

      TLS termination

      In the standard deployment model, AWS Network Load Balancer (NLB) terminates TLS, and then forwards unencrypted packets to The Things Stack via AWS internal network.

      Since the NLB does not support verifying client TLS certificates, the proxy performs TLS termination, only on the relevant ports. This includes 8987 (LoRa Basics™ Station CUPS mTLS) port.

      Certificate Requirements

      The CN (Common Name) field of the certificate must be the 16 digit gateway EUI. Example: 0123456789ABCDEF

      Update Instructions

      The SupportProxyTLS and LBSCUPSmTLSEnabled options are available in multiple templates. Make sure to set it to true in all the templates if the proxy should terminate LoRa Basics™ Station CUPS mTLS Connections.

      If not, it should be set to false in all the templates (this is the default). Setting it true in one and false in another will lead to to errors.

      Preparation

      • If you are deploying TTS for the first time, complete the steps with SupportProxyTLS and LBSCUPSmTLSEnabled set to false (this is the default).
      • If you are updating to a version with mTLS support from a previous version, first deploy the following templates in this exact order with SupportProxyTLS and LBSCUPSmTLSEnabled set to false.
        • 3-1-security-group-rules: The template will be deployed with no changes.
        • 3-2-load-balancer-rules: The template will be deployed with no changes.
        • 4-1-secrets: The template will be deployed with no changes.
        • 4-2a-configuration: CloudFormation will mark GSConfiguration for a change but this is only on the template. No resources will be changed.
        • 5-4-ecs-services: There will be changes to the GS and GCS Task roles. The GS/GCS services/tasks will not be updated.
        • 5-7-ecs-proxy: The template will be deployed with no changes.
        • 5-8a-certs-le: The template will be deployed with no changes.

      Enabling mTLS Support

      Once the preparation step is complete, run the following steps in the exact same order as described here.

      Step 1: Configuration of the CA Certificate Store

      First, deploy template 2-4c-mtls-s3. This generates an S3 bucket to store CA certificates.

      After this template has been deployed, navigate to S3 and find the bucket that was created. You can check the Resources tab in Cloud Formation to find the bucket.

      Create an index.yml file that contains the CA store configuration and upload it to the root of the bucket.

      The contents of this file are as below.

      common:
        - common.pem
      

      Create a common folder at the root of this repository. This folder contains the common trusted root certificates that are used to verify the client certificate. In the future, there may be specific groups of trusted root certificates for specific purposes.

      Note:

      If you want to set the certificates later, create an empty index.yml file.

      Step 2: Fetch and Store Server TLS Credentials

      The Things Stack AWS ECS deployment uses Certbot to obtain a TLS server certificate from Let’s Encrypt and stores it in AWS Certificate Manager.

      As certificates stored in ACM cannot be retrieved, a copy of the TLS server certificate is stored in AWS Secrets Manager. The proxy loads the TLS server certificate from AWS Secrets Manager.

      Update 4-1-secrets with the following setting:

      • Set SupportProxyTLS set to true.

      This creates a ProxyTLSSecret for the TLS server certificate secrets and also a GCS token hash key secret, which is used by the Gateway Configuration Server and the Identity Server.

      Next, update 5-8a-certs-le with the following settings:

      • Set SupportProxyTLS to true. If ExistingCertARN is set, clear this before running the update.
      • Fetch new certificates by manually requesting it using the instructions.
      • When fetching the certificates, make note of the CertificateArn field from the logs where the new certificates are written.
      • Once the certificates are successfully queried, rerun the same 5-8a-certs-le template but this time set, the ExistingCertARN with the CertificateArn value. This ensures that Certbot renews these certificates and doesn’t replace them.

      Step 3: Update the Load Balancer and the Proxy

      We now forward traffic from 8987 on the AWS NLB to 8987 on Envoy.

      First update 3-1-security-group-rules. Set LBSCUPSmTLSEnabled to true. This will allow traffic from 8987 to flow into the cluster.

      Next Update 3-2-load-balancer-rules.

      • Set SupportProxyTLS to true.
      • Set TLSCertificateARN with the ARN from the previous step and update.
      • Set LBSCUPSmTLSEnabled to true.

      The new certificate is picked up by all TLS listeners and new listener and target group for LoRa Basics™ Station CUPS mTLS is created.

      Now, update 5-7-ecs-proxy. Set SupportProxyTLS and LBSCUPSmTLSEnabled to true.

      Make sure that the image of the proxy is minimum v3.30.0. Images from earlier releases do not contain changes necessary for mTLS support.

      You can check if the connection is successful by querying the certificates presented by the 8987 listener.

      $ openssl s_client -connect <domain>:8987
      

      Step 4: Update The Things Stack Components

      Since mTLS is supported for LoRa Basics™ Station CUPS , The Things Stack Identity Server and Gateway Configuration Server components require additional configuration.

      In 4-2a-configuration, set SupportProxyTLS to true and deploy it.

      Once that is completed, deploy 5-4-ecs-services with SupportProxyTLS set to true.

      Step 5: Testing

      Now that the above setup is complete, we can now test if everything worked. For testing instructions check the mTLS Authentication page.

      Additional Operator Notes

      Reloading Certificates

      Envoy (proxy) supports reloading TLS certificates without dropping active connections.

      This is done by using the static SDS configuration.

      The 5-8b-ecs-certbot-scheduled-task runs a periodic task to fetch new server TLS credentials.

      If there are new credentials, they will also be stored in Secrets Manager.

      The -aws-proxy images used for the proxy contain a cronjob that queries Secrets Manager (daily by default) and updates the proxy configuration.

      ← Interoperability Troubleshooting AWS ECS Deployment →

      On this page

      • Mechanism
      • Update Instructions
      • Additional Operator Notes

      Sections

      Getting Started

      Concepts

      Cloud

      Enterprise

      Integrations

      API

      Hardware

      Sitemap

      View our Sitemap

      Contributing

      GitHub

      About Us

      The Things Industries

      Privacy Policy

      Cookie settings

      Cookie Settings

      We use cookies to enhance your experience, analyze site traffic and deliver personalized content. Read our Privacy Policy.

      Cookie categories