Telemetry

The Things Stack exports Prometheus telemetry on the /metrics endpoint. This reference gives an overview of the most important metrics.

Warning:
Metrics are not covered by our compatibility commitment. This means that metrics may be changed or removed in major, minor, or patch releases. This page only lists metrics that are considered relatively stable, but even these metrics may be changed.
Note:
Metrics are exported and available only when a corresponding action takes place in The Things Stack. For example, to get the connected gateway’s metric (ttn_lw_gs_connected_gateways), a gateway should be connected to propagate that metric.

Process and Go Metrics

The Things Stack uses the Prometheus instrumentation library for Go that includes a collector for the state of the process (on Linux) and metrics from the Go runtime.

gRPC metrics

The Things Stack uses the Prometheus gRPC instrumentation library that exposes metrics about gRPC method calls.

  • grpc_server_conns_opened_total and grpc_server_conns_closed_total can be used to see how many gRPC connections to The Things Stack are opened, and how many are currently open.
  • grpc_server_started_total and grpc_server_handled_total can be used to see how many RPCs to The Things Stack are started, active and finished (including response code).
  • grpc_server_msg_sent_total and grpc_server_msg_received_total can be used to see how many RPC messages are sent/received by the server.

Similar metrics exist for gRPC client connections opened by The Things Stack, and RPCs made by The Things Stack.

General Metrics

  • ttn_lw_license_expiry_seconds can be used to keep track of license expiry
  • ttn_lw_log_messages_total can be used to track the log messages written by different log namespaces at different log levels.
    • Before version 3.13.1 this metric was called ttn_lw_log_log_messages_total.
  • ttn_lw_events_publishes_total can be used to track the published events by event type.
  • ttn_lw_events_channel_dropped_total can be used to watch for dropped events, which typically indicates that a consumer (such as a user’s web browser) can’t keep up.
  • ttn_lw_workerpool_workers_started and ttn_lw_workerpool_workers_stopped indicate the total number of workers started, and stopped, in the worker pools.
  • ttn_lw_workerpool_workers_idle indicates the number of workers which are idle (not currently processing any work item) in the worker pools.
  • ttn_lw_workerpool_work_queue_size indicates the number of elements found in the queue of the worker pools.
  • ttn_lw_workerpool_work_processed and ttn_lw_workerpool_work_dropped indicates the total amount of work processed, or dropped, by the worker pools.

For the Gateway Server:

  • ttn_lw_gs_connected_gateways indicates the number of connected gateways.
  • ttn_lw_gs_uplink_received_total indicates the number of uplink messages received from gateways.
  • ttn_lw_gs_downlink_sent_total indicates the number of downlink messages sent to gateways.
  • ttn_lw_gs_status_received_total indicates the number of status messages received from gateways.
  • ttn_lw_gs_uplink_forwarded_total indicates the number of uplink messages forwarded to the Network Server or Packet Broker.
  • ttn_lw_gs_uplink_dropped_total indicates the number of uplink messages that are dropped for various reasons.
  • ttn_lw_gs_uplink_failed_total indicates the number of uplink messages for which the processing has failed for various reasons.
  • ttn_lw_gs_downlink_tx_success_total and ttn_lw_gs_downlink_tx_failed_total indicates the number of successful transmissions, and failed transmissions for various reasons, reported by the gateways.

For the Packet Broker Agent:

  • ttn_lw_pba_uplink_received_total indicates the number of uplink messages received from Packet Broker (as a home network).
  • ttn_lw_pba_downlink_forwarded_total indicates the number of downlink messages sent to Packet Broker (as a home network).
  • ttn_lw_pba_downlink_received_total indicates the number of downlink messages received from Packet Broker (as a forwarder).
  • ttn_lw_pba_uplink_forwarded_total indicates the number of uplink messages sent to Packet Broker (as a forwarder).

For the Network Server:

  • ttn_lw_ns_uplink_received_total indicates the number of uplink messages received from the Gateway Server or Packet Broker.
  • ttn_lw_ns_uplink_duplicates_total indicates the number of duplicate uplinks that are merged into a single uplink.
  • ttn_lw_ns_uplink_processed_total indicates the number of processed uplinks, meaning uplinks that are matched to a device.
  • ttn_lw_ns_uplink_forwarded_total indicates the number of uplinks forwarded to an Application Server.
  • ttn_lw_ns_uplink_dropped_total indicates the number of uplinks that are dropped for various reasons.
  • ttn_lw_ns_downlink_attempted_total indicates the number of downlink attempts.
  • ttn_lw_ns_downlink_forwarded_total indicates the number of downlinks forwarded to the Gateway Server.
  • ttn_lw_ns_uplink_gateways is a histogram that indicates the number of gateways that received a given uplink.

For the Application Server:

  • ttn_lw_as_subscriptions_started_total and ttn_lw_as_subscriptions_stopped_total indicate the number of started/stopped uplink subscriptions from integrations and external applications.
  • ttn_lw_as_pubsub_integrations_started_total and ttn_lw_as_pubsub_integrations_stopped_total indicate the number of started/stopped pub/sub integrations.
  • ttn_lw_as_uplink_received_total indicates the number of uplink messages received from the Network Server.
  • ttn_lw_as_uplink_forwarded_total indicates the number of uplink messages forwarded to integrations and external applications.
  • ttn_lw_as_uplink_dropped_total indicates the number of uplink messages dropped for various reasons.
  • ttn_lw_as_downlink_received_total indicates the number of downlink messages received from integrations and external applications.
  • ttn_lw_as_downlink_forwarded_total indicates the number of downlink messages forwarded to the Network Server.
  • ttn_lw_as_downlink_dropped_total indicates the number of downlink messages dropped for various reasons.
  • ttn_lw_as_subscription_sets_started_total and ttn_lw_as_subscription_sets_stopped_total indicates the total number of subscriptions sets started and stopped.
  • ttn_lw_as_packages_processed_total and ttn_lw_as_packages_failed_total indicates the total number of uplink messages processed, or which have failed processing for various reasons, by application packages.
  • ttn_lw_as_packages_storage_upstream_messages_stored and ttn_lw_as_packages_storage_upstream_messages_dropped indicates the total number of uplink messages stored, or dropped for various reasons, by the Storage Integration.
  • ttn_lw_as_webhook_sent_total and ttn_lw_as_webhook_failed_total indicates the total number of webhook messages sent, or failed for various reasons.
  • ttn_lw_javascript_run_latency_seconds is a histogram that indicates the duration of executing JavaScript payload formatters.

For the Join Server:

  • ttn_lw_js_join_accepted_total and ttn_lw_js_join_rejected_total indicate the number of accepted and rejected join requests.
← Deleting, Restoring and Purging Entities Tenant Management →