Matrix
Forgejo
Open Collective

    Metrics

    Continuwuity records metrics with OpenTelemetry and can export them in two ways, both disabled by default:

    • Prometheus scraping from /_continuwuity/metrics
    • OTLP push, alongside traces

    Prometheus

    [global]
allow_prometheus_metrics = true
prometheus_metrics_token = "a-long-random-secret"

    The endpoint is served on the same listener as the rest of the server. Either set prometheus_metrics_token or block /_continuwuity/metrics at your reverse proxy so it is not publicly readable.

    scrape_configs:
  - job_name: continuwuity
    scheme: https
    metrics_path: /_continuwuity/metrics
    authorization:
      credentials: a-long-random-secret
    static_configs:
      - targets: ["matrix.example.com"]

    OTLP

    Requires the otlp_telemetry feature, enabled in default builds.

    [global]
allow_otlp_metrics = true

    Configure the exporter with standard OpenTelemetry environment variables:

    OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf # or "grpc"
OTEL_METRIC_EXPORT_INTERVAL=15000

    service.instance.id defaults to the server name, and OTEL_RESOURCE_ATTRIBUTES or OTEL_SERVICE_NAME override the defaults.

    Prometheus can receive OTLP directly when started with --web.enable-otlp-receiver, using http://prometheus:9090/api/v1/otlp as the endpoint.

    Exported metrics

    Names below are as they appear in Prometheus. OTLP uses the dotted OpenTelemetry names, such as continuwuity.sender.events. HTTP and process metrics follow the OpenTelemetry semantic conventions.

    MetricTypeLabelsDescription
    target_infogaugeservice_version, service_instance_idRunning server version and name
    http_server_request_duration_secondshistogramhttp_request_method, http_route, http_response_status_code, network_protocol_name, network_protocol_version, url_schemeHandled HTTP requests, by route template
    http_server_active_requestsgaugehttp_request_method, url_schemeRequests currently being handled
    http_server_request_body_size_byteshistogramHTTP request duration labelsRequest body sizes, when Content-Length is supplied
    http_server_response_body_size_byteshistogramHTTP request duration labelsResponse body sizes, when known
    continuwuity_federation_incoming_transaction_duration_secondshistogramoutcomeProcessing time of incoming transactions
    continuwuity_federation_incoming_transactions_activegaugeIncoming transactions currently being processed
    continuwuity_federation_incoming_pdus_totalcountertype, outcomeIncoming PDUs by event type and outcome: accepted, forbidden, invalid, error or unparsable
    continuwuity_federation_incoming_pdu_duration_secondshistogramoutcomeTime to handle an incoming PDU
    continuwuity_federation_incoming_pdu_lag_secondshistogramTime between an accepted PDU's origin_server_ts and its arrival
    continuwuity_federation_incoming_edus_totalcountertypeIncoming EDUs by type
    continuwuity_federation_incoming_edu_duration_secondshistogramtypeTime to handle an incoming EDU
    continuwuity_federation_outgoing_request_duration_secondshistogramoutcomeOutgoing federation requests
    continuwuity_federation_destinationsgaugestateRemote servers in retry backoff or with a stale destination cache entry
    continuwuity_sender_transaction_duration_secondshistogramdestination, outcomeOutgoing federation, appservice and push transactions
    continuwuity_sender_events_totalcounterdestination, kind, outcomePDUs and EDUs sent in outgoing transactions
    continuwuity_sender_events_queued_totalcounterdestination, kindPDUs and EDUs queued for sending
    continuwuity_sender_queue_depthgaugeMessages waiting for the sender workers
    continuwuity_usersgaugeoriginKnown local users, including deactivated, and remote users
    continuwuity_invites_totalcounterdirection, destination, outcomeInvites sent by local users or received over federation
    continuwuity_media_write_size_byteshistogramoriginMedia files written, from local uploads or cached remote media
    continuwuity_media_read_size_byteshistogramoriginMedia files served
    continuwuity_media_filesgaugeStored media files, including thumbnails
    continuwuity_antispam_checks_totalcountercheck, outcomeAntispam service checks; denials include the service being unreachable
    continuwuity_database_size_bytesgaugeTotal size of RocksDB SST files
    continuwuity_database_memory_bytesgaugenameRocksDB memtable, table reader and row cache usage
    continuwuity_database_block_cache_bytesgaugecolumnRocksDB block cache usage
    continuwuity_cache_entriesgaugenameEntries in the auth chain and state info caches
    continuwuity_jemalloc_memory_bytesgaugekindjemalloc statistics, only with the jemalloc_stats feature
    process_uptime_secondsgaugeTime since the server started
    process_cpu_time_seconds_totalcountercpu_modeCPU time used by the process
    process_memory_usage_bytesgaugeResident memory (Linux)
    process_memory_virtual_bytesgaugeVirtual memory (Linux)
    process_unix_file_descriptor_countgaugeOpen file descriptors (Linux)
    continuwuity_tokio_workersgaugeTokio runtime worker threads
    continuwuity_tokio_alive_tasksgaugeAlive tokio tasks
    continuwuity_tokio_global_queue_depthgaugeTasks in the tokio global queue
    continuwuity_tokio_worker_busy_time_seconds_totalcounterTime worker threads spent busy
    continuwuity_tokio_worker_parks_totalcounterNumber of times worker threads parked

    User and media file counts are estimates taken from RocksDB (rocksdb.estimate-num-keys), so they may lag slightly or drift after deletions.

    Grafana dashboard

    Import grafana-dashboard.json under Dashboards → New → Import, or provision it from a file.