Cequence uses the Open Telemetry framework for metrics, logging, and tracing. This article outlines the architecture, configuration methods, and implementation approaches used with the OpenTelemetry framework.

Architecture

The Cequence Defender and Cequence Bridge components use Open Telemetry through the OTEL collector service, which centralizes the collection of telemetry data with the following methods.

• Scraping metrics from existing exporters
• Using the Prometheus and Nginx stats exporters as data sources
• Incorporating an Nginx OTEL module for trace publication
• Implementing a file collector plugin to gather logs from all Cequence Defender services

Currently, Cequence services expose statistics APIs to the Prometheus exporter, which publishes statistics on port 9122. The nginx statistics exporter module publishes nginx statistics separately, on port 9145.

Refer to the Open Telemetry Collector documentation for specifics.

Environment Variables

Note that Cequence Bridge does not support the OTEL_TRACES_EXPORTERS environment variable.

Configuration Methods

You can configure your OTEL collector exporters with a YAML configuration file or keep the default values.

exporters:
  prometheus:
    endpoint: "0.0.0.0:9122"

exporters:
  debug:
    verbosity: "detailed"

To configure a custom exporter, create a file at /opt/cequence/otel/conf/exporters.yaml that specifies the configuration, as in the following example. 

exporters:
  otlphttp:
    endpoint: "https://example.com/otlp/trace"
    tls:
      insecure: true
    headers:
      Authorization: "Bearer token123"
  otlp/logs:
    endpoint: "otlp.example.com:4317"
    tls:
      insecure: true

Lists of supported exporters are available at Github repositories.

Configuring extensions

To configure OTEL collector extensions, create a file at /opt/cequence/otel/conf/extensions.yaml that specifies the configuration, as in the following example.

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: 0.0.0.0:1777
  oauth2client:
    client_id: your_client_id
    client_secret: your_client_secret
    token_url: https://oauth.example.com/token
    timeout: 10s

Lists of supported extensions are available at Github repositories.

Usage Examples

When you use custom exporters or extensions, make your configuration files (exporters.yaml or extensions.yaml respectively) available at the /opt/cequence/otel/conf directory.

Basic Configuration

OTEL_METRICS_EXPORTERS=prometheus,debug
OTEL_TRACES_EXPORTERS=debug

Custom Configuration

OTEL_PIPELINE_LOGS_EXPORTERS=otlp/logs
OTEL_PIPELINE_METRICS_EXPORTERS=prometheus,otlp
OTEL_PIPELINE_TRACES_EXPORTERS=otlphttp OTEL_PIPELINE_EXTENSIONS=oauth2client