Skip to main content

Observability

SolidPing integrates with popular observability tools to help you monitor the monitoring platform itself. All three integrations are configured independently and can be enabled side by side.

Prometheus Metrics​

SolidPing exposes a Prometheus-compatible metrics endpoint, gated behind a bearer scrape token. Per-org check and incident activity is sensitive on any internet-facing instance, so /metrics answers 404 — indistinguishable from a disabled endpoint — until you set a token; it never serves unauthenticated.

VariableDefaultDescription
SP_PROMETHEUS_ENABLEDtrueMaster switch for the metrics endpoint. false always 404s, regardless of the token below
SP_PROMETHEUS_PATH/metricsPath of the metrics endpoint
SP_METRICS_SCRAPE_TOKEN-Bearer token required to scrape /metrics. Unset means 404; set means every request must carry Authorization: Bearer <token> or get 401. Can also be set as the metrics.scrape_token system parameter from the database, which takes effect on the next restart
Upgrading from a version that scraped without a token

Set SP_METRICS_SCRAPE_TOKEN (or the metrics.scrape_token system parameter) once, then add it to your scrape config as shown below. A security fix that stays off by default is not a fix, so this is a deliberate breaking change — there is no grace period during which the endpoint accepts both authenticated and unauthenticated requests.

Scrape Configuration​

scrape_configs:
  - job_name: solidping
    static_configs:
      - targets: ['solidping:4000']
    metrics_path: /metrics
    authorization:
      credentials: <token>

Available Metrics​

Metrics cover check execution, worker health, job scheduling, and application performance.

Region liveness metrics​

solidping_workers_active{region} is the number of live workers serving each cloud region, and solidping_region_dark{region} is 1 while a region has checks assigned and no live worker, 0 otherwise. Both are refreshed every minute and exist even when the platform watchdog is disabled. An alert on solidping_region_dark == 1 catches a dark region within about 6 minutes of its last worker heartbeat.

Database query metrics​

solidping_db_query_duration_seconds observes SQL query latency, labeled by operation (SELECT, INSERT, ...), backend (postgres/sqlite), status (ok/error) and callsite — the code path that issued the query (e.g. uptimebar.bucket_availability, results.list), or unlabelled when the calling package hasn't been annotated. Only a small, fixed set of hot read paths currently set callsite, so the label's value set stays bounded; it will never contain raw SQL text or request-specific data.

If you built dashboards or alerts against an earlier version: callsite widens this metric's label set, so a selector that used to return one series per operation/backend/status combination now returns one series per combination per distinct callsite value — a panel that plotted a single line can now plot several, and an alert written against a single series now evaluates once per callsite. The concrete breakages this can cause: binary operations and on()/ignoring() matching that assumed identical label sets between two vectors (these can now fail with "many-to-many matching not allowed" or silently drop samples); recording rules and alerts that assumed one series per operation/backend; and a histogram_quantile over a sum by (...) that doesn't include every distinguishing label. If you want the pre-upgrade shape back, aggregate the new label away explicitly rather than adding a wildcard matcher — e.g. sum by (operation, backend, status, le) (rate(solidping_db_query_duration_seconds_bucket[5m])).

Queries slower than db.slow_query_threshold (default 500ms, SP_DB_SLOW_QUERY_THRESHOLD; see Database configuration) are also logged at WARN as slow SQL query, throttled to at most one line per normalized statement per minute (further occurrences are counted in a suppressed field on the next line). The logged statement always has literal values stripped, regardless of the Verbose SQL-logging setting.

solidping_results_rows gauges the total row count in the results table, labeled by period_type (raw, hour, day, month). It's refreshed on the background aggregation job's own cadence (at most once every 5 minutes, not per-request), so it's cheap to scrape but may lag a live row count by a few minutes — useful as an early warning that ingest is outgrowing Postgres shared_buffers well before a table scan turns disk-bound.

Kubernetes ServiceMonitor​

Store the token in a Secret and reference it with authorization.credentials — never inline it in the ServiceMonitor itself, which is usually much less tightly access-controlled than the Secret store.

apiVersion: v1
kind: Secret
metadata:
  name: solidping-metrics-token
stringData:
  token: <token>
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: solidping
spec:
  selector:
    matchLabels:
      app: solidping
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
      authorization:
        credentials:
          name: solidping-metrics-token
          key: token

Sentry​

SolidPing supports Sentry for error tracking and performance monitoring. Set a DSN to enable it.

VariableDefaultDescription
SP_SENTRY_DSN-Sentry DSN (empty = disabled)
SP_SENTRY_ENVIRONMENTproduction (test under SP_RUN_MODE=test)Environment name (development, staging, production)

Leaving SP_SENTRY_ENVIRONMENT unset never produces environment-less events: it defaults to production, or to test when the server runs with SP_RUN_MODE=test. Set it explicitly to distinguish staging from production.

SP_SENTRY_DSN=https://your-key@sentry.io/your-project
SP_SENTRY_ENVIRONMENT=production

Additional tuning such as the traces sample rate and debug logging is available through the sentry section of config.yml, or via SP_SENTRY_TRACES_SAMPLE_RATE / SP_SENTRY_DEBUG:

sentry:
  dsn: https://your-key@sentry.io/your-project
  environment: production
  traces_sample_rate: 0.0   # 0.0 to 1.0, default 0.0 (errors/panics are always captured at 100%; this only controls transaction/performance sampling, which duplicates the OpenTelemetry tracing above for a self-hostable product)
  debug: false

OpenTelemetry​

SolidPing supports OpenTelemetry for distributed tracing, metrics, and log export over OTLP. Traces, metrics, and logs are toggled independently.

VariableDefaultDescription
SP_OTEL_ENABLEDfalseEnable OpenTelemetry export
SP_OTEL_ENDPOINT-OTLP collector endpoint
SP_OTEL_PROTOCOL-OTLP protocol: http or grpc
SP_OTEL_INSECUREfalseSkip TLS verification for the collector
SP_OTEL_TRACES-Export traces
SP_OTEL_METRICS-Export metrics
SP_OTEL_LOGS-Export logs
otel:
  enabled: true
  endpoint: otel-collector:4317
  protocol: grpc
  traces: true
  metrics: true
  logs: false