Grafana Dashboard Templates¶

Acteon ships with pre-built Grafana dashboards and a Prometheus scrape configuration that provide immediate observability over the gateway. Two dashboards cover the full metric surface: an Overview dashboard for gateway-wide throughput and outcomes, and a Provider Health dashboard for per-provider latency percentiles and success rates.

No external dependencies are required beyond Prometheus and Grafana. The gateway exposes a lightweight Prometheus text-format endpoint (GET /metrics/prometheus) with zero third-party crate overhead.

Quick Start¶

The fastest way to get the full monitoring stack running is with Docker Compose:

docker compose --profile monitoring up

This starts four services:

Grafana is available at http://localhost:3000 with default credentials configured in deploy/grafana/grafana.ini. Both dashboards are provisioned automatically under the Acteon folder.

Prometheus Endpoint¶

GET /metrics/prometheus

Returns all gateway metrics in Prometheus text exposition format (text/plain; version=0.0.4). No authentication is required for this endpoint.

The endpoint reads atomic in-memory counters and serializes them directly to the text format. There is no dependency on the prometheus crate -- the exporter is a hand-written ~80-line Axum handler that formats # HELP, # TYPE, and metric lines.

Example Response¶

# HELP acteon_actions_dispatched_total Total number of actions dispatched to the gateway.
# TYPE acteon_actions_dispatched_total counter
acteon_actions_dispatched_total 15482

# HELP acteon_provider_success_rate Provider success rate percentage (0-100).
# TYPE acteon_provider_success_rate gauge
acteon_provider_success_rate{provider="email"} 99.12
acteon_provider_success_rate{provider="slack"} 97.50

Exported Metrics¶

All metrics use the acteon_ prefix. Counters are monotonically increasing from server start; gauges reflect current computed values.

Gateway Dispatch Counters¶

LLM Guardrail Counters¶

Chain (Workflow) Counters¶

Circuit Breaker Counters¶

Recurring Action Counters¶

Quota Counters¶

Retention Reaper Counters¶

Embedding Cache Counters¶

These metrics are only emitted when an embedding provider is configured.

Per-Provider Metrics¶

These metrics carry a provider label and are emitted for each registered provider.

Overview Dashboard¶

The Acteon Overview dashboard (acteon-overview.json) provides a high-level view of gateway activity across seven collapsible row sections:

Throughput¶

• Action Throughput (timeseries) -- Dispatched, executed, and failed action rates over time using rate(...[5m]).
• Action Outcomes (stacked) (timeseries) -- Stacked area chart showing the breakdown of all outcome types (executed, suppressed, deduplicated, rerouted, throttled, failed).
• Totals (stat) -- Absolute counter values for all nine action outcome types. Failed and throttled counters turn red/orange when non-zero.

LLM Guardrail¶

• LLM Guardrail Decisions (timeseries) -- Allowed, denied, and error rates.
• LLM Guardrail Totals (stat) -- Absolute counts with sparkline area graphs.

Chains (Workflows)¶

• Chain Throughput (timeseries) -- Started, completed, failed, and cancelled chain rates.
• Chain Success Rate (gauge) -- Completed / started ratio. Green above 95%, orange 90-95%, red below 90%.
• Chain Totals (stat) -- Absolute chain lifecycle counts.

Circuit Breaker¶

• Circuit Breaker Activity (timeseries) -- Open rejections, state transitions, and fallback reroutes.
• Circuit Breaker Totals (stat) -- Absolute counts; non-zero values turn red.

Recurring Actions¶

• Recurring Action Totals (stat) -- Dispatched, errors, and skipped counts. Error count turns red when non-zero.
• Recurring Action Rate (timeseries) -- Dispatched and error rates over time.

Quotas & Retention¶

• Quota Totals (stat) -- Exceeded (red when > 0), warned, degraded, and notified counts.
• Retention Reaper Totals (stat) -- Deleted, compliance hold (skipped), and error counts.

Embedding Cache¶

• Embedding Cache Hit/Miss Rate (timeseries) -- Topic and text cache hit/miss rates.
• Topic Cache Hit Rate (gauge) -- Green above 80%, orange 50-80%, red below 50%.
• Text Cache Hit Rate (gauge) -- Same thresholds as topic cache.

Provider Health Dashboard¶

The Acteon Provider Health dashboard (acteon-provider-health.json) provides per-provider observability across three row sections:

Provider Success Rates¶

• Success Rate by Provider (stat) -- Large stat panels showing each provider's success rate percentage. Color-coded: green (>= 99%), yellow (>= 95%), orange (>= 90%), red (< 90%).

Request Volume¶

• Request Rate by Provider (timeseries) -- Per-provider request rate using rate(acteon_provider_requests_total[5m]).
• Failure Rate by Provider (timeseries) -- Per-provider failure rate.
• Total Requests by Provider (stat) -- Absolute request counts per provider.

Latency¶

• Average Latency by Provider (timeseries) -- Average latency in milliseconds per provider over time.
• p99 Latency by Provider (timeseries) -- 99^th percentile latency per provider over time.
• Latency Percentile Summary (table) -- Combined table with columns: Provider, p50 Latency, p95 Latency, p99 Latency, Avg Latency, and Success Rate. Success Rate column is color-coded by threshold.

Standalone Setup¶

If you are not using Docker Compose, you can set up monitoring manually.

1. Configure Prometheus¶

Add an Acteon scrape job to your prometheus.yml:

scrape_configs:
  - job_name: "acteon"
    metrics_path: "/metrics/prometheus"
    scrape_interval: 15s
    static_configs:
      - targets: ["localhost:8080"]
        labels:
          service: "acteon-gateway"

For multiple Acteon instances, list all targets:

static_configs:
  - targets:
    - "acteon-1:8080"
    - "acteon-2:8080"
    - "acteon-3:8080"

2. Import Dashboards into Grafana¶

Import the dashboard JSON files from the deploy/grafana/dashboards/ directory:

1. Open Grafana and navigate to Dashboards > Import.
2. Click Upload JSON file and select acteon-overview.json.
3. Select your Prometheus datasource for the DS_PROMETHEUS variable.
4. Repeat for acteon-provider-health.json.

Alternatively, use Grafana provisioning by copying the files from deploy/grafana/provisioning/ into your Grafana configuration directory:

/etc/grafana/provisioning/
  datasources/
    prometheus.yml          # Points to your Prometheus instance
  dashboards/
    dashboards.yml          # Points to the dashboard JSON directory

The provisioning datasource config (deploy/grafana/provisioning/datasources/prometheus.yml):

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

The provisioning dashboard config (deploy/grafana/provisioning/dashboards/dashboards.yml):

apiVersion: 1
providers:
  - name: "Acteon"
    orgId: 1
    folder: "Acteon"
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: false

3. Verify¶

1. Open Prometheus at http://localhost:9090/targets and confirm the acteon job shows status UP.
2. Open Grafana at http://localhost:3000 and navigate to the Acteon folder. Both dashboards should be listed.
3. Dispatch a few actions through the gateway and watch metrics populate.

Customization¶

Adding Custom Panels¶

Both dashboards use a DS_PROMETHEUS template variable for the datasource. To add custom panels:

1. Open a dashboard in Grafana and click Edit.
2. Add a new panel and select ${DS_PROMETHEUS} as the datasource.
3. Use any acteon_* metric in your PromQL queries.
4. Save the dashboard.

Since provisioned dashboards are set to editable: true, changes persist in Grafana's storage. To make changes permanent across deployments, export the modified dashboard JSON and commit it to deploy/grafana/dashboards/.

Useful PromQL Queries¶

Overall success rate:

(acteon_actions_executed_total / clamp_min(acteon_actions_dispatched_total, 1)) * 100

Action failure rate (5-minute window):

rate(acteon_actions_failed_total[5m]) / clamp_min(rate(acteon_actions_dispatched_total[5m]), 0.001) * 100

Provider with highest p99 latency:

topk(1, acteon_provider_p99_latency_ms)

Quota usage trending toward limit:

rate(acteon_quota_exceeded_total[1h])

Alerting¶

Grafana supports alerting directly from dashboard panels. Recommended alert rules:

To configure alerts, edit a panel, switch to the Alert tab, and define thresholds. See the Grafana alerting documentation for details on notification channels and routing.

Configuration Reference¶

Prometheus Scrape Config¶

Grafana Configuration¶

Grafana is configured via deploy/grafana/grafana.ini, which is mounted into the container at /etc/grafana/grafana.ini. Key settings:

Dashboard Settings¶

Both dashboards share these settings:

Production Hardening¶

Authentication¶

The default Docker Compose setup uses credentials from deploy/grafana/grafana.ini. For production:

1. Change the admin password in grafana.ini or override it via the Grafana UI on first login.
2. Enable Grafana's built-in LDAP, OAuth, or SAML authentication.
3. Consider placing Prometheus behind a reverse proxy with authentication -- the /metrics/prometheus endpoint exposes operational data.

# deploy/grafana/grafana.ini
[security]
admin_password = your_production_password

[auth.generic_oauth]
enabled = true
# ... OAuth config

Data Retention¶

Prometheus defaults to 30 days of retention. Adjust based on your storage budget:

command:
  - "--storage.tsdb.retention.time=90d"
  - "--storage.tsdb.retention.size=10GB"

For long-term storage, consider Thanos or Cortex as a remote write backend.

High Availability¶

For production HA deployments:

• Prometheus: Run two independent Prometheus instances scraping the same targets. Use Thanos or similar for deduplication and unified querying.
• Grafana: Grafana supports HA mode with a shared PostgreSQL or MySQL database for session/dashboard storage.
• Acteon: Each Acteon instance exposes its own /metrics/prometheus endpoint. List all instances as Prometheus targets and use sum(rate(...)) in PromQL to aggregate.

Network Security¶

• Restrict Prometheus scrape access to internal networks only.
• Use TLS between Prometheus and Grafana (GF_SERVER_PROTOCOL=https).
• The /metrics/prometheus endpoint does not require authentication by default. If your Acteon server is publicly accessible, use a reverse proxy or firewall rule to restrict access to the metrics endpoint.

File Layout¶

deploy/
  grafana/
    grafana.ini                      # Grafana server configuration (auth, security)
    dashboards/
      acteon-overview.json           # Gateway overview dashboard
      acteon-provider-health.json    # Per-provider health dashboard
    provisioning/
      dashboards/
        dashboards.yml               # Dashboard provisioning config
      datasources/
        prometheus.yml               # Prometheus datasource config
  prometheus/
    prometheus.yml                   # Prometheus scrape configuration