Documentation Grafana Cloud Observe and act Agent Observability Observe and troubleshoot Analytics

Monitor agents with analytics

Agent Observability includes built-in analytics that visualize agent activity, performance, cost, and quality. The analytics views use Prometheus metrics and Agent Observability data to surface actionable insights.

Before you begin

• Configure your agent to export generations and OpenTelemetry metrics to Grafana Cloud. For instructions, refer to Get started with Grafana Cloud.
• Select a time range that includes your agent traffic.
• To use skill analytics, enable the sigil.skills-analytics feature toggle and configure a Prometheus data source that receives SDK tool metrics. Configure Tempo separately for related recorded conversations and traces.

Access dashboards

Select Analytics in Agent Observability. Under Explore by area, expand:

• Agents: generation counts, latency, time to first token, errors, tokens, and estimated cost by agent.
• Usage and cost: token usage, estimated cost, cache efficiency, and conversation activity.
• Evaluations: evaluation scores, pass rates, and quality trends.
• Tools: tool call frequency, duration, and error rates.

Use the time picker to set the query window. The refresh picker lets you set an auto-refresh interval, for example, 30s, 1m, or 5m. The selected interval is saved for the session.

Find items that need attention

The Needs attention section highlights agents, conversations, tools, and evaluations that may need review. Open Health overview settings and select Sudden changes to compare with the previous time window, or High levels to check absolute values. Turn on Show only entities that need attention to hide healthy cards.

Open a card to investigate its supporting data in the relevant Agent Observability view.

Include or exclude traffic

Use the Provider, Model, and Agent filters to focus on specific traffic. Select one or more values in each menu, or enter a custom value. By default, the filter includes the selected values.

To hide those values instead, select Exclude selected in that filter’s menu. Each filter has its own mode, so you can include one provider while excluding a model or agent. Clear Exclude selected to return to inclusion mode.

Close the menu or click outside the dropdown to apply your selection. Selected filter chips stay visible while the URL updates, and your selections stay in the dropdown options list across data reloads. To remove values, remove chips directly or clear the filter.

Dropdown options cascade to reflect your selections:

• Selecting a provider limits Model options to models observed for that provider.
• Selecting a provider and model limits Agent options to agents that match both selections.

In coding-agent analytics views, Provider, Model, and Agent dropdown options are scoped to metrics from known coding-agent launchers. Repository, User, and added label filters use the same coding-agent scope. This prevents unrelated application names or unused provider and model combinations from appearing. Other analytics pages provide unscoped options across all agents.

These controls also apply when you browse conversations or review evaluation results. Your selections and exclusion modes stay in the URL when you share the view, and follow you across pages. For details, refer to Share filters across pages.

Share filters across pages

When you select a provider, model, agent, or custom label filter (for example, service_name = api), your filter selection follows you to other supported Agent Observability pages and persists across page reloads and new browser sessions. Agent Observability saves the selection in your browser’s local storage, so it applies only to that browser and isn’t shared with other users. Time ranges remain scoped to your current browser tab session.

Shared filters apply across pages as follows:

How filter persistence works

Agent Observability restores and saves shared filters by these rules:

• The URL is the source of truth: Saved filters are restored on initial load only when the URL doesn’t already contain filters for that view. When you open a shared link, bookmark, or drill-down link with URL parameters, the page uses those URL parameters and doesn’t overwrite your saved filters. When saved filters are restored, they’re written to the browser address bar so the URL remains shareable.
• Manual edits update saved filters: Only changes you make directly on a page save to your browser storage, and only the filter you change. If you open a link that sets an agent and then add a label filter, the label filter is saved and your saved agent stays as it was.
• Pages save only what they display: When you edit or clear filters on a page, only the filters supported on that page are updated. Changing or clearing an agent on an evaluation page, for example, doesn’t remove a provider or model filter saved from Analytics.
• Conversation label mapping: On Conversations, supported Prometheus metric labels (for example, service_name) automatically translate to conversation resource attributes (for example, resource.service.name), and vice versa. Conversation-only filters, such as conversation title, user, status, or rating, remain local to Conversations. A label filter that uses an operator conversation search doesn’t support, such as !~, stays saved for the metric pages but doesn’t apply on Conversations.
• Evaluation exclusion behavior: Evaluation pages don’t have an exclusion switch. If an agent filter with exclusion mode is saved from another page or loaded from a link, evaluation pages leave the agent filter unscoped rather than filtering out or showing an exclusion, and don’t overwrite or clear your saved exclusion setting.

Set time ranges in page URLs

When you share an Agent Observability view or link to it from a Grafana dashboard, you can control the active time range with the from and to URL query parameters.

URL time parameters accept the following formats:

• Epoch milliseconds: Numbers representing milliseconds since the Unix epoch, for example, ?from=1790812800000&to=1792972800000. Grafana dashboard links and panel data links write this format when you use the ${__from} and ${__to} variables.
• Relative expressions: Relative time strings, for example, ?from=now-7d&to=now or ?from=now-1h&to=now.
• ISO timestamps: ISO dates and timestamps, for example, ?from=2026-10-01T00:00:00Z&to=2026-10-02T00:00:00Z or ?from=20261001&to=20261002.

If a URL parameter contains an invalid or unreadable time value, the page falls back to its default range instead of failing to load. Unreadable values aren’t saved to your session.

Break a chart down by one or more dimensions

Each chart has a Break down switch that splits a single line into one line per dimension. Turn it on and choose dimensions from the picker next to it.

The picker lists common dimensions such as Agent, Model, and Provider first, followed by every other Prometheus label present on your gen_ai_client_* metrics in the current time range. Both kinds work the same way, so you can compare staging against production, or one team’s spend against another’s.

Select more than one dimension to see their combinations. Grouping by Agent and deployment_environment gives you one line per agent per environment, named checkout-agent / production.

You can combine up to three dimensions. Each one multiplies the number of lines on the chart, so the picker stops offering more once you reach the limit. If a chart becomes hard to read, remove a dimension or narrow the filters above it.

A label appears in the picker only if your SDK or collector attaches it to the metrics. The simplest way to add one is to set resource attributes on the service that emits the telemetry:

Bash Copy

export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=ai

Dots become underscores in Prometheus label names, so deployment.environment appears as deployment_environment.

You can also set client tags in the SDK, either with the AGENTO11Y_TAGS environment variable or the tags field in the client configuration:

Bash Copy

export AGENTO11Y_TAGS=team=ai,env=production

Each client tag becomes a label named agento11y_tag_<KEY>, for example, agento11y_tag_team. The SDK also stores the value on every generation.

For coding-agent sessions, refer to Attribute cost to a user, repository, or branch to add user, repository, and branch labels automatically.

Keep the number of distinct values small, because each value adds a metric series. For the behavior of each mechanism, refer to Client tags.

Tags that you pass on a single generation, and generation metadata, aren’t metric labels, so they don’t appear in the picker. Filter conversations by metadata.<KEY> instead.

Your selection is remembered per chart area.

Identify performance issues

Use the performance dashboard to spot problems:

• High latency: filter by agent or model to find slow generations. Drill into traces for specific conversations to identify bottlenecks.
• Error spikes: the error rate panel shows failures over time. Click through to conversations with errors to inspect the call_error payload.
• Slow time to first token: for streaming agents, the TTFT panel reveals which models or prompts have poor streaming performance.

Inspect tool failures

On the Tools page (Analytics > Tools), review tool executions, error rates, and latency.

Select Error count to chart the number of failed tool executions instead of the percentage shown by Error rate. You can also select Error count in the Tools area under Analytics; its table ranks tools by the highest error count.

Click Tool, Executions, Exec %, Errors, Error rate, or Latency to sort the table. Click the same header again to reverse the order. The sort menu and column headers control the same ordering.

Analyze skill operations

Use skill analytics to compare named loads, resource reads, and commands reported by the SDK, including failures. The view reads the existing tool-duration histogram from Prometheus. These operations don’t establish successful loads, installed skills, active state, or influence on later model calls.

Open the skills overview at /a/grafana-agento11y-app/analytics/skills in your Grafana URL. The view requires the sigil.skills-analytics feature toggle and a configured Prometheus data source. If the toggle is disabled, the URL redirects to Analytics.

Tempo is optional for metrics and agent suggestions. Configure Tempo to open related recorded conversations and traces from a skill’s detail page. Those actions remain available when metrics are unavailable, provided you have a valid skill name and Tempo configured.

Overview and activity

The skills overview summarizes skill usage across your agents:

• Skills: distinct skill names with positive estimated activity.
• Executions: estimated tool executions associated with eligible skill operations.
• Errors: estimated executions with a nonempty error_type metric label.
• Error rate: percentage of those executions with a recorded error type.
• Avg P95: approximate 95th percentile latency from histogram buckets pooled across the selected skills, not an average of per-skill percentiles.

The activity chart uses the same controls as Tools. Choose one of four chart modes:

• Executions: estimated eligible tool executions and errors as separate series.
• Error rate: percentage of executions with a recorded error type.
• Error count: estimated executions with a recorded error type.
• Latency: approximate P50 and P95 duration.

To segment activity, choose a dimension from the breakdown menu: None, Skill, Agent, or Tool.

Activity points estimate counts over chart intervals. Chart sums can differ from whole-window totals because the counter estimator depends on grouping and sampling density. Missing chart measurements appear as gaps, not zeros. Expand About these metrics for the definitions, coverage limits, and counter-estimation behavior.

Skills table

The skills table lists returned skill names with the following measurements:

• Skill: the exact skill-name metric label. Click a name to open its detail page.
• Executions: estimated executions.
• Exec %: this skill’s percentage of estimated executions in the selected scope.
• Errors: estimated executions with a recorded error type.
• Error rate: this skill’s estimated errors divided by its estimated executions, before display rounding.
• P95 latency: approximate whole-window 95th percentile duration from histogram bucket increases.

Counts, errors, shares, and rates use the same accepted count distribution. A zero denominator leaves the rate unavailable. P95 uses an independent histogram query, not an average of chart percentiles.

Click any sortable column header to sort the table. By default, the table shows the first 50 skills. Click Show all skills to expand the list.

Filter and search skills

The filter bar provides controls specific to skill analytics. It doesn’t inherit filters or time ranges from the general analytics dashboard:

• Time range: defaults to the last 1 hour, with no three-hour maximum. Auto-refresh is disabled; click the refresh icon to request data again, including for absolute ranges. Invalid resolved ranges don’t issue queries.
• Search skills: enter a case-sensitive literal substring of the skill name. Search applies after 500 ms without typing.
• Agent: select an exact gen_ai_agent_name metric-label value, or enter a custom value. Suggestions use the selected range and other supported filters, excluding the selected agent. Suggestions don’t prove activity within the exact window. Custom entry remains available if discovery fails.
• Service: a skillService URL parameter filters the exact service_name metric label and appears as a removable chip.
• Source: a skillSource URL parameter appears as a removable chip, including empty or whitespace values. SDK metrics don’t include source URI. Aggregates and agent suggestions stay blocked until you explicitly remove this chip. Related trace searches retain the exact source scope.

Skill-name, agent, and service filters preserve exact metric-label values without trimming or interpreting encoded-looking text. Removing a source chip clears the source scope; changing another filter doesn’t remove it.

Drill into skill details

Click any skill name in the table to open its detail page (/analytics/skills/detail?name=<SKILL_NAME>). The detail page requires the exact skill name in the URL query parameter.

The detail view focuses on the selected skill:

• The KPI band displays Executions, Errors, Error rate, and P95 latency.
• The activity chart shows activity for the selected skill, with breakdown options for None, Agent, and Tool.
• Click Find related recorded conversations to browse related recorded conversations. This requires a conversation ID on the matching span and stored Agent Observability metadata.
• Click Search related recorded traces in Tempo Explore to search related raw traces, including tool-only traces without conversation metadata.
• Click Back to skills to return to the overview while preserving your active filters.

Trace destinations retain the exact supported skill name, service, agent, source, and selected time range. Links also retain your Grafana URL subpath. The metric-only execution kind doesn’t become a trace filter.

Related records aren’t an exact list of metric contributors. Export time can differ from execution time, and trace sampling, capture, retention, and identity normalization can change the population. For example, Python trims agent metric labels but preserves tool-span agent names. The link keeps the exact agent predicate rather than guessing another identity or removing the filter.

For recorded skill fields, tool payloads, and process details, refer to Browse and debug conversations.

Requirements and limits

Check producer coverage and measurement limits before interpreting results:

• Use an upgraded SDK client that adds gen_ai_skill_name and agento11y_tool_execution_kind to the existing operation-duration histogram. Older producers and standalone Go otelgenai don’t automatically supply these labels. SDK clients normalize skill names before export, omit blank names, and retain eligible identity on failures.
• Eligible observations require gen_ai_operation_name="execute_tool", a nonempty gen_ai_skill_name, and execution kind load_skill, read_skill_resource, or command. Generic tools and unnamed operations don’t qualify. Process exit codes don’t determine metric errors.
• Counts use gen_ai_client_operation_duration_seconds_count; percentiles use the classic gen_ai_client_operation_duration_seconds_bucket. Skill analytics doesn’t add a separate skill histogram, native-histogram fallback, or support for combining representations.
• To filter by service, promote the service.name resource attribute to the service_name metric label in your metrics pipeline. Promotion affects new ingestion, not historical samples. Don’t substitute job for service identity.
• You need data:read permission in Agent Observability.

Operation counts are estimates. The sparse-counter estimator uses counter increases, with a cumulative-value fallback when increases are absent for a group. It can omit singleton series in mixed groups, extrapolate changes, or substitute a lifetime cumulative value.

Cloud Adaptive Metrics can omit the fallback entirely. Empty results don’t establish zero skill use.

Histogram percentiles are approximate and can cover fewer observations than estimated counts. If bucket increases supply no finite measurement, latency remains Unavailable, even when counts exist. A finite zero duration is valid.

If an aggregate response contains warnings, the view identifies the data as incomplete and displays the annotations. Incomplete counts withhold summary totals, shares, rates, and dependent percentiles; partial row counts can remain visible. Partial activity can show labeled operation or error observations. Operation legends retain sums labeled as partial observations. Partial error-count legends omit totals so missing samples don’t appear as zero. Rates and percentiles remain unavailable.

Informational annotations also appear. A latency failure or warning doesn’t invalidate otherwise complete counts.

Longer windows can increase backend query cost despite coarse chart resolution. Runtime cost is unmeasured.

Optimize costs

The tokens and cost dashboard helps you find optimization opportunities:

• Cost by model: compare estimated cost across models and providers. Consider switching expensive calls to cheaper models where quality is acceptable.
• Cache efficiency: the cache read ratio shows how effectively prompt caching reduces token usage. Low cache rates may indicate prompts that change too frequently.
• Token usage trends: spot unexpected increases in token usage that may indicate prompt regression or unnecessary verbosity.

Select Cache hit rate in Usage and cost to compare prompt cache reuse over time. Break down by Agent, Model, or Provider to find differences. If you select Token type, the chart shows cache read and cache write token counts instead of a percentage. The conversations table also shows Cache hit rate so you can inspect individual sessions.

The overview cost total covers your selected time range. It uses recorded per-request costs where available and estimates earlier periods from token usage and catalog prices. Agents totals and Needs attention cost charts use the same approach. Recorded costs include your configured model prices. If no recorded costs are available, Agent Observability estimates the whole range. Token-based estimates don’t account for per-request long-context pricing tiers and can understate costs for long prompts.

Cost charts mark the transition from token-based estimates to recorded costs. A step at that marker doesn’t necessarily mean spending changed. Changing chart time buckets doesn’t change the cost total for the selected range.

When Cost shown at list prices appears, your selected filter or breakdown requires token-based estimates instead of recorded costs. Configured model prices and long-context pricing don’t apply to these estimates. A notice below the filters applies to the cost totals and charts on the page, not to the cost recorded for each conversation; a notice on a chart applies only to that breakdown. Remove the named filter or breakdown to return to recorded costs where available. For details, refer to Configure your model prices.

When you ask Grafana Assistant about costs from Analytics or Agents, it receives the page’s cost source and selected scope. Recorded costs include your configured prices. Estimates for older periods or unsupported labels use public list prices instead; check which source the answer uses before comparing figures.

Track quality

The quality dashboard visualizes evaluation scores alongside operational metrics:

• Score trends: monitor if quality improves or degrades after agent version changes.
• Score distributions: identify if responses cluster around high or low scores.
• Correlation: compare quality scores with latency and cost to find the right balance.

Use Prometheus metrics directly

If you need custom dashboards, query the Agent Observability OpenTelemetry metrics in Prometheus:

For custom cost queries, use these stack-scoped metrics from your configured Prometheus data source:

Multiplying aggregated token usage by current observed prices doesn’t reproduce recorded costs: historical rates, long-context tiers, and per-request fees can differ. Cost estimates aren’t provider invoices.

Reuse dashboard panels

On panels with More actions, click Open in Explore to open the Prometheus query with the selected time range. Click Add panel to dashboard to copy the panel into a Grafana dashboard. Adding panels requires a configured Prometheus data source, dashboard create or write permission, and Grafana’s add-to-dashboard integration.

Some panels also offer Create alert or links to related conversations and agents. Alert creation requires Grafana Alerting, alert-creation permission, and Grafana’s create-alert integration.

Ask scoped analytics questions

When Grafana Assistant is available, ask about the analytics page’s selected time range and filters, for example, cost, latency, errors, or tool usage. The page supplies this scope and its query context to Assistant.

A conversation can contain several model calls. Specify whether you want conversation totals or model-call totals so the answer uses the measure you need.

Open conversations from exemplars

Dashboard exemplars connect metric spikes to example conversations. When exemplars are enabled, histogram panels can show exemplar markers with links that open the matching conversation.

Use exemplar links to move from a latency or error spike to the conversation that produced it. This feature requires exemplars to be enabled for your stack.

Set up alerts

Create Grafana alerts on Agent Observability metrics to proactively catch issues:

• Alert on error rate exceeding a threshold.
• Alert on p95 latency exceeding SLO targets.
• Alert on estimated cost per day exceeding budget.
• Alert on evaluation score drops below a quality threshold.

Configure alerts in Grafana using the standard alerting workflow with the Prometheus data source.

Schedule recurring reports

Use reports to receive recurring summaries of your agents’ cost, latency, errors, or quality. You need Grafana Assistant available in your stack and permission to create automations.

1. On the Agent Observability analytics page, click Add report.
2. Describe the summary you want, for example, a weekly cost report, then click Continue in Assistant. You can also select a template under Starting points.
3. Review the report instructions, schedule, and delivery settings in Grafana Assistant, then create the automation.

Describing a report or selecting a template opens a draft; it doesn’t schedule a report until you create the automation. Click Manage automations to manage schedules, delivery, and run history in Grafana Assistant.

The Weekly cost digest compares recorded costs for the last seven days with the previous seven days. These costs use your configured model prices where available and catalog prices otherwise. Unlike dashboard totals, this report doesn’t estimate older periods from token usage. If it can’t read recorded costs, it reports No data instead of estimating spend.

Next steps

• Optimize cost and performance
• Browse and debug conversations