Introducing DigitalOcean Signals and Simulations: see how your agents and models really behave in production, then test the fix before it ships

You ship an agent. It mostly works. Then the feedback loop becomes a support ticket: a user got the wrong answer, a tool call looped four times, a refund went out that shouldn’t have. Standard observability tells you the system was up and the latency was fine. It can’t tell you the agent chose the wrong tool, used stale context, or made a frustrated user give up. And when you do find the problem, writing realistic test cases to validate a fix means hours of digging through traces or inventing scenarios you hope resemble reality.

Today we’re launching two capabilities, part of DigitalOcean Agent Learning Services, that address both halves of that problem. Signals watches production traffic on DigitalOcean Managed Agents and DigitalOcean Inference Engine, and automatically highlights potential problems: frustrated users, failing tools, infrastructure constraints. Simulations, a new capability of DigitalOcean Evaluations, lets you test a change through realistic multi-turn scenarios, grounded in that same production traffic, on the same harness and tools your agent uses in production, before it reaches a user.

Both run natively on the infrastructure you already use to operate models and agents on DigitalOcean, and both feed the same Evaluations grader you already trust. Signals shows you what went wrong, and Simulations lets you test the fix before it ships.

DigitalOcean Signals: stop reading transcripts to find out what broke

Agent and inference failures often look technically successful. The tool call returned 200; it just returned the wrong region. The conversation completed; the user left annoyed. Signals catches that class of failure at scale by leveraging the infrastructure provided by DigitalOcean Inference Engine, Harness Runtime and Action Gateway. Signals stitches conversation turns into sessions and classifies each one across three dimensions. Interaction signals that track user sentiment (corrections, escalations, users quitting), Execution signals that track tool reliability (invalid arguments, retry loops), and Environmental signals that track infrastructure (timeouts, rate limits, context overflow). Trends show up on the Signals view on Harness Runtime and Serverless Inference, and you can drill from any trend into the sessions behind it.

Raw trace volume can exceed 100 GB/s, so grading every session with an LLM judge is prohibitively expensive and manual review doesn’t scale at all. Signals instead runs an inexpensive first-pass triage to find the sessions worth a closer look. The method, described in our paper Signals: Trajectory Sampling and Triage for Agentic Interactions, achieved an 82% informativeness rate and a 1.52x efficiency gain over random sampling in a controlled study on the τ-bench benchmark. Flagged sessions become evaluation scenarios in one click, so a cluster of wrong-region database queries becomes a targeted eval with no manual test-data prep. Signals is opt-in, is designed to redact PII before processing, and keeps traces and trace-derived evaluations within your DigitalOcean Team boundary; they’re never used for cross-customer model training. It works best when your agent runs on Managed Agents and your models use DigitalOcean Inference Engine.

Simulations in DigitalOcean Evaluations: test the whole conversation, not one response

Evaluations scores agent and model performance against defined test cases, which is the right check for single responses. But agents fail across turns: a prompt tweak fixes one step and quietly breaks a later one, a fine-tuned checkpoint scores well on its training objective and regresses on real multi-turn behavior. Hand-built test cases are narrow, synthetic sets aren’t grounded in your agent or your users, and live traffic is realistic at the cost of real mistakes. Simulations give you that validation layer without the tedious build. It starts with crafting the right scenario to test your model or agent candidate: every scenario starts from your agent’s configuration, so it only exercises goals and tools the agent actually supports. When Signals is enabled, scenario generation is weighted toward where production traffic already shows friction and toward what your change is most likely to affect; when there’s no production data yet, guided refinement builds a baseline set from your configuration. Or you can just describe one in plain language: “a refund request from an angry customer with missing order details.”

A user simulator plays the persona turn by turn against your candidate, which runs in an isolated sandbox on Harness Runtime with the same configuration and Action Gateway tools it uses in production. Tools that write data or cost money can run as mocks, so tests don’t need to touch your live systems. Each scenario plays several times, and each play is an episode: one scenario against one candidate, producing one trajectory. The user simulator can take a different path each time, which is how a single scenario surfaces several ways a conversation can go. An LLM judge scores every trajectory on Evaluations metrics such as user-goal completion, knowledge retention, and tool-use accuracy, plus any custom metrics you define. Because each trajectory records every tool call, Simulations can tell a reasoning failure from a tool-use failure, and running a baseline against a candidate shows exactly what improved and what regressed. It can be used to validate any change, from instructions and tool selection to fine-tuned or RL-trained checkpoints, on any harness supported by Harness Runtime, including Claude Code, Codex, and OpenCode. Third-party agents such as LangChain, Hermes, and CrewAI aren’t supported in this preview, and if you only want to compare two models on fixed prompts, single-turn Evaluations remains the right tool.

In this preview, episodes are created by a simulation run: you pick a scenario set and a candidate, and the run plays every scenario the number of times you set. The three pieces are already separate underneath, which is where this is going. We’re working toward letting you run an episode on its own, any one scenario against any one agent or model, scored the same way, so you can change one thing and re-test without starting a new run.

How Signals and Simulations work together

At a high level, the loop runs like this:

• You run an agent on Managed Agents. DigitalOcean Harness Runtime and DigitalOcean Action Gateway already emit events as part of their architecture. There’s nothing extra to instrument.
• Signals picks up those events and stitches individual turns into sessions, triages them cheaply to find the informative ones, then annotates them against a three-part taxonomy: Interaction, Execution, and Environmental. Trends land in the Signals tab on the Harness Runtime page.
• Flagged sessions become evaluation scenarios in one click.
• Simulations takes those scenarios, or ones you describe in plain language, and runs your candidate change through them as full conversations in an isolated sandbox, one episode per play, and then grades each run.
• Ship the validated change. Signals keeps watching. Scenarios worth keeping become regression tests for the next change.

Get started

DigitalOcean Signals and Simulations are now available in Public Preview. Turn on Signals for an agent you’re already running, and run your next change through Simulations before your users do.

Simulations is fully integrated with DigitalOcean Agent Droplets plans, allowing you to leverage your plan’s discounts and monthly allowances when testing agents. Agent Droplets plans now also apply discounts (Pro: 15% discount and Team: 20% discount) to scenario generation, user simulation and LLM judging during simulations.

Enable Signals

1. In the DigitalOcean Console, access Signals in the Data & Learning section. Alternatively, you can go to Signals from Harness Runtime -> Signals.
2. Enable Signals for one or more Managed Agents. Data processing can take up to an hour to begin.
3. Review trends in user sentiment, tool execution, and infrastructure, then open any flagged session and export it to Evaluations as a test scenario.
4. You can also download Signals data to fine-tune your models or use it as reward and preference data for reinforcement learning.

Run a Simulation

1. In the DigitalOcean Console, go to Evaluations.
2. In Simulation Scenarios, pick a pre-defined scenario, describe one in plain language, or pull one from production traces via Signals.
3. Click Run Simulation. Then select your candidate, the user simulator model, and the metrics to grade against. Set episodes per scenario and the turn limit.
4. Run, then review graded scores and the full trajectory for each episode. Save the scenarios that matter to your library as regression tests. Once a run has used a scenario set, the set locks so every past run still reads against the exact scenarios it played; duplicate the set to revise it. Scenario versioning is next, so you can edit in place and keep each run tied to the version it used.