Making an AI Agent Feel Fast on Serverless Inference

Telegram (long polling, no webhook, no public IP)
     │
     ▼
Hermes Gateway ── runs as a launchd/systemd service
     │
     ▼
Hermes Agent loop
 ├── SOUL.md ................ personality + intake rules
 ├── foodie-trail skill ..... workflow: intake → research → plan → narrate
 ├── Linkup (MCP, stdio) .... live web research
 └── text_to_speech tool .... ElevenLabs → Telegram voice notes
     │
     ▼
DigitalOcean Inference Engine (Serverless Inference)
https://inference.do-ai.run/v1  (OpenAI-compatible, 70+ models)

Notice that the system has two very different jobs, and they need very different hardware. The agent program itself is small: it receives a message, calls some APIs, and passes text around. Any cheap machine can do that. The expensive job is running the model, and that never touches the agent’s machine, because each model call goes out to the Inference Engine and gets billed per token. This is why the setup costs so little to run: you pay a few dollars a month for the small machine, and for GPU time only in the seconds the model is actually working.

Two things about this setup matter for everything that follows. First, the agent can run almost anywhere because the model does not run with it. Telegram bots use long polling, so nothing needs to reach the machine from outside. The agent works the same on a laptop or a Droplet. Second, every model call goes through one OpenAI-compatible endpoint. The Inference Engine serves models from Anthropic, OpenAI, Meta, DeepSeek, Qwen, and others behind https://inference.do-ai.run/v1 with a single access key. Changing models or providers is a config change, not a migration.

The step-by-step setup lives in DigitalOcean’s Serverless Inference docs and the Hermes docs, so this section covers only the essentials and the common mistakes.

Create a model access key in the control panel (under Serverless Inference) and check it works before anything depends on it:

curl https://inference.do-ai.run/v1/models \
  -H "Authorization: Bearer $DO_MODEL_KEY"

This returns the live model catalog. Trust it over any blog post, including this one, because catalogs change. From that response on this build: anthropic-claude-5-sonnet reports a 1,000,000-token context and 128,000-token max output, and anthropic-claude-haiku-4.5 is available as a cheaper option for testing.

Three rules for picking a model for agent work.

Pick for tool-calling reliability, not benchmark scores; an agent that fumbles function calls invents restaurants instead of researching them.

Check the max output limit; several strong catalog models cap at 8K tokens, and a cut-off itinerary fails in a confusing way. Keep a cheap model for iteration, because you will re-run test prompts dozens of times.

Also, skip the catalog’s router:* entries while debugging; you want the same model on every run.

Wiring the framework: install Hermes, point it at the endpoint as a custom OpenAI-compatible provider (pick Chat Completions mode explicitly instead of auto-detect), and connect Telegram through the built-in gateway (Hermes Telegram docs). Two traps from this build are worth knowing: after switching models in chat, /model <id> --global is required or the gateway starts with the old model, and since the bot can run commands on its host, keep the default deny-unknown-users behavior and allow only your own Telegram ID.

Research and voice are configurations, and both choices are worth explaining.

Linkup, because restaurant data goes stale fast, and the agent needs a search API that returns current results with sources, built for machine use. Any similar search API would slot in the same way. MCP instead of a hand-written integration, because Hermes finds the server’s tools at startup and registers them like its own. Swapping search providers later is a config change. In ~/.hermes/config.yaml:

mcp_servers:
  linkup:
    command: "npx"
    args: ["-y", "linkup-mcp-server"]
    env:
      LINKUP_API_KEY: "${LINKUP_API_KEY}"
    supports_parallel_tool_calls: true

tts:
  provider: "elevenlabs"
  elevenlabs:
    voice_id: "pNInz6obpgDQGcFmaJgB"
    model_id: "eleven_multilingual_v2"

Keys live in ~/.hermes/.env. The ${LINKUP_API_KEY} placeholder is filled in at runtime, so secrets stay out of the config file you might share. And supports_parallel_tool_calls: true tells the framework the searches are safe to run at the same time. That one line becomes the biggest speed win below.

Everything above is plumbing anyone can copy. The agent’s behavior lives in two files, and the choices inside them are also speed choices.

SOUL.md sets the personality, and the personality is a product requirement. Foodie profiles the person before planning the trip (street-food hunter, biryani purist, vegetarian, café hopper, fine dining), guesses the profile from clues when it can, and asks at most one question when it cannot. The one-question limit helps in two ways. People lose patience and quit when a bot keeps asking questions before doing anything. And each question makes the whole run slower: the agent has to wait for the person to reply, then process that reply with another model call, before any real work starts.

The foodie-trail skill sets the workflow, and its most important lines are limits. At most six searches per itinerary, with the queries written out in the skill. No restaurant in the output unless a search result confirmed it. Each day is a walkable route through one neighborhood. Narration scripts are 60 to 90 seconds, written to be spoken. The search limit matters beyond cost: an agent with no limit keeps searching to settle its own doubts, and every extra round adds a full search-plus-model cycle. Writing the queries into the skill turns an open-ended search into a fixed set of steps that can run in parallel.

Two numbers shape how the agent feels, and they are improved in different ways. Time to first token (TTFT) is how long before the user sees anything. Total generation time is how long before the full result arrives. Mixing them up is the most common mistake in this kind of work.

Time to first token

For a single model call, TTFT is mostly network time plus prefill: the time the model spends reading your input before it writes anything. Prefill grows with input length. So TTFT is, more than anything, a question of how much you are sending.

Watch what enters the context on every turn. An agent framework quietly stacks the system prompt, personality file, memory, skill list, and chat history into each request. Hermes keeps skill descriptions short and only loads a full skill file when it is needed, which helps. Do your part: keep the personality file short (Foodie’s is under 400 words), keep skill descriptions brief, and let conversations reset (this build resets after 4 hours idle). A long conversation carrying 30K tokens of history adds real wait time compared to a 6K one, so the framework’s compression command and idle resets save time as much as money.

Match thinking time to the task. Reasoning models spend their first seconds on thinking the user never sees. For a turn that just reads a city and a taste profile out of a message, that is wasted waiting. Hermes can adjust this at runtime (/reasoning low). The pattern: planning steps deserve thinking time, acknowledgments do not.

Send a quick first line. The trick that matters most for a slow task: have the skill send one line (“Biryani purist, Bangalore, two days. Researching now.”) before its first search. The itinerary arrives no sooner, but the app stops feeling frozen. That is most of what users mean by fast.

Keep connections warm. In production, put the agent’s host in a region near the inference endpoint, and let the long-running gateway reuse its connections. The same goes for tools: npx starts the Linkup server cold on first use, so set the framework’s recycling options generously for tools you need quickly. The first search of the day should pay the cold-start cost, not the first search of every conversation.

Use a small model for background jobs. Hermes sends side tasks like naming conversations and compressing history to a separately configured model. Point those at a Haiku-class model so slow calls you never see stay out of the way.

Total generation time

The full run is a pipeline: read the request, search, write the itinerary, write the narration, render the audio, deliver it all. The gains come from reshaping the pipeline, not from speeding up any single call.

Run the searches in parallel. The biggest single win. Six searches at 3 to 5 seconds each cost 20 to 30 seconds run one after another. Marked safe to run together, they take about as long as the slowest single search. The rule: any read-only tool your agent calls more than twice per task should be marked parallel-safe, and the queries should be written so they do not depend on each other, which is what lets them run together.

Cap the loop and write out the queries. You know how long research takes because you decided how long it takes.

Send the text first, render audio in the background. The user reads Day 1 while its audio renders. As far as they are concerned, the answer arrived when the text did. Per-day audio files beat one big file for two reasons: they overlap with reading time, and a text-to-speech failure costs one day’s audio instead of everything.

Keep the output short on purpose. Generation time grows with output length, so wordiness is slowness. The skill’s format rule (each day under about 15 lines) roughly halves writing time compared to what the model produces without instruction, and it reads better on a phone anyway. When you do need long output, make sure the model’s output limit fits it in one pass. Hitting the limit and continuing in a second call is the slowest way to generate text.

Measure before you tune. The bottleneck is rarely where you assume. On this build it was audio upload on a home connection, not the model.

This agent runs on serverless inference, and at this scale that is not a compromise. It is the correct choice. Hosting it on a GPU machine would mean renting an H100 GPU Droplet at around $3.39 per hour, roughly $2,500 a month, to pay for a GPU that sits idle while all the real work happens per token on the Inference Engine. The $12 Droplet does the same job.

Self-hosting starts to make sense in four situations. The first is cost: once your monthly token bill gets close to the flat cost of renting a GPU, the math flips, because a self-hosted open-weight model costs the same whether it serves one request or a million. The second is data control: if prompts cannot leave your infrastructure for compliance or privacy reasons, per-token APIs are off the table no matter the price. The third is a custom model: if you have fine-tuned weights for your domain, you need somewhere to serve them. The fourth is latency: an API gives you typical performance, while your own endpoint gives you a floor you control.

The downsides are just as real. The fixed cost runs whether traffic comes or not, and GPU Droplets bill even while powered off. You take on running the serving stack yourself: vLLM or similar, model updates, scaling, monitoring. And open-weight models still trail frontier models on tool-calling reliability, which matters more for agents than for most workloads. An agent that saves money on tokens but fumbles function calls is a bad trade.

There is a middle step that skips most of the operational cost: the Inference Engine’s Dedicated Inference tier provides a dedicated GPU endpoint with bring-your-own-model support, managed by the platform. For most teams, the sensible path is serverless first, dedicated when the numbers or compliance rules demand it, and fully self-hosted only when you need control the managed options cannot give. Because the whole stack speaks the same API, each move is a base URL change, not a rewrite.

The agent stack is settling into a familiar shape: an open-source runtime you install, a managed model service you rent, outside abilities like search and voice attached through a shared protocol, and on top, the thin layer that is actually yours. In this build that layer was two Markdown files of about a thousand words, and it was also, clearly, the product.

Where your models run is becoming a normal infrastructure decision, made on latency, model choice, and price, and easy to reverse because the OpenAI-compatible API has become the shared standard. And speed, not model quality, is now where products separate. The levers in this piece are ordinary engineering habits: send less context, run searches in parallel, deliver text before audio, use small models for background work, measure before tuning. The model is the easy part now. The experience is the work.

Stack: Hermes Agent (MIT), DigitalOcean Inference Engine — Serverless Inference, Basic Droplets, Linkup, ElevenLabs.