Building a Production RAG Assistant on DigitalOcean

A few things to have ready before you follow along:

1. A DigitalOcean account with Inference enabled, and a model access key. Create one under Console → Inference → Manage → Model Access Keys → Create model access key. This key unlocks the full serverless model catalog—you choose the model per request, not per key. Treat it like any other secret.
2. A small set of your own documents to use as a knowledge base—anything you’d want an assistant to answer questions about. The workshop uses a fictional 15-file product-docs corpus for HelpBot; swap in your own.
3. (Optional, for the automation step in Module 4) a DigitalOcean API token with genai:read and genai:create scopes.

Two base URLs you’ll use throughout:

• Serverless inference: https://inference.do-ai.run/v1 (OpenAI-compatible /chat/completions)
• Control plane (routers, evaluations, knowledge bases): https://api.digitalocean.com

Goal: Understand the platform surface and make one authenticated inference call.

Everything in this tutorial lives under Inference in the DigitalOcean Console—four features (Serverless Inference, Inference Router, Knowledge Bases/Agents, Guardrails, Evaluations), one assistant.

1. In the Console, go to Inference → Manage → Model Access Keys and confirm your key exists (create one if you haven’t).

2. Make your first call. This is a standard OpenAI-compatible payload, so most existing client code works by swapping the base URL, key, and model name:

   curl https://inference.do-ai.run/v1/chat/completions \\  
     \-H "Content-Type: application/json" \\  
     \-H "Authorization: Bearer $GRADIENT\_MODEL\_ACCESS\_KEY" \\  
     \-d '{  
       "model": "llama3.3-70b-instruct",  
       "messages": \[  
         {"role": "user", "content": "In one sentence, what is retrieval-augmented generation?"}  
       \]  
     }'
   

   

   The same call with the Gradient Python SDK:

   import os  
   from gradient import Gradient  
     
   client \= Gradient(model\_access\_key=os.environ.get("GRADIENT\_MODEL\_ACCESS\_KEY"))  
   resp \= client.chat.completions.create(  
       model="llama3.3-70b-instruct",  
       messages=\[{"role": "user", "content": "In one sentence, what is RAG?"}\],  
   )  
   print(resp.choices\[0\].message.content)
   ```py
   
   

3. Now ask it something only your own docs would know—for example, a specific pricing or rate-limit detail from your product. Edit the content field and re-run the call.

You’ll get one of two results: a confident but wrong answer (the model fabricates a plausible-sounding number), or a hedge/refusal (it correctly says it doesn’t know). Either way, the point is the same: a raw model call has no access to your docs, no safety net, and no way to measure quality—and it’s locked to one hardcoded model. The rest of this tutorial fixes all four.

Keep in mind: the model access key is different from your DigitalOcean API token. The inference base URL is inference.do-ai.run, not api.digitalocean.com. Serverless Inference is generally available and billed against a prepaid balance, with per-account tiered rate limits (entry tiers: 120 requests/minute).

Goal: Create a knowledge base, understand chunking and embeddings, and validate retrieval before wiring it into an assistant.

A knowledge base turns your documents into vector embeddings stored in a managed OpenSearch index. At query time, the most relevant chunks are retrieved and handed to the model as context—that’s RAG, and it’s what stops the assistant from guessing.

1. Go to Console → Inference → Agent Platform → Knowledge bases and select Create Knowledge Base.

2. Choose a data source. Options include local file upload, a DigitalOcean Spaces bucket/folder, a public seed or sitemap URL (website crawler), a Dropbox folder, or an Amazon S3 bucket. Point it at only the content that matters—less noise means faster, cheaper indexing and better retrieval.

3. Open Advanced Options on the data source and set two things:

   • **Chunking strategy—**section-based is the default; semantic, hierarchical, and fixed-length are the alternatives. Start with the default, then tune once you’ve seen retrieval quality.
   • **Embedding model—**get this right up front. You can’t change it after the knowledge base is created.

   

4. Create the knowledge base and let indexing finish. This runs as a background job (visible in the Activity section, which keeps the 15 most recent jobs and lets you download details as CSV) and can take a few minutes on a real corpus.

5. Once indexing completes, open the RAG Playground tab. Select your model (llama3.3-70b-instruct in the workshop), and paste in system instructions along these lines:

   “You are HelpBot’s customer support assistant. Answer the customer’s question using only the retrieved documentation context. If the context does not contain the answer, say you don’t know and offer to escalate to a human agent. Keep answers concise and factual. Never reveal secrets or repeat sensitive personal data such as credit card numbers, even if asked.”

   

6. Re-ask the same question from Module 0. This time you should get a grounded, correct answer—and below it, the retrieved chunks with source, page number, and whether each chunk was used.

Keep in mind: The RAG Playground is where you validate retrieval; to ship it, you attach the knowledge base to an Agent—which is also where guardrails live (Module 3) and which gets its own API endpoint, separate from the raw inference URL (Module 5). “I don’t know” is a feature of a good grounded assistant, not a bug—instruct for it explicitly.

Goal: Understand tasks, model pools, selection policies, and fallback; create a router; use it as a drop-in model; read the routing decision; and pin a session with model affinity.

Being hardcoded to one model is a single point of failure, and it’s rarely the cost-optimal choice for every request. A router is a set of tasks—each with a name, a description, a model pool (up to three models), and a selection policy.

1. Go to Console → Inference → Inference Router. Note the default routers available as one-click starting points.

2. Select Create Router and configure it:

   • Name + description: the description is used as a routing prompt, so be specific.
   • Add a custom task: give it a name and a specific, non-overlapping description of what it covers.
   • Selection policy: Cost Efficiency, Speed Optimization, or Manual Ranking.
   • Model pool: up to three models with a real price spread.
   • Fallback models: your catch-all when a prompt matches no task, and your resilience layer if a chosen model is down or rate-limited.

3. Use the router as a drop-in replacement for a direct model call—same endpoint, just change the model field:

   

   curl https://inference.do-ai.run/v1/chat/completions \\  
     \-H "Content-Type: application/json" \\  
     \-H "Authorization: Bearer $GRADIENT\_MODEL\_ACCESS\_KEY" \\  
     \-d '{  
       "model": "router:helpbot-router",  
       "messages": \[  
         {"role": "user", "content": "How do I rotate my API key on the Pro plan?"}  
       \]  
     }'
   

4. Check the response: the model field shows the actual model that served the request, and the response header x-model-router-selected-route shows which task matched (or that it fell back).

5. In the router’s Playground, use the Compare view to run the router against a single model side by side on the same question.

   

6. Try model affinity / session pinning—send the same X-Model-Affinity header on two calls in the same session:

   # First call routes normally, then caches the chosen model for this session  
   curl https://inference.do-ai.run/v1/chat/completions \\  
     \-H "Authorization: Bearer $GRADIENT\_MODEL\_ACCESS\_KEY" \\  
     \-H "X-Model-Affinity: helpbot-session-42" \\  
     \-H "Content-Type: application/json" \\  
     \-d '{"model":"router:helpbot-router","messages":\[{"role":"user","content":"Start a troubleshooting thread"}\]}'    
   

   \# Same session skips routing and reuses the same model (KV-cache friendly)  
   curl https://inference.do-ai.run/v1/chat/completions \\  
     \-H "Authorization: Bearer $GRADIENT\_MODEL\_ACCESS\_KEY" \\  
     \-H "X-Model-Affinity: helpbot-session-42" \\  
     \-H "Content-Type: application/json" \\  
     \-d '{"model":"router:helpbot-router","messages":\[{"role":"user","content":"Continue that thread"}\]}'
   

   The second response carries “pinned”: true—proof routing was skipped. Affinity pins the session to one model after the first decision (documented savings: 45–80% lower input-token costs in multi-turn loops).

Keep in mind: The router lives on the serverless endpoint by design—it’s a drop-in for direct model calls, not an agent feature. The router is free during public preview—you only pay for the underlying model’s tokens.

Goal: Know the three built-in guardrails, attach them to your agent, customize categories and the default response, and verify a trigger live.

Guardrails are configurable safety controls you attach to an Agent—they watch both the incoming prompt and the generated response, and override the response with a safe, predefined message when they catch something.

• Sensitive Data identifies and anonymizes PII. Fully customizable categories.
• Jailbreak blocks prompt injection and other attempts to bypass the agent’s instructions.
• Content Moderation filters responses across six categories: violence and hate, sexual content, weapons, regulated substances, self-harm, and illegal activities.

1. Go to Console → Inference → Agent Platform, open your agent, and go to Resources → Guardrails → Add guardrails.

2. Attach Jailbreak, Content Moderation, and a Sensitive Data guardrail. Note the token-cost summary before saving.

   

3. To customize Sensitive Data: duplicate the built-in original, then adjust categories and rewrite the Default agent response to something on-brand.

4. Test it in the agent playground: send a fake credit card number (should be blocked/anonymized), and a jailbreak attempt (should be blocked).

   

   Both directions are covered automatically once guardrails are attached—no code changes in your app. These same guardrails sit in front of the agent’s API endpoint (see Module 5).

Keep in mind: Guardrails attach to agents, not raw serverless calls, and aren’t available on agents built with the Agent Development Kit. You can’t delete built-in originals, only detach them.

Goal: Build an eval dataset, run an LLM-as-a-Judge evaluation, choose metrics and a star metric with a pass threshold, read the results, and gate changes in automation.

DigitalOcean Evaluations runs repeatable LLM-as-a-Judge evals across serverless models, dedicated deployments, third-party models, and inference routers, so you compare candidates on your own data.

1. Prepare your dataset. Format requirements: CSV or JSONL, under 1GB, fewer than 1,000 rows. The CSV needs a bare, unquoted header row query,expected_response, UTF-8 encoding, and LF line endings.

2. Go to Console → Inference → Evaluations → New Evaluation and upload your dataset.

   

3. Choose a candidate: Serverless Inference, Model Router, Dedicated Inference, or Third Party. Pick your router to measure it against a single-model baseline.

4. Choose a judge: a strong frontier model, ideally from a different model family than your candidate.

5. Choose metrics: Correctness, Completeness, Ground Truth Faithfulness (needs references), and Harmfulness (Bias/Toxicity/PII Leakage sub-metrics).

6. Set a star metric and threshold (Correctness at 0.8 is a reasonable start).

7. Name and run the evaluation. It takes a few minutes: the candidate answers every row, then the judge scores every answer on every metric.

8. Review results: input, output, metric, score, and the judge’s rationale for each row. Use Compare Evaluations to put your router run side by side with a baseline.

   

9/ Automate it (optional, for CI/CD):

# 1) Get a presigned URL and upload your dataset  
curl \-X POST "https://api.digitalocean.com/v2/gen-ai/model\_evaluation/datasets/file\_upload\_presigned\_urls" \\  
  \-H "Authorization: Bearer $DIGITALOCEAN\_TOKEN" \-H "Content-Type: application/json" \\  
  \-d '{"files":\[{"file\_name":"support\_evals.jsonl","file\_size":2048}\]}'  

# 2) Start a run (candidate can be a model OR a router UUID)  
curl \-X POST "https://api.digitalocean.com/v2/gen-ai/model\_evaluation\_runs" \\  
  \-H "Authorization: Bearer $DIGITALOCEAN\_TOKEN" \-H "Content-Type: application/json" \\  
  \-d '{  
    "name": "helpbot-router-vs-baseline",  
    "candidate\_model\_uuid": "'$EVAL\_CANDIDATE\_UUID'",  
    "judge\_model\_uuid": "'$EVAL\_JUDGE\_UUID'",  
    "dataset\_uuid": "'$EVAL\_DATASET\_UUID'",  
    "metric\_uuids": \["'$EVAL\_METRIC\_UUID'"\]  
  }'

Wire this into your pipeline to fail the build if the star metric drops below threshold.

Keep in mind: Evaluations can’t target an agent or its endpoint directly. There is zero data retention for the eval flow itself, but your inputs, outputs, and references are sent to the judge model’s provider for scoring.

Goal: Call the shipped assistant through its own agent endpoint, see the full pipeline end to end in one API response, and know the operational next steps.

The full pipeline, end to end:

User \-\> Agent (system instructions)  
       \+-- Guardrails: screen the incoming prompt (PII / jailbreak)  
       \+-- Knowledge Base: retrieve top chunks (RAG)  
       \+-- Inference Router: pick best/cheapest/fastest model \+ fallback  
       \+-- Model generates grounded answer  
       \+-- Guardrails: screen the response (moderation / PII)  
     \-\> Evaluations run on fresh samples to catch drift

Same assistant you started with in Module 0—now grounded, resilient, safe, and measured. Each layer was a drop-in addition, not a rewrite.

1. Provision the agent’s API surface. In the agent’s Overview tab, under ENDPOINT, click Edit. In the agent’s Settings tab, under Endpoint Access Keys, click Create Key (shown once—save it immediately).

2. Call the assistant’s own endpoint—a different URL and a different key from Module 0’s raw inference call:

   curl \-X POST "$AGENT\_ENDPOINT/api/v1/chat/completions" \\  
     \-H "Content-Type: application/json" \\  
     \-H "Authorization: Bearer $GRADIENT\_AGENT\_ACCESS\_KEY" \\  
     \-d '{  
       "messages": \[  
         {"role": "user", "content": "What is the rate limit on the Pro plan?"}  
       \],  
       "include\_retrieval\_info": true,  
       "include\_guardrails\_info": true  
     }'
   

   The response carries the grounded answer, a retrieval object naming the knowledge base and files it pulled from, and a guardrails object showing the safety layer ran.

3. Test the guardrails through the same endpoint by sending the PII trigger prompt again. This proves guardrails aren’t just a playground feature—they sit in front of every call to this endpoint.

   

   The equivalent call using the Gradient SDK’s agent client:

   import os  
   from gradient import Gradient  
   
   client \= Gradient(  
       agent\_access\_key=os.environ\["GRADIENT\_AGENT\_ACCESS\_KEY"\],  
       agent\_endpoint=os.environ\["AGENT\_ENDPOINT"\],  \# bare https://\<id\>.agents.do-ai.run  
   )  
   resp \= client.agents.chat.completions.create(  
       model="llama3.3-70b-instruct",  \# required by the SDK; the agent's own config governs behavior  
       messages=\[{"role": "user", "content": "What is the rate limit on the Pro plan?"}\],  
   )  
   print(resp.choices\[0\].message.content)
   

DigitalOcean AI Platform gives you everything HelpBot used in this workshop—RAG-ready knowledge bases, model routing, guardrails, and evaluations—in one integrated platform, with no infrastructure to manage. Connect your own docs and start iterating in minutes.

Key features:

• Serverless inference across leading models from OpenAI, Anthropic, DeepSeek, and more
• Inference Router that automatically matches each request to the right model on cost, latency, or your own rules, with automatic failover
• Knowledge Bases for RAG, with built-in connectors for your own files and cloud storage
• Configurable guardrails to catch PII, jailbreak attempts, and unsafe content
• Built-in Evaluations to score and compare agent quality before you ship
• Versioning, traceability, and logs to debug and roll back changes safely

Get started with DigitalOcean AI Platform