Notes from production.

1. Why logs don't help when an agent fails

   The log shows the action happened, which parameters it used, what came back. The one thing it doesn't show is why those parameters were chosen.

   • Agentic AI
   • Production

2. Filtering, not benchmarks, decides your vector database

   Comparison tables open with query latency. On our deployment those numbers decided nothing at all.

   • RAG
   • On-Premise

3. In RAG the problem isn't the answer, it's the document

   The complaint arrives as "the model is making things up". Look closely and it isn't. It is summarising the document it was handed, accurately. The document is wrong.

   • RAG
   • Production

4. Where an MCP server belongs on a corporate network

   MCP genuinely lowers integration cost. But what you are standing up is, by definition, a service with programmatic access to internal systems.

   • AI Security
   • On-Premise

5. Before production: where the first 50 eval cases come from

   The hard part of building an eval set isn't choosing metrics. It's having nothing to measure yet.

   • Agentic AI
   • Production

6. Before you buy a bigger model: look at what's in the context

   Scaling the model when quality drops is the most expensive button on a self-hosted setup. It is usually not the button that needs pressing.

   • On-Premise
   • Production

7. One agent's mistake is the next agent's input

   Every step in the chain does its own job correctly and the result is still wrong. Finding where it started takes longer than fixing it.

   • Agentic AI
   • Production

8. Don't give your agent a "run command" tool

   A tool signature is the definition of what an agent can do in your system. One broad tool erases the guarantees thirty narrow ones give you.

   • Agentic AI
   • AI Security

9. Where we drew the automation line

   Deciding how far the agent goes looks like a question about model quality. It isn't. It is a question about what breaks when it gets things wrong.

   • Agentic AI
   • Production

10. You can't solve prompt injection with a prompt

    The model is doing exactly what it was built to do - treat the text in front of it as instruction. Where that text came from is invisible to it.

    • AI Security
    • Agentic AI

11. Ollama or vLLM: we're asking the wrong question

    The choice usually gets argued along a "which one is faster" axis. The right axis is the concurrency profile.

    • On-Premise
    • Production

12. Errors should surface where they're made, not at the end

    The most expensive failure in an agent chain isn't producing a wrong output. It's a wrong output being taken for a right one and passed along.

    • Agentic AI
    • Production

No articles published yet.