Agentic AI is has come a long way in the past few years. LangGraph, tool calling, retrieval, workflow-aware models, the whole lot. But it isn’t just the what we deploy which has changed, it’s also how we deploy it.
Gone are the days where every agent needed to be stuffed into a container running on a server somewhere, requiring individual management and governance. Databricks’ Mosaic AI Agent Framework and Model Serving has come along and fixed that by providing a first-class path to running proper agentic systems in production without duct-taping together servers, containers, and cron jobs.
But the developments in this space haven’t stopped there. As agents have become more sophisticated, the limitations of previous design patterns have become very apparent. New questions have arisen such as:
- How do we more efficiently pass data to, from, and between agents?
- How do we govern what these agents are meant to be able to access?
- How do we make sure our agents can scale effectively?
So today, we’re looking into how to serve and govern these agents the right way, what on Earth makes a ResponsesAgent different to a ChatAgent, and why serverless functions alone simply aren’t enough.
Let’s get into it!
ResponsesAgent vs ChatAgent
If you’ve spent any time building with LLMs, you’re likely familiar with Chat Models and Chat Agents, the classic message‑in, message‑out conversational pattern. They’re ideal for assistants, copilots, chatbots, and any UI where the user is meant to “talk” to the model.
Recently, though, you may have noticed a new term appearing in tooling and documentation: ResponsesAgent. This newer interface is now the recommended approach for production‑grade deployments on Databricks.
A ResponsesAgent standardises how an agent receives inputs and produces outputs. It enforces structured schemas, provides predictable response shapes, and integrates cleanly with Mosaic for routing, logging, tracing, and evaluation. In short, it acts as the production‑ready contract for agentic workloads.
Here’s the practical takeaway:
- ChatAgents are great for conversational interfaces and chat‑centric UIs.
- ResponsesAgents are built for systems that need to scale reliably, evaluate output, ensure consistency, or operate deterministically
If you’re deploying to Mosaic AI Model Serving, a ResponsesAgent is almost always the right choice.
Authenticating Agents
When you’re developing locally, authentication feels like an afterthought. But when you're shipping a production agent? It’s the difference between a smooth, reliable agent endpoint, and your Team inbox blowing up because “the bot stopped working again”.
Databricks endpoints enforce authenticated access using workspace identities and service principals. The challenge is token rotation, whereby if you're calling your endpoint from an app, you must ensure credentials stay fresh, otherwise you may end up in a situation where a RAG agent is deployed to Model Serving, works fine for one hour, then fails due to expired credentials.
However, Service Principals still require some level of management and extra management overhead. But worry not, because we have another trick up our sleeves – automatic authentication!
Automatic authentication allows you to simply provide a list of resources at the point of logging your agent in MLflow, at which point a Databricks-managed service principal will be created which grants the agent access to those resources (only if the person deploying the agent has access to them as well). This is kind of like a On-Behalf-Of model, except going forwards the agent will still operate under its own identity, just with permissions that you have allowed it to have.
These automatic service principals re fully managed for you, which means you don’t have to worry about them at all after the moment of logging your agent. When you retire your agent, the SP disappears and all permissions get revoked!
Why Endpoints Beat Serverless Functions
Many teams try to deploy agents using AWS Lambda or Azure Functions. And sure, they can work for simple inference. But agentic AI is not simple inference, so here’s why Mosaic endpoints win:
Cold Starts Will Destroy Your Latency Budget
Cold starts on serverless functions can be wildly inconsistent, especially when loading model runtimes and the plethora of supporting tools and helper functions. Databricks endpoints are warm, optimized, and designed for high-throughput LLM inference. Even if you have them set to scale-to-zero when not in use, they are optimised for fast spin up times and provide rigid status checking endpoints to allow you to communicate their status to users – and other agents.
Agents Need Stateful Tracing and Observability
Mosaic AI Agents have built-in:
- Tracing
- Step‑level logs
- Evaluation tools
- AI judges
- MLflow integration for metrics and quality evaluations
Sure, these can be stitched together on your serverless function, but each one will need to be individually maintained, configured, and updated whenever anything changes. Databricks, on the other hand, abstracts all of this away for you through great services such as the Unity AI Gateway.
Autoscaling That Actually Understands Model Workloads
As mentioned earlier, Mosaic Model Serving endpoints support:
- Automatic scaling
- Scale‑to‑zero
- Multi‑model routing
- Versioned deployments
- Readiness states (READY, SCALING, STOPPED, etc.) built directly into the platform.
Serverless functions don’t know they’re running AI agents, and therefore they treat your model as arbitrary compute. That means instability, inconsistent cold starts, and no model lifecycle management. Furthermore, if you want to deploy a new agent using LLMOps best practices, that multi-model traffic routing is going to make your canary deployments a lot easier…
Secure, Governed, UC-Integrated
Agents and tools can be registered in Unity Catalog and deployed with fully governed lineage and access control. Standard serverless functions on the other hand just execute code, they don't give you:
- Fine‑grained permissions
- Lineage tracking
- Model version governance
- Enterprise-level auditing
If you want to track anything about who is using your agent, what they’re allowed to do, or where that information gets stored, that’s a whole load of extra overhead you’re going to have to manage yourself.
Native Support for Multi-Step Agents
Mosaic is designed specifically to handle and enable multi-step agents with a lot of moving parts and processes. Tool calling, multistep reasoning, workflow planning? These things all take time, and depending on the nature of the agent and the request, latency can be rather unpredictable.
Mosaic was designed with this in mind. Serverless functions on the other hand? One word, timeouts.
Conclusion
Agentic systems are only going to get more complex, with more tools, more steps, and more orchestration. But Databricks’ Mosaic AI endpoints give you the operational backbone to deploy them with real-world reliability.
Here’s what a clean production workflow looks like today on Databricks for deploying your agentic systems:
- Author your agent using LangChain / LangGraph, the OpenAI API, or even DSPy. Databricks supports all three.
- Wrap it up as a ResponsesAgent for standardised I/O and easier serving.
- Deploy to Mosaic AI Model Serving to get a scalable, monitored endpoint with autoscaling, versioning, and UC governance.
- Authenticate your agent to access the resources it needs.
- Use Agent Evaluation for high-quality monitoring and iterative improvement (AI judges, metrics, feedback loops).
This is how you take an agent from “cool demo” to “actual enterprise product”. If you're building serious production AI, Mosaic AI endpoints aren’t optional - they’re foundational.
Author
James Bentley