Building a prototype AI agent that runs on your laptop is one thing. Moving that agent into production where it handles real workloads, scales horizontally, recovers from failures, and integrates with other services is entirely different. The real challenge isn’t the model or the reasoning loop, it’s the infrastructure that keeps it running reliably at 3 AM when you’re asleep.
This guide covers the practical patterns we use to deploy stateful AI agents at scale using Docker and Kubernetes. You’ll learn how to containerize agents properly, manage their lifecycle in orchestration, handle tool invocation isolation, scale replicas with load balancing, implement health checks, and debug distributed workflows. These patterns apply whether you’re running LLM-based agents, tool-using systems, or multi-agent workflows.
Why Containerization and Orchestration Matter for AI Agents
AI agents are stateful services. They maintain context, manage tool state, handle long-running workflows, and need to recover gracefully from failures. Unlike stateless microservices, agents can’t simply be killed and replaced without losing important state or mid-execution work.
Docker gives you reproducible, isolated environments where your agent and all its dependencies (LLM SDKs, vector databases, custom tools) run consistently across development, staging, and production. Kubernetes adds orchestration, automatic scaling, health management, and rolling updates. Together, they let you treat agents as reliable infrastructure rather than fragile scripts.
The key challenges are different from typical microservices:
- Agents often hold state across requests (conversation history, tool results, execution context)
- Tool invocations need isolation to prevent one agent’s tool failure from crashing the entire container
- Graceful shutdown is critical: you can’t forcibly kill an agent mid-workflow
- Debugging distributed agents requires proper logging and tracing, not just container logs
- Resource allocation is harder to predict because agent behavior depends on model outputs and tool complexity
Containerizing Your AI Agent
Start with a clean Dockerfile that captures your agent’s runtime environment. Here’s a practical template:
FROM python:3.11-slim
WORKDIR /app
# Install system dependencies for tools and libraries
RUN apt-get update && apt-get install -y \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy requirements and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy agent code
COPY . .
# Create non-root user for security
RUN useradd -m -u 1000 agent
USER agent
# Health check endpoint
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
# Run agent service
CMD ["python", "-m", "uvicorn", "agent_service:app", "--host", "0.0.0.0", "--port", "8000"]
Key points: use a slim base image to keep the container size manageable, install only what you need, run as a non-root user for security, and include a health check that Kubernetes can use to monitor the agent’s status.
Your agent should expose an HTTP API, even if it’s internal. This makes it observable and controllable from Kubernetes. A minimal FastAPI wrapper looks like this:
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import logging
app = FastAPI()
logger = logging.getLogger(__name__)
class ExecuteRequest(BaseModel):
task: str
context: dict = {}
class ExecuteResponse(BaseModel):
result: str
state: dict
@app.post("/execute")
async def execute_task(request: ExecuteRequest) -> ExecuteResponse:
try:
result, state = await agent.run(request.task, request.context)
return ExecuteResponse(result=result, state=state)
except Exception as e:
logger.error(f"Execution failed: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.get("/health")
async def health_check():
return {"status": "healthy", "ready": agent.is_ready()}
@app.post("/shutdown")
async def graceful_shutdown():
await agent.cleanup()
return {"status": "shutting down"}
This gives Kubernetes three critical endpoints: execute for your main workload, health for monitoring, and shutdown for graceful termination.
Managing Agent Lifecycle in Kubernetes
A Kubernetes Deployment for your agent might look like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ai-agent
spec:
replicas: 3
selector:
matchLabels:
app: ai-agent
template:
metadata:
labels:
app: ai-agent
spec:
containers:
- name: agent
image: my-registry/ai-agent:1.0.0
ports:
- containerPort: 8000
env:
- name: LOG_LEVEL
value: "INFO"
- name: MODEL_NAME
valueFrom:
configMapKeyRef:
name: agent-config
key: model-name
- name: API_KEY
valueFrom:
secretKeyRef:
name: agent-secrets
key: api-key
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "2Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 5
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15 && curl -X POST http://localhost:8000/shutdown"]
terminationGracePeriodSeconds: 30
Let’s break down the critical parts:
Replicas and scaling: Three replicas give you redundancy and horizontal scaling. Kubernetes will restart failed containers automatically.
Resource requests and limits: Requests tell Kubernetes what to reserve for this pod; limits prevent runaway resource consumption. For agents, be conservative with memory because LLM operations can spike.
Liveness and readiness probes: Liveness detects dead containers and restarts them. Readiness tells the load balancer when the agent is ready to receive traffic. The readiness probe should check not just HTTP connectivity but also whether the agent’s dependencies (database, model cache) are available.
Graceful shutdown: The preStop hook gives the agent time to finish in-flight requests before Kubernetes sends SIGTERM. The terminationGracePeriodSeconds sets an upper limit. This is crucial for agents because abrupt termination can corrupt state or lose work.
Handling Tool Invocation Isolation
One of the hardest problems with AI agents in production is that tool invocations can fail, hang, or behave unexpectedly. A single bad tool call shouldn’t crash the entire agent or block other agents in the same pod.
Implement tool invocation as a separate service or sidecar container. Here’s a pattern using a tool executor service:
import asyncio
from concurrent.futures import TimeoutError
import logging
logger = logging.getLogger(__name__)
class IsolatedToolExecutor:
def __init__(self, timeout_seconds: int = 30):
self.timeout_seconds = timeout_seconds
async def execute(self, tool_name: str, **kwargs) -> dict:
try:
# Look up the tool
tool = self.get_tool(tool_name)
if not tool:
return {"error": f"Tool {tool_name} not found", "success": False}
# Execute with timeout and exception handling
result = await asyncio.wait_for(
tool(**kwargs),
timeout=self.timeout_seconds
)
return {"result": result, "success": True}
except asyncio.TimeoutError:
logger.warning(f"Tool {tool_name} timed out after {self.timeout_seconds}s")
return {"error": f"Tool timed out", "success": False}
except Exception as e:
logger.error(f"Tool {tool_name} failed: {e}")
return {"error": str(e), "success": False}
def get_tool(self, name: str):
# Your tool registry
return TOOLS.get(name)
The agent calls the executor instead of tools directly. If a tool fails or times out, the agent gets a structured error response and can handle it gracefully. The agent doesn’t crash, and other agents aren’t affected.
For even stronger isolation, run tools in a separate container (a sidecar) so a tool crash doesn’t touch the agent container at all:
spec:
containers:
- name: agent
image: my-registry/ai-agent:1.0.0
# ... agent config ...
- name: tool-executor
image: my-registry/tool-executor:1.0.0
ports:
- containerPort: 8001
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "1Gi"
cpu: "500m"
The agent container calls the tool executor via localhost:8001. If the tool executor crashes, Kubernetes restarts just that container; the agent stays up.
Scaling Agent Replicas with Load Balancing
Expose your agents via a Kubernetes Service:
apiVersion: v1
kind: Service
metadata:
name: ai-agent-service
spec:
type: LoadBalancer
selector:
app: ai-agent
ports:
- protocol: TCP
port: 80
targetPort: 8000
The Service acts as a load balancer, distributing requests across all healthy agent replicas. Kubernetes automatically updates the Service endpoints when pods are added or removed.
For more control, use a Horizontal Pod Autoscaler to scale replicas based on metrics:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: ai-agent-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ai-agent
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
This scales the deployment to keep CPU around 70% and memory around 80%. If load increases, Kubernetes spins up new replicas; if load drops, it scales down.
Implementing Health Checks and Graceful Shutdowns
Health checks are your first line of defense. A proper health check endpoint should verify not just that the container is running, but that the agent is actually functional:
@app.get("/health")
async def health_check():
checks = {}
# Check agent state
checks["agent_initialized"] = agent.initialized
# Check dependencies
try:
checks["database"] = await db.ping()
except Exception as e:
checks["database"] = False
logger.warning(f"Database check failed: {e}")
try:
checks["model_loaded"] = agent.model is not None
except Exception as e:
checks["model_loaded"] = False
logger.warning(f"Model check failed: {e}")
all_healthy = all(checks.values())
status_code = 200 if all_healthy else 503
return {
"status": "healthy" if all_healthy else "degraded",
"checks": checks
}, status_code
Graceful shutdown is equally important. When Kubernetes sends a SIGTERM signal (which happens during rolling updates or scale-downs), you have a limited window to finish work cleanly:
import signal
import asyncio
class Agent:
def __init__(self):
self.active_tasks = set()
self.shutting_down = False
async def shutdown(self):
logger.info("Graceful shutdown initiated")
self.shutting_down = True
# Give active tasks time to complete
while self.active_tasks and time.time() < deadline:
logger.info(f"Waiting for {len(self.active_tasks)} tasks to complete")
await asyncio.sleep(1)
# Cancel remaining tasks
for task in self.active_tasks:
task.cancel()
# Clean up resources
await self.cleanup_resources()
logger.info("Shutdown complete")
# In your FastAPI app
if __name__ == "__main__":
agent = Agent()
def handle_shutdown(signum, frame):
asyncio.run(agent.shutdown())
exit(0)
signal.signal(signal.SIGTERM, handle_shutdown)
signal.signal(signal.SIGINT, handle_shutdown)
The preStop hook in your Kubernetes config should call an explicit shutdown endpoint to trigger this gracefully before the container is terminated.
Debugging Distributed Agent Workflows
When agents are distributed across multiple containers and replicas, debugging becomes harder. Structured logging is essential:
import logging
import json
from uuid import uuid4
class JSONFormatter(logging.Formatter):
def format(self, record):
log_entry = {
"timestamp": self.formatTime(record),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
"trace_id": getattr(record, "trace_id", None),
"pod_name": os.getenv("HOSTNAME"),
}
if record.exc_info:
log_entry["exception"] = self.formatException(record.exc_info)
return json.dumps(log_entry)
# Add trace ID to every request
@app.middleware("http")
async def add_trace_id(request, call_next):
trace_id = request.headers.get("X-Trace-ID", str(uuid4()))
request.state.trace_id = trace_id
response = await call_next(request)
response.headers["X-Trace-ID"] = trace_id
return response
Structured logs let you search and correlate events across all agent instances. Use a centralized logging system like ELK, Datadog, or CloudWatch to aggregate logs from all containers.
Add tracing to understand request flow through your agent system:
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
@app.post("/execute")
async def execute_task(request: ExecuteRequest):
with tracer.start_as_current_span("execute_task") as span:
span.set_attribute("task", request.task)
with tracer.start_as_current_span("agent_run"):
result, state = await agent.run(request.task, request.context)
with tracer.start_as_current_span("save_result"):
await save_result(result, state)
return ExecuteResponse(result=result, state=state)
Distributed tracing shows you exactly where time is spent and where failures occur across your agent infrastructure.
Putting It Together: A Complete Example
Here’s a minimal but complete example of a production-ready agent deployment:
apiVersion: v1
kind: ConfigMap
metadata:
name: agent-config
data:
agent-config.yaml: |
model: gpt-4
temperature: 0.7
max_tokens: 2000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ai-agent
spec:
replicas: 3
selector:
matchLabels:
app: ai-agent
template:
metadata:
labels:
app: ai-agent
spec:
containers:
- name: agent
image: my-registry/ai-agent:1.0.0
imagePullPolicy: Always
ports:
- containerPort: 8000
env:
- name: LOG_LEVEL
value: "INFO"
volumeMounts:
- name: config
mountPath: /etc/agent
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "2Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 5
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10 && curl -X POST http://localhost:8000/shutdown"]
terminationGracePeriodSeconds: 30
volumes:
- name: config
configMap:
name: agent-config
---
apiVersion: v1
kind: Service
metadata:
name: ai-agent-service
spec:
type: LoadBalancer
selector:
app: ai-agent
ports:
- protocol: TCP
port: 80
targetPort: 8000
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: ai-agent-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ai-agent
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
Deploy this with kubectl apply, and you have a self-healing, auto-scaling, production-ready AI agent system.
Common Pitfalls and How to Avoid Them
Ignoring state persistence: If your agent holds state in memory, scale-down or pod restarts lose it. Use external storage (Redis, a database) for state that matters. Design your agent to be stateless where possible, or use persistent volumes for critical state.
Underestimating resource needs: LLM operations are memory-intensive. Test your agent under realistic load before setting resource requests and limits. Monitor actual usage and adjust.
Skipping graceful shutdown: If you force-kill agents mid-task, you lose work and corrupt state. Always implement preStop hooks and shutdown endpoints. Test rolling updates to make sure they work smoothly.
Poor observability: You can’t debug what you can’t see. Implement structured logging, distributed tracing, and health checks from day one. Don’t wait until production to figure out what’s failing.
Tight coupling between agent and tools: If a tool fails, the whole agent shouldn’t crash. Use the isolated tool executor pattern. Fail gracefully and let the agent decide what to do.
Next Steps
Start by containerizing your agent with the Dockerfile template above. Test it locally with Docker, then deploy a single replica to Kubernetes. Add the health checks and shutdown hooks, then test rolling updates. Once that’s solid, add multiple replicas and an autoscaler.
Invest in logging and tracing early. It pays off immediately when debugging production issues. Use the tool isolation pattern if your agents invoke external services or unreliable code.
Monitor your agents in production. Track response times, error rates, resource usage, and scaling events. Use that data to fine-tune replicas, resource limits, and autoscaling thresholds.
The patterns in this guide are proven in production systems. They’re not perfect for every use case, but they give you a solid foundation to build on. Start here, measure what matters for your workload, and iterate.
Can I run multiple agents in the same pod?
Yes, but be careful. Multiple agents in one pod share resources and share a fate (if the pod dies, all agents die). It can make sense for tightly coupled agents that must coordinate, but generally it’s simpler to run each agent in its own pod and let Kubernetes manage scaling. If you do run multiple agents per pod, give each its own port and implement proper resource isolation.
How do I handle agent state across pod restarts?
Store state externally. Use a database, Redis, or object storage for anything you need to survive a pod restart. Design your agent to reload state on startup. If state is small and doesn’t need to survive pod death, in-memory is fine, but be explicit about that tradeoff.
What if my agent needs persistent storage?
Use Kubernetes PersistentVolumes. Define a PersistentVolumeClaim in your Deployment and mount it as a volume. This gives your agent access to storage that persists across pod restarts. Be aware that PVs are often slower than in-memory storage, so use them only for data that truly needs persistence.
How do I update my agent without downtime?
Use rolling updates. Kubernetes updates one pod at a time, waiting for new pods to be ready before terminating old ones. Implement graceful shutdown so in-flight requests complete before termination. Test rolling updates in staging first to catch issues before production.
How do I debug an agent that works locally but fails in Kubernetes?
Check logs with kubectl logs. Add structured logging to your agent. Use kubectl exec to shell into a running pod and inspect its state. Check readiness and liveness probe failures in kubectl describe pod. Verify environment variables, config maps, and secrets are set correctly. Start with the simplest possible agent and add complexity incrementally, testing each step in Kubernetes.
