Building an LLM-powered chatbot that works in a production environment requires different architectural decisions than a prototype. You need to isolate tenant data, stream responses for a smooth user experience, track costs per customer, handle rate limits gracefully, and scale across concurrent users without breaking the bank.
I’ve shipped several multi-tenant chatbot systems at scale, and the transition from “it works locally” to “it works for 500 concurrent users across 50 tenants” involves real architectural choices. This guide walks through the patterns, code patterns, and operational decisions that matter.
The Core Challenge: Multi-Tenant Isolation at the LLM Layer
Single-tenant systems are simpler to build. You call OpenAI, Anthropic, or a local model, get a response, and return it. But in a multi-tenant SaaS product, you need to guarantee that tenant A’s conversation history never leaks into tenant B’s context, that rate limits are applied per tenant, and that costs are tracked accurately per customer.
The isolation starts at the connection level. Rather than a single LLM client for your entire application, you need per-tenant or per-organization client instances that carry tenant context through the entire request lifecycle.
// Bad: Single shared client, easy to mix up tenant context
const sharedLLMClient = new ChatOpenAI({
apiKey: process.env.OPENAI_API_KEY,
temperature: 0.7,
});
// Good: Tenant-aware client factory with context isolation
class TenantLLMClientFactory {
private clientCache = new Map();
getClientForTenant(tenantId: string, config: TenantLLMConfig) {
if (!this.clientCache.has(tenantId)) {
this.clientCache.set(
tenantId,
new ChatOpenAI({
apiKey: config.apiKey || process.env.OPENAI_API_KEY,
temperature: config.temperature || 0.7,
// Tenant ID in metadata for logging and cost tracking
metadata: { tenantId },
})
);
}
return this.clientCache.get(tenantId);
}
}
Caching clients per tenant reduces connection overhead and makes it easier to apply per-tenant rate limits and cost caps later. Each cached client carries its tenant identity, so any logs or cost tracking automatically know which customer is incurring the expense.
Streaming Responses: Why It Matters and How to Implement It
When a user sends a message to a chatbot, they expect to see the response appear word by word, not wait 5 seconds for the full response. This is streaming, and it’s not just a UX nicety. It reduces perceived latency, makes the interface feel responsive, and lets users start reading while the model is still generating.
In Node.js with Express or Fastify, you stream responses over HTTP using Server-Sent Events (SSE) or raw chunked transfer encoding. Here’s a practical pattern using LangChain’s streaming support:
import { Readable } from 'stream';
import { RunnableConfig } from '@langchain/core/runnables';
async function streamChatbotResponse(
req: Request,
res: Response,
tenantId: string,
userMessage: string
) {
const llmClient = clientFactory.getClientForTenant(tenantId, config);
const conversationHistory = await loadConversationHistory(tenantId);
// Build the chain with streaming enabled
const chain = conversationHistory.pipe(llmClient);
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
let totalTokens = 0;
let fullResponse = '';
try {
// Stream tokens as they arrive
for await (const chunk of await chain.stream(
{ input: userMessage },
{ runId: generateRunId() } as RunnableConfig
)) {
const token = chunk.content || '';
fullResponse += token;
totalTokens++;
// Send SSE event with token
res.write(`data: ${JSON.stringify({ token, type: 'content' })}
`);
}
// Track cost after streaming completes
await trackTokenUsage(tenantId, totalTokens, 'completion');
res.write(`data: ${JSON.stringify({ type: 'done', tokens: totalTokens })}
`);
res.end();
} catch (error) {
res.write(
`data: ${JSON.stringify({ type: 'error', message: error.message })}
`
);
res.end();
}
}
On the client side, you consume these events with a simple EventSource listener:
const eventSource = new EventSource('/api/chat/stream?tenantId=xyz&message=hello');
let responseText = '';
eventSource.addEventListener('message', (event) => {
const data = JSON.parse(event.data);
if (data.type === 'content') {
responseText += data.token;
document.getElementById('response').textContent = responseText;
} else if (data.type === 'done') {
console.log(`Received ${data.tokens} tokens`);
eventSource.close();
} else if (data.type === 'error') {
console.error(data.message);
eventSource.close();
}
});
The key is that tokens arrive in real time, and the UI updates as they do. No waiting for the full response.
Token-Aware Cost Tracking and Rate Limiting
LLM API costs scale with token usage. A single 10k-token conversation might cost $0.15 at current rates, but scale that across 500 concurrent users and 50 tenants, and you’re looking at significant daily expenses if you’re not tracking usage carefully.
You need two layers of control: tracking tokens per tenant so you can bill them accurately, and enforcing rate limits so a single tenant doesn’t consume their entire monthly budget in an hour. As noted in the algorithmine.com production playbook, cost control is a security control, unbounded spend is an availability risk.
// Token counter using tiktoken (OpenAI's tokenizer)
import { encoding_for_model } from 'js-tiktoken';
class CostTracker {
private encoding = encoding_for_model('gpt-4');
private costPerInputToken = 0.00003; // Model pricing (update as needed)
private costPerOutputToken = 0.0001;
async trackUsage(
tenantId: string,
inputText: string,
outputText: string,
model: string
) {
const inputTokens = this.encoding.encode(inputText).length;
const outputTokens = this.encoding.encode(outputText).length;
const cost =
inputTokens * this.costPerInputToken +
outputTokens * this.costPerOutputToken;
// Store in database for billing
await db.insert('tenant_usage_logs', {
tenantId,
model,
inputTokens,
outputTokens,
cost,
timestamp: new Date(),
});
// Update tenant's current month spend
await db.query(
`UPDATE tenant_billing SET month_spend = month_spend + ? WHERE id = ?`,
[cost, tenantId]
);
}
async checkRateLimit(tenantId: string, hardLimit: number) {
const monthlySpend = await db.query(
`SELECT month_spend FROM tenant_billing WHERE id = ?`,
[tenantId]
);
if (monthlySpend[0].month_spend > hardLimit) {
throw new Error(
`Tenant ${tenantId} has exceeded monthly LLM budget of $${hardLimit}`
);
}
}
}
Before each LLM call, check the tenant’s remaining budget. This prevents surprise bills and gives you a clean way to enforce fair usage policies.
// Middleware: check rate limit before processing
async function enforceRateLimitMiddleware(
req: Request,
res: Response,
next: NextFunction
) {
const tenantId = req.headers['x-tenant-id'] as string;
const tenantConfig = await db.query(
`SELECT monthly_budget FROM tenant_config WHERE id = ?`,
[tenantId]
);
try {
await costTracker.checkRateLimit(tenantId, tenantConfig[0].monthly_budget);
next();
} catch (error) {
res.status(429).json({ error: error.message });
}
}
Connection Pooling and Request Queuing
When you have 50 tenants sending concurrent requests, you can’t just spawn a new LLM API call for each one. You’ll hit rate limits, waste connections, and create unpredictable latency. Instead, use connection pooling and request queuing.
For database connections, this is standard practice. For LLM API calls, it’s equally important. Here’s a simple queue-based approach:
import PQueue from 'p-queue';
class LLMRequestQueue {
private queues = new Map<string, PQueue>();
private defaultConcurrency = 5; // Concurrent requests per tenant
getQueueForTenant(tenantId: string) {
if (!this.queues.has(tenantId)) {
this.queues.set(
tenantId,
new PQueue({
concurrency: this.defaultConcurrency,
interval: 60000, // 1 minute
intervalCap: 100, // Max 100 requests per minute per tenant
})
);
}
return this.queues.get(tenantId);
}
async executeWithQueue<T>(
tenantId: string,
task: () => Promise<T>
): Promise<T> {
const queue = this.getQueueForTenant(tenantId);
return queue.add(task);
}
}
// Usage in your chat endpoint
async function handleChatRequest(req: Request, res: Response) {
const tenantId = req.headers['x-tenant-id'] as string;
const userMessage = req.body.message;
try {
const response = await requestQueue.executeWithQueue(tenantId, async () => {
return await llmClient.chat.completions.create({
model: 'gpt-4',
messages: [{ role: 'user', content: userMessage }],
});
});
res.json({ response });
} catch (error) {
if (error.code === 'RATE_LIMIT_EXCEEDED') {
res.status(429).json({ error: 'Request queued, please retry' });
} else {
res.status(500).json({ error: error.message });
}
}
}
This ensures that each tenant’s requests are serialized and rate-limited independently. Tenant A hitting their limit doesn’t affect Tenant B.
Conversation History and Context Management
Stateless APIs are simpler, but a chatbot needs memory. You need to store conversation history per tenant, retrieve it efficiently, and avoid sending the entire history on every request (which wastes tokens and costs money).
A practical approach: store full history in a database, but only send the last N messages plus a summary of older context to the model.
class ConversationManager {
async getRecentContext(
tenantId: string,
conversationId: string,
maxMessages: number = 10
) {
// Get last N messages from database
const recentMessages = await db.query(
`SELECT role, content FROM messages
WHERE tenant_id = ? AND conversation_id = ?
ORDER BY created_at DESC
LIMIT ?`,
[tenantId, conversationId, maxMessages]
);
// Reverse to chronological order
return recentMessages.reverse();
}
async addMessage(
tenantId: string,
conversationId: string,
role: 'user' | 'assistant',
content: string,
tokenCount: number
) {
await db.insert('messages', {
tenant_id: tenantId,
conversation_id: conversationId,
role,
content,
token_count: tokenCount,
created_at: new Date(),
});
}
async getSummaryOfOlderMessages(
tenantId: string,
conversationId: string,
beforeMessageId: string
) {
// Get older messages and summarize them in a single context window
const olderMessages = await db.query(
`SELECT role, content FROM messages
WHERE tenant_id = ? AND conversation_id = ?
AND id < ?
ORDER BY created_at DESC`,
[tenantId, conversationId, beforeMessageId]
);
if (olderMessages.length === 0) return '';
// Summarize for context without bloating token count
const summary = olderMessages
.slice(0, 5) // Take oldest 5
.reverse()
.map((m) => `${m.role}: ${m.content}`)
.join('
');
return `Earlier in the conversation: ${summary}`;
}
}
When you call the LLM, construct the prompt with recent messages plus a summary, not the entire history. This keeps token counts predictable and costs linear rather than exponential as conversations grow.
Error Handling and Graceful Degradation
LLM APIs have rate limits, and models can be temporarily unavailable. Your chatbot needs to handle these conditions gracefully with clear feedback to users.
async function chatWithFallback(
tenantId: string,
userMessage: string
) {
const maxRetries = 3;
let lastError;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const response = await requestQueue.executeWithQueue(
tenantId,
async () => {
return await llmClient.chat.completions.create({
model: 'gpt-4',
messages: [{ role: 'user', content: userMessage }],
});
}
);
return response;
} catch (error) {
lastError = error;
// Rate limit: exponential backoff
if (error.status === 429) {
const backoffMs = Math.pow(2, attempt) * 1000;
await new Promise((resolve) => setTimeout(resolve, backoffMs));
continue;
}
// Server error: retry
if (error.status >= 500) {
continue;
}
// Client error: don't retry
throw error;
}
}
// If all retries fail, return a fallback response
if (lastError) {
console.error(`Chatbot failed for tenant ${tenantId}:`, lastError);
return {
fallback: true,
message:
'I am temporarily unavailable. Please try again in a moment.',
};
}
}
Exponential backoff for rate limits, retries for transient server errors, and a fallback message for the user. This keeps your application resilient without requiring manual intervention.
Monitoring and Observability
In production, you need to see what’s happening. Track response times, token usage per tenant, error rates, and cost trends.
class ChatbotMetrics {
async recordRequest(
tenantId: string,
durationMs: number,
inputTokens: number,
outputTokens: number,
success: boolean
) {
await metrics.histogram('llm_request_duration_ms', durationMs, {
tenantId,
success,
});
await metrics.counter('llm_tokens_input', inputTokens, { tenantId });
await metrics.counter('llm_tokens_output', outputTokens, { tenantId });
if (!success) {
await metrics.counter('llm_errors', 1, { tenantId });
}
}
async getTenantCostTrend(tenantId: string, days: number = 30) {
return await db.query(
`SELECT DATE(created_at) as date, SUM(cost) as daily_cost
FROM tenant_usage_logs
WHERE tenant_id = ? AND created_at > NOW() - INTERVAL ? DAY
GROUP BY DATE(created_at)
ORDER BY date ASC`,
[tenantId, days]
);
}
}
With these metrics, you can alert on cost spikes, identify slow tenants, and optimize your infrastructure based on real usage patterns.
Putting It Together: A Complete Handler
Here’s a complete request handler that ties together isolation, streaming, cost tracking, and error handling:
app.post('/api/chat/stream', enforceRateLimitMiddleware, async (req, res) => {
const tenantId = req.headers['x-tenant-id'] as string;
const conversationId = req.body.conversationId as string;
const userMessage = req.body.message as string;
const startTime = Date.now();
let inputTokens = 0;
let outputTokens = 0;
// Set up streaming response
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
try {
// Load tenant config and conversation history
const tenantConfig = await getTenantConfig(tenantId);
const llmClient = clientFactory.getClientForTenant(tenantId, tenantConfig);
const recentMessages =
await conversationManager.getRecentContext(tenantId, conversationId);
// Count input tokens
inputTokens = estimateTokens(recentMessages) + estimateTokens(userMessage);
// Queue and execute LLM request
let fullResponse = '';
await requestQueue.executeWithQueue(tenantId, async () => {
for await (const chunk of await llmClient.stream({
model: 'gpt-4',
messages: [
...recentMessages,
{ role: 'user', content: userMessage },
],
})) {
const token = chunk.content || '';
fullResponse += token;
outputTokens++;
res.write(`data: ${JSON.stringify({ token })}
`);
}
});
// Store conversation and track usage
await conversationManager.addMessage(
tenantId,
conversationId,
'user',
userMessage,
inputTokens
);
await conversationManager.addMessage(
tenantId,
conversationId,
'assistant',
fullResponse,
outputTokens
);
await costTracker.trackUsage(
tenantId,
userMessage,
fullResponse,
'gpt-4'
);
// Record metrics
const duration = Date.now() - startTime;
await metrics.recordRequest(tenantId, duration, inputTokens, outputTokens, true);
res.write(`data: ${JSON.stringify({ type: 'done' })}
`);
res.end();
} catch (error) {
const duration = Date.now() - startTime;
await metrics.recordRequest(tenantId, duration, inputTokens, outputTokens, false);
res.write(
`data: ${JSON.stringify({ type: 'error', message: error.message })}
`
);
res.end();
}
});
Conclusion
Building a production LLM chatbot is more than calling an API and returning the response. You need to isolate tenant data, stream responses for UX, track costs accurately, enforce rate limits, handle failures gracefully, and monitor everything.
The patterns here are battle-tested across multiple production systems: per-tenant client instances, streaming over SSE, token-aware cost tracking, request queuing, and comprehensive error handling. These aren’t theoretical, they’re the difference between a chatbot that works for one customer and one that scales profitably across dozens.
Start with these foundations, measure what matters to your business, and iterate from there. The cost control and reliability improvements compound as you scale. Smart token management and per-tenant rate limiting directly impact your unit economics and customer experience.
How do I prevent one tenant’s conversation history from leaking into another tenant’s context?
Isolate at the data layer, not just the presentation layer. Use per-tenant client instances from a factory, always filter queries by tenant ID in your database, and never pass conversation history between tenants. Store tenant ID in request context and validate it on every database operation. The isolation is only as strong as your weakest query.
What’s the performance impact of streaming responses compared to returning the full response at once?
Streaming has negligible performance overhead on the server side. The LLM generates tokens sequentially anyway, so you’re just forwarding them as they arrive instead of buffering them. The real benefit is user experience: perceived latency drops dramatically because users see the first token in 200ms instead of waiting 5 seconds for the full response.
How accurate is token estimation for cost tracking?
Using tiktoken (OpenAI’s tokenizer) is highly accurate for GPT models, typically within 1-2 tokens of actual usage. For other models like Claude or Llama, use their respective tokenizers. Don’t estimate; use the actual tokenizer. The difference between a guess and actual token count compounds quickly at scale.
What happens if a tenant exceeds their rate limit mid-request?
Check the rate limit before queuing the request, not after. This prevents wasted API calls and gives you a clean error boundary. Return a 429 status code to the client so they can retry or contact support. Store the check result in a short-lived cache to avoid database queries on every request.
Should I use a local LLM model or a cloud API for cost optimization?
Both have trade-offs. Local models (Llama, Mistral) reduce per-token costs but require GPU infrastructure and maintenance. Cloud APIs (OpenAI, Anthropic) have higher per-token costs but zero infrastructure burden. For most SaaS products, cloud APIs are more cost-effective when you factor in operational overhead. Optimize token usage first, then consider infrastructure.