Building a chat interface that talks to an AI API sounds straightforward until you ship it. The moment you hit production, you realize that streaming responses need careful handling, state management gets messy fast, and users notice every re-render delay. I’ve built several of these, and the patterns that separate a prototype from a solid production interface are concrete and learnable.
This guide walks through the core challenges: handling streaming data from LLM APIs, managing conversation state without prop drilling chaos, recovering gracefully from errors, and keeping the UI responsive even under load. We’ll work through real patterns you can adapt to OpenAI, Anthropic, or any self-hosted LLM endpoint.
Understanding Streaming Responses
Most AI APIs support server-sent events (SSE) or chunked transfer encoding for streaming. Instead of waiting 10 seconds for a full response, tokens arrive incrementally. This creates a better UX: users see text appearing in real time, and perceived latency drops dramatically.
The fetch API handles this natively. Here’s the basic pattern:
async function streamChatResponse(messages: Message[]) {
const response = await fetch('/api/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ messages }),
});
if (!response.ok) {
throw new Error(`API error: ${response.status}`);
}
const reader = response.body?.getReader();
const decoder = new TextDecoder();
let fullText = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value, { stream: true });
fullText += chunk;
// Emit chunk to UI layer
}
return fullText;
}
This reads the response body as a stream, decoding chunks as they arrive. The key detail: TextDecoder with the stream flag handles multi-byte UTF-8 sequences that might split across chunks.
If your API uses newline-delimited JSON (like OpenAI’s chat completions), parse each line separately:
const lines = chunk.split('
').filter(Boolean);
for (const line of lines) {
if (line.startsWith('data: ')) {
const data = JSON.parse(line.slice(6));
if (data.choices?.[0]?.delta?.content) {
fullText += data.choices[0].delta.content;
}
}
}
Managing Conversation State
A chat interface needs to track messages, handle pending responses, and let users send new messages while one is streaming. Prop drilling through deeply nested components becomes painful fast. A custom hook with context makes this cleaner:
interface ChatContextType {
messages: Message[];
isLoading: boolean;
error: string | null;
sendMessage: (content: string) => Promise<void>;
clearError: () => void;
}
const ChatContext = createContext<ChatContextType | null>(null);
export function ChatProvider({ children }: { children: React.ReactNode }) {
const [messages, setMessages] = useState<Message[]>([]);
const [isLoading, setIsLoading] = useState(false);
const [error, setError] = useState<string | null>(null);
const sendMessage = useCallback(async (content: string) => {
setError(null);
const userMessage: Message = { role: 'user', content, id: crypto.randomUUID() };
setMessages(prev => [...prev, userMessage]);
setIsLoading(true);
try {
let assistantContent = '';
const assistantMessage: Message = {
role: 'assistant',
content: '',
id: crypto.randomUUID(),
};
// Stream the response
const reader = await streamChatResponse([...messages, userMessage]);
for await (const chunk of reader) {
assistantContent += chunk;
setMessages(prev => {
const updated = [...prev];
const lastMsg = updated[updated.length - 1];
if (lastMsg?.id === assistantMessage.id) {
lastMsg.content = assistantContent;
} else {
updated.push({ ...assistantMessage, content: assistantContent });
}
return updated;
});
}
} catch (err) {
setError(err instanceof Error ? err.message : 'Unknown error');
} finally {
setIsLoading(false);
}
}, [messages]);
return (
<ChatContext.Provider value={{ messages, isLoading, error, sendMessage, clearError: () => setError(null) }}>
{children}
</ChatContext.Provider>
);
}
export function useChat() {
const context = useContext(ChatContext);
if (!context) throw new Error('useChat must be used inside ChatProvider');
return context;
}
This pattern centralizes state, prevents prop drilling, and makes testing easier. The key is updating the message content as chunks arrive, so the UI reflects streaming in real time.
Error Handling and Resilience
Network errors, rate limits, and API timeouts are not edge cases, they’re normal. Users expect the chat to handle them gracefully:
async function sendMessageWithRetry(
content: string,
maxRetries: number = 2
): Promise<void> {
let lastError: Error | null = null;
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
await sendMessage(content);
return;
} catch (err) {
lastError = err instanceof Error ? err : new Error(String(err));
if (attempt < maxRetries) {
const delay = Math.min(1000 * Math.pow(2, attempt), 5000);
await new Promise(resolve => setTimeout(resolve, delay));
}
}
}
throw lastError;
}
Exponential backoff with a cap prevents hammering a struggling API. For frontend errors, give users actionable feedback:
function getErrorMessage(error: Error): string {
if (error.message.includes('429')) {
return 'Rate limited. Please wait a moment and try again.';
}
if (error.message.includes('401')) {
return 'Authentication failed. Check your API key.';
}
if (error.message.includes('timeout')) {
return 'Request timed out. The API might be slow.';
}
return 'Something went wrong. Please try again.';
}
Distinguish between retriable errors (rate limits, timeouts) and permanent ones (auth, invalid input) so the UX responds appropriately.
Performance Optimization
Chat interfaces can re-render aggressively. Every token arrival triggers a state update, which can cause performance issues if not handled carefully. A few patterns help:
1. Memoize message list rendering:
const MessageList = React.memo(({ messages }: { messages: Message[] }) => (
<div>
{messages.map(msg => (
<div key={msg.id} className={`message ${msg.role}`}>
{msg.content}
</div>
))}
</div>
), (prev, next) => {
// Only re-render if message count changes or content of last message changes
if (prev.messages.length !== next.messages.length) return false;
return prev.messages[prev.messages.length - 1]?.content ===
next.messages[next.messages.length - 1]?.content;
});
2. Batch state updates during streaming:
const [pendingContent, setPendingContent] = useState('');
const updateTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const flushPendingContent = useCallback(() => {
if (pendingContent) {
setMessages(prev => {
const updated = [...prev];
const lastMsg = updated[updated.length - 1];
if (lastMsg?.role === 'assistant') {
lastMsg.content += pendingContent;
}
return updated;
});
setPendingContent('');
}
}, [pendingContent]);
const onChunk = useCallback((chunk: string) => {
setPendingContent(prev => prev + chunk);
if (updateTimerRef.current) clearTimeout(updateTimerRef.current);
updateTimerRef.current = setTimeout(flushPendingContent, 50);
}, [flushPendingContent]);
Batching updates every 50ms reduces re-renders from hundreds per second to dozens, keeping the UI smooth.
3. Use useCallback for handlers:
const handleSend = useCallback((text: string) => {
sendMessage(text);
}, [sendMessage]);
const handleClear = useCallback(() => {
setMessages([]);
setError(null);
}, []);
This prevents child components from re-rendering when parent state changes but their props stay the same.
Handling Token Limits
LLMs have context window limits. Conversations that grow indefinitely will eventually fail. Implement a strategy early:
function trimConversation(
messages: Message[],
maxTokens: number = 4000
): Message[] {
// Rough estimate: 1 token approximately 4 characters
let tokenCount = 0;
const trimmed: Message[] = [];
// Always keep the most recent message
for (let i = messages.length - 1; i >= 0; i--) {
const msg = messages[i];
const msgTokens = Math.ceil(msg.content.length / 4);
if (tokenCount + msgTokens > maxTokens && trimmed.length > 1) {
break;
}
trimmed.unshift(msg);
tokenCount += msgTokens;
}
return trimmed;
}
Before sending to the API, trim the conversation history. This is a rough estimate, but it prevents most context window errors. For production, integrate a proper tokenizer library like js-tiktoken.
Putting It Together
Here’s a complete chat component that ties these patterns together:
export function ChatWindow() {
const { messages, isLoading, error, sendMessage, clearError } = useChat();
const [input, setInput] = useState('');
const messagesEndRef = useRef<HTMLDivElement>(null);
useEffect(() => {
messagesEndRef.current?.scrollIntoView({ behavior: 'smooth' });
}, [messages]);
const handleSubmit = useCallback(
async (e: React.FormEvent) => {
e.preventDefault();
if (!input.trim() || isLoading) return;
try {
await sendMessage(input);
setInput('');
} catch (err) {
console.error('Failed to send message:', err);
}
},
[input, isLoading, sendMessage]
);
return (
<div className="chat-window">
<div className="messages">
{messages.map(msg => (
<div key={msg.id} className={`message ${msg.role}`}>
<p>{msg.content}</p>
</div>
))}
{isLoading && <div className="message assistant loading">Waiting for response...</div>}
<div ref={messagesEndRef} />
</div>
{error && (
<div className="error-banner">
<p>{error}</p>
<button onClick={clearError}>Dismiss</button>
</div>
)}
<form onSubmit={handleSubmit} className="input-form">
<input
value={input}
onChange={e => setInput(e.target.value)}
placeholder="Type your message..."
disabled={isLoading}
/>
<button type="submit" disabled={isLoading || !input.trim()}>
Send
</button>
</form>
</div>
);
}
Conclusion
Real-time AI chat interfaces in React require attention to streaming mechanics, state management, error resilience, and performance. The patterns above, grounded in production experience, handle the common pitfalls. Start with the context provider and streaming handler, add retry logic and token trimming, then optimize re-renders as needed. Most users won’t notice the engineering, but they’ll feel the difference when the chat is smooth, responsive, and doesn’t break under load.
How do I handle backpressure when streaming responses arrive faster than the UI can render?
Batch state updates using a timer, as shown in the performance section. Collect chunks for 50ms before updating React state, reducing re-renders from hundreds per second to dozens. This keeps the UI responsive while still delivering near-real-time text appearance.
What’s the difference between fetch streaming and EventSource?
Fetch with getReader() works for any response stream, including chunked transfer encoding and newline-delimited JSON. EventSource is simpler for server-sent events but only supports GET requests and is less flexible. For AI APIs, fetch is usually the better choice.
How do I prevent token limit errors?
Before sending to the API, trim the conversation history to stay within the context window. Use a rough token estimate (1 token per 4 characters) or a proper tokenizer library. Keep the most recent messages and drop older ones if needed. Always validate on the backend too.
Should I store the conversation in local storage or a database?
For prototypes, local storage is fine. For production, use a database so users can resume conversations across devices and sessions. Encrypt sensitive data in transit and at rest, and implement proper auth to prevent cross-user access.
How do I handle rate limiting gracefully?
Implement exponential backoff with a cap (1s, 2s, 4s, max 5s). Catch 429 status codes and show a user-friendly message. On the backend, track rate limit headers and respect them. Better yet, queue messages client-side and send them at a safe rate.
