OpenTelemetry Python: Production Microservices Observability

Implement distributed tracing, metrics, and log correlation in Python FastAPI/Django services. Real patterns for production AI pipelines with Jaeger and Prometheus.

sohail
By Sohail Ahmad
Posted
0

Last month, I spent four hours tracking down why an AI inference pipeline was dropping requests at the third service in a chain of five. Log files were scattered across three servers. Timestamps didn’t align. The only clue was a generic timeout error. Without proper observability, debugging distributed systems becomes exponentially harder.

OpenTelemetry changed that. Within a week of instrumenting our services, the same issue showed up immediately in Jaeger as a trace spanning all five services, pinpointing the exact service and line of code causing the bottleneck. This article walks you through the same patterns we now use in production.

Why OpenTelemetry Matters for Python Microservices

When you have one Python service, logging to stdout works fine. When you have five services talking to each other, databases, and external APIs, logs alone tell you almost nothing. You see an error in service C, but you don’t know if it came from service A, B, or an upstream call. You don’t know how long each step took. You don’t know if the issue is latency, a timeout, or a silent failure.

OpenTelemetry solves this by giving you three things:

  • Distributed traces that follow a request across all services
  • Metrics that measure performance, errors, and business outcomes
  • Logs correlated with traces, so you see context instead of noise

Unlike older APM tools, OpenTelemetry is vendor-neutral and open source. You instrument once, then export to Jaeger, Datadog, New Relic, or any backend. No lock-in.

Setting Up Tracing in FastAPI

Let’s start with the fundamentals. Here’s how to add tracing to a FastAPI service.

First, install the dependencies:

pip install fastapi uvicorn opentelemetry-api opentelemetry-sdk opentelemetry-exporter-jaeger opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-requests

Now initialize tracing in your app:

from fastapi import FastAPI
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

app = FastAPI()

jaeger_exporter = JaegerExporter(
    agent_host_name="localhost",
    agent_port=6831,
)

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(jaeger_exporter)
)

FastAPIInstrumentor.instrument_app(app)
RequestsInstrumentor().instrument()

@app.get("/health")
async def health():
    return {"status": "ok"}

That’s it. Every HTTP request to your FastAPI service now generates a trace. Outgoing HTTP calls via requests are also traced. You can start Jaeger locally with Docker:

docker run -d --name jaeger -p 6831:6831/udp -p 16686:16686 jaegertracing/all-in-one

Hit your endpoint and navigate to http://localhost:16686 to see traces in real time.

Creating Custom Spans for Business Logic

Automatic instrumentation gets you HTTP and database calls. For your own logic, you need custom spans. This is where tracing becomes powerful.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.post("/process_image")
async def process_image(image_id: str):
    with tracer.start_as_current_span("fetch_image") as span:
        span.set_attribute("image_id", image_id)
        image_data = await fetch_from_storage(image_id)
    
    with tracer.start_as_current_span("run_inference") as span:
        span.set_attribute("model", "resnet50")
        predictions = await model.predict(image_data)
    
    with tracer.start_as_current_span("save_results") as span:
        span.set_attribute("prediction_count", len(predictions))
        await save_predictions(image_id, predictions)
    
    return {"image_id": image_id, "predictions": predictions}

Each span is a unit of work. In Jaeger, you see exactly how long fetch_image took, then run_inference, then save_results. If one is slow, you spot it immediately. Attributes let you filter and search traces later.

Collecting Metrics with Prometheus

Traces show you what happened to a single request. Metrics show you patterns across all requests. Install Prometheus instrumentation:

pip install opentelemetry-exporter-prometheus opentelemetry-instrumentation-sqlalchemy

Set up metrics export:

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from prometheus_client import start_http_server

prometheus_reader = PrometheusMetricReader()
metrics.set_meter_provider(MeterProvider(metric_readers=[prometheus_reader]))
start_http_server(port=8000)

Now create custom metrics for your business logic:

from opentelemetry import metrics

meter = metrics.get_meter(__name__)

inference_counter = meter.create_counter(
    "inference_requests_total",
    description="Total inference requests",
    unit="1"
)

inference_duration = meter.create_histogram(
    "inference_duration_seconds",
    description="Time spent on inference",
    unit="s"
)

@app.post("/process_image")
async def process_image(image_id: str):
    start_time = time.time()
    
    try:
        predictions = await model.predict(image_data)
        inference_counter.add(1, {"status": "success"})
    except Exception as e:
        inference_counter.add(1, {"status": "error"})
        raise
    finally:
        duration = time.time() - start_time
        inference_duration.record(duration)
    
    return {"predictions": predictions}

Prometheus scrapes these metrics at http://localhost:8000/metrics. You can then build dashboards to track error rates, latency percentiles, and throughput.

Correlating Logs with Traces

Logs are still useful, but they’re most useful when tied to traces. Python’s logging module can include the trace ID and span ID automatically.

import logging
from opentelemetry import trace

logging.basicConfig(
    format='%(asctime)s - %(name)s - %(levelname)s - trace_id=%(trace_id)s span_id=%(span_id)s - %(message)s'
)

class TraceContextFilter(logging.Filter):
    def filter(self, record):
        trace_context = trace.get_current_span().get_span_context()
        record.trace_id = format(trace_context.trace_id, '032x')
        record.span_id = format(trace_context.span_id, '016x')
        return True

logger = logging.getLogger(__name__)
logger.addFilter(TraceContextFilter())

@app.post("/process")
async def process():
    logger.info("Starting processing")
    # Your logic here
    logger.info("Processing complete")

Now every log line includes the trace ID. When you see an error in logs, you can paste the trace ID into Jaeger and see the full distributed trace. This is the magic: logs, traces, and metrics all connected.

Instrumenting Multiple Services

The real power comes when you have multiple services. Let’s say your AI pipeline has a service that fetches data, another that runs inference, and a third that saves results. Each needs tracing.

The key is that trace context must flow between services. When service A calls service B, it must pass the trace ID. FastAPI auto-instrumentation handles this for HTTP calls via the W3C Trace Context standard.

Here’s what happens automatically:

import httpx

# In service A
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("call_inference_service"):
    # OpenTelemetry automatically injects trace context into the request
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://inference-service/predict",
            json={"data": data}
        )

Service B receives the trace context in the HTTP headers and continues the same trace:

@app.post("/predict")
async def predict(request: Request):
    # FastAPI automatically extracts trace context from headers
    # The trace from service A continues here
    with tracer.start_as_current_span("model_inference"):
        result = await run_model(request.data)
    return result

In Jaeger, you see one trace spanning both services with clear parent-child relationships. This is how you debug across service boundaries.

Handling Async and Concurrency

Python’s async/await can complicate tracing. Context must flow through tasks. OpenTelemetry handles most of this, but be careful with concurrent operations:

import asyncio
from opentelemetry.context import attach, detach
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def process_batch(items):
    tasks = []
    for item in items:
        # Each task needs its own span context
        task = asyncio.create_task(process_item(item))
        tasks.append(task)
    
    results = await asyncio.gather(*tasks)
    return results

async def process_item(item):
    with tracer.start_as_current_span("process_item") as span:
        span.set_attribute("item_id", item.id)
        # Your logic here
        return result

For more complex scenarios like thread pools or multiprocessing, you may need to manually propagate context. Check the OpenTelemetry Python documentation for your specific use case.

Real-World Debugging Example

Your AI pipeline has five services, and inference is timing out for 10% of requests. Where do you start?

With OpenTelemetry:

  1. Filter Jaeger traces by error status or duration threshold
  2. Open a slow trace and see which service is slow
  3. Expand that service’s span and see which operation is the bottleneck
  4. Check the logs for that trace ID to see detailed error messages
  5. Look at Prometheus metrics for that service to see if it’s consistently slow or intermittent

Without this, you’re guessing. With it, you have facts.

Production Considerations

Before deploying, keep these things in mind:

  • Sampling: Tracing every request can be expensive. Use sampling in production. Start with 10% and adjust based on volume and budget.
  • Span processors: Use BatchSpanProcessor, not SimpleSpanProcessor, to avoid blocking your application.
  • Exporters: Jaeger is great for development. For production, use a managed service or Jaeger in production-grade setup with persistent storage.
  • Resource attributes: Add service name, version, and environment to all spans so you can filter in Jaeger.
  • Error handling: Ensure spans are closed even when exceptions occur. Context managers handle this automatically.
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider

resource = Resource.create({
    "service.name": "inference-service",
    "service.version": "1.2.0",
    "deployment.environment": "production"
})

trace.set_tracer_provider(TracerProvider(resource=resource))

Conclusion

OpenTelemetry gives you visibility into distributed systems. It’s not a silver bullet, but it transforms debugging from guesswork to engineering. Start small: instrument one service, send traces to Jaeger, and get familiar with the tool. Then expand to your entire pipeline.

The investment pays off the first time you find a production bug in minutes instead of hours. Your team will thank you.

Do I need to use Jaeger? Can I use another backend?

No, Jaeger is optional. OpenTelemetry is vendor-neutral. You can export to Jaeger, Datadog, New Relic, Grafana Tempo, or any OTLP-compatible backend. The instrumentation code stays the same. Only the exporter changes.

Will tracing slow down my application?

Not significantly if configured correctly. Use BatchSpanProcessor instead of SimpleSpanProcessor, enable sampling in production, and use async exporters. The overhead is typically minimal for well-tuned setups.

How do I sample traces without losing important data?

Use probabilistic sampling for routine requests (for example, 10%) and always-on sampling for errors. OpenTelemetry supports custom samplers. You can also use tail sampling at the collector level to keep traces that match certain criteria.

Can I use OpenTelemetry with Django instead of FastAPI?

Yes. Install opentelemetry-instrumentation-django instead of the FastAPI instrumentation. The patterns for custom spans, metrics, and log correlation are identical.

How do I correlate logs from different services in one trace?

Include the trace ID in all logs. OpenTelemetry automatically propagates trace context across service boundaries via HTTP headers. Parse the trace ID from logs and filter by it in your log aggregation tool (ELK, Loki, etc.), or use the trace ID to jump from logs to Jaeger.

sohail

Sohail Ahmad

Leave a Reply

Your email address will not be published. Required fields are marked *