OpenTelemetry SDK: Comprehensive Documentation

Introduction: The Foundation of Modern Observability

OpenTelemetry (OTel) is an open-source, vendor-neutral observability framework designed to standardize and collect telemetry data (metrics, logs, and traces) from cloud-native applications.

The OpenTelemetry SDK is a core component of this framework, providing developers with tools and libraries to:

Instrument applications,
Generate, process, and export telemetry data,
Integrate with various backend systems.

This documentation serves as a comprehensive guide to OpenTelemetry SDK, covering everything from architecture and core concepts to language-specific implementations and advanced usage scenarios.

1. OpenTelemetry Architecture and Core Concepts

OpenTelemetry has a layered architecture consisting of the following components:

API (Application Programming Interface) → Used to instrument code without coupling to a specific SDK.
SDK (Software Development Kit) → The implementation of the API, responsible for processing and exporting telemetry data.
Exporter → Sends processed telemetry data to backends (Jaeger, Prometheus, Zipkin, etc.).
Collector → (Optional) A centralized service for receiving, processing, and exporting telemetry data.

Core Observability Signals

Traces: Represent the end-to-end journey of a request in a distributed system.
Metrics: Numerical measurements (e.g., CPU usage, request count, error rate).
Logs: Textual records of discrete events.

2. SDK Implementation Across Languages

OpenTelemetry SDK provides a consistent experience across multiple languages. Below are examples for Java, Python, and Node.js.

2.1. Java

Dependencies (Maven):

<dependencies>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.38.0</version>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk</artifactId>
    <version>1.38.0</version>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
    <version>1.38.0</version>
  </dependency>
  <dependency>
    <groupId>io.grpc</groupId>
    <artifactId>grpc-netty-shaded</artifactId>
    <version>1.58.0</version>
  </dependency>
</dependencies>

Initialize SDK:

OtlpGrpcSpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
    .setEndpoint("http://localhost:4317")
    .build();

2.2. Python

Install dependencies:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

Initialize SDK:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317")
processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

2.3. Node.js

Install dependencies:

npm install @opentelemetry/sdk-node @opentelemetry/api @opentelemetry/exporter-trace-otlp-grpc

Initialize SDK (instrumentation.js):

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const traceExporter = new OTLPTraceExporter({ url: 'grpc://localhost:4317' });
const sdk = new NodeSDK({ traceExporter });

sdk.start();

3. Manual Instrumentation: Traces and Spans

3.1. Creating a Span (Java Example)

Span span = tracer.spanBuilder("my-custom-operation").startSpan();
try (Scope scope = span.makeCurrent()) {
    // Your business logic here
} catch (Throwable t) {
    span.setStatus(StatusCode.ERROR, "An error occurred");
    span.recordException(t);
} finally {
    span.end();
}

Python and Node.js examples available in documentation.

3.2. Adding Attributes and Events

span.setAttribute("user.id", "12345");
span.addEvent("Order confirmed");

3.3. Context Propagation

OpenTelemetry automatically propagates context using the W3C Trace Context standard.

4. Manual Instrumentation: Metrics

Python Example:

from opentelemetry import metrics

meter = metrics.get_meter("my-meter")
request_counter = meter.create_counter("http.requests")
request_counter.add(1, {"http.method": "GET", "http.route": "/api/users"})

5. Manual Instrumentation: Logs

OpenTelemetry integrates with existing logging frameworks (e.g., Log4j, Logback, Python logging).

Java (Logback Example):

<appender name="OTEL" class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
  <captureExperimentalAttributes>true</captureExperimentalAttributes>
</appender>

6. Exporters and Backend Integration

Supported exporters:

OTLP (recommended)
Jaeger
Prometheus
Zipkin
Logging/Console

In production, always use the OpenTelemetry Collector for centralized processing.

7. Advanced Concepts

Sampling: AlwaysOn, AlwaysOff, TraceIdRatioBased, ParentBased.
Resource Detection: Automatically adds attributes like service.name, k8s.pod.name, cloud.provider.
Context Propagation: Supports W3C Trace Context, W3C Baggage, and B3 formats.

8. Best Practices

Follow Semantic Conventions.
Define meaningful service.name.
Use manual instrumentation for critical business logic.
Avoid high cardinality attributes.
Use Collector in production.
Keep SDK dependencies up-to-date.