Skip to content

feat: add OpenTelemetry instrumentation - #672

Open
kahirokunn wants to merge 1 commit into
cloudevents:mainfrom
kahirokunn:opentelemetry-tracing
Open

kahirokunn wants to merge 1 commit into
cloudevents:mainfrom
kahirokunn:opentelemetry-tracing

Conversation

@kahirokunn

@kahirokunn kahirokunn commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Proposed Changes

Add explicit helpers for injecting and extracting the CloudEvents Distributed Tracing extension through cloudevents/otel.

Description

injectDistributedTracingExtension(context, event) copies a valid W3C trace context into a new event, leaving the original unchanged. Explicit injection replaces an existing tracing extension; if the supplied context has no valid span context, the original event is returned.

extractDistributedTracingExtension(context, event) returns a context containing the event's valid tracing extension. A missing or invalid extension leaves the base context unchanged. The returned context is not activated automatically.

The helpers do not create spans and can be used independently of transport instrumentation. OpenTelemetry dependencies are optional for applications using the package root. The README shows injection followed by sending the event with await emit(tracedEvent).

This PR is based directly on main and does not depend on #657. Existing-span enrichment, span-creating wrappers, metrics, and exception logs are deferred to separate follow-up work. Related to #355.

Validation

  • Lint, builds, conformance (7 scenarios / 38 steps), and 247 integration tests passed on Node.js 24.22.1.
  • GitHub Actions passed on Node.js 22.x and 24.x, including coverage.
  • The README example was executed against a local HTTP receiver to verify the event carries the active trace context and the original event remains unchanged.

@thompson-tomo thompson-tomo left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So on reviewing this, to me it seems over engineered and ultimately longterm fragile due to the coupling of messaging, rpc & http conventions.

My suggestion is as follows:

  • have the encode operation add the cloud event attribute values in baggage
  • have the decode operation remove the cloud event baggage items
  • develop either of the below which converts baggage to attributes
    • cloud events log/span processor
    • baggage to attribute log/span processor

With this approach we can have the following scenarios:

Integrated transport

  • application calls send
  • ce lib start emit process
  • ce otel extension adds cloud event info to baggage
  • transport lib send invoked
  • transport instrumentation creates span
  • processor reads baggage & adds it as attributes
  • propagator attaches info to carrier
  • transport sends request
  • receiver transport receives request
  • propagator attaches info from carrier
  • processor reads baggage & adds it as attributes as well as removes it from baggage
  • handling of event completes.

Encoder

  • ce encoder library called for a payload
  • ce otel extension adds cloud event info to baggage
  • application has an encoded payload
  • application invokes transport lib send
  • transport instrumentation creates span
  • processor reads baggage & adds it as attributes
  • propagator attaches info to carrier
  • transport sends request
  • receiver transport receives request
  • propagator attaches info from carrier
  • application creates span
  • processor reads baggage & adds it as attributes as well as removes it from baggage
  • ce decoder library called for a payload
  • application handles decoded payload

This way the solution works for any transport & the responsibility for defining the telemetry resides with the instrumentation libraries rather than cloud events.

@kahirokunn

Copy link
Copy Markdown
Contributor Author

I think CloudEvents instrumentation should represent the logical event lifecycle independently of the underlying transport.

OpenTelemetry's CloudEvents Semantic Conventions recommend following the Messaging Span model, including span relationships and message creation contexts.

I'd prefer to follow these conventions directly, keeping CloudEvents-level observability separate from transport-level instrumentation.

@thompson-tomo

Copy link
Copy Markdown

I think CloudEvents instrumentation should represent the logical event lifecycle independently of the underlying transport.

This is not documented anywhere in semantic conventions and in fact it goes the opposite direction by not documenting the lifecycle ie no span but rather what attributes can be added to spans.

OpenTelemetry's CloudEvents Semantic Conventions recommend following the Messaging Span model, including span relationships and message creation contexts.

The conventions are for the instrumentation which in the case of sending http request that would http, it would make no sense to have a messaging publish span for http as you don't have a messaging system.

I'd prefer to follow these conventions directly, keeping CloudEvents-level observability separate from transport-level instrumentation.

I am following the conventions, as per https://opentelemetry.io/docs/specs/semconv/cloudevents/cloudevents-spans/#span-attributes key points bei g

can be attached to spans when performing operations with CloudEvents, regardless of the protocol being used.

Which means the attributes are added to the protocol ie if you are using rabbitmq then you have your rabbitmq span and cloud events attributes added to it.

If CloudEvents are instrumented independently of the above conventions, instrumentations can rely on the CloudEvents Distributed Tracing Extension as means to propagate the trace context.

That is exactly what i am proposing, using the extension to add to context and pulling it out when needed.

Key thing is, convention is not for a span but rather attributes added to applicable spans.

@kahirokunn

Copy link
Copy Markdown
Contributor Author

The Go SDK already creates CloudEvents-level PRODUCER and CONSUMER spans alongside HTTP instrumentation.

I’d like to follow the same separation in the JavaScript SDK rather than limit CloudEvents instrumentation to adding attributes to transport spans.

@thompson-tomo

Copy link
Copy Markdown

The go sdk is actually quite different to both the current otel spec & this pr. Go is just creating a span with the cloud event attributes and nothing else so no connection to messaging etc.

Note I can see the benefit to this approach on client side as it avoids the need to attach attributes after the payload has been encoded but on receiver side it fells un-necessary as what do you gain?

Also on looking at the otel spec history it appears that Go is following a much older spec which was removed in open-telemetry/semantic-conventions#1182. In fact it was written by the same author of the go sdk, just 3 years later.

@kahirokunn

Copy link
Copy Markdown
Contributor Author

Thanks for pointing out the spec history.

On the receiver side, the value I’m aiming for is batch-aware processing: one processing span linked to each event’s available creation context, with batch size and per-event attributes.

This follows a similar opt-in, library-level instrumentation pattern to Prisma for ORM operations and BullMQ for job processing.

I’d like to provide this as reusable, opt-in instrumentation so applications don’t have to implement that logic themselves. Where existing instrumentation already covers this, the wrapper can simply be omitted.

@thompson-tomo

Copy link
Copy Markdown

On the receiver side, the value I’m aiming for is batch-aware processing: one processing span linked to each event’s available creation context, with batch size and per-event attributes

Should be already covered out of the box as per messaging spec https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/#batch-receiving in particular https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/#recording-per-message-attributes-on-batch-operations which describes how to capture the message level details.

This can all be done without instrumentation & instead achieved at the propagation layer via a custom propagator.

I’d like to provide this as reusable, opt-in instrumentation so applications don’t have to implement that logic themselves.

The issue is not about providing the functionality but rather what is the responsibility of cloudevents to provide.

Where existing instrumentation already covers this, the wrapper can simply be omitted.

Existing instrumentation will only cover the standard messaging use case ie creating spans however a cloudevents propagator is needed to create the links based on the payload & processor to add attributes to the messaging send span.

As such i feel that the extension should just be:

  • injecting the traceinfo into the carrier (payload)
  • adding details of message into baggage
  • extracting the traceinfo from the carrier (payload) and creating links

@kahirokunn

Copy link
Copy Markdown
Contributor Author

My concern is using Baggage solely to pass CloudEvents metadata to a local processor. With baggage propagation enabled, that metadata could also be sent to downstream services. Opting into instrumentation shouldn’t implicitly opt applications into propagating event metadata downstream.

I’d prefer to attach the batch attributes and links directly when the opt-in wrapper creates the processing span. This keeps the integration simple, avoids introducing Baggage and an additional processor, and doesn’t depend on other instrumentation providing an appropriate processing span.

@thompson-tomo

Copy link
Copy Markdown

My concern is using Baggage solely to pass CloudEvents metadata to a local processor. With baggage propagation enabled, that metadata could also be sent to downstream services. Opting into instrumentation shouldn’t implicitly opt applications into propagating event metadata downstream.

Then have the processor remove before it sending. Alternatively we can add it to the context instead that way it is not propagated. I do like baggage so that reciever spans can have access to the properties without parsing the messages.

I’d prefer to attach the batch attributes and links directly when the opt-in wrapper creates the processing span.

I don't follow how this differs to what i said, under my proposal when processing a message the propagator extracts the context from the message, once extracted links can be created with the desired attributes.

This keeps the integration simple, avoids introducing Baggage and an additional processor, and doesn’t depend on other instrumentation providing an appropriate processing span.

You are right it doesn't rely on other instrumentation creating the span but i see that as a major positive as that way cloud events doesn't need to duplicate implementation & avoids chance of spans being double instrumented.

I don't see what is complex from an app developer side, the steps would be:

  • add cloudevents processor to otel config (single line) in side generating messages
  • add cloudevents extension which internally does either:
    • creates link for each message being processed adding the attributes
    • adds message detail to context/baggage when creating message

Nothing else needs to be done so it is almost as simple (1 additional line) as directly creating the sending span but significantly decreases the code base needing to be maintained by cloudevents.

@kahirokunn

Copy link
Copy Markdown
Contributor Author

Thanks for clarifying. I’d like to support enriching existing spans while keeping handlerWithTracing() as an opt-in option when existing instrumentation doesn’t provide an appropriate processing span.

The wrapper’s purpose is to capture the event or batch handler’s own duration and failures, with links to each event’s available creation context. That processing can be a distinct operation from the enclosing HTTP request.

@thompson-tomo

thompson-tomo commented Nov 15, 2026 •

Copy link
Copy Markdown

I’d like to support enriching existing spans while keeping handlerWithTracing() as an opt-in option when existing instrumentation doesn’t provide an appropriate processing span.

If a transport is not instrumented it would be much better to instrument that transport following otel in either a personal or otel-js-contrib repo. Note as per the spec the messaging span represents the logical operation of calling send on the transport layer.

The wrapper’s purpose is to capture the event or batch handler’s own duration and failures, with links to each event’s available creation context. That processing can be a distinct operation from the enclosing HTTP request.

That is what the old Otel spec said but it was collectively decided that it shouldn't be the case and instead the messaging/http conventions should be used to represent the messaging/http operation. If you are really interested in the encoding/decoding operation duration that could be expressed as dedicated internal child spans or events.

My recommendation for this pr given it is not following the otel spec or cloud events as well as the fact that it is doing an extremely lot of different things, is to split into manageable/reviewable chunks.

The way i would do it is dedicated pr's for:

  • trace propagator to inject & extract trace context
  • cloud events extension lib which triggers propagation & on receiver side also adds links to the current span.
  • solution to add message attributes to sender span which is started after message is constructed

Independent of the above if a transport you need is not instrumented instrument. Note the first 2 pr's are following the current spec.

@kahirokunn

Copy link
Copy Markdown
Contributor Author

handlerWithTracing() wraps the application handler for already-decoded events and measures its execution.

#1182 removed the separate CloudEvents span definitions, but its merged text recommends following the Messaging span structure. That recommendation remains in the current CloudEvents conventions.

The Messaging conventions describe Process spans covering handler execution. They also explicitly show a Process span beneath an HTTP server span when a broker delivers messages over HTTP, linked to the message creation context.

That Messaging Process span, enriched with CloudEvents attributes and links, is what I intend this opt-in helper to provide when existing instrumentation does not already cover the processing operation.

Whether this SDK should maintain that helper, and how to split the PR, are separate questions. I don’t think #1182 establishes that this processing span structure should be avoided.

@thompson-tomo

Copy link
Copy Markdown

handlerWithTracing() wraps the application handler for already-decoded events and measures its execution.

Thats not a messaging processing span as the operation has already started and the description states it is to represent the callback/handler.

#1182 removed the separate CloudEvents span definitions, but its merged text recommends following the Messaging span structure. That recommendation remains in the current CloudEvents conventions.

Yes because it is assumed there is already a span for the messaging system. Cloudevents spec even states the focus of ce is on preparing the content https://github.com/cloudevents/spec/blob/main/cloudevents/SDK.md#object-model-structure-guidelines and not wrapping the sdk.

The Messaging conventions describe Process spans covering handler execution.

A handler excludes decoding.

They also explicitly show a Process span beneath an HTTP server span when a broker delivers messages over HTTP, linked to the message creation context.

It does but key thing is the complete flow:

  • client publishes a message to a broker
  • broker initiates a http request (http client span)
  • server initiates a http span as broker sent a request to it
  • server now routes it a handler hence creates a process span.

Key thing is you have a broker who initiates a http request to the server. This process span would be very difficult to accurately create as the same endpoint could be called from many brokers.

That Messaging Process span, enriched with CloudEvents attributes and links, is what I intend this opt-in helper to provide when existing instrumentation does not already cover the processing operation.

Whereas for me the first thing the extension needs to do is cover the cloud events use case which is how to propagate distributed tracing otel. This applies to all transports as even already instrumented transports will not support the ce distributed tracing extension, hence that needs to be implemented

Whether this SDK should maintain that helper, and how to split the PR, are separate questions.

It is not a question of if a distributed tracing helper should be maintained by ce but rather is it ce responsibility to instrument transports which are not instrumented. Note, the pr can't be merged without answering that question but If the pr was focussed on a single delivery it would be much more likely to progress and is not tied into that bigger discussion.

I don’t think #1182 establishes that this processing span structure should be avoided.

What it does specify is the narrow use case of when it should be used and tbh if you are using that architecture, it would be more beneficial to instrument the framework.

@kahirokunn
kahirokunn force-pushed the opentelemetry-tracing branch 2 times, most recently from efdd469 to a778b4a Compare November 16, 2026 01:43

@thompson-tomo thompson-tomo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for greatly descoping this PR, it made it much easier to review.

Comment thread src/otel/propagation.ts
Comment on lines +19 to +49
/**
* Copy a valid span context into a new CloudEvent, replacing its tracing extension.
* @param {Context} context the trace context to inject
* @param {CloudEvent} event the event to copy
* @returns {CloudEvent} an event copy for a valid span context, otherwise the original event
*/
export function injectDistributedTracingExtension<T>(context: Context, event: CloudEvent<T>): CloudEvent<T> {
const extension: Record<string, string> = {};
propagator.inject(context, extension, defaultTextMapSetter);
if (!extension.traceparent) {
return event;
}

// cloneWith() merges properties, so remove the previous tracestate before copying the extension.
const properties: Partial<CloudEventV1<T>> = { ...event };
delete properties.traceparent;
delete properties.tracestate;
Object.assign(properties, extension);
// Preserve events that were constructed without validation.
return new CloudEvent<T>(properties, false);
}

/**
* Extract the event's trace context while preserving other values from the supplied context.
* @param {Context} context the base context
* @param {CloudEventV1} event the event to read
* @returns {Context} the combined context, or the supplied context for a missing or invalid extension
*/
export function extractDistributedTracingExtension(context: Context, event: CloudEventV1<unknown>): Context {
return propagator.extract(context, event, getter);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason not to define this as an actual otel propagator & then have ce helpers which call this propagator plus adds the links?

This way the client application can be simpler & you could have it automatically extract the ce context on Web hooks at http span creation time if w3 context is not available & headers are used.

@kahirokunn
kahirokunn force-pushed the opentelemetry-tracing branch 3 times, most recently from bc106f8 to 7c35873 Compare November 16, 2026 06:29
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
kahirokunn force-pushed the opentelemetry-tracing branch from 7c35873 to e344c0e Compare November 16, 2026 06:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants