Internal telemetry routing WIP DRAFT #1672

jmacd · 2025-12-20T00:57:51Z

Related to #1663.

This is a proposal to support an internal route for telemetry that lets us process OTAP-Dataflow telemetry using our own pipeline support. This requires special protections from self-induced telemetry, and it requires options to route telemetry in many ways to remain compatible with OpenTelemetry and keep our options. The proposal to accompany this PR is in the ARCHITECTURE.md draft.

…d/otel_sdk_logs_bridge

codecov · 2025-12-20T01:01:16Z

Codecov Report

❌ Patch coverage is 56.05048% with 592 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.02%. Comparing base (3ce308b) to head (38bda49).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1672      +/-   ##
==========================================
- Coverage   84.27%   84.02%   -0.25%     
==========================================
  Files         464      471       +7     
  Lines      133941   135276    +1335     
==========================================
+ Hits       112884   113672     +788     
- Misses      20523    21070     +547     
  Partials      534      534

Components	Coverage Δ
otap-dataflow	`85.42% <56.05%> (-0.43%)`	⬇️
query_abstraction	`80.61% <ø> (ø)`
query_engine	`90.25% <ø> (ø)`
syslog_cef_receivers	`∅ <ø> (∅)`
otel-arrow-go	`53.50% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

andborja · 2025-12-20T01:01:30Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+internal telemetry pipeline is rigorously safeguarded against these
+pitfalls through:
+
+- OTAP-Dataflow components downstream of an ITR cannot be configured


ITR - first time usage. Define.

andborja · 2025-12-20T01:26:21Z

rust/otap-dataflow/crates/telemetry/ARCHITECTURE.md

+with:
+
+- No-op logging
+- Raw logging


If "Raw logging" means console output, It should be through the SDK. It is yet another exporter, and should not be considered by separate.

for stdout/console, we cannot (and should not) use OTel. Need to use tracing::fmt or develop customer.

Yes the OTel exporter is not specified as a stable output format by OTel.

I see that it is a JSON-line oriented exporter, and I would welcome such a thing (@gouslu and I discussed deriving such a thing from the AzMon exporter code, which is similar to JSON-line).

In this PR as proof of concept I replaced tracing::fmt and developed an OTLP-bytes-to-console exporter, that's essentially what you're saying I think @cijothomas.

Actually, I've seen logging SDKs emit directly to protobuf bytes before! One reason we can't use tracing::fmt over the OTLP bytes representation in this case (AFAICT) is that the tracing Event struct does not contain a timestamp, there's no way to format a log statement recorded in the past. This is not the case for OTel SDK, why in this proposal we're able to reconstruct an OTLP SDK log event and process/export as an alternative to the ITR.

andborja · 2025-12-20T01:28:50Z

rust/otap-dataflow/crates/otap/src/internal_telemetry_receiver/mod.rs

@@ -0,0 +1,72 @@
+// Copyright The OpenTelemetry Authors


avoid the "mod.rs" format. Use a file internal_telemetry_receiver.rs at the same level as lib.rs instead.

jmacd · 2025-12-20T15:36:12Z

I apologize-- the code in this PR is not acceptable for review, we may view it as a feasibility study to accompany the ARCHITECTURE.md document.

cijothomas · 2025-12-22T19:38:57Z

rust/otap-dataflow/crates/config/src/pipeline/service/telemetry/logs.rs

+    pub max_record_bytes: usize,
+
+    /// Maximum number of records in the bounded channel.
+    /// When full, new records fall back to raw console logger.


why fallback here, instead of keeping count of the drops?

cijothomas · 2025-12-22T19:39:51Z

rust/otap-dataflow/crates/config/src/pipeline/service/telemetry/logs.rs

+    #[serde(default = "default_max_record_bytes")]
+    pub max_record_bytes: usize,
+
+    /// Maximum number of records in the bounded channel.


"bounded" channel feels like internal implementation details; so we should avoid exposing them to public config..

cijothomas · 2025-12-22T19:42:23Z

rust/otap-dataflow/crates/engine/src/effect_handler.rs

+    pub(crate) resource_bytes: OtlpProtoBytes,
+    pub(crate) scope_name: String,
+    pub(crate) flush_threshold_bytes: usize,
+    pub(crate) overflow_sender: mpsc::UnboundedSender<Bytes>,


is it okay to use a unbounded sender for overflow purposes?

This PR is a draft, so it is probably a temporary shortcut. The rule I personally follow is to systematically use bounded channels together with a policy that specifies what should happen in case of overflow, that is, drop the incoming message or block the sender. This policy can be configurable, or chosen directly in the code depending on whether it is worth making configurable.

lquerel · 2025-12-26T18:18:41Z

rust/otap-dataflow/crates/config/src/pipeline/service/telemetry/logs.rs

+    /// Internal collection for component-level logs.
+    /// 
+    /// When enabled, component logs (otel_info!, otel_warn!, etc.) are routed through
+    /// an internal telemetry receiver in the OTAP pipeline, allowing use of built-in
+    /// batch processors, retry, and exporters (console, OTLP, etc.).


I'm a big fan of this approach.

lquerel

I mainly focused on the architecture document for now, which is very detailed and informative. A few general remarks:

In the long term, we should be able to define TelemetrySettings at a global level, at the engine level, with the possibility to override them in a specific pipeline configuration.
In the case of a tight integration with the otap-engine, I think we need to find a way to reuse the AttributeSet concept already used for metrics. This offers several advantages:
1. we get a common language and definition for attributes,
2. they are populated by the engine and registered only once, which avoids redefining them on every call. We only need to provide an attribute set ID, which is essentially just a number. Additional "dynamic" attributes could be added when needed.
We should be able to compile the engine in a way that eliminates all overhead for a given logging level, essentially the same behavior we have today with the current macros. I believe this is already the plan, but I wanted to make this rule explicit.

lquerel · 2025-12-26T18:41:37Z

rust/otap-dataflow/crates/config/src/pipeline/service/telemetry/logs.rs

+    /// When disabled (default), component logs are routed to the OpenTelemetry SDK,
+    /// using the same export path as 3rd party logs from tokio-tracing-rs.
+    #[serde(default)]
+    pub internal_collection: InternalCollectionConfig,


Long term, I think this approach will apply not only to logs, but also to metrics and traces. At some point, we might promote this configuration to a more general level.
For now, I suggest that this field be:

an option, so we can remove the enabled field from InternalCollectionConfig, which I think would simplify things

renamed to something more explicit. I'm not fully satisfied with my proposal otap_pipeline; there is probably a better name to find

lquerel · 2025-12-26T18:47:36Z

rust/otap-dataflow/crates/engine/src/local/processor.rs

+    ///
+    /// This method never fails - errors are silently dropped to prevent recursion.
+    /// If the telemetry buffer is not configured, this is a no-op.
+    pub fn log_event(&self, log_record: &impl otap_df_pdata::views::logs::LogRecordView) {


We need to find a way to reuse the NodeAttributeSet that we already use for metrics. That will let every log emitted by a node share a common context with the metrics.

jmacd added 14 commits December 18, 2025 09:59

planning stateful encoder

152f55e

tokio-tracing-rs to OTLP bytes

fc70f4a

logs and traces

8175afc

otlp bytes ftw

729d8af

Starting ITR

3a879eb

ITR flushed

17be0b7

remove docs

203c73a

Merge branch 'main' of github.com:open-telemetry/otel-arrow into jmac…

0b01d28

…d/otel_sdk_logs_bridge

fixed

ab2714a

remove code snips

e6051c9

ITR

880ff22

less docs

08cff09

diagram

aa63561

revert a lot

38bda49

github-project-automation bot added this to OTel-Arrow Dec 20, 2025

github-actions bot added the rust Pull requests that update Rust code label Dec 20, 2025

andborja reviewed Dec 20, 2025

View reviewed changes

cijothomas reviewed Dec 22, 2025

View reviewed changes

lquerel reviewed Dec 26, 2025

View reviewed changes

Internal telemetry routing ** WIP DRAFT ** #1672

Are you sure you want to change the base?

Internal telemetry routing ** WIP DRAFT ** #1672

Uh oh!

Conversation

jmacd commented Dec 20, 2025

Uh oh!

codecov bot commented Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmacd commented Dec 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lquerel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Internal telemetry routing WIP DRAFT #1672

Internal telemetry routing WIP DRAFT #1672

codecov bot commented Dec 20, 2025 •

edited

Loading