Skip to content

Conversation

@lalitb
Copy link
Member

@lalitb lalitb commented Dec 22, 2025

fixes: #1678

Summary

Implements HTTP CONNECT proxy support for OTLP/OTAP gRPC exporters:

  • HTTP/1.1 CONNECT tunneling to the proxy for both http:// targets (gRPC over h2c inside the tunnel) and https:// targets (TLS inside the tunnel).
  • Environment variable support (HTTP_PROXY, HTTPS_PROXY, NO_PROXY, ALL_PROXY)
  • NO_PROXY with CIDR notation (e.g., 192.168.0.0/16, 10.0.0.0/8)
  • Configures TCP socket options (TCP_NODELAY + keepalive). When using a proxy, these are applied to the TCP connection to the proxy (the same connection that carries the CONNECT tunnel).
  • Explicit rejection of https:// proxy URLs with helpful error messages

Why OTLP/OTAP exporters need custom proxy implementation

Azure Monitor and Geneva exporters use the reqwest HTTP client, which provides built-in proxy support via reqwest::Proxy::all() - no custom code needed.

OTLP/OTAP exporters use gRPC (tonic), which:

  • Has no built-in proxy support
  • Requires custom TCP connectors with tower::service_fn
  • Needs manual HTTP CONNECT tunnel implementation for proxy traversal

This PR implements the missing proxy infrastructure for gRPC-based exporters.

Changes

  • proxy.rs: HTTP/1.1 CONNECT tunnel implementation with socket option handling
  • client_settings.rs: Integration with tonic via custom connector
  • Dependencies: Added socket2 and ipnet crates

How HTTP CONNECT Tunneling Works

Step 1: THE HANDSHAKE
The Exporter creates a TCP connection to the Proxy and sends a plaintext request. The Proxy establishes a TCP leg to the Backend.

┌───────────┐  TCP + HTTP/1.1 CONNECT backend:PORT       ┌───────────┐
│ Exporter  │ ─────────────────────────────────────────>.│   Proxy   │
└───────────┘ <─────── 200 Connection Established ────── └───────────┘

Step 2: THE DATA TUNNEL

Once the 200 is received, the Exporter uses the socket for the actual OTLP data. The Proxy merely moves bytes back and forth.

┌───────────┐       ┌───────────┐       ┌───────────────┐
│ OTLP/OTAP │  TCP  │   Proxy   │  TCP  │    Backend    │
│ Exporter  │══════>│ (relays)  │══════>│               │
└───────────┘       └───────────┘       └───────────────┘
          ║                                     ║
          ╚═════════════════════════════════════╝
     On-the-wire protocol inside the tunnel (opaque to proxy):
     - Case 1 (TLS): TLS + gRPC over HTTP/2 (HTTP/2 negotiated via ALPN)
     - Case 2 (h2c): gRPC over HTTP/2 cleartext (HTTP/2 prior knowledge)

Test Setup:

The implementation was verified via a manual end-to-end test setup (not included in the PR). The architecture below was used to validate proxy traversal for both h2c and TLS traffic:

┌─────────────────┐
│ Fake Data Gen   │ Generates test telemetry (logs/traces/metrics)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ OTAP Exporter   │ gRPC client with proxy configuration
│ (df_engine)     │ • HTTP_PROXY=http://localhost:8080
│                 │ • grpc_endpoint=http://remote:4317
│                 │ • admin_port: 8081 (changed to avoid proxy conflict)
└────────┬────────┘
         │
         │ HTTP CONNECT tunneling
         ▼
┌─────────────────┐
│   mitmproxy     │ Intercepts and logs proxy traffic
│   :8080         │ (validates CONNECT requests are working)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ OTel Collector  │ Receives telemetry via gRPC
│   :4317         │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Debug Exporter  │ Displays received data
└─────────────────┘

@github-actions github-actions bot added the rust Pull requests that update Rust code label Dec 22, 2025
@codecov
Copy link

codecov bot commented Dec 22, 2025

Codecov Report

❌ Patch coverage is 87.29690% with 86 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.44%. Comparing base (8e81cc8) to head (7c26273).

Additional details and impacted files
@@           Coverage Diff            @@
##             main    #1679    +/-   ##
========================================
  Coverage   84.43%   84.44%            
========================================
  Files         465      466     +1     
  Lines      135552   136229   +677     
========================================
+ Hits       114460   115036   +576     
- Misses      20558    20659   +101     
  Partials      534      534            
Components Coverage Δ
otap-dataflow 85.95% <87.29%> (-0.01%) ⬇️
query_abstraction 80.61% <ø> (ø)
query_engine 90.39% <ø> (ø)
syslog_cef_receivers ∅ <ø> (∅)
otel-arrow-go 53.50% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Limit HTTP headers in proxy response to prevent DoS (100 headers, 8KB max)
- Fix IPv6 address parsing in NO_PROXY (handle bracketed literals)
- Redact credentials from proxy URLs in debug logs
- Clarify performance documentation for has_proxy/effective_proxy_config
- Restrict visibility of internal proxy helper methods
- Fix test compilation error in otlp_exporter_proxy_tls.rs
- Make effective_proxy_config private as it's only used internally
- Remove unused has_proxy method from GrpcClientSettings
Remove unused field 'rusage_thread_supported' from struct.
let tcp_keepalive_retries = self.tcp_keepalive_retries;

service_fn(move |uri: http::Uri| {
let proxy = Arc::clone(&proxy);
Copy link
Member Author

@lalitb lalitb Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use Arc because tonic/tower requires Send + 'static for connectors.

let tcp_keepalive_interval = self.tcp_keepalive_interval;
let tcp_keepalive_retries = self.tcp_keepalive_retries;

service_fn(move |uri: http::Uri| {
Copy link
Member Author

@lalitb lalitb Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This closure runs per TCP connection, not per RPC. With HTTP/2 multiplexing, connections are long-lived (startup + reconnects only), so the cost here is negligible. The cost here involved are:

  • Arc::clone for proxy object,
  • get_proxy_for_uri lookup
  • string operations in should_bypass (have TODO in code to optimize this later).

Also this code path is only invoked when a proxy is configured. Without a proxy, tonic uses its default connector directly, so above overhead is not there.

@lalitb lalitb marked this pull request as ready for review December 26, 2025 17:30
@lalitb lalitb requested a review from a team as a code owner December 26, 2025 17:30
@lalitb
Copy link
Member Author

lalitb commented Dec 26, 2025

this is ready for review now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rust Pull requests that update Rust code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Add HTTP proxy support for OTAP and OTLP gRPC exporters

1 participant