New component: Prometheus Remote Write receiver

[tomatopunk]

The purpose and use-cases of the new component

The new component, Prometheus remote write receive, is designed to enable OpenTelemetry collectors to receive metrics data from Prometheus instances via the remote write protocol. This allows for seamless integration of Prometheus metrics into the OpenTelemetry ecosystem, enabling centralized metrics collection, processing, and exporting to various backends.

Example configuration for the component

receivers:
  prometheusremotewrite:
    endpoint: "0.0.0.0:12345"  # The address to listen on for Prometheus remote write 

Telemetry data types supported

metric

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@tomatopunk

Sponsor (optional)

@andrzej-stencel

Additional context

The tracking issue #10358, #14751 was closed because this component did not have a sponsor.

[dashpole]

See #14751 (comment). The lack of transactionality in PRW 1.0 makes it difficult to implement a working receiver for PRW. IMO, we should wait to introduce this until PRW 2.0, which was just released as experimental: https://prometheus.io/docs/specs/remote_write_spec_2_0/. It solves the issues raised in the previous proposal.

[ChrsMark]

Example configuration for the component

receivers:
prometheusremotewrite:
   endpoint: "0.0.0.0:12345"  # The address to listen on for Prometheus remote write 

Should we consider tls+https support as well instead of just plain http?

[andrzej-stencel]

Thanks @dashpole, this is very valuable information.

Let's make sure the new receiver supports the PRW 2.0 spec.

[vipinvkmenon]

Looking forward to this feature. Id be more than happy to test and provide feedbacks if any once merged

[JustinMason]

#34196

I have a great initial implementation of a remoteWrite receiver, this foundation implemented all the configurations supported by the oltp receiver. So HTTP, gRPC, CORS, TLS and mTLS, and auth settings are all in there.

This mirrored the https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver and its initial implementation is limited to sum and gauge metric mapping, but it is stable. The receiver has been running reliability in a production environment getting remoteWrites from prometheus and Victoria metrics.

[rakesh-p]

until this remotewrite receiver is ready, is there no other way to send metrics from prometheus to opentelemetry collector please?

we have several k8s clusters and each with its own prometheus running to collect the metrics from the cluster. we would like to send the metrics collected by all prometheus instances (1 in each cluster) to central opentelemetry collector.

So far, the only way i found is that i can add a prometheus receiver to opentelemetry pipeline which can go and scrap metrics directly from targets. Also the opentelemetry collectors run as a cluster behind load balancer. Does it mean, all the prometheus receivers will scrap the metrics from targets and hence will result in duplication?

Thanks

[rakesh-p]

for anyone using splunk-otel-collector, there is a remote write receiver https://docs.splunk.com/observability/en/gdi/opentelemetry/components/signalfx-gateway-prometheus-remote-write-receiver.html

[JustinMason]

@rakesh-p Im not sure about the signalfx receiver, it may be worth testing.
Consider building your own collector using the framework and include this module.
OTel Collector Builder

Here is a example of a collector and a initial implementation: https://github.com/JustinMason/otel-collector/tree/main/receiver/prometheusremotewritereceiver

[ArthurSens]

Hi @andrzej-stencel

I'm Arthur, a Prometheus team member and contributor to the otel-prometheus-wg!
I've noticed that this component's bootstrap has become stale, so I'm wondering if you're still interested in sponsoring the work here.

I'd love to work on a prototype that focuses on Prometheus Remote-Write v2 since we already know that v1 has a few limitations for good OTLP compatibility.

[dashpole]

I'm also happy to help with the review of a PRW 2.0 receiver

[dashpole]

I talked with @andrzej-stencel. I'm going to take over of the sponsor of the component to unblock development

[andrzej-stencel]

Thank you @dashpole!

[tomatopunk]

Thank you for all. Regarding the sponsor's not to proceed with reviewing and merging the PR, I will no longer be continuing its development.

[ArthurSens]

No worries @tomatopunk! Thanks for quickstarting the conversations here, I'm volunteering to continue your work :)