Prometheus provides great short-term metrics storage for the Grafana stack. However, it has its scalability limitations. Using remoteWrite to transition to a new architecture while preserving the existing flow of metrics is a great option for some.
When enabled, remote write increases the memory footprint of Prometheus. On average the increase is about 25%, but that number depends on the shape of the data. It is because each remote write destination starts a queue which reads from the write-ahead log (WAL), writes the samples into an in memory queue owned by a shard, which then sends a request to the configured end-point.
|--> queue (shard 1) ---> remote end-point
WAL ---|--> queue (shard 2) ---> remote end-point
|--> queue (shard n) ---> remote end-pointBelow example is in Helm and demonstrates how to enable remoteWrite.
remoteWrite:
- url: https://<target end-point e.g. mimir>/api/v1/push
basic_auth:
username: <user>
password: <password>
tls_config:
insecure_skip_verify: false