Datadog Observability Pipelines is a Datadog product that runs a Worker inside your own infrastructure to collect, process and route logs and metrics before they reach Datadog or another destination. You build and manage pipelines from the Datadog UI, API or Terraform, and Datadog bills it per GB of uncompressed data the Worker ingests, from $0.095 per GB per month on an annual plan, or per vCPU for large volumes. It fits best when Datadog is already the center of your observability stack; processing at the source fits better when data starts at many sites, has to stay where it is produced, or is not observability data at all.
This guide covers what the product does, the cost questions buyers ask about it, and where a source-side pipeline is the better tool. Every Datadog fact below comes from Datadog’s own documentation and pricing pages as of September 27, 2026, and links to them. The Expanso example at the end was run, and its output is shown as produced.
What Datadog Observability Pipelines does
The product has two parts. The Observability Pipelines Worker is software you install in your environment. In Datadog’s words, it “runs within your infrastructure to aggregate, process, and route data.” The pipeline definitions live in Datadog, where you build them from templates, deploy them to Workers and watch their health.
Each pipeline has a source, a group of processors that run from top to bottom, and one or more destinations:
- Sources. The sources page lists log sources including the Datadog Agent, OpenTelemetry, Kafka, Syslog, Fluentd and Fluent Bit, Logstash, Splunk HEC and Splunk forwarders, Amazon S3 and Amazon Data Firehose. Metrics come from the Datadog Agent or OpenTelemetry.
- Processors. The processors page lists filter, sample, quota, throttle, deduplicate, edit fields, parse JSON and XML, Grok parsing, enrichment tables, generate metrics, remap to OCSF and a Sensitive Data Scanner processor for logs, plus aggregation and tag cardinality controls for metrics.
- Destinations. The destinations page lists Datadog Logs and Datadog Archives alongside Amazon S3, Azure Storage, Google Cloud Storage, Splunk HEC, Microsoft Sentinel, Google SecOps, CrowdStrike Next-Gen SIEM, Elasticsearch, OpenSearch, Kafka, ClickHouse and others.
Datadog’s product page adds more than 100 ready-to-use Packs for common sources and Live Capture for inspecting events as they pass through a pipeline.
Where the Worker runs and what it needs
The install guide covers Docker, Kubernetes, Linux, CloudFormation and ECS Fargate. Two requirements shape where it can go:
- A connection to Datadog. The Worker starts with a Datadog API key and a pipeline ID, and the guide notes that “the API key must be enabled for Remote Configuration.” Firewalled deployments must allow outbound traffic to Datadog endpoints, including
api.<DD_SITE>andconfig.<DD_SITE>on port 443. - Aggregator-class compute. The scaling guide recommends 8 vCPUs per instance, at least 4, and at least 2 GiB of memory per vCPU. In Datadog’s benchmark, a pipeline using 12 processors handled “approximately 1 TB per vCPU per day.”
On placement, Datadog and source-side processing agree. The same guide says: “Datadog recommends the decentralized approach of deploying the Workers as close to the data source as possible,” meaning within each region, cluster or datacenter where data originates. The open question for most buyers is how close is close enough, and what it costs to run a Worker there.
How much does Datadog Observability Pipelines cost?
Datadog offers two ways to pay. These are US list prices from the pricing list and the Observability Pipelines pricing page:
| Billed | Per ingested GB, per month | Plus, per vCPU, per month |
|---|---|---|
| Annually | $0.095 | $1,950 |
| Month-to-month | $0.10 | $2,340 |
| On-demand | $0.12 | $2,808 |
The pricing list names the per-vCPU plan Observability Pipelines Plus. The per-GB plan is “billed per GB of uncompressed data ingested.” Datadog positions it for volumes below 30 TB per month and the per-vCPU plan for volumes above that, and suggests provisioning “1 vCPU for every 30TB/month” as a general rule. At list price, 10,000 GB a month on the annual per-GB plan is $950 a month. Your contract, volume discounts and region can change every number here, so treat the table as a starting point for the conversation with Datadog, not a quote.
Cost questions to settle before you buy
The list price is the easy part. These are the questions that decide what the product costs you in practice.
Which meter will you be on? Per GB is simple and scales linearly with volume. Per vCPU is billed on the vCPUs allocated to Workers, so ask how redundancy counts. The scaling guide recommends at least 4 vCPUs per instance, and a highly available deployment runs more than one instance.
Does data the Worker drops still count? The per-GB plan meters uncompressed data the Worker ingests. Read literally, an event the Worker filters out was still ingested before it was dropped. Ask Datadog how your contract counts it, because the answer decides whether aggressive filtering lowers the pipeline fee or only the destination fees.
What does each destination charge? Observability Pipelines sits in front of your destinations; it does not replace their bills. If Datadog Logs is a destination, Log Management charges its own ingestion ($0.10 per GB) and indexing (from $1.06 per million events at 3-day retention, billed annually). The pipeline reduces what reaches those meters. A SIEM or archive destination has its own pricing too.
Who pays for the Workers? You run them, so their compute, storage and operations are on your infrastructure bill in addition to the Datadog fee.
Where does the Worker sit relative to the network? A Worker in each region avoids moving data between regions. A Worker in a central cluster means every byte crosses the network before anything is dropped.
Is it available on your Datadog site? The pricing page states that Observability Pipelines “is currently unavailable on the US-FED site.” Teams on US1-FED need a different approach.
Our Datadog alternatives guide covers the other side of the bill: Log Management exclusion filters, custom metrics governance and commitment plans.
When Datadog Observability Pipelines is a good fit
It is a strong choice when Datadog is where your team already works. You get pipelines built from templates in the same product as your dashboards and monitors, Sensitive Data Scanner rules you may already use, one vendor relationship and one support contract.
It also suits SIEM migrations and dual shipping. With Splunk, Sentinel, Google SecOps and CrowdStrike among its destinations, and Splunk forwarders among its sources, it can feed an old and a new platform from the same stream while you move. If your logs already flow through Datadog Agents into a regional cluster, adding a Worker there is a small architectural step.
When processing at the source is the better fit
Source-side processing means the pipeline runs on or next to the machine that produces the data, so the record that leaves is already the record you meant to send. It is the better fit in five situations.
Your data starts at many small sites. Stores, plants, vehicles and cell sites each produce modest volumes, and an aggregator sized to Datadog’s guidance of 4 to 8 vCPUs may be more than one site needs. An agent on hardware you already run at each site is a different trade.
Links are intermittent. The Worker takes its configuration from Datadog through Remote Configuration, so ask Datadog how it behaves at a site whose link drops for hours. Expanso Edge nodes keep running the pipelines they already have when they lose their connection to Expanso Cloud, and reconcile their state with the control plane when it returns.
The data has to stay where it is produced. When residency or contractual rules say a field must never leave a site, removing it at the source is simpler to reason about than removing it after the first hop. We covered the pattern in Remove PII From Logs Before You Export Them.
Not all of it is observability data. Sensor readings over MQTT, database extracts and business events need the same filtering and routing, and often go to warehouses and object stores rather than an observability backend. Expanso’s component catalog lists 50 inputs, 80 processors and 55 outputs, documented on docs.expanso.io.
You want the drop decision before any per-GB meter. Data a source-side pipeline drops never reaches a pipeline or destination meter, and never crosses the network.
The two are not exclusive. A source-side pipeline can shape and trim logs, then deliver them straight to Datadog’s intake, or to an Observability Pipelines Worker through its HTTP Server source. The example below does the first.
A runnable example: shape logs for Datadog at the source
This job reads application logs on the machine that wrote them, drops debug output and health checks, removes an email address from a message, keeps only the fields worth sending, and posts the result to Datadog’s HTTP logs intake in the format its API documents: a JSON array of log objects with ddsource, ddtags, hostname, service and message, sent with a DD-API-KEY header.
We ran it on September 27, 2026 with expanso-edge v2.1.21 on macOS, using a local-mode node that needs no account. To keep a real API key out of a demo, the Datadog intake was replaced by a small local receiver that records each request and answers 202 Accepted, the status Datadog’s API documents for accepted logs. Nothing was sent to Datadog.
The input
Save these twelve synthetic lines as app.jsonl. They include three debug lines, three health checks, an email address in an error message, bulky user_agent and headers fields, a stack trace and one line that is not JSON.
{"ts":"2026-09-27T12:00:00Z","host":"store-042","service":"checkout","level":"INFO","msg":"order placed","order_id":"o-1001","user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15","headers":{"accept":"application/json","accept-language":"en-US,en;q=0.9","x-request-id":"r-5a1"}}
{"ts":"2026-09-27T12:00:01Z","host":"store-042","service":"checkout","level":"DEBUG","msg":"cache lookup","key":"cart:88"}
{"ts":"2026-09-27T12:00:02Z","host":"store-042","service":"gateway","level":"INFO","msg":"GET /healthz 200","path":"/healthz"}
{"ts":"2026-09-27T12:00:03Z","host":"store-042","service":"gateway","level":"INFO","msg":"GET /healthz 200","path":"/healthz"}
{"ts":"2026-09-27T12:00:04Z","host":"store-042","service":"checkout","level":"ERROR","msg":"card declined for [email protected]","order_id":"o-1002","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36","headers":{"accept":"application/json","x-request-id":"r-5a2"}}
{"ts":"2026-09-27T12:00:05Z","host":"store-042","service":"checkout","level":"DEBUG","msg":"retrying payment","attempt":2}
{"ts":"2026-09-27T12:00:06Z","host":"store-042","service":"gateway","level":"INFO","msg":"GET /api/orders 200","path":"/api/orders","latency_ms":38,"user_agent":"okhttp/4.12.0","headers":{"accept":"*/*","x-request-id":"r-5a3"}}
{"ts":"2026-09-27T12:00:07Z","host":"store-042","service":"gateway","level":"INFO","msg":"GET /healthz 200","path":"/healthz"}
{"ts":"2026-09-27T12:00:08Z","host":"store-042","service":"inventory","level":"WARN","msg":"stock sync slow","latency_ms":2210}
checkout: connection reset by peer
{"ts":"2026-09-27T12:00:10Z","host":"store-042","service":"checkout","level":"FATAL","msg":"worker crashed","order_id":"o-1003","stack":"Error: worker crashed\n at processQueue (worker.js:88:11)\n at runNext (worker.js:41:5)"}
{"ts":"2026-09-27T12:00:11Z","host":"store-042","service":"inventory","level":"DEBUG","msg":"sku scan","sku":"sku-778"}Before running it, write down the expected result: six lines dropped, one rejected with a reason, and five sent to Datadog, two of them as errors.
The pipeline
Save this as datadog-at-source.yaml:
name: datadog-at-source
type: pipeline
restart_policy: never
config:
input:
file:
paths: ["./app.jsonl"]
scanner:
lines: {}
pipeline:
processors:
- mapping: |
let rec = content().string().parse_json().catch(null)
let raw = $rec.level.or("").string().lowercase()
let sev = match $raw {
"fatal" => "error"
"warning" => "warn"
_ => $raw
}
root = if $rec == null {
{"rejected": true, "reason": "not valid JSON",
"raw": content().string()}
} else if $sev == "debug" || $rec.path.or("") == "/healthz" {
deleted()
} else {
{
"ddsource": "app",
"service": $rec.service,
"hostname": $rec.host,
"status": $sev,
"ddtags": "env:prod,site:" + $rec.host,
"timestamp": $rec.ts,
"message": $rec.msg.re_replace_all(
"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]+",
"[email redacted]"),
"order_id": $rec.order_id,
"latency_ms": $rec.latency_ms,
"error": if $rec.stack != null {
{"stack": $rec.stack}
} else { null }
}.filter(kv -> kv.value != null)
}
output:
switch:
cases:
- check: this.rejected.or(false)
output:
file:
path: "./out/rejects.jsonl"
codec: lines
- output:
broker:
pattern: fan_out
outputs:
- file:
path: "./out/sent.jsonl"
codec: lines
- http_client:
url: ${DD_LOGS_URL}
verb: POST
headers:
Content-Type: application/json
DD-API-KEY: ${DD_API_KEY}
batching:
count: 1000
period: 5s
processors:
- archive:
format: json_arrayThe Bloblang mapping does the work. It normalizes severity, rejects lines that are not JSON as data with a reason, drops debug and health-check lines, and builds a new record from named fields only, so user_agent and headers never leave the host. It replaces anything shaped like an email address in the message and moves the stack trace to error.stack, the attribute Datadog Error Tracking reads. Records it cannot parse go to a rejects file instead of disappearing, a trap we covered in Filter Noisy Logs Without Losing the Errors.
The output sends every kept record to two places: a local file you can read, and the http_client output, which batches records into one JSON array per request. Datadog’s API accepts up to 1,000 entries and 5 MB uncompressed per request, hence count: 1000. The URL and key come from environment variables resolved when the pipeline starts, so the same file points at a test receiver today and at Datadog tomorrow.
The stand-in receiver
Save this as capture.py. It plays the part of Datadog’s intake: it records the path, content type, whether the API key header arrived, and the body.
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
class H(BaseHTTPRequestHandler):
def do_POST(self):
body = self.rfile.read(int(self.headers["Content-Length"]))
rec = {
"path": self.path,
"content_type": self.headers.get("Content-Type"),
"api_key_header_present": "DD-API-KEY" in self.headers,
"body": json.loads(body),
}
with open("captured.jsonl", "a") as f:
f.write(json.dumps(rec) + "\n")
self.send_response(202)
self.end_headers()
HTTPServer(("127.0.0.1", 19311), H).serve_forever()Run it
Install expanso-edge and expanso-cli from the installation guide. Work in one directory throughout, because the file paths in the job resolve against the directory the node starts in. Check the job offline first:
mkdir -p out
expanso-edge validate datadog-at-source.yaml[OK] datadog-at-source.yaml: validStart the receiver in one terminal:
python3 capture.pyIn a second terminal, start a local-mode node with the two variables set. DD_API_KEY is a placeholder here, not a real key:
export DD_LOGS_URL=http://127.0.0.1:19311/api/v2/logs
export DD_API_KEY=not-a-real-key
expanso-edge run --local --data-dir ./edge-dataIn a third terminal, deploy the job to the node and count the results:
expanso-cli job deploy datadog-at-source.yaml \
--endpoint http://localhost:9010
wc -l out/*.jsonl captured.jsonl
wc -c app.jsonl out/sent.jsonlStop the node and the receiver with Ctrl+C when you are done.
What the run produced
The job ended in the completed state, and the counts matched the prediction:
1 out/rejects.jsonl
5 out/sent.jsonl
1 captured.jsonl
7 total
1978 app.jsonl
1114 out/sent.jsonl
3092 totalTwelve lines went in. Six were dropped as noise, one was rejected with its reason, and five were sent. On this synthetic file, 1,978 bytes of input became 1,114 bytes of output, because the dropped lines and the user_agent and headers fields never left the host. That is a property of this file, not a benchmark; your ratio depends on your logs.
These are the five records, exactly as sent. Their order can differ from the input:
{"ddsource":"app","ddtags":"env:prod,site:store-042","hostname":"store-042","message":"order placed","order_id":"o-1001","service":"checkout","status":"info","timestamp":"2026-09-27T12:00:00Z"}
{"ddsource":"app","ddtags":"env:prod,site:store-042","hostname":"store-042","message":"card declined for [email redacted]","order_id":"o-1002","service":"checkout","status":"error","timestamp":"2026-09-27T12:00:04Z"}
{"ddsource":"app","ddtags":"env:prod,site:store-042","hostname":"store-042","latency_ms":38,"message":"GET /api/orders 200","service":"gateway","status":"info","timestamp":"2026-09-27T12:00:06Z"}
{"ddsource":"app","ddtags":"env:prod,site:store-042","error":{"stack":"Error: worker crashed\n at processQueue (worker.js:88:11)\n at runNext (worker.js:41:5)"},"hostname":"store-042","message":"worker crashed","order_id":"o-1003","service":"checkout","status":"error","timestamp":"2026-09-27T12:00:10Z"}
{"ddsource":"app","ddtags":"env:prod,site:store-042","hostname":"store-042","latency_ms":2210,"message":"stock sync slow","service":"inventory","status":"warn","timestamp":"2026-09-27T12:00:08Z"}The receiver recorded one request: a POST to /api/v2/logs with Content-Type: application/json, the DD-API-KEY header present, and a body that was a JSON array of those five records. The rejects file held the one line that was not JSON:
{"raw":"checkout: connection reset by peer","reason":"not valid JSON","rejected":true}Pointing it at Datadog
To send to Datadog instead, start the node with DD_API_KEY set to a real key from your secret store and DD_LOGS_URL set to your site’s logs intake. For the US1 site, that is:
https://http-intake.logs.datadoghq.com/api/v2/logsTwo limits of this run are worth stating plainly. We proved the request shape against a stand-in, not Datadog’s acceptance of it, so send a small batch and confirm it in Datadog’s Log Explorer first. And the example does not compress requests; Datadog’s API documentation recommends gzip for larger volumes.
To run the same job on real hosts, deploy it through Expanso Cloud with a label selector, so it runs on the nodes next to the logs it reads. The first five nodes are free; see pricing. For larger log-reduction patterns, the log reduction recipe and our log processing use case go further, and What Is an Observability Pipeline? and Cribl alternatives cover the wider category.
FAQ
What is Datadog Observability Pipelines?
It is a Datadog product that runs a Worker in your own infrastructure to collect, process and route logs and metrics. Pipelines are built and managed in Datadog and can send data to Datadog and to other destinations, such as Splunk, Microsoft Sentinel and Amazon S3.
How much does Datadog Observability Pipelines cost?
At US list price, $0.095 per GB of uncompressed data ingested per month billed annually, $0.10 month-to-month or $0.12 on-demand. A per-vCPU plan, listed as Observability Pipelines Plus at $1,950 per vCPU per month billed annually, is recommended for volumes above 30 TB per month. Destination costs, such as Datadog Log Management, are billed separately.
Does Observability Pipelines replace Datadog Log Management?
No. It is a pipeline that sits in front of destinations. If you send logs to Datadog, Log Management still bills its own ingestion and indexing for what arrives.
Where does the Observability Pipelines Worker run?
In your environment, on Docker, Kubernetes, Linux, CloudFormation or ECS Fargate. Datadog recommends deploying Workers as close to the data source as possible, in each region, cluster or datacenter where data originates.
Can I filter logs before they reach Datadog without Observability Pipelines?
Yes. Any pipeline that can call Datadog’s HTTP logs intake can shape and filter logs first. The example in this post does it with Expanso Edge running next to the source.
Is Datadog Observability Pipelines available for government customers?
Datadog’s pricing page states that the product is currently unavailable on the US-FED site. Check with Datadog for your specific site and requirements.
