Telemetry data is any measurement or event transmitted automatically from a remote source to a receiving system for monitoring, analysis, or storage. The term comes from the Greek “tele” (remote) and “metron” (measure). Your Kubernetes cluster emitting pod metrics, a cell tower reporting signal strength, a SaaS application logging API response times: all telemetry.
The concept is straightforward. The challenge is volume. A single Kubernetes cluster can produce 10-50 GB/day of logs alone. A telecom operator monitoring 3,847 cell sites generates terabytes of infrastructure telemetry per day. IoT fleets with thousands of sensors produce millions of data points per minute. Collecting this data is the easy part. Processing, routing, and paying for it at scale is where organizations struggle.
This post covers what telemetry data is, the types you’ll encounter, how collection works, and how to build a telemetry pipeline that doesn’t bankrupt your observability budget.
Telemetry Data Meaning: A Precise Definition
Telemetry data is machine-generated information collected and transmitted from a source to a remote system without manual intervention. It records what happened, when it happened, and where it happened. It does not require a human to initiate or approve each transmission.
Three properties distinguish telemetry from other data types:
Automated collection. Telemetry flows continuously or at defined intervals without human action. An agent on a server reports CPU usage every 10 seconds. A network switch exports flow records every minute. An application traces every HTTP request. The system generates the data as a byproduct of its operation.
Machine origin. Telemetry comes from infrastructure, applications, devices, and sensors. Not from users filling out forms or analysts writing reports. The source is a machine, and the data describes machine behavior.
Remote transmission. The data moves from where it’s generated to where it’s analyzed. That gap might be a few network hops within a data center or thousands of miles between an oil rig sensor and a cloud backend. The transmission is what makes it telemetry rather than a local log file.
In practice, the term covers a broad range of data: application logs, infrastructure metrics, distributed traces, network flow records, IoT sensor readings, security events, and business transactions. Any system that reports its own state or behavior to another system is producing telemetry.
Types of Telemetry Data
Telemetry splits into four primary categories. Most observability and monitoring systems are built around these types, and understanding the differences matters for pipeline design.
Logs
Logs are timestamped text records of discrete events. An application logs an error. A firewall logs a blocked connection. A container runtime logs a pod restart. Each entry describes a single event with enough context to understand what happened.
Logs are the most verbose telemetry type. A busy web server might produce 500 MB to 2 GB of access logs per day. A Kubernetes cluster with 200 pods can generate 10-50 GB/day of combined container logs. Debug-level logging in a microservices architecture can push that number to 100+ GB/day.
The challenge with logs is that most of them are noise. Health-check pings from a load balancer hitting every endpoint every five seconds. Debug output that was useful during development and never got turned off. Informational messages that confirm normal operation. Studies of enterprise deployments show that 60-80% of log data is never queried within 30 days of collection.
Metrics
Metrics are numerical measurements captured at regular intervals. CPU utilization at 73%. Memory usage at 4.2 GB. Request latency at 23ms. Error rate at 0.3%. Each metric has a name, a value, a timestamp, and usually a set of labels that identify the source.
Metrics are compact compared to logs. A single metric data point might be 100 bytes. But the cardinality problem makes them expensive at scale. If you track 500 metrics across 10,000 hosts with 20 label dimensions, the combinatorial explosion creates millions of unique time series. High-cardinality metrics are one of the fastest-growing cost drivers in observability platforms.
Common metric formats include Prometheus exposition format, StatsD, OpenTelemetry metrics, and platform-specific formats like CloudWatch Metrics or Azure Monitor Metrics.
Traces
Traces follow a single request as it moves through a distributed system. A user clicks “checkout,” and the trace captures the path through the API gateway, the order service, the payment processor, the inventory service, and back. Each step is a “span” with a start time, duration, status, and metadata.
Traces are the newest of the core telemetry types and the hardest to manage at scale. A single user request can produce dozens of spans. Multiply that by millions of requests per hour and the volume gets large fast. Tail-based sampling (keeping 100% of error traces while sampling 1-5% of successful ones) is a standard approach to control volume without losing visibility into failures.
OpenTelemetry has become the dominant standard for trace instrumentation, replacing vendor-specific SDKs from Jaeger, Zipkin, and proprietary tools.
Events
Events are structured records of significant state changes or actions. A deployment completed. A scaling event triggered. A certificate expired. A user account was locked. Events differ from logs in that they represent discrete, meaningful occurrences rather than a continuous stream of operational output.
Events tend to be low-volume but high-value. A Kubernetes cluster might produce thousands of events per hour (pod scheduling, scaling decisions, health-check failures), but each one represents an actionable state change. Security events (authentication failures, privilege escalations, policy violations) are the most operationally critical telemetry category for many organizations.
How Telemetry Data Is Collected
Collection methods vary by source type, but they fall into two patterns: push and pull.
Push-Based Collection
The source sends data to a collector or backend. The source decides when and how often to transmit. Examples:
- Application SDKs (OpenTelemetry, Log4j, Serilog) push logs, metrics, and traces to a collector endpoint.
- Syslog streams push log events over UDP or TCP to a syslog receiver.
- Webhooks push event notifications to HTTP endpoints.
- SNMP traps push alerts from network devices to a trap receiver.
Push-based collection works well when the source knows something happened and needs to report it immediately. It struggles when sources overwhelm the receiver with volume, creating backpressure and potential data loss.
Pull-Based Collection
A collector periodically scrapes or queries the source for data. The collector controls the timing. Examples:
- Prometheus scraping pulls metrics from /metrics endpoints at a configured interval.
- Log file tailing reads new lines from log files on disk.
- API polling queries cloud provider APIs (CloudWatch, Azure Monitor) for new data.
- SNMP polling queries network devices for interface counters and system stats.
Pull-based collection gives the collector control over volume and timing. It struggles with high-frequency events where the polling interval might miss short-lived spikes.
Agent-Based Collection
Most production deployments use agents: lightweight processes that run on or near the source and handle both push and pull collection. The agent collects data locally, buffers it, and forwards it to a central destination.
Agents range from heavy (Splunk Universal Forwarder at 200+ MB memory footprint) to light (Expanso’s agent, a single binary whose footprint scales with the pipeline it runs). The footprint matters when you deploy agents to thousands of edge devices, IoT gateways, or resource-constrained environments. An agent that stays small on 3,847 telecom sites is feasible. One that demands hundreds of megabytes at every site is a capacity problem.
Telemetry Pipeline Architecture
A telemetry pipeline is the infrastructure between data sources and destinations. It collects, processes, and routes telemetry to the systems that need it. The pipeline has three layers.
Collection Layer
Agents and receivers ingest telemetry from all sources. This layer handles protocol translation (converting syslog to structured JSON, Prometheus metrics to OTLP, proprietary formats to standard schemas) and initial buffering.
The number of supported connectors determines how much custom integration work your team avoids. A pipeline with 200+ connectors covers most enterprise sources out of the box. A pipeline with 20 connectors means your engineers are writing and maintaining glue code for every source that doesn’t fit.
Processing Layer
This is where telemetry gets shaped before it reaches its destination. Processing includes:
- Filtering: Dropping debug logs, health checks, and known noise patterns.
- Sampling: Keeping 1 in N events for high-volume, low-value streams.
- Enrichment: Adding metadata (geo-IP, Kubernetes labels, business unit tags) to events in flight.
- Transformation: Normalizing timestamps, renaming fields, converting formats.
- Masking: Redacting PII, credit card numbers, or PHI before data leaves a compliance boundary.
- Aggregation: Rolling up raw metrics into summaries (per-minute averages instead of per-second samples).
The processing layer is where cost control happens. Every event dropped or aggregated here is an event you don’t pay to ingest, store, and query downstream. A DSL (domain-specific language) for writing processing rules, like Bloblang, makes this work accessible to operations teams without requiring custom code for every transformation.
Routing Layer
Processed telemetry goes to one or more destinations based on content, source, severity, or any field in the event. Security logs route to the SIEM. Application metrics route to Datadog or Grafana. Raw archives route to S3. Business events route to a data lake.
Routing turns a single telemetry stream into multiple purpose-built feeds. Hot data goes to expensive, fast platforms. Cold data goes to cheap, deep storage. The same event can go to multiple destinations simultaneously.
A financial institution used this three-layer architecture to split 14.3TB/day of telemetry across Splunk (for real-time security analysis) and S3 (for compliance archives). Splunk costs dropped from $3.7M to $1.4M annually, a 62% reduction. The data wasn’t lost. It was routed to the right destination at the right cost.
The Scale Problem: Why Raw Telemetry Breaks Budgets
Telemetry volume grows 30-40% annually in most enterprises. The math is relentless.
If your observability platform costs $500K/year today and telemetry grows 35% per year, you’re paying $675K next year and $911K the year after. No new features. No new insights. Just more data hitting the same platform at the same per-GB rate.
The problem compounds in three ways:
Platform pricing punishes volume. Datadog charges $15/host/month for infrastructure monitoring, but APM adds $31/host/month, and log management adds per-GB ingestion fees. Splunk charges per GB ingested. New Relic charges per GB of data plus per-user fees. Every new source, every new service, every new host adds to the meter.
Most telemetry is never queried. Across enterprise deployments, 60-80% of ingested data goes unsearched within 30 days. Debug logs from production systems. Health-check responses that confirm normal operation every five seconds. Verbose trace spans for routine requests. Organizations pay full price to ingest, index, and store this data, then never look at it.
Edge and IoT multiply the problem. Distributed architectures generate telemetry far from centralized backends. Shipping raw data from thousands of edge locations means massive egress costs, bandwidth contention, and latency that makes real-time alerting impractical. A telecom operator with 3,847 sites can’t afford to ship every SNMP trap and syslog message to a cloud backend at full fidelity.
The result: observability becomes one of the fastest-growing line items in the IT budget, often outpacing the growth of the infrastructure it monitors.
Processing Telemetry at the Edge
The traditional approach ships all raw telemetry to a centralized processing cluster. Every event traverses the network, gets processed, and then gets forwarded to its destination. This works for small deployments. At scale, it creates a bandwidth bottleneck, a single point of failure, and double the egress costs (data travels to the processor, then to the destination).
Edge processing flips the model. A lightweight agent at each source handles filtering, transformation, and routing locally. Only the data that survives processing ships to central destinations. The benefits are concrete:
Bandwidth reduction. A telecom operator reduced telemetry volume by 78% at the source by filtering duplicate SNMP traps and redundant status messages. Instead of shipping terabytes of raw data from 3,847 sites, the edge agents sent only unique, actionable events. The network infrastructure that had been saturated with telemetry traffic got immediate relief.
Cost reduction. An enterprise reduced observability costs from $240K to $71K per month, a 70% reduction, by processing telemetry at the edge before it reached their observability platform. The platform received a focused, high-signal dataset. Average query latency dropped from 45 seconds to 2.8 seconds (a 16x improvement) because the indexed data was cleaner.
Real-time capability. Processing at the source means decisions happen in single-digit milliseconds, not hundreds of milliseconds round-trip to a cloud backend. For use cases like industrial monitoring, broadcast systems, or security event detection, that difference matters.
Compliance at the boundary. PII masking and data redaction happen before data leaves the source. Sensitive fields get hashed or stripped at the edge agent, so they never traverse the network in cleartext. For regulated industries (healthcare, finance, telecom), this turns a compliance risk into a solved problem.
Edge processing requires agents that are small enough to deploy everywhere. A lightweight agent can deploy to cell sites, retail locations, factory floors, and IoT gateways. A heavyweight agent that demands gigabytes of memory can’t.
Building a Telemetry Pipeline That Scales
A telemetry pipeline that works at 100 GB/day and a pipeline that works at 10 TB/day are different architectures. Here’s what separates the two.
Start with Connectors, Not Code
Every custom integration is maintenance debt. If your pipeline supports 200+ connectors out of the box, your team spends time writing processing rules instead of writing parsers for every new source. Check connector coverage before committing to a platform. The gap between “we support that” and “you need to write a custom input” shows up in the first month.
Filter Close to the Source
The most expensive byte is one that travels the full path from source to backend only to be dropped by a retention policy 30 days later. Move filtering as close to the source as possible. Debug logs, health checks, and duplicate events should never leave the machine that generated them if nobody is going to query them.
One financial institution discovered that 73% of their Splunk ingest was noise. They had been paying $3.7M annually to store data that served no operational or security purpose. Filtering at the source brought their Splunk costs down to $1.4M, saving $2.3M per year without losing any data their analysts actually used.
Use a DSL for Processing Rules
Processing rules written in a general-purpose language (Python, Go) are flexible but hard to audit, test, and hand off between teams. A purpose-built DSL for telemetry processing lets operations engineers write, read, and modify rules without the overhead of a full development workflow. Rules propagate to agents in under 30 seconds, so changes take effect across thousands of endpoints within a minute.
Route by Value, Not by Source
The default instinct is to route all logs to the log platform, all metrics to the metrics platform, and all traces to the trace platform. A better approach: route by business value. High-severity security events go to the SIEM at full fidelity. Routine operational logs go to cheap storage. Error traces go to the APM tool. Successful request traces get sampled at 5% and sent to the data lake.
This turns your telemetry pipeline from a dumb pipe into a cost optimization layer. Each event ends up at the destination that matches its operational value, at the price point that matches its query frequency.
Plan for Config Propagation
At scale, the ability to push a new filtering rule to thousands of agents matters. If propagating a config change takes hours, your pipeline can’t respond to incidents (a log source suddenly spiking 10x) or cost events (a new deployment doubling your ingest). Config propagation in under 30 seconds across your entire fleet means you can react in real time.
Expanso’s architecture handles this by treating the processing layer as a distributed control plane. Rules defined centrally propagate to every agent, with its 180+ connectors, in under 30 seconds. The result is a telemetry pipeline that adapts to changing conditions without manual intervention at each endpoint.
FAQ
Q: What is the difference between telemetry data and observability data?
Telemetry data is the raw material. Observability is the capability you build on top of it. Telemetry includes logs, metrics, traces, and events generated by your systems. Observability is the ability to understand system behavior by querying and analyzing that telemetry. You can collect telemetry without achieving observability (if the data is too noisy or poorly structured), and you can’t have observability without telemetry.
Q: What are the most common telemetry data formats?
For logs: JSON, syslog (RFC 5424), and plaintext. For metrics: Prometheus exposition format, StatsD, and OTLP (OpenTelemetry Protocol). For traces: OTLP, Jaeger Thrift, and Zipkin JSON. OpenTelemetry is converging the industry around OTLP as a universal format for all three signal types. Older formats (syslog, StatsD) remain common in legacy infrastructure.
Q: How much telemetry data does a typical enterprise generate?
It depends on the size and architecture. A mid-size company with 500 servers and a microservices architecture typically generates 1-5 TB/day of combined telemetry. Large enterprises with thousands of hosts, Kubernetes clusters, and edge deployments can generate 10-50 TB/day. IoT-heavy environments (manufacturing, telecom, energy) push even higher. A telecom operator monitoring 3,847 sites can produce multiple terabytes per day from infrastructure telemetry alone.
Q: How do I reduce telemetry costs without losing visibility?
Filter noise at the source. Debug logs, health checks, and duplicate events are safe to drop in most environments. Sample high-volume, low-value streams instead of collecting everything. Route data by value: high-priority events to expensive real-time platforms, archives to cheap object storage. This approach routinely delivers 50-70% cost reductions while maintaining full visibility into the data that matters.
Q: What is a telemetry pipeline?
A telemetry pipeline is a data processing layer between your telemetry sources and your analysis platforms. It collects data from agents and receivers, processes it (filtering, enrichment, transformation, masking), and routes it to the right destination. The pipeline gives you control over what data goes where and at what cost. Without one, every byte your infrastructure generates lands directly in your observability platform at full price.
Q: Can telemetry be processed at the edge?
Yes. Edge processing is becoming the standard for distributed environments. A lightweight agent at each source handles filtering, transformation, and routing locally. Only processed data ships to central destinations. This reduces bandwidth, lowers egress costs, enables real-time processing, and keeps sensitive data from traversing the network. Edge processing requires a small-footprint agent. Expanso’s agent is built for that: a single binary whose footprint depends on the pipeline it runs, so it deploys to resource-constrained edge environments.
Bottom Line
Telemetry data is the foundation of every monitoring, observability, and security program. The question isn’t whether to collect it. The question is how to collect, process, and route it without letting volume dictate your costs. The organizations that solve this build a processing layer between their sources and their platforms, filtering noise at the source and routing data by value.
If you want to see how much of your current telemetry is noise and what a processing layer would save, book a data consultation with Expanso. We’ll map your telemetry sources, quantify the waste, and show you the before-and-after math for your environment.
