Performance Benchmarks

400K Events/Second: How Expanso Edge Handles Extreme Data Throughput

We put Expanso Edge through rigorous performance testing across multiple input methods. Here's what we learned about processing high-velocity data streams at the edge.

400K
Max signals/second
12M
Signals per test
35/38
Tests passed

The Challenge: Processing Data at the Edge

Modern edge computing demands more than just running code close to data sources. When you're processing sensor data from manufacturing lines, log streams from distributed systems, or real-time telemetry from IoT devices, throughput matters. A lot.

We designed a comprehensive benchmark suite to test Expanso Edge under real-world conditions. Our goal: understand the performance characteristics of different input methods and identify the limits of what's possible when processing data at the edge.

What we tested: Three distinct input methods (HTTP, stdin pipe, file tailing), each at multiple throughput levels from 50K to 400K signals per second. Each signal is a JSON object that gets timestamped and filtered in real-time.

Key Findings

The results exceeded our expectations. Expanso Edge sustained 400,000 signals per second through stdin pipes with 100% test pass rate. Even the HTTP input method, which has the highest overhead, sustained 250K signals/second.

Stdin Pipe

400K
signals/second
Fastest ingestion method

File Tailing

300K
signals/second
Log ingestion pattern

HTTP Server

250K
signals/second
Full API stack

Input Method Comparison

Each input method has different performance characteristics based on how data enters the processing pipeline:

Input Type Max Rate Use Case Tests Passed
Stdin Pipe 400K/s Process piping, streaming ingestion 16/16
File Tailing 300K/s Log files, batch processing 12/12
HTTP Server 250K/s REST APIs, webhooks 7/10

Throughput by Input Method

250K
100K
HTTP Server
400K
400K
Stdin Pipe
300K
300K
File Tailing
Drop (discard output)
File (write to disk)

Why These Numbers Matter

Processing 400,000 events per second isn't just a vanity metric. Here's what it enables:

Manufacturing Telemetry

Process sensor readings from thousands of machines simultaneously. Real-time anomaly detection before issues reach the cloud.

Log Aggregation

Filter and enrich log streams at the source. Reduce bandwidth by 10-100x by processing before transmission.

IoT Data Pipelines

Handle bursts of sensor data during peak events. Add timestamps, filter noise, and route to appropriate destinations.

Security Monitoring

Process network events in real-time. Detect threats at the edge before they propagate through your infrastructure.

The Pipeline

Each benchmark used a realistic processing pipeline that adds timestamps and filters data: 

input:
  stdin:
    codec: lines

pipeline:
  threads: -1  # Use all available cores
  processors:
    - mapping: |
        root.processed_at = now()
        root.value = this.value
        root.type = this.type
    - mapping: |
        root = if this.value > 5 { this } else { deleted() }

output:
  file:
    path: output.jsonl
    codec: lines

This isn't a synthetic benchmark - it's a real-world pattern. Add metadata, filter based on conditions, write results. The fact that we can do this 400,000 times per second opens up entirely new possibilities for edge data processing.

What We Learned

Key insight: Input method selection matters significantly. When maximum throughput is critical, stdin pipes outperform HTTP by 60%. But HTTP provides better flexibility for distributed architectures where you need to receive data from multiple sources.

The HTTP input method showed an interesting pattern: it handles the "drop" scenario (discarding output) much better than buffered writes. At 150K/s with buffered output, disk I/O becomes the bottleneck. This suggests that for maximum HTTP throughput, pair it with async output sinks or memory-based queues.

File tailing proved remarkably consistent - 100% pass rate across all test configurations. This makes it ideal for log ingestion scenarios where you're processing files as they're written.

Ready to Process Data at the Edge?

Expanso Edge brings high-performance data processing to wherever your data lives. No cloud round-trips. No compromises on throughput.

Learn More About Expanso

Methodology

For complete transparency, here's how we ran these benchmarks:

  • Test duration: 30 seconds per rate level
  • Rates tested: 50K, 100K, 150K, 200K, 250K, 300K, 350K, 400K signals/sec
  • Success criteria: Processing completes within expected time (5% tolerance)
  • Signal format: JSON lines with random values (1-10)
  • Processing: Add timestamp, filter values > 5

All tests were run on standard hardware to represent realistic edge deployment scenarios. The benchmark code is open source and reproducible.