Every logging bill eventually produces the same meeting. Someone shows a chart, someone says the word “ingest”, and the team agrees to filter out the noise. A rule gets written that afternoon, ships that week, and the chart goes down. Everyone moves on.
The problem surfaces months later during an incident, when the one log line that would have explained the failure turns out to have been dropped by a filter nobody has looked at since. The rule worked exactly as written. It was simply written against an assumption that stopped being true.
Filtering logs is worth doing. It is also a change that quietly stands between you and every future investigation, which makes it worth more care than a one-line config usually receives.
The filter is a policy decision wearing a config file
The instinct is to treat severity as a proxy for importance: keep errors and warnings, drop everything at INFO and below. It is a reasonable default and it is wrong often enough to matter.
An INFO event can record a deployment, a configuration change, a permission grant, a completed payment, or a user’s consent choice. None of those are errors. Several of them are the first thing an auditor asks for, and at least one may be the only record that a business transaction happened at all. Severity describes how alarmed the developer was when they wrote the line. It does not describe how useful the record is to someone else, later, asking a different question.
So the rule is not “which severities do we keep”. It is “which events does this specific destination need”, and severity is one signal that helps answer it. A service-health dashboard, an incident archive, and a compliance store have genuinely different requirements, and the same source stream can legitimately feed all three with different rules. Dropping an INFO record from a performance export is not a claim that it has no value anywhere. The OpenTelemetry specification for severity is deliberate about this: it defines a numeric range precisely because the mapping from severity to action is left to the consumer.
Write down the expected output before you write the rule
The discipline that prevents most filtering accidents costs about five minutes.
Take three or four representative records. Write down, by hand, which ones should survive and which should not. Then write the rule and check it against that list. If the rule and your written expectation disagree, you have learned something before it reached production rather than after.
A minimal fixture might be an informational health check, a warning about a slow query, and an error with a stack trace. A rule that keeps warnings and errors should produce exactly two records, with their context intact. Confirming that takes seconds and it establishes the baseline everything else is measured against.
The cases that actually break filters
Once the happy path passes, the interesting work starts. Real log streams are messier than fixtures, and a severity filter meets all of the following within a week of shipping:
Severity arrives lowercase, because one service writes error and another writes ERROR, and a string comparison treats them as different values. Severity is missing entirely, because a library logs without setting the field. Severity carries an unexpected value like CRITICAL or FATAL or 35, which the rule was never told about. The record is not valid JSON at all, because something upstream truncated it mid-write.
For each of those, decide deliberately whether the record should be kept, dropped, or sent somewhere else for inspection. The dangerous default is a rule that silently drops anything it does not recognize, because that turns a parser failure into a quieter stream, which looks exactly like success. A filter that cannot distinguish “no errors occurred” from “errors stopped being parseable” is a filter that will eventually lie to you during an outage.
Routing unrecognized records to a separate low-volume destination is usually the right answer. It costs almost nothing and it converts an invisible failure into a visible one.
A runnable example with a known result
We put together a small pipeline that does exactly this, with the input and the expected output published alongside the configuration.
The fixture is three records at different severities. The selection rule keeps the error and discards the rest, and the verified local output is a single record. That is a deliberately unimpressive result, and it is the point: with three records and one surviving line, you can hold the entire behavior of the rule in your head and confirm it by reading.
The example runs locally against synthetic data, needs no account, and takes a few minutes. Once it works, change the severity on one record and watch the output change. That is the loop worth having before you point a rule at a real stream.
Measure the reduction honestly
When the rule goes live, count input and output records over the same window. Then count bytes as well, because the two numbers tell different stories. A stack trace and a health-check ping each count as one record and differ by three orders of magnitude in size. A filter that drops ninety per cent of records may reduce volume by a fraction of that, or by considerably more, depending entirely on which records it dropped.
And be careful about the claim that follows. A three-record fixture that keeps one record demonstrates how the rule handled those three records. It does not establish a production reduction percentage, and it certainly does not establish a saving on your bill, which depends on your actual traffic mix, your destination’s pricing model, and whatever processing and storage sits behind it. Our benchmarks page publishes measured figures with their conditions stated, which is the standard any number in this area should meet.
For a real rollout, run the candidate rule alongside the existing stream before replacing it. Compare what an investigation would have had access to under each. That comparison is more informative than any percentage.
Filter where the data is produced
There is a structural point underneath all of this. Filtering after data has been shipped to a central platform reduces what you store and query, but you have already paid to move it, and the vendor has already counted it.
Applying the rule at the source changes the economics, because the record that never leaves costs nothing to transmit, index, or retain. It also changes the governance picture, since data that stays put cannot leak from somewhere it was copied to. That is the architecture we build around, and the cost optimisation use case goes into the specifics of where the savings actually come from.
Start small enough to verify
Begin with a rule you can hold in your head and a fixture you can read. Confirm what survives, confirm what does not, decide explicitly what happens to the records that fit neither category, and only then connect it to something that matters.
Open the log filtering example and start from a result you can check.
