A log line is the most casually created record in any engineering organization. Someone adds a field to help debug a checkout failure on a Tuesday afternoon, and from that moment every system downstream of the application receives it: the observability platform, the security data lake, the analytics warehouse, the vendor doing anomaly detection, and the backup of each. Nobody approved that. Nobody reviewed it. It happened because logging is a one-line change and data governance is a quarterly meeting.
The gap between those two speeds is where most personal data ends up in places it was never meant to go. The fix is not a bigger meeting. It is a transformation that runs before the data leaves, with a result you can read.
Deletion and selection are different strategies
There are two ways to remove a field from a record, and they fail in opposite directions.
Deletion names the fields you do not want. Drop email, drop ip_address, drop payment_card, forward the rest. This is the natural first instinct and it works well when you need to preserve a large, stable schema and remove a handful of known offenders. Its weakness is that it is a blocklist, and a blocklist is only as current as the last person who remembered to update it. The day an application starts emitting customer_phone, that field flows straight through, because nothing in the pipeline was ever told to stop it. You find out in an audit.
Selection names the fields you keep. Construct a new record containing a timestamp, an operation, and a duration, and whatever else arrives is simply not part of the output. New fields stay out by default. The cost is real and worth stating plainly: a genuinely useful new field also stays out until somebody adds it deliberately, so selection trades silent leakage for silent omission. That is a trade most teams should take for a narrow export, because a missing column announces itself the moment an analyst goes looking for it, while a leaked column does not announce itself at all.
The UK Information Commissioner’s Office guidance on data minimisation frames the same idea in regulatory language: personal data should be adequate, relevant, and limited to what is necessary. Selection is what that principle looks like when it is expressed as code rather than as policy.
Start from the question the export has to answer
Before choosing an approach, decide what the destination is for. This sounds obvious and it is routinely skipped, because the default is to forward everything and decide later.
A latency report answers “which operations are getting slower?” That needs a timestamp, an operation name, and a duration. It does not need to know who the user was. A fraud investigation answers a different question, needs different fields, and probably belongs in a different destination with different access controls and a different retention period. A security audit trail needs identity precisely because identity is the point.
Once you write the question down, the field list follows from it, and you have a defensible reason for every column that survives. That reasoning is the artefact worth keeping, more than the config itself. When someone asks in eight months why user_email is not in the performance dataset, the answer is not “we forgot”, it is “performance questions do not require it”.
What a small, checkable example looks like
Here is a synthetic event of the kind an application might emit during a checkout:
{
"event_id": "demo-001",
"email": "[email protected]",
"ip_address": "192.0.2.10",
"payment_card": "SYNTHETIC-CARD",
"action": "checkout"
}The transformation that selects rather than deletes is four lines:
- mapping: |
root.event_id = this.event_id
root.action = this.actionAnd the output, verified by running it:
{ "action": "checkout", "event_id": "demo-001" }Three fields are gone, and they are gone because they were never named, not because anyone remembered to exclude them. Add a customer_phone field to the input and run it again; the output is unchanged. That is the behavior worth confirming with your own eyes before you trust it with a real stream.
We published this as a runnable example with the input, the configuration, and the expected result together. It runs locally against synthetic data, needs no account, and takes about three minutes. The value is not the configuration, which you could have written yourself. It is that you can change one field and watch the boundary move.
Test what disappears, not only what survives
Most pipeline testing checks that the good case produces the expected record. That is half the job. The interesting cases are the ones where the input is not what you assumed.
Run a normal event, an event missing an optional field, and an event carrying an unexpected personal field. Then run malformed input as a separate exercise, because error handling is its own design problem and mixing the two hides both. For each, compare the actual output against the record you intended to create, not against a vague sense that it looked fine.
One caveat deserves more attention than it usually gets: neither selection nor deletion finds personal data hidden inside a field you kept. A free-text message column is the classic offender, because a developer debugging a support ticket will paste a customer’s email into an error string without thinking about where that string is going. Structural field control does nothing about this. It needs content inspection, which is a different tool and a different conversation.
Hashing is a separate decision, not a shortcut
Teams often reach for hashing as a way to keep an identifier while claiming it is no longer personal data. Be careful here, because the claim is frequently wrong.
A stable hash of an email address still links every event about that person to every other event about that person, which is exactly the capability that makes an identifier an identifier. And the input space for most identifiers is small enough to enumerate: there are only so many phone numbers, and a hash of one is recovered in seconds on commodity hardware. The NIST guidance on de-identification treats this as a design problem with a threat model, not a transformation you apply and declare finished.
Salted hashing, pseudonymisation with a key you control, and generalisation each have a place. None of them substitutes for not collecting a field you do not need.
Where the transformation runs matters as much as what it does
A pipeline that removes personal data after the data has crossed a network boundary has already lost the argument, because the copy on the other side of that boundary exists. Removing fields is most useful when it happens at the point the data is produced, before anything is forwarded, replicated, or cached.
That is the architecture Expanso is built around: processing that runs next to the data source, so the record that leaves is already the record you intended to send. Our data governance use case covers the broader pattern, and the security and governance page explains what the control plane does and does not see.
Before connecting anything real, check the paths that are easy to forget. Error outputs frequently contain the full original record. Retry queues hold it. Debug logging at the wrong level prints it. A successful transformed record on the happy path proves nothing about any of those.
Start with something you can read
The useful first milestone is small and concrete. You supplied a known record, a pipeline transformed it, and you can explain why each field survived or disappeared. That is a much better foundation for a governance conversation than a policy document nobody has tested.
Run the PII removal example, change a field, and see what the boundary does.
