Smart sampling: Jev scores what is interesting, a counter samples the rest.

SourcePOST /events

Expanso nodeone pipeline, 6 steps, deterministic

1. Receive
2. Shape
3. Ask Jev
4. Gate
5. Sample
6. Route

Jev judgment

• interesting

• keep
• sampled

1. Expanso, deterministic

   Same fixed rules every time, no model involved.

2. Jev, judgment

   The one step a rule cannot write.

3. Receivelines 11 to 16

   An HTTP server input accepts events on POST /events.

4. Shapelines 17 to 26

   Parses the POST body if it arrived as a string, then stamps received_at. It applies the same fixed rules every time, with no model involved; the timestamp is the one value that differs.

5. record + questionsanswers + confidence
6. Ask Jevlines 27 to 68

   Expanso packs the whole record and the typed questions into one request. Jev answers. If the call fails, a catch substitutes empty answers marked jev-unavailable, and the pipeline keeps going.

   • interestingscore
7. Gatelines 69 to 78

   One fixed threshold over Jev’s score marks each event keep or sampled.

8. Samplelines 79 to 82

   Keeps every keep event, and of the rest exactly one in every hundred, using a counter rather than a random draw. The other ninety-nine are deleted.

9. Routelines 83 to 100

   A switch output writes kept and sampled events to separate files.

name: jev-smart-sampling
type: pipeline
description: Smart sampling with Jev — every event is scored for interestingness. Keep 100% of the interesting ones, 1% of the noise. Volume reduction depends on your signal-to-noise ratio.
namespace: production
labels:
  category: log-processing
  pattern: ai-decision
  model: jev

config:
  input:
    http_server:
      address: "0.0.0.0:8080"
      path: /events
      allowed_verbs: ["POST"]

  pipeline:
    processors:
      - mapping: |
          # http_server already parses JSON bodies; only parse raw strings
          root = if this.type() == "string" { this.parse_json() } else { this }

      - mapping: |
          root = this
          root.received_at = now()

      # ── Ask Jev: how interesting is this event? ──
      - mutation: |
          meta jev_start = timestamp_unix_milli()

      - branch:
          request_map: |
            root = {
              "state": this.string(),
              "model": "jev-latest",
              "questions": {
                "interesting": {
                  "type": "score",
                  "instructions": "How interesting or unusual is this event for an engineer investigating an incident?",
                  "criteria": [
                    "Routine noise, safe to drop",
                    "Mildly interesting context",
                    "Notable, worth keeping",
                    "Highly unusual, keep for investigation",
                    "Critical signal, never drop"
                  ]
                }
              }
            }
          processors:
            - http:
                url: "${JEV_API_URL:https://api.typesafe.ai/v1/systemone}"
                verb: POST
                headers:
                  Content-Type: application/json
                  Authorization: "Bearer ${TYPESAFE_API_KEY}"
                timeout: 2s
                retries: 1
            - catch:
              - mapping: |
                  root = {"answers": {}, "model": "jev-unavailable"}
          result_map: |
            root.jev = {
              "answers": this.answers,
              "model": this.model.or("jev-latest"),
              "ms": timestamp_unix_milli() - metadata("jev_start")
            }

      # ── Judgment-based sampling: 100% of interesting, 1% of noise ──
      # (annotate first, drop in a separate mapping: assigning fields after
      # root = deleted() aborts the mapping, and `this` is frozen at mapping
      # entry so re-assigning it would wipe the annotations above)
      - mapping: |
          let score = this.jev.answers.interesting.score.or(0)
          root = this
          root.jev_score = $score
          root.jev_decision = if $score >= 3 { "keep" } else { "sampled" }

      # ── Deterministic 1% sample of low-interest events via counter ──
      - mapping: |
          root = if this.jev_score >= 3 || count("sampled") % 100 == 0 { this } else { deleted() }

  output:
    broker:
      pattern: fan_out
      outputs:
        - stdout:
            codec: lines
        - switch:
            cases:
              - check: this.jev_decision == "keep"
                output:
                  file:
                    path: ./data/jev-smart-sampling/kept-${! now().ts_format("2006-01-02") }.jsonl
                    codec: lines
              - check: "true"
                output:
                  file:
                    path: ./data/jev-smart-sampling/sampled-${! now().ts_format("2006-01-02") }.jsonl
                    codec: lines

100 lines. Copy and Download both give you the file byte for byte.