The pipeline and its ten workers

The pipeline is ten small Go programs, from Trawl at stage 01 to Bellbuoy at stage 10, each in its own MIT-licensed repository and each with one job. Together they take an idea for a tool from first sighting to a listing in the catalog, and keep watching it afterward. This page describes each worker and the rules they share. Each name in the table links to that worker’s full specification, and the naming registry lists what each one used to be called.

The ten workers#

Finding and recording#

Trawl collects candidates from four places: public notices on SAM.gov, issues opened on GitHub, scheduled sweeps of public government and vendor sites, and the form on this site. Its job is to make sure each one is recorded exactly once.

Cleaning the address. The same tool is often reachable at several addresses. Before anything is saved, the address is put through fixed rules: the scheme and host are made lowercase, tracking and session parameters are removed along with any fragment, and trailing slashes and default file names are tidied so that equivalent addresses become one. Addresses that are not public, that point inside a private network or that need a login are refused.

Checking for a duplicate. A SHA-256 hash of the cleaned address is compared with the ones already recorded. If it matches, the sighting is noted against the existing record. If it is new, the candidate is given a permanent identifier of the form PA-0001 and passed on.

What intake never does. It does not create accounts on other sites, ask for sales demonstrations, get around a paywall or a CAPTCHA, ignore robots.txt, or accept controlled information or real bid figures.

Observing#

Sextant reads what a candidate says about itself in public and turns that into a structured record: the job it does, its inputs and outputs, whether it needs a login, whether it tracks its users, its license, its price model, the public rule it cites and the date it was looked at.

A small program first, a browser only when needed. Most public pages and documents can be read with a plain web request, by a program that needs about 30 MiB of memory. Only when a page is built entirely by scripts does the worker start a temporary browser to render it, and that browser is discarded as soon as the page has been read.

Public pages only. The worker reads what anyone can see. It does not sign up, start a trial, fill in a contact form, enter real data or get past a login. Where a page says something the worker cannot see for itself, the fact is recorded as Stated instead of Seen. Anything else is Unknown.

The raw page stays behind. The worker keeps the page it fetched only long enough to extract the facts. The copy is sealed and no later worker can read it. Only the structured record moves on. Clean room and credit explains why this matters.

Screening, scoring and deciding#

Assay applies the five knockout gates and, for candidates that pass, the 100-point rubric and the tie-breakers. It produces a recommendation. Countersign holds the queue in which the owner makes the decision. Both are described in How we decide what to build.

Crediting and specifying#

Once the owner approves a build, Waybill writes two things: the note that credits where the idea came from, and the specification the tool will be built from. The specification contains the official rule, the inputs and outputs, and worked examples. It contains nothing from the product that was observed. See Clean room and credit.

Building, verifying and listing#

Building. Moldloft reads the specification and generates the tool’s own repository from one of four templates.

Every template produces plain files that run in the browser, with no server behind them and nothing loaded from elsewhere.

Verifying. Shakedown runs five checks. A tool is not listed unless all five pass.

1. It works offline. The tool loads once and keeps working with the network disconnected. Every table and date it needs ships with it.
2. It cannot send anything. Every page carries a content security policy that includes connect-src 'self' and form-action 'none', so the browser itself refuses any outside request.
3. It tracks nothing. No sign-in, analytics, cookies or counters. If the tool keeps work in the browser, it says so and offers a button to clear it.
4. Its answers match the worked examples. The tool is run against examples calculated by hand, including the awkward cases: zero, an empty file, a date that crosses a boundary. All test data is made up.
5. It shows its sources and its limits. Each result cites its regulation, paragraph and revision date. The tool says what it does not cover and that it is not professional advice, and carries an MIT license.

Listing. Chartroom adds the verified tool to the catalog. It also handles the other outcomes: linking to an existing tool, merging an idea into another entry, and retiring a tool whose rule has been repealed.

Watching and routing#

Tidegauge keeps the catalog accurate after a tool is published. On a schedule, it asks each official source whether anything has changed, using the cheapest check that works: a header that says whether a file is newer, a structured feed, or a comparison of a page’s structure.

The monitor does not change a tool itself. It reopens the item so that the pipeline handles it: a live tool whose rule changed goes back to be rebuilt, or is retired if the rule was repealed. A parked idea is looked at again when the thing blocking it clears. A linked tool that breaks or starts tracking its users goes back to be reconsidered.

Bellbuoy turns failures, paused items and approved edits into work for the right repository. Its main job is to avoid noise. If two hundred items fail on the same unfamiliar table, one task is opened, not two hundred: the worker builds a fingerprint from the worker’s name, the error code and the pattern of the failure, and adds to the count on an existing task when the fingerprint matches.

Why no worker uses AI#

Many pipelines reach for an AI model when a page is messy. This one does not, anywhere.

AI is used only while the rules are being written, under review. How AI is used has the detail.

Sub-tools: one rule for each case#

Real pages do not look alike. One agency publishes a neat table and another buries the same figures in a brochure. Each variation is handled by a small, separate rule called a sub-tool, kept in the repository of the worker that needs it, under tools/<case>/. A sub-tool has three parts:

• A matching rule that says which pages or files it applies to.
• The logic that extracts, scores or builds, written as plain code.
• Fixtures: saved examples of the input together with the exact output expected, so that a test fails if the behaviour ever changes.

When no sub-tool matches, the worker stops work on that one item and marks it needs_subtool. Everything else in the queue carries on.

1. 1Pause the itemThe item is marked needs_subtool, with the error code E_NEEDS_SUBTOOL.
2. 2Open one taskBellbuoy opens a task in the worker’s repository describing the pattern, or adds to an existing one.
3. 3Write the ruleA sub-tool and its fixtures are added and reviewed.
4. 4Build and deployThe worker’s tests run, including the new fixtures, and the new version is rolled out.
5. 5Run the paused items againEvery item that was waiting on that pattern is replayed automatically.

Each awkward case is solved once and tested from then on.

How the workers stay in step#

Workers do not talk to each other, and they do not touch the database. Each one asks a single service, Muster, for its next item, and reports the result back to it.

• Leases. A worker claims an item for a limited time. If the worker stops, the lease runs out and the item is offered again.
• One shared contract. The hub repository owns the definition of the handful of calls a worker makes (lease, heartbeat, complete, fail, needs_subtool), the fourteen item states and the error codes. There is no separate shared-code repository for the ten workers to depend on.
• Versions. Each worker states which version of the contract it speaks, and the hub checks every message against it. That lets one worker change and ship without the other nine being redeployed.
• Three error codes. E_TIMEOUT means something outside took too long. E_PARSE means the data was broken. E_NEEDS_SUBTOOL means the data was valid and unfamiliar.
• The wall is checked here too. The hub rejects any message to Moldloft that carries a raw page captured by Sextant.

Cluster architecture describes the queue service and the database behind it.