Sextant specification

Design

Technical specification for the deterministic Sextant worker that inspects public pages, spawns on-demand browser Jobs, and seals raw captures behind the clean-room wall.

J.4.94 min readLast revised November 2026

Status: Design. A specification. It describes how the component is meant to work, not something you can use on this site today. What the labels mean.

Registry entry · Stage 02

Reads a candidate's public pages and writes down what job it does, copying nothing.

Formerly: cs-02-observe, then Spyglass
Namespace: cs-workers
Repository: sextant
Container image: lazarette.clearedstack.svc.cluster.local:5000/sextant
Depends on: Muster
Depended on by: Nothing else in the platform
Boundaries: Leases work from Muster over cluster-private HTTP
Fetches public pages over HTTPS, and starts a temporary browser job when a page needs one

From the naming registry, which is the one place this component’s name is written down.

On this page

Overview & Role
Runtime & Resource Budget
Interfaces & State Transitions
Sub-Tools & Failure Modes
Security & Clean-Room Boundaries
Related pages

Overview & Role#

Sextant (v0.1.0-draft) is the deterministic observation worker in the ClearedStack v3.1 pipeline, replacing the original Stage 3 (Observe). Implemented as a Go service in the Sextant repository under the MIT License, it inspects public web pages, .gov portals, and agency PDFs using repeatable extractors and zero runtime AI.

Its core responsibilities are:

Public-only inspection: Fetch open-web vendor marketing pages, government portals, and procurement PDFs without logging in, bypassing paywalls, or submitting forms.
Deterministic fact extraction: Produce strict, schema-validated JSON capturing a candidate tool's core job, inputs, outputs, login requirement, tracking profile, license or copyright, price model, cited public rule, observation date, and confidence tier (Seen, Stated, or Unknown).
Clean-room sealing: Seal all raw HTML, DOM snapshots, screenshots, and downloaded PDFs behind the clean-room wall so downstream specification and build workers (Waybill and Moldloft) never see vendor markup, stylesheets, or client-side scripts.

Runtime & Resource Budget#

Sextant runs in the cs-workers Kubernetes namespace under the Standard (Workers) priority class, with ephemeral headless browser Job pods scheduled under Low (Ephemeral Jobs).

Attribute	Value
Worker ID / Repo	Sextant (`sextant`)
Namespace	`cs-workers`
Kind / Version	`worker` · `v0.1.0-draft`
Baseline RAM Budget	`30 MiB` (always-on Go HTTP pod)
Ephemeral Footprint	On-demand headless browser `Job` pods (created and deleted per JS-heavy target)
Runtime AI Calls	`0` (100% deterministic Go + YAML rules)
Local Spool	`/var/spool/cs` (persists completed extraction payloads across Muster restarts)
Build & Deploy	Built with `ko` (no Docker) by Keelworks, signed with `cosign` in Lazarette, deployed by Shipkeeper via Server-Side Apply

Keeping headless browsers out of the always-on worker pool preserves the ~300 MiB baseline worker footprint so the full 17-pod ClearedStack platform fits on a single 4 GiB k3s node.

Interfaces & State Transitions#

Sextant communicates exclusively with Muster in the clearedstack namespace over cluster-private HTTP authenticated via its Kubernetes ServiceAccount (CONTRACT_VERSION=v1). It never connects directly to Logbook, Plainforge, or Lazarette.

Queue consumption: Polls Muster using lease, heartbeat, complete, fail, and needs_subtool, draining priority lanes in strict order: urgent → normal → backfill.
Inbound states:
- recorded — Newly deduplicated PA-XXXX candidates from Trawl.
- observed (re-queued) — Items returned from needs_subtool replay after a sub-tool fix ships, parked items whose blocker cleared in Tidegauge, or linked items flagged for drift.
Outbound transitions:
- observed — Extraction succeeded; sanitized structured JSON facts are stored in core.* and raw captures remain sealed behind the clean-room wall.
- needs_subtool — Fail-closed pause (E_NEEDS_SUBTOOL) when no registered sub-tool matches the target layout.
Spool durability: Extracted payloads are written to /var/spool/cs before acknowledging complete to Muster, preventing lost work during control-plane restarts.

Sub-Tools & Failure Modes#

Per-site and per-format variations live inside the Sextant repository under tools/<case>/, where each sub-tool pairs a match.yaml rule, deterministic extract.go logic, and golden fixtures/.

Sub-Tool	Target Pattern & Behavior
`saas-marketing`	Public vendor landing pages and feature tables; extracts stated inputs, outputs, pricing model, and login gates.
`gov-portal`	Federal `.gov` and `.mil` portals and calculators; captures cited FAR, DFARS, or NIST clauses and offline behavior.
`agency-pdf`	Downloadable procurement PDFs, worksheets, and instructions; extracts form fields, rules, and publication dates.
`on-demand-browser-job`	Ephemeral headless browser `Job` spawned only for JavaScript-heavy single-page apps to render the DOM and inspect client-side network calls using synthetic test inputs.
`<vendor-slug>`	Dedicated extractors for specific vendors whose public page structure differs from standard layouts.

Sanitized Failure Modes

Sextant never guesses when an extraction fails or encounters an unknown layout:

Error Code	Trigger	Platform Behavior
`E_NEEDS_SUBTOOL`	Page layout, table, or PDF format matches no registered `tools/<case>/` sub-tool.	Fails closed to `needs_subtool`. Bellbuoy opens a deduplicated task in Sextant on Plainforge; once a new sub-tool and fixture deploy, Muster auto-replays paused items.
`E_TIMEOUT`	Upstream public page, PDF download, or ephemeral browser `Job` exceeds deadline.	Item remains safely queued for bounded retry; Shipkeeper strips cluster internals before surfacing status.
`E_PARSE`	Malformed HTML, corrupted PDF, or schema validation failure on extracted JSON.	Halts item cleanly and reports fingerprinted error via Bellbuoy without emitting partial data.

Security & Clean-Room Boundaries#

Passive public boundaries: Sextant never creates accounts, starts free trials, submits lead-capture forms, enters real contracting data, or circumvents paywalls, robots.txt, or CAPTCHAs. If a tool sits behind a login or anti-bot gate, observation stops at the public surface and records vendor claims as Stated rather than Seen.
Strict clean-room wall: Raw HTML, DOM trees, stylesheets, client-side scripts, and binary PDFs are sealed at Sextant and never cross into Waybill or Moldloft. Muster enforces this contract boundary by rejecting any build payload containing raw observation artifacts.
Pod hardening & network policy: Runs as an unprivileged non-root user with dropped Linux capabilities, no privilege escalation, and a read-only root filesystem (writable only at /var/spool/cs). Default-deny Kubernetes NetworkPolicy allows outbound HTTP/HTTPS for public fetches and cluster-private HTTP to Muster, while blocking direct access to Logbook, Plainforge, and Lazarette.

Found a mistake, or something that has gone out of date? Open an issue or write to [email protected]. This page is a text file you can read and correct: Source of this page.