Cluster architecture

The pipeline is designed to run on a single small Kubernetes node, with everything it needs inside the cluster: its own Git server, its own image registry, its own build runner and its own deployer. This page describes that design.

One node, two namespaces, seventeen pods#

The whole platform is sized to fit on one machine with 4 GiB of memory, running k3s with containerd.

Under 1 GiB at rest leaves room for the operating system, the database’s storage and the occasional build.

An earlier draft gave every worker its own namespace, eleven in all. Two keeps the network rules simple and the manifests few, which matters to one person maintaining the stack.

The control plane#

Plainforge, Lazarette, Keelworks and Shipkeeper are written from scratch as small Go programs instead of being assembled from large existing projects. Clean room and credit explains why.

From a push to a running pod#

Every change follows the same path, and no step uses Docker.

1. PlainforgeA commit landsOn main or as a tag. Plainforge mirrors the code to GitHub and tells Keelworks.
2. KeelworksA build job startsA temporary, unprivileged job is created for this one build.
3. Build jobTest, build, signThe repository’s tests run first. Then the image is built, signed and pushed to Lazarette.
4. KeelworksRecord what was builtThe image’s digest is written into the app’s manifest in Plainforge.
5. ShipkeeperDeployShipkeeper sees the new digest and rolls the pods forward one at a time.

Why no Docker. A Docker daemon runs with root-level privileges and uses memory a small node cannot spare. Building images inside a cluster with Docker usually means giving the build job control of the host. k3s already runs containers with containerd, so all the cluster needs is a standard image, and there are builders that produce one without a daemon:

Tests come first. A build job runs the repository’s tests, including the saved examples for every sub-tool, before it builds anything. Code that fails never reaches the registry.

Images are signed. Each image’s digest is signed with cosign before it is stored.

Code on GitHub, images inside#

Plainforge is the primary copy of every repository. It sits inside the cluster and cannot be reached from the internet. Everything that writes code talks to it first: an approved edit, a task for a missing rule, a new image digest. On each change it pushes the code outward to GitHub, so that anyone can read, check or fork it.

GitHub is a mirror of the code only. Built images never leave the cluster.

Keeping images inside means no registry passwords in the worker pods, and a rollback that works even when the outside network does not.

Install, upgrade, pause, roll back#

Shipkeeper reads each app’s manifest from Plainforge and applies it to the cluster with Kubernetes’ own server-side apply. It adds no custom resource types. For the owner it offers four actions on any service or worker:

• Install applies the app’s manifests for the first time.
• Upgrade moves it to the newest signed image.
• Pause scales it down without deleting its configuration or data.
• Rollback returns it to one of the five previous images kept in Lazarette.

It also watches the health of every pod and reports it, after removing anything that would reveal the layout of the cluster. See what stays private.

One database and one gatekeeper#

All state lives in a single Postgres database: the items, the queue, the decisions and the edit proposals. Keeping the queue and the state in one place means a change of state and the next step in the queue happen together or not at all. An item is never marked done in one system while still waiting in another.

The data is split in two. core holds the full working records. pub holds views of them with only the columns meant to be public.

Workers ask Muster for work over the cluster’s private network and identify themselves with their Kubernetes service account. A worker that reads untrusted public pages therefore has no database credentials to leak. Only Muster knows the layout of the tables, so a change to a table does not ripple through ten repositories.

Nothing finished is lost. Each worker has a small disk of its own. When it finishes a step it writes the result there first, then reports it. If Muster is restarting, the result waits on the worker’s disk and is sent again until it is acknowledged. Unfinished work is covered the other way round: the worker’s lease on the item runs out and the item is offered again.

Network, priority and privilege#

Closed unless opened. Both namespaces block all traffic by default. Only Conn accepts connections from outside. Workers may call Muster, and the workers that read public sources may fetch them from outside. Workers cannot reach the database, Plainforge or Lazarette.

Who keeps running under pressure.

No root. Every pod runs as an ordinary user, with all Linux capabilities dropped, no way to gain privileges and a read-only filesystem wherever nothing needs to be written.

Starting from nothing#

The design calls for the cluster to set itself up from one manifest applied to a fresh k3s node. A setup job that runs once then:

Because every repository is MIT licensed and written from scratch, the design can be forked without asking: rebranded, given different scoring weights, or run privately with the GitHub mirror turned off.

Running it with no internet connection#

Nothing in the design depends on an outside service while it runs, so the whole cluster can sit inside a closed network. Four things change.

What crosses the boundary is a bundle of signed image archives, the manifests and the repositories, with a checksum to prove it arrived intact. Working on disconnected networks covers that discipline.