This post was written by Claude Fable. It did all the work, it seemed only fair. Edits for clarity by me.
I run a little platform called Bespoke. It hosts the one-off apps I write for myself: a mail client, a calendar, a journal, a wiki, a queue for my 3D printer. Each one is a Go binary behind a subdomain on my tailnet, and each one exists because I wanted something that no product would ever build for an audience of one.
Early on I split it in two. The framework is public. My apps are private. That boundary is the whole reason the thing is pleasant to work in: I never think about whether a half-finished journal app is embarrassing, because nobody can see it.
Then I wanted to give a friend my GitHub tracker, and discovered I’d built a wall with no door.
The hook was already there
Every app in an instance is a directory with an app.toml:
name = "GH Tracker"
slug = "gh-tracker"
port = 4107
icon = "git-pull-request"
description = "Open PRs and issues across your GitHub projects"The manifests are the registry. There’s no database of installed apps, no
registration step. dev, deploy, the generated systemd units, the Caddy
routes, the Litestream config — all of it comes from globbing apps/*/app.toml.
This is the single best decision in the codebase and I can’t take credit for it
being deliberate; I just got tired of keeping two lists in sync.
The manifest has one optional field I’d added days earlier for a different reason:
package = "github.com/bketelsen/bespoke/apps/builder"It tells the build to compile that Go package instead of ./apps/<slug>. I
wrote it so the framework could ship an opt-in app without every instance
vendoring its source. And nothing in the build path cares whose module it
points at.
So the door existed. Sharing an app is: extract it to its own Go module, go get it, write an app.toml naming the package. Three steps, no new
infrastructure.
The registry already existed too
The thing I kept almost building was a registry. Accounts, uploads, versions, integrity. Then I noticed I’d have been rebuilding the Go module proxy, badly.
proxy.golang.org gives immutable versions. sum.golang.org gives integrity —
a transparency log that makes a version’s content tamper-evident. GOPRIVATE
covers the apps I don’t share. And minimum version selection negotiates the
platform version between my instance and someone else’s app.
That last one has a sharp edge worth naming: installing an app can upgrade your framework. If an app requires a newer Bespoke than you’re running, MVS quietly raises you to it. That’s not wrong, but it means “install this small app” is a bigger action than it looks, and the installer should say so out loud.
What actually broke
The apps compiled. They ran. They rendered completely unstyled.
Bespoke compiles one stylesheet per instance, so that my theme css and every app’s markup go through a single Tailwind pass. Tailwind only emits utilities it can see, and it sees them by scanning source files:
@import "/path/to/bespoke@v0.4.0/design/base.css";
@import "/home/bjk/projects/bespoke-home/design/theme.css";
@source "/home/bjk/projects/bespoke-home/apps/**/*.templ";
@source "/home/bjk/projects/bespoke-home/apps/**/*_templ.go";An installed app’s templates live in the Go module cache, which is in neither tree. So every class it used got pruned, and the failure was completely silent — no warning, no error, just a page with no CSS.
The fix is to derive the scan roots from the registry instead of hardcoding
them: for every app with a package, resolve its module directory and add it.
cmd := exec.Command("go", "list", "-f",
"{{with .Module}}{{.Dir}}{{else}}{{.Dir}}{{end}}", pkg)The module directory rather than the package directory, because a shared app’s templates may sit above its main package, and scanning a little extra costs nothing while missing a template costs the app its styling.
I made the unresolvable case a hard error. A build tool that half-works and tells you nothing is worse than one that stops.
To prove the fix I added a single distinctive class — tracking-widest, used
nowhere else in my instance — to the extracted app, then compiled the stylesheet
with the old CLI and the new one. Absent, then present. That canary mattered
more than it sounds: every other class the app used was also used by some
other app, so a naive “does it look right?” check would have passed against a
completely broken build.
go get -tool, and why a plain require won’t do
Wiring the extracted app into my instance, I hit something I didn’t expect. I
added the module, everything worked, and then go mod tidy silently removed it.
Of course it did. Nothing in my instance imports the app. It’s a main
package, which Go forbids importing, so from the module graph’s perspective the
dependency is garbage.
Go 1.24’s tool directive is the answer, and it’s a better fit than it first
appears:
tool (
github.com/bketelsen/bespoke-app-gh-tracker
github.com/bketelsen/bespoke/cmd/bespoke
)A tool directive means “this module provides an executable I use.” An installed
app is literally an executable my instance builds and runs. It survives
tidy, it pins a version, and it’s the same mechanism already pinning the CLI.
This is the sort of thing you only find by running the whole loop. The failure mode — works today, unpinned tomorrow, mystery build failure next week — is exactly the kind that never shows up in a demo.
A phone book, not a registry
For discovery I made bespoke-apps: one TOML file mapping short names to module
paths.
[[app]]
module = "github.com/bketelsen/bespoke-app-journal"
name = "Journal"
description = "One stream for every journaling moment"
author = "bketelsen"
source = "https://github.com/bketelsen/bespoke-app-journal"bespoke search lists it. bespoke add journal resolves the name, pins the
module, reads an app.toml.example the app publishes, picks a free port, writes
the manifest, and recompiles the stylesheet. BESPOKE_INDEX points at any other
list, and a full module path installs an app that was never indexed at all.
CI checks two things: entries are well-formed, and modules resolve. That’s the entire review. Nobody reads the code, nobody runs it, nobody rechecks it later. The README says so in those words, because an index that implies more curation than it performs is worse than no index.
The deployment methodology
Worth a detour, because the sharing work leaned on it hard.
Everything derived is generated into dist/gen/ from the manifests, and every
generated file carries a # GENERATED by bespoke — do not edit header. systemd
units, the Caddy route file, the Litestream config. Adding an app changes one
app.toml; the routes, unit, and backup entry follow automatically.
Deploy is deliberately boring:
- Regenerate artifacts, cross-compile every binary
CGO_ENABLED=0 GOOS=linux rsyncbinaries intobin.new/, plus manifests, units, and the stylesheet- Wait for the LLM gateway to go idle, so a restart never lands mid-completion
- Per app: swap the binary, keep the old one as
.prev, restart, poll/healthzfor ten seconds, and roll back to.previf it doesn’t come up - Reconcile: retired apps get stopped and their units removed, databases deliberately preserved
The health gate has saved me more than once, and it’s about fifteen lines of
shell. Every process is a systemd user unit. No root anywhere in the normal
path — which I didn’t fully appreciate until I tried to install Litestream to
/usr/local/bin and discovered my app host has no passwordless sudo. It went to
~/.local/bin instead, next to the LLM CLI, and the runbook lost a step.
The other decision that shows up everywhere: generated files are never hand-edited, and the manifest is the only input. A lot of the “just works” feeling is really just refusing to keep a second source of truth.
The part I’d been avoiding
Here’s the thing about making apps installable: it turns “code I wrote” into “code someone else wrote, running as me.”
Bespoke runs every app as one user, under one systemd user manager, with a shared data directory. Process-per-app is real isolation for crashes. It is not isolation for malice. An installed app could read every other app’s SQLite file, and I’d published a mechanism that made installing strangers’ code easy.
Two findings made this concrete faster than I expected.
Every unit read one shared ~/bespoke/env. So my mail app’s encryption key was
in the environment of every other app — including the third-party one I’d just
installed. Not a compromise; just a shared file nobody had thought about since
there was only ever one author. Splitting it took ten minutes: units now read
~/bespoke/env and then an optional ~/bespoke/env.d/<slug> that only that unit
sees.
And the app units had no sandboxing at all. systemd-analyze --user security
scored them 9.4 UNSAFE.
Hardening user units, and one trap
Most systemd hardening advice assumes a system unit and a privileged manager. Some of it silently does nothing for user units, and one directive is actively misleading.
I reached first for ProtectProc=invisible, which stops a process reading its
neighbours’ /proc/<pid>/environ. It configures procfs hidepid=, which filters
by UID — and every app here runs as the same user. It would have hidden
exactly nothing while scoring well on the audit.
The directive that actually does it is PrivatePIDs=yes (systemd 257+), which
gives the unit its own PID namespace. I verified it rather than trusting the
docs, with a probe that does what a real app does — listen on TCP, write SQLite,
make an outbound HTTPS call — and reports what it can see:
ok listen tcp
ok sqlite write
ok outbound https
FAIL net.Interfaces (netlink) address family not supported by protocol
visible pids in /proc: 1
entries in /tmp: 0One visible PID. Empty /tmp. And a casualty I’d half-expected:
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX blocks AF_NETLINK, which Go’s
net.Interfaces() needs. Nothing in the platform calls it, so I kept the
restriction and documented the exact error string — netlinkrib: address family not supported by protocol — where an app author will meet it.
The score went to 5.0 MEDIUM. Worth knowing what the remaining 5.0 is.
Actually separating the data
Process hardening doesn’t stop the mail app reading journal.db. That needs
filesystem scoping, and it turned out pkg/db already had the hook: it reads
BESPOKE_DATA before falling back to a shared directory. So no Go changes, just
different generated units:
Environment=BESPOKE_DATA=%h/bespoke/data/mail
ProtectSystem=strict
ProtectHome=tmpfs
BindReadOnlyPaths=%h/bespoke/bin/mail
BindReadOnlyPaths=%h/bespoke/apps
BindReadOnlyPaths=%h/bespoke/assets
BindPaths=%h/bespoke/data/mailReplace the home directory with an empty tmpfs, then mount back exactly four
things: the app’s own binary, the manifests, the stylesheet, and its own data.
Figuring out which four required reading the framework rather than guessing —
web.Run loads its own manifest for the port, the app switcher reads every
manifest to build the nav, and pkg/ui serves the compiled stylesheet from disk
with the embedded copy as fallback. Miss any one and the app fails at startup or
renders naked.
The result, from inside the sandbox versus on the host:
inside mail's sandbox — data/: [mail]
on the host — data/: [bookmarks builder calendar family-walks
gh-tracker journal mail personal-wiki
print-projects todo platformd.db]A sibling’s database isn’t denied. It’s absent. That distinction matters: there’s no error to handle, no path to probe, nothing to get clever about.
Migrating a live host to that layout is the unglamorous part. Databases move
into per-app directories with their -wal and -shm files, mail’s attachment
directory moves with it, and platformd stays at the root because it isn’t
scoped. I wrote that procedure into the runbook, ran it, and immediately found a
bug in my own loop: it moved platformd.db too. Fixed on the host in seconds;
fixed in the runbook because the next person doesn’t get to watch me catch it.
Backups, and four things the docs didn’t tell me
While I was in there I noticed something worse than any of the above: Litestream
was configured but not installed. Zero backups. The generated config was also
still emitting the replicas: array that v0.5 removed — a config that parses
fine and replicates nothing.
I don’t want my data offsite, so the target is a TrueNAS box on the LAN, with its existing snapshot replication to a Synology as the second copy. That last part matters more than it sounds, because v0.5 writes exactly one replica per database. Fanning out is the storage layer’s job now, not Litestream’s.
Three more things I only learned by doing it:
Retention defaults to 24 hours. Notice a bad delete on Wednesday and Monday is already gone. My databases total about 20 MB; a week costs nothing.
SFTP key auth can’t ride in the URL. The sftp://user:password@host/path
form has no room for a key path, and any NAS worth backing up to has password
auth disabled. The generator had to learn a key-path field.
The host key must be the ECDSA one. Without a pin, Litestream logs sftp host key not verified and connects anyway — which means anything on the LAN can
impersonate the backup target and receive your mail database. Pinning it failed
with host key mismatch until I tried all three key types and found that
Litestream’s Go SSH client negotiates ecdsa-sha2-nistp256. The ed25519 key
ssh-keyscan hands you by default is simply the wrong key.
And one I caused: after repointing the replica from a local path to the NAS,
every shutdown took the full 90-second TimeoutStopSec and ended in SIGKILL.
Litestream keeps per-database state in a hidden .<db>-litestream/ directory
that refers to the old target, and its shutdown sync kept failing on files that
no longer meant anything. Clearing it took shutdown from 90 seconds to zero.
Then the only test that counts: restore from the NAS and compare to live. Integrity ok, same tables, same row counts. A directory listing is not a backup.
What I still can’t do
The honest limit is egress. A scoped app still reaches the network and can send
its own data anywhere. IPAddressAllow=/IPAddressDeny= would fix it, but they
need the service manager to install BPF programs, which an unprivileged user
manager can’t do — they’d appear in the unit file and do nothing, which is the
worst possible outcome for a security control.
Doing it properly means one UID per app, which rewrites deploy, data ownership, and the internal services plane. That’s the real argument for per-UID: not “more isolation” in the abstract, but that it’s the only route to the one control I actually want.
So the sandbox bounds the blast radius and never makes installing a stranger’s app safe. The index says so, the CLI says so on every run, and I’d rather ship that sentence than a reassuring one.
The thing I keep relearning
Nearly every design decision here survived contact with reality. Nearly every factual assumption did not.
ProtectProc looked right and did nothing. Password auth looked fine until the
NAS refused it. The ed25519 host key was the obvious choice and the wrong one. I
blamed PrivatePIDs for the 90-second shutdown and was wrong — it reproduced
without it. Each of those took one experiment to settle and would have taken a
long time to debug in production.
The cheapest tool in this entire effort was a twenty-line probe binary that printed what it could see. Not the architecture, not the ADRs. A program that answers “is this actually true on this machine?”
The code is at bketelsen/bespoke, and the app index — vouching for nothing, as advertised — is at bketelsen/bespoke-apps.