The story rc1 did not tell
The previous post stopped at v0.9.0-rc1. Two more candidates
have landed since, and each one is the kind of change that resets the
six-month soak clock: not because anything broke, but because the
compile-time refusal surface grew (rc2) and a host dependency was removed
wholesale (rc3). The public Kern API is back-compatible in both. No existing
program needs a line changed.
- rc2 made injection a property of the grammar. SQL, shell, URL, regex and template injection are compile errors, not runtime filters you have to remember to call.
- rc3 pulled the container engine inside the binary. It now fetches and runs real OCI images with nothing on the host but Kern, a Linux kernel and libc.
Why the soak clock restarts, and why that is the honest call.
v0.9.x exists to prove six months of zero breaking changes
before v1.0. rc2 and rc3 add no breaking changes, but they each change the
set of programs the compiler accepts or the dependencies a deploy requires.
Counting that as a fresh soak window is the conservative reading of the
promise. 17 of 21 v1.0 gates are now closed or have their harness
shipped; the four that remain are the foundation incorporation, the
real-account KMS exercise, the parser-fuzzer 24 h run, and the soak itself.
Three of those four are not code.
Injection prevention as a syntactic category (rc2)
Most languages treat injection as a runtime discipline: parameterize your queries, escape your shell arguments, validate your URLs, and hope every call site remembered. rc2 takes a different position. The promise is that an AST shape an attacker would need to land the bug does not exist in valid Kern source. Five orthogonal compile-time rules carry it.
The foundation is a family of @<kind>_literal annotations.
A parameter decorated with one of them must receive a string literal AST node
at every call site. A value assembled from user input is a different AST node,
so it does not type-check.
import db.postgres as pg
async fn find_user(conn: pg.Conn, id: str) -> Result<Row>:
# @sql_literal: the query text must be a literal.
# User input only ever arrives as a bound parameter.
return await pg.query(conn, "SELECT * FROM users WHERE id = $1", [id])?
async fn broken(conn: pg.Conn, id: str):
# pg.query(conn, "SELECT * FROM users WHERE id = " + id)
# COMPILE ERROR: @sql_literal expects a string literal,
# got a concatenation. The classic SQLi shape will not parse
# into a valid program.
pass
- Five kinds:
@sql_literal,@shell_literal,@url_literal,@regex_literal,@template_literal. The stdlib SQL, SSRF and ReDoS surfaces are decorated. Primitives that took a table or column name as a plainstrwith a "must not contain user input" warning were removed outright; the structural rule replaces the warning. - TLS-mandatory sub-rule:
@url_literalrejects any literal that does not begin withhttps://.parse_urlandtrust_urlreject plainhttp://at runtime. Url<Trusted, Untrusted>type-state: for URLs built at runtime,parse_url(raw)?yieldsUrl<Untrusted>, and onlytrust_url(raw, allowed_hosts)?promotes it after a host-allowlist check. Thehttp_get_url_typedsink rejects bare strings, untracked variables, andUrl<Untrusted>, which closes SSRF for the runtime-built case.SecureandPersonalDatanow ban every sink: the existingSecure<_, Plain>andPersonalData<T>rules covered database and HTTP calls. They now also cover display sinks (print/log_*) and file sinks (write_file), and taint propagates through string concatenation. Without this,log_info("user " + name)would leak personal data into a log shipper undetected, and GDPR Article 30 discipline rested on developer memory alone.auth_jwt_verify: HS256 is hardcoded in the C runtime, so the OWASP "alg confusion" attack family is structurally absent from the JWT path.
A self-contained container engine, no docker, no podman (rc3)
In rc1 the native container runtime sandboxed a process with namespaces,
cgroups and overlayfs, but the broader subsystem still shelled out to a
host-installed Docker or Podman to pull and run images. rc3 closes that gap.
The engine is now compiled into the language. It has been verified end to end
against debian:bookworm-slim and alpine:3.19 from the
Docker registry, on hosts that have had docker, podman, skopeo, runc, iproute2
and iptables explicitly purged.
import sys.container as ctr
async fn main() -> Result<str>:
# Pull + run a real image. No docker, no podman on the host.
# Signature checked against a cosign-compatible Ed25519 key,
# manifest digest pinned to the blob we just fetched.
out = await ctr.oci_run_image_verified(
image: "alpine:3.19",
pubkey: env("KERN_OCI_PUBKEY"),
cmd: ["echo", "hello from a kern-pulled container"]
)?
print(out)
return Ok(out)
What that one call does without touching the host toolchain:
- In-tree HTTPS and tar: manifests and layer blobs come down over the vendored mbedTLS the rest of the runtime already uses; layer tarballs are unpacked by a statically linked libarchive 3.7.4 (tar + gzip + zstd + xz read paths only, around 3.5 MB). The final binary shows zero libarchive entries under
ldd. - Hand-rolled netlink:
runtime/kern_netlink.cbuildsRTM_NEWLINK/SETLINK/NEWADDR/NEWROUTEmessages straight against anAF_NETLINKsocket, replacing everyfork+exec("ip")andfork+exec("nsenter"). - Native nftables NAT: outbound masquerade installs an
inet kern_nattable and postrouting chain overNFNETLINK_NFTwith the full payload, bitwise, cmp, meta and masq expression set. NoiptablesornftCLI. - Namespaces, cgroups v2, overlayfs:
clone()with the fullCLONE_NEW*set, direct writes to/sys/fs/cgroup/kern/<name>/, andpivot_root(2)into the merged overlay so the command sees the image's own filesystem. - Cosign-compatible verification:
oci_pull_image_verifiedfetches the companion signature, verifies the Ed25519 sig over the simplesigning blob, and pinscritical.image.docker-manifest-digestto the manifest it just fetched.KERN_OCI_PUBKEYis honoured. - Full lifecycle, native: every
popen("podman ...")andpopen("docker ...")is retired.run,stop,logs,exec,stats,inspect,pauseand the rest run against cgroup, overlay and state-dir bookkeeping under/var/kern/containers/<name>/.
The dependency list that a production Kern deploy no longer needs is concrete: docker, containerd, runc, crun, podman, buildah, skopeo, iptables, nftables, iproute2 and libarchive-dev. What remains is the Kern binary, a Linux kernel 5.15 or newer with cgroup v2 and user namespaces, and libc.
The macOS path is honest about the constraint. The native engine is Linux-only by kernel design. On macOS the runtime returns a clear "containers require Linux" error rather than silently falling back to Docker Desktop. Develop against a Linux VM locally, target a Linux host in production.
Supply-chain tripwires that keep the shellout gone (rc3)
Removing the shellout once is easy. Keeping it gone across future commits is the real work, so rc3 ships the gates that make a regression fail loudly.
make check-container-no-shellout: a source-level grep gate. Apopen(orsystem(in any of the container, network, namespace, cgroup, overlay, fs or netlink runtime sources fails the build.make runtimedepends on it.- CI on a privileged Debian runner: the container leg apt-purges every container and network shellout binary first, then asserts
command -v docker podman ...returns nothing before the suite runs. strace -f -e execvewrapper: the conformance run greps the trace for any of docker, podman, iptables, nft, skopeo, buildah, runc, crun, ip, nsenter, mount or unshare, and exits non-zero if one appears.
The bootstrap stopped trying to allocate 145 GB
This one is on main, heading into rc4, and it is worth stating
plainly because the old number was embarrassing. The self-bootstrap, the step
where the Kern compiler compiles itself, used to climb toward 145 GB of memory
and OOM on anything short of a very large machine. It now self-bootstraps in
about a minute in a couple of gigabytes of RAM. That is the difference between
"needs a dedicated build box" and "runs on a laptop or a normal CI
container." A few changes did most of the work:
- O(1)
gc_find_header: an O(N) walk through every live allocation became pointer-offset arithmetic with a magic-word check. The GC mark phase dropped from O(M*N) to O(M+N). Single biggest win, roughly 7x, and it made the rest of the optimizations visible. - Native C string primitives:
kern_str_sliceand friends (contains,starts_with,index_of) replaced pure-Kern versions that ran a full UTF-8 encode of the source on every call. Around a 130x reduction in bootstrap memory. - Cached length and interned single chars: the lexer's
len(source)andchar_athot loop became O(1). - Byte-based GC throttle: the collector now triggers on a 256 MB allocation delta rather than an allocation count, so a million small allocations cannot drift before a collect.
Also on main, heading into rc4
- CapTP over the network: three-vat introductions, promise pipelining, and a
std.captp.nettransport over TCP, with Ed25519-signed gift tokens for forgery resistance. A TCP loopback Bootstrap and Resolve roundtrip passes end to end. - A formally verified lexer track: Coq was chosen as the proof assistant (INRIA, EU-native, the CompCert track record), and
proofs/lexer.vscaffolds the token alphabet and the correctness theorems. A grammar-versus-parser diff harness lints the spec EBNF against the lexer keywords. - EU compliance scaffolding: a
kern ai-audit verifyCLI that replays the chain hash from the shell (EU AI Act Article 12), NIS2 Article 23 reporting helpers (early_warning,notification,final_report), a DORA SPDX 2.3 SBOM emitter alongside the existing CycloneDX one, a GDPR Article 30 processing-records generator, and a 90-day soak monitor that samples RSS, fd-count and audit records into CSV.
What v0.9.x is for
v0.9.0-rc3 is not v1.0. The point of the v0.9.x line
is soak time:
- Parser fuzzer 24 h clean. Harness shipped (commit
4b700d2), five mutators (bit flip, insert, delete, truncate, token splice), smoke run 521 iters / 0 crashes / 0 timeouts in 120 s. CI gate command for v1.0:tools/fuzz_parser.sh 86400. - HSM / PKCS#11 signing. Adds a PKCS#11 binding for hardware-backed Ed25519 release signing.
- Full DORA Art. 9/10/11 risk mapping. Currently mapped at the operation level; the runtime risk register is the v0.9.x deliverable.
- Real-account exercise of cloud KMS providers. Code interfaces ship here; running them against real Scaleway / Azure tenants is the regulatory check.
- External security audit. Scheduled inside the
v0.9.xwindow. - 6 months of zero breaking changes. Then v1.0.
Every item above is on the ROADMAP with a checkbox. None are aspirational; all are blockers.
By the numbers
- 944 tests at 100% pass, 728 conformance and 216 negative; both gates CI-blocking
- 17 of 21 v1.0 ROADMAP gates closed or harness shipped
- 5 injection classes (SQL, shell, URL, regex, template) eliminated as a syntactic category
- 0 container daemons on the host: real OCI images pulled and run by the Kern binary alone
- 102 stdlib modules across
std,data,net,db,sys - 31,694 lines of Kern in the self-hosted compiler
- 36,797 lines in the C runtime, around 300 FFI bridge functions
- Self-bootstraps in about a minute in a couple of GB of RAM, down from a 145 GB OOM
- EUPL-1.2, hosted on Codeberg, governed from the Netherlands
Try v0.9.0-rc3
Install on Linux or macOS. Single binary, no runtime dependencies.
curl -fsSL https://kern-lang.eu/install.sh | bash