It's the right question, because it's the one that should break the pitch if we got the architecture wrong. A verify-loop is convincing when the thing it checks is cheap and total to check. Side effects are neither.

So let me answer it honestly. Which means starting with what we don't do.

The honest part first​

When we say Skipper closes the loop, we mean it the way control theory means it: the feedback signal has to be trustworthy enough to act on.

Type-checking and deterministic execution give us that. They're cheap to run, and the verdict is total. The check passes for every input or it doesn't, and we don't have to believe a story about it.

A side effect gives us none of that. You can't cheaply verify that an email arrived. You can't verify it totally either, because the answer lives on someone else's mail server on a Tuesday.

So the loop doesn't check whether the effect happened. It checks whether the effect was described correctly. That distinction is the whole post.

We already do this. It's called a compiler.​

We didn't invent this for AI. Compilers have done it for 50 years.

A C compiler verifies that your call to send() matches its declared signature, every argument type and the return shape, and it does that without ever opening a socket. It has never once sent a packet to find out whether your code is correct. That's the whole trick.

Treat Agent Output Like Compiler Output argued we should treat agent output the same way: stop reading each artifact by hand, build the verification into the process that produces it. Side effects are where that argument has to prove it isn't hand-waving. They're the one place people assume the compiler move can't reach. The three pieces below are what reaching looks like.

1. Effects are typed values​

Anything that touches the outside world (an email send, an HTTP call, a hash, a database write) isn't an opaque call the model drops wherever it likes. You declare it up front as a typed external: a fixed contract of inputs, outputs, and params.

There's a rule the reactive core enforces with no exceptions. The core can't do I/O at all. The deterministic, reactive part of a Skipper service never touches the network. An effect executes in exactly one sanctioned place: a separate subprocess at the edge, reached through a generated wrapper.

So the loop's job here is narrow and decidable. It checks that the constructed effect type-checks against its contract. Execution gets deferred across the boundary. The model describes the email send. It can't smuggle a socket into the part of the program we're trying to prove things about.

2. The sandbox runs against typed doubles​

For the loop to run a service end to end, the boundary needs something on the other side. That something is a stub. Each external call gets pinned to a typed mock record: given these params, return this value, or this error.

An email send becomes called with {to, subject, body}, returns sent, and the assertion lives in the handler that consumes the result. The model's call site has to match the declared intent or the test fails. You can't hide a malformed payload, because the payload's shape is part of the contract.

Reactive computation, the Skip Runtime the whole thing runs on, makes those outputs reproducible from the same inputs. Same inputs, same graph, same result, every run. So the verification pass is deterministic without ever touching the network. We're not flaking on some third-party rate limit while trying to decide whether the code is correct. We keep those problems apart on purpose.

3. The contract is the unit of verification​

Here's where the big-company question gets its real answer.

You hand Skipper an OpenAPI spec, or a prompt it derives one from, and from that point, for that run, the spec is frozen. It's the fixed reference. The agent codes against it. The loop type-checks against it, unit-tests against it, validates every call site against it. The typed stubs derived from the externals are what let the generated code actually run.

Look at what that does to the "no cross-company sandbox" problem. It dissolves it. There was never going to be a live internal service inside the verification loop, and there doesn't need to be. The interface is fixed, so the model's stochasticity is contained even when the implementation is remote, proprietary, and sitting behind 3 VPNs you've never heard of.

Verification happens against the contract. The live service never enters the loop, so its absence costs you nothing. You need their types, and that's it.

What the loop will not tell you​

Here's the part I'd rather say out loud than have you find out later.

The loop does not claim the effect succeeded in production. It doesn't promise the email reached an inbox, that Sentry ingested the event, that the internal service was even reachable when your code ran for real. Delivery, idempotency, retries: that's the subprocess at the edge running against real credentials, outside the verification loop.

We keep that integration layer visible and separate instead of folding it into the word "verified." I'm doing that on purpose. A closed loop with a dishonest sensor is just an unstable system with extra steps, flying the plane confidently into the ground.

"Verified" has to mean exactly one thing, or it means nothing. Here it means the code is type-correct against the contracts and runs deterministically against them. Whether the third party 3 time zones away is having a good day is a separate question, and the loop doesn't pretend to answer it.

The model is stochastic. The pipe around it isn't. Side effects don't change that. Quietly pretending the loop covers delivery would, so it doesn't, and it says so.

PARIS, FRANCE — June 1, 2026 — SkipLabs today launched Skipper, a closed-loop coding agent that takes a single prompt and returns a running, validated service. Skipper was built by Julien Verlaguet, creator of Facebook's Hack programming language, alongside a team of engineers drawn from Facebook, Microsoft, Microsoft Research, Meta and the broader systems community. It serves as the architectural layer sitting between AI models and shipped software. Skipper is designed specifically for technical builders: product managers running their own AI workflows, engineering teams shipping production services, and developers who demand architectural discipline without architectural overhead.

The Missing Link in AI Coding​

Skipper arrives at a critical inflection point. While AI coding tools have made the first draft of a service nearly free, what remains is the gap between almost running and actually running in production. This is an architecture problem—one that doesn't disappear just because a language model gets smarter.

"Building correct software has always been an architecture problem disguised as a coding problem," said Julien Verlaguet, Founder and CEO of SkipLabs. "AI did not change that; it just made the problem more urgent." Skipper sits beneath the foundation models. Rather than competing with Claude, GPT, or Gemini, it acts as the substrate that turns their output into finished software rather than a draft awaiting review. When a builder describes the service they want, Skipper handles the entire pipeline autonomously: decomposing the work into manageable pieces, routing each piece to the specific AI model best suited for that task, generating and validating the code, and returning a reliable, running service. No developer review, no oversight cycles, no back-and-forth.

Fixing State and Concurrency​

State management and concurrency are the hardest parts of writing correct software, and the exact areas where AI most often produces broken code. Skipper solves this via its reactive runtime—drawn from the open-source Skip framework Verlaguet built at Facebook in 2017. Because the runtime automatically handles state management and concurrency, the AI model never has to reason about cause and effect across a sprawling state graph. By shrinking the surface area where AI can make mistakes, Skipper ensures more of what it produces actually works.

"Closed loop is not a feature," Verlaguet added. "It is a different theory of what an AI coding tool is supposed to do. The current generation makes the developer faster. The next generation makes the developer's involvement optional. Describe what you want and Skipper builds it."

What Ships Today​

Launching today, Skipper delivers the core closed-loop experience: a single prompt becomes a running service. A model-agnostic harness routes each task to the AI model best suited for it, with no lock-in to any single foundation model. And Skipper-built services integrate with the outside world from day one — calling external APIs, fetching live data, and posting to other systems to ensure the software fits into a real stack, not just a sandbox.

The Product Roadmap​

In the weeks following launch, SkipLabs will roll out SKJS, a sound, TypeScript-compatible type system that lets Skipper understand how changes to one part of the code affect the rest. Next comes in-place incremental updates, allowing builders to change a specification and have Skipper modify the running service without a full rebuild. By late 2026, SkipLabs will deliver precise, zero-pollution awareness of the full codebase at any scale.

Who Skipper is For​

Skipper is built for those who experience the frustration of watching an AI generate a near-working idea, yet possess the technical knowledge to see exactly why the last 10% breaks. It is designed for technical product managers wiring AI into production tools via real API integrations rather than sandboxed demos. It also serves engineering teams shipping production backend services who want architectural discipline without the per-line review cycle, and software development firms building AI-driven systems for clients whose code absolutely must ship.

A Decade Making Code Provably Correct​

The architecture beneath Skipper predates the current AI coding wave and is backed by over a decade of infrastructure-first philosophy. Verlaguet created Hack, the gradually typed programming language that powers Facebook's business logic and over 100 million lines of production code. Amplify Partners, which led SkipLabs' $8 million seed round, noted in its investment memo that Verlaguet is "one of the top two to three programming language designers in the world." His work on programming language design has also been publicly endorsed by Turing Award winner and former Meta Chief AI Scientist Yann LeCun. Skipper is available today at skipperai.dev.

About SkipLabs​

SkipLabs builds the infrastructure layer that makes AI-generated code work in production. Founded in 2022 by Julien Verlaguet, creator of Facebook's Hack programming language, the company develops tools that close the loop between intent and running software for technical product managers, engineering teams, and the firms that build production systems on top of AI. SkipLabs is backed by Amplify Partners, with angel investors including Yann LeCun and Spencer Kimball, co-founder and CEO of Cockroach Labs. For more information, visit skiplabs.io.

What closed-loop meant before we borrowed it​

Closed-loop control comes from control theory. An autopilot reads altitude, compares it to the target, nudges the elevators based on the error. The defining property isn't that a loop exists. It's that the feedback signal is trustworthy enough to act on.

An autopilot with a broken altimeter will fly the plane into the ground. Confidently. Autonomously. In perfect mechanical closure.

Control theorists know this. They obsess over sensor fidelity, because a closed loop with a bad sensor isn't a stable system. It's an unstable one with extra steps.

Medical devices go further. An artificial pancreas is closed-loop. A glucose monitor that beeps at a human to dose insulin is not. The line isn't drawn at the loop. It's drawn at whether the signal is sound enough to take the human out. You earn closed-loop autonomy through signal quality. You can't declare it through architecture.

What most agent loops actually close on​

So look at what a typical coding agent feeds back into its next decision:

• Test output. Tests are samples, not proofs. The agent can pass them without satisfying the intent. Sometimes by editing the test.
• Linter and type-checker output. Useful, but TypeScript's type system is unsound by design. A green check means "I couldn't find a contradiction in the bit I can see." It doesn't mean correct.
• Runtime logs. Noisy, incomplete, silent on the failures you actually care about.
• Model-based judgment. The agent's own confidence, or a second model with the same blind spots. Either way, the signal is the thing you were trying to check.

Every one of these is a lossy sensor. It drops information about the thing it's supposed to measure. The loop still closes mechanically. Output feeds back into the next decision. But what flows through it is a low-fidelity proxy for correctness.

And because the agent acts on that proxy with full autonomy, small signal errors compound into the failure modes we all recognize:

• The agent deletes the failing test and declares victory.
• The agent ships code that type-checks fine but violates an invariant the types can't express.
• The agent loops for 40 minutes polishing a solution that was already correct on attempt 3, because nothing in the signal told it to stop.
• The agent drifts. Each step is locally plausible. The trajectory is nonsense.

These aren't bugs in the agents. They're what closed-loop control does when the sensor is lossy. Control theory called this decades ago. We're rediscovering it the hard way.

A physics problem, not a craftsmanship problem​

Here's the part worth sitting with. No amount of prompt engineering, scaffolding, or agent-framework cleverness fixes a lossy feedback signal. You can tune the controller all day. If the altimeter is wrong, the altitude is wrong. The ceiling on an agent's reliability is the trustworthiness of what it closes the loop on.

So the real question when you're evaluating an AI coding tool isn't how autonomous is the loop. It's what does the loop close on, and how much can you trust it?

A sound type checker closes a better loop than an unsound one. A deterministic execution environment closes a better loop than a flaky one. A reactive system that recomputes only what changed closes a better loop than one that re-runs everything and re-derives confidence from noise. These are sensor upgrades. They're what lets the loop actually control something.

This is what Skipper is built on. SKJS is a fully sound TypeScript-compatible type checker, so when Skipper's loop closes on a type signal, that signal means what it says. Execution is deterministic, so "it worked" is reproducible instead of probabilistic. Computation is reactive, so feedback is precise instead of a fog of re-runs. Three sensor upgrades. Together they're what lets Skipper close the loop at all.

Why this matters​

The industry has been iterating hard on the controller. Better prompts, better scaffolds, better planning. And treating the feedback signal as given. The feedback signal was never given. It's the variable that decides whether your autonomous loop is closed-loop or just an unstable one in a costume.

Skipper closes the loop because Skipper took the sensor seriously. Everything else in the category is running open-loop with confidence it hasn't earned.

That essay argued that the right response to AI-generated code isn't more code review — it's the same trick the industry already pulled with compilers: build the verification into the process that produces the output, then stop reviewing each artifact by hand.

The internet had thoughts. Or at least X had thoughts... a lot of them, often in flowery language, as you can imagine. I read every one. Most of them landed on three or four arguments worth addressing head-on.

Before I go through them, I want to restate the thesis, because most of the responses were answering a question I didn't ask.

Restating the take​

The point of the original piece was not that LLMs are compilers. The point was that fifty years of compiler R&D — type systems, static analysis, formal semantics, abstract interpretation, symbolic execution, model checking — is exactly the apparatus we now need to wrap around coding agents.

Agents on their own cannot be trusted to ship code. They need a straightjacket. Static analysis is the straightjacket. The compiler community spent five decades building tools that prove things about code before it runs. That toolkit is the answer. Agents need it more than humans ever did, not less.

The model is stochastic. The pipe around it isn't.

The analogy isn't "agents are compilers." The analogy is "the R&D tradition that produced the trust apparatus around compilers is the R&D tradition I need to point at agents now."

With that on the table, let me go through what landed.

What the critics got right​

Two objections genuinely move the piece, and both of them make the case for static analysis stronger, not weaker.

Hallucination has no compiler analogue​

The cleanest version of the point, paraphrased from one of the more direct replies: compilers don't hallucinate, and they don't recommend functions that don't exist.

Right. This is the failure mode that doesn't exist in the compiler world, and I should have addressed it explicitly. A compiler translates a formally specified input into a formally specified output. An LLM produces semantics, sometimes out of thin air.

But notice what catches a hallucinated function call: static analysis. What catches a reference to a nonexistent symbol: type checking and name resolution specifically — passes that have shipped in every serious compiler since the 1970s. Hallucination is exactly the failure mode this machinery was built to catch. The agent will invent a function; a sound type system will reject the program before it runs. That's the apparatus doing its job.

The reliability gap is real, and it's the entire engineering challenge​

The failure-rate gap between a modern C compiler and a frontier LLM isn't 1.5× or 10×. It's many orders of magnitude. Compilers fail rarely enough that individual miscompilations are notable events, tracked, and fixed across the toolchain. LLMs fail often enough that "did the model hallucinate" is a routine line item in any agent pipeline.

This is not an argument against the analogy. It is the argument for aggressive static guarantees on agent output. When the author fails at parts-per-million, you can get away with light verification. When the author fails at parts-per-hundred, you cannot. The reliability gap is precisely why every tool the compiler community has produced — and several it hasn't yet — needs to be pointed at coding agents now.

Both of these objections are correct. Both of them strengthen the case for treating agent output as something to be verified by serious infrastructure, not eyeballed by a human.

What the critics got right that doesn't touch the argument​

A second cluster of objections is technically true and rhetorically beside the point.

"Compilers have formal specifications. LLMs don't." True. The analogy is not "LLMs are formally specified like compilers." The analogy is that the apparatus I built around compilers — much of which doesn't depend on the compiler itself being formally specified — is what agents need. Type checkers, sanitizers, fuzzers, model checkers: most of this apparatus operates on the output program, not on the source-to-machine translation. It would be useful even if the upstream artifact had no spec at all. Which is convenient, because in the agent case the upstream artifact doesn't.

"Humans actually do read compiler output." Yes — in embedded, GPU, security, and performance-critical contexts. The honest version of the original claim is narrower: most code shipped in production today is not produced by a human reading the assembly first, and that's possible because of the apparatus. I should have written the narrower version. But this correction is about scope, not about the structure of the argument.

"Compilers are extremely reliable. LLMs constantly fail." Already covered above. Yes — and this is why the apparatus matters more, not less.

These are real points. They're not the points the determinism arguments were trying to make.

Where the critics missed​

The single largest cluster of objections — and it was a very large cluster — was effectively one critique repeated three ways: compilers are deterministic, LLMs aren't, therefore the analogy fails.

This is arguing about a property of the artifact. The original piece was arguing about the apparatus around the artifact. They are different things, and conflating them is the actual category error.

You don't trust a compiled binary because the compiler is deterministic. You trust it because of types, tests, fuzzers, sanitizers, ABI checks, monitoring, and rollback. Most of that apparatus would be useful even if the compiler weren't deterministic — which is convenient, because in practice it often isn't. Compiler output shifts under PGO, LTO, parallel link order, optimization-level changes, GPU floating-point reordering, and timestamps. Nobody has decided compilers are therefore not analogous to themselves. The apparatus absorbs the non-determinism.

The validators, the type checkers, the test runners, the static analyzers, the CI gates: those are generally not deterministic either (some may be), and none of them are individually robust. What they are is a stack — each one trims a class of failure, and together they push the failure rate of the output close enough to zero that you trust the artifact. Build the stack. The model is stochastic. The pipe around it doesn't have to be deterministic — it has to be robust enough that the artifact is.

The "just use temperature zero" variant of the criticism misses in the other direction. Greedy decoding with a fixed seed gets you bit-identical output for a single prompt at a single checkpoint. It does not get you a deterministic function from natural-language intent to working code. The variance lives upstream, in the prompt, the retrieval context, the tool transcripts. You can pin the sampler and still get a different program tomorrow. Sampler determinism is not the determinism property anyone actually wants.

If your response to "build the verification apparatus" is "but LLMs aren't deterministic," you're describing why the apparatus is harder to build, not why you shouldn't build it.

Worth noting: since the original post went up, several of the people running the determinism critique have publicly walked it back. The strongest correction came from inside the LLVM community — the observation that real-world compilers are wildly non-deterministic by default. Internal passes use unordered containers keyed on pointer addresses, which shift every run thanks to ASLR. Non-stable sorting algorithms reorder things between invocations. Filesystem inode allocation dictates enumeration order. Reproducible builds are a massive engineering effort precisely because the underlying toolchain is so far from deterministic to begin with. The mathematical theory of a compiler is deterministic. The actual production toolchain is not. This is exactly the distinction the original piece was trying to make, made better by people who initially thought I was wrong about it.

The most rigorous version of the rebuttal came from outside the thread entirely. Isaac Van Doren's Nondeterminism's not the problem makes the point cleanly: imagine a compiler that picks register assignments by Math.random — it would still be useful. Then set an LLM to temperature zero — it's still untrustworthy. Determinism is irrelevant. The real difference is that programming languages have semantics and prompts don't. I agree, and that reframe is the spine of the next section.

The position the debate has settled into is the right one: compilers are not deterministic, LLMs are more non-deterministic than compilers, and the gap is large.

That gap is the engineering challenge, it is not a category boundary.

What static analysis actually buys you​

The argument so far is abstract. Here's the concrete version.

The underlying problem is straightforward: programming languages have semantics; prompts don't. The nearly 900 pages of the Java Language Specification tell you exactly what promises the language makes about your code's behavior. A prompt makes no promises. To give a prompt the kind of guarantee a language gives you, you would need an external tool to validate that the output upholds the prompt's intended semantics — and the cost of doing that at the level of full theorem proving is famously prohibitive.

The honest version is more interesting. Programming languages don't actually specify everything either. Java doesn't tell you when the garbage collector runs. Haskell doesn't tell you what gets evaluated next — that's a function of the runtime, not the language. C doesn't tell you whether a variable lives on the stack or in a register. These languages succeed not because their specifications are exhaustive, but because the apparatus around them — the GC, the runtime, the register allocator — makes the underspecified parts irrelevant to the correctness of your program. The abstraction holds.

That's the move we need for agents. The prompt is underspecified, and at one level it always will be. The apparatus has to make that underspecification stop mattering — to enforce, at the level of the output program, the properties the prompt couldn't pin down.

You don't need full theorem proving to get most of the way there. The compiler R&D tradition has produced a layered stack of tools that impose semantic guarantees on programs without needing the prompt to be specified at all:

• A sound type checker rejects programs that reference functions that don't exist, that pass arguments of the wrong shape, that mix incompatible types. That's a hallucination-detection system, caught statically, no test runs required.
• Data-flow analysis catches uninitialized reads, use-after-free, leaked resources, and a long list of memory and ownership errors that no amount of LLM "review" will reliably find.
• Abstract interpretation proves bounds on values — array indices, integer overflow, null dereference — across all execution paths, not just the ones a test happened to exercise.
• Symbolic execution explores program paths a test suite would never reach, finding inputs that violate assertions.
• Bounded model checking exhaustively explores program behaviors up to a finite bound. Counterexamples it finds are real bugs; the absence of one means any remaining bug has to be deeper than the bound you checked. This is a stronger guarantee than "we tested some inputs," and weaker than full verification.

None of this is speculative. It is what has been shipping in industrial compilers and verifiers for decades — Coverity, Infer, Astrée, KLEE, CBMC, Frama-C, the seL4 verification effort. The agent era is not asking for new science. It is asking us to point existing science at a new producer of code.

The semantic ground that the prompt fails to establish gets re-established by analysis of the output program. The agent produces; the apparatus verifies; the program ships only if the apparatus signs off.

It's worth being precise about the topology here, because it's the place where the compiler analogy is genuinely loose. In the compiler world, verification is intricated in the production pipeline: the type checker runs before codegen, the optimizer is constrained to semantics-preserving transformations, the linker validates symbols. By the time you have an output, the apparatus has already done its work. In the agent world, the topology is looser. The agent can generate and ship straight to the apparatus as a downstream gate, or it can interleave — generate a fragment, run the type checker, take the errors, continue — with verification stepping in at whatever granularity the loop allows. Different topology, same principle: the trust lives in the pipeline, not in the artifact. The asymmetry that does remain is one of defaults: the compiler's verification is non-optional and built in, while the agent's has to be deliberately wired into the loop. That's an argument for tighter integration over time — not for adding verification to the loop, but for making it the default that the agent can't opt out of.

This is what we are building at SkipLabs. SKJS is a TypeScript-compatible type checker that is sound — it doesn't have the escape hatches that make standard TypeScript ergonomic for humans and unreliable as a verification layer. The reactive runtime enforces explicit dependency contracts that an agent has to satisfy before code runs. This is pedantry that would feel like friction to a human author and is exactly right for an agent that needs to be held accountable for what it generates before anything reaches production.

The apparatus exists. Some of it ships today. More of it is being ported from the compiler world into the agent world right now. The point of the original piece — and the point I should have made more clearly — is that this is where the engineering work is. Not in scaling code review. Not in waiting for better models. In building the static guarantees that make trust warranted.

If agents are now the primary authors of code, should the tools and languages those agents use still be optimized for humans?

I don't think they should. And the history of programming languages tells us exactly why we'll resist that change — and why we'll embrace it anyway.

Programming Was Always a Human-Readability Project​

Every major leap forward in programming language design has been, at its core, a leap toward human legibility. Assembly gave way to C because humans could reason about it. C gave way to higher-level languages because abstraction made programs easier to hold in a person's head. And at the far end of that arc, you get Python and Ruby: languages whose defining virtue is that they read almost like English prose.

Python's indentation rules aren't a technical necessity — they're a readability convention enforced by the language itself. Ruby's syntax is deliberately soft, forgiving, close to natural language. The TIOBE index (a widely cited ranking of programming language popularity) doesn't measure what's technically optimal. It measures what engineers want to write and read. For decades, those were the same thing.

The tools followed the same logic. Error messages were written for people. Outputs were formatted for terminal reading. Type systems were made optional or loosened to reduce friction for the human author. Everything was, and still is, optimized for the person sitting at the keyboard.

Hack and the Last Time We Had This Argument​

We've been here before, though the stakes were smaller.

When Julien Verlaguet and the team at Facebook created Hack, they were solving a specific problem: PHP, the language powering the entire Facebook codebase, was fundamentally type-unsafe. At the scale Facebook operated, that wasn't just an aesthetic problem, it was an engineering liability. You couldn't refactor with confidence. You couldn't catch entire classes of bugs statically. The language was optimized for the speed of a solo developer writing web pages, not for the discipline of thousands of engineers maintaining hundreds of millions of lines.

So Julien built Hack: a gradually-typed, rigorously annotated replacement that was strictly less pleasant to write than PHP. It was more verbose. It demanded precision. It required you to say exactly what you meant. And then they got every engineer at Facebook to switch.

The rollout did not go smoothly at first. Engineers pushed back hard. They asked for escape hatches — ways to keep writing PHP in the parts of the codebase they owned. They requested exemptions, delayed migration timelines, special-case bypasses. Some wanted help porting their code, others simply wanted to be left alone. The friction was real and sustained.

But the org held the line. The resistance gave way not because engineers were eventually won over by argument, but because the environment made staying on PHP progressively harder. The CI pipeline was tightened incrementally — more checks, stricter gates, less tolerance for untyped code — and the hierarchy didn't offer the opt-outs people asked for. The path of least resistance slowly became compliance.

Agents Don't Need the Training Wheels​

A coding agent doesn't need readable syntax. It doesn't need forgiving type semantics. It doesn't need terse, human-friendly error messages. It needs the opposite.

An agent benefits from verbosity. Long, precise, unambiguous tool outputs are easier to parse than short, clever, human-optimized ones. An agent benefits from pedantic execution — systems that fail loudly and specifically when something is wrong, rather than silently coercing types or swallowing errors in the name of developer experience. An agent benefits from strong type systems not because it needs guardrails, but because types are the most information-dense way to describe what code is supposed to do.

When a human reads an error message, you want it to be concise, sympathetic, maybe even helpful in a narrative sense. When an agent reads an error message, you want it to be complete, structured, and machine-parseable. These are different optimization targets that have been collapsed into one because until recently, the reader was always human.

The same applies to tool APIs, CLI outputs, documentation formats, error codes. All of it was designed for a human consumer. That assumption is now in question.

The Coming Divergence​

I expect we'll see a bifurcation in tooling. On one side, languages and tools that continue to optimize for human authors — and there will always be human authors. On the other, a new generation of tools designed with the agent as the primary consumer: strict, verbose, formally specified, hostile to ambiguity.

This is what we're building toward at SkipLabs. SKJS, our TypeScript-compatible type checker, is sound in ways that standard TypeScript isn't. That makes it harder for humans and better for agents. Our reactive runtime enforces explicit dependency contracts that would feel over-engineered to a developer writing code by hand, and are exactly right for an agent that needs to reason about what depends on what.

The pattern is consistent: pedantry that felt like friction when humans were the authors becomes an asset when the author is an agent operating at scale.

Why We'll Resist This, and Why We'll Do It Anyway​

Julien's engineers resisted the switch to Hack. PHP felt natural. Hack felt bureaucratic. And they made the switch because the system-level argument was overwhelming.

We'll resist this transition for the same reason: the new tools will feel worse to write, and they will be worse to write, by human standards. That's not a design failure — it's a deliberate optimization for a different consumer. The discomfort is the point.

But the system-level argument is, again, overwhelming. If agents are generating the bulk of code, and the tools those agents use are designed around human readability, we're leaving enormous capability on the table. We're asking the agent to work in a medium optimized for someone else.

We made the tools more readable to get more developers. We'll make them less readable — more precise, more formal, more machine-native — to get better agents. The readability era of programming languages was long and productive, and is now coming to an end.

This post follows Treat Agent Output Like Compiler Output.

The answer, I think, is hiding in something we already used once before and then promptly forgot we did: the compiler.

Nobody Reviews Compiler Output​

Think about how you relate to your compiler. You write C++, Rust, Go, and the toolchain spits out a binary. Do you open that binary and read through the assembly? Do you schedule a meeting with a colleague to review the object code before shipping?

Of course not. That would be absurd. And not because you blindly trust compilers, you don't. Compilers have bugs. Compilers have had famously catastrophic bugs. But you've constructed an entire apparatus that makes reviewing the output unnecessary: you write tests against observable behavior, you have type systems that constrain what the output can do, you have reproducible builds, you have fuzzing and sanitizers and formal verification in high-stakes domains. You trust the process, not the artifact.

We haven't built that apparatus for coding agents. And that, not the output itself, is what's actually missing.

The Real Diagnosis​

Consider that a developer like Michael Novati landed 417 PRs in a single day in February. That's enough to argue that reviewing AI-generated code is volumetrically impossible. And it is. But I'd push the diagnosis further: the volume isn't the problem, it's a symptom that exposes the problem.

The problem is that we're still treating agent output the way we used to treat junior developer output, as something that needs a human to eyeball it before it's real. That made sense when code reviews were the primary quality gate. It makes no sense when they can't be.

The compiler analogy is clarifying here not because it tells us to trust agents blindly, but because it shows us what a mature pipeline looks like once you stop treating artifacts as things to be read and start treating them as things to be verified. We don't review compiled binaries. We run them against test suites, we check them against specifications, we instrument them in production. We moved the quality control upstream (type systems, linters, formal specs) and downstream (testing, monitoring, rollback), and eliminated the manual middle.

That's exactly the move we need to make with agents. We're not there yet. And the reason the lights-out framing makes engineers nervous isn't irrationality, it's that the upstream and downstream apparatus barely exists.

What's Actually Missing​

If compiler output requires no human review because of what surrounds it, then agent output will require no human review when we've built the equivalent. What does that actually mean?

Upstream: Compilers have type systems, formal contracts about what code means before it runs. For agents, this equivalent is still primitive. We have prompts, we have some scaffolding, but we don't yet have robust formal specifications that an agent is verifiably executing against. The closest analogs, TDD, contract testing, design-by-specification, exist but are not yet standard practice when agents are the author.

The verification layer: Compilers are deterministic. Given the same source, you get the same output. Agents are not. This means the test suite that used to catch human mistakes needs to be substantially more comprehensive when the author can produce plausible-but-subtly-wrong code at 50x the rate. The idea of using AI to check AI, pre- and post-action reviews, dedicated security agents, is the right direction. But it's nascent, and we're nowhere near treating it as infrastructure.

Downstream: The production monitoring and rollback culture that makes deploying compiled binaries safe is reasonably mature. Canary deploys, feature flags, observability tooling, this stuff works. Applying it rigorously to agent-generated changes is tractable. It just isn't yet habitual.

The Uncomfortable Part​

Here's the contrarian edge: the engineers most resistant to lights-out codebases are often the same engineers who would resist, if they could time-travel back, the idea that you don't need to review your linker's output. The intuition feels protective. It is actually just unfamiliarity with where the trust has been relocated.

Trusting compiler output isn't naïve. It's the result of decades of investment in making that trust warranted. We didn't skip the hard work, we did the hard work in a different place. The same logic applies here. The question isn't "should we trust agent output?" It's "have we built the infrastructure that makes that trust reasonable?"

Right now, mostly, we haven't. That's not an argument against moving toward lights-out codebases, we're being pushed there regardless. It's an argument for being honest about what "moving there responsibly" actually requires.

Hardware chip companies already work with black-box components verified by acceptance tests rather than human review. That's the model. But notice: chip verification is a discipline, with tooling, with formal methods, with teams of people whose entire job is designing the test harness. We are going to need the software equivalent, and it is probably going to be harder, because software has far less formal specification culture than hardware.

What To Build​

The lights-out codebase is coming. Su is right about that. But rather than simply accepting it as an inevitability to brace for, we should treat it as a design target, one that tells us exactly what we need to build:

• Formal specification layers that agents execute against, not just prompts
• Test infrastructure robust enough to substitute for the code comprehension that reviews provide
• AI-checks-AI pipelines as first-class CI infrastructure, not bolt-on curiosity
• Production instrumentation good enough that bad agent behavior is caught fast and rolled back faster

The compiler didn't make us stop caring about correctness. It moved where correctness is enforced. That's the project in front of us. The scary part isn't the lights-out codebase. The scary part is how few teams are treating what replaces the review as serious engineering work.

This post was written in response to Philip Su's "No More Code Reviews: Lights-Out Codebases Ahead"

How long have you been working at SkipLabs?​

I joined SkipLabs in November 2022, so it's been a little over two and a half years now.

In a few words, what do you do at SkipLabs?​

At SkipLabs, we develop tools focused on reactive programming. Our goal is to solve complex data synchronization problems while making this approach accessible and cost-effective in terms of infrastructure.

In practical terms, reactive programming is a declarative way of expressing computations: instead of manually handling state transitions, you simply describe state as a function of multiple inputs.

To make this approach tangible and usable on a daily basis, we created the Skip Framework—a reactive programming framework designed to be both powerful and easy to adopt.

How would you describe the engineering culture at SkipLabs?​

I’d say the engineering culture at SkipLabs is built first and foremost on trust and individual responsibility. Everyone pulls their weight and doesn’t hesitate to get their hands dirty when needed—even with tasks that may be a bit thankless. There’s a real collective drive to move things forward, and it shows in the general vibe: everyone is experienced, which allows us to work in parallel in a smooth and efficient way.

We operate in a fairly open and independent manner, with very few meetings. And despite this autonomy, there’s a lot of mutual support—whenever a question arises, someone is always there to help. Our expertise is quite diverse, which is a real strength. Several people on the team have a strong background in programming languages and a deep interest in low-level systems—especially compilers and LLVM, which sit at the heart of our stack.

On the tech side, the Skip Framework is currently exposed via a TypeScript API, and soon also via a native API in SkipLang. At the core of it all is a programming language, SkipLang, designed by Julien (previously at Facebook). The language was built to provide strong guarantees about data mutability—making it clear what can and cannot change, to enable safe and efficient cache invalidation. We use this language today. It has its own compiler, written in SkipLang itself, which targets LLVM as a backend. This gives us strong performance while maintaining high portability: the framework can be compiled natively or to WebAssembly, which makes it possible, for example, to run it directly in a browser. In addition to SkipLang and TypeScript, we also use some Python and Bash—classic tools to keep everything running smoothly.

So how do you approach scalability challenges?​

For me, scalability starts with the ability to accurately measure performance. Before even thinking about optimization, you need reliable metrics—and ideally, with minimal manual intervention. Manual handling quickly becomes a source of errors, especially in systems as complex as ours.

It’s crucial to measure what you actually want to observe—not just loosely correlated metrics. Once you have a solid measurement foundation, you can start large-scale analysis, deep profiling, pinpoint real bottlenecks, and invest where it really matters. Right now, we’re specifically working on load testing Skip to better understand how our system behaves under different loads, and to validate our horizontal scalability. It’s a progressive process, but essential to ensuring strong performance in varied environments.

And in your latest tests, what were the main bottlenecks you identified?​

One of the first bottlenecks we found was in the creation of streams. Once a stream is set up, performance is excellent, even with multiple users subscribing in parallel. But the initial creation phase can be a bit too slow. We’re currently working on a fix to improve that.

This kind of issue really shows how important fine-grained, automated tracing is—to avoid local changes negatively affecting other parts of the system. Working as a team also means writing code that helps—or at least doesn’t hinder—others’ work. This is why we’re implementing recurring load tests that run automatically in our CI. The goal is to detect performance regressions early and make sure different optimizations continue to work well together, even as the codebase evolves.

And in those load tests, what metrics do you prioritize?​

There are several, but the most important is: how many users can we serve under acceptable conditions for a given budget? We aim to stay close to real-world use cases. Depending on the services being tested, some loads involve heavy writing, others are mostly read-heavy. Some may involve lots of shared data, others significant server-side computation before results are pushed out.

One key metric is replication time: when a client writes data, how long does it take for that data to be visible to all other affected clients? This delay depends on the associated computation graph, so it can vary a lot—but it’s an essential metric for us. Once we’ve defined what counts as an acceptable client-side response time, we check how many clients we can serve with a single instance. Then we measure how that capacity scales when adding more servers—that’s our horizontal scalability curve.

And on the server side, what do you measure?​

There, we focus on how well we’re using available resources. For example: how many server instances can we run efficiently on a given machine? Are we fully utilizing RAM, CPU cores? There’s no point in paying for unused capacity. We complement that with regular profiling to ensure there are no avoidable bottlenecks in our server stack. It’s an ongoing effort, but it gives us real visibility into the system’s actual performance.

Beyond technical challenges, what do you see as the bigger issues in this field?​

I see three fundamental challenges:

• Correctness – We want clients to receive data that is both fresh and accurate, with mathematical guarantees that what they see is consistent with the system’s state. That’s non-negotiable.

• Performance – All of that has to happen as fast as possible, even at large scale. Offering low-latency reactivity is a bold promise—and a serious technical challenge.

• Developer ergonomics – This is often underestimated but just as crucial. Our goal is for developers to express business logic as declaratively and clearly as possible. Ideally, they focus only on what they want to produce, not how data will sync or update. It’s our job to ensure that works smoothly behind the scenes.

To achieve this, we need a sound and efficient computation model, solid implementation, and a clean codebase. That’s what we’re building with Skip: a technical foundation that lets you offload complexity without compromising rigor.

What do you enjoy most about your work right now?​

Benchmarking itself isn’t necessarily what excites me most—but right now it’s pretty thrilling, because we’re entering a new phase of the project. After months of prototyping and refining the core of the framework, we’re finally putting it through real-world load testing. Now we can truly measure how far this new paradigm can go under real conditions. It’s gratifying to see that what we’ve built—brick by brick—is starting to hold up at scale, and that the choices we’ve made are beginning to pay off.

Taking a step back, what would you like to see evolve in computing today?​

There are quite a few things I’d like to see change, but if I had to summarize, I’d highlight three key areas that matter most to me:

Valuing open source as a shared public good:​

I believe open source—or more broadly, digital commons—are still vastly underdeveloped relative to their importance. These are software foundations that benefit everyone and deserve more collective investment.

At SkipLabs, we work in open source under MIT license, and that’s one of the reasons I joined the team.

Rethinking how we interact with code:​

Today, as developers, our interaction with code is still quite “primitive”: we’re just editing letters on a page. Even though modern editors add powerful features, it’s still largely syntax-level work. I’d like us to move toward tools that support semantic interaction with code—like version control that understands the intent behind a change, not just a diff of characters. That would unlock deeper collaboration, better understanding, and smarter analysis tools. It’s a major area for innovation.

Mainstreaming cryptography in the digital public sphere:​

I think we should make broader use of asymmetric cryptography for things involving public speech, traceability, or authorship—without necessarily resorting to blockchain. Digital signatures alone are enough in many cases.

Right now, these tools are still largely inaccessible to the general public, which is a real problem. I think it’s crucial to defend a web architecture that respects individual freedoms—and that includes broad adoption of tools like encrypted email, and digital signatures, ideally within an open-source ecosystem.

Thanks, Lucas​

Thanks to you !

Here's the thing about cache invalidation: it sits right at the center of the classic performance versus consistency trade-off. Mess it up, and your users are stuck looking at stale data. Maybe they see yesterday's stock prices, or worse, they're making business decisions based on outdated metrics. But nail the consistency part while implementing it poorly, and you've just thrown away all the performance benefits that made you add caching in the first place. This fundamental tension has been driving innovation in caching strategies for decades, from simple Time-to-live (TTL) expiration to increasingly complex dependency tracking systems.

The Infrastructure Caching Challenge​

Modern applications cache everywhere, from CPU L1/L2 caches to CDN edge caches to application-level query caches, and each layer brings its own invalidation headaches that get exponentially worse across distributed systems. You've probably dealt with the classic cache invalidation strategies, and you know they all suck in their own special ways: TTL forces you to choose between stale data and cache misses, manual invalidation becomes impossible to maintain as dependencies grow, and event-driven invalidation couples your business logic to caching infrastructure.

Then there's the distributed systems reality check. Network partitions happen, clocks drift, and eventual consistency models mean that even your best-designed invalidation strategies can fail spectacularly. Your invalidation message gets delayed, lost, or arrives out of order, and suddenly half your cache nodes are serving stale data while the other half got the memo. These edge cases are the worst: they're nearly impossible to test properly and usually only surface when you're already under production load.

If you're running database-backed applications, you're dealing with multiple cache layers between your data store and your users. A single database update might need to invalidate entries in Redis, Memcached, your application's memory cache, and your CDN caches. Coordinating this invalidation across all these different systems while maintaining performance and consistency? That's the kind of engineering challenge that makes you question your career choices.

Ok, I may be a little dramatic here, but now that we agree on the problem we're talking about...

SkipLabs: A Reactive Approach to Cache Invalidation​

At SkipLabs, we take a different approach to the cache invalidation problem through reactive backend architecture. Instead of treating caching as a separate concern requiring manual coordination, SkipLabs builds reactivity right into the core of your data processing pipeline. When your underlying data changes, all dependent computations and cached results automatically update in a cascading fashion.

The reactive model that SkipLabs uses draws from functional reactive programming and incremental computation research. When data changes, the system automatically figures out which computations depend on that data and updates them incrementally. You don't need manual cache invalidation because the cache is always in sync with the underlying data through the reactive dependency graph.

SkipLabs pulls this off through a sophisticated runtime that tracks fine-grained dependencies between data and computations. Unlike traditional caching systems where invalidation is something you bolt on afterward, the SkipLabs runtime makes invalidation an intrinsic part of how computation works. When data changes, the runtime precisely identifies which cached computations are affected and updates them incrementally.

The platform handles the complex scenarios that make traditional caching systems cry: joins across multiple data sources, aggregations over massive datasets, deeply nested dependency chains. By modeling these relationships explicitly, SkipLabs can provide strong consistency guarantees while keeping the performance benefits of caching.

Pros and Cons of the Reactive Approach​

The reactive approach has some compelling advantages. First, you can finally stop writing manual invalidation logic. No more tracking complex dependency relationships or trying to remember which cache keys to invalidate when data changes. Your codebase becomes more maintainable, and you'll have fewer bugs related to stale data keeping you up at night.

Performance can be significantly better than traditional caching in many scenarios. Because the system understands your dependency graph, it can perform minimal incremental updates rather than nuking entire cache regions. For applications with complex data relationships, this means much more efficient cache utilization and faster response times when data changes.

The consistency guarantees are typically stronger than what you get with eventually consistent caching systems. Updates propagate automatically through the dependency graph, so there's less risk of serving stale data or having different parts of your application showing inconsistent views.

But let's be real about the trade-offs. The complexity doesn't disappear, it just moves from your application code into the runtime system. This can make debugging and performance tuning more challenging when things go wrong. You need to understand the reactive model and how your code will execute within it, which represents a significant learning curve for you and your team.

Converting existing code to the Skip runtime without properly structuring and decomposing its dependency structure can incur more resource overhead than it saves through incrementality. The benefits of reactive systems shine when computations are well-structured for incremental updates, but poorly designed dependency graphs can actually hurt performance compared to simpler caching strategies.

The reactive model also puts constraints on how you can structure your applications and access data. Not every existing codebase can be easily adapted to work within a reactive framework, you might need significant architectural changes to realize the benefits. To me, this is the hardest problem to solve.

Wrap-up​

Cache invalidation continues to be one of the most challenging problems in software engineering, with traditional approaches forcing difficult trade-offs between performance, consistency, and complexity. The reactive approach that platforms like SkipLabs are pioneering offers a promising alternative that could fundamentally change how we think about caching and data consistency.

Reactive systems aren't a silver bullet, they come with their own complexities and trade-offs. But they represent an important evolution in how we build data-intensive applications. As the technology matures and developer tooling improves, reactive backends may become viable for a much broader range of applications.

The key insight is that cache invalidation isn't just a technical problem to solve, it's a fundamental design challenge that shapes how we architect our systems. Whether you're using traditional invalidation strategies or exploring reactive approaches like SkipLabs, the goal remains the same: giving users fast, consistent access to data while maintaining system scalability and developer sanity. The reactive approach gives us a new tool for tackling this enduring challenge in computer science.

What's Next?​

I'd love to hear what aspects of cache invalidation and reactive systems you'd like me to dive deeper into. Are you interested in:

• Detailed performance results for a reactive approach?
• Deep technical implementation details of how SkipLabs tracks dependencies?
• Real-world case studies of teams migrating from traditional caching to reactive systems?
• Practical migration strategies for existing codebases?
• Other distributed systems challenges that reactive architectures could solve?

Drop me a line and let me know what you'd find most valuable, your feedback helps shape what I write about next.

The conventional solution? Event-driven architectures and streaming systems. But here's the problem: these approaches force developers to manually create and manage streams, handle async complexity, deal with event schemas, and debug distributed streaming pipelines. You end up spending more time on streaming infrastructure than on your actual business logic.

Think of a financial portfolio app: instead of calculating portfolio performance each time a user loads their dashboard, the server maintains continuous streams where position changes flow through pricing calculations, which flow through performance metrics, which flow through sector aggregations. Or an e-commerce site where product price changes automatically update product views, category aggregations, search indices, and recommendation scores in real-time. Rather than computing results on demand, reactive systems keep computational pipelines running continuously, so the answers are always up-to-date and ready for immediate use—reducing latency and spreading load over time instead of spiking under demand.

Precomputed Reactive Structures​

A reactive approach maintains continuous streams: position changes flow through pricing calculations, which flow through performance metrics, which flow through sector aggregations. The portfolio view stays current automatically, and user requests return instantly regardless of portfolio complexity.

And here's the key insight that changes everything: we should be able to treat complex computations as persistent data structures rather than transient operations. These structures exist in memory, stay synchronized with source data, and serve requests without additional computation. This is the fundamental principle behind Skip's approach, which we'll explore in detail later.

Cache Invalidation as Dependency Management​

Traditional caching introduces the notorious cache invalidation problem: when source data changes, systems must remember to invalidate all dependent cached values. This leads to either stale data or overly aggressive invalidation that negates caching benefits.

Reactive systems eliminate manual cache invalidation by treating cached values as materialized views with explicit dependencies. When source data changes, the system automatically pushes updates through the dependency graph to wherever they are needed—whether that's materialized views, API endpoints, or system edges. Cache invalidation becomes dependency propagation.

For example, when a product price changes in an e-commerce system, the reactive pipeline automatically updates product views, category aggregations, search indices, and recommendation scores. Hand-written reactive systems typically handle simple event propagation but require a lot of extra code written by developers in the backend to manage complex stateful views and aggregations. Skip enables this ideal: the developer declares dependencies once; the system maintains consistency thereafter through automatic dependency management of complex computational states.

How can Skip enable this ideal? Three words for you: granular incremental updates.

Granular Incremental Updates​

Skip reactive backends can perform granular, incremental updates rather than full recalculation when source data changes. If one product among thousands changes price, only the affected portions of aggregated views require updates. Category totals recalculate only for the affected category. Search indices update only relevant entries.

This granularity reduces computational overhead and memory churn. Systems can maintain complex materialized views that would be prohibitively expensive to recalculate entirely on each change.

Granular updates also improve system responsiveness. Instead of blocking while recalculating entire datasets, systems can process changes incrementally and continue serving requests throughout the update process.

I know what you're thinking: "This sounds great, but what's the catch?" Well, like most things in engineering, there are trade-offs to consider.

Memory-Computation Trade-offs​

We make a trade-off: memory for computation time and request latency. Maintaining precomputed structures requires more memory than computing results on demand. However, memory costs have decreased significantly relative to computation costs and user experience requirements.

The trade-off becomes particularly favorable for read-heavy workloads where the same complex computations serve multiple requests. Computing once and serving many times provides better resource utilization than recomputing identical results repeatedly. So we've covered the theory and the benefits, but what about the practical side? Well, implementing this isn't without its challenges.

Implementation Considerations​

Implementing reactive backends requires careful consideration of data flow patterns and dependency graphs. Systems must handle backpressure when computation cannot keep pace with data changes. They must also manage memory usage as materialized views grow with data volume.

Monitoring remains important for reactive systems, though the metrics of interest shift from traditional patterns. Traditional metrics like request latency remain relevant, but reactive systems require additional visibility into stream processing rates, backpressure events, and materialized view sizes to understand system health.

Skip's Approach to Reactive Systems​

What if there was a completely different way to tackle this complexity? What if we could get all the benefits of reactive systems without having to build or manage the streams ourselves? This is where Skip comes from.

Skip is a reactive framework that automatically creates and manages streams for you based on declarative computations. You write code as if time is frozen—defining what your outputs should look like based on your inputs at a single point in time—and Skip automatically turns this into live reactive streams that update clients in real-time. No manual stream wiring, no event handling complexity, no dependency tracking. Just declarative logic that automatically becomes reactive infrastructure.

Think of it this way: instead of programming reactive streams directly, you write functions that describe relationships between data. Skip automatically generates the optimal reactive streams to keep everything synchronized and delivers updates to clients seamlessly.

This approach fundamentally changes how we think about backend pressure because you're no longer computing results on demand or managing complex event flows. Skip handles the reactive infrastructure while you focus on the business logic relationships between your data.

Wrap up​

Reactive programming addresses backend pressure by shifting computation from request time to background processing. This approach enables instant response times, automatic cache consistency, and granular updates that scale with system complexity. The fundamental insight is treating complex computations as persistent, automatically maintained data structures rather than transient operations. This perspective changes how systems handle scaling, caching, and data consistency, often simplifying architecture while improving performance.

If you're dealing with read-heavy workloads where the same complex computations serve multiple requests, reactive approaches offer a compelling alternative to traditional request-response patterns. Start by identifying your most computationally expensive read operations—those are prime candidates for reactive transformation. Look for scenarios where you're repeatedly calculating the same results, dealing with complex cache invalidation logic, or struggling with response times under load. The challenge has always been the complexity of building and managing these reactive systems manually. Skip changes this by automatically handling the stream creation, dependency tracking, and incremental updates that make reactive backends practical. Instead of spending weeks building reactive infrastructure, you can focus on declaring the relationships between your data and let Skip generate the optimal streaming architecture.

The reactive paradigm isn't a silver bullet, but for systems facing mounting backend pressure from concurrent requests and complex computations, Skip provides a viable path toward managing computational load without sacrificing response times or data consistency. As your system complexity grows, the benefits of automatic dependency management and incremental updates become increasingly valuable. In our next post, we'll dive deep into load testing Skip-powered systems and share concrete performance data that demonstrates these benefits in practice. We'll explore how Skip's reactive architecture behaves under various load patterns and what metrics matter most when evaluating performance.

The technical foundation that makes this possible is Skip's hyper-efficient reactive computation graph, which is written in the Skip programming language, runs natively, and takes great pains to efficiently represent and manipulate the data underlying your reactive system.

However, that still requires memory, compute, and other resources -- so what do you do when traffic spikes or grows beyond the capacity of even a powerful single machine? Scale out and distribute your reactive service across more machines, of course!

We've recently built out some capabilities to make this easy, using a distributed leader/follower architecture to dramatically increase the number of concurrent resource instances that a Skip service can support.

Leader/Follower architecture​

A single Skip service consists of:

• a shared computation graph representing the portion of that service's computations that is common among all of its clients: some data structures, aggregations, partially-computed results, or the like that are always maintained up-to-date regardless of client requests, and

• one or more resources which can be instantiated by clients, dynamically extending the computation graph as needed to produce data streams customized by user-specific context or request parameters.

In practice, the work of maintaining the shared computation graph should not massively increase under spiking load, but the work of maintaining resource instances and serving their data streams will increase in proportion to the number of concurrent clients.

This dynamic affords an opportunity for horizontal scaling: we can maintain the shared computation graph just once on a leader and mirror it to each of any number of followers, among which resource instances are evenly distributed, as illustrated in the following diagram:

This diagram shows the structure of a distributed Skip service and the data flow for a single client request. When client A requests a live data stream, a reverse proxy forwards that to an available follower, selecting follower 2 in this example. That follower then sets up a reactive compute graph to maintain the requested data, incorporating user context and query parameters from client A as well as any shared inputs from the leader.

This design allows to instantiate many more resources for many more clients than a single Skip service could handle alone, while maintaining the clean/simple semantics and low latency of a single-service deployment.

To see this in action, you can pull our example, run it locally, and navigate to localhost in your browser:

npx create-skip-service hackernews --example hackernews --verbose
docker compose -f hackernews/compose.distributed.yml up --build

To see more options or run the application in alternative configurations, consult the README.md online or in the hackernews directory created by create-skip-service.

Kubernetes​

Running a distributed reactive service is a great way to handle larger amounts of traffic and/or more complex reactive computations, but what's really important is to be able to easily scale your reactive system up and down when your product goes viral, traffic spikes, and your pager goes off in the middle of the night.

We've recently built out some Kubernetes configuration to make this as easy as running kubectl scale --replicas=$X ... (or the GUI/dashboard equivalent if you prefer or are running on a hosted platform) without breaking any clients or requiring any changes in your reactive service or other backend components.

The core idea is simple: your reactive Skip service is a Kubernetes StatefulSet, giving each pod a stable and unique network identity. When a new pod is added (either at startup or when scaling up), it registers itself with the cluster's ingress load balancer.

When a resource is instantiated, the resulting data stream's identifier encodes the follower hosting the stream, allowing the load balancer to route external traffic to the proper Skip instance. When traffic subsides and the deployment scales down, the pod is taken out of rotation by the load balancer, until a subsequent scale-up brings it back into use.

Wrap-up​

Everyone's infrastructure and application are different, so let us know if there are other frameworks or tools you'd like to see supported or used in our examples and demos!

We're also happy to help you scale out your reactive service using Skip, either by adapting these tools to your environment or advising on your setup. Reach out and show us what you're building, or come join the Discord!

npx create-skip-service my-skip-chat --template with_react_vite

What's Inside the Template?​

Our React + Vite template for Skip Services comes with everything you need to get started on building a modern web application: an easy to manipulate end-to-end data flow from the user interface to the backend, and vice-versa (reactive programming enthusiasts, I have your attention now).

Getting Started​

To create a new project with our React + Vite template, simply run:

npx create-skip-service my-skip-chat --template with_react_vite --verbose

cd my-skip-chat   # enter the project directory
./setup.sh        # and set up the project

Setting up the project will end up with:

✅ Setup complete!

To run the application:
1. Start the reactive service:
   cd reactive_service && pnpm start

2. In a new terminal, start the frontend:
   cd frontend && pnpm dev

The frontend will be available at http://localhost:5173

This could hardly ever be simpler :-p

If you open the project directory​

In both subprojects, frontend and reactive_service, we've hardcoded some values, and taken shortcuts here and there in order to be straight to the point: making reactive programming easy to get, easy to grasp, easy to adopt, easy to embrace. easy to marry... ok, one step too far.

One or two cool things you will want to try:

• plugging into a DB from the backend, we can give you some help : Building a Real-time Blog with Skip and PostgreSQL
• having several conversations on screen, one per tab
• integrating your user-management system
• integrating an AI agent to discuss into the chat

We have a Discord server, why don't you swing by and say hi, ask a question or two?

What's Next?​

For the next Skip article, what should I tackle first? You tell me!

• Scaling your Skip service horizontally?
• Having an AI agent interacting in the chat ?
• Managing authorization and privacy per user?
• What else would be most useful to you now?

npx create-skip-service <project-name> <options>

What are we talking about?​

create-skip-service is a CLI tool that helps developers quickly set up new Skip service projects with customizable templates, saving the time you'd need to write boilerplate code. It's designed to follow best practices and provide a consistent starting point for all Skip service development.

Key Features​

• Quick Project Setup: A one-liner to have a fully working service
• Template System: Pick the right template for you, ask us for templates you need
• Git Integration: Automatic Git repository initialization (optional)
• Customizable: Once installed, templates are meant to be enhanced for YOUR project
• Verbose Logging: Detailed output for debugging and understanding the setup process, so you know what's going on

Getting Started​

KISS - here is the one-liner to get started:

npx create-skip-service my-service

Basic Usage​

Of course, we've kept in mind that not all projects are alike, so here are a few options to get what's right for you:

# Create a new service with default template
npx create-skip-service my-service

# Create a service with a postgres integration
npx create-skip-service my-service --template with_postgres

# Create a service without Git initialization
npx create-skip-service my-service --nogit

# Run with verbose logging
npx create-skip-service my-service --verbose

Available Options​

• --template, -t: Specify the template to use (default: "default")
• --nogit: To NOT initialize a Git repository, otherwise git is initialized with a first commit
• --verbose, -v: Run with verbose logging
• --quiet, -q: Run with minimal logging
• --force, -f: Force overwrite if directory exists

Why create-skip-service?​

We created this tool to address several common challenges in Skip service development:

1. Efficiency: Reduce the time spent on project setup and configuration
2. Standardization: Provide a unified way to create and manage Skip services
3. Flexibility: Support different templates and configurations while maintaining consistency

Contributing​

We welcome contributions to create-skip-service! Whether it's bug reports, feature requests, or code contributions, your input helps make this tool better for everyone.

• GitHub Repository: SkipLabs/create-skip-service
• Issues: GitHub Issues

What's Next?​

We're actively working on improving create-skip-service with plans for:

• Additional template options
• Enhanced configuration options
• More detailed documentation
• Community-driven template contributions

Try It Out!​

Give it a go! And keep us posted on what you are building. Feedback is bliss, right?

npx create-skip-service my-service

Wrapping Up​

create-skip-service represents our commitment to making Skip service development more efficient and enjoyable. By handling the boilerplate setup automagically¹, we believe this tool will help developers focus on what matters most - building great services.

As of today, we are already on our way to improving and augmenting the number of options and templates. Come back and help us sort priorities!

We have a Discord server, why don't you swing by and say hi?

What's Next?​

For the next Skip article, what should I tackle first? You tell me!

• Scaling your Skip service horizontally?
• Integrating with frontend frameworks like React?
• Managing authorization and privacy per user?
• What else would be most useful to you now?

Before we get too far ahead, two things…(1) what you need and (2) what you will learn:

What you need:​

• Node.js (Latest LTS version recommended)
• pnpm package manager
• Docker (for PostgreSQL)
• (optional) a Json pretty printer in the terminal, I use jq

What you will learn:​

• Real-time post streaming using Skip's reactive data system
• PostgreSQL integration for data persistence
• RESTful API endpoints for post management
• Support for streaming data access

Okay..ready?….let's go!​

We are starting from a REST server for the simplest blog post system and we are going to add real-time updates features to it. Let's check out the project:

git clone https://github.com/SkipLabs/postgres_and_skip_poc.git
cd postgres_and_skip_poc/
git checkout $(git rev-list --max-parents=0 main)

And yes, you are right we are not starting on top of the main branch, but don't worry we'll get there in no time.

Run the init_and_start_server.sh script, it will start a PostgreSQL database in a Docker container and fill it in with some data. It will then install and build the whole project with pnpm to finally start it!

./init_and_start_server.sh

The last lines should look something like:

🚀 Starting server...


> skip-postgres@1.0.0 start /your/path/skip-postgres
> node dist/index.js

Server running at http://localhost:3000

In a new terminal, we are going to check that the REST API is up: First, let's display one of the articles:

curl localhost:3000/posts/7 | jq

Note that here there's an easy shortcut - I use jq, which will make life more convenient to read JSON output.

As you can see, the article is not published:

  "published_at": null,
  "created_at": "<some_date>",
  "updated_at": "<some_date>"

We are going to publish and unpublish it:

curl -X PATCH http://localhost:3000/posts/7/publish | jq
curl -X PATCH http://localhost:3000/posts/7/unpublish | jq

You should see the modification as results of the calls.

A working REST API.

And now here comes the fun part, let's make it a reactive API

The Skip Service​

First of all, kill your running server. We are about to checkout the top of the main branch and install the necessary bits:

git checkout main
pnpm add -D @skipruntime/core @skipruntime/helpers @skipruntime/server
pnpm add @skip-adapter/postgres

A Skip service has collections as inputs and mappers to transform them into resources as outputs.

In our case, in file skipservice.ts you will find:

1. A type you want to be notified about: PostWithAuthor which contains information from two tables: Users and Posts (see schema.sql).

2. A mapper PostMapper which creates instances of this type from an EagerCollection (an EagerCollection is always kept up-to-date).

3. A resource PostResource which implements a function from a PostsResourceInputs to an EagerCollection<number, PostWithAuthor>

And it is when we instantiate our service that we plug it to the PostgreSQL, when we define an instance of SkipService, we receive a context as input. This context provides a method useExternalResource.

Remember that in Skip, Resources are the outputs of your service that clients can subscribe to, while Collections are the inputs that feed data into your service. When you create a graph using createGraph, you define how these Collections are transformed into Resources. Once you've set up your graph and connected it to your external services (like PostgreSQL in our case), you just need a few lines of code in your server to expose these Resources to clients:

app.get(
 '/streams/posts/:uid',
 asyncHandler(async (req, res) => {
   const uid = Number(req.params.uid);
   const uuid = await serviceBroker.getStreamUUID('posts', uid);
   res.redirect(301, `${SKIP_READ_URL}/v1/streams/${uuid}`);
 })
);


app.get(
 '/streams/posts',
 asyncHandler(async (req, res) => {
   const uuid = await serviceBroker.getStreamUUID('posts');
   res.redirect(301, `${SKIP_READ_URL}/v1/streams/${uuid}`);
 })
);

You can find these lines in src/index.ts#L65-L80

We are adding two endpoints:

• /streams/posts/:uid which is a stream of changes for a given post of id uid
• /streams/posts which is a stream of changes for the entire posts table

Let's play with it, in three terminals:

Terminal 1: Start the Server​

./init_and_start_server.sh

Server and services are ready when you see:

Skip control service listening on port 8081
Skip streaming service listening on port 8080
Server running at http://localhost:3000

Terminal 2: Subscribe to Post Updates​

curl -LN http://localhost:3000/streams/posts/7 | \
  while read -r line; do
    echo "$line" | grep '^data:' | sed 's/^data: //' | jq .
  done

💡 Note: The shell command above includes some formatting magic to make the output readable. It filters for data lines and pretty-prints the JSON using jq.

Terminal 3: Modify the Post​

curl -X PATCH http://localhost:3000/posts/7/publish | jq
curl -X PATCH http://localhost:3000/posts/7/unpublish | jq

Watch Terminal 2 while you run these commands in Terminal 3 - you'll see the real-time updates!

Wrap-up​

In our previous blog post, we have walked you through how to create a Skip service for a reactive social network, with code! Check it out here: Reactive Social Network Service (Proof of Concept).

Now we have learned how to plug a PostgreSQL service using Skip!

What's Next?​

For the next Skip article, what should I tackle first? You tell me!

• Scaling your Skip service horizontally?
• Integrating with frontend frameworks like React?
• Managing authorization and privacy per user?
• What else would be most useful to you NOW?

In this tutorial, we walk through building a small proof-of-concept social network backend using Skip and TypeScript. It tracks users, groups, and their friendships—with automatic updates to "active friends" using Skip’s reactive computation graph.

You’ll learn how to:

• Set up a reactive Skip service
• Define users and groups as live-updating resources
• Connect it all to a REST API using Express
• Observe real-time updates as the data evolves

"The simpler, the merrier" — this project keeps things minimal, focused, and easy to explore.

From Alice adding a new friend to live data reactions, this guide makes reactive systems tangible.

The Simpler, the Better​

Ever heard the saying, "The simpler, the better"? Well, that especially holds true when it comes to learning—particularly in reactive programming. Think of it like what we expect from a social network: reactivity in response to change. Sure, a full-blown social network has more bells and whistles than a carnival, but here we're just tackling a small piece of it. This guide keeps things light and breezy, focusing on the essentials to show how to build a reactive system using Skip. It's a proof of concept.

By keeping things simple and working right in the terminal, we can easily follow how data flows and how changes propagate without getting lost in complexity.

We'll start by setting up a Node project with TypeScript. Then, we'll define our data types. Since this is a social network, we want to represent friends and determine if they're active in a given group. We'll define members with names and associated groups, and for each group, we'll track whether a given member is active. After that, we'll build the Skip service, and finally, we'll wrap things up with a web server that uses the reactive service. All this will be wrapped up in an example where Alice makes a new friend.

You can follow along with the full code on GitHub, structured with a commit for each step:

Step 1: The TypeScript Project​

The whole first step can be found here on GitHub.

This first project really is Basic TypeScript 101, but watch out for some details such as the configuration of package.json and tsconfig.json. These two files can be sneaky little configuration traps if you're not careful. They might seem straightforward, but trust me, they can trip you up when you least expect it. So, keep an eye out and make sure everything is set up just right.

Let's create a directory for this proof-of-concept:

mkdir reactive_social_network_service
cd reactive_social_network_service

And now the TypeScript project:

npm init -y
npm install --save-dev typescript @types/node
mkdir -p src
npx tsc --init \
    --target ES2022 \
    --module nodenext \
    --rootDir "./src" \
    --outDir "./dist" \
    --moduleResolution nodenext

Configuration of package.json to prepare the project for what's about to come:

npm pkg set type="module"
npm pkg set scripts.build="tsc"
npm pkg set scripts.start="node dist/index.js"
npm pkg set scripts.clean="rm -rf dist node_modules"

We are now ready to enter the Skip world!

Step 2: The Skip Service​

The whole second step can be found here on GitHub.

We are going to need SkipLabs packages:

npm add @skiplabs/skip

So you better understand where we are heading, here is what the source directory will eventually look like:

src
    activefriends.mts
    data.mts
    index.ts
    skipservice.mts
    types.mts

And we start with the necessary types. In a nutshell, Skip defines a resource as the output of a Skip service. A resource is updated and maintained reactively. A collection is the core data structure over which reactive computations operate. An EagerCollection is always kept up-to-date. LazyCollection exists as well for evaluation upon queries only, but we are not using them in this proof-of-concept.

Here the code for src/types.mts:

import {
    type EagerCollection,
} from "@skipruntime/core";

type UserID = number;
type GroupID = number;
type User = { name: string; active?: boolean; friends: UserID[] };
type Group = { name: string; members: UserID[] };

type ServiceInputs = {
    users: EagerCollection<UserID, User>;
    groups: EagerCollection<GroupID, Group>;
};

type ResourceInputs = {
    users: EagerCollection<UserID, User>;
    activeMembers: EagerCollection<GroupID, UserID>;
};

export type {
    UserID,
    GroupID,
    User,
    Group,
    ServiceInputs,
    ResourceInputs,
};

Now some data to get started. Outside this proof-of-concept, this would come from your database.

Four users and two groups e.g. Carol is in both groups but inactive.

Here is the code for src/data.mts:

import { InitialData } from "@skipruntime/core";
import { ServiceInputs } from "./types.mjs";

// Initial data for the social network service
const initialData: InitialData<ServiceInputs> = {
    users: [
        [0, [{ name: "Bob", active: true, friends: [1, 2] }]],
        [1, [{ name: "Alice", active: true, friends: [0, 2] }]],
        [2, [{ name: "Carol", active: false, friends: [0, 1] }]],
        [3, [{ name: "Eve", active: true, friends: [] }]],
    ],
    groups: [
        [1001, [{ name: "Group 1", members: [1, 2, 3] }]],
        [1002, [{ name: "Group 2", members: [0, 2] }]],
    ],
};

export { initialData };

Let's talk about active friends! For that, we need to define a filter for that active boolean. We are going to use a mapper. In Skip, a mapper describes a computation from keys/values in one collection to keys/values in another collection. It forms the edges of the Skip reactive computation graph, specifying transformations and compositions of data to produce intermediate results and outputs.

We are defining two mappers and a resource: one mapper to retrieve active members out of a given group, one mapper to retrieve friends of a given user out of a given group, and a resource which will hold active friends for a given user.

Here is the code for src/activefriends.mts:

import {
    type EagerCollection,
    type Json,
    type Mapper,
    type Resource,
    type Values,
} from "@skipruntime/core";

import { GroupID, Group, UserID, User, ResourceInputs } from "./types.mjs";

// Mapper functions for reactive data transformations
class ActiveMembers implements Mapper<GroupID, Group, GroupID, UserID> {
    constructor(private users: EagerCollection<UserID, User>) { }

    // Maps group members to active users only
    mapEntry(gid: GroupID, group: Values<Group>): Iterable<[GroupID, UserID]> {
        return group
            .getUnique()
            .members.flatMap((uid: UserID) =>
                this.users.getUnique(uid).active ? [[gid, uid]] : [],
            );
    }
}

// Filters group members to only include friends of a specific user
class FilterFriends implements Mapper<GroupID, UserID, GroupID, UserID> {
    constructor(private readonly user: User) { }

    mapEntry(gid: GroupID, uids: Values<UserID>): Iterable<[GroupID, UserID]> {
        return uids
            .toArray()
            .flatMap((uid: UserID) => (this.user.friends.includes(uid) ? [[gid, uid]] : []));
    }
}

// Resource that provides active friends for a given user
class ActiveFriends implements Resource<ResourceInputs> {
    private readonly uid: UserID;

    constructor(params: Json) {
        if (typeof params != "number")
            throw new Error("Missing required number parameter 'uid'");
        this.uid = params;
    }

    // Creates a collection of active friends for the specified user
    instantiate(inputs: ResourceInputs): EagerCollection<GroupID, UserID> {
        const user = inputs.users.getUnique(this.uid);
        return inputs.activeMembers.map(FilterFriends, user);
    }
}

export { ActiveMembers, ActiveFriends };

And now the Skip service itself. It encompasses a server and a broker.

Here is the code for src/skipservice.mts:

import { runService } from "@skipruntime/server";
import { SkipServiceBroker } from "@skipruntime/helpers";

import { ResourceInputs, ServiceInputs } from "./types.mjs";
import { initialData } from "./data.mjs";
import { ActiveFriends, ActiveMembers } from "./activefriends.mjs";

// Service configuration and reactive graph definition
const service = {
    initialData,
    resources: { active_friends: ActiveFriends },
    // Creates the reactive data flow graph
    createGraph(input: ServiceInputs): ResourceInputs {
        const users = input.users;
        const activeMembers = input.groups.map(ActiveMembers, users);
        return { users, activeMembers };
    },
};

// Start the reactive service with specified ports
const server = await runService(service, {
    streaming_port: 8080,
    control_port: 8081,
});

// Initialize the service broker for client communication
const serviceBroker = new SkipServiceBroker({
    host: "localhost",
    control_port: 8081,
    streaming_port: 8080,
});

export { server, serviceBroker };

Step 3: The Server​

The whole first step can be found here on GitHub.

To use the Skip service, we are creating an Express server:

npm install express
npm install --save-dev @types/express

This server will expose an API to monitor active friends, and modify users and groups.

Here is the code for src/index.ts:

import express, { Request, Response } from "express";
import { server, serviceBroker } from "./skipservice.mjs";

// Initialize Express app
const app = express();
app.use(express.json());

// Store a reference to the Express server
const expressServer = app.listen(8082, () => {
    console.log(`Web server listening at port 8082`);
});

// Utility function to handle errors
const handleError = (res: Response, error: unknown) => {
    console.error("Error: ", error);
    res.status(500).json(error);
};

// Route handlers
const getActiveFriends = async (req: Request, res: Response) => {
    try {
        const uuid = await serviceBroker.getStreamUUID("active_friends", Number(req.params.uid));
        res.redirect(301, `http://localhost:8080/v1/streams/${uuid}`);
    } catch (error) {
        handleError(res, error);
    }
};

const updateEntity = async (entity: string, idParam: string, req: Request, res: Response) => {
    try {
        const id = Number(req.params[idParam]);
        await serviceBroker.update(entity, [[id, [req.body]]]);
        res.status(200).json({});
    } catch (error) {
        handleError(res, error);
    }
};

// Define routes
app.get("/active_friends/:uid", getActiveFriends);
app.put("/users/:uid", (req: Request, res: Response) => updateEntity("users", "uid", req, res));
app.put("/groups/:gid", (req: Request, res: Response) => updateEntity("groups", "gid", req, res));

// Graceful shutdown handler for:
// - SIGINT: Ctrl+C in terminal
// - SIGTERM: System termination requests (kill command, container orchestration, etc.)
["SIGTERM", "SIGINT"].forEach((sig) => process.on(sig, async () => {
    await server.close();
    expressServer.close(() => {
        console.log("\nServers shut down.");
    });
}));

Let's install, build and start it!

npm install
npm run build
npm run start

Starting the server will display something like:

> reactive_social_network_service@1.0.0 start
> node dist/index.js

Skip control service listening on port 8081
Skip streaming service listening on port 8080
Web server listening at port 8082

The Express server listens to the steaming service for updates, and it is now time see it in action!

Step 4: Let's See It Work​

We are going to use three terminals: one is already running the server and the service, one will issue writes on port 8081, and one will watch changes on port 8082.

While the service is running, in a fresh terminal, we are going to listen for changes on Alice's friends (index 1):

curl -LN http://localhost:8082/active_friends/1 # Alice's active friends

The previous command will continue listening after displaying the following, hence you won't get any prompt back.

event: init
id: <SOME_HASH>
data: [[1002,[0]]]

Leave it like that and open the third and last terminal in which we are going to add and remove values to the initial data: the list of Alice friends is about to evolve:

curl http://localhost:8081/v1/inputs/users \
    -X PATCH \
    --json '[[1, [
    {
        "name": "Alice",
        "active": true,
        "friends": [0, 2, 3]
        }
    ]]]'

What do you see in the second terminal? The data is being reactively updated. Go ahead and try other modifications e.g. Bob turns inactive, Alice removes Carol from her group of friends.

Wrapping Up​

Reactive programming doesn't have to be intimidating. With Skip, we've seen how simple it is to wire up reactivity—from raw data all the way to live updates in an API. The idea is simple: when something changes, everything that depends on it updates automatically. And that's exactly what we built.

Of course, this is just the beginning. You now have a foundation for reactive services, and from here, the possibilities are wide open. Want to plug it into a real-time dashboard? Hook it into a game? Build collaborative tools?

What's next?​

For the next Skip article, what should I tackle first? You tell me!

• Listening to live changes from a PostgreSQL database?
• Scaling your Skip service horizontally?
• Integrating with frontend frameworks like React?
• Managing authorization and privacy per user?
• What else would be useful to your projects?

We've exposed reactive programming in the nude, 'scuse my french. The reactive world can be vast and complex but SKIP will be your guide and now you've got a solid start.

How did we get here?​

One of the most powerful and consistent trends in software for the past decade has been the move from single stack to cloud native applications. Cloud native applications are inherently distributed and have become popular as developers are drawn to the convenience of containers and platform-as-a-service infrastructure.

The API-ification of important subsystems means today hardly anyone would consider implementing their own payments, shipping, SMS, chat, billing or shopping cart functionality. Instead they’ll lean on Stripe, Twilio, Shopify, Shippo, etc; accelerating time-to-value and further distributing the functions of the application.

In the last few years developers have moved quickly to incorporate more and diverse AI features into their applications, but AI models and application logic compute requirements are quite different from one another and therefore seldom run in the same stack, further distributing the application.

Some have called for a “back to basics” approach of a single stack or modular monolith, but given the list of external demands on modern application development, distributed is unlikely to go away. Instead it’s becoming a more common approach for application architectures, not less, especially for “successful” applications that need to modularize to support more customers, users and features.

10 years of bad advice​

One can sympathize with the desire for a “back to basics” approach to cloud-native applications. There are many “pros” to writing cloud-native, distributed applications but living with them has been miserable for developers and operators alike. “Cloud native” is effectively synonymous with “distributed,” and “distributed” is another way to say a network is parked in the middle of the application. This changes engineering and operations in many big and subtle ways. The most common and questionable advice to tackle these challenges has been “event-driven.”

Events are the biggest reason why engineering teams try to postpone distributed or microservice architectures for as long as possible. “Cloud native” and “microservices” have unwittingly become synonymous with “event-driven” and sullied their reputations in the process.

Events remain a useful technical mechanism to transmit state and data among application resources but they are a miserable experience for developers and operators alike. Events:

• Put application state and data in dozens of queues that make the application prone to failures, race conditions and incorrect results
• Force developers to think and program asynchronously which is more complex and time consuming than synchronous code
• Require developers know more than just their portion of the code and application domain: they also need to understand how to work with queues, caches and event schemas
• Make applications difficult to troubleshoot as there is no one place to understand the state and data for an application

Event-driven vs event-hidden​

The event-driven approach was necessary in 2020 but it isn’t any longer in 2025. Today’s modern abstractions ensure it’s unnecessary to make a faustian bargain between the flexibility & scale of distributed and the existential dread inspired by event-driven architectures.

There are three innovations that together allow us to engineer applications as event-driven but where developers and operators are fully abstracted away from the events themselves.

1. React paired with client state management frameworks like Redux, Xstate or Zustand create a simple, productive experience for frontend engineering. Frontend engineers are able to implement simple, performant UI components that can support complex application behaviors while outsourcing state and DOM management to the framework.

2. Durable execution systems like Temporal, Restate and DBOS ensure distributed backend application writes are always correct within and across services and network boundaries because each function call is durable by default. All of the convenient guarantees of an ACID transaction are injected into the everyday programming experience.

3. Reactive frameworks for incremental computation like Skip make it simple to implement distributed application reads from multiple sources and latencies to disparate clients and views. Developers can blend data from multiple REST and/or streaming sources but write their logic as simple map functions free from references to infrastructure or events. Because the services are reactive, performance and correctness concerns like cache invalidations are outsourced to the framework.

All three technologies share many of the same design principles. Each technology caters to a different area of the application (client changes, backend writes, backend reads) but they all:

• Provide a declarative developer experiences free from references to infrastructure
• Are event-driven “under the covers” but don’t leak these details to users
• Work asynchronously, supporting high levels of concurrency but still make the developer experience feel effectively synchronous
• Manage and in some cases enforce immutability of functions and/or data
• Handle application state as a first class concern
• Allow for modularity and incremental adoption - they don’t enforce “one pure approach” on an entire application
• Can run locally or globally
• Are open source

Collectively, these frameworks provide a substantial upgrade to applications across developer experience, feature velocity, correctness, efficiency and operational reliability. Software development returns to a simpler, low context model; developers do not need to learn the unique quirks of low level infrastructure or understand the overall application architecture just to build or enhance their part of the application.

New powers​

Burying events doesn’t just simplify engineering, it grants engineers with new powers in the process.

Transparency

Ironically, with event-hidden engineering, every behavior of the application is now more visible in an actionable way because events are held in lower layers of the stack that understand their semantics and context.

State Handling

Events are not the only thing that get hidden. By extension state handling, state changes and their implications for correctness are also buried in this new model. The new developer experience is just implementing as if forever at “init time.” All the progressions to get from 1 init time state to a future init time state are handled by the platforms powering the event-hidden architecture.

Replayability

Lastly, event-hidden architectures nearly always enable replayability. Keeping events and state in a separate, well abstracted place makes it possible to give developers and operations teams a general way to time travel through the application’s computation.

Conclusion​

In 2025 it turns out you can go home again: you can build feature-rich, scaled, distributed applications without having to interact with events, queues and the industry of patterns, frameworks and consultants that came with them.

Event-hidden will become the default web app architecture for the next 10 years.

While Skip reactive services compose naturally with each other, they must also coexist with other backend systems with non-reactive semantics and APIs.

This blog post describes some recent enhancements we've made to the Skip framework to make it easy to bridge the reactive/non-reactive interface for popular systems like PostgreSQL and Kafka, and shows how you can build similar integrations with other systems and APIs.

Postgres​

PostgreSQL is one of the most popular relational databases in use today, serving as a highly scalable and resilient source-of-truth for critical application data.

For incremental computation, it can be tricky to maintain an up-to-date view of an application's state over a relational data model. Different data sources add and modify values in an interconnected relational data model: you can either re-query and re-build the world periodically, or build some ad-hoc logic to propagate deltas.

Using Skip's Postgres adapter, developers can define some computation over their Postgres state and use the Skip framework to watch for changes in the relevant tables and recompute only the minimal affected outputs.

To use the Postgres addapter, when defining your Skip service, specify one (or more) databases in the externalServices field:

const service = {
  initialData: ...,
  resources: ...,
  createGraph: ...,
  externalServices: {
    postgres: new PostgresExternalService({ host, port, database, ... }),
    ...,
  },
}

Then, anywhere in that service, you can pull in a database table as a Skip collection by specifying a table name and a column to use as the key of the collection. For example, given a PostgreSQL table my_table with schema

CREATE TABLE my_table (id SERIAL PRIMARY KEY, value TEXT);

we can use the table as a collection myTable in a Skip reactive service as follows.

const myTable: EagerCollection<number, { id: number, value: string }> =
  context.useExternalResource({
    service: "postgres",
    identifier: "my_table",
    params: { key: { col: "id", type: "SERIAL" } },
  });

Under the hood, Skip will maintain this collection up-to-date by watching the database for changes to my_table; the collection myTable will update incrementally the same as any other Skip collection and can be used in reactive computations combining multiple tables or any other Skip collection.

Although the key column specified here is the primary key column id of the table, it need not be a primary key or even unique -- it can be convenient to use a foreign key or other column to group multiple rows per key in the resulting Skip collection.

A more-involved example of a reactive service with computations over several Postgres tables is available here.

Kafka​

Apache Kafka is another backend component with widespread adoption for high-throughput messaging and event streaming, often used to power real-time features.

Integrating a Kafka cluster with a Skip service is a natural way to build reactive logic on top of streaming data. As such, we provide an ExternalService adapter which handles the plumbing for you and abstracts Kafka message streams as Skip collections.

First, specify connection information and configuration in the externalServices field of your service:

const service = {
  initialData: ...,
  resources: ...,
  createGraph: ...,
  externalServices: {
    kafka: new KafkaExternalService({ clientId, brokers, ... }),
    ...,
  },
}

then, consume messages into a reactive collection with a usExternalResource call, e.g.

const myKafkaTopic: EagerCollection<string, string> =
  context.useExternalResource({
    service: "kafka",
    identifier: "my-kafka-topic",
    params: {},
  });

By default, messages are interpreted as their string key and value, but a "messageProcessor" can be provided to parse message payloads into other types, manipulate key structure, or pull in "topic" identifiers or other message metadata.

Other external APIs​

Of course, there are many other technologies and systems and you may need to integrate those with Skip. The simplest option for many non-reactive APIs is to periodically poll an HTTP endpoint; we provide a simple interface to specify a polled dependency on a non-reactive API:

const service = {
  initialData: ...
  resources: ...
  createGraph: ...
  externalServices: {
    myExternalService: new PolledExternalService({
      my_resource: {
        // HTTP endpoint
        url: "https://api.example.com/my_resource",
        // Polling interval, in milliseconds
        interval: 5000,
        // data processing into `Entry<K, V>[]` key/values structure
        conv: (data: Json) => Array.from(data, (v, k) => [k, [v]])
      }
    }),
  },
};

This PolledExternalService specifies a single endpoint polled every 5 seconds, but in general can include any number of endpoints with different polling intervals and converter/parser functions.

Of course, polling necessarily introduces some latency in the reactive service: data can become stale between polls. On the other hand, increasing the request frequency can place undue load on the target non-reactive API. As such, care should be taken when setting up a polled dependency to choose a reasonable interval, and where possible polling should be avoided in favor of genuinely reactive updates.

What's next?​

These three adapters (PostgresExternalService, KafkaExternalService, and PolledExternalService) are provided to make it easy to integrate with common backend systems, but all three are built using only public APIs and can easily be extended or tweaked in user code.

If your application would benefit from additional adapters for other components and technologies, you can implement an ExternalService to wrap non-reactive systems as inputs to your Skip service.

We welcome and support open-source contributions and feature requests and are always happy to answer questions or help out on Discord.

We’re pleased to announce that SkipLabs has raised $8 million to bring incremental computing to applications everywhere. We intend to use the entirety of this funding to enhance the open source Skip project and grow the developer community.

This financing also means we get to work with Lenny Pruss at Amplify Partners who have invested in some of today’s most prominent developer platforms such as dbt Labs, Prisma and Temporal to name a few and Alex Mackenzie at Tapestry VC. Check out Amplify’s blog post that shares their view on SkipLabs and the future of reactive systems and Tapestry’s blog post on why they chose to invest.

We are also humbled to earn the support of individual investors like Adam Gross, Spencer Kimball, Yann LeCun, Tom Occhino, Olivier Pomel and Nicolas Vasilache.

2025 so far…

We launched Skip at the start of this year and are very happy with the reception we’ve received so far. In the first two months since its alpha release, the Skip project has received more than 1,000 stars on github, more than 1,000 unique installations while our Discord community grew by 400%.

What has been most gratifying are the discussions working with users to apply Skip and incremental computing to making their backend applications fast, secure and easy to enhance. Developers have used Skip to power real-time crypto trading applications, CRM content management systems and sports tournament management.

We look forward to hundreds more of these collaborations because the opportunity for incremental computing is everywhere, it just wasn’t easy enough to do before Skip. Thanks to Skip, incremental computing will become the simplest way of building services that:

• Power real-time features
• Source data from more than one place with more than one latency
• Serve distinct views to diverse clients that are a combination of browser, mobile, agent and API
• Move security and privacy implementations from the client to the server

The Road Ahead

The Skip Framework is new, but it’s built on very mature technical foundations developed while we were at Meta. We plan to incorporate feedback from the community for a few months more before we ship a beta release so we can be confident in our backward compatibility commitments.

In the meantime we have lots of exciting new features to ship. In the past few weeks we’ve already added prebuilt external resources for Postgres and Kafka, a more efficient natively compiled runtime and native OAuth support.

In the months ahead we plan to add:

• Horizontal autoscaling
• A built-in debugger that builds on Skip’s transactional heap to provide unprecedented visibility at runtime
• Support for additional backend programming languages

Thank You

Our deepest gratitude to our users, investors, advisors and early employees who have believed in and supported SkipLabs’ mission of incremental computing since the earliest days. We’ll keep shipping and proving you were right all along. Back to work!

Facebook allowed us to open source both Hack and Skiplang so we could share both with the larger engineering community. I eventually decided to leave Facebook (now Meta) to build on the work of Hack and Skiplang and develop technologies that would address the same challenges we had at Facebook but for engineers generally. To do this, SkipLabs needed to incorporate two new technologies to make the work we did at Facebook usable in a normal engineering context.

1. A way to manage objects transactionally as their state changes

This is an essential and unique design choice in Skip’s approach to incremental computation. Without it, developers have to program against streams and assume a large and insidiously subtle amount of complexity. Incremental computation will only become mainstream if the dev and ops time experience is simpler and easier than the more common request/response paradigm, not just faster & continuous.

Object handling needed to go beyond simple caching. We needed an in-memory system of objects and collections that could maintain an up-to-date view of the data as it changed which we call **reactive collections**.

Most of the objects in a program can be managed with memoization: storing and reusing the results of expensive computations. This can be fast, simple and efficient provided you know when to evict the memoized values.

In Skip, memoized objects are held in “Lazy Collections.” They compute and cache results only when needed clients and will automatically evict data from the cache to save memory.

Other objects serve as indices or dictionaries. These objects realistically cannot be memoized as every value in the dictionary needs to be up to date at all times. Skip manages these objects as “Eager Collections,” keeping them in memory at all times and recomputing whenever a source updates.

2. An ability to program services that compute incrementally but in familiar languages.

Skiplang was built from the ground up to support incremental computing and it’s very well suited to that purpose. Realistically engineering teams outside of Meta have already picked their languages and stacks and it would be unrealistic to expect everyone to change to meet Skip.

We realized we could bring the most essential characteristics of Skiplang to already widely used languages, starting with TypeScript. Skip aims to allow developers to implement programs declaratively while the framework automatically manages the state and decides what parts of the graph of computation need to be re-run. For this to work, programs need to be deterministic, effect free and assume they are interacting with objects in a transactional heap.

The Skip Runtime integrates with popular TypeScript runtimes (e.g. Node, Bun) and using proxy objects gives TypeScript a reactive, transactional heap. We then added a TypeScript API that gives developers a simple programming model of map functions applied to the objects Skip is now holding in its transactional heap. In this way, developers get to stick with their preferred language while Skip can transparently manage both the state of the objects and directs when different parts of a program are recomputed.

We added higher level features to the API to more easily manipulate those collections (filter, slice, join, etc ...), and added support for sub-collections (collections created while computing the entry of a collection), among other things. In the future we think there’s an opportunity to add other (optional) higher level primitives to make building complex services even easier.

Today the Skip Framework exclusively supports TypeScript. We plan to add support for Python and Java in the near future and other languages in the medium term.

The Skip Framework

By combining a programming model that enforced effect-free operations on immutable objects with a runtime that abstracted away state and data management details, Skip has grown from a language to a full framework that we’ve recently released.

Why Skip is useful

Skip can power services that perform any combination of reads and writes but the main benefits are experienced with reads. Skip gives read-based features & functions several advantages over the more typical request/response model.

Skip programs will:

Continuously react to changes in data and clients - by managing the state of objects and the impact of functions on those objects, Skip services can incrementally recompute and stream new values to clients without recomputing unchanged values unnecessarily.

Automatically manage their own state - a feature may need to fetch data from multiple sources, compute intermediate results in a specific order while hydrating & invalidating different caches. In Skip all of this complexity is handled by the framework. Developers just write functions against collections for a single point in time.

Let you introspect their execution - reactive programs are historically challenging to inspect and debug. Because the Skip runtime manages both the objects and their graph of computation, Skip programs are very easy to introspect. We plan to add a debugger to the framework to capitalize on this part of the design.

Work alongside traditional request / response programs - Unlike past approaches to state management and reactivity, we wanted to make sure Skip was not an “all or nothing” engineering decision. Vertically, Skip can run in one or several layers of an existing software stack. It can read and write data from preexisting REST services, streaming sources or some legacy backend. Skip can also power a new service where it reads and writes from a traditional database.

Horizontally, Skip can power some or all the features of an application’s backend and coexists with other business logic written in other frameworks, languages, etc. Within Skip’s context it's very easy to extend what you’ve implemented in a service through a feature called “mirroring” that lets a new service import and synchronize data from a pre-existing peer service.

Why it feels awesome to use Skip

Skip lets developers focus on the "what" instead of the "how." By defining the desired outcome declaratively, Skip generates the necessary logic to keep data live and reactive. It handles complexities like cache invalidation and recomputation efficiently without adding any cognitive burden for the developer.

We took a graph-based approach to the Skip programming model. You define your computation as a series of operations on collections, creating a **graph of reactive computations**. Each node in the graph can be a lazy or eager collection, and the framework automatically manages dependencies, updates, and cache invalidation for you.

Here’s the magic: You write your program as if **time is frozen**, defining static outputs in terms of static inputs. Skip then turns this into a live, streaming system. If an input changes, the framework efficiently propagates updates to the affected parts of the graph, ensuring everything stays consistent. It even supports transactional updates, so you can modify multiple inputs simultaneously without breaking consistency.

This approach not only makes building incremental features easier but also scales better as the system grows. By focusing on what you want to build and letting Skip handle the rest, developers can deliver live, interactive features without the typical headaches.

Why Skip matters

Skip isn’t just about solving caching or problems of incremental computation, it’s about rethinking how we build systems that deal with constantly changing data. By giving developers tools to handle immutability, caching, and reactivity seamlessly, Skip makes it easier to write reliable, efficient, fast and scalable applications.

By combining the lessons learned from Facebook’s early challenges with the innovation of Skip, we’ve charted a path toward solving some of the most complex problems in real-time and reactive systems. This is just the beginning, and we’re excited to see where this journey takes us next.

I joined a much-smaller-than-today’s Facebook in 2011 to work on what would eventually become the developer tools team. At the time I joined, the engineering team was in the hundreds and the tools we had for development were, well, not great. PHP was the backbone of almost everything, with millions of lines of code. JavaScript was secondary, and only a few critical services ran on other languages. The company was scaling rapidly, and our development practices weren’t keeping up.

As the codebase grew, so did the difficulty of maintaining and improving it. Facebook became much more feature rich and simple enhancements became complicated and error prone. Often new features needed to operate conditionally on how other preexisting features were already operating. Essentially with each new layer of features teams added, state management became a greater concern.

Power of type systems

Facebook’s codebase relied heavily on dynamic constructs called “gatekeepers” that determined which features were live for which users under different conditions. But these gatekeepers could change which code paths were executed in ways that weren’t always obvious during development.

The inability to recreate the state of a path made testing and refactoring a nightmare. Imagine this: you tweak a method, and three weeks later, you’re woken up at 3 AM because your change broke something deep in the application. Over time developers stopped touching critical parts of the codebase. Instead of refactoring and improving existing code, they’d build parallel versions—hacks on top of hacks. The technical debt was piling up, and we needed a way to make changes safely and confidently.

We believed we could use types to make deeper static analysis and automatic refactoring possible in the PHP language and this would help engineers bring some order to the chaos. The system was called Hack. Adoption wasn’t immediate—there was some resistance—but over time, developers saw the value. We intentionally designed the type system to be more flexible than strictly “correct.” This decision allowed us to cover more of the codebase quickly, even if it meant making some trade-offs in precision. It worked. Hack became the default way to write PHP at Facebook, and it’s still in use today.

Incrementality

As Facebook applications grew denser and more layered, more teams were looking to add real-time features where users’ actions immediately impact others, like clicking a "Like" button and instantly updating the count across everyone’s apps. Such features are essential because performance and interactivity grew user engagement. When an application feels fresh and interactive, it’s much more appealing than static content. However, implementing this in a scalable way is not straightforward.

A typical scenario: you fetch a Facebook post. Normally, the server retrieves data like the post’s text, the number of likes, and the comments. It applies rules to filter and prioritize what’s shown (e.g., highlighting the most relevant comments or enforcing privacy settings). This logic runs once and produces a static result—the post you see on your screen.

Making this "live" means the post updates automatically. If someone likes it, the count changes instantly. If a comment gets edited or deleted, it reflects for everyone in real time. Achieving this requires "inverting the arrow." Traditionally, a user’s actions—like adding a comment— produces a write to the database. To make things reactive, engineers needed to turn these writes into continuous streams of updates. For example:

1. User likes a post → Create a stream of updates for that post’s like count. 2. User adds a comment → Create a stream for the comments section.

Typically reactive frameworks compose these streams into live updates. While conceptually straightforward, this approach fell short because:

Excessive recomputations: Each small change could require recalculating the entire post, which is expensive and inefficient.

Caching problems: To compute incrementally but avoid excessive fetching and recomputation, you need caches. But managing caches as objects are mutated becomes a nightmare—how do you ensure they’re up-to-date without introducing inconsistencies?

Caches can work well for "hot" data—frequently accessed data with a short lifespan. But outside this narrow use case, caching comes with trade-offs:

1. Long TTL (Time-to-Live): Keeps data in the cache longer, but risks showing stale or inconsistent data. 2. Manual Invalidation: Requires custom logic to invalidate caches when data changes. This is extremely error-prone and hard to maintain.

For large-scale systems like Facebook, neither approach was sustainable. We needed a better solution.

We created Skiplang to better serve our needs for better state management and real-time features & services. We believed we could use our experience developing type systems to flip the way we think about reactive systems. Instead of starting with database writes and building logic from there, Skiplang let you:

1. Define the Desired Output: Begin with what you want to build (e.g. a Facebook post). 2. Use Declarative Rules: Specify what data is needed and how it’s combined.

Skiplang was originally designed to handle incremental computation and caching, but its capabilities quickly expanded to address the needs of reactive features and efficient data handling. We built Skiplang on the premise that cached objects were immutable - unchangeable once created. If we could guarantee that immutability, it would be easier to use static analysis of Skiplang’s typed language to track dependencies and know exactly when to invalidate the cache and where to recompute. Traditional programming languages didn’t give us the tools to enforce these guarantees.

Skip gives TypeScript developers a simple declarative way to implement and run read-mostly features and services that are performant, transparent and continuously updated. The framework handles all the complexities of state management, integrations and failure handling in the process.

You can install via:

npm i @skiplabs/skip

This is an alpha release - we are happy with its stability and functionality but are looking forward to feedback on the interfaces and developer experience.

If you have time to check it out over the holidays and have feedback or just questions, come talk to us in discord.

Happy holidays,

The SkipLabs team