Whenever Factorio comes up on Hacker News, some software engineer inevitably complains "it's too much like my day to day work". Building things by hand, designing new architectures, blueprints and interfaces, the endless refactoring, coding and Factorio are very much the same. The complaint is that it's like a truck driver playing Truck Driver Simulator at night after driving the truck all day. It will just feel like too much of the same motions.

Well, that was before AI coding. Now the complaint of developers is "I just chat to Codex/Cursor/Claude and hit enter all the time, the craft is gone!". We no longer experience this coding, puzzling, refactoring loop that we really enjoyed pre Opus 4.6.

So lately, I've gone back to playing Factorio a couple of hours per month. Startup + family life doesn't allow for much more unfortunately.

Coding as it was is no longer economically viable, but playing Factorio was never about that. Also its game mechanics are probably better than those in your IDE ;)

you can discuss here but due to all the blogger spam, the thread on HN is better

This post was written by GPT-Sol 5.6 mimicking my writing style. After multiple 12h+ days of optimizing CI I could not muster the energy to write a full on blog post, but I did want to share our optimizations with the world. The contents are correct and it's based on our commit history and our codebase.

Ultimately it's just a list of things we did to optimize our Rust compilation, which is a problem that everyone that builds large Rust projects has. Our compilation jobs alone sometimes took 9 minutes. Now we reliably do it in just over 2 minutes. The tests then run for another 2.5.

Introduction

Comper currently contains 176,496 lines of Rust code across 975 files and 18 crates.

The test suite contains:

• 762 Rust unit tests
• 377 Rust integration tests
• 985 frontend unit and component tests
• 91 Playwright end-to-end tests
• 9 additional ignored Rust tests

Our merge request pipeline builds the frontend, runs the frontend tests, compiles every Rust test binary, runs three backend test shards, and runs two end-to-end test shards.

The complete pipeline now takes about 4 minutes and 40 seconds. It used to take around 20 minutes.

There was no single compiler flag that fixed it. The largest improvement came from running GitLab runners on our own Kubernetes cluster and deliberately using their local disks as persistent caches.

The biggest hack: node-local caches

GitLab normally restores a cache before a job and uploads it afterwards. This works well when a cache consists of a few large files. It works much less well for Rust directories containing tens of thousands of small files.

For every job GitLab had to:

1. Walk the cache directory
2. Package or compress it
3. Upload it
4. Download it in another job
5. Extract it again

At some point we were spending almost as much time moving Rust caches as compiling Rust.

Our GitLab runners run on a dedicated Kubernetes node pool. Every job pod mounts a directory from the local filesystem of its node:

[[runners]]
  request_concurrency = 4
  cache_dir = "/cache"

  [runners.kubernetes]
    namespace = "gitlab-runner"

    [runners.kubernetes.node_selector]
      nodepool = "gitlab-runners"

  [[runners.kubernetes.volumes.host_path]]
    name = "cache"
    mount_path = "/cache"
    host_path = "/var/lib/gitlab-runner/cache"

This is an anti-pattern. Kubernetes pods are supposed to be disposable and independent of the node on which they run. Our cache is neither.

We accepted this because the cache is only an optimization. When a node disappears, the next build starts cold but still works. Our jobs are scheduled on a small dedicated runner pool, so they usually return to a node that already has the data.

Cargo and kache use separate directories on this mounted disk:

variables:
  CARGO_HOME: "/cache/comper/cargo"
  KACHE_CACHE_DIR: "/cache/comper/kache"
  KACHE_MAX_SIZE: "100GiB"
  CARGO_INCREMENTAL: "0"

CARGO_HOME contains the registry and git dependencies. Kache contains compiler outputs.

We no longer cache the complete Cargo target/ directory. Kache gives us most of its useful contents without requiring GitLab to zip, transfer and unzip the whole directory.

Coverage is the exception. It still caches a limited set of Cargo fingerprints and dependencies.

Replacing sccache with kache

We originally used sccache. It worked, but its Rust cache keys included the compilation directory.

Every GitLab job gets a fresh checkout path. We also use lots of local Git worktrees. In both cases sccache only achieved about a 60% cross-directory hit rate.

We replaced it with kache, a new and mostly vibe-coded compiler cache. Kache's keys are independent of the checkout directory, so an artifact compiled in one worktree can be reused in another.

Our measurements were:

• Cross-worktree build: 84 seconds with sccache, 40 seconds with kache
• Warm CI compile: 59 seconds with sccache, 34 seconds with kache

CI wraps every rustc invocation with kache:

variables:
  RUSTC_WRAPPER: "kache"
  KACHE_S3_BUCKET: "comper-sccache"
  KACHE_S3_ENDPOINT: "https://s3.sbg.io.cloud.ovh.net"
  KACHE_S3_REGION: "sbg"
  KACHE_S3_PREFIX: "kache"
  KACHE_NAMESPACE: "comper"

The node-local directory is the fast path. S3 shares artifacts between nodes and keeps the cache available when a node is replaced.

Kache requires a daemon for remote cache access:

.ensure_kache: &ensure_kache
  - mkdir -p "${CARGO_HOME}" "${KACHE_CACHE_DIR}"
  - command -v kache
  - kache --version
  - kache daemon start

We also tried bulk-prefetching the S3 cache at the start of every job. It downloaded too much and was not reliably faster, so we removed it.

Incremental compilation is disabled in CI

Incremental compilation is great when a developer repeatedly changes the same checkout. CI jobs start with a clean Cargo target directory.

In that situation incremental compilation added 15 to 20% overhead, created larger artifacts and prevented kache from caching some compiler outputs.

We explicitly disable it:

variables:
  CARGO_INCREMENTAL: "0"

Locally we make the opposite choice. More on that later.

Compile all tests once

Originally every backend test shard compiled the Rust workspace. Each shard also restored and uploaded its own caches.

That multiplied several minutes of compilation and cache transfer by the number of shards.

Now one job compiles all test binaries and creates a nextest archive:

build-test-binaries:
  stage: test
  image: ${CI_DOCKER_IMAGE}
  needs: []
  variables:
    FF_USE_FASTZIP: "true"
    ARTIFACT_COMPRESSION_LEVEL: "fastest"
  script:
    - *ensure_kache
    - cargo build --profile test -p migration --bin migration-cli --locked
    - >-
      cargo-nextest nextest archive
      --workspace
      --cargo-profile test
      --profile ci
      --archive-file nextest-archive.tar.zst
  artifacts:
    paths:
      - nextest-archive.tar.zst

The archive contains:

• All Rust test binaries
• comper-server
• The migration CLI

This is the only large Rust artifact transferred between jobs.

We initially built comper-server separately before creating the nextest archive. That caused Cargo to compile the root crate twice because feature unification differed between the normal build and the test build.

Removing one innocent-looking cargo build command saved a complete compilation of the application.

Our nextest fork

Our integration tests make HTTP requests to a real Comper server. Starting a separate server for every test would give excellent isolation, but it would also make the suite extremely slow.

Starting one server outside nextest was not ideal either. Every test would have to wait for it, including tests that did not need it. We would also lose nextest's normal process management, output capture and test scheduling.

We built a server-wrapper feature in our nextest fork. A server wrapper belongs to a group of tests rather than one test. Nextest starts it lazily when the first matching test is ready, probes it until it is healthy, captures its output and shuts it down when the run finishes.

Our configuration looks like this:

experimental = ["server-wrappers"]

[scripts.server-wrapper.test-server]
command = ["./scripts/start_test_server.sh"]
probe = {
  url = "http://127.0.0.1:5150/_readiness",
  interval = "100ms",
  timeout = "500s"
}
capture-stdout = true
capture-stderr = true

[[profile.default.scripts]]
filter = "all()"
server-wrapper = "test-server"

One Comper server is started per CI shard and shared by the tests in that shard. The tests still run in parallel, which also exercises concurrency inside the application.

At the time we introduced it, more than 150 integration tests ran in about 40 seconds with one shared server. Running every test with its own server would have taken around 15 minutes.

There are trade-offs. A test can corrupt shared in-process state and affect another test. Our database and fixtures have to support concurrent tests. A crashed server also affects the rest of the shard.

For our suite this is a good trade-off. The shared server is part of the system under test, and concurrent requests are closer to how it runs in production.

We proposed the feature upstream in nextest discussion #3330: Feature request: sidecar / server wrapper for client/server tests. The discussion also covers alternatives such as creating an on-disk seed and starting a fresh service for each test, process lifecycle questions, and use cases with heavyweight servers that can take 90 seconds to start.

The maintainers were interested in accepting some form of the feature. Until its lifecycle and isolation contracts are worked out, we are happy to maintain the fork.

Run the backend tests without Cargo

The backend tests are divided between three GitLab jobs.

Those jobs download the archive and run their assigned partition:

back-end-tests:
  stage: test
  image: ${CI_DOCKER_IMAGE}
  parallel: 3
  needs:
    - job: build-test-binaries
      artifacts: true
  script:
    - mkdir -p "${NEXTEST_ARCHIVE_DIR}"
    - >-
      cargo-nextest nextest run
      --archive-file nextest-archive.tar.zst
      --extract-to "${NEXTEST_ARCHIVE_DIR}"
      --extract-overwrite
      --workspace-remap .
      --partition "count:${CI_NODE_INDEX}/${CI_NODE_TOTAL}"
      --profile ci
      --no-fail-fast

These jobs never invoke rustc.

They extract the binaries, start PostgreSQL, Keycloak and Mailhog, and run their part of the suite.

Reuse the binaries for end-to-end tests

Our Playwright jobs use the same precompiled server and migration CLI. There is no additional Rust build for E2E.

The browser suite runs in two GitLab shards. Each shard runs two Playwright workers:

const ciNodeIndex = process.env.CI_NODE_INDEX
const ciNodeTotal = process.env.CI_NODE_TOTAL

export default defineConfig({
    fullyParallel: true,
    retries: process.env.CI ? 1 : 0,
    workers: process.env.CI ? 2 : undefined,
    shard:
        ciNodeIndex && ciNodeTotal
            ? {
                  current: Number(ciNodeIndex),
                  total: Number(ciNodeTotal),
              }
            : undefined,
})

We also changed the fixtures to create one board per Playwright worker instead of one board per test. Creating and deleting a board took between 300 and 600 milliseconds. Repeating that for every test added up.

Navigation helpers use domcontentloaded where waiting for every image and asset provides no additional confidence.

Mold and the parallel rustc frontend

CI uses LLVM for code generation, mold for linking and rustc's parallel frontend:

variables:
  RUSTFLAGS: "-C link-arg=-fuse-ld=mold -Z threads=8"

The parallel frontend reduced a full compilation by about 25%.

Eight and sixteen threads performed about the same in our measurements. We use eight in CI to reduce peak memory when multiple jobs are running on the same node.

Mold only changes the linker. It does not change Rust's code-generation backend or reduce the quality of the generated code.

Linux and macOS development

Local development has different priorities. We care about the edit-compile-run loop more than runtime performance.

Linux uses wild and sixteen rustc frontend threads. Apple Silicon uses the platform linker and ten frontend threads:

[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=--ld-path=wild", "-Z", "threads=16"]

[target.aarch64-apple-darwin]
rustflags = ["-Zthreads=10"]

We tested multiple combinations with the compile cache disabled:

config                          full build   incremental
cranelift + wild                145s / 155s  8s
cranelift, no fast linker       161s         20s
LLVM + wild                     158s / 169s  8s
LLVM, no fast linker            182s         23s
LLVM + mold                     157s         8s
LLVM + wild + threads           121s         8s
LLVM + wild + threads, deps O3  285s         6s

The fast linker saves approximately 12 to 15 seconds on every rebuild. The parallel frontend mainly improves full builds.

We previously used Cranelift for local development. A year ago it made a large difference. With our current nightly toolchain, Cranelift and LLVM both produced an eight-second incremental result. Cranelift only saved about 8% on a full build.

We removed it.

Local development, CI and production now all use LLVM. The difference is in profiles, linkers and caching, not in code-generation quality.

Cache dependencies locally, keep our crates incremental

Wrapping every local rustc invocation with kache caused another problem.

Kache cannot cache incremental output, so it strips Cargo's incremental compilation flags. This meant every edit to one of our own crates paid for a full code-generation pass.

Kache's own exclude patterns did not solve this. Excluded crates still went through its passthrough path, which also removed the incremental flags.

Cargo only enables incremental compilation for workspace crates. Registry and git dependencies are not incremental. We use that distinction in a small dispatcher:

case "${CARGO_MANIFEST_DIR:-}" in
  */registry/src/*|*/git/checkouts/*) exec kache "$@" ;;
  *) exec "$@" ;;
esac

Dependencies go through kache. Our workspace crates go directly to rustc and keep incremental compilation.

This also allows a new worktree to reuse dependency artifacts compiled in another worktree without slowing down the normal editing loop.

A warm full local build measured around 112 seconds. A one-line edit measured around 19 seconds before the final linker changes and eight seconds for an edit isolated to comper-api afterwards.

Splitting one giant crate into 18 crates

We also split the application into 18 Rust crates.

This was by far the largest change.

The original root crate contained HTTP controllers, database access, background workers, LLM pipelines, authentication, repository analysis and runtime infrastructure. Everything depended on everything else.

The root crate went from approximately 54,600 lines to 34,300 and eventually to 5,700 lines. The workspace now contains crates such as:

• comper-api
• comper-db
• comper-domain
• comper-workers
• comper-runtime
• comper-auth
• comper-agent
• comper-repo-intelligence

This helped local compilation by limiting invalidation when changing code near the top of the dependency graph. It also gave rustc more independent compilation units that can be built in parallel.

It did not produce the spectacular clean-CI improvement we expected.

CI still builds the complete workspace. Changing comper-db, for example, invalidates most crates above it. Splitting a large crate does not help when all the resulting crates still need to be rebuilt.

The architectural improvement was enormous. The direct clean-build improvement was much smaller. Linkers, caching and removing duplicated builds produced clearer benchmark results than the monumental crate split.

We do not optimize dependencies in development

A commonly recommended Cargo configuration is:

[profile.dev.package."*"]
opt-level = 3

This makes dependencies run faster but also makes them much slower to compile.

In our benchmark it changed a cold build from 121 seconds to 285 seconds. The incremental build only changed from eight to six seconds.

Dependencies are not rebuilt when editing our own crates, so we would pay more than two additional minutes on a cold build for a two-second improvement that rarely matters.

Normal development therefore uses unoptimized dependencies. We have a separate release-profile task when we need the server itself to run quickly.

Production still gets full LLVM optimization

Production is built with Cargo's release profile:

build-release:
  script:
    - *ensure_kache
    - cargo build --release --bin comper-server --locked
    - cp target/release/comper-server ./comper-server
    - cargo build --release -p migration --bin migration-cli --locked
    - cp target/release/migration-cli ./migration-cli

This is a normal optimized LLVM release build. It uses mold for linking, but mold does not change LLVM's optimizations or the generated machine code before linking.

The development optimizations do not reduce production quality.

Smaller improvements

The large changes account for most of the reduction, but we removed smaller pieces of unnecessary work too:

• Only one pipeline runs per change. An MR does not also get a redundant branch pipeline.
• Backend jobs use rules: changes and can be skipped for frontend-only changes.
• Coverage only runs for tags.
• Merge request pre-commit checks only inspect changed files.
• The CI image contains Chromium instead of Chromium, Firefox and WebKit.
• The CI image is built with nix2container and split into approximately 80 reusable OCI layers.
• Jobs use a content-hashed CI image and do not wait for the image-building job.
• libeatmydata disables fsync inside CI containers.
• The Mermaid validation bundle is stored by content hash and reused.
• We transfer one nextest archive instead of the archive plus duplicate server binaries.
• Artifact compression uses the fastest setting because the compiled binaries do not compress enough to justify the CPU time.
• We removed Axum's debug_handler attributes from frequently changed API code. That saved another one or two seconds per check.
• The backend model drift check runs once rather than once per test shard.
• Playwright browsers are already present in the CI image, so E2E jobs do not download Chromium.

None of these changes reduced the pipeline from 20 minutes to less than five by itself. Together they removed a lot of waiting.

The resulting pipeline

The critical path is now:

1. Start jobs immediately with a prebuilt CI image.
2. Read Cargo dependencies and kache artifacts directly from the runner node.
3. Compile all Rust test binaries once using LLVM, mold and the parallel frontend.
4. Transfer one nextest archive.
5. Run three backend test shards without a compiler.
6. Run two Playwright shards using the same precompiled server.
7. Run frontend tests, linting and other checks in parallel.

The full merge request pipeline, including the browser tests, now finishes in approximately 4 minutes and 40 seconds.

The biggest improvement did not come from making Rust dramatically faster. It came from compiling things once, keeping caches close to the CPU, and no longer transferring the same thousands of files between every job.

Legal implications

What follows is a bit of a legal quagmire. Easee would not be able to attack IBIS, and neither could IBIS attack easee, because both had valid patents for the same invention. Who would be able to expand into other regions though? And could one of the parties license their patent to other entities? So many questions.

The invention

We were trying to figure out how to estimate the distance of the user to the screen at which they performed our vision test. We relied on heel-to-toe steps and the shoe size, but this was a bit clunky. So I came up with a better method using the blind spot in your vision.

The ingenious parts of this invention are:

• Pre-calibrating the screen with a physical measure (like a plastic bank / ID card)

• Blind spot distance check

• the angle between the eye's blind spot vs center-of-vision (fovea) is very uniform in the population

• you can do a blind spot check (click when the circle in the screen disappears) on a pre-calibrated screen to determine the exact distance to the screen

• Then use the webcam to determine the pupillary distance (the distance between the two pupils) and track that to measure the distance to the screen while taking the rest of the vision test

This magical combination is identical in both patents.

Inventing this required a couple of weeks of deep thinking and doing research on my side, and I'm still pretty proud of it. We did not have any communication with IBIS so no clue what the process for invention looked like on their side.

Better execution

Just a couple of days ago the US patent in the family was granted as US20260151029A1. I'm still the inventor but hold no rights of course. This event triggered this blog post, which had been on my to-do list for quite some time. Since this invention a lot happened. I quit my job as CTO of easee back in 2023, and in 2025 easee was bought by its main competitor in the USA - Visibly. This was after years of fighting each other via their respective patent portfolios. Notably the landmark case of Visibly vs Easee in 2020 in which Bird & Bird and Taylor Wessing went head to head, and which I remember from being on vacation in August 2020 in the south of France, and taking calls with very expensive lawyers from my tent. Fun times ;)

It's clear that easee (and now Visibly) executed their patent strategy way better. Where easee has expanded this family from NL into WO, EP, and now the US, IBIS is stuck with their GB patent. Yves was very strategic from the start. Invention is one, but executing on your strategy is another thing, as this case shows.

Now I'm not a patent attorney and have not studied this field, but I do know that this case makes for a rare legal anomaly. We can only wonder what would have happened if IBIS decided to execute international expansion in parallel with easee, and both would have hit all their filing deadlines.

Welcome, everyone. Today, we’re going to talk about "dark code," the value of system visibility, and what a classic real-time strategy game can teach us about managing modern software organizations. We’re living in one of the wildest eras of technology shifting under our feet, and as technology leaders, we need a new mental model to survive it.

To give you a bit of context, I'm 38 years old and have spent my career navigating different software paradigms. I spent seven years at Mendix, a low-code platform pioneer, which started in the Netherlands and then moved it's HQ to Boston. After that, I became the CTO of an online eye-testing medical device company in Amsterdam, where I learned a massive amount about regulations, building teams, and writing patents. I then spent three years working as a fractional CTO across 8 different companies, and today, I am the co-founder and CTO of Comper, which, if you look at my career timeline (7, 5, 3 years), will likely only be around for 1 year ;)

Jokes aside, we’re building a company to solve the very visibility issues I faced at every single company I was at before. At Comper, we make maps for software systems. It’s a simple philosophy: if you are lost, the very first thing you want to get your hands on is a map so you know where you’re going. We are trying to bring that exact utility to software.

We started the company in October, and things are moving incredibly fast. We are currently a team of seven full-time people, plus a few part-time contributors.

Our product provides a collaborative digital canvas that ingests your Git data to visually surface various aspects of your code landscape. You can zoom out to see the macro architecture or zoom all the way down to individual code details, allowing both human teams and AI agents to explore and explain code systems together.

Before diving into the core analogy, I want to pose a couple of questions. These aren't for immediate answers, but we'll come back to them later:

• What organizational issues have you seen stem from people simply not understanding how the technology in their own company works?
• How are you dealing with the enormous, unprecedented volume of code being created today via AI?
• What do you wish you had deep insights into that currently stands as a blind spot for you; and what do you wish other business leaders understood about your stack?

If you’ve ever played Age of Empires, you know it's a game entirely driven by a mechanic called the Fog of War.

When the game starts, the entire map is pitch black, it is completely unknown. When you send a scout out on a horse, you can suddenly see exactly what that unit sees in real-time. But as soon as your scout moves away, that territory relapses into a state of gray fog. You can still see the permanent outline of the terrain and buildings, but you have no idea if enemy units are moving through it. To maintain visibility, you have to strategically build watchtowers and position outposts to ensure the enemy cannot maneuver in the dark.

The absolute best Age of Empires players master this mechanic to dominate the game. They possess distinct traits:

• Extreme Velocity: They move so fast that they hit you before you even realize they're coming.
• Proactive Control: They dictate the pace of the game and take the fight to you; they are virtually impossible to surprise because they see threats miles away.
• Resource Efficiency: If you look at their resource graphs at the end of a match, they don't sit on vast, unspent hoards of gold or wood. They gather exactly what they need and build an army that is just good enough to win.

As CTOs and VPs of Engineering, our CEOs always demand that we build things good, fast, and cheap. We usually tell them to pick two. Yet, top strategy players manage to achieve all three simultaneously.

Let’s look at three perspectives on how we can apply these lessons to our software organizations.

In the grand scheme, the game map is the broader market, and you are the player. Your army is your engineering team, your resources are capital (raised via investments or generated through sales), and your tech tree is your software stack.

To win against the world, the best companies optimize their internal feedback loops. They eliminate friction ruthlessly, ensuring pull requests are merged rapidly and customer feedback is captured instantly. In game terms, they place their villagers directly next to the gold mine to accelerate the economic engine.

They stay in control by investing heavily in research and scouting. They don't just build blindly; they set up watchtowers to evaluate shifting technology. This is incredibly critical right now, as the tech industry has never been more in flux due to AI. Furthermore, they maintain strategic reserves of time and money to absorb sudden market disturbances. Finally, they build cheaply by validating and selling a concept before over-engineering a massive full-feature product when a simpler solution would have sufficed.

Interestingly, the top companies also use the Fog of War to their advantage outside their walls. Keeping your internal roadmaps and strategic pivots hidden from competitors and vendors provides immense bargaining power.

While the Fog of War is a useful defensive tool against the outside world, it is an absolute disaster when it exists inside your own company.

Whenever I’ve stepped into a new company as a fractional CTO, I've felt like I was navigating in the dark with nothing but a tiny torch. You can see a small sliver of code here, a tiny team interaction there, and you slowly try to construct a holistic mental model.

In a massive, complex enterprise, it takes an incredibly long time to piece that puzzle together.

The real danger occurs when different leaders operate on completely conflicting mental models of the same technology stack. I've witnessed immense organizational drama because of this misalignment:

• The Oblivious CEO: I once worked with a CEO who spent five years funding a single-tenant, gig-economy application. He suddenly decided he wanted to instantly pivot it into a multi-tenant, white-labeled platform, genuinely believing it would be an effortless switch. He had zero grasp of the profound architectural implications.
• The Blind CPO: I saw a Chief Product Officer make a disastrous decision to merge two entirely separate platforms. One stack was built on C# and Azure; the other was Java on AWS. He assumed that simply shifting engineers between teams would allow them to blend the platforms seamlessly. The engineering teams knew it would fail due to wildly disparate APIs, stacks, and cultures, but leadership pushed ahead blindly.
• The Reckless VP: I watched an incoming VP of Engineering immediately lay off a handful of engineers solely to make a visible corporate impact, entirely unaware that he had just fired the explicit linchpins holding the core infrastructure together.

We now went into a roundtable discussion on this topic, an audience member noted that a massive portion of a technology leader's role is balancing tech clarity for the C-suite without causing their eyes to glaze over. Another participant pointed out that architectural analogies are our best defense. Explaining that you wouldn't try to build ten concrete floors on top of a single-family residential house because it would obviously implode.

Another leader shared the acute friction of running a nimble, greenfield AI startup division inside a larger scale-up. Business executives frequently point to tiny, three-person external competitors and demand to know why an enterprise engineering organization supporting a ten-year-old codebase can't match that exact turnaround speed.

So my conclusion is that this is how the best technology organizations work. They have systems for onboarding and documentation, and measure and maintain it very well.

The third perspective centers on how we manage AI agents.

A couple of months ago, this blogpost did the rounds. The "five" or actually six levels of agentic engineering. It takes inspiration from the car industry where there are also distinct levels of autonomy:

• Level 1: Spicy Autocomplete — The initial wave of IDE code completion assistants we grew used to a few years ago.
• Level 2: The Coding Intern — Delegating minor isolated tasks, such as asking an AI to rewrite a specific function.
• Level 3: The Junior Developer — Handing off simple, end-to-end tasks and reviewing the output later.
• Level 4: The Developer — Treating the AI as a functional team member that autonomously investigates an issue and submits a pull request for human verification.
• Level 5: The Agentic Team — Deploying multiple sub-agents that collaborate uninterrupted for 12 hours or more on complex objectives, updating documentation, and fixing code across boundaries.
• Level 6: The Dark Software Factory — An operating model where code is autonomously generated, compiled, and deployed without a human ever reading or reviewing it.

The clear litmus test for moving into the Dark Software Factory is token expenditure; companies operating at this level burn $1,000 a day in LLM tokens per engineer. Very few companies are there, I am certainly not.

This brings us to the core definition of Dark Code: software that no human has ever written, read, or reviewed. While this is accelerating rapidly due to AI, dark code also exists in legacy systems. Many enterprises rely on ancient mainframe software where the original human authors left the company decades ago, leaving a grey box that nobody truly understands.

The terrifying reality of agentic velocity is that our human management tools have failed to keep pace.

Consider this real-world example from our own team: our designer used an AI agent to adjust frontend layouts and test the visual output directly on screen. Once satisfied, the agent generated a single pull request spanning 211 files and over 7,000 lines of code.

When Git platforms present this change as a standard, flat wall of code text, a comprehensive human line-by-line review becomes literally impossible. In practice, engineers look at the green checkboxes, see that the automated tests passed, verify the UI looks fine, and hit merge. We are losing granular visibility of what is happening under the hood. For a hardcore software engineer who prides themselves on elegant architecture, this reality is deeply uncomfortable, even if the resulting development speed provides an undeniable dopamine hit.

So again, in the agentic world, we can learn a couple of things from AoE. Make sure your codebases are top-notch: CI/CD, linters, quick reviews. Keep docs up to-date automatically. Explore new solutions. Make sure tech debt is tracked, and keep your context windows up to date.

During our discussion, an audience member posed a fundamental question: "If AI is generating and parsing code at this velocity, is traditional, human-consumable documentation even relevant anymore?" One participant revealed that their organization completely abandoned writing standard internal readmes for humans. Instead, they write "meta-documentation" specifically formatted for AI consumption; explicitly mapping out service boundaries, architectural rules, and information flows so agents don't waste millions of tokens rediscovering the environment. To combat the sloppy code duplication that agents naturally introduce when working in isolation, they run nightly cleanup agents to ruthlessly refactor and prune the codebase.

However, as another roundtable member argued, "epistemic debt"—the risk of completely losing the human understanding of how a system functions, is incredibly dangerous. We have already seen major cloud outages extend for hours simply because the on-call human engineers didn't understand the underlying automation well enough to fix it when it broke. Senior developers must maintain high-level conceptual maps of the stack, even if they cede line-by-line execution to the machines.

We have arrived at a fascinating, paradoxical crossroads: as a society, we have never relied more on software, yet we have also never understood it less. To understand this transition, we can look back at early 20th-century manufacturing.

When electrical power first became available to industrial factories, managers didn't immediately change how they built plants. For the first 20 to 30 years, factories kept the exact layout of the old steam-engine era. They maintained a single, massive central drive shaft running through the ceiling of the factory, powered by a massive central motor, and mechanically hooked individual electric tools to that turning shaft.

It took decades for industrialists to realize the true paradigm shift of electricity: you could entirely eliminate the complex central shaft, run flexible wires through the grid, and place a tiny, independent electromotor inside every single machine.

Right now, we are in that exact same awkward middle phase with AI engineering. We are using incredibly advanced agentic capabilities, but we are still forcing them to work within the legacy "central drive shaft" paradigms of flat file repositories and line-by-line text pull requests. We desperately need entirely new visual and architectural tools to manage this paradigm. Fortunately, unlike Age of Empires, the rules of the real world aren't fixed. We have the agency to invent the new tools ourselves.

This brings us back to the map. To navigate this sea of dark code, tech leaders need an altitude-adjusted view of their infrastructure.

When Comper connects to your GitHub or GitLab accounts, it parses the entire system into a live, multiplayer canvas. Instead of cloning fifty separate repositories and running throwaway terminal commands to understand an ecosystem, teams can look at a unified landscape.

We allow you to toggle various data overlays instantly:

• Code Complexity & Health: Highlighting highly complex or problematic areas in red so you can visualize structural risk

• The Bus Factor: Color-coding the canvas based on authorship so you can instantly see knowledge silos. For example, you can filter by a specific engineer to see exactly which components they've touched over the past year.

• Code Velocity & Age: Shading components based on how recently they were modified, providing an instant heatmap of where active engineering hours are being spent.

We also auto generate documentation and keep it up to date when the codebase changes.

This was the end of the talk, we then had another 20 minutes or so of discussion.

When an audience member asked how to translate these visual "red zones" into language a CEO actually cares about, the answer came down to product roadmapping. If we map explicit software features directly onto this canvas as first-class citizens, a CTO can tangibly show business leadership: "You want to build feature X. To do that, we have to cut directly through this highly complex, technical-debt-heavy red zone. This is exactly why it will take 20% longer and cost more money." Visualizing true cross-application dependencies gives engineering leaders the exact empirical backing they need to justify architectural refactoring to boards and executives.

When you begin measuring lines of code or raw engineering hours directly, the metrics inevitably break down under Goodhart's Law, especially when AI can generate thousands of lines in seconds. But by tracking feature cycle times across a visual system map, we can accurately measure true organizational efficiency.

To leave you with a final bit of irony: when we originally started building Comper, we built a mini-game inside it inspired by Geoguessr. The tool would show an engineer a random snippet of code from their company's repository, and the engineer had to guess where on the corporate software map it belonged. It was an amazing, fun way to test codebase literacy.

But as AI adoption skyrockets and engineering teams produce vast oceans of code they've never personally read, we've noticed something funny: the game has become completely impossible to play. Nobody knows the literal text anymore. The era of dark code is officially here, and it’s time we built the maps to navigate it.

Thank you very much for your time.

Dark Code: lines of software that no human has written, read or even reviewed.

Two weeks ago I presented a talk "Mapping out Dark Code" at the OpenAI Codex meetup in Amsterdam, and now people start conversations with me and mention Dark Code. So before this goes out of control, let me write some thoughts.

Tldr:

1. Will my codebase become an unholy mess if no one looks at the code?: Yes, this is an unsolved problem.
2. Will there be new programming languages if humans don't care about code anymore?: Doubtful, and that's a shame!
3. Dark Code is already here: In enterprises no one knows the code anyways.

1 - Will my codebase become an unholy mess if no one looks at the code?

Agents take prompts and implement this in your codebase. I see parallels with evolution chipping away at our DNA ACGT base pairs. Evolution is just trying something randomly until it works. Likewise, in software, an agent will just find the easiest way to achieve its goal.

If the system is big enough, a random request in a prompt will not trigger a full refactoring or a new architecture. I feel architecture will get stuck in local maximums quite quickly.

Yes, without new tooling for architecture, backpressure on the agents when complexity goes too high, your codebase will become an unholy mess. However, there is a lot of development in this field now, so some things will emerge.

2 - Will there be new programming languages if humans don't care about code anymore?

The SWE Bench performance on new or unknown languages by LLMs is abysmal. I saw an article on this recently but can't find it anymore. There's no or little training data on these new languages, so the LLMs have to use their reasoning and can't rely on "muscle memory", to put it in layman's terms.

What does this mean for programming language research?

A) All coding agents are generating ever more code in current, general purpose, languages, and it's a bit of a vicious cycle. The more code is generated, the more models will be trained on that. It's hard to get good model performance on a new language, and thus it's harder to generate enough code to be trained on.

B) Previously humans cared about their code. When they saw a way to create a better language, some people were crazy enough to create a new one! Because they cared so much about the software systems they built. No wonder, they were living in their codebases all day. Now that we let most code be generated by our agents, do we still care enough to create a new language? I don't think so.

But, as I've written in the future of programming systems I believe there's a LOT of room left for better programming languages. I believe the Low-Code movement was partially about this. Standardizing what we were trying to build.

A language / linter / syntax should know exactly where your application talks to the outside world. Where there is Personally Identifiable Information flowing through your systems. Instead, all major current languages are general purpose languages, and unconstrained. Unconstrained sounds good, but actually, constraints allow you to speed up tremendously.

Will we ever get nice things, or are we casting the current status quo in stone. Will Rust be the best language forever?

3 - Dark Code is already here

Dark Code is not that binary. Code I've written 1 year ago is already a lot darker than the code I'm writing today. Enterprises are full of code written by teams that are no longer there. That code is pretty dark.

My startup is trying to shed a light on large software systems that are too complex to understand, unless you take a birds eye view. I'm a bit scared that it might not be relevant in the world of Dark Code, because people don't seem to care about code anymore. On the other hand, there is now more code than ever, so I'm betting that people will want to work on this higher level. Maybe we're building the IDE of the future?

Thanks for reading. If you liked this post, you might also enjoy my thoughts on the future of programming systems.

Discuss on HN, comments on the blog are mostly spam.

It was a lot of fun and the audience seemed to like it. My CMO gave me a 7-8 out of 10.

This is not an exact representation of the talk, I improvised, went off-script and forgot things. Also the organizers told me to say the f-word less (4 is bad, 1 would be OK), so in this transcript I've redacted it. At least if you say the f-word, you're quite sure it's not AI-slop ;)

Hi, I'm Jouke and today I'm going to talk to you about Mapping out Dark Code. This is the very first Codex meetup here in Amsterdam, this is the very first presentation of the evening and it's also a first time we're showing Comper to a real-world audience. Lots of firsts. We're happy to be here!

[the sentences appeared one by one, I do that all the time to make sure people follow my train of thought, and don't try to read everything at once]

So a bit about me, not to brag, but so you have something to related to. "Affordances" is what they call this in design language.

I'm now 38 years old. [Someone in the audience: I can relate to that]

I started at Mendix as an engineer, then became CTO here in Amsterdam for a tiny startup in online eye testing. Then I left to become a Fractional CTO for 3 years. Now I'm co-founder of Comper. If you follow the sequence, I'll be doing this for just one year ;)

Today I want to talk about how I think the world will change when intelligence is for rent. Then talk about Dark Code, a new concept that I came up with. It logically follows from the Dark Software Factory, so it's not super inventive, but I do like it a lot. Then the assignment for today was to show you what we built with Codex, so I'll give a quick demo of our product.

Then the last thing why I'm here today is that I'm actually scared. The future is more unpredictable than ever. I've just started a software company. I'm very good at software, a skill that is now being commoditized. Most of my friends are in software or other white collar professions.

Let me give you a mini 10 second demo of my product, we're mapping all software in your team and allow you to zoom in, and have smart overlays etc. It helps you to rapidly understand a complex software landscape.

We started in October and now have 7 people. Did a pre-seed in January and are hiring. And it's very cool, but it's also Scary to be responsible for a software company in these times.

[dramatic pause with an empty slide for effect]

So why am I scared? Well, let me tell you a bit of a story about something that also changed the world.

Back in the day, human muscles were worth a lot. These farmers harvesting the crops did useful work. Hard work, but if you look at the guy on the right it's clearly work hard / play hard.

Humans were the only things that could harvest food or move things around.

Then later, we got horses, oxen and mules for a lot of muscle work. They used horses to move things around because they were better than humans. This is the next level of muscle power, and the human is now the driver. I now want to talk about this guy.

Imagine that you're this person. He's clearly very good at his job. The coat even matches the carriage!

This, dear audience, is someone who knows their shit. They know how hard the horses can be driven, knows when to grease the wheels, which roads to avoid because of traffic, all of that. There were 100's of thousands of these people in the 18th century.

And you know what's funny. I couldn't even tell you now what this profession is called. A horse rider? A carriage driver? Whatever. It's obsolete. But people were so proud to be doing this job.

Now let's talk about the horse.

First a question. How many HP does one horse have? [many in the audience: one!]

One right?

Well actually, no. It's 14.9. [confusion all around]

You see, a typical horse can output 14.9 horse power at max intensity. HOWEVER, over the average of a day, you can get one horse power from one horse. So if you said one, you are still right.

It's actually a genius term, horse power. When James Watt invented the steam engine, he needed to sell a bulky, ugly, expensive machine to people. How did he do that? He said: this machine replaces X horses. People loved replacing horses because they needed care, food and pooped everywhere.

Think about that. Watt invented a new unit of measurement, not to deeper understand physics, but to sell his product. Literally "this replaces N horses". Genius.

In our history of muscles, we started with humans, then horses (etc) and only much later, together with the steam engine we got Horse Power.

Ok first of all, this is not pee. It's gasoline. We're now years further, and this is the muscle in the ultimate condensed form. Gasoline. It replaced all the horses. It's energy productized to the max.

But if you look behind the scenes, this product is part of HUGE global infrastructure that is needed to create that simple product, like this dirty refinery.

Humans were proud of our muscles, and industry took it away.

We were also very proud of our intelligence. I was very proud to get a very very high report card at the end of primary school, but my parents didn't care too much. Child trauma. Anyway.

Intelligence used to be, until VERY recently, exclusively human.

And yes I know, LLMs are not truly intelligent.

Purists will say, it's not AGI and definitely not ASI etc.

But, to me this is a bit like saying this back in 2007 when the iPhone came about. "It can do video calling". "Ok but if you look at inter-human communication this is actually only the first step, we'll get VR and brain-to-brain in the future". Sure, you may be right, but right now I have this smartphone, it's awesome and I just want to build cool things with it!

So while we don't have AGI, it's clear that LLMs are some type of intelligent, it's just a bit unclear how much.

I truly believe that we now have intelligence, priced per token. But how much? We don't have a unit for it yet. We have IQ, and SWE Bench, but how much brain time or humans can it replace?

However, I am starting to get a gut feeling for it. For some tasks in my codebase I'm like "Composer can handle this fine, it's super fast, let's go". And for others I'm like: let's see if Opus 4.6 can figure it out.

So what is happening in the world right now is that we're taking something that we took something that was exclusively human, into a product. (Again, not pee.)

And let's not forget, ultimately the companies that build these gigantic data centers are just trying to push more of that product to us. Interesting side note: I'm curious if we are currently demand or supply-constrained.

So let's see where that brings us.

Have you seen this graph? It was popular on Hacker News a couple of months ago. There were also about 25 million horses in the USA before the car came along. If you haven't read this blog post "Horses" by Andy Jones, look it up, it's really great. It's about when something is good enough to replace something else. Spoiler alert: the horses lost to the car and got slaughtered.

I started with "I'm scared". And just like horses, there are about 40 million software developers in the world right now. Some say 20 million full time professional, and a lot do it part-time or for hobbies so 40 million seems right. So where will this "let's replace human intelligence with LLMs" take us? I don't think we'll get slaughtered, but something has to change.

Melanie, our CMO just deployed our new website. Never touched code before. I gave her Cursor and she just did it with minimal help. Is everyone now becoming a developer?

So another post that was popular recently was this one. About five levels of Agentic development. He says 5 levels, but whether you start at 0 or 1, the count is still 6 so that is weird. It goes from Spicy Autocomplete to Junior Developer, to Developer, to Engineering Team to Dark Software Factory.

Just a show of hands, where are you at professionally?

[most said 3, some said 4, other said 5. I think I should have added "At 5 you are spending 1k on tokens per day per engineer".]

So Dark Software Factories produce something. I'm calling it Dark Code.

The definition is that it's code that no human has written, read or reviewed. So it's dark like, there are no lights.

Now I have thoughts and feelings about this. I think code is not the right abstraction. In Low-Code we put abstractions in place for forms, database entities etc. It was beautiful. Our IDE could reason and check high-level concepts. Agents nowadays just focus on adding code in languages that don't have these abstractions. Good luck on creating a new elegant language now. LLMs are not trained on it and people are not going to write it themselves anymore. In a way, we are cementing our current state of software language development.

Also, I'm often proud of my architecture, code hierarchy or highly performant code. I CARE about some of my code. And now it's just a prompt "Make it faster" and no one cares? My mastery feels hurt.

But the good news for me, is that there already is a lot of Dark Code out there in huge organizations. Sure, someone once wrote it, but that person has long since left. So now no one knows how this works.

In fact, that's how my startup Comper got off the ground.

Does anyone know this game? It's called GeoGuessr and we were playing it in my office a lot when I was CTO. [a lot of people said yes!]

You see a place in google streetview and you have to guess where you are, it's super fun.

Someone from marketing saw us playing this and said "why don't you create that for code?". I thought it was brilliant. You see some code that you or someone in your team once wrote and you have to guess where it is. I started building the game, and figured out how to make a map out of all the software code in the company. Then I figured out it was useful for a lot of other things.

Now, one thing we had was the bus factor: an area is in bad shape if all the authors have left. Because then no one knows about the code anymore. Both the game and the bus factor don't really matter anymore in a world where everything is Dark Code....

That was my last point. Now I'll give you a quick demo, because that was the original assignment for tonight.

[ here I gave a 5 minute demo of Comper, including the game ]

Alright, thanks a lot. Questions? [there were questions but I forgot the details].

Quick context: I’ve interviewed for about three jobs in my life and interviewed somewhere between 300 and 500 candidates for teams I led. So yes, my view is from the interviewer’s chair. I’m in software, which has its own hype cycles, but most of this applies broadly.

What you’ll get here:

• A short glossary so we’re speaking the same language

• Three real hiring scenarios I’ve run

• What each one means for you

• A practical checklist to be the candidate they remember

How companies talk about hiring (so you can decode it)

This is how we talk about the hiring process, you'll might hear these terms but not all of them might mean something to you.

• Hiring Manager: Owns the budget and the final decision. Might not run the process day-to-day.

• Internal Recruiter: Works at the company, screens and coordinates. Often your main contact. Can be a gatekeeper without deep technical knowledge.

• External Recruiter: Paid a percentage of your salary if you get hired. Useful network, misaligned incentives. Treat claims about budget with skepticism. Make sure you check if the recruiter is external or internal.

• Referral: Someone inside vouches for you. Often faster process, better odds, sometimes bonus for the referrer.

• Interview Committee: Everyone weighing in. A founder can overrule. Decisions after each round.

• The Team: The people you’ll actually work with. Weirdly, sometimes they have little say.

• Rounds: Each live step (screens, interviews, take-homes). Usually 2–4 rounds; sometimes more.

• Take-home assignment: Later-stage homework to test motivation and skill; gives you something concrete to discuss.

• The Market: If talent is plentiful, companies get picky (and add hoops). If scarce, they cut steps and sweeten offers.

• Pipeline: How many candidates are in play. A full pipeline means more competition and faster, colder decisions.

• ATS (Applicant Tracking System): The tool tracking every note, score, and email. A one-line “reject” from a manager becomes a canned rejection to you.

• Evaluation: Scorecards (1–5 on skills, values, etc.). Imperfect, but used to create “objective” comparisons.

Alright, now that you know the terminology, let's take a look at some examples.

Scenario 1: The scale-up with money to hire and a high bar

Context: At Mendix circa 2015, we’d just raised a Series B. We needed cloud engineers. Money wasn’t the bottleneck; talent was. We used an internal recruiter to find candidates. The team only saw profiles after recruiter screening.

Process we ran:

• 20–30 minute screen with recruiter or hiring manager: vibe check and basics. Not deep technical discussion.
• 60-minute interview with hiring manager + team member(s): skills and experience.
• Optional take-home (~1 day) and a 90-minute review.
• Decision rule: If anyone serious said “I have doubts,” we didn’t proceed. Feedback to candidates was often generic.

What this means for you:

• Treat the first screen as a clarity and professionalism test. Keep complex answers simple. Don’t go too technical unless invited.

• Internal recruiters can be gatekeepers. Make your resume skimmable: crisp summary, measurable outcomes, relevant tech. Layout matters more than it should.

• A referral here is gold. It gets you past the first filter faster.

• Ask politely about how they decide: “What are the top 3 things you’re evaluating for this role?” and “What would cause a no-go at the end?”

• Take-homes are work. If you do one, over-communicate assumptions, document trade-offs, and propose next steps. That artifact can carry you.

• Expect limited feedback. Ask once if you want concrete tips to improve.

Scenario 2: The startup drowning in applicants

Context: At easee, we needed a QA engineer. I was CTO. QA draws lots of entry-level applicants. We didn’t want to use external recruiters (expensive, sometimes game-y), so we posted the role and got ~200 CVs fast. I screened ~40 candidates, did ~10 one-hour interviews, and brought 3 to final.

What I learned: it’s a ton of work. Without a system, you forget people. Generic cover letters (now often AI-written) don’t help you stand out.

What this means for you:

• Assume you have 10–20 seconds to earn a deeper look. Lead your CV with a 2–3 line summary that matches the job. Include one punchy proof point.

• Show effort early. Examples:

• A 60–90 second Loom introducing yourself and one relevant project.

• A short test plan or bug report you’ve written (for QA), or a concise repo/mini-case for other roles.

• Three crisp bullets: why this team, why now, and what you’ll do in the first 30 days.

• Don’t submit AI-flavored mush. If I can’t hear your voice, I assume I’ll have to manage you closely.

• Make it easy to remember you. Use a distinctive, relevant example and repeat it: “I cut regression escapes by 40% by reworking the smoke suite—happy to walk you through how.”

• It’s fair to ask about the pipeline: “How many candidates are in later rounds?” This helps you calibrate your effort and timeline.

Scenario 3: The tiny, chaotic company with a founder-driven process

Context: ~10-person company. The CEO sources on platforms like honeypot.io, runs loose interviews focused on vision, and sometimes hires on the spot. Why it works for them: they can place you at client companies if you have the right certs—team fit and long-term role clarity are secondary.

What this means for you:

• Expect inconsistency and gut calls. If you need structure, tread carefully.

• Test for chaos with specific questions:

• Who do I report to? How often do we meet?

• What does success look like at 30/60/90 days?

• What’s the plan if client work slows?

• How many people have left in the last 12 months, and why?

• Protect yourself: clarify pay, bench policy, and benefits in writing. Consider a short paid trial or contract-to-hire if the vibes are shaky.

• If something feels off, it probably is. You’re not crazy.

What almost no one tells you

• Incentives aren’t aligned. External recruiters get paid more when your salary is higher. Hiring managers want fairness across the team and budget sanity.

• The fastest path is to no. Efficient funnels reject early and often to save time.

• “We picked someone with a better fit” is usually true—and also a catch-all. It can mean timing, salary, team dynamics, internal politics, or a quiet “no hire” policy.

• Feedback quality is low by design. Too much detail invites arguments and risk.

• Not knowing what you want is risky for managers. If you’re still exploring, say so and show how this role fits a clear hypothesis.

• Desperation level at the company changes everything. When the market favors companies, they add hoops. When it favors candidates, they skip them.

• Hiring is expensive, and it takes months before you're truly productive. If there are concerns you might not stay long with the company, this is a red flag for many roles. In your resume, short stints are a bad sign, so if there are good reasons, explain them. If there are promotions, they are a very good sign, so clarify those too.

Questions that change the interview

Try using a few of these in your next call:

• What’s something in your code/process/product you’re a bit embarrassed about?

• What are you most proud of in the company?

• What would make you say “no” to me after this call?

How to be the candidate they remember

• Lead with a story: why this company, why this role, why now. Make it feel inevitable.

• Make the recruiter’s life easy. Send a 3–4 sentence summary they can paste into the ATS note: who you are, why you fit, one concrete result, link to proof. Gatekeepers remember people who cut their admin time.

• Be a pleasant calendar event. On time, tech checked, concise answers (60–90 seconds), finish with “does that answer your question or should I go deeper?”

• Calibrate confidence. Clear, calm, and prepared beats hype.

• Confidence without bragging is contagious. It’s very easy to hire a person who believes in what they can do and really looks forward to working in this team. It feels like you won’t have to hand-hold this person while they do their jobs, which is always nice for a manager.

• Ask real questions. Show you’re also evaluating them. People advocate for candidates who have their own standards.

• Following up after the interview with a short message shows that you really want this position. Make it positive and not desperate.

Reality check on rejection

• Rejections are often about team dynamics, timing, or budgets you can’t see. Don’t overfit your entire approach to one no.

• Ask for feedback once. If you get a generic response, move on.

• Keep your own pipeline full. Options create calm.

If you made it this far

Thanks again for listening to the interview and reading this “other side” view. And if you’re mid-process somewhere, remember to make it easy for the right people to say yes.

Spring is my favorite season, summer is actually better, but in spring it's clear that winter is over and summer is coming up. In early 2014 I was driving from the Mendix office in Rotterdam to our co-location data center (DC for short) in Amsterdam. I felt fantastic, and for a couple of reasons: I was driving my new car, I was paid to do an important job with a $10k 1U server in the boot, and as the cherry on top, it was finally spring and the weather was great! There have not many times in my life I felt this wonderful. Little did I know that I was now at the top of a rollercoaster, and on the way back I would feel very different.

All of the physical servers at Mendix had Pokemon names. The one in my car was called Flareon. It had broken down a couple of weeks before and had gotten fixed by Hans, our lead systems engineer. We now had to put it back in the rack with its fellow Pokemon. Even though I was the team's product manager and on paper, driving to the data center and replacing a server was not really my job, I had gotten my DC access badge a couple of months before and I was happy to do it. This was my first solo mission.

When I got through the security gate, the floor looked deserted. I drove the cart to Flareon's aisle and stopped at the rack. As soon as I opened it, I noticed something. The empty slot where the server was supposed to be, was placed quite high up. I could only just reach it if I stood on my toes.

The rack looked roughly like this (sorry for the rough sketch, it's from memory).

| Switch 1 | | Server 1 | | Server 2 | | Server 3 | | Server 4 | | Server 5 | | Server 6 | | Switch 2 | | Server 7 | | Server 8 | | Server 9 | | Server 10 | | Server 11 | | Server 12 | | Netapp SAN (4U?) |

Flareon was server 2 in this rack, and was about 2m / 7ft up. No matter, I grabbed a little step ladder and slid out the rack rails. I went down to get the server and noticed it was a bit difficult to climb the ladder with an unwieldy and heavy server that weighs about 15kg / 33lbs.

When the server and I reached to the top, I also noticed that the rail mount had a system with four latches that had to click all together. I knew this already, but somehow I had never been as aware as now. A further complication was that when it latches, you need to click the latch on the left or right side to release it. But you're probably holding the server with one hand in the front, and one hand in the back (so yes, not left or right). For the lower placed servers you could hold the server with one hand and a knee, and use your spare hand to release the latch but at 2m in the air, balancing on a step ladder, I had no knees or spare hands available. I suddenly realized I had come to a two-person job alone and felt very lonely.

I now had a choice. I could go down and give up, or see if there was someone available to help. Alternatively I could just go for it and try to do it by myself. What's the worst that could happen? My instincts have been shaped by software development, and there the worst that can happen is not very bad. You can always do git reset.

So after a second or two of deliberation while balancing on the step ladder with a server that was not getting any lighter, I decided to go for it. Standing on the left side, I would lift the server over the rails, and I'd try to drop the two latches on one side into place. Then I would move my hands slowly to the other side and push the other two carefully in. I never got that far. One latch clicked, and the other one missed its opening. To fix it, I’d need one hand to hold the server and another to release the latch. Impossible. Within half a second the 15kg server started falling 2 meters towards to the ground. The rails twisted and released the latch. I tried to grab the server on its way, but lost my balance and fell after it like Gandalf fell after the Balrog.

I don't think the data center had ever been exposed to this amount of sound during normal operations. It was deafening. Within 10 seconds I realized the floor was in fact not deserted, as two guys who had been working three aisles down came running towards me. "What the f*ck happened?!" they shouted. Actually the scene was quite straightforward and after a moment they had figured it out. I was so shocked I didn't even answer. They asked if I was alright and then offered help. I noticed I was not hurt myself, and the server had a dented front but otherwise looked alright.

With the help of the other guys we put Flareon in its rack. I wasn't sure if it was broken or not, but it couldn't stay on the ground so why not put it up there now that I had two volunteers?

When the server was in place, my new friends went back to their own servers and I tried to calm my nerves. I looked at my phone and saw a lot of Nagios alerts coming in. What the hell?

After deciphering some of them and realizing another server had suddenly gone offline, I looked up and sure enough, the ON led of the server underneath Flareon looked like it was very much OFF. In my fall I must have hit the power button... I called Hans for help, explained what had gone wrong and he calmly reassured me that we would fix this first, and figure out the rest later. What a great team mate!

Hans set out to restart all the applications on this server. About 50 Mendix apps were down for half an hour or so (I tried to find this incident on the status page but we only started using it a couple weeks after this incident. Coincidence?). In the meantime I hooked up Flareon's power and network and started it up. Hans did a system test and everything looked alright.

I locked up the rack, packed the cart in my car and started driving back. On the way home I felt incredibly stupid. I had dropped a $10k machine, caused a serious production outage and suddenly realized this could have ended much, much worse.

See, the server fell down and could even easily hit the Netapp with 40+ disks spinning at 10k RPM. Those things don't like vibrations. There are stories of disk arrays being destroyed by way less than what happened today. Rebooting a server is annoying, but destroying ALL primary data for hundreds of customers would have been a disaster of unimaginable consequences for the company. I got away easy, but the realization made me nauseous.

That spring day in 2014 is burned into my memory - a real rollercoaster. Here are three things that I've learned:

- Hardware is not software, there is no git reset

- Never do a two-person job solo

- Disable the power button on servers

The story became infamous within our team. A couple months later we went back for some maintenance and had to do a re-enactment.

Flareon went through some stress testing and then happily served our customers for years. I believe it was finally decommissioned in 2021/2022.

P.S. I looked up some instruction videos for the latching mechanism for the DL360p. I was probably doing it wrong as you're supposed to take the latches apart, attach half to the server, and just slide it in! So lesson number four:

- Read the instructions.

Discuss on HN.

What personally gives me energy is if there are "deep problems" that require an elegant solution, so if you let me focus on those things, you get a well-organized codebase out of it, making all future development a lot easier.

I believe that if you have good engineers with a product mindset, they will have a well-developed intuition for what to work on. So if you're managing a backlog and are stuck in analysis paralysis, ask the engineers what gives them the most energy and let them work on that. It'll be great for motivation and ownership.

• Team wants to work for someone who cares about what they do.
• There is focus and clear priority for the work to be done.
• The team has the right resources at hand.

You probably can't achieve 10x performance consistently, but most mission-driven startups get very close. If you have the feeling that something is wrong with your team's output, you can use the following flow-chart to find problems.

Here's a PDF version.

Here's a very beautiful antique rotary phone. It's a Dutch PTT phone model called simply "1950".

But, what's this at the back? A USB-C port?

A peek inside shows us some weird things! How did that LiPO battery get there?

Ok let's quit the jokey intro, this was all me. I've had this phone for about 15 years and over the last couple of weeks I've converted into a fully functioning Bluetooth headset using an ESP32 microcontroller.

In this post:

- 1 - Features

- 2 - Lessons Learned

- 3 - Pictures and story! of how it got made (scroll down!)

but first...

Does it work?

This morning was the first Google meet I've used it in:

A participant reported on the sound quality

"I didn't know how much I missed the different sound quality! Beautiful. We could hear you very well, not distorted. I wish I recorded your speech ehehe"

I could hear them very well too, so I consider this a great success!

Get the source code on Github jtwaleson/bakelite-to-the-future-esp32

Features

If you count the hours and multiply it by my hourly rate, this is BY FAR the most expensive phone I've ever touched. But you get a lot:

• The buzzer works when you are dialed (but not if you are dialed on Whatsapp?)

• The rotary dial can be used to dial complete phone numbers

• The red button connects / disconnects my phone

• The red button connects / disconnects my laptop

• The two indicators indicate Bluetooth usage (bottom) and an active audio connection (top)

• Lifting the horn or putting it down picks up / hangs up call. If no number is dialed you are connected too Google Assistant/Siri.

• It has a 1500mAh battery lasting for about 1 day. I didn't implement any power savings. As it's mostly a gimmick and easily connected to usb-c for charging this is fine.

• Most importantly: I can once again slam down the phone if I'm having an angry phone call. Which I never have, but still.

Lessons Learned

I have some new skills under my belt:

• I2S protocol and its many flavors, and how finding the right components is tricky (the software only supports variant left-shifted, but this chip only support Philips! So have to order new parts, look at datasheets and wait 3 days)

• An oscilloscope and function generator are great tools!

• This is my first ESP-IDF project, so getting to use that stack was fun. It works quite well!

• You are on your own. Even though the ESP32 community has tens of thousands of people, there's not many people that have tried out this part of the stacks. Poorly documented functionality and missing integrations all around. However, Espressif has great people on Github.

• Why flyback diodes are needed and what inductive loads are. (Yes I am a beginner).

• The difference between MOSFETs (voltage controlled) and BJTs (current controlled) and NPN (low-side switching) and PNP (high side switching).

• How simple rotary dials are.

• How class-d amplifiers work. I vaguely knew this, but now I learned why there's no common ground in class-d amps and that you need that if you have a headset, because the mic and speaker will share a common ground over a three-wire connection.

• Super glue can be removed with nail polish remover.

Pictures and the story

The un-opened phone and connection box. I bought it for like 10 euros in a second hand shop 15 years back. Actually it had 3.5mm audio jacks because I already tried to convert it 10 years ago into a wired headset. Now we are going to do much better!

It's pretty full on the inside! You can see my wires for tapping off the audio signal made with UTP wires. The phone originally had two lines and you could switch between red and black. I don't know what the white buttons were used for. I'm going to use them for * and # DTFM codes and some other hacky stuff.

The indicators on the bottom left look weird, but we'll get to that.

The speaker and microphone look like this. I suppose they can be upgraded later.

It was made by a company Standard Electric in The Hague.

And of course the PTT (Dutch postal service, for Post, Telegraph and Telephone)

Was it made in 1965? The model is called 1950.

Ok let's start dismantling it with my fantastic soldering iron that heats up in 4 seconds (pro-tip, buy this 150 euro Aixun T3A T245, it will blow your mind)

Wires are loose, I'll re-use them later.

An old Capacitor.

And this? I don't know what it does but there were two of them. If you want them feel free to reach out.

There is no sensor for the horn, but I would like one, would this work? The answer is no.

Hooking them up to power and controlling them with some simple BJTs. They also make a nice clicky sound when switching on/off.

It works. The controlling wire (yellow) takes 0 mA. This is safe to use by the GPIO on the ESP32.

Let's try a lot of switching with a signal generator. It works, but what are these huge voltage spikes I seen on my scope?

Hmm, I could fit a usb-c port in there.

Let's prepare all the buttons etc for connecting.

My family was really happy with my maniacal focus on this project for three weekends straight.

Ok let's start to see if we can get sound to work with this INMP441.

Ok the signal that the HFP_HF library sends is weird, the microphone is not compatible. I went with a WM8782 ADC based board instead. I also bought a DAC , because the first I2S board I bought was a class D amp and I need a common ground one. The new DAC/ADC boards can both be figured to "standard" left-aligned 16 bit i2s signals and don't need any further amplification.

The microphone in the headset was an electret and with +5V and a 4.7k Ohm resistor, hooked up via a tiny capacitor to the ADC, produced good results.

I spent like two full days trying and learning about the microphone and I2S and could tear my hair out at some points. Without on oscilloscope to see what's actually going this would have been impossible.

On the other hand, the rotary dial was super simple. One power (or ground) and the other one line gives a long pulse (while not at rest position) and the other pulses for each digit. The code to interpret this was like 10 lines.

Starting the put everything together. I used an ESP32 T-Koala dev-board. The newer ESP32 chips (like C3/C6 etc) don't have classic bluetooth so you'll need to pick one that has both BLE and classic BT.

On the top you see the buzzer which needed 7 volts, so underneath there's a voltage booster from 5 to 7 volts configured with a switching NPN MOSFET.

Top view with the battery added.

The bar in the middle has a spring underneath it which presses the horn up. I isolated the spring and the bar with some plastic and added solder. If the spring (hooked to a GPIO) is up it touches the bar (hooked to ground). Now it acts like a switch! It bounces around (literally) so I need to add some debounce in the software.

I modified the original back plate to add a USB-C and on/off switch. I had to cut off a bit of the steel with my Dremel, but it fit seamlessly into the original case. No bakelite was hurt during the making of this phone.

In this picture you see a simple "power-only" usb port, but I switched it for a fully functioning one, so now I can update the ESP32 firmware without opening the phone again.

The completed thing. It's wireless and has about 1 day of standby time. This one sparks joy :)

Links

- HFP_HF example code https://github.com/espressif/esp-idf/blob/master/examples/bluetooth/bluedroid/classic_bt/hfp_hf/README.md

- debug issue 1 https://github.com/espressif/esp-idf/issues/14700

- debug issue 2 https://github.com/espressif/esp-idf/issues/14231

- debug issue 3 https://github.com/espressif/esp-idf/issues/13738

- debug issue 4 https://www.esp32.com/viewtopic.php?t=40279

- my code https://github.com/jtwaleson/bakelite-to-the-future-esp32

- Hacker News post https://news.ycombinator.com/item?id=41986452

Ok! Thank you for inviting me and sorry for not keeping to the original format as requested. I proposed this idea instead and it was accepted, so in part Matthijs (the organizer) is to blame and I hope it is OK with all of you as well ;)

When cycling here, I realized I needed to make a last minute change and added "Systems" to the title. I believe that limiting ourselves to the activities of the programmer is too narrow, and we should have solutions that involve everyone who is in the lifecycle of software. So it's about the System and that includes the designers, architects, users, database administrators, operators, and yes, also developers.

Your request was to talk about one of the classic texts of the software engineering / software architecture field. Now I didn't pick a classic, and I didn't pick a book. Instead I picked this semi-random blog post I found on Hacker News a couple of years ago.

Why this blog post? It's not so revolutionary or great that it is going to be the next classic work. But, it had some imagery that I just could not get out of my head. Open the link and read the first parts until the cave man animation. Here are some stills if you don't want to open the link for some reason.

Editing code on a screen is just a window to a very large codebase!

We are trying to form a mental image of the codebase in our head by staring at a tiny isolated part of it

It is a lot like a cave man seeing just a small part of the cave wall. With that we mean: it is primitive as hell!

So why did I pick it? Well, after a 7 year stint at Mendix, a Low-Code pioneer where I left in 2018 (see footnote 1), I am obsessed with better ways of programming. It was already better in many ways, but we still only scratched the surface with that product. It made me see the light, at least in my mind. This blog author seems to have some of the same ideas. Over the years I've collected bookmarks of all interesting hacker news posts about this field, and this is one of the top ones in the list. And it had nice visuals that made for a good story.

So let's go to the four thoughts.

Today we are going to explore four ideas that lead to the way I view the future of programming systems. The middle box will remain a question mark, because it is currently a quest that I am on and I do not have the answers yet. The ideas around it will give some kind of direction for the center box though. I think you'll get it at the end.

The first idea is that the way we program now is stuck in a local maximum due to the way that software development evolved.

The human body has an appendix. It is a useless vestigial* organ that served our predecessors and now sometimes kills us. That was the story at least, now science seems to say it is useful. But for today's story let's ignore that new science ;) The point is that software is riddled with historic vestigial artifacts. If we would start over from scratch, we could build something much better.

*Vestigial: [....] having become functionless in the course of evolution. - Oxford Languages definition

[frame 1 & 2] In the fifties we programmed by connecting wires between vacuum tubes to make our circuits. Then computers became more flexible with punch cards, and then punch cards could be written with high level languages like Assembly and FORTRAN. Yes, in those days even Assembly was high level.

[frame 3] Fast forward 40 years and we are still manipulating source code files with instructions. Or maybe not instructions if you use nice functional languages, but it's still very similar.

[frame 4] But programs today are much more complex! Here is some code that connects to the internet and lets another program do something by sending it some data. And this is just one example of things that became more complex. In your daily lives you see how complex it can get, from self-driving cars to ChatGPT to the software that controls our international electricity grid.

[frame 5] Nevertheless, a typical project is still structured like this. Folders with source code files, config files, makefiles, documentation etc. They looked like this 30 years ago and still do so today. The fact that "you can throw anything here and call it a project" is in my view a vestigial aspect of coding. It screams to me that no one is running the asylum. If I stick a random image in a random directory, very few IDEs or CI/CD systems will complain about that. Isn't that weird?

[frame 6] Another vestigial aspect of code is the fact that we still have to bother with tabs vs spaces!

(Audience member) - but.... that's now taken care of right? Everyone has linters and autoformatters?

No! The fact that a linter has to fix something that shouldn't be there in the first place is horrible!

It is like we have built a whole bundle of tools that we've connected to each other and compensate for each others leaky abstractions to get the situation under control. To programs we've added databases, logging, monitoring, error tracing, compilers, linters, formatters, ORMs, test frameworks, mocks, LSPs, and all of these things as separate tools. And now if we want to change the way we program substantially, we have to change all of these things! It's very optimized and very powerful, but it feels like everything has been glued on.

The point is: we've evolved and are now at a sub-optimal point. Let's go to thought 2.

The next point is that we can understand things much faster with the right medium.

Understanding is crucial for being productive, and it goes for both machines and humans. If an IDE recognizes what some code or config actually does, it can apply special rules to it. For example, the Laravel framework in PHP is highly structured in what Events, Database Models, and Routes the application has. The framework treats these as High Level concepts. It's really great! But, because it's just coded as Class-based PHP, virtually every IDE still treats the code defining e.g. Events as regular PHP code. The IDE just sees the characters, and maybe some of the PHP, but knows nothing about Laravel!

Understanding is crucial for humans too.

Here is a text based description of what a combustion engine does. Thanks ChatGPT!

Of course I much prefer a visual animation. Check this gorgeous website for this and more examples. I grasp what it does much faster and much more accurately with a proper medium. That medium took way longer to generate than some text, but it pays off!

Or when I'm asking for directions in a city. It's super confusing.

I much prefer a map!

When architects or developers are trying to make sense of a software landscape, the current mediums for that seem to be:

• reading powerpoint presentations by architects that are probably not at all grounded in reality
• walking to a whiteboard with some seniors
• reading documentation which is very likely out of date

I would really like a better medium for that!

So the second thought is: other mediums can often explain things much, and we should try to find the right ones.

Let's continue to the next point.

Programming systems were originally built for the silicon. Humans were cheap and computers were expensive. Of course this has changed significantly as now the situation is completely reversed. We have powerful IDEs and there are some programming systems that really tried to do better (Mendix, etc), but nothing mainstream goes far enough.

As a piece of silicon I just need to know how I can execute a piece of code through my circuits.

As a programmer I have all the questions on the right. Please take a look, I'm not going to repeat them here. I think virtually no IDE helps you with these questions. Yes, some have dead code detection, but there is a lot of false negatives and if you stop using one of the entry points of the application (an API route for example) and you forget to delete it in the code, no IDE will warn you about that. Your monitoring tool knows it's not used. Why doesn't your IDE? I suspect that the fancy programming systems in Meta or Google will help you with a lot of these things, but those are not accessible to 98% of the software engineers. VS Code also highlights who wrote a piece of code and when, but if a function got moved once over the last years, good luck! Again, a sign of a bolted-on technology, which fundamentally does not capture meaning but code characters stored on disk.

So, the conclusion here is that programmers have a lot of questions while programming that A) need a lot of discipline to do right and B) lead to a lot of manual work. They have unmet needs.

The last thought is one of hope.

I think software is truly truly special. It's in a category of its own if you compare it to any other industry out there. (Please note that I have not tried them all ;) )

These are some of the reasons that make software different and that make people fall in love with software. I fell a long time ago, and I still am in love with it. The last point is made incredibly well by Fred Brooks. I managed to sneak in a reference to a proper classic text of the software engineering field after all!

Because software is so inherently different we have the power to re-invent our programming systems radically.

So please, help me out by 1) not accepting the status quo, 2) keep an eye out for the innovators and try their solutions, 3) always look for better abstractions in your code and architecture.

The four thoughts once again. These ideas are central to my belief that the future of software systems will be very different from where we are now. In the future, programmers, users, designers, enterprise architects, etc, will be supported by much better tooling the caters to all of their needs. And maybe most importantly we will finally get rid of tabs vs spaces ;)

Like I said, over the years I have curated a long list of articles that resonated with the way I was thinking. I'm now considering starting a newsletter or series of blog posts with the best links of the month with my commentary or something. Then again, maybe that's just a distraction and I should just start building my own company that actually builds the future of programming systems.

Thanks again for having me and letting me talk about my passion. What questions do you have?

[ discuss on HN ]

Footnotes:

1. Mendix made a lot of things better through offering the right high-level concepts. I think criticasters would say that we were artificially limiting the developers. I would say that we put concepts like Forms, Database entities, Scheduled events, API endpoints etc as first class citizens in the development environment, so that it could help you. And help you in ways that make more sense than stupid-simple auto-complete. I love AI coding tools like Cursor or CoPilot, but the fact that they use text character based completion makes me mad as hell! It should be so much better!

1. Security is limitless. You can always spend more effort to make things more secure. The same goes for quality, safety, employee happiness, etc.

2. The needs of security are opposed to the needs of a convenient user experience. Improving one typically hurts the other.

Now some organizations say "Security is our #1 priority". Really? You want to make something that has no limits your number one priority? I mean security is a good thing, but this seems a bit too simple? In fact, hollow marketing claims like that can make me a bit angry. In this post I'll help you understand what to make of statements like that, and how to deal with security in real life. I'll cover:

• A philosophical intro
• What does "Security is our #1 priority" actually mean?
• How far should you take security?
• What should companies say instead?

A philosophical intro to help our thinking

The limitless nature of security and the balancing with user experience (UX) reminded me of something. I had to stop and think for a bit, but then I remembered. When I was about 19 or 20 I discovered GK Chesterton. I loved him as he made me see there were actually formidable Christian thinkers, and in my sort-of evangelical background there weren't many of those. Unfortunately for what 20 year old me would have liked, and despite Chesterton, my personal faith didn't survive, but I did still learn a lot from his writing that I appreciate to this day. Here's one quote that relates to this topic:

“The modern world is not evil; in some ways the modern world is far too good. It is full of wild and wasted virtues. When a religious scheme is shattered (as Christianity was shattered at the Reformation), it is not merely the vices that are let loose. The vices are, indeed, let loose, and they wander and do damage. But the virtues are let loose also; and the virtues wander more wildly, and the virtues do more terrible damage. The modern world is full of the old Christian virtues gone mad. The virtues have gone mad because they have been isolated from each other and are wandering alone. Thus some scientists care for truth; and their truth is pitiless. Thus some humanitarians only care for pity; and their pity (I am sorry to say) is often untruthful.” - GK Chesterton, Orthodoxy, 1908

What Chesterton is saying is that good things (virtues) can become bad if they are going too far or without including other good things. Seems like a no-brainer, and it is. But once you grasp this pattern you see it happening all the time. In this blog post it's about security freaks that forget about UX, but it's the same in our political world. Some people take DEI so far that they vilify white men. Some people take family values so far that they make others feel like misfits. Both family values and inclusivity are good things, but taken apart they can lead to extremes and can foster polarizing between the groups.

So what we need is a healthy mix between all the good things, without letting one become extreme and neglecting the others. This isn't easy! If you're a security extremist, and you still get hacked, at least you can say you did everything in your power to make it more secure. On the other hand, suppose you have balanced UX and Security and you took some decisions to make things less secure but for a better UX. You get hacked. How do you justify that? Saying "Reality is complex and we took a decision to prioritize UX in this case" is not a good soundbite. It takes guts to defend non-extremist positions, because the extremists will always have more powerful one-liners.

What does "Security is our #1 priority" really mean?

Is Security actually limitless? I think yes. In fact the safest way a bank can operate is to shut down their online presence, buy a big vault and put a small army outside the door. Even then, the amount of security checks and the size of the army can be increased infinitely. However, most people will prefer to send money using FaceId on their banking app and think that that security is good enough.

And do companies actually say this? Yes. Here's Microsoft "Prioritizing Security Above All Else", AWS "Cloud security at AWS is the highest priority", Meta "Safeguarding your data is our highest priority" and there are many more.

It sounds nice, but does this in practice mean that whenever anyone has an idea to improve security, at the expense of UX, consumer prices, etc, you still implement it? Because that is what it sounds like. Of course the answer is no, so I'm not taking the statements too seriously, although I understand it gives your customers a warm fuzzy feeling. I'd prefer organizations to be honest and clear, but I can only dream about a world where this is true.

In reality these companies mean something like:

"We have a balanced framework of priorities, we categorize issues and give each a weighed score. We give UX issues 20%, security 25%, tech debt 10%, new features 20%, etc.. As you can see risk is the number 1 priority because 25% is higher than 20%."

Or maybe they mean:

"Security (to a level that is the norm in our industry) is our number #1 priority. Once we have implemented enough we focus on other issues."

If you have examples of completely honest security statements like "we care about security because we need you to trust us, and therefore it's on of our top priorities" I'd be very interested. Please comment!

How far should you take security?

When I gave the talk in 2018, where to actually draw the line was still a mystery to me. Having worked with risk frameworks like ISO 14971 and ISO 27001 since then, I now have some tools in my belt to help me deal with this.

It all starts with a scoring mechanism: figure out what do we have that needs to be protected, what risks are there and how likely / severe are the risks. You multiply the likeliness and severity and you get a risk number for each risk that you can map to low, medium, high.

• If the risk number is low you can safely accept the risk.
• If it is medium you should consider risk controls unless justified.
• If high you must implement risk controls.

Then once the risk controls are in place, verify that they actually mitigate the risks and if the remaining risks are now low or non-existent. Then you are done.

Sounds simple, but where do we put the line of low, medium, high? I always put some decision makers together, give examples of things that could go wrong and ask them to come up with a scoring system. Then ask what kind of risks they are willing to accept, considering what the product or proces would like like with a risk control applied.

This simple scoring system has a risk matrix, a risk appetite / risk acceptance policy, and a risk inventory. It's a simple tool that can help you make decisions.

What should we say instead?

The entire world has been handling safety, quality and security in a grown-up manner for decades. Let's promote the companies that do that without saying it's their top priority. It's a meaningless statement.

In my ideal world, companies would say "We maintain a state of the art security system, because without our customer's trust we would have no right to exist as a company. As such, it is one of the most important things we work on and we spend a large amount of our effort on it.". Probably doesn't sound as good to most people, but companies that say this would certainly score points with me.

Have good examples of honest security statements? Comment on HN.

Speed, I am Speed. -- Lightning McQueen

Some thoughts about the impact of feedback loops in startups and software.

Parts:

• Intro
• Framework
• Observations
• Take-away

Intro

Feedback delay is the time between doing something and seeing whether it was successful.

If the feedback is slow, we have a "lagging indicator". It becomes hard to link causes and effects, and in consequence it becomes hard to optimize the performance. Fast feedback makes things easy.

How quickly the feedback arrives is critical. If it takes milliseconds to seconds, it works at the speed of the brain and enables a state of flow. If it takes decades to centuries, it needs a multi-generational organization like a government or organized religion to run it. Somewhere in the middle is the realm of persons, teams and companies.

A framework of feedback time

Here's a quick table of feedback times, in three groups, with each roughly an order of magnitude difference.

C is the level where (large) organizations can run experiments without seeing the results for a long time

• C4 centuries
• C3 decades
• C2 years
• C1 months

At B a person or team can have the patience to execute this

• B4 weeks
• B3 days
• B2 hours
• B1 minutes

A is the speed of the brain, the feedback arrives so fast that it feels effortless.

• A4 seconds
• A3 100+ milliseconds
• A2 10+ milliseconds
• A1 1+ milliseconds

Observations

With this mini framework in mind, here are a couple of observations.

1 - Startups

Startups are viable if they do something 10x better. Each of the steps is a 10x improvement, so if you manage to create a solution that brings feedback time down one step on the ladder, you have a viable startup. At least according to Peter Thiel.

2 - Faster is always better

Faster feedback is always better and never worse, but there are diminishing returns. E.g. when I'm typing and it takes 4 milliseconds to render to the screen, it's already imperceptible, and improving it more offers no benefit.

When looking to improve something, check what level it is at right now and what would count as imperceptible. The difference is the space that a new product or service could enter.

For example, in a regulated industry I will get feedback from an auditor at C1 (a couple of months) but my life would be so much better if they replied at A4 (seconds) and I can have a dialogue. So: there's an enormous opportunity here for a regulatory body that promises responses in B1/B2 (see e.g. Scarlet) or for a group of experts you can chat to instantly (OpenRegulatory Slack).

3 - Multiple orders-of-magnitude improvements shake the world

Improving by 1 step is great, but something weird happens when we take 2 steps down the ladder or more. Things that were previously impracticable or downright impossible, now become in reach.

When jetliners were invented, we could travel around the world in B3 (days) where in 1930 it still took my grandfather C1 (months) to go from Indonesia to The Netherlands. Jet travel has certainly changed international business, politics and tourism.

A recent example are LLMs. Getting a book summarized would take a human B2 (hours), and ChatGPT does it in A4 (seconds). By making it two orders of magnitude faster, we just went into flow-state territory!

When a new technology makes multiple orders of magnitude improvements, the world needs a couple of decades to adjust to this new reality.

4 - Software vs Hardware

Most software development features are in B, while hardware is in the lower Cs. They are wildly different industries because of this, with different approaches to QA.

Software developers that go into regulated industries see their feedback time going from B1 (when automated tests tell me there is a problem) and B3 (when customers tell me if they like something) to C2 (the regulators will review your documentation and give you feedback in about a year). Talk about culture shock!

5 - Hunt for the feedback loop

Whenever I am stuck on a problem, I have bad tendency to get blocked and not knowing where to start. When I notice this happening, I've learned to try to find something to do that works towards the goal and that gives me feedback the fastest.

In programming there are feedback loops that will always get me into a hyper productive state:

• Getting compiler / linter errors in my IDE. When I've refactored code or when I introduce a linter to help clean things up, I get a bunch of new errors that I just have to fix one by one.

• I execute the program and get an error, then I fix it and go to the next error.

• Getting CI test results (e.g. speed of the program, memory usage, code coverage). Changing something and seeing the result of my change gets me feedback pretty quickly, but CI is often in B1/B2 (minutes/hours) and unfortunately frequently gets me out of flow state.

• Create a to-do list of things to finish, and try to get it to 0 as quickly as possible.

SpaceX famously achieves rapid progress through rapid feedback cycles compared to the upfront design work at e.g. NASA.

Their long-term goal is to colonize Mars, which is clearly in C4 / C3 (centuries / decades). They won't know if this will work until it's done. Those kind of grandiose projects have a bad habit of not getting started. SpaceX brilliantly broke the problem up into multiple milestones, and got a feedback loop going towards milestones 1 - launching Falcon 9, 2 - making the boosters re-usable, 3 - building a bigger rocket which is entirely re-usable.

Their results are phenomenal.

This is a simple thing to apply when stuck on big projects: break it down, and get a feedback loop going that brings you towards the first milestone.

Take-away

Always look with a fresh view to your current way of working. How much better would things get if you got feedback sooner? That's where the opportunities are.

P.S. when finishing this I realized I could have written the same things about other cost factors. Time is a cost, but so are money, energy and materials. The framework could be the same and very similar observations could be made.

2018 - Mendix Cloud v4 - Part 2 - How to launch a complicated product

Note: This was a multi-year mega project (at least for my standards) and ran from about 2015 till 2018 when I left Mendix. It's still one of my proudest professional achievements. If you haven't yet, read part 1 about Cloud v4.

During 2017/2018 our team was doing a lot of things at the same time:

• Handle the operational fires in Mendix Cloud v1/v2/v3.

• Add new hardware with 50%-100% YoY growth, as well as handle all kinds of customer requests, like resizing VMs or adding more storage. These were all manual operations by our engineers.

• Migrate v1/v2 apps to v3 to ensure standardization.

• Build v4 which would make everything standard, scalable and self-service. It would be the solution to solve the chaos from v1/v2/v3.

Fires? What fires?

Running a 24/7 hosting operation for customers that are not so technical is tricky. Our brand promise was to make hosting Mendix apps simple for customers, so that meant that WE had to do a lot of hard things. We had lots of crisis situations:

• Unexpected security vulnerabilities like HeartBleed that had to be patched. For HeartBleed we patched everything within about 4 hours after the announcement, thanks in a large part to Hans.

• Physical servers breaking down, leading to about a hundred apps going offline at a time.

• Operating system migrations (e.g. from Debian Squeeze to Wheezy), that had to be manually upgraded across thousands of VMs.

• Programming errors leading to data corruption, on hypervisor level and also in the Mendix app that managed the entire cloud.

This may sound strange, but I loved the adrenaline rush when a crisis started. I now have a sort of spider sense for it. As Product Manager I often took the lead, jumped into organizing mode and got customer service, the 2nd & 3rd line engineers and management lined up. I tried to hold regular status overviews via organized excel sheets and write customer communication updates. After a couple of crisis situations we set up https://status.mendix.com to streamline our communication, that helped a lot. I built pretty good relationships with our people in security, customer support and CloudOps. At easee, my later company, the server setup was super boring and I secretly missed the adrenaline rushes quite a bit.

Needed features and the people that built them

But back to v4. We had expanded our team with Emir, Hans T., Cenk, Raphael, and Diogo. Unfortunately Riccardo left for ~consultancy~ HFT pastures at some point, and Emir left too a bit later IIRC. The team started building v4 services. We could spin up empty Cloud Foundry clusters quite easily, but to get feature parity with v3 we needed:

• Backup services (nightly + PITR as a high-end option)
• Service brokers for S3 and RDS
• Reaper / resumer for Free Apps
• A Cloud Portal that could support both v3 and v4
• A lot of improvements to the buildpack (client certificate support etc)
• A logging service (+ live logging would be nice)
• A monitoring service
• A load balancer / SSL/TLS terminator
• Dashboards for monitoring

That's a lot of things we had to build! We also needed to up our documentation game because we were really lagging behind there. I managed to poach Kasja as a technical writer and she started creating a lot of articles, as well as bringing a lot of fun to the team. For some reason the atmosphere in our office was better on the days that she was working. It's nice to have diverse teams and not only smart introverted nerds (she is also a smart introverted nerd, but that aside).

The plan to avoid the Second-system effect

By this time v3 was pretty mature, and I knew enough about the software industry that I was dead-set on avoiding the Second-system effect. So: no new features in v4 except for scaling horizontally, which we would get out of the box with Cloud Foundry. In fact, we would start with less features than v3, and start migrating the customers for whom the limited feature set in v4 edition was enough.

I figured we would launch v4 in four phases:

1. Launch a v4 beta for Sandboxes / Free Apps without any guarantees. This way we could remove the hacky Sandboxes implementation in v3 and get some real world experience with running v4.

2. Launch some small non-critical apps on v4.

3. Make v4 the default for new apps (v4 unless you need VPNs, 99,99% uptime, etc.)

4. Start phasing out v3 and move everything to v4.

I think I had the plan outlined in my head, but I didn't communicate this at all. People outside our team were really confused about what was happening and when, especially as all the other migrations from v2 were also still going on. It turns out Project Management is a skill, and I didn't have it yet. My approach was "It's done when it's done and today we are just trying to survive". I didn't put any concrete dates on launches, because in that chaotic environment it was hard to be predictable.

Struggles & Highlights

Nevertheless, the plan continued but some stressful situations followed. Here are some highlights:

1 ) Making Free Apps work with the Web Modeler. The Free Apps cluster had to be made faster and compatible with the Web Modeler. Read about that in 2015 - InstaDeploy. Getting this more or less stable was a lot of work and had a lot of management eyes on it.

2 ) Sales team briefing. At some point I sat down with the Head of Marketing (Hans de V.) and we decided I should brief the sales teams on the new v4 system that was coming up. I expected some kind of dialogue, but instead sat next to Hans (who is really great at giving presentations), and was given the microphone in front of a Zoom call with 40 sales people. It was one of the absolute worst performances I've ever done. I had given plenty of mediocre presentations where I could look the audience in the face and get some kind of feedback. That interaction compensated my chaotic presentation style a bit, but here, talking into the webcam and just seeing my own slides was throwing me off. These sort of things require adaptability, preparation and experience and I had none.

3 ) Latency issues. Our great new "High Availability" feature where apps were hosted Multi-AZ turned out to be a performance disaster for some apps. We had customer apps where 1 API call would result in 1000s of sequential database requests . The latency to the database could jump between 0.1ms and 5 ms depending on the AZ, so after re-deploys the performance of apps would be dramatically different. Oops. This did not make our Customer Service and Professional Services teams happy.

4. Migrating 6000 apps single-handedly in one night. After a while I wanted to migrate all Free Apps from the first and outdated Cloud Foundry cluster to a new and more professional cluster. These were about 6000 apps that needed their database, application code, dns entries and files moved to another Cloud Foundry cluster. Doing this properly with the team, checklists and reviewed migration scripts would take weeks. I decided to do it by myself in one night with a python script from my home after a long working day. Keeping track of this huge migration hit the limits of my brain and the stresslevel during the migration was insane. This turned out well but was very stupid, something could easily have gone wrong.

5 ) AWS Costs. The costs at AWS were going through the roof. After a couple of weeks I discussed going for up-front investments with the Head of Finance and we had to transfer 300k EUR just for the first batch. Nowadays whenever I need anything from AWS NL I mention that I was (for a large part) responsible for getting Mendix on AWS and they are always happy to help me for landing a huge account.

6. Road Trip. Mendix R&D employees had a 1500 euro yearly training budget. Most people would not use it, or go to a nearby boring conference, spend 500 on the conference ticket, 500 on travel, and 500 on food & hotel. I wanted something more epic. With about 8 colleagues we pooled all our resources, obtained free conference tickets (because the company sponsored Cloud Foundry) and went on an epic 10 day trip. First by plane from Amsterdam to Las Vegas, then by fancy convertible to Death Valley, Yosemite, San Francisco, the conference in San Jose, on to Los Angeles and back to Vegas. I'll write another post about this in the future, but it was one of the greatest trips of my life. We also learned a lot about Cloud Foundry.

Wrapping up and moving on

In early 2018 I was getting ready to leave Mendix to be CTO at a tiny startup called easee. But by this time we had finished two important milestones, so I could leave without a sense of abandoning the project.

Some unsuspecting new customer was the first to go live on v4, the CSM team figured that this customer wouldn't need any of the v3 features and would not do mission critical stuff any time soon. Then more new, non-critical customers joined. Around this time we also publicly announced Cloud v4 with a blog post that is now lost.

About half a year after that, we decided (together with CSM, Support and Professional Services) that all new apps would go to v4 by default!

This completed step 3 of my 4-step plan. It had taken years of discipline of not giving out guarantees of VPNs, of not promising NL-based hosting, of promoting 12-factor architecture best practices, but now it was ready. Most apps could "just" be deployed to v4 and scaled horizontally, and we could scale the underlying infrastructure with just our credit card!

The verdict

Mendix Cloud v4 brought a lot of benefits:

• Global deployment: we could offer locations in all AWS Regions within days, while before it would have been a huge pain to find a co-location provider, get approvals, buy hardware, fly there and set up the hardware. Let alone maintaining and growing it.

• Standardization: we offered a standard set of functionality and therefore could operate the apps very well.

• High availability: apps were running on multi-AZ 12-factor apps combined with highly available AWS RDS and AWS S3. Compare this with single-VM apps that could easily be down for an hour in case of hardware failure.

• App scalability: apps could scale horizontally and could be paired with gigantic RDS instances, something that we would have struggled with in v3 because of how tightly we packed apps onto physical machines.

• Organizational scalability: we could upgrade clusters without application disruptions (so no need to align with all customers) and could shrink and grow our cloud without doubling our engineering team. App resizes were also largely self-service, reducing the load on Customer Service.

There were also some downsides:

• Ultimately Cloud Foundry lost the OSS cloud war. In 2015/2016 we decided for Cloud Foundry. Kubernetes was in its infancy and seemed overly complex and not enterprisey enough. Despite that it turned out to be the big winner and we were stuck with Cloud Foundry. For the end user it wouldn't matter, as Cloud Foundry is just the engine underneath Cloud v4 and it can be swapped out for Kubernetes without customers noticing the difference. It was a sane choice at the time and we couldn't have predicted the future.

• It was expensive. I estimate that our infra costs were about 5 times higher on AWS than they were on own hardware because of how optimized we were running things on our own before.

I think the pros far outweighed the cons.

To seasoned engineers this makes total sense, but just to re-iterate what worked well in the process:

1. We kept the feature set of v4 as limited as possible to avoid the Second-system effect.

2. We prepared for the fundamental architecture changes for years. E.g. adding horizontal scalability support to the runtime, and offloading storage to S3 instead of the local file system. These had to be planned long in advance, without any direct benefit.

3. We didn't "shout" about v4 or commit publicly to a launch date until we were very close to ready.

4. We kept the communication lines open between all departments: R&D, sales, CSM, Customer Service, CloudOps. All were actively involved in getting the first customers on v4.

5. We always launched with non-critical applications first to gain experience.

What happened afterwards

After I left, the Mendix Cloud team (now called the Cloud Unit) grew to about 100 people. They continued improving the software, expanding the global footprint and offering dedicated Mendix Cloud regions for big customers. After the last apps on v3 turned off in early 2023, I believe the v4 name was officially retired, but it's still there in the documentation URL: https://docs.mendix.com/developerportal/deploy/mxcloudv4/ . Note that there are 16 public regions now! In v3 we just had four regions, two of which were in The Netherlands. Also note that even 6 years after launching v4, there is still no v5, as the "chaos" of all the previous non-standard versions has been solved, there is no need to have multiple different versions.

My personal take-away

The crowning achievement for me personally were three applications that really needed the scale or fault-tolerance that Cloud v4 offered. The first was the PostNL Order Management System which had a huge landscape that needed to be deployed with horizontally scaled applications. The other was an app for the Huishoudbeurs in 2018, this was a companion app for the fair which needed to scale rapidly with thousands of concurrent users for a couple of days. The final one was a Japanese customer which needed a solution hosted very near Japan for latency reasons. Luckily AWS was there and we could spin up a new region within a couple of days. None of these could have worked on v3.

What makes me especially proud of the product is A) that it took years of disciplined, coordinated effort to make it possible B) that we managed to successfully host v3 and scale with the business needs year over year, while C) also spending just enough engineering capacity to build the much better solution with v4. The team was also really fantastic, I was having the time of my life with these guys.

We did all that with just 12 people.

I still remember this logo fondly

P.S. if you want your own "Mendix Cloud"-like hosting on Kubernetes ask my friend Xiwen who's building low-ops.com .

next: 2018 - The easee Wizard Engine

previous: 2017 - The Reaper

2005 - Dat Narrenschip

Around 2005 I started creating a website for my friend Jaap who deals in antique maps at datnarrenschip.nl

I wrote most of this here: see How we ended up owning a framing shop

next: 2005 - Porting an Atari tax program to Delphi

previous: 2004 - The HP 49g

Shortly after taking over the framing shop, the local newspaper wrote an article about it.

When I was 16 and living in Middelburg my mom introduced me to a bit of an odd man. His kids went to the same school as my siblings and his wife told my mom they needed a new website. My mom suggested I could help because I was always on my computer and she wanted me to do something useful. Thanks mom!

Jaap (as he was called) smoked a lot, had long hair, and always listened to hippie music like Pink Floyd. Most importantly, he had a lot of interesting antique maps which he dealt in. His passion for the maps and the stories he told about them were incredible. His knowledge is encyclopedic, he can tell you exactly which edition of a map it is by subtle variations in the paper. I really liked him and we are friends to this day.

One of the maps he showed me was only one of thirteen surviving editions in the world. It is known that both Columbus and Hitler had at one point owned one. He gave it to me so I could touch it. No latex gloves or glass covers or anything. I was holding it in my bare hands and there was a chance that this map had been owned by either Hitler or Columbus, or maybe both ;). That was a mind blowing experience!

The exterior of Dat Narrenschip in Middelburg

So after some time I was done creating the site, we re-launched it on my php stack, and customers could view his collection. I have maintained the site ever since even though I do more "important" things now. Over the last 20 years it has gone through 2 more complete rewrites (php, php + jquery, now shopify). Now on Shopify it is virtually maintenance free, and if needed, other people can handle it. You can also buy maps online now, which I heartily recommend that you do. I personally have an antique map of every place I've lived in.

The website and hanging out with Jaap started my love for antique maps. In my view they are:

1. Beautiful pieces of art.
2. Full of interesting history, geography, and exploration.
3. Still pretty reasonably priced, you can buy great maps for 500 - 5000 USD.

When I see an antique map I think of all the people that must have collaborated to make this come true. The explorers who went all over the world, the surveyors who measured all the distances, the businessmen who brought the resources together to print hundreds of maps. The workers who carved the copper plates, the millers who made the paper in their mills, the painters who colored the printed sheets of paper. And finally of course the customers who bought the atlases, the rich merchants for their enjoyment or the seafaring navigators who needed proper maps to navigate the world.

Whenever I created something for the website I was paid either in cash or with an antique map. Later I had some more money and I would buy one every other year or so. The one I like most is a 1782 edition of the United States by Klockhoff. One of the very first editions recognizing the new country with their 13 states and their own flag. I think it is just absolutely beautiful, and from just 6 years after the declaration of independence. I often ask visitors to point out Washington DC (which is not on the map because it wasn't built yet!).

The very rare 1782 Klockhoff

Anyway, when I wanted to frame it, the local framing shop in Utrecht asked about 300 euros for it. I thought that was a lot of money, but still had it framed. It's on my wall today and many people who have had virtual meetings with me will have seen it behind me.

A couple years later I bought another map. Coming from a very DIY family, I thought I could frame it myself for way less money. How hard could it be? So I ordered some materials and tried it. It turned out that it was very difficult to get the corners in exactly 45 degrees and cutting the mats (passe-partouts) was pretty difficult too. Making sure the art work is kept safe and the materials are acid free turned out to be a requirement we didn't think about back then. Nevertheless, I managed to do it for about 80 euros and in my ignorance I was happy with the result. Then we framed some more maps together with Stephanie (my wife) and we both enjoyed doing it! Since then she has been selling antique prints and also framing a couple, but always as an extra activity besides taking care of our family, and with non-professional equipment. Which means it was slow and needing a lot of rework to get the quality right.

On weekends she would sell her framed art on fairs etc.

Now that the kids are a bit older, she wanted to go full-time into selling art and framing and began looking for equipment on Marktplaats (the Dutch ebay). She looked for some professional equipment like a Morso miter chopper. To her surprise she found an entire shop for sale that also mentioned it had a Morso. That's how she found it! The owner was asking for a reasonable price for the store. We went over and it looked like this:

The workshop in Soest

Apparently the owner was 88 years old and finally retiring. We went over once, and on the way back decided that we should just do it. We had some extra money as I was freelancing that year. Buying a store wasn't really "the plan" as Stephanie first wanted to finish her professional framing course which would take a year, and I wanted to start my own software company, but the seller was in a hurry and chances like this don't come often. So I decided to work a bit less and Stephanie could do her course and work in the shop, while I took more care of the kids. The deal was finalized quickly.

The Art of the Deal

By now in mid 2024 she is a certified framer and runs the shop about 4 days a week. It's a lot of fun and she's really good at it. You can visit the store in Soest, look up her website here.

Life is a bit weird sometimes, and this string of random events is how we ended up owning a framing shop.

In the summer of 2020, Yves Prevoo had got easee in the first Techleap batch. Pretty cool! I was invited to join one of the sessions for the CTOs in which Ali Niknam talked about his adventures. I didn't know anything about him then but I liked him right away as he also wore a black t-shirt and a Casio F91w, my standard outfit.

I had been struggling with innovating in a highly regulated industry. I dreaded the yearly ISO 13485 audits and always had the feeling we were doing everything wrong or breaking some law that we didn't know about yet. It didn't help we didn't have anyone with Medical Device experience and it didn't help that we had a very strict auditor. To be frank, I really had no idea how to develop anything new and get it certified without going bankrupt from the clinical trials. I didn't know what to do or how to lead my team in an inspiring way because of all the regulations.

Hearing Ali speaking about Bunq in such an optimistic sense was surprising. Wasn't banking also highly regulated? How did he keep the can-do attitude that I lost? So I decided to ask him about this during the Q&A. As I stood up he complimented my Casio and I made some stupid joke that didn't land, but then I asked "How do you keep innovation and fun alive in a highly regulated industry?"

I think he understood the pain where my question was coming from and his answer was short and spot on. This is how I remember it:

"Auditors are not omniscient, they are typically pretty boring and their job is to go through checklists. This is how the system works and it's a necessary evil. Now, it's your job to figure out what's on their checklists and how to give an answer to each item on the list. But don't start by focusing on doing everything right, first focus on what you want to achieve, and then figure out the checklists. After all, auditors are not leading industry advances, we need you to do that."

What a fantastic call to action! I felt very empowered and my whole way of thinking about auditors made a 180 in that moment.

In the next year I felt better, was able to file two innovative patents and ship a lot of improvements with my team. There was still quite some residual fear, and the EU really put a hard break on innovation with the "no significant changes" clause in the MDR.

But now, in every new project I take on in a regulated industry, I see my role vs the auditor's role crystal clear. My job is to innovate, their job is to make sure the necessary checklists are checked.

Thanks Ali!

(this is a cross-post from LinkedIn https://www.linkedin.com/feed/update/urn:li:activity:7200829829997359104/ )

In June 2023 I started freelancing as a Fractional CTO. It's for complicated projects or for companies that don't need a full-time CTO yet. It's been really great. In the past year I've:

• Posted my availability once in my "goodbye to easee" LinkedIn post.
• Been hired by 5 companies, who all found me through my own network.
• Networked more than ever before. I suddenly have time for random meetings throughout the day.
• Have not started an LLC (BV), nor got insurance, nor opened a separate business bank account.
• Have signed only 1 contract for my freelance work (but some more NDAs) and almost never committed to a certain duration of the work. The paperwork takes time (and expensive lawyer overhead) and so far I've been fine with operating on a trust basis.
• Have had all my invoices paid (by at most a week later than the due date).
• Earned 2.5x what I earned before, while frequently working for only 3 days a week and took more vacations.
• Had a great vacation to Egypt with my family.
• Bought a framing shop with my wife where she's now working full-time.
• And maybe most importantly: I've spent much more time with my kids and their school activities than ever before. I've been on school trips, did fun things with my kids after school, was invited to teach my son's class about electricity and inventions.

As for the work, it's always been interesting and most of the time fun. I've:

• Made plans and a technical architecture for a new product proposition, did competitive analysis and made plans to hire a team of about 40 engineers.
• Set up an ISO 27001 ISMS in about 3 weeks, while working 2 days a week. That was a challenge but there was a lot of support from the full-time staff.
• Interviewed about 40 people and hired a couple of them.
• Set up k8s deployment with a helm chart and operator based on kopf, bringing my cloud skills back to excellence. (I had never touched kubernetes before).
• Coached a software team on their way of working and individual engineers on their career tracks.
• Helped management teams get a better grip of their technology stack.

I'm doing this so I can start my own product company, and I don't find being a freelancer really fulfilling because there's no long-term perspective for me. But objectively speaking, going freelance is very very comfortable.

Although social media and climate change are giving us major anxiety, it sure beats hunger, plagues, world wars or nuclear gloom. Humans have evolved with constant discomfort, and now we're comfortable.

This is a reason for boredom and restlessness in the modern world. We want to be challenged and experience risks. Instead of life on easy mode we need struggle.

Perhaps this explains why seemingly normal people that have lots of other options start companies that do not make a lot of financial sense. There's a need to prove oneself, to work really hard, and to experience risks. Startups definitely let you do that. Whenever I see SaaS, I think Struggle as a Service.

My answer so far has always been: "No, bare LLMs are not going to compete with this product." I think that people are failing to understand the distinction between different classes of AI companies. I see it like this:

• Foundational AI - creating models like GPT-4 (text), Sora (video) etc.
• Applied AI - using existing models to create smart applications.

Note that there are also companies not focusing on AI - they will start lagging behind, let's forget about them ;)

There are orders of magnitude (!) more companies that will deal with Applied AI than Foundational AI, and they will be very busy for decades to come. Here's why:

1. To get the most out of LLMs, you have to ask the right questions.
2. LLMs have access to very limited context by default.
3. LLMs are not AGI and agents need to programmed manually.
4. LLMs don't understand specialized problems.
5. Integrating AI in everything is a lot of work.

1 - To get the most out of LLMs, you have to ask the right questions.

In "The Hitchhikers Guide to the Galaxy", a bunch of philosophers decide to ask a computer the answer to "The Ultimate Question of Life, the Universe, and Everything". After eons of calculations the computer famously answers "42" . The lesson is that we have to ask the right questions to get good answers. How does this relate to LLMs? Currently ChatGPT has an easy time looking intelligent because we are driving the conversation and we are responding pretty intelligently in the dialog with the LLM. A large part of the intelligent conversations with ChatGPT are due to the intelligence of the human.

When applying LLMs to a product, we have to be very clear about what answer we want answered, for all users of that product. This is difficult, and will require quite some brain power of the engineers at Applied AI companies.

2 - LLMs have access to very limited context by default

There's a whole world of knowledge in your head whenever you are working in a product. Assume you work in HR and are trying to process a contract change with one of your employees and you want a review of your proposal. Instead of asking the HR manager, you decide to first ask ChatGPT. When you start typing the question into ChatGPT, you will quickly notice you'll need to give it a lot of context for it to give a correct response:

- What was the contract before and why are we making this change?

- In which country are you located and are there any recent/upcoming regulations that might make this contract change problematic from a legal perspective?

- Is this change OK standard company HR policy or is it an exception?

- How will this contract change affect the performance/output/tenure of the employee?

- How fair is this change towards the rest of the organization? How likely is the employee to discuss this with others and what are the possible effects of this?

A lot of this context will be present in the heads of the HR manager you'd normally ask. If we want LLMs to give an answer without all the manual context copy/pasting, we need Agents to pass the right context to it.

3 - LLMs are not AGI and agents need to programmed manually.

Large Language Models are not Artificial General Intelligence. Listen to some Lex Fridman podcasts with Yann LeCun or Sam Altman if you need convincing on this one. LLMs are pretty limited in their abilities and "just" spit out text based on context. Within that capability they are already incredibly useful, but this is only the start. If we do some clever prompt engineering and we allow their response to issue commands, we can set up an Agent structure to perform tasks. One of the most straightforward agent structures is ChatGPT where it acts as a chat assistant to a user. Others can perform Retrieval Augmented Generation (RAG) to dynamically fetch external information into context.

Think of the LLM as the book-smart intern that will be handling questions, they are able to give text-book answers based on their information about the world at large. Think of the agent structure as the manager that gives instructions, checks the answers etc. To get the most out of the intern we need to give it the right instructions, gives it the right context and permissions to look up information and finally check the answers. For the foreseeable future, all this agent logic has to be programmed manually for the specific use-case you want to use it for. Read: quite a lot of work. The good news is that you can expect a TON of improvements and out of the box integration in this space over the next couple of years, but an LLM without an agent structure is not going to be very useful for most products and hard work will be required.

4 - LLM's dont understand specialized problems

LLMs are trained on public information on the internet, and English currently dominates there. There is a lot more non-public information and a lot more non-English content.

One of my friends (hi Fedor!) has a company that applies AI to speed up tender management. Their software scans the documents with details about the tender, and teams can collaborate to come back with a bid. They apply AI to find information quicker and answer questions. This is a very specialized expertise where one line on page 85 of document 14 might be critical to the viability of the bid. There's a ton of very specific terms that normal humans will have never heard of. What's more, the languages and rules vary wildly by country. We can not expect a generic LLM to give expert level answers in this context.

5 - Integrating AI in everything is a lot of work.

We've seen that thinking about the right questions is a lot of work. We've seen that getting the right context and prompts is a lot of work. We've seen that creating Agents is a lot of work. If we do all of these it gives us AI-power in one part of our product. So if we imagine a product that wants a simple "smart input field" that supports AI powered auto-complete, we have now built one "smart input field". Unfortunately your product still has 100 dumb input fields. You'll need to think about how to make each of those smart. Right now each of those 100 fields will require manual work, as you'll have to think about what specific context the LLM should get for that place in the UI, who is able to get what context into the autocomplete functionality (e.g. an admin prompt might get different context than a guest user). Making your whole application smart will take a lot of effort.

Years ago only laptops had wifi. Then came the smartphones, then the TVs, then the thermostats, then the washing machines, then the doorbells, and we're not nearly near the end.

For AI there's a similarly long tail of applications where AI can add value, and we'll be working on incorporating them many many years. It's an incredible time to be in software engineering!

Discuss on HN.

Needless to say, that hit home.

I've seen many LinkedIn posts that go like "here are ten things that take 0 talent: following up on commitments, being on time, ... " etc. . But I think that's wrong. Making sure you "follow up" is hard work. I'm a total pleaser by nature, I want to be agreeable and I dread confrontation. Telling someone "I'm not going to do anything with this" is very difficult. So my natural tendency is to say "Hm yes that's a good point, it would be great if we did something with it, perhaps we can do ..." and then go off and brainstorm something nice in a fantasy world. People go along with this most of the time, because it is very nice to go off into fantasy mode. It's just that at some point they have to admit reality. In this instance it took a couple of meetings to see that nothing was happening.

In the years afterwards, running a regulated quality management system and managing a team, I've had to unlearn my pleasing nature and get down to earth. I never want anyone to tell me again that I'm not keeping my commitments. I've noticed that I'm promising less, I say "no" more, and I have a simple checklist system to track all the things that I actually committed to. I smile a bit when I see other people go into fantasy mode and typically steer the conversation back to reality.

So thanks for the feedback Nilco, I'm definitely not perfect at following up, but a lot better than before.

(this is a cross-post from LinkedIn)

The infamous Joel Test is a list from the year 2000 of 12 things all great software companies do. Since then most companies have implemented Git and CI/CD, checking of three items, so we have some space left in the 2024 update ;) I believe "Do you allow your developers to use AI assisted development environments?" is a necessary addition.

I get that you don't want your source code to end up on some OpenAI / Microsoft / Github server somewhere, sure, but find a way to use your own models or learn to live with it. Note that this is often not the same as "#9 - Do you use the best tools money can buy?" as blocking AI tools is about data security, not money.

So why do I think developers need AI programming tools? I'm a seasoned developer with about 15 years of serious programming experience. In the last 10 years I've done less of it, as I was busy with Product Manager / CTO things. I'm now starting my own company, so I'm programming a lot, and I'm exploring whether some technologies really have a 2-10x impact on my work (supabase does, shout-out to you guys!). I wanted to give AI assistants a try, so I spun up Github Copilot, plandex.ai and aider.chat .

There's plenty of things wrong with these tools: they are often wrong, are slow and the GPT4 ones are really expensive. I have many opinions on how AI programming tools should work instead of the brute-force LLMs we use today, but I'll reserve those for another post. Because even in this early phase I can say that I will never go back to not using them. And this is just the start, it will only get better from here. Programming with Copilot is different from normal programming, a bit like singing a duet I guess. You have to adapt to having a partner, give it the right cues, move out of the way when it's their turn etc. But it takes the boring repetitive tasks out of programming, and I love it. I feel less tired, can do better work, and feel like I'm often going 2x faster.

So if you're an engineer, find a place that does not block you from using AI tools. Ask it during your interview process!

If you're a CTO: find a way to get your engineers to use them, or make one.

And if you're Joel Spolsky, please release an update of the Joel Test, would love to know what you think in 2024 ;)

Discuss on HN

Speech patterns that rhyme are pleasing and resonate with our brain.

Could there be an evolutionary benefit to it?

Cultures without writing systems told stories to pass knowledge from one generation to the next.

As we know from the Chinese whispers game, a lot of information is lost that way.

During oral story telling, information is added, removed or changed, it is wildly unreliable.

In computer systems we add checksums or parity bits to lossy mediums to ensure reliable transmission.

In way, the constraint of rhyming words can be seen as a parity bit on sentences.

The amount of words that can be placed in a rhyming sentence is significantly lower than in an unconstrained one.

Perhaps humans got better at reliable data transmission when they evolved to appreciate rhyme.

The first episode I listened to was by 99% Invisible. I believe that by the time I got back to Amsterdam a few days later I had listened to about 10 episodes. Half a year later and I had gone through 80% of all episodes. I'm a bit obsessive, and as it turns out, picky. For me to love a podcast it has to be:

• Authentic. I have a soft spot for the lone podcaster who loves what he/she is doing.
• Informative. I want to learn new things, and be slightly entertained in the process.
• Orchestrated. I can't stand random people talking about something without a clear direction.

Orchestrated and Authentic can sometimes conflict. It's a sign of good artistic leadership if that doesn't happen.

Since my first podcast in 2018, virtually no day has gone by without me listening to a podcast. Most nights I fall asleep while listening to one, which I resume the next day. Either I fall asleep and it doesn't matter, or I lie awake and learn something new. Both are good outcomes.

Here are the podcasts that I've listened to a LOT. Thank you to all the hosts, producers and other people involved in creating these.

The A / I / O are the aspects above on a 1-5 scale.

A+ - Have to listen to any new episode:

• The Memory Palace - A5, I5, O5
• The Fall of Civilizations - A5, I5, O5
• Hardcore History - A5, I5, O5
• The Anthropocene Reviewed - A5, I4, O4

If you know these podcasts you will recognize a common theme. Nostalgia. Apparently it runs deep with me.

A - Loved most of the episodes but not all, or I'm slightly tired of it by now.

• The Rest is History - A5, I5, O4
• The History of Rome - A5, I4, O5
• In Our Time - A4, I4, O5

These podcasts have taught me so much about the world. I feel a bit of Renaissance man because of them.

The Rest is History is actually A+ to me right now, but I know I'll get a bit tired of it eventually. There's there's just so much! I've listened to about 50% of the episodes now. Around 10% of the episodes are not really interesting.

One of the best gifts I've ever received is a signed copy of the Rest Is History. Thank you dearest wife! Here it is:

B - Absolutely loved these in the past but lately I've stopped enjoying them as intense as before

• 99% Invisible - A3, I2, O5
• This American Life - A4, I3, O5
• The Lex Fridman Podcast - A5, I3-5, O3
• Revisionist History - A3, I3, O5

The style of Revisionist History get tiring a bit too quickly. 99% Invisible and TAL have become a bit too woke for my tastes.

The Lex Fridman podcast is sometimes great, sometimes so-so depending on the guest and the topic. Lex isn't the best interviewer (also definitely not the worst) but the insane level of the guests and high frequency of the podcast is amazing.

C - Pretty enjoyable but too infantilized

• Cautionary Tales - A2, I3, O3
• 50 Things That Made the Modern World - A3, I5, O5

Recently I had the pleasure to work with a CEO and got some feedback that I shared plans with our suppliers too clearly. He said he didn't like that at all and he wanted things to be as vague as possible. My gut reaction was that this was ridiculous. If they don't understand our plans and motives how can they deliver a good service? It goes against everything I've read about leading people.

But, when I reflected a bit, I thought of the idiom "Knowledge is Power".

If knowledge is power, then it follows that the absence of knowledge must be weakness. By keeping the people around you weak, you keep the upper hand.

I don't know which way is right when dealing with your own team or with suppliers (although I know which I personally prefer). But: the insight has opened my mind and I now see the pattern of hiding information all the time, so I am really grateful to the CEO for teaching me this lesson. It gave me a new mental tool to understand how the world works.

• In strategy games, the Fog of War is crucial. You can't see what your enemies are doing and they can attack you from any corner of the map, until you put up scouts, watchtowers, etc.

• By not sharing salary ranges, companies keep power to compensate people how they wish. If they are asking you to give your desired salary, you lose power by giving it.

• In a consultancy company you have no one-size-fits-all product so you might want to have a vague website saying things about "digital transformation" etc. It sounds very important without sharing any information!

• By not publicly committing to a roadmap, your company keeps the power to change its mind when it wants to. On the other hand, some customers might want to see whether your roadmap aligns with their needs and you might not close the deal without sharing a roadmap.

• If you lead ambitious people and you want to keep your cozy management position, you might want to hide information from your team to prevent them from being promoted. On the other hand, if your team is stable, your team's power is the sum of the power of all the members. By giving your team the full information available, you can create amazingly powerful teams that do the right things the right way.

• By not sharing plans with a supplier they might suspect you have other options, and they might work harder to get the contract.

• By setting clear objectives for people and linking them to bonuses, promotions and salary increases, you give your people a LOT of power. If you can't set the right goals this will mess up your organization. By keeping these things vague you keep the power with management. This is often the right thing to do, because setting the right goals for a year up front is downright impossible.

• In project planning, there is a 5d chess interplay of estimations, budgets, deadlines, scope. Think about the role of information the next time you are trying to figure out what is going on.

In stable, high-trust, mission-oriented organizations power is not so important and knowledge is shared freely. In highly political organizations with lots of infighting, power is everything and information is guarded closely.

Knowledge is power.

My desktop has looked like this since 2008. I love the picture of the gearbox, it's a testament to the hidden precision engineering that goes on inside the built world all the time.

I don't see much of the gearbox though, because most of the time my screens look like this:

The environment around my background picture has changed a little more, but has been stable for the last 10 years. My experience is very different to my wife's constantly changing system. In 2009 I moved from Windows to Ubuntu, then to Debian using Gnome. Then finally to i3 in 2013 after a brief affair with XMonad in 2012. So by now in early 2024, I've had the same minimal UI for more than 10 years, and if I'd only had the guts to adopt i3 earlier it could have been 15. I have made small changes but if I didn't like it I would revert back to the previous version. All the changes were conscious efforts and I had the freedom to go back when I wanted to.

This short post is in praise to boring stable systems, and a thank you to all the open source contributors that made my desktop environment possible.

• OS: Debian (since 2011)
• Window manager: i3 (since 2013)
• Browser: Chrome (since 2008)
• Editor: vim (since 2011, now actually neovim but I haven't noticed a difference with vim)
• Terminal: gnome-terminal (since 2009)
• Shell: zsh / oh-my-zsh (since 2014)
• Mail: Gmail (since 2005)

Here are some new improvements to my home computer systems.

• Backup management: syncthing (since 2022)
• Home automation and energy management: home-assistant (since 2022)
• Microcontroller management: espthing (since 2024)
• Shell history management: atuin (trying it out since early 2024)

Being boring with tools is great. It gives you the freedom to be revolutionary in areas where you actually want to make a difference.

2021 - The WaterRower Camera

I was getting more and more heavy during this period, almost reaching 90kg / 200lbs. I never exercised but I had bought a rowing machine with a water tank because I wanted to exercise, in principle. The machine was bought in 2019 just before the pandemic and during the boring early pandemic months it was a welcome way to distract myself. For some weeks I started rowing like a maniac but of course it wasn't sustainable. The machine started collecting dust.

It had a small "workout computer" displaying the stats, but it was very dumb and not connected to anything. For some reason I thought it would be a good idea to measure my stroke speed, depth, and power more closely. The (black) elastic band is wound up around a cylinder. By painting the cylinder white and tracking with a camera how much black/white pixels are visible, it shows how far you've extracted the cord. Doing that a lot of times per second you can see stroke depth, speed, power, and recovery speed. I attempted to do this with a cheap ESP32 board with a camera but found that I could only get 1 FPS due to the way the library worked. I have since learned more about interacting with the hardware, and think that I can get it much faster because I only need to count pixels that match a certain threshold in brightness.

Anyway, that would take a lot of time. Instead I hooked up a raspberry pi zero 2 w and reached 60FPS with brute force. Then I built a simple web interface which I used to exercise about 5 extra times. It felt really good to exercise and have it monitored intensely at 60FPS.

Then my wife's cousin Johannes came by, saw me rowing and laughed his a** off. Apparently my rhythm was completely wrong and I should do short and very fast pulls, then a slow recovery. I had taught myself rowing in the wrong way and now couldn't get used to the right away anymore. I haven't rowed since, but I discovered that running is very nice. In the summer of 2022 I lost 13 kgs and got down to 77kg and feel a lot healthier.

next: 2021 - FDA Crawler

previous: 2020 - QMS Integrity Checker

The remote control was nice, but it's a separate one from my tv. Using the optical toslink output the volume buttons on my TV do nothing. Annoying for me and my family and very confusing for guests who wonder why the TV has no sound.

I started using some esphome microcontrollers for various home automation tasks, and I got an idea. Why don't I listen on IR to my TV remote and if the volume buttons are pressed, mimic the Aiyima remote and send the volume signals there too. I actually ordered some IR Transmitter and IR Receiver break-out boards some time ago but never used them before.

On a Saturday night I hooked everything up and started debugging the signals from the Philips and Aiyima remote controls. I took a quick look with my oscilloscope but then found how to come to a conclusion quicker using this blog.

The Philips one was easy enough and showed up right away. The Aiyima uses some custom prefix and codes, or at least one not known to the esphome libraries. Many manufacturers use overlapping protocols (even though the keys might not overlap, not all the address space is used) so it recognized JVC, Pronto, Pioneer and some other codes.

Through trial and error I got it to work by using transmit_pioneer 0x4d80 through 0x4dff.

So if you ever want to accomplish the same, here are all the buttons on the remote mapped out.

Action - #nr - pioneer equivalent hex code

bas- 0 0x4d80

tre- 1 0x4d81

mode 2 0x4d82

bas+ 8 0x4d88

previous 25 0x4d99

next 65 0x4dc1

tre+ 69 0x4dc5

vol+ 74 0x4dca

mute 78 0x4dce

vol- 82 0x4dd2

off 92 0x4ddc

Having examined how similar the different protocols are, I feel like there might be a better way of doing this than emulating pioneer but I'm happy with this solution for now.

I was hoping that there would be secret codes on the amp that were not present on the remote control, to idempotently switch off the amp, switch to a specific input or go to a pre-set volume. That would be great for some more automation. I tried all 256 codes from 0x4d00 to 0x4dff but did not get any results. It's possible that an idempotent "switch on" is present but I didn't want to try all codes twice. I thought of Shenzen IO and realized that of course no one had time for this. That's what I deserve for always saying YAGNI to my engineers.

If you want to plug this into esphome, you can do it like so. This contains just one action but the rest can be extrapolated from the table above:

button:
- platform: template
    name: amp mute toggle
    on_press:
      - remote_transmitter.transmit_pioneer:
          rc_code_1: 0x4dce # amp mute tuggle

Update on January 4th 2024: In the end I managed to fix the light. Check the bottom of the post for the conclusion. Thanks to reddit user icannotfindagoodname for the help in finding the issue!

After chatting with Cowboy support they offered me a new battery for 490 euros and that they would also send an offer with a discount, which appeared a couple days later. The discounted price was 367 euros. A bit much for a broken LED module! The battery was still totally fine.

So I'll attempt to fix it myself and I'll show you how to disassemble the battery here.

WARNING - DO NOT DO THIS AT HOME

Now I need to warn you to not try this at home. I have been working with electronics for a long long time and I know what to do and what not to do, and most importantly, know what I should not even try. Battery packs are not a joke and could lead to serious fire and health problems if not handled correctly. Real professionals will probably even look at my photos and say that I'm an idiot for doing this in my living room. I do not assume any responsibility for anything you do in your life, including opening up dangerous lithium ion battery packs.

WARNING - DO NOT DO THIS AT HOME

We start with the 4 screws at the bottom.

Great success.

Then remove these 4 screws here.

Here it gets complicated. There's only a tiny hole through which to operate.

I also pried open the top thinking that there would be easy access to the PCB connectors, but no. It's just the cylinder lock. The two compartments are totally isolated. No need to open this up unless there's any problem with your lock. Coincidentally the tiny hole that looks like a "insert paperclip to reset CMOS" hole is just a water(?) drain. There's no reset button behind it.

I also removed the long screw, which you DO NOT NEED TO DO. It's just for the structural integrity of the battery pack and is not attached to the frame.

After a lot of prying through the tiny hole I managed to disconnect the connectors and remove the outside interface. This was the hardest part of the disassembly! It's really difficult to operate through a tiny hole and still be careful because you're near the main output terminals of the battery pack.

Now only the light is still attached. If you push REAL HARD through the hole you can eject the rear light which is glued down. Then gently remove the connector for the light. I didn't figure out how it worked before so I slightly damaged some plastic of the lights. All in the name of science I guess.

Ok now how do you get the battery pack out of the enclosure? The fit is very tight and there's no surface to apply pressure to. You can't push from the inside and you definitely DO NOT want to stick metal tools anywhere near the cells. The side with the PCB has the most clearance from the edge of the frame. I figured that by slamming the frame down onto a hard edge (like the side of a workbench), the momentum of the cells will do the work and slowly slide out. And after 4 or 5 "gentle" slams the pack was indeed ejected.

Here are some shots of the pack and the PCB.

The white "paste" covers the leads that go to the batteries. I'm not replacing any cells so I'm not touching it.

There's a small push button here. Not sure what it's for.

Also an on/off switch for the entire pack.

Here's the pack.

This is where it's attached to the rear lights.

And here next to the enclosure.

Ok now we get to what we came here for...

The rear light module in all its (broken) glory.

It's easily pried open.

Alright so here we just have 8 LEDs. It seems the voltage is about 1.5V tot 2V per LED.

Now I tested my LEDs and they all still work. Hm.. Could be the rear light module or the signal from from the bike's controller.

The red and black terminals of the LED module are hooked directly to the main battery terminals, so there's some power modulation from 36V down to a usable level for the LEDs. The green wire is controlled from the main PCB in the frame of the bike, the connection goes through one of the small terminals of the battery connector and then directly to the led module. The signal is a 3V PWM at 1khz. So it's probably the LED module that's broken. This is actually the first time I'm seriously using my oscilloscope and it was awesome!

The LEDs are wired in serial. When lighting up the first or last 7 of them with 12V it works, but when connecting all 8 of the LEDs I got a short circuit, which means that there's something wrong with my LED PCB. I suspect it's the power inductor that overheated and will try to fix it.

Here are some close-ups of the LED PCB.

Update: after some helpful comments on the reddit thread and chatting to user icannotfindagoodname, I desoldered some components and tested them in isolation. The diodes and resistors all seemed fine but the fuse (F1) showed weird behavior. As I was seeing a short if I put a voltage across all of the LEDs I traced the circuit to see where those points connect. This turned out to be the capacitor. I shorted the fuse and put in a beefier 10 uF electrolytic capacitor I had lying around, I applied 32V and a PWM signal and voila, there was light!

I ordered new capacitors, a resettable fuse and a new 100uH inductor (the specs of the ZXLD1362 LED driver suggested a bigger one than the 68uH that was used. The battery voltage can apparently reach 42V.)

Here is what I ordered at Farnell:

It arrived within 2 days, pretty cool. Now I put everything in place and re-assembled the battery.

I put in the LED module last, so that I could reach the connectors through the hole. Then I glued it shut using hot glue. Then I noticed some small cracks in the black plastic enclosure for the LED module. This probably happened because of the force I applied when removing it. As another reddit user pointed out that his battery died in the end because some water got in that way, I put super glue on the cracks to hopefully prevent the same fate. On close inspection the glue is visible and it doesn't look as clean as before, but better than the entire battery becoming e-waste.

After the whole ordeal the light seems a bit more dim than before, though I might just be imagining that. It could also be because of using a slightly different resistor at R4 because I damaged it while desoldering and testing. I really need to buy some SMD soldering equipment.