Way back in the age of the dinosaurs, if you wrote a program and wanted to test that it was correct, you mostly had two choices: you could either manually construct malformed inputs and try them in succession, or hook up a program which would randomly generate inputs and try them automatically ¹. Those programs were called fuzzers, or sometimes generators, and this was “fuzz testing”. It also for quite a long time wasn’t very good: the generator would have no feedback from the program under test except “did this random input crash or not”. This feedback, although useful, doesn’t allow the fuzzer to tell if it is making progress relative to the last random input or doing a good job, and so your fuzzer would just repeatedly hit the same shallow code. You’d leave it running for a week and then when you come back realize it was never generating inputs that would hit interesting parts of your program, and they were all immediately being thrown out or never progressing past a if(packet_header == 0xdeadbeef) check.

American Fuzzy Lop revolutionized the field by introducing coverage-guided fuzzing. Instead of having the only feedback to your fuzzer be whether an input crashed or not, it also recoded what code coverage it hit over the course of a program execution; if an input executes a piece of the program that it never reached before, then it hit new code, and probably is now able to also test some new interesting surface area of the program which might have some bugs. This was immensely more effective than “dumb fuzzing” of yesteryear.

AFL collects coverage for programs by compiling the program under test with a custom clang compiler pass, which adds coverage feedback on CFG edges: if you hit a new basic block in the code, you get a new coverage bitmap entry set². If running the program on an input sets results in a bitmap which has new bits set than previously, it hit new code; that input can be saved to use as the seed input to mutate for future executions.

You can also do feedback based fuzzing for blackbox binaries that you don’t have the source code to. AFL++ for example has a qemu_mode backend: it executes the program under test with qemu-user, which will lift your x86 assembly to QEMU’s TCG IR and then lower it back with its JIT compiler to x86, but now which can safely execute under its type-2 hypervisor environment. AFL++ has a fork of QEMU which adds a custom TCG hook that adds coverage instrumentation to blocks in the middle-end; the IR for each block is extended with TCG operations that write a coverage bitmap entry the same way that AFL enhances clang IR with extra LLVM operations. Despite sounding like a lot of work, this qemu_mode is only like ~3-5x slower than running a program natively.

The gold standard for blackbox binary coverage is to let hardware do it for you. Instead of needing to add extra operations to the program under test, which need to be executed when you run a fuzz testcase, you can instead use Intel PT. Enabling certain performance counter configuration registers will start processor tracing: your CPU will then automatically record metadata as part of its execution pipeline, and the fuzzer can look at the resulting metadata to pull out a trace of what basic blocks it executed. This is unreasonably effective, with Intel PT based tracing only being something like ~10% slower than normal native execution.

The issue, of course, is that if you’re like me your laptop has an AMD processor and not Intel. AMD have their own magic performance counting features, including a branch trace buffer that records what code is being executed, but its 1) much worse documented 2) much slower than Intel PT 3) impossible to use in a precise manner - that is, the counters are either sampling based and so only give you 1/n events, or may end up dropping events if a metadata buffer fill up before you’re able to drain it. In either case that makes it sadly unsuitable for fuzzing (you technically could build a fuzzer on top of merely stochastic coverage, and it might even not be that bad. But in practice no one has done so, and it’s probably not worth it.)

Which brings me to my bad idea. I wanted to write my own fuzzer for a side-project, and so wanted to collect precise blackbox binary coverage. But didn’t want to write my own dynamic recompilation engine (again), or buy a new laptop. What is a software engineer to do?

The Wrong Way

We can instead instrument the guest code by replacing all the branches with INT3 instructions. Wait come back I can explain.

In order to collect precise coverage we need the program to do something whenever it hits a branch. We can’t insert new behavior in each block, because it already is using all of the bytes in the block for its normal instructions: trying to resize blocks requires fixing up code pointers, and then you’re just back at QEMU-but-bad. The minimal length of a branch instruction is two bytes long: Jcc has a form which is only a single opcode byte plus a displacement to branch with. INT3 is only one instruction long, hijacks control flow in a way that doesn’t clobber GPRs which would prevent continuing execution after the instruction, and gives us the location of the branch that we overwrite as context so we can compute the coverage bitmap entry that we want to record in our coverage map. It’s perfect!

Unfortunately, if you tried to do this on Linux with ptrace you’d probably want to die. Linux will turn hit INT3 into a SIGTRAP. That can be caught by a ptrace based debugger that is supervising your process. That signal will include the tracee’s PC and register state, which you could use to implement the coverage hook, and then PTRACE_CONT to continue the process. It’s just that this entire dance would be several context switches: guest program in ring 3 -> INT3 trap to x86 interrupt handler in ring 0 -> Linux kernel delivers a ptrace event in your debugger in ring 3 -> your debugger PTRACE_CONT via syscall to ring 0 -> Linux kernel syscall implementation resumes the guest in ring 3. Each of those context switches are very expensive, especially because context switching between privilege levels requires even more work from your processor than same-ring context switches. Oh, if only there was another way!

The answer, of course, is virtualization extensions. Your computer can run a smaller computer inside of it through the magical power of Intel VT-x (or, as we call it on Linux, kvm). The code inside the smaller computer thinks it is running completely bare metal: that guest code can configure interrupt handlers and MSRs and paging however it wants, just like a normal kernel, and even run userspace code under the guest kernel ³. All while, in reality, the guest kernel is being ran in a virtualized machine hosted by your normal Linux kernel; you can configure it so that when the guest kernel tries to do I/O to the real world, they actually trap to the host kernel, which turns them into a userspace event that the “virtual machine manager” can implement and then resume the virtual machine with the result of. Crucially, while the virtual machine is executing, it is executing bare metal - your processor is actually running all of the instructions for the guest VM, and they all use the normal processor state like general purpose registers and hardware page tables, just with some extra bits that say to also do a hypervisor translation for the physical memory so it can’t reach outside the bounds of the VM and things. The important thing is that it means the VM runs at basically 100% native speed, except for when it needs to VMExit back to the host kernel because the guest kernel tried to do something that traps to the hypervisor. It turns out this works so well, in fact, that basically the entire internet runs on top of kvm in the form of cloud based hypervisors like Firecracker.

TinyKVM is a neat project by the author of Varnish⁴. It implements a VMM as a C++ library, which you can use to run arbitrary Linux binaries: it sets up a minimal guest VM with the required x86 processor state to execute usermode code, with a tiny guest kernel that has interrupts which mostly just trap to the VMM in the host. That means that you can run a guest binary in the VM, and when it does a syscall like write it bubbles up to a VMExit, the host VMM sees that it did a syscall with rax=1 and all the other GPRs for arguments, and then do its own write syscall with the real FD and arguments on the host.

This doesn’t automatically lets us solve the issue of too many context switches for the INT3 hooks. But because the guest kernel is a full kernel, it can configure the interrupt handle that INT3 context switches to however it wants - and I was able to fork tinykvm into something I call TinyCOV, which hacks up the guest kernel a bit. x86 actually has the privilege level for interrupts configured as part of the IDT data structure describing the interrupt gate! That means that we can have INT3 trap from the guest userspace code to the guest kernel interrupt handler but stay in ring 3, and even put our coverage hook implementation directly in the guest “kernel” code to be executed before returning from the interrupt back to continue execution. This cuts down each INT3 coverage hook down to only two context switches, to and from the guest interrupt routine, and zero privilege level changes. And it means we never even need to VMExit either, because the guest kernel is able to handle the interrupt entirely itself instead of needing the VMM to emulate some privileged operation.

Because we overwrite the branch itself with an INT3, just returning from the interrupt back to where we trapped doesn’t actually work and we’d end up skipping the actual branch operation. Since INT3 is only one byte but the minimum length of a branch is two however, we can use the second byte to record some metadata for the branch hook: we can store an index there, and then use the index plus the offset of the INT3 into the page to uniquely identify the location, and then JIT compile the same branch instruction that we overwrote (but with some jump instructions for either side of the branch back to the original successor instructions). Handling dynamic jumps, like JNZ rax is a bit trickier, but still possible.

Since we still have the ability to VMExit from the kernel back to the VMM if we want, we also are able to leverage that in order to implement code discovery: tinycov hooks branch instructions a block at a time, starting from the binary entrypoint. It writes a sentinel value as the metadata byte for the INT3 the first time it hooks a branch, and then in the guest kernel traps to the hypervisor if it sees the coverage hook still has the sentinel value and thus its the first time it’s ran. The VMM is then able to disassemble the successors of the basic block and hook their branches, pushing the code discovery frontier forward, and replace the sentinel value with the correct trampoline index value for the JIT code so that future hits of the branch can stay entirely in the guest kernel.

Results

Despite how scuffed all of this sounds, it does actually work: I’m able to run random binaries under tinycov and collect a coverage bitmap of code that was hit, plus a mode to instead generate a full program trace. Because it works by hijacking control flow entirely, we can even put arbitrary code in the INT3 kernel handler: I was able to implement CmpCOV style dictionary building by recording if a branch ends in a cmp instruction and recording the register operand value, since the kernel interrupt handler (and the VMM VMExit handler) have access to the full set of GPRs from the program under test synchronously, which you can’t quite do with Intel PT style asynchronous coverage traces.

That said, how fast is tinycov compared to QEMU? The answer is, uh, really slow. I ran some benchmarks, and it’s like 700% slower - and worse on branch heavy code like bzip4, which will be taking more INT3 handlers per instruction ran. CPU out-of-order pipelines really don’t like taking an interrupt every few instructions, even if the context switch stays at the same privilege level. I am happy that the idea of using userspace interrupt handlers under kvm works; I did some benchmarks and it’s like 50% faster than using “proper” ring 3 -> ring 0 interrupts instead! But even 50% faster than doing it the slower way is not enough for this to really be usable when QEMU is so much faster.

Technically, there are some neat things that this would enable you to do which you can’t with QEMU or Intel PT based coverage. For one, because we’re trapping to a single coverage hook which is actually implemented in native code, it’s really easy to add custom coverage hooks: for some fuzzing targets you want to introduce some extra state into the coverage bitmap calculation in order to get the fuzzer to treat identical control flow traces as new program states to continue exploring from, like if you’re fuzzing a language runtime with a virtual PC or some other data-driven loop. If you want to do this with QEMU you have to either write your own TCG hook (hard), or set code callbacks which end up taking a software-VMExit from the JIT code (slow).

Because we’re hooking code in place we also have the ability to unhook code, which I think you could do something cool with. With QEMU the codegen will emit direct branches between JIT blocks to be fast, but the instrumentation being emitted in TCG and thus part of the translated block instructions means that you can’t remove the hook in a block afterwards as you’ll still have those incoming edges. With tinycov we could remove the coverage hook from branches which we decided “aren’t worth it”, or that we have seen both the true and false exits for and so know that it can’t ever lead to new code - meaning that those branches can be rewritten back to their original bytes, and so execute at literally native speed again.

In practice, I think this is mostly just a toy and cute idea, although I am pleased that it works at all. I’ll probably try plugging it in to be used as the coverage collection implementation for LibAFL just to see how much worse it does than QEMU as part of a full fuzzer, but unfortunately no one should really try to use this in production. It was pretty fun to get working though, and I was pleasantly surprised how easy it was to hack up tinykvm to support the functionality I wanted. Any excuse to do very cursed x86 kernel dev things in a side-project, and staying small enough to actually finish, is a win in my books.

Footnote:

Lunacy is a Lua 5.1 interpreter I’ve been working on as a side-project in Rust. Unlike the average Lua interpreter side-project, Lunacy does type specialization of bytecode operations via Lazy Basic Block Versioning (LBBV), plus includes a Just-in-Time compiler.

LBBV is a pretty cool compilation technique that came out of Maxime Chevalier-Boisvert’s research while at University of Montreal, culminating in her PhD thesis, around developing a JavaScript JIT, Higgs. Critically for me, her approach focuses a lot on simplicity: the goal of LBBV is that it is an effective compilation strategy that can be implemented by just one PhD student in a reasonable amount of time, or one engineer as a side-project.

The original LBBV paper is very approachable, and I highly recommend reading it. To summarize, it enables optimizing bytecode operations, such as ADD, which naively would need to branch on the types of its operands to figure out if it’s adding two numbers or two strings or what have you. Whenever the operation would inspect the type of an operand and check if it matches a type, it can statically be given a pass/fail based on a specialization context for the basic block. If the context doesn’t know the type of the operand, instead of needing to conservatively fall back to a dynamic operation which can handle any value, compilation can instead be suspended to a thunk: the thunk will eventually be forced by execution hitting it at runtime, at which point you can continue compiling with the context now filled with the runtime type of the value promoted to a static type, guarded by a runtime check to validate that later executions of the same codepath use the same type. In the failure case of the guard, you can suspend another thunk that will specialize the operation for the next runtime type you see.

This is very powerful! The end result is that you have blocks which seemingly by magic have type guards only for the types of values that actually were observed at runtime, and for things like loops “for free” unroll one iteration which contains all the type guards, and then have entirely known types inside the hot loop (because that block reached a fixpoint of the specialization context for the loop header, and so reuses the block to form a loop). Because you have the powerful primitive of blocks specialized for a static context based off observed runtime information, you can stuff whatever information in the static context you want and always have a correct “guess” for the value of speculative information. And if you ever end up with too many blocks, no matter what reason, you can fallback to running generic code instead of having exponential block blowup.

Comparison of Lunacy and Higgs specialized blocks for sum. Notice how the inner loop body, block 7, ends up with fully specialized ADD operations for the accumulator and loop induction variable. Lunacy doesn’t do small-integer optimization and so has no overflow checks.

Lunacy does a few things different from Higgs. For one, Higgs is a pure JIT: it has no interpreter, and immediately compiles all bytecode operations directly to assembly. Lunacy instead is implemented as an interpreter first and a JIT second: bytecode operations are implemented as Rust coroutines, which yield effects. Those effects may be things like “guard on stack slot X being type Y”, which are resumed with results like “the slot matches/fails the guard” - and like LBBV, if the type isn’t currently known, we instead suspend the coroutine inside a thunk via a closure. It may also yield an effect like “add operands A and B together”, which can be implemented so it only cares about how to add numbers since the coroutine would only reach the yield if the required guards matched.

Runtime behavior is emitted as separate residual operations, separate from the actual Lua bytecode operations which are implemented as coroutines that are driven at compile time. In order to avoid having to define and implement runtime behavior for every combination of operand type etc., Lunacy also uses a closure generating interpreter: complex runtime behavior, like adding two numbers, is instead implemented by a closure that captures the bytecode operands such as the stack slot indexes it uses, and at runtime we execute the closure in order to perform the operation. The interpreter repeatedly executes the residual operations that were the result of compiling blocks by running all of the coroutines for the Lua bytecode.

This approach means that the bytecode coroutines in Lunacy were very easy to implement. All of the behavior for an operation, including the different Residual::Exec variants, are located in a single coroutine which intermixes the static compilation portion via yielded effects and the runtime execution portion via closures which can capture arbitrary static information. Because Rust coroutines even implement Clone, when we suspend a coroutine to a thunk in order to discover extra information from the runtime behavior we can even push a clone of the same coroutine suspended at the same yieldpoint down into the failure case - essentially forking it, and each fork independently being resumed to compile the remainder of the operation and block with a different observed type. The interpreter itself is very small and simple, because it only has a small set of simple residual operations to handle.

This residual scheme also made implementing the JIT side of the interpreter very easy as well, which was a large portion of why I chose to use closure generation as well: because our set of residual operations is very small, only things like type guards or executing a closure or executing a thunk (itself also a closure), for the JIT we only need to handle how to compile those same small set of residual operations. Instead of needing to maintain a second assembly implementation for all bytecode operations, we can emit a static call to the closure function pointer address in the JIT, which crucially eliminates dynamic dispatch and other things that make modern OoO processors sad. Meanwhile, the remainder of the residual operations such as runtime typeguards we can implement in assembly in ~4 instructions as a native branch, which also can get their own branch predictor entry. The actual JIT is tiny, and mostly consists of some DynASM-rs assembly in a match.

Because Lunacy is a template JIT where most of our operations are function calls, it doesn’t do any register allocation. It just pins the interpreter state to a register in order to move into the function argument ABI for the closure call repeatedly. However, because the closure call ABI has a number of preserved registers, it does leave open opportunities to do Copy-and-Patch style register windowing using those preserved registers in the future and different residual operations, or use multiple different copies of closures which expect arguments in different registers we can select from during JIT emission.

Unlike Higgs, which immediately JITs all code, we only trigger the JIT after a block hits a hotness threshold because it’s been executed enough times. This means that we will have forced all of the thunks on executed codepaths, unlike Higgs: Lunacy currently doesn’t invalidate or modify JIT code at all, which to me is much easier to reason about, because of this. Because the interpreter and JIT are both running the same residual operations for blocks, bailing out of the JIT back to the interpreter is very easy: the PC for the JIT bailout is the same PC that the interpreter should continue execution at. Thunks which weren’t yet forced when we trigger JIT compilation are handled as bailouts back to the interpreter, which will run the thunk as normal and rewrite it to a jump residual to a new block: if the new block is ran enough jumping to it may end up triggering JIT compilation again. In the future if the JIT->interpreter->JIT transition for thunks ends up being a bottleneck we could fixup the bailout to jump to the newly JIT’d block as well.

JIT code for sum. Each residual exec operation turned into a native CALL to a static address, with a static closure object pointer. Despite this looking pretty bad for a JIT, it’s still several times faster than the interpreter!

More context

Lunacy doesn’t only specialize the types of stack slots in the specialization context, but also will 1) track the concrete function value for function values 2) specialize table keys to static types and constant offsets.

The first one is fairly basic: just like we can suspend execution in order to discover the runtime type of a value, we can also suspend execution to discover the function pointer of things that we know are functions, again emitting another guard on the pointer value matching our expection. We can do this for both native functions, such as print, as well as Lua functions: this not only lets us in the JIT emit a native call to the C function directly, but also avoid needing to dynamically check the value and switch behavior depending on if its a native call or not, and lets us emit nested calls to more JIT code using the native processor stack instead of the interpreter call stack if the Lua call target is also already JIT’d. Because we can use all of the existing LBBV infrastructure in order to promote dynamic values to static context, this was surprisingly easy to implement.

For two, it was slightly more difficult. V8 implements table specialization in the form of hidden classes with shapes and transition maps: tables are assigned a pointer to a shape, which describes the layout and type of the table, and then at runtime you can emit a shape guard if table.shape != expected: fail; success in order to then emit loads from the table, which result in typed values, at the offset described by the shape in order to avoid hashmap lookups in the JIT. Tables are assigned shapes incrementally based on the order they are built. Code such as

local t = {}
t.a = 1
t.b = 2

would repeatedly transition the shape of the table according to a series of maps saying the next shape for each key that was added, such as

{} -> {a: int} -> {a: int, b: int}

I didn’t want to implement this in Lunacy, however. Because shapes describe the entire layout of a table, it means that any inheritance-like behavior must be compiled as polymorphic blocks: Dog and Cat have different shape pointers, and you could only insert a guard that they are both Animals by implementing more logic that finds the common ancestor and walks shape pointers at runtime, and figuring out when to do that as part of LBBV where all compilation has to progress incrementally (unlike a method or tracing JIT, which looks at multiple blocks at a time upfront) appeared hazardous. Shapes also requires following the transition maps at runtime if you have dynamic keys, and V8 also implements things like shape invalidation which requires being able to migrate JIT stack frames back to the interpreter (which itself necessitate emitting stack maps) to handle some cases.

Because Lua function environments, which contain non-local variables, are themselves tables I would also have the issue of shape pointers transitioning even from unrelated keys potentially invalidating the shape guards for every function just because a new global was added late, requiring a separate scheme to also be added to handle environments.

Instead, Lunacy implements something closer to LuaJIT’s hash slot specialization where the static context only contains the individual keys that are accessed in the block. A function which only accesses properties for an Animal would specialize on the hash slots for those properties being the first discovered type and offset, and for a Dog and Cat which were constructed in the same order for those shared prefix of keys be able to use the same specialized blocks. Likewise, adding a new variable to an environment wouldn’t cause existing guards to fail because the type and offset for all the previous keys stay intact.

LBBV IR for local t = {} t.a = 1 print(t.a). gettable_href does O(1) access with the hash witness index that was initialized by href_init, and the stack slot containing print was specialized to the concrete function pointer.

I did slightly tweak the LuaJIT scheme, adding a runtime “hash witness” table that is populated with the dynamic index of the key instead of including a constant index based on the first observed key index; the idea was that animal.age might be at different indexes even for subclasses because the hashmap storage resized based on different number of keys, and so initializing the offset the first time you see the key in each function would maintain the desired behavior. The hash witness table also includes a “table epoch”: like the shape transitions, when Lunacy sets the key on a table and can’t prove via static information that the value stays the same type, it increments a monotonic epoch counter, and the initial value of that epoch counter is saved when we initialize the hash witness entry. The epoch is needed in order to catch invalidation of table keys, where t.a = 1; t[dynamic] = "string"; t.a + 1 can’t assume that t.a stayed a string, and we must emit a guard that if t.epoch != witness[0].epoch: bail; success to verify the table stays the same. If the epoch increments, because the table had a new key added, we can do an “epoch repair” operation which do a full check that the value is still the expected type and either update the witness epoch value to the new table epoch now that we’re sure it’s still safe, or fallback to compiling a new block with the newly observed type.

But Lunacy now instead just uses IndexMap to implement the hash part of Lua tables, which have stable indexes based off insertion order, and removes the issue of indexes potentially being unstable based off resizing; I will probably remove the index from the witness and emit the constant index in the JIT directly, which will remove one data-dependent load from table property accesses while only introducing a few redundant blocks. The epoch scheme may also not be needed once I move to a better value representation (NuN boxing) which would be able to check the types of values in a single instruction, and so be as simple as our epoch guard is.

JIT IR for: local function say(animal) print(animal.name .. " is ", animal.age, "years old") if animal.barks ~= nil then print("bowwow") end end. Despite calling it with local dog = {name = "Fido", age = 5, barks = "a lot"} and local cat = {name = "Mittens", age = 3, meows = "sometimes"}, the first print is monomorphic instead of needing to specialize per-shape.

The other wrinkle with doing table shape optimization is around aliasing. If you have x.a = 1; y.a = "string"; x.a + 1, even though you never saw a store invalidate x.a it may now have the wrong type if x == y. The initial thing Lunacy was doing was just emit the epoch guard on every table access to catch these types of aliasing hazards; I eventually discovered the PyPy does a “type-based aliasing optimization”, where you can infer that y.b = "string" can’t invalidate x.a, because they’re different keys or different types - V8 is likewise able to do the same thing based on two variables having different know shapes I believe. With this we’re able to elide a lot of redundant epoch guards. Next up I want to try implementing caching of table property values in registers in the JIT, in order to remove redundant loads and perform store-to-load forwarding. Ideally this would allow code like t.a = t.a + 1 to stay entirely in a register without needing to touch memory inside a hot loop. This likewise has aliasing hazards like we had above for type stability, where print(other.a) potentially may observe the delayed store to t.a if other == t: I need to mess around to see how actually feasible this is, and how often these kinds of aliasing hazards would negatively affect benchmarks.

Performance

It’s not great?

Summary
  ./target/unsafe/bench nbody.bin 10 ran
    1.01 ± 0.02 times faster than lua5.1 bench.lua -- lua_benchmarking/benchmarks/nbody/bench 10
    1.11 ± 0.02 times faster than ./target/release/bench nbody.bin 10
    2.77 ± 0.04 times faster than ./target/interpreter/release/bench nbody.bin 10

Summary
  lua5.1 bench.lua -- lua_benchmarking/benchmarks/life/bench 1000 ran
    1.66 ± 0.07 times faster than ./target/unsafe/bench life.bin 1000
    2.03 ± 0.06 times faster than ./target/release/bench life.bin 1000
    2.62 ± 0.05 times faster than ./target/interpreter/release/bench life.bin 1000

Currently, Lunacy runs about the same speed as PUC Lua 5.1 on nbody. nbody is heavy on table property accesses, and so we sure would hope we do better than PUC Lua, which will do actually-for-real hashmap lookups each time. Other benchmarks are a bit slower. This is very modest performance - but it is in fact occasionally faster than vanilla Lua, which was my initial end goal and I consider an achievement.

More notably, Lunacy’s interpreter-only mode is about 2.75x slower than Lua, which means the JIT does actually work in being substantially faster than the interpreter, which was my second big goal!

Why is it slower? Profiling is difficult, but it appears to be a mix of:

• Our value representation is bad, and is literally just a Rust enum. Lunacy doesn’t do any pointer tagging, NaN boxing, or (the goal) NuN boxing currently - so our values are 2x more bytes than they ideally would be, which makes everything slower and precludes trying to keep them in registers as each value would consume two of them.

• The LBBV specialization is kinda slow (~20% of runtime), in particular because the hash slot specialization creates a lot of thunks despite many of them being immediately forced. It is probably possible to avoid repeatedly switching back and forth between running and compiling for some of those thunks.

• Likewise, the LBBV specialization clones a bunch of contexts and coroutines everywhere, and uses a lot of HashMaps. These can probably be optimized a lot as well to bring down the 20% overhead number.

• The table hash slot specialization does a full hashmap lookup to initialize the hash witness index, and needs to load the index from the hash witness for accesses.

• Lunacy doesn’t implement the function entrypoint specialization described in the sequel paper to LBBV, which means each function has no known types at its entrypoint again - and so has to re-initialize hash witnesses with the slow hashmap lookup. Oops! Performing entrypoint specialization with the continuation specialization for return values as well requires being able to invalidate continuations in the JIT, which is unfortunately kind of complicated.

• Some of the residual operations which aren’t easy to implement in assembly (because they e.g. call Rust methods) are implemented by emitting calls to JitHelper functions. These calls require shuffling values into ABI registers and can’t bake constants into the assembly, instead needing them to be passed as arguments. This isn’t actually a big percentage of wallclock time but it still feels bad.

And, of course, the big caveat of I’m mostly doing this as a fun side-project, and so haven’t been focusing on actual Lua feature compatibility in favor of doing neat JIT things. Notably, Lunacy doesn’t actually implement metatables yet, although I will need to implement them soon in order to run more benchmarks. Likewise it’s missing a couple bytecode operations, and there’s a decent amount of codepaths in existing bytecode operations that are unimplemented!(). I only recently bothered to even implement a basic mark-and-sweep garbage collector, and was relying on Rc<T> for garbage collection up until then.

Overall however, I’m very happy with the current progress on Lunacy. I’ve wanted to write a “real” JIT compiler for an interpreter for a while now, so it’s great to finally get around to it and reach the point I can get some ok-ish times on benchmarks.

I share that story to make clear: this is a post about a cost of dropping out, but not regretful post, or a post discouraging the practice at all.

However, I believe it’s important to understand the tradeoffs you’re making with major life decisions. I certainly didn’t at the time, and from talking to other dropouts or people considering it, most of the tradeoff considerations center around professional life and their career progression. Those are not the only things college offers.

For most people, college is the foundational building block in their social circles for years to come. This seems to hold even for people who don’t really try at making friends, which is somewhat surprising to me, as is how enduring those social circles tend to be.

For most kids, college is their first opportunity for meaningful freedom from the yoke of their parents. I distinguish this from actually busy adults in the developmental sense of the word, as the modern college experience is akin to daycare and has few axes of success¹. Famously, many Asian teens will briefly become party animals, then settle down again after getting that out of their system (or not), but the effect exists more broadly. This intense socialization and increased freedom happens among a peer group that is more similar to you, and with more common interests, than any you’ve had before. With that framing, it’s not so surprising that by default even relatively antisocial people will find a friend group. The rotating groups in classes each semester means a huge amount of forced comingling between people who are likely to get along.

For many, this is also when they can develop a more serious long-term relationship, and get physical with it. The meme of college students fucking like rabbits in the dorms freshman year is, as far as I can tell, basically true. The freedom combined with increased opportunity for a relationship, on top of continued teens/early 20s hormones, also helps force people to become better socially adjusted and more apt at interacting with the other sex. This effect is particularly pronounced for guys. Many people stop wearing ill-fitting clothes, kick a nervous tic or two, and learn to flirt².

Even after graduation, people often end up in similar cities to their existing friend group. This becomes their initial, primary social circle, and they discover the horrible truth: that the college social experience is basically unique, and you will never again be constantly exposed to different people who share your age and interests by default. And the people you now spend the most time around, your coworkers, are discouraged as objects of romantic interest³.

When you drop out, you give this all up. And often the process leading up to dropping out consists of making this tradeoff to a lesser extent. I spent much of my high school and limited college years programming, and while that focus was great for skill acquisition, it definitely precluded the standard high school/college social experience. I’m not convinced this tradeoff is inherent, but it seems to be incredibly common. I suspect I would have tried to navigate it differently if I’d understood the full implications, but it’s hard to say in hindsight.

None of this is to say you can’t be social, make new friends, and find fantastic partners after college! It just doesn’t happen by default, especially for introverts, and requires a level of agency to make happen that is rarer than you’d think. Many people coast off their college circles for years, and look back on those as the best experiences of their life.

When you drop out, you suddenly have no choice: you must take responsibility for this aspect of your life. This is tough. It takes time, energy, and learning from failure, and you may not be comfortable pursuing it until you’re settled career-wise; I certainly didn’t fix this for a while. It’s also particular tough immediately after dropping out, as people your age are mostly still in school and their leisure time is dominated by school-centric social events. Even after they graduate, you end up at pretty different levels of life experience and often pretty different values, which makes dating weird. This goes away after a few years, and I’d generally encourage people to push past it, but you will probably end up with an older peer group and a few years of awkward dates.

For most of my friends, this all more or less worked itself out, so if you’re considering dropping out it’s not all gloom and doom. And the benefits of dropping out can be massive - it’s a variance-increasing decision, for good or for ill. But this is an important aspect to be aware of, and if you do drop out, to try and address as soon as you have some breathing room to do so.

If you’re thinking of dropping out, or recently pulled the plug and you want to chat, shoot me an email. I have limited generic advice to offer, but I’m usually happy to make time to sit down and talk things over.

Thanks to Ryan Hill for reading a draft of this post.

This paper was submitted to SIGBOVIK 2025.

I found a remote code execution vulnerability in the Minecraft mod Integrated Scripting. If you are running Integrated Scripting, and the version is:

<=1.21.1-1.0.16
<=1.21.4-1.0.9-224
<=1.20.1-1.0.11
<=1.19.2.-1.0.9

your server is vulnerable to being hacked by any player who is able to craft some items from Integrated Scripting. If you have a public or semi-public server, you should update the mod to a newer version.

Several popular modpacks contain Integrated Scripting. If you are running:

AllTheMods10 <=2.36
Craftoria <=1.14.0
FTB Presents Direwolf20 1.21 <=1.9.0
(and many more)

your server is also vulnerable, and you should update.

Technical details

Versions of Integrated Scripting released before February 24th, 2025 are vulnerable to a critical remote code execution vulnerability. Leveraging Java reflection it’s possible to escape the GraalJS sandbox and call arbitrary methods on arbitrary Java objects. The initial vulnerability disclosure, with more details, is available as GHSA-2v5x-4823-hq77. The vulnerability has also been assigned CVE-2025-27107.

This means there’s no substitute for actually looking at the results. If you are a video engineer, you must look at sample output and ask yourself if you like what you see. You should do this regularly, but especially if you’re considering changing anything, and even more so if ML is anywhere in your pipeline. You cannot simply point at metrics and say “LGTM”! In this particular domain, if the metrics and skilled human judgement are in conflict, the metrics are usually wrong.

Netflix wrote a post on their engineering blog about a “deep downscaler” for video, and unfortunately it’s rife with issues. I originally saw the post due to someone citing it, and was incredibly disappointed when I clicked through and read it. Hopefully this post offers a counter to that!

I’ll walk through the details below, but they’re ultimately all irrelevant; the single image comparison Netflix posted looks like this (please ‘right-click -> open image in new tab’ so you can see the full image and avoid any browser resampling):

Note the ringing, bizarre color shift, and seemingly fake “detail”. If the above image is their best example, this should not have shipped – the results look awful, regardless of the metrics. The blog post not acknowledging this is embarrassing, and it makes me wonder how many engineers read this and decided not to say anything.

The Post

Okay, going through this section by section:

How can neural networks fit into Netflix video encoding?

There are, roughly speaking, two steps to encode a video in our pipeline:

1. Video preprocessing, which encompasses any transformation applied to the high-quality source video prior to encoding. Video downscaling is the most pertinent example herein, which tailors our encoding to screen resolutions of different devices and optimizes picture quality under varying network conditions. With video downscaling, multiple resolutions of a source video are produced. For example, a 4K source video will be downscaled to 1080p, 720p, 540p and so on. This is typically done by a conventional resampling filter, like Lanczos.

Ignoring the awful writing¹, it’s curious that they don’t clarify what Netflix was using previously. Is Lanczos an example, or the current best option²? This matters because one would hope they establish a baseline to later compare the results against, and that baseline should be the best reasonable existing option.

2. Video encoding using a conventional video codec, like AV1. Encoding drastically reduces the amount of video data that needs to be streamed to your device, by leveraging spatial and temporal redundancies that exist in a video.

I once again wonder why they mention AV1, since in this case I know it’s not what the majority of Netflix’s catalog is delivered as; they definitely care about hardware decoder support. Also, this distinction between preprocessing and encoding isn’t nearly as clean as this last sentence implies, since these codecs are lossy, and in a way that is aware of the realities of perceptual quality.

We identified that we can leverage neural networks (NN) to improve Netflix video quality, by replacing conventional video downscaling with a neural network-based one. This approach, which we dub “deep downscaler,” has a few key advantages:

I’m sure that since they’re calling it a deep downscaler, it’s actually going to use deep learning, right?

1. A learned approach for downscaling can improve video quality and be tailored to Netflix content.

Putting aside my dislike of the phrase “a learned approach” here, I’m very skeptical of “tailored to Netflix content” claim. Netflix’s catalog is pretty broad, and video encoding has seen numerous attempts at content-based specialization that turned out to be worse than focusing on improving things generically and adding tuning knobs. The encoder that arguably most punched above its weight class, x264, was mostly developed on Touhou footage.

2. It can be integrated as a drop-in solution, i.e., we do not need any other changes on the Netflix encoding side or the client device side. Millions of devices that support Netflix streaming automatically benefit from this solution.

Take note of this for later: Netflix has many different clients and this assumes no changes to them.

3. A distinct, NN-based, video processing block can evolve independently, be used beyond video downscaling and be combined with different codecs.

Of course, we believe in the transformative potential of NN throughout video applications, beyond video downscaling. While conventional video codecs remain prevalent, NN-based video encoding tools are flourishing and closing the performance gap in terms of compression efficiency. The deep downscaler is our pragmatic approach to improving video quality with neural networks.

“Closing the performance gap” is a rather optimistic framing of that, but I’ll save this for another post.

Our approach to NN-based video downscaling

The deep downscaler is a neural network architecture designed to improve the end-to-end video quality by learning a higher-quality video downscaler. It consists of two building blocks, a preprocessing block and a resizing block. The preprocessing block aims to prefilter the video signal prior to the subsequent resizing operation. The resizing block yields the lower-resolution video signal that serves as input to an encoder. We employed an adaptive network design that is applicable to the wide variety of resolutions we use for encoding.

I’m not sure exactly what they mean by the adaptive network design here. A friend has suggested that maybe this just means fixed weights on the preprocessing block? I am, however, extremely skeptical of their claim that the results will generate to a wide variety of resolutions. Avoiding overfitting here would be fairly challenging, and there’s nothing in the post that inspires confidence they managed to overcome those difficulties. They hand-wave this away, but it seems critical to the entire project.

During training, our goal is to generate the best downsampled representation such that, after upscaling, the mean squared error is minimized. Since we cannot directly optimize for a conventional video codec, which is non-differentiable, we exclude the effect of lossy compression in the loop. We focus on a robust downscaler that is trained given a conventional upscaler, like bicubic. Our training approach is intuitive and results in a downscaler that is not tied to a specific encoder or encoding implementation. Nevertheless, it requires a thorough evaluation to demonstrate its potential for broad use for Netflix encoding.

Finally some details! I was curious how they’d solve the lack of a reference when training a downscaling model, and this sort of explains it; they optimized for PSNR when upscaled back to the original resolution, post-downscaling. My immediate thoughts upon reading this:

1. Hrm, PSNR isn’t great³.
2. Which bicubic are we actually talking about? This is not filling me with confidence that the author knows much about video.
3. So this is like an autoencoder, but with the decoder replaced with bicubic upscaling?
4. Doesn’t that mean the second your TV decides to upscale with bilinear this all falls apart?
5. Does Netflix actually reliably control the upscaling method on client devices⁴? They went out of their way to specify earlier that the project assumed no changes to the clients, after all!
6. I wouldn’t call this intuitive. To be honest, it sounds kind of dumb and brittle.
7. Not tying this to a particular encoder is sensible, but their differentiability reason makes no sense.

The weirdest part here is the problem formulated in this way actually has a closed-form solution, and I bet it’s a lot faster to run than a neural net! ML is potentially good in more ambiguous scenarios, but here you’ve simplified things to the point that you can just do some math and write some code instead⁵!

Improving Netflix video quality with neural networks

The goal of the deep downscaler is to improve the end-to-end video quality for the Netflix member. Through our experimentation, involving objective measurements and subjective visual tests, we found that the deep downscaler improves quality across various conventional video codecs and encoding configurations.

Judging from the example at the start, the subjective visual tests were conducted by the dumb and blind.

For example, for VP9 encoding and assuming a bicubic upscaler, we measured an average VMAF Bjøntegaard-Delta (BD) rate gain of ~5.4% over the traditional Lanczos downscaling. We have also measured a ~4.4% BD rate gain for VMAF-NEG. We showcase an example result from one of our Netflix titles below. The deep downscaler (red points) delivered higher VMAF at similar bitrate or yielded comparable VMAF scores at a lower bitrate.

Again, what’s the actual upscaling filter being used? And while I’m glad the VMAF is good, the result looks terrible! This means the VMAF is wrong. But also, the whole reason they’re following up with VMAF is because PSNR is not great and everyone knows it; it’s just convenient to calculate. Finally, how does VP9 come into play here? I’m assuming they’re encoding the downscaled video before upscaling, but the details matter a lot.

Besides objective measurements, we also conducted human subject studies to validate the visual improvements of the deep downscaler. In our preference-based visual tests, we found that the deep downscaler was preferred by ~77% of test subjects, across a wide range of encoding recipes and upscaling algorithms. Subjects reported a better detail preservation and sharper visual look. A visual example is shown below. [note: example is the one from above]

And wow, coincidentally, fake detail and oversharpening are common destructive behaviors from ML-based filtering that unsophisticated users will “prefer” despite making the video worse. If this is the bar, just run Warpsharp on everything and call it a day⁶; I’m confident you’ll get a majority of people to say it looks better.

This example also doesn’t mention what resolution the video was downscaled to, so it’s not clear if this is even representative of actual use-cases. Once again, there are no real details about how the tests with conducted, so I have no way to judge whether the experiment structure made sense.

We also performed A/B testing to understand the overall streaming impact of the deep downscaler, and detect any device playback issues. Our A/B tests showed QoE improvements without any adverse streaming impact. This shows the benefit of deploying the deep downscaler for all devices streaming Netflix, without playback risks or quality degradation for our members.

Translating out the jargon, this means they didn’t have a large negative effect on compressability. This is unsurprising.

How do we apply neural networks at scale efficiently?

Given our scale, applying neural networks can lead to a significant increase in encoding costs. In order to have a viable solution, we took several steps to improve efficiency.

Yes, which is why the closed-form solution almost certainly is faster.

The neural network architecture was designed to be computationally efficient and also avoid any negative visual quality impact. For example, we found that just a few neural network layers were sufficient for our needs. To reduce the input channels even further, we only apply NN-based scaling on luma and scale chroma with a standard Lanczos filter.

OK cool, so it’s not actually deep. Why should words have meaning, after all? Only needing a couple layers is not too shocking when, again, there’s a closed-form solution available.

Also, while applying this to only the luma is potentially a nice idea, if it’s shifting the brightness around you can get very weird results. I imagine this is what causes the ‘fake detail’ in the example above.

We implemented the deep downscaler as an FFmpeg-based filter that runs together with other video transformations, like pixel format conversions. Our filter can run on both CPU and GPU. On a CPU, we leveraged oneDnn to further reduce latency.

OK sure, everything there runs on FFmpeg so why not this too.

Integrating neural networks into our next-generation encoding platform

The Encoding Technologies and Media Cloud Engineering teams at Netflix have jointly innovated to bring Cosmos, our next-generation encoding platform, to life. Our deep downscaler effort was an excellent opportunity to showcase how Cosmos can drive future media innovation at Netflix. The following diagram shows a top-down view of how the deep downscaler was integrated within a Cosmos encoding microservice.

Buzzword buzzword buzzword buzzword buzzword. I especially hate “encoding stratum function”.

A Cosmos encoding microservice can serve multiple encoding workflows. For example, a service can be called to perform complexity analysis for a high-quality input video, or generate encodes meant for the actual Netflix streaming. Within a service, a Stratum function is a serverless layer dedicated to running stateless and computationally-intensive functions. Within a Stratum function invocation, our deep downscaler is applied prior to encoding. Fueled by Cosmos, we can leverage the underlying Titus infrastructure and run the deep downscaler on all our multi-CPU/GPU environments at scale.

Why is this entire section here? This should all have been deleted. Also, once again, buzzword buzzword buzzword buzzword buzzword.

What lies ahead

The deep downscaler paves the path for more NN applications for video encoding at Netflix. But our journey is not finished yet and we strive to improve and innovate. For example, we are studying a few other use cases, such as video denoising. We are also looking at more efficient solutions to applying neural networks at scale. We are interested in how NN-based tools can shine as part of next-generation codecs. At the end of the day, we are passionate about using new technologies to improve Netflix video quality. For your eyes only!

I’m not sure a downscaler that takes a problem with a closed-form solution and produces terrible results paves the way for much of anything except more buzzword spam. I look forward to seeing what they will come up with for denoising!

Thanks to Roger Clark and Will Overman for reading a draft of this post. Errors are of course my own.

History

First, I have to explain the historical workflow. Everyone in a group has access to an FTP server, where the workraw¹ and premux² are both uploaded to a folder for the episode as they are encoded. The translator will generally type out the initial script into an Etherpad clone³, along with all the signs and timestamps. The typesetter(s) will complete the signs in the pad, marking them off as they go, and uploading the file containing the signs to the FTP when completed with a filename like showname_06[ts-ryan].ass and all the fonts they used. The timer will take the dialog from the pad and time it in Aegisub, splitting where appropriate, and upload a showname_06[time].ass file along with their chosen dialog fonts, and usually copy over the OP and ED files from previous episodes as part of this. Editing can happen either in the pad (my preference) or after timing, in which case there’d also be a showname_06[edit].ass file dumped in the folder. Translation checking was also handled both ways, when applicable. Finally, the QC will merge the typesetting and finalized dialog and upload a showname_06[merged].ass, and they or others will upload showname_06[qc].ass and potentially successive versions with changes in response to the QC pass⁴⁵. Finally, someone took the final version, muxed everything together in mkvtoolnix, made a torrent, and threw it up on Nyaa. Show released!

Changes

Since then, there have been two major changes to the workflow.

The first major change is migrating to Git for a lot of the process. This was, as best I can tell, done in response to the host of different versioned files that would be dumped on the FTP. Some people felt that version control was a better solution, and Git was chosen for that purpose as it was by this point effectively the standard in software engineering.

The second is automating the actual release process much more heavily, in an effort to stamp out issues like forgetting to mux fonts, incorrect naming, and merging the wrong scripts. This workflow change seems conceptually positive to me; even if I have gripes with some of the specific tools used for it⁶, we were previously at a suboptimal level of release automation. I want to be clear that this is theoretically separate from the migration to Git, and the benefits of automated releases can be accrued independently. The existing tools tie the two together, but I consider them overcomplicated and don’t believe the marriage is intrinsic. There’s nothing stopping people from using a standardized naming scheme on FTP and a more straightforward tool to simplify the merge, muxing, and release processes.⁷

My core contention is that because the same person made the general tooling improvements and some of the tooling that enabled a Git workflow, people mentally link them and adopt the less useful portion (Git) unnecessarily. I also claim that the existing tools are overly generic and difficult for most people to use.

Strengths

I’ll start with what Git’s theoretical strengths: it offers a single latest version of any file going into a release, with a clear history of changes to it, and enables concurrent editing to a file. If you’re using Github, the pull request process also offers a way to review QC changes, creating a middle ground between the QC purely taking notes and the QC applying changes directly. It also creates nice web-based diffs automatically, which makes it generally easier to review others’ work.

Unfortunately, I’m not convinced these are particularly great benefits. Having a single latest version of a file is useful, but wasn’t a big issue in practice on the FTP, even if showname_ep06[qc][final][v3][no but for real this time].ass looks goofy, and could have been dealt with by standardizing naming (enforced by a release bot). Similarly, the concurrent editing is nice in theory but in practice does not come up very often if you’re separating dialog and TS.

I’m also not sold that the version history matters much at all for subbing. I’ve almost never seen it referenced due to the fundamentally short-term nature of these releases. In the scenario you’re looking to change something for the Blu-rays, it’s usually not long after the project and the staff can just search in the channel to read the discussion, if one ensued. You do ensure that the version history is never accidentally overwritten, which is much easier to do on accident with an FTP, but if it doesn’t matter much either way I don’t consider the occasional mistakes to be of much consequence.

Finally, the pull request process seems mildly beneficial but again not all that great. Most of the changes aren’t actually discussed in context on the site, and instead are discussed in the staff channel in Discord, and even a competent QC will probably leave some changes to the original typesetter, so you’ll end up with the good ‘ol process of pinging the typesetter to push changes anyway. This, and similar workflows borne of the diffs being intrinsic to git, is probably the strongest argument for Git, though I personally know a lot of people that don’t like it and don’t take advantage of this at all. It seems great for some groups, particularly ones composed heavily of programmers comfortable with Github, but probably not great for the median project.

Problems

Now that I’ve talked about the benefits and why I consider them minimal, what about the downsides?

The biggest one is that Git is really difficult to use. If you think otherwise, you probably don’t understand it very well. Software engineers are overrepresented in fansubbing, so basic familiarity is not uncommon for many of the staff, but even then in practice every group has someone whose part-time job is unfucking the inevitable messes Git produces. And for people who don’t work in software, which includes many of the translators⁸, Git means punching in arcane commands or dealing with an extremely complicated GUI, and occasionally getting into a bad state and begging a team member to help⁹.

It also necessitates further mandatory tooling setup to contribute to a project. In practice, everyone still uses an FTP for the premuxes¹⁰, so getting Git installed locally along with a GUI, making a Github account, and getting this all running is an extra step for every non-programmer to muddle through. You’re also putting your questionably legal fan translations on Github, which has so far been fine but I personally have reservations about. Also unlike when typesetting gained increased tooling requirements, these offer no improvement (hopefully) to the final output. It’s theoretical minor efficiency, and minor consistency, improvements, in exchange for a more complicated toolchain pushed onto the entire group. The tradeoff is quite different.

It also makes it surprisingly easy to leak personal information if a group wants to make a repo public. There’s an extra manual step of checking to see if anyone used identifying information in their commits, and rewriting the history if so.

Finally, as mentioned earlier, Git adoption seems to have been tied originally to SubKt¹¹, and more recently also muxtools. Both of these are monsters, and are so generic that they force every group to have someone comfortable in Kotlin or Python, respectively, to set up and maintain them for every project. In practice this means that similar to Git, every project ends up needing someone whose part-time job is tech support. These tools are difficult to set up and CLI-only, so good luck to anyone who has to interact with them that isn’t a programmer!

Looking forward

I think the situation has gotten sufficiently bad that groups have actually ended up in a worse place than the historical FTP-based workflow. I view the current situation as programmers responding to their dislike of manual processes by pushing incredibly complex tools on non-technical users for very marginal benefits (and very likely negative time saved).

What would a better world look like? A more opinionated tool, with YAML or some equivalent for configuration, running as a bot on a server with an FTP and enforcing a standardized naming scheme, and with less magical mega-merging and more checks to make sure everything necessary is present as part of a release, alongside the more rote aspects like muxing and uploading. It should opt for manual processes over automated ones for anything too complicated, and try hard to be straightforward for a nontechnical user. This means it has to be a single binary on Windows/MacOS, packaged in an installer, and have a GUI for editing the configuration and generating a release. No more “just learn python lol”. There’s a world where we have tooling that’s strictly better than the old workflow, but we aren’t there today.

Thanks to Petzku for reading a draft of this post.

If you have a thing that looks like source code->magic box->program you run, compilers are the magic box. If you pass source code around everywhere and run the magic box late enough¹ people call it “just-in-time” and then act very impressed. If the magic box just runs the source code and doesn’t output a program you run then they call it an interpreter, though, and are no longer impressed anymore.

Sometimes those interpreter magic boxes have another magic box inside them that do bytecode->second, smaller magic box->program you run. If you run the second, smaller magic box on snippets of the program that you notice you’re running a lot, then you call it a hotspot JIT that you integrated into an interpreter, and I’m pretty sure Google or Oracle immediately offers you $300k/yr to suffer through absolutely unhinged CS PhD code for a living².

We’re gonna write a magic box today.

Someone else’s magic box

I could write a JIT background introduction post here, but really there are a bunch of them already and I don’t think I would add very much writing another near-identical one. Instead, I’ll be referring to this blog post: none of this post will make sense unless you read it. I’ll wait.

Ok cool you’re back. Good blog post, right? That’s basically JIT compilers 101: you can JIT compile an interpreter by abstractly interpreting what operations the program would do if you started at the first bytecode operation and ran each sequential one, but instead of doing the operations immediately you instead record what the operation would do, and then at the end have one linear set of operations that, if you jump to the entry, will have the same result of executing all the operations in sequence. This final set of operations is compiled assembly code, and so Blazing Fast(tm).

Or, well. That’s the hope. The final set of operations is only as fast as what you generate: you could in fact record every single instruction that was executed while you were doing the interpreter part, and then run all those instructions again for your “JIT compiled output”, and while it would technically be JIT compiled it wouldn’t be a very good magic box and probably wouldn’t actually be much faster or different³.

A smarter magic box

Smart JITs record what information they were able to glean statically from the first pass, in order to emit final output code that doesn’t have to do operations that are redundant or useless for the output. Very smart JITs like V8 or HotSpot have so many of these that they are basically as complex, or more complex, internally as LLVM or GCC are for normal compilation, because the job is essentially the same: optimizing code. That’s too hard so we’re going to mostly ignore it for now.⁴ There is one thing we can make slightly smarter, though.

That BF JIT from the blog post you read (you did read it, right?) put all the operands for each operation at a constant location each time. BF is simple enough that this isn’t a problem for it at all, since it’s really a 1-register machine: > implicitly depends on the BF data pointer being in register r13 for it to manipulate, and there are no other live values in the VM other than values stored in the data array. The problem you then have if you were, say, making a calculator, is how would you JIT compile +, which has two operands and not just a static “current register”? For this example, we’ll be using the expression (0+1)+(2+3), which you can evaluate with a simple program in Rust:

enum Expr {
    Num(i64),
    Add(Box<Expr>, Box<Expr>),
    // ... more operations
}
type Value = i64;

fn eval(e: &Expr) -> Value {
    match e {
        Expr::Num(u) => *u,
        Expr::Add(left, right) => {
            let left = eval(left);
            let right = eval(right);
            left + right
        }
    }
}

First, a possibly-obvious observation: a recursive evaluation of a tree can be turned into an equivalent set of linear operations, by instead recording what operation you need to evaluate in order⁵ to a linear list. (0+1)+(2+3) becomes 0, 1, +, 2, 3, +, + in this way. I’ve always called it “linearization”, presumably after stealing it from a paper I read once upon a time; other people call it “flattening” an AST or (if they’re nerds) “A-Normal Form conversion”. This is part-and-parcel of JIT compilation of expression trees, or any non-JIT compiler: you can think of the expression tree as identical to this instead:

produce 0
produce 1
evaluate +
produce 2
produce 3
evalaute +
evaluate +

And, instead of first recording your traversal and then processing the linear list later, you can just do your processing in the recursive traversal directly.

If the equivalent code you are emitting for your JIT compiler for + expects that the left and right operands and the resulting value are always in the same register each time (say, rax and rdi, and the output in rax), then you also have to make sure that trees of operations correctly put the operands in the correct place in between operations, and none of them ever clobber values that you put in those registers and depend on being kept the same across operations. Consider how you would evaluate (0+1)+(2+3) if each + was emitting identically: you evaluate the left tree 0+1 which come in rax and rdi, and set the output as rax = 1 as a result, and now have to execute 2+3 which…needs rax = 2 and rdi = 3 when it evaluates. Messy!

Here a lot of simple JIT blog posts cheat. They just use the processor stack. + takes two operands and outputs one result; you could write it in assembly as pop rax; pop rcx; lea rax, [rax+rcx]; push rax so the operands are results need to be at the top of the stack for each assembly snippet, and then whatever registers it uses are anything you want because none of them have to be kept untouched across operations. You only have to make sure that you don’t clobber any stack values, which is pretty easy - your stack when running the right subtree of the above example is then [3][2][1] and the + only twiddles the top of the stack and leaves 1 untouched, producing [5][1].

Here’s a neat trick, which someone told me is technically a variant of “Sethi–Ullman local register allocation” but I can never remember and so call “Oberon-0 abstract stack machine register windowing stuff”⁶ or similar every time I mention it: Keep track of what stack offset you would be running at, if you did the push/pop stack stuff for operation inputs and outputs. And then map values at each stack offset to a register, and instead of needing to shuffle values to different registers to keep them alive, you emit a different assembly snippet for each operation depending on what offset it would be executing at, if it was using the stack.

Here’s an example, using the yaxpeax_86 and dynasm-rs crates for dynamic assembly creation.

use dynasmrt::x64::Assembler;
use dynasmrt::{dynasm, DynasmApi};
use yaxpeax_x86::long_mode::RegSpec;
use once_cell::sync::Lazy;

static regmap: Lazy<Vec<RegSpec>> = Lazy::new(|| vec![RegSpec::rax(), RegSpec::rdi(), RegSpec::rsi()]);
fn compile(e: &Expr, offset: usize, ops: &mut Assembler) -> usize {
    match e {
        Expr::Num(u) => {
            dynasm!(ops
                ; mov Rq(regmap[offset].num()), QWORD *u
            );
            offset + 1 // abstract stack push
        },
        Expr::Add(left, right) => {
            let after_left = compile(left, offset, ops);
            let after_right = compile(right, after_left, ops);
            let pop_right = after_right - 1; // abstract stack pop
            let pop_left = pop_right - 1; // abstract stack pop
            dynasm!(ops
                ; add Rq(regmap[pop_left].num()), Rq(regmap[pop_right].num())
            );
            pop_left + 1 // abstract stack push
        },
    }
}

fn jit_and_run(e: &Expr) -> Value {
    let mut ops = Assembler::new().unwrap();
    let start = ops.offset();
    let res = compile(e, 0, &mut ops);
    assert_eq!(res, 1); // all tree of operations should boil down to one result
    dynasm!(ops
        ; ret
    );
    let end = ops.offset();
    let buf = ops.finalize().unwrap();
    let compiled: extern "C" fn()->Value = unsafe { core::mem::transmute(buf.ptr(start)) };
    compiled()
}

If we had the stack->register map as rax, rdi, rsi for example⁷, then the final assembly for (0+1)+(2+3) would instead be

mov rax, 0; // 0
mov rdi, 1; // 1
add rax, rdi; // +
mov rdi, 2; // 2
mov rsi, 3; // 3
add rdi, rsi; // +
add rax, rdi; // +
ret

Does that set of operations look familiar? It’s the same as the linearized traversal of the tree from one of the above paragraphs. This program doesn’t use any stack, only registers, and is Blazing Fast(tm). If you run out of registers because you abstract stack offset gets too high, then you can fall back to normal push and pop like the “cheating” example - or implement some dumb last-recent-used cache for the registers and insert spilling when you need values at abstract stack offsets that don’t map to registers and you’ve accidentally re-created Linear Scan Register Allocation (which is a good thing).

This is a specific case of optimizing an interpreter when JITting, where you are able to pre-compute some unique location for each operation inputs and outputs and emitting code tailored for those specific locations, instead of emitting extraneous operations shuffling data around unneeded - and a case that isn’t quite touched on in the original BF JIT series of blog posts, which more talks about optimizing sequences of operations themselves via peephole optimizations, and then skipping to using LLVM which handles global register allocation and removing extraneous moves for entire functions, removing a need for doing this style of optimization yourself. Most other JIT-related blog posts I’ve seen online also don’t handle anything more complex than BF, or implement anything more than the push/pop machinery.

A cute paper that extends this idea is “Copy-and-Patch Compilation: A fast compilation algorithm for high-level languages and bytecode” by Haoran Xu, Fredrik Kjolstad. It uses C++ to generate several different versions of functions which implement bytecode operations, using templating in order to create at compile-time “stencil” versions of the function which pad the argument and result values with dummy values. This has the result of essentially tricking the compiler into generating the various shifted assembly snippets itself, with the register used for each abstract stack index being the corrosponding register in the function ABI.⁸

This is, in my head at least, also closely related to the “Delayed Code Generation in Smalltalk-80” paper which not only allocates a unique location of values while doing code generation, but also delays emitting the code to evaluate those values until they are observed by some use, which itself has where it would like the value to be put at the end - and thus can emit code that computes values in that desired destination location directly. This technique was used in the V8 JavaScript JIT for their one-pass code generation at one point in time: they had a JIT that was “being stupid”, and they replaced it with something “not being stupid”, and reaped some performance gains despite being fairly conceptually simple. And, hopefully, now that you know this exists, you can too.

Footnote:

In the majority of cases, this is approximately trivial: you create some objects, get pointers to them and do some set of operations, complete the operations and all the pointers go away, and then destroy all the objects. More complex programs have problems, however, especially if the answer to the question “who is in charge of destroying this object?” doesn’t have a great answer, and requires a bit more work.

C++ has a shared_ptr<T> type for a reference counted pointer to an object. Every time you copy the shared_ptr, it increment a counter; every time you destruct a shared_ptr, you decrement the counter. Once the counter hits zero, you can destroy the object the shared_ptr is pointing to because you know you were the last person that has access to the object, and can lock the door on your way out.

Rust has the same thing in its Rc<T> and Arc<T> types (with Rc being “reference counted” and Arc being “atomically reference counted” - Arc is safe to share between threads, with the count increment/decrement using atomic operations). The difference is that C++ doesn’t fix all memory safety bugs with shared_ptr: dereferencing the shared_ptr gives you a raw pointer, which you can erroneously hold past destroying the shared_ptr it came from, which may destroy the object. Rust’s Rc smartpointers instead give you a borrow to its contents, and thus are guaranteed by the compiler not to outlive the object they’re borrowing from (i.e. the Rc).

Rust also prevents data races through its Arc type: shared_ptr is like Arc in that incrementing and decrementing the count across different threads that are pointing to the same object are safe for both of them, but shared_ptr is still vulnerable to data races, for example if two threads are attempting to write to the object at the same time, or one is writing while another is reading the same memory location. Rust only gives you a read-only pointer to the contents of an Rc or Arc pointer - in order to mutate an object behind an Arc, you need to use one of Rust’s “interior mutability” types, such as RefCell<T>, which does a dynamic check that there is either several read-only pointers to the object, or one read-write pointer, but not both, or else it will panic and crash the program. Usually Rust enforces the read xor write access invariant using its borrow checker system, but because you have non-trivial ownership for objects you can’t use the borrow checker, and have to have those checks at runtime instead of compile time.

This is all well and good. With shared_ptrs or Arc you can make big tangled data structures all pointing to one another. You can make a doubly-linked list, or a graph with edge that are pointers to nodes, or a red-black tree with “parent” links. The problem is that it will leak memory, and eventually your program will get very slow and then die. What went wrong?

Cycles

shared_ptr and Arc have the same problem: they have a counter, which they maintain as the number of existent pointers to this object. If you have a cycle of objects like A<->B, where A and B are two different objects but both have pointers to eachother, then both have a count of 1 and thus can’t be deleted, because they’re still accessible from something. If your program has destroyed every other reference to A or B, so they are unreachable except through the internal pointers to eachother, than they can safely be destroyed! They’re garbage, because there is no possible (correct) way for your program to access that memory. But they won’t be, because the reference counting isn’t smart enough to know the difference between pointers that are reachable and unreachable, and just sees that both objects have a reference-counted pointer somewhere and can’t be destroyed yet.

This is a pretty bad problem to have. The normal solution for this in Rust or C++ is to use an arena allocator, where you allocate a memory region upfront, and then can have objects within that region that can point to other arena objects, and then at the end you free the entire arena. This isn’t a great solution, since you are essentially still leaking memory, it’s just eventually you will free that leaked memory when you teardown the entire arena, and not any time sooner. You can also use a real garbage collector, like shredder or the boehm-demers-weiser GC. These will occasionally do a scan of all reachable objects, freeing everything else - they don’t care about cycles, because if the cycles of objects aren’t reachable from any threads they won’t be marked as reachable, and thus freed because they don’t keep precise reference counts. You can really think of an arena allocator as a bad version of a semi-space garbage collector, where you can only ever do one collection and it always frees everything: this is obviously still useful in that it guarantees that all your memory is reclaimed, but has obvious downsides.

My problem with shredder, or rust-gc, or any of the several other Rust garbage collector libraries is that they’re complex. A lot of them have novel schemes to make sure that your program maintains a set of initial reachable objects (“roots”) correctly at all times, since if it doesn’t you will incorrectly free memory and crash (or worse), and don’t have very high confidence that they are managing those roots correctly and thus have warnings about not being production ready. All(?) of them also have stop-the-world phases where they have to pause every running thread in your process in order to scan all the roots from a coherent view (or don’t support multithreading at all!).

Collection

Turns out there’s a neat variant of garbage collection called cycle collection. I was ruminating on how complex all the Rust garbage collectors were a few months ago, and how there isn’t just an Arc-but-doesnt-leak replacement smartpointer you can drop-in to programs that need it, and my addled brain dredged up half-remembered thoughts of when I read Concurrent Cycle Collection in Reference Counted Systems, 2002 a few years before. I then brashly and confidently tweeted about how you could make a very simple Arc-but-doesnt-leak replacement smartpointer library in Rust, and banged out a design based on the general principals I remembered. And then I actually re-read the paper and realized it had object color state transition diagrams that look like this. Which I had, by pure coincidence, been using as the punchline for jokes about how complex concurrent garbage collection systems since, uh, probably the same several years timespan since I last read the paper. Woops.

The basic idea with the cycle collection scheme is that by default, objects are the same as reference-counting. The difference is that when your count is decremented to a number other than 0, you add the object to a list, and then when doing a garbage collection you scan all the objects in your list (recursively visiting all objects reachable from each object) to see if any of them form graphs of objects where all the objects’ references are accounted for with only internal pointers to eachother. The idea is that in order to form a cycle, you have to have a mutator thread:

• Create an object A
• Create an object B
• Increment A’s refcount and create A2, and set B.ptr = A2
• Set A.ptr = B (the original B)
• Destroy A, which is the last external pointer into the cycle of objects.

Cycle collection handles this by making the “destroy A” step, that is the point where you make a group of objects unreachable, instead essentially mark the objects as potentially unreachable. The scan step then finds all objects that are unreachable, because you know none of the objects in some graph have external pointers, and thus nothing can acquire a new pointer to any of them, and not unreachable, because your scan didn’t find all the pointers, and thus something else by definition has at least one pointer and reference count.

The nice thing about this is that, in the best-case of all your objects just being created with a reference count of 1, and then being used a bit and then decremented from 1->0, objects are just the same as Arc and freed immediately, with no extra work being needed. Even if the Arc’s count was incremented and then decremented twice instead (1->2->1->0) you can still immediately free the object: you know you had the last reference to the object, and thus it can’t possibly be leaked via cycle, and so you can just remove the object from the list and then free it immediately. The recursive scan is only ever needed to free objects that were actually leaking memory via cycles.

We visit objects in order to count how many pointers to them we’re able to find, because if we can find an equal number of pointers to an object as it’s reference count then we know the mutator doesn’t have any pointers, and if we find a disconnected group of those objects then we can free all of them at once since they’re collectively unreachable.

There’s a bunch of complex stuff that you have to handle in order to have correct visiting in the face of concurrrency, unfortunately - if you’re scanning a graph of objects, by the time you scan the last object that first object’s count may have changed. We use a scheme where objects have a bitmap of flags, and are marked VISITED when we first see it, and then if the object is decremented while VISITED it is transitioned to DIRTY. When checking to see if an object is leaked or not we can then also see if it is DIRTY or not, where we know that a mutator had a live reference to the object, and thus it and all reachable objects can be marked not leaked. The object is queued to be added to the next collection’s list instead (since the decrement may have been for the last external reference).

Additionally, consider what happens if we visit struct Foo { a: Mutex<Option<Gc<Bar>>>, b: Mutex<Option<Gc<Bar>>> } concurrently with a mutator swapping the contents of foo.a and foo.b - we could count a as one pointer to whatever it is referencing, and then the mutator swaps the field, and then we could count b as a second pointer to…the exact same object as before, accidentally double-counting the first pointer’s object and under-counting the second’s, which would be very bad. The collector normally takes a read-lock on the Gc<T> objects while it scans them, which either fails if object has an outstanding &mut-reference from a mutator (in which case we obviously know it’s still reachable from something) or succeeds and blocks mutators from acquiring new &mut-references which are required to swap around fields of objects. The entire point of interior mutability, however, is being able to modify fields with only a &-reference! The simple solution, and what I did, is just require users to mark any type that has interior mutable fields containing Gc<T> references, and take a write-lock instead of a read-lock if so, which guarantees the object is completely inaccessible even via &-reference while we visit it.

Samsara

Samsara is my library that provides a Gc<T> type which is a reference counted with cycle collection smartpointer. It is a concurrent garbage collector, with the potentially-unreachable list scanning happening on another thread (only via manual trigger, not automatically via heuristiscs for now). And, unlike every(?) other Rust garbage collector that I know of, it doesn’t have a global stop-the-world phase: it will occasionally block mutator threads if they try to acquire a read-write reference from a Gc object while the collector is scanning it, but that should be very short (since it only holds the read-lock while visting that object, and not the entire subgraph). It doesn’t interrupt threads via signals or anything for safepoints, either, unlike some other garbage collectors.

Right now it’s actually implemented with only safe Rust, plus some unsafe libraries (im for an immutable datastructure, and qcell for an ergonomic get/set API to try and prevent user deadlocks). It is, literally, just Rust’s Arc<T> behind the scenes, plus an RwLock<T> for the interior mutability, with the collector list holding Weak<T> references to objects that need to be scanned. Even if the collector incorrectly frees a set of objects that it thinks aren’t reachable any more, it would panic the user via an Option::unwrap instead of giving a use-after-free.

It isn’t the most efficient currently. When doing a cycle collection it will probably use a lot of memory, especially if every live Gc object is reachable from one of the objects on the list, since it creates a big HashMap of objects and adjacency list of edges. It also currently has the problem where objects are sent to the list, or removed from the list, via sending a message on a channel to the collector thread, which means that if you trigger a collection and then decrement a bunch of Gc objects before it finishes, it will block on the channel since it isn’t being drained while a collection happens. I think that the “list” could instead be maintained via an instrusive doubly-linked-list in the Gc<T> header, in order to allow the mutators to add or remove their objects from the list without needing the collector to do anything; unfortunately it would be hilariously unsafe and a lot of work to make sure it’s sound, and without a concrete usecase that is held back by the current design I probably won’t get around to trying that scheme out for a while.

I don’t claim to be doing anything novel here, for the record! For one, I’m obviously basing this off the paper from 2002, though doing things a little bit different. For another, there are several other Rust crates that also implement the paper! fitzgen’s bacon-rajan-cc and cactusref both are probably equally good if not better fits for anything you’d want to use samsara for - I mostly just nerdsniped myself and thought it would be neat to write this, and so I did.

JIT stands for just-in-time compiler. When programming languages are written, they usually first start off as an interpreter. Once they reach a critical amount of usage and support (and people complaining about how slow they are), they sometimes adopt JIT compilers. The interpreter side is usually written as a big case/switch over language operation types (Add, If, Break, Function call, etc.) which run a specific part of code; the JIT side, for code that is ran a sufficient amount of time, instead abstractly run a snippet of code, and record what operations it would run if it was being interpreted…and instead emit the equivalent operations as assembly in some compiler-ish way, so that jumping to the start of the assembly is equivalent to running the list of operations all at once without the overhead of having to switch and dispatch to the correct interpreter operation handler at each step.

There are a few problems to this.

For one, JITs are really fucking hard to write. They are the subject of PhD theses, like Maxime’s Higgs VM. Very good JITs like Java’s HotSpot or Javascript’s V8 have possibly millions of man-hours dedicated to them to try and make them as fast as possible, and LuaJIT (which is regarded as one of the best and “simplest” JITs ever made) is so extremely impressive it is a running joke that Mike Pall, its author, is a robot sent back in time from the future because it is so unbelievable that a normal human being wrote it himself.

Because they are so hard, the second problem falls out: not many languages reach that step. Most languages only ever had an interpreter, because the amount of effort to write a JIT outweighs the benefits that could be reaped by just trying to squeeze out some speed from the existing interpreter, or adding new features.

And the third problem is that the actual act of writing a JIT is fraught with peril: you have to make sure that the assembly you are emitting is exactly equivalent, from an external point of view, as running the interpreter operations naively. And the process of writing the assembly emission via abstract interpretation usually looks an awful lot like the normal interpreter, but slightly different - you have a giant case/switch over operations, and the operations do something that kinda looks like an If or a Break or a Function call, but just defer the actual act for a bit later. It’s tedious work, and because the JIT and interpreter operations are essentially 1:1, it’s work to keep them in sync whenever you touch interpreter code. But even worse is that it’s bespoke. V8 is one of the most complex programs ever written, and Google has poured more money than God into it, but all that effort is almost completely unusable for any other JIT because of how extremely specialized the work is to Javascript and V8 itself.

Lineiform is my attempt to solve this at least a little bit. The idea is that instead of having an interpreter, and eventually having to start nearly from scratch implementing an equivalent JIT, you write an interpreter in a slightly special way and make sure to call some of Lineiform’s library functions at certain points. Lineiform will then automatically do JIT-esque things: optimize hot code into runs of operations without dispatch overhead, or getting rid of extraneous work that an intelligent JIT compiler would’ve been able to elide. This is probably the utter height of hubris; the closest equivalent is something like Truffle/Graal or RPython, wherein you write an interpreter in some subset of Java or Python and, through just normal JITting, it also optimizes your interpreter: your language semantics are in terms of Java or Python operations, and it knows how to emit assembly and optimize those, so it comes along for the ride.

I don’t want to write an interpreter in Java or Python though, and they have some annoying constraints (such as Truffle/Graal requiring a commercial license from Oracle in order to use, and most people don’t trust Oracle with a 20ft pole… edit: Woops, that’s a lie - the Community version of Graal is GPL, with an Enterprise version with extra features). I want to write an interpreter in Rust. And Rust doesn’t have a bytecode interpreter that it targets, so you can’t just write a JIT for that and JIT arbitrary interpreters “for free”. So what do you do?

Closure generation compilers

Well, one piece of the puzzle is something called a “closure generation compiler”. They don’t seem to be very well-known; I was introduced to the concept by a random Facebook engineer’s compiler presentation, and most of the prior art for it is by a single paper’s author, Marc Feeley. A normal tree-walking interpreter flow for an evaluate function is something like switch on Increment -> [recurse on argument -> switch on Constant(3) -> return 3] -> Increment 3, return 4. A closure generation interpreter would instead be a build function, which does switch on Increment -> [recurse on argument -> switch on Constant(3) -> return a closure that when evaluated returns 3] -> return a closure that when evaluated runs the constant 3 closure and adds 1. And then at the very end, to evaluate instead of build, you call the returned closure, which just runs the increment closure and then the argument closure: it separates the interpreter step into two parts, generating a big tree of closures that call other closures for each operation, and actually evaluating which only run the closures, which jump directly into each language operation runtime behavior. This reduces the interpreter overhead, since you don’t have the case/switch in the operations you have to do run the language.

I read that and thought it was neat, and then I thought a bit more and went: Huh, that sounds a little like a JIT. You are deferring the actual evaluation of language operations until later, and thus requiring the evaluation do less work because you were able to do some things ahead of time (specializing the runtime behavior to a closure you can just call). And how closures work in Rust is that they are essentially structs, which have a function pointer you jump to in order to run it, and a closed environment which is some list of values in memory which is passed to the function pointer as an argument: this is why let three = ||(return 3); ||(three() + 1)() works, because three is put in the closure environment and fetch out and called when it is run.

Forth and direct threading

Another piece of the puzzle is the programming language Forth. Forth is kinda neat, in that it has an interpreter design that isn’t used very often outside of it: it uses a direct threading interpreter. This is designed so that instead of a big case/switch of operations, where it jumps to the implementation of the operation and then jumps back to the case/switch for the next one, you instead build a big table of function pointers to call, each one its own operation. And critically, each operation ends with a jump to the next entry in the table. Running the interpreter is then just jumping to the first entry in the table, which runs its body and then it jumps to the next entry, which runs its body, which…etc. Constants, for example, are just snippets that return the constant value, and Forth is a stack based language, where operations store values on the stack or pop them to use as values. 3 5 + then is a program that jumps to a snippet that pushes 3 to the stack and then runs the next operation, which is a snippet that pushes 5 to the stack and then runs the next operation, which is to pop two values (3 and 5) from the stack and then add them and return.

One insight that direct threaded interpreters use is that you can compile direct threaded functions, however! If you have the function 3 5 +, you can just stitch all the functions together into one big function, and not need tailcalls at the end of operations. If you have a list of operations, then the sequence of instructions that it will end up running will by definition be all of them sequentially, and you could just emit and run that directly. If you have an optimizing compiler in front, then maybe it would be able to notice when it is emitting code that there’s no reason to push 3 and then pop 3 from the stack into a register for the add later when it can just use the constant 3 in a register directly, and the same for 5, and then maybe it can notice that it can just fold 3+5 to 8 directly and emit 8 instead. Because you had extra knowledge, namely the full list of operations that the function would run, you could do global optimizations and get rid of some of them.

This is one of the key insights that interpreters are missing, and why they are slow in a lot of cases: if you have a sequence of language operations that, say, add values together, those operations might have to check if the value they are adding is an int or a float, or an object with an overloaded + operator, and so has to do some work to figure it out when it is evaluating. If you have a concrete operation, such as 3 5 +, then you know what work is unneeded: your values are always numbers, and so you can specialize the add to only add together integers, and maybe even further to just compile down to a constant 8. Language primitives have to be maximally permissive, because they have to cover all possible inputs; if you know inputs, even only some of them, then you can reduce the range of possible cases they need to cover, and do global optimizations to remove redundant things. You can think of this as a special case of partial application, where you are fixing the function pointer that the tailcall at the end of snippets will run to concrete values, and so can apply it at compile time instead of having to do it at runtime.

Putting them together

Lets return to Rust closures. When you run the closure generation interpreter, you are taking some dynamic input and returning a tree of closures that have other closures they call in their environment; if you have the rust closure let three = ||(3); let five = ||(5); let add = ||{ three() + five() }; add() then it is one closure which has two entries in its environment (three and five). It jumps to their implementation, which return values. If you squint, this is very similar to how a direct threaded interpreter works, but the calls happen in the middle of the function instead of before the add, and so leave their data in immediates in the function instead of on a global stack. You still have concrete implementations of snippets of code you know you will be running if you call the closure, it’s just harder to figure out what they are since it’s not a linear array. In fact, in order to figure out what closures the add closure would be calling, you pretty much have to just run it, since the closure can be doing arbitrary x86 instructions, not just maybe some list of interpreter operations that you could special-case on how to put together and optimize. If you aren’t crazy, this is about when you start thinking about how rustc emits LLVM IR and how you can maybe leverage that, and invent holyjit. Unfortunately, I’m crazy, and don’t want to do things like vendor a custom rustc fork to build programs with.

Well, it turns out there is maybe hope. There is a concept of a dynamic recompilation engine. QEMU uses this, for example: when you run a QEMU aarch64 VM on an x86 host, behind the scenes it is translating those ARM instructions into an internal representation that keeps the same operational semantics as the ARM program, and then emitting x86 that from the outside has the same runtime behavior. QEMU is JIT compiling arbitrary ARM code from ARM to x86. And this isn’t restricted to only QEMU; things like DynamoRIO also use dynamic recompilation in order to JIT from x86 to x86, so that you can do neat things like add code before and after any branch instruction in a binary: it lifts x86 instructions to an IR, adds whatever operation you want done before branches, and then lowers the IR back to x86, making sure to fixup memory accesses so that nothing can tell that it was moved and modified.

And now that we have the background, we have Lineiform: Lineiform is just a wrapper function that takes a closure as input, and returns a closure as output. In the middle however, it will dynamically inline calls to other closures that the input might be doing, using dynamic recompilation to lift, optimize, and then lower x86 instructions to a set of x86 instructions with the same operational semantics, but hopefully doing less work such as inlining small closure snippets so there isn’t call/ret and ABI overhead, or folding constants like 3+5. We can have a performant closure generation interpreter, which is already plenty fast, and hopefully optimize out whatever work we can from hot closures, such as commonly called methods or loops, to avoid things like type checks, in order to incrementally JIT code faster and faster. And critically, this should all be very easy to integrate with an existing codebase or arbitrary new interpreter, since it doesn’t depend on any interpreter specific concepts or require re-implementing of your interpreter as an entirely new JIT like existing projects: you only have the one interpreter case/switch dispatch, which builds a reified tree of snippets it jumps between, which Lineiform collapses down into a single code block and optimizes.

One of the inspirations for this project is stumbling upon, years ago, a cool GitHub repo called redmagic that is kind of similar: it’s a C library that you would annotate a normal interpreter with, and it would automatically build program traces for a tracing JIT by just stitching together the assembly instructions that singlestepping would hit. In practice, that’s very different than how Lineiform work, but it’s also really cool and I’ve been thinking about it on and off ever since; it’s unfortunately also not very useful (with the quoted performance gains ~10% in very specific cases, and development stalled), and I’m hopeful that being a lot more structured will help Lineiform do a lot better.

And there are lots of cases where this could be useful - a hot interpreter loop that indexes into an array, for example, would just be calling an Index operation that needs to do bounds checks each time, because it has no way of knowing across interpreter operations that it has already done a check, or calling a method on some dynamic object multiple times having no way of knowing if the method exists on the object or not at each invocation. With Lineiform, the hope is that it exposes the multiple Index operations to a single function optimizer, which can remove the redundant bounds checks for example. And the pie-in-the-sky goal is to do some really tricky stuff with persistent caching of the IR in order to extract an ahead of time compiler for interpreters too, so that you only have to pay for the dynamic recompiling and optimization once, ever, for a program.

The goal is to provide a modest speedup for whatever interpreter is able or willing to use Lineiform, with a fraction of the effort of writing a normal JIT. It’s not going to be as performant as an artisanal hand-rolling JIT…but hopefully it will be close, and nearly free.

So far it doesn’t do much: it can inline some simple closures and fold simple math operations across them, such as:

let a: Wrapping<usize> = Wrapping(1);
let mut jit = crate::Lineiform::new();
use core::hint::black_box;
let clos = jit.speedup(move |()| { black_box(a) + Wrapping(2) });

optimizing down to

push rbp
mov rbp, rsp
mov eax, 0x3
mov rsp, rbp
pop rbp
ret

Forward from here

I was trying to use Cranelift, a small and fast codegen backend that is mostly used by Fastly for their WebAssembly JIT, for this but the usecase that I’m targeting is sufficiently weird that it didn’t work all that well. I’ve been working on an alternate codegen backend, handrolled specifically for Lineiform’s constraints, that I’m fairly hopeful will work out ok - the scope is a lot more limited than a normal compiler in various ways, so it’s not like I’m going to have to reimplement all of LLVM. I’ll probably implement a simple closure generation Scheme interpreter, and then try to implement operations as needed to make it work until I have a PoC; I don’t know how long this will take, but it’s probably a while. In the mean time, if you’re interested you can check out the (extremely work in process) GitHub repo.

And if it works, and if it’s fast, and if it’s useful? Who knows. I’d like if Apple or Facebook paid me a lot of money for this library in order to make things fast (think being able to JIT compile build scripts or DSLs), so I’m not sure if I’ll opensource or only source-available it, just in case. If you’re the type of person that cares about licensing (or just want to chat about compilers or systems programming things!), reach out to me on Twitter.