Conservation of Compute

Last year, we were building software that helps you find engineers through GitHub. Not through resumes, through the code they actually wrote.

The obvious way to do that is to build a search engine on top of GitHub. Crawl everything, embed everything, and every time a recruiter types in a query you go hit the index. Which is a lot of compute, and you’re paying it on every search.

We didn’t do that. We just asked the model.

Describe the role, the one you’re actually trying to fill, and ask the model which repos would be most relevant for it. Then pull the latest contributor list from those repos and rerank. No index, no embeddings, no crawl.

And it worked. I think the reason is kind of obvious in hindsight. Models get trained on the popular, active parts of GitHub, and those are the repos with maintainers and code review and actual signal. So the model already knows the part of GitHub you’d want to search anyway. We kind of just got that for free.

Honestly, nothing really stuck at the time. You build it, it works, you move on. But recently I got reminded of it, and I had a thought. The interesting part wasn’t that it worked. It was where the work went.

Someone has already read all of GitHub. Well, all of everything, really. The model provider did that during training.

So the heavy lifting didn’t go away. It just moved. Someone did it once, and now anyone who can ask a question gets to use it.

And that sounded a lot like a conservation law to me. Energy doesn’t disappear, it just goes somewhere. The compute to read GitHub didn’t go away, someone just spent it earlier. Conservation of compute, right?

I liked that. I sat with it for a bit.

But it doesn’t really hold up, because compute isn’t conserved at all.

Every time a recruiter ran a query we paid for inference. Not a lot, but not zero. And sure, our bill went down, we weren’t running a whole vector database anymore. But the training compute is still there, it’s spent, and our queries just sit on top of it.

If this were an actual law the numbers would add up. Training plus our inference should land somewhere near what building the search engine ourselves would’ve cost.

It doesn’t, not even close.

That model is answering questions for millions probably billions of people, so the cost of reading GitHub gets split so many ways that our share is basically nothing. That’s not conservation, that’s just, I don’t know, leverage?

Okay, so compute isn’t the thing. But something did get saved, because we really did skip a lot of work. So what was it?

I think what I was missing is that there are two kinds of compute and I was lumping them together.

One is search. You don’t know the answer, so you spend cycles finding it. Try stuff, throw out what doesn’t work, narrow it down. Figuring out which repos matter is search. Figuring out how to sort a list is search. Figuring out a sort that runs in n log n was also search, it just happened in Tony Hoare’s head in 1959.

The other is just doing the thing. You already know the path, you walk it. Running quicksort today is that. Running inference on a model that’s already trained is that. You pay for it every time, and that’s fine, it’s cheap, nobody’s figuring anything out.

And that’s the split I guess. The compute to run things you pay every time. The compute to figure things out you pay once. When the AI providers trained on GitHub, the figuring out got done, and everyone after that just runs.

So maybe it’s not conservation of compute. Maybe it’s conservation of search. You build something, and the search is done, and from then on it’s just execution.

And once I had the word search, it got kind of obvious, because there’s already a thing every programmer uses that conserves search. A cache.

That’s all a cache is. Compute something once, keep it, don’t compute it again. Same question, same answer, no search the second time.

An index is a cache, just over a class of lookups instead of one. You’re not storing one answer, you’re storing enough structure that a whole family of queries skips most of the work.

An algorithm is a cache over a method. Quicksort doesn’t store any sorted list, it stores the search for how to sort, and it works on lists nobody’s ever seen. Hoare paid for the search, and every call since is a hit.

A model is a cache over a function, sort of. It doesn’t store answers or methods, it stores an approximation of a whole space of questions, including the ones nobody’s asked yet. Training is the search, the weights are the cache.

So I guess building is caching. A library is a cache of design decisions. A binary is a cache of compiling. A model is a cache of reading the internet.

And that clears up something else that was bugging me. A conserved quantity shouldn’t be able to pay out forever, and this one does. Someone spends a few weeks on an algorithm and the world saves, I don’t know, centuries. But that’s just how caches work.

There’s that old joke about the two hard problems in computer science, cache invalidation and naming things. It’s a bit less funny now.

Because if building is caching, then going stale is cache invalidation. Every model has a training cutoff, so anything that happened after that is a miss. And invalidation is just what you do about it. Pushing an update. Bumping a version. Training a new model and retiring the old one. You’re evicting the old search and paying for a fresh one.

So the law I actually believe in is smaller than the one I started with. Compute isn’t conserved. Search is. You build something, the search gets cached, and the cache keeps paying out as long as people keep asking that shape of question.