Artisanal Chess Engines

I was genuinely surprised to learn that the chess engine community harbors some hostility toward LLM assistance. I first realized it from this GitHub issue, in which a Stockfish contributor expresses their demoralization as an open-source developer when they see "agentic engineers" adapting their ideas with little to no attribution, and claiming spots on public leaderboards.

During my college years over a decade ago, I wrote a decent chess engine and was involved in ICGA competitions. I'm certainly not qualified to speak on cutting-edge engine development, but I can speak to the attitudes of the chess programming community back then, and the striking similarity to the attitude today.

The Rybka controversy#

Rybka was a chess engine that won four World Computer Chess Championships (WCCC), and was stripped of all victories in 2011 after the International Computer Games Association (ICGA) decided that it contained unattributed code from other chess engines (Fruit and Crafty).

Fruit was a pioneering and highly influential open-source chess engine released by Fabien Letouzey in 2004. Within six months of Fruit's release, HIARCS had its biggest Elo gain in over a decade, Junior improved even more sharply, and no serious engine released after Fruit followed the old slow growth curve. Studying the strongest open program had become how the field worked. Rybka absorbed the same ideas and then pulled 150 Elo ahead of everyone on equal hardware, which is hard to square with the theory that it was mostly borrowed code.

What is an Elo point?show me →↓

An Elo point is a unit of measurement for the relative strength of a chess engine. It is a logarithmic scale, where each point is worth approximately 100% of the previous point. For example, an engine that is 100 Elo points stronger than another engine is about 100 times stronger.

Rybka's author Vasik Rajlich gave an interview explaining his side of the story:

I did not list Fruit on the entry form because there is no direct game playing Fruit code in Rybka. Rybka uses a different board representation than Fruit and uses a different structure of search routines, so this should be quite clear. Now it's true that I did take a number of things from Fruit which are above the level of source code. This is something that I discussed many times, something which is perfectly normal practice, and it's not something which is asked about on the entry form.

Rajlich had publicly said as early as 2005 that he had gone through the Fruit 2.1 source "forwards and backwards and took many things." Rybka's readme also thanked Fruit, "which shattered a number of computer chess myths, demonstrated several interesting ideas, and made even the densest of us aware of fail-low pruning." Rajlich also posed a question that applies to LLM-assisted development today:

...if the ICGA wants to open this Pandora's Box of what exactly is an acceptable level of similarity, why is Rybka being singled out? Why is Rybka being the only engine which is investigated?

Fabien Letouzey (author of Fruit) gave a 2005 interview where he said that whether an engine is someone's "own work" made little sense to him, since all engines already share most of their techniques from a body of research going back three decades. When asked about Fruit itself, he couldn't think of a search feature or evaluation term in it that hadn't been described before.

A quick defense of Vasik Rajlich#

The evidence against Rajlich was not his actual source code, but a functional reconstruction based on the compiled binary. At the time, Rajlich's defenders argued that rewriting Rybka's operations using Fruit-like structure, ordering, and terminology would naturally produce something visually similar to Fruit, without proving that Rajlich originally wrote it that way.

A more precise similarity between engines can be measured by their ponder-hit rates - how often two engines pick the same move across a large set of test positions.

Each engine starts at the left edge, and the brackets join engines into clusters from left to right, most similar first. The horizontal position of a junction is the distance between the two sides it connects, rescaled so the most distant merge lands at 25. The colored dots mark the junctions that matter for the case:

• Strelka 2.0, a reverse-engineered copy of Rybka 1.0 Beta, merges with it at a distance of about 1. This is the baseline for a genuine clone.
• Fruit 2.1 joins Loop, Onno, and Toga (an acknowledged Fruit derivative) at that same distance of 1.
• Critter 1.2 joins Houdini and Strelka 5 at a distance of 3, and nobody accused Critter of copying either one. Strong engines simply agree on a lot of moves.
• The Fruit family and the group containing Rybka 1.0 Beta only meet at a distance of 10, further apart than plenty of engines with no history between them.

If Rybka was a Fruit clone, it would sit next to Fruit the way Strelka sits next to Rybka. Miguel Ballicora, who wrote the engine Gaviota, reran the analysis on a larger dataset and found no significant relationship between Fruit and Rybka at all.

Rajlich ended up declining to participate in the ICGA's proceedings, because the public accusations and apparent prejudgment convinced him that he would not be treated fairly.

An abridged history of chess engines#

In the early nineties, world champion Gary Kasparov doubted that a machine could ever beat the human world champion, and most of the chess world agreed with him. In 1997, IBM's Deep Blue beat Kasparov in a six game match.

Kasparov's proposed "advanced chess" the following year, where human and engine play as a team. For a while, the hybrid pairs were capable of beating an unaided chess engine. However, it wasn't long before engines resoundingly defeated any hybrid pairing, and could only be competitive when playing against each other.

Chess engine development went through a similar cycle recently:

1. 2017

   1. AlphaZero

      december 2017

      deep reinforcement learning

      Beat Stockfish 8 over a hundred games with an evaluation it taught itself from scratch.

2. 2018

   1. Leela Chess Zero

      open source · distributed training

      Volunteers rebuilt the idea in the open, training the network on vast collections of chess games using donated GPU power.

3. 2020

   1. Stockfish adopts NNUE

      september 2020

      stockfish 12

      Neural Networks Used Efficiently (NNUE) are small enough to evaluate on a CPU, and run alongside the handcrafted evaluations collected from decades of development.

4. 2023

   1. Handcrafted evaluation deleted

      july 2023

      stockfish 16

      Stockfish maintainers delete 2.7k lines of expert chess knowledge at the cost of around 2 Elo, so development can focus on improving the NNUE.

Decades of hand-crafted positional knowledge, the exact kind of artisanal code being defended in that GitHub thread, was removed from the strongest engine in the world because measurement is the culture.

Stockfish development is driven by fishtest, which validates changes to the engine based on Elo gained across tens of thousands of games. It allows the development community to place its trust in the scoreboard over the author. Trust doesn't depend on the author's reputation, past contributions, and the software development tools that they decided to use.

What can an unassisted human accomplish?#

The question of "what can an unassisted human accomplish" is a legitimate question, but it's a different question from "what is the most that can be accomplished". The ranking of chess engines has only ever measured the latter.

Organized sports is particularly interested in the former, and therefore bans performance enhancers which can be detected with standardized tests. It can be much more difficult to detect plagiarism in computer programs, and the Google vs. Oracle litigation demonstrates how difficult it is even when the source code is available.

It is unrealistic to have any sort of objective test that determines whether a clever move-ordering heuristic came from the author's intuition, a 2009 TalkChess thread, a page of the Chess Programming Wiki, or a model. The Rybka verdict was decided by human judgment, and the investigation was led by Robert Hyatt - author of Crafty, one of the two engines Rybka was accused of copying. Essentially, Rybka's competitors sat in judgment of the reigning champion on a question with no objective test.

What the frontier is for#

In my opinion, the natural direction for this hobby is an "establishment of the frontier". This has been taking place in virtually all fields of human expertise over the past year, where LLMs are applied by domain experts to vastly enhance what humans were capable of accomplishing without assistance.

Ideas have always been abundant in engine development. If you spend time on chessprogramming.org, you'll eventually find a fresh direction to look. At least for me personally, the scarcity has been in human stamina: every theory costs weeks to implement and validate before you learn whether it was worth an Elo point or not. A model collapses the cost of implementation, so human attention can shift toward choosing the theories to test and judge. Software development is fundamentally just theory-building, and being allowed to focus on theory-building actually makes the hobby a lot more fun for me. It also makes the community approachable by the game theorist who struggles to write C++.

A caveat on amateur operators#

An LLM will produce a plausible alpha-beta enhancement whether or not it's fundamentally sound, because plausibility is what it optimizes for. The error correction has to come from outside the model, through rigorous testing and expert review. An LLM works best as a force multiplier in the hands of someone who already understands the field. However, if expertise is the safety mechanism, a community of experts using these tools in the open is exactly where the tool belongs.

I also recently saw this popular article on using LLMs to visualize chip fabrication. The author asks a model to build a knowledge base around a topic he doesn't have deep expertise, asks the same model to review its own accuracy, then has it generate a RollerCoaster Tycoon style simulation, and publishes the result with the claim that it is "100% accurate and free of hallucinations." It is surprisingly easy in that loop to diverge from ground truth.

As a private study aid, it's a genuinely fun idea, but when it is published as teaching material, it can be a distorted view of reality. An engine developer using an LLM is still responsible for testing every idea they generate against test suites, and the ground truth is the difference in Elo points. Whether verification is possible, and whether anyone is doing it, matters far more than whether a model was involved.

Who gets credit#

The Rybka affair has a second half, which Riis documents. In the years before the ICGA acted, Rybka itself was taken apart wholesale. In 2007 a programmer using the pseudonym "Jury Osipov" released Strelka, a reverse-engineered Rybka 1.0 Beta, and published its source code a year later. In 2009 a group working under fake names disassembled Rybka 3 and released it as IPPOLIT, which spawned a family of strong derivatives. Robert Houdart, whose engine Houdini sat at the top of the rating lists, credited them directly: "Without many ideas and techniques from the open source chess engines Ippolit and Stockfish, Houdini would not nearly be as strong as it is now."

So the author who named his sources in his readme got a lifetime ban, while the pseudonymous disassemblers whose work fed every leading engine faced nothing. The ICGA itself published a disassembly of Rybka 2.3.2a, a commercial product still under license, to make its case.

The workable norm is scrutiny and disclosure: publish your methods, name your sources, and let rating lists distinguish engines by what their authors disclose. That asks more of LLM users, and it's a reasonable ask. A ban asks nothing of them except silence.

The Stockfish contributor's complaint, that optimizations he spent real time ideating and validating were absorbed without attribution, is legitimate, and a byline in a list of engines the model learned from is thin recompense. But this is a provenance problem, and the hobby has fought provenance battles for many years before LLMs.

Nobody loses the craft#

Using an LLM to generate the finished piece doesn't make us craftsmen; it just robs us of the craft.

If the process is the product for you, every part of the process is still there. You can derive your evaluation terms by hand the way people still cut wood by hand. The pleasure of artisanal woodworking remains the same, regardless of what people build with power tools.

Every engine author already stands on a common corpus of knowledge: chess engineering wikis, decades of forum archives, and strong open-source models like Stockfish and Leela Chess Zero.