The 16 Top Chess Engine Championship: TCEC16: Reading, UK and Maryland, USA
The 16 Top Chess Engine Championship: TCEC16: Reading, UK and Maryland, USA
TCEC Season 16 started on July 14th 2019 with a revised structure as active engine authors are forming
a longer queue to join the action. ‘Divisions’ are now ‘leagues’: divisions 4 and 3 were replaced by a
larger Qualification League. Leagues 1 and 2 with 16-18 engines were double the size of their predeces-
sors. Fig. 1 and Table 1 provide the logos and details on the enlarged field of 46 engines. Elo figures
seem to be getting higher but it is of course only the Elo differences that are significant.
PD
L1
{
L2
{
QL
{
Fig. 1. Logos for the TCEC 16 engines (CPW, 2019) as in their original leagues and divisions.
Promotion/relegation rules have changed but otherwise the rules are as for TCEC15. In the event of
network breaks, if both engines were in the 7-man and/or TCEC win (or draw) zone, the game was
Four engines of different kinds were taking part in a TCEC event for the first time: ASYMPTOTE,
LCZEROCPU, CHESSFIGHTERNN and STOOFVLEES II. The latter two are neural network engines utiliz-
ing GPUs. LCZEROCPU is a non-GPU version of LCZERO running on conventional CPUs: it was
participating only for demonstration purposes: its results were not ultimately considered.
1 Qualification league: 1 DRR phase, 2 RRs, 34 rounds, 306 games @ 30+5/m
This time, each engine played both White and Black from four-ply openings defined by the second
author here using random choice from an opening book. The results are as in Table 2: ‘P%’ is the %-
score before LCZEROCPU’s results were discounted. Generic stats are in Tables 9 and 10: the Elo dif-
ference of 932 between highest- and lowest-rated heralded many decisive results and 2-0 scores.
The non-GPU LCZEROCPU performed remarkably well, seemingly evaluating well with low node-
counts. It sustained only one loss, to STOOFVLEES II, and made third place its own. While it may have
affected the authors psychologically if not their engines, the removal of its results did not change the
list of promotees: SCORPIONN, STOOFVLEES II, MARVIN, WASP, TOPPLE and CHESS22K. Would they
finish in this order in League 2? Wool (2019) covered this tournament overture somewhat lightly but
notes that SCORPIONN and TOPPLE seem to have improved on their TCEC15 performances. Would the
march of the neurons continue?
Table 2. The TCEC16 Qualification League cross-table: one DRR phase, 2 RRs, 34 rounds, 306 games.
League 2 games were launched with random 4-move rather than 2-move openings. The two neural
network engines here, STOOFVLEES II and SCORPIONN continued on to League 1 but in different ways.
SCORPIONN was stolid, unbeaten but relatively win-shy. STOOFVLEES II won seven more games but
clearly has its vulnerabilities, losing games 27, 39 and 192. In the last, 40…Ng4 against lowly MARVIN
is inexplicable, maybe a last but tardy attempt at complication. (The second author exclaimed in chat
that STOOFVLEES’ defeats were “sudden and violent”.) The lack of QPP-Q endgame table support for
GULL in game 63 was decisive.
PEDONE and a greatly improved ARASAN occupied the other promotion spots. FRITZ, though unbeaten,
missed out because it fell short on the tiebreaker, number of wins. TOFFLE, MARVIN and CHESS22K
returned to the Qualification League with PIRARUCU in attendance. The remaining promotee, WASP,
made the grade reasonably comfortably. Wool (2019) commented lightly on games 22, 26, 27, 39, 63,
84 and 108 from the first round robin.
Table 3. The TCEC16 League 2 cross-table: one DRR phase, 2 RRs, 30 rounds, 240 games.
And so we arrived at the third all-play-all, newly renamed ‘League 1’.2 In a large field of 16 engines,
would STOOFVLEES II and SCORPIONN continue their upwardly mobile transit of TCEC? Perhaps we
should apologise now that, faced with such large divisions, we are almost certainly missing excellent
performances by many engines: a set of event-triggers would be useful. Because of the enlarged field,
TCEC introduced a 4-way play-off for the two promotion spots, not quite the EFL’s way but close.
Table 4 covers the league results and Table 5, the play-off.
In the first half, STOOFVLEES II beat ETHEREAL, ETHEREAL beat LASER: SCORPIONN lost to both.
XIPHOS did not make hay against what became the back end of the field. PEDONE’s win over
STOOFVLEES II was a clear outlier which kept things close at the top. In the second half, little changed
at the top though SCORPIONN beat STOOFVLEES II in the return game. Down the field, CHIRON,
ANDSCACS and GINKGO shipped some unexpected losses to back-markers from CHESSBRAINVB to
ARASAN. So, with all games played, there was a clear gap between the top four heading for the play-
off, and XIPHOS, CHIRON and ANDSCACS in the next places. The neural network version of FIRE,
replacing its venerable Shannon-AB namesake, was a disappointment. Whether this was due to neural
net immaturity or some other defect is unclear. It was demoted to League 2 with PEDONE and ARASAN.
There were six technical defaults, always anticlimactic; three by BOOOT led to its automatic demotion,
saving ROFCHADE from relegation.
In the play-off, two double round robins, 24 games, STOOFVLEES II won three games, defaulted once
but still won. SCORPIONN came second by virtue of losing only one game whereas ETHEREAL and
LASER lost three each. So, STOOFVLEES II and SCORPIONN march on from the Qualification League to
the Premier Division: the audience was keen to know if they would stay there.
‘Kingscrusher’ (2019a-2019d), aka CM Tryfon Gavriel and a fount of contextual knowledge in his
amiable video coverage is particularly following the ‘new machines’. He commentated on ARASAN–
STOOFVLEES II game 73, STOOFVLEES II–SCORPIONN g88 and STOOFVLEES II–ETHEREAL, both g103
and play-off g24): he was inspired to conjecture new opening and middle-game theory. Assaf Wool
2 The ‘EFL’ English Football League now has ‘League Division 1’ - which used to be League Division 3.
(2019), or should that be ‘Assaf Wool’,3 opens doors onto quite a few games.4 It is worth following this
blog: it is an easy introduction to the games played.
Table 4. The TCEC16 League 1 cross-table: two DRR phases, 28 rounds, 112 games.
Table 5. The TCEC16 League 1 Play-off cross-table: two DRR phases, 12 rounds, 24 games.
The Premier Division featured a minority of classic Shannon-AB engines for the first time. SCORPIONN
and STOOFVLEES II have joined LEELA CHESS ZERO, ALLIESTEIN and KOMODOMCTS. These engines
have come through the ranks remarkably quickly and now challenge for podium places. As a result,
there is some interest in every game. Would the engines STOCKFISH and LC0 take their burgeoning
relationship to the next step or would one of the more recent arrivals make it to the Superfinal? How
would the Shannon-AB engines fare against the new wave of Monégasque ‘MCTS-ers’ and ‘Neural
Networks’, sometimes in combination? We give the headlines on each of the three ‘DRR’ double round
robins in turn.
DRR1: STOCKFISH beat STOOFVLEES II in game 2, SCORPIONN in g9 and ALLIESTEIN in g46 – the
longest 1-0 win in the division. Meanwhile KOMODOMCTS beat ALLIESTEIN in g6 and ALLIESTEIN
beat KOMODO in g52. STOCKFISH’s only loss was a technical default in a defensible position to
HOUDINI, g51: half a point gone. Despite this, STOCKFISH and LEELA led after 14 games with
ALLIESTEIN, KOMODO and STOOFVLEES II in joint third, one point behind. KOMODOMCTS and
SCORPIONN were well tailed off with no wins against the other engines.
Table 6. The TCEC16 Premier Division cross-table: three DRR phases, 42 rounds, 168 games.
Table 7. The TCEC16 Premier Division figures: head-to-head and round-by-round scores.
TCEC themselves chose g113, LEELA’s win over SCORPIONN, as their top game here – 122 moves
closed out by a 6-man ‘endgame adjudication’ in a position a novice could have won. ALLIESTEIN–
STOOFVLEES II, g123, was an important 1-0 win, unusually ending in two rooks mating a lone King.
Kingscrusher’s (2019e-2019k) video commentaries covered games 01, 22, 50, 64, 103, 109 and 123.5
Perhaps to correct the apparent bias towards STOOFVLEES II above, he featured LEELA CHESS ZERO,
SCORPIONN, KOMODOMCTS and ALLIESTEIN, sometimes facing each other, at other times, victorious
over the Shannon-AB engines, the three amigos as it were. Wool (2019) featured the key 1-0 wins,
5In the shorthand of Table 1, these games were LC0-SN, LC0-KO, KM-LC0, LC0-ST, LC0-HO, SN-AS & AS-SV. Games 22,
64 and 109 were also TCEC’s top three picks.
games 02, 52, 64, 102, 136 and 158.6 He also provided brief notes on a further 44 games, a prodigious
contribution which was much appreciated.7
STOCKFISH stepped on stage having barely avoided disqualification: its supporters breathed again.
Entr’acte change can increase technical risk. A defective .dll was not in the recompile.
We can identify three dimensions today in the chess engine space:
1) the engine: code driving (CPU, GPU, TPU …) servers and neural network topology,
2) the search algorithm: minimax/alpha-beta or some form of Monte Carlo Tree-Search,
3) the training regime used, both data and process:
data - tabula rasa null input or some selection of previously played games,
process – the sequencing and batching of input, aspects of the training code.
It is worth re-introducing ALLIESTEIN in this 3D-content, particularly as the engine is only in its second
TCEC season. It is a synergy of ALLIE from Adam Treat and STEIN, a neural network trained by Mark
Jordan in a supervised learning process,8 in this case on 3m+ CCRL and 40m+ LCZERO self-play games.
The excellent Chessprogramming Wiki site on ALLIE (CPW, 2019b) and Chessdom’s (2019) interview
with Adam and Mark provide useful graphics and more information.
With regard to the second dimension, search, ALLIESTEIN can use either MCTS or minimax/alpha-beta
to search for and decide its move. Both approaches have their merits, one taking the broader view, the
other betting the house on a narrow and possibly treacherous path. Perhaps MCTS and minimax can be
used in conjunction in some future engine.
The training process relies on the quality of the input and some aspects of the training itself. For
example, is the choice of CCRL games as input not only to avoid human fallibility over the board but
to inherit some tactical sharpness from decades of Shannon-AB evaluation functions honed on the
experience of classical minimax/alpha-beta search?
Both engines came to the Superfinal in new versions, STOCKFISH with code improvements and
ALLIESTEIN two weeks older and wiser. ALLIESTEIN played as White first. Kibitzing on the match, we
had two engines, REDFISH9 (stronger than the competing STOCKFISH) and BLUELEELA.10 For the first
time, we include in our report the evaluation graphs for the decisive games. STOCKFISH’s evaluations
may be benchmarked against those of REDFISH, and ALLIESTEIN’s against those of BLUELEELA.
Readers are invited to provide feedback as to the specific value of these evaluation graphs.
Also, for the first time, we can defer to GM Matthew Sadler’s (2019) perspective which is written in his
inimitably accessible style. He provides rich insight and context, including the 2018 ALPHAZERO–
STOCKFISH match, for this event. His focus is particularly on games 3, 5-6, 14, 24, 25-26, 41-42, 75 and
100. As usual, we commend the ‘My Experience’ blog (Wool, 2019) which covered all the decisive
games in some detail and touched on others. In his TCEC16 Superfinal playlist, ‘GM Thechesspuzzler’
6 These games were ST-SV, AS-KO, LC0-ST, ST-AS, KO-AS and ST-AS.
7
Wool also provided brief notes on games (DRR1) 1-3, 6, 9, 17, 23, 25, 34, 41, 46, & 49-52, (DRR2) 58, 62, 65, 67, 69,
73-4, 78, 81, 88-90, 95, 100, 103 & 109, (DRR3) 113-4, 116, 123, 135, 139, 141, 144, 149-151, 159 & 163.
8 DEUSX was the first neural network employing supervised learning to compete in TCEC (Season 13).
9 Shannon-AB STOCKFISH 2019100810, 256 threads on 8 Xeon Scalable 8168s, plus sub7man EGTs.
10 NN-based LEELA CHESS ZERO v0.22.0-dev-nJ13B.2-136 (24x320) on NVIDIA GPU 2080 TI with s8man EGTs.
(2019) covered games 6, 14, 19, 20 and 100. Other commentators – ‘Agadmator’, ‘ElzChess’ and
Jozarov are getting involved and all are welcome, especially if they make it clear where their TCEC
videos are!
g75 g99
g61 g93
Fig. 3. The decisive games’ evaluation graphs: ALLIESTEIN’s 5 wins, all 1-0, games 03, 27, 39, 75 & 99;
STOCKFISH’s (a) 12 1-0 wins, games 06, 12, 14, 26, 32, 42, 46, 56, 60, 64, 80 & 100 and (b) 2 0-1 wins, games 61 & 93.
The contestants stood +3=2-1 in their Premier Division head-to-head: advantage STOCKFISH. This did
not make ALLIESTEIN the favourite for the title and there was some concern that STOCKFISH would
soon be well ahead on the scoreboard. It was not to be.
After two draws, first blood went to ALLIESTEIN in game three. It was already more confident of a win
than the kibitzing BLUELEELA at move 23 when STOCKFISH apparently eased its grip on the position.
Evaluations increased and by move 36, all engines were seeing the first result of the match. ALLIESTEIN
exploited its space advantage, created a passed a-pawn, and STOCKFISH, in an unavailing defence, found
itself down rook-vs-knight and on the wrong end of a TCEC adjudication. Nevertheless, it would have
been good to see the terminal, pawnful position played out for a few more moves. It has been: our
Superfinal pgn file provides plausible continuations of the decisive games.
The 19-ply opening of games five and six favoured White but STOCKFISH found a perpetual check in
game 5 with its rook-sac 17…Rxb2. Game 6 (Sadler, 2019) went away from ALLIESTEIN by move 22
and was a done deal by move 33. Honours even: 1-1.
Table 8. The TCEC 16 Superfinal match of 100 games: the decisive games, Black wins underlined.
# S uperfinal Elo Pts P% Elo ± # of games won (underline = '0-1') Openings # of win-pairs
Stockfish 06, 12, 14, 26, 32, 42, 46, 56, B48, A65, C19, E16, C27, D70, A10,
1 3892 54½ 54.5% -170 14
19092522 60, 61, 64, 80, 93, 100 B06, C75, A45, C11, C96, D35, B35 99-100
1
AllieStein (1-0, 1-0)
2 3823 45½ 45.5% +170 05 03, 27, 39, 75, 99 E98, B01, C11, B07, B35
v0.5-dev_7b41f8c-n11
Chart Title Chart Title
20
8
15
6
10
4
5 2
0 0
0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100
-2
Fig. 2. The TCEC 16 Superfinal: the incidence of decisive results and STOCKFISH’s lead.
Game 12 was even until ALLIESTEIN, after running down its thinking time, lost the plot with 33…Kf1
and 34…Nd6. GM Matthew Sadler covers STOCKFISH’s wins in game 14 (resulting from 33…Qc6?)
and game 26, a rare and dramatic mate on move 34, so we skip to game 27. On move 39, ALLIESTEIN’s
confidence in a result really rockets but the other engines do not see anything at all. By the way,
Chessbomb’s (2019) kibitzing STOCKFISH 10 is not the best engine to spot the affine and DNA-adjacent
TCEC STOCKFISH’s errors: the game was lost before any ‘red moves’ were flagged there.
Game 32 was a STOCKFISH win and here Chessbomb (2019) puts big question marks against Black’s
5…Nc6 which was actually in the provided opening. ALLIESTEIN seems at odds with these first few
moves and there are questions about its 12…h5 and 18…f4. Opportunity comes over the horizon for
STOCKFISH after ALLIESTEIN played 32…Kc7 without much thought: time-management was beginning
to look like an issue. It remained blissfully unaware of problems until move 39 when it realised that its
three connected passed pawns were not enough.
In game 39, after the defined opening of ten moves, ALLIESTEIN’s prescience was never matched by
the kibitzers but proved justified: confidence was high by move 19. This game saw White use the ‘thorn
pawn’ strategy – pawn to h6, see various Kingscrusher videos – and reduced the efficacy of Black’s two
bishops. It all worked: 1-0. So, after the first 40 games, the score was 21-19, an Elo difference of only
18.
However, after this, STOCKFISH went on a roll – six wins without reply starting with game 42 where
Sadler (2019) made comparison with ALPHAZERO’s play (DeepMind, 2018; Sadler and Regan, 2019,
esp. p230). The other wins were in games 46, 56, 60, 61 (as Black) & 64.11
In game 46, STOCKFISH’s attack centred on its e-pawn and both engines had no doubts on move 45 that
44. e7 Qa2 sealed the game for White. In game 56, ALLIESTEIN was curiously ahead of STOCKFISH in
thinking it was lost. It happens: these engines think they are playing themselves and that they’re the
only ones in the room. Game 60 was in some ways similar to game 46: STOCKFISH pushed a passed
pawn to the seventh rank and tied up valuable ALLIESTEIN resources in defence. Again, it was shortly
after move 40 that both engines saw the game as decisive. Game 61 was the second half of a combination
punch, the only case of two wins in a row and the first win for Black. Jozarov (2019b) covers this game.
Here, a turning point was 24. a4 Bxf4 where ALLIESTEIN missed STOCKFISH’s decisive discovered
attack and mate threat.
Jozarov (2019c) covers game 64. Here again, ALLIESTEIN missed another Bishop-sac with benefits, this
time a confident 23. Bxh7+ which started a classic assault on the Black castle, admittedly from a pos-
ition of strength. ALLIESTEIN won in game 75, see Sadler (2019). Game 80 looked drawish for 160
moves until STOCKFISH’s 164. Ree1 seduced Black’s wandering king one further step into 164…Kxf3.
Further mistakes by ALLIESTEIN simply shortened a well-fought game which ended with unbalanced
heavy artillery, QR-vs-RR, on the field. The match was formally over scorewise after STOCKFISH’s
second win as Black in game 93.
Jeroen Noomen chose the exciting Sicilian Dragon opening to end the Superfinal and we were rewarded
for the first time with two wins by White, a triumphant climax to this festival of chess. Sadler and
Jozarov (2019d) both cover game 100.
6 In conclusion
Congratulations are once again due to the STOCKFISH team as their engine in its various versions has
become TCEC Grand Champion for the seventh time. The trophies for seasons 6, 9, 11-14 and 16 should
make a fine sight in their cabinet. Plaudits should also go to ALLIESTEIN’s team as their engine did far
better than expected. ALLIESTEIN’s Superfinal performance rating is only some 31 Elo points behind
STOCKFISH: as it is relatively new out of the box, it presumably has some potential. We imagine this is
not ALLIESTEIN’s last TCEC Superfinal.
As ever, we thank the small TCEC team who keep everything together with increasing success: their
success is inversely proportional to their visibility. We thank all participants in this season’s events,
both the leading TCEC commentators and the chat room. So much good chess would go by, especially
in the drawn games, without getting the appreciation it deserves if the online audience was not kibitzing
and sharing their perspectives. Even at Rapid tempo as in the early leagues here, in TCEC Cup 4
(Haworth and Hernandez, 2019b) and arguably at Blitz tempo, most if not all of the TCEC engines are
playing at super-GM levels. All the games and supplementary data can be revisited at leisure (Haworth
and Hernandez, 2019a) with key decisive ones, including all those in the Superfinal, played out. Some
11 Some ALLIESTEIN parameters were discovered around game 45 to be in error: it’s not clear what the effect was.
of these completions are less obvious than others. The generic stats and shortest/longest games are in
Tables 9 and 10.
With such an excess of riches, it would be useful to have some tools to alert us to the events that we
variously look forward to – specific engines and contests, opening novelties, minor exchanges, sacri-
fices, unbalanced forces, the leaps in the evaluation curves, endgames, unexpected results and so on.
What toolkit is to hand today? Is there a new AI challenge here? We end with that thought while looking
forward to Season 17.
Table 9. Generic statistics for each phase of TCEC16: results, terminations and average game-length.
1-0 114 37.3 75 31.3 61 25.4 5 20.8 35 20.8 17 17.0 307 28.5
0-1 91 29.7 45 18.8 29 12.1 3 12.5 13 7.7 2 2.0 183 17.0
White performance 164.5 53.8 135.0 56.3 136.0 56.7 13.0 54.2 95.0 56.5 57.5 57.5 601 55.8
Black performance 141.5 46.2 105.0 43.8 104.0 43.3 11.0 45.8 73.0 43.5 42.5 42.5 477 44.2
TCEC draw 34 11.1 62 25.8 70 29.2 8 33.3 41 24.4 21 21.0 236 21.9
3x repetition 28 9.2 20 8.3 38 15.8 3 12.5 30 17.9 20 20.0 139 12.9
50-move rule 4 1.3 3 1.3 1 0.4 0 0.0 8 4.8 14 14.0 30 2.8
Stalemate 0 0.0 0 0.0 0 0.0 0 0.0 1 0.6 0 0.0 1 0.1
Terminations
EGT adj., 'draw' 35 11.4 35 14.6 41 17.1 5 20.8 40 23.8 26 26.0 182 16.9
EGT adjudication 110 35.9 70 29.2 64 26.7 7 29.2 56 33.3 26 26.0 333 30.9
TCEC win 94 30.7 74 30.8 54 22.5 3 12.5 20 11.9 18 18.0 263 24.4
EGT adj., 'win' 75 24.5 35 14.6 23 9.6 2 8.3 16 9.5 0 0.0 151 14.0
Tech. default 1 0.3 2 0.8 6 2.5 1 4.2 2 1.2 0 0.0 12 1.1
M anual adj. 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
M ate 33 10.8 8 3.3 5 2.1 1 4.2 4 2.4 1 1.0 52 4.8
Loss on time 2 0.7 1 0.4 0 0.0 0 0.0 0 0.0 0 0.0 3 0.3
Resignation 0 0.0 0 0.0 2 0.8 1 4.2 6 3.6 0 0.0 9 0.8
M oves 69.0 69.4 66.6 71.5 83.5 89.0 72.7
Length
Table 10. The shortest and longest 1-0, drawn and 0-1 games in each phase of TCEC16.