user17540: Chess engines are so strong that it is becoming increasingly hard to find positions that they are demonstrably evaluating incorrectly. I'm interested in collecting examples—if there are any—of positions where we know the correct evaluation, but that are very difficult for engines.
Below is a beautiful study that I encountered recently in Chapter 5 of the book Barrycades and Septoku: Papers in Honor of Martin Gardner and Tom Rodgers, by Thane Plambeck and Tomas Rokicki (eds.), MAA Press, 2020.
[Title "Carlos Pereira dos Santos, White To Play And Win"]
[FEN "bR2r3/2nK4/kp1pp1pp/p1R2p2/1P1P1P1P/p7/P2P3P/8 w - - 0 1"]

The engines I have access to are old and out of date, but after 1.b5+ Ka7 they want to play 2.Rxe8? and miss the winning continuation 2.Rxa8! Nxa8 3.Rc8! Rxc8 4.Kxc8 with a win for White. I would be curious to know if stronger engines are able to find this win.  [EDIT: Thanks to Oscar Smith for pointing out that I can download Stockfish 11 for free. It had some difficulty at first, but after searching about 2.5 billion nodes it finally found 2.Rxa8!]
Years ago, there was a ChessBase article on this topic.  They give some examples of positions, such as fortresses, where the engines' numerical evaluation is way off.  However, sometimes the engine plays the position correctly anyway, finding the drawing moves even though it thinks it's losing, and such examples are not so interesting to me.  What is more interesting to me is a position where the engine loses a position that a perfect player would draw, or draws (or even loses) a position that a perfect player would win.
Probably the best example from that article is a study by Behting, shown below.
[Title "K. K. Behting, Baltische Schachblätter 1908, White To Play And Draw"]
[FEN "8/8/7p/3KNN1k/2p4p/8/3P2p1/8 w - - 0 1"]

As of 2012, careful analysis demonstrated that the study is sound but that engines had serious trouble with it.  Is that still true today?  [EDIT: It seems so; after searching 25 billion nodes, Stockfish 11 was still unsuccessfully trying the 1.Ng7+ line that is analyzed in detail here.]
There is one other type of example that I know about, namely tablebase positions where White has an extremely subtle win.  For example, below is a position that I found on Wikipedia.
[Title "Lomonosov Tablebase, White To Play And Win"]
[FEN "8/r6n/2Q5/8/5k2/3K4/7N/3b3q w - - 0 1"]

Clearly, if an engine without tablebases plays White and an engine with tablebases plays Black, the engine playing White is not going to win.  Sooner or later it is bound to blunder away the win.  I would be curious if there are any humanly comprehensible examples of this type.
In the process of typing up this question, I found another question on this site that asks a very similar question.  Unfortunately, some of the answers there seem to be (no longer?) valid.  Also, let me emphasize again that for my purposes, a fortress position that the engine thinks is losing but that it plays correctly anyway does not count.
In short, my question is this:

Are there other good examples of positions that stump engines?


user15223: There are plenty of positions that stump engines, and it's not too difficult to find some from engine tournaments. Example, this position stumped Stockfish:
[FEN "1k1r4/pp4pp/4pb2/2Q5/2P5/8/P5PP/6K1 w - - 1 28"]

Stockfish actually reached +2.76 eval later, even though White has no way to break through. In fact Black can afford to give up the e6-pawn and the position is still drawn. Lc0 evaluated the position accurately as a draw, which doesn't mean Lc0 is some kind of goddess at evaluating fortresses, since this position stumped Lc0:
[FEN "5k2/5p2/p3pPp1/3pN2p/b6P/2P5/P5P1/7K w - - 0 35"]

Black plays ...Bb5 and defends along the f1-a6 diagonal, and White will never get anywhere, especially since if the Knight moves too much Black advances the e6-pawn with counterplay. This time Stockfish accurately assessed the position as drawn, while Lc0 was at +1.6 - or about 60% chance to win - or so for quite a while.
If you're wondering about positions with study-like solutions that humans think is pretty - that's a different question.

17540: <p>Chess engines are so strong that it is becoming increasingly hard to find positions that they are demonstrably evaluating incorrectly. I'm interested in collecting examples&mdash;if there are any&mdash;of positions where we know the correct evaluation, but that are very difficult for engines.</p>

<p>Below is a beautiful study that I encountered recently in Chapter 5 of the book <i>Barrycades and Septoku: Papers in Honor of Martin Gardner and Tom Rodgers</i>, by Thane Plambeck and Tomas Rokicki (eds.), MAA Press, 2020.</p>

<pre><code>[Title "Carlos Pereira dos Santos, White To Play And Win"]
[FEN "bR2r3/2nK4/kp1pp1pp/p1R2p2/1P1P1P1P/p7/P2P3P/8 w - - 0 1"]
</code></pre>

<p>The engines I have access to are old and out of date, but after 1.b5+ Ka7 they want to play 2.Rxe8? and miss the winning continuation 2.Rxa8! Nxa8 3.Rc8! Rxc8 4.Kxc8 with a win for White. I would be curious to know if stronger engines are able to find this win.  [<b>EDIT:</b> Thanks to Oscar Smith for pointing out that I can download Stockfish 11 for free. It had some difficulty at first, but after searching about 2.5 billion nodes it finally found 2.Rxa8!]</p>

<p>Years ago, there was a <a href="https://en.chessbase.com/post/can-we-still-deceive-computers-in-chess" rel="nofollow noreferrer">ChessBase article</a> on this topic.  They give some examples of positions, such as fortresses, where the engines' numerical evaluation is way off.  However, sometimes the engine plays the position correctly anyway, finding the drawing moves even though it thinks it's losing, and such examples are not so interesting to me.  What is more interesting to me is a position where the engine loses a position that a perfect player would draw, or draws (or even loses) a position that a perfect player would win.</p>

<p>Probably the best example from that article is a study by Behting, shown below.</p>

<pre><code>[Title "K. K. Behting, Baltische Schachblätter 1908, White To Play And Draw"]
[FEN "8/8/7p/3KNN1k/2p4p/8/3P2p1/8 w - - 0 1"]
</code></pre>

<p>As of 2012, careful analysis demonstrated that <a href="https://en.chessbase.com/post/john-nunn-the-behting-study-is-sound" rel="nofollow noreferrer">the study is sound</a> but that engines had serious trouble with it.  Is that still true today?  [<b>EDIT:</b> It seems so; after searching 25 billion nodes, Stockfish 11 was still unsuccessfully trying the 1.Ng7+ line that is analyzed in detail <a href="https://en.chessbase.com/post/john-nunn-the-behting-study-is-sound" rel="nofollow noreferrer">here</a>.]</p>

<p>There is one other type of example that I know about, namely tablebase positions where White has an extremely subtle win.  For example, below is a position that I found on <a href="https://en.wikipedia.org/wiki/Endgame_tablebase#Computer_chess" rel="nofollow noreferrer">Wikipedia</a>.</p>

<pre><code>[Title "Lomonosov Tablebase, White To Play And Win"]
[FEN "8/r6n/2Q5/8/5k2/3K4/7N/3b3q w - - 0 1"]
</code></pre>

<p>Clearly, if an engine <i>without</i> tablebases plays White and an engine <i>with</i> tablebases plays Black, the engine playing White is not going to win.  Sooner or later it is bound to blunder away the win.  I would be curious if there are any humanly comprehensible examples of this type.</p>

<p>In the process of typing up this question, I found <a href="https://chess.stackexchange.com/q/6297">another question on this site</a> that asks a very similar question.  Unfortunately, some of the answers there seem to be (no longer?) valid.  Also, let me emphasize again that for my purposes, a fortress position that the engine thinks is losing but that it plays correctly anyway does not count.</p>

<p>In short, my question is this:</p>

<blockquote>
  <p><b>Are there other good examples of positions that stump engines?</b></p>
</blockquote>
 15223: <p>There are plenty of positions that stump engines, and it's not too difficult to find some from engine tournaments. Example, <a href="https://tcec-chess.com/#div=sf&amp;game=61&amp;season=17&amp;x=archive" rel="noreferrer">this position stumped Stockfish</a>:</p>

<pre><code>[FEN "1k1r4/pp4pp/4pb2/2Q5/2P5/8/P5PP/6K1 w - - 1 28"]
</code></pre>

<p>Stockfish actually reached +2.76 eval later, even though White has no way to break through. In fact Black can afford to give up the e6-pawn and the position is still drawn. Lc0 evaluated the position accurately as a draw, which doesn't mean Lc0 is some kind of goddess at evaluating fortresses, since <a href="https://tcec-chess.com/#div=sf&amp;game=40&amp;season=17&amp;x=archive" rel="noreferrer">this position stumped Lc0</a>:</p>

<pre><code>[FEN "5k2/5p2/p3pPp1/3pN2p/b6P/2P5/P5P1/7K w - - 0 35"]
</code></pre>

<p>Black plays ...Bb5 and defends along the f1-a6 diagonal, and White will never get anywhere, especially since if the Knight moves too much Black advances the e6-pawn with counterplay. This time Stockfish accurately assessed the position as drawn, while Lc0 was at +1.6 - or about 60% chance to win - or so for quite a while.</p>

<p>If you're wondering about positions with <em>study-like solutions</em> that humans think is pretty - that's a different question.</p>
