user28405: Is there a fast way to generate FEN strings for every move in a PGN?
I'm generating an opening book using the Lichess elite database and Python Chess, but Python Chess is taking a surprisingly long time to analyze games and generate FEN strings for each move. Like, 1/1000th the speed I expected.
This is how I've done it - it works, but again, slow. board = game.board() is the primary slowdown - but I don't see a way to get FENs directly from the game object.
# PARSE PGNS and create an opening book using a giant dictionary

import chess
import chess.pgn
import io
import datetime

#START WITH SOME DUMMY DATA FOR DEBUGGING

games = '''[Event "Rated Rapid game"]
[LichessURL "https://lichess.org/YcFqJqyM"]
[Date "2020.06.01"]
[Round "-"]
[White "Poecraft94"]
[Black "Germanvince"]
[Result "1-0"]
[WhiteElo "2323"]
[BlackElo "2415"]
[ECO "B50"]
[Opening "Sicilian Defense"]
[TimeControl "600+0"]
[UTCDate "2020.06.01"]
[UTCTime "00:00:01"]
[Termination "Normal"]
[WhiteRatingDiff "+9"]
[BlackRatingDiff "-8"]

1. e4 c5 2. Nf3 d6 3. Nc3 Nf6 4. g3 g6 5. Bg2 Bg7 6. O-O O-O 7. d4 cxd4 8.
Nxd4 a6 9. Re1 Nbd7 10. Bg5 Rb8 11. Nd5 e6 12. Nxf6+ Bxf6 13. Bxf6 Nxf6 14.
Qd2 Qc7 15. Rad1 Kg7 16. Nf3 Rd8 17. e5 Ne8 18. Re3 b5 19. Rd3 d5 20. Rc3
Qe7 21. Nd4 Bb7 22. Rf3 Rbc8 23. g4 h6 24. Rh3 g5 25. f4 gxf4 26. Rh5 Nc7
27. Qxf4 Rh8 28. Rf1 Rcg8 29. Rf5 Rf8 30. g5 h5 31. Bh3 b4 32. b3 Nb5 33.
Rf6 Nxd4 34. Qxd4 Bc6 35. R1f2 Bd7 36. Qa7 Rh7 37. g6 1-0

[Event "Rated Blitz game"]
[LichessURL "https://lichess.org/AY9kSWFt"]
[Date "2020.06.01"]
[Round "-"]
[White "sandstorm00"]
[Black "yendorzerep"]
[Result "1/2-1/2"]
[WhiteTitle "NM"]
[BlackTitle "IM"]
[WhiteElo "2497"]
[BlackElo "2426"]
[ECO "B06"]
[Opening "Modern Defense: Standard Line"]
[TimeControl "180+0"]
[UTCDate "2020.06.01"]
[UTCTime "00:00:19"]
[Termination "Normal"]
[WhiteRatingDiff "-1"]
[BlackRatingDiff "+1"]

1. e4 g6 2. d4 Bg7 3. Nc3 c6 4. Be3 d5 5. e5 f6 6. exf6 Nxf6 7. Bd3 O-O 8.
h3 Nbd7 9. Nf3 Qc7 10. Qd2 e5 11. dxe5 Nxe5 12. Nxe5 Qxe5 13. O-O-O Bf5 14.
Bxf5 Qxf5 15. g4 Qf3 16. Rhg1 Ne4 17. Nxe4 Qxe4 18. h4 a5 19. h5 a4 20. a3
Rae8 21. Rde1 Qe5 22. c3 Qd6 23. f4 Re4 24. hxg6 hxg6 25. f5 Rfe8 26. Bf2
Rxe1+ 27. Bxe1 d4 28. Bf2 Re4 29. fxg6 Qxg6 30. Bxd4 Bxd4 31. cxd4 Rxg4 32.
Rxg4 Qxg4 33. Qe3 Kf7 34. Kd2 Qe6 35. Qf4+ Ke7 36. Qc7+ Qd7 37. Qxd7+ Kxd7
38. Kc3 Kd6 39. Kb4 b5 40. b3 axb3 41. Kxb3 c5 42. dxc5+ Kxc5 43. Kc3 Kc6
44. Kb4 Kb6 45. a4 bxa4 46. Kxa4 1/2-1/2'''

# If you have the Lichess elite database, yoou can parse the entire thing by uncommenting here

#with open('/lichess_elite_2020-06.pgn') as f:
#    games = f.read()

games = games.replace("\n\n[", "||||[")
pgns = games.split("||||")

import time

fens = {}
start = datetime.datetime.utcnow()

for i, pgn in enumerate(pgns):
    game = chess.pgn.read_game(io.StringIO(pgn)) #parse each pgn string
    if game.errors:
        continue

    parent_pos = None
    move_num = 0
    
    while game:
        move_num += 1
        if move_num > 40: #practically, we only need to first 20 moves for our book
            break

        game=game.next()
        
        if game:
            board = game.board()
            pos = board.fen(en_passant='fen') #grab fen string of current position
            move = game.san() #what was the last move made?

            if pos not in fens:
                fens[pos]={}

            if parent_pos:
                #in the previous position, the current move was made X times
                if pos in fens[parent_pos]:
                    fens[parent_pos][move] += 1
                else:
                    fens[parent_pos][move] = 1

        parent_pos=pos

print('Run time: ', (datetime.datetime.utcnow()-start).seconds)

This yields a data structure with FEN strings as keys, and the most common replies as values. E.g. after parsing the 2 games, e4 is followed by g6 once and c5 once.
{
    'rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1': {
        'c5': 1,
        'g6': 1}
...
}```


user12951: If you are not tied to using PythonChess and simply want FEN encodings of each position, you might like to consider using my free, open-source pgn-extract program with its -Wepd option to output in EPD format. It is a compiled C program so likely to be significantly faster for this use case. For instance:
pgn-extract --quiet -Wepd games.pgn

Here are the first few lines of its output for one of the games quoted in the OP:
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
rnbqkbnr/pppppp1p/6p1/8/4P3/8/PPPP1PPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;

If you wish to suppress irrelevant en passant squares then there is also the --nofauxep option to do that.

28405: <p>Is there a <strong>fast</strong> way to generate FEN strings for every move in a PGN?</p>
<p>I'm generating an opening book using the Lichess elite database and Python Chess, but Python Chess is taking a surprisingly long time to analyze games and generate FEN strings for each move. Like, 1/1000th the speed I expected.</p>
<p>This is how I've done it - it works, but again, slow. <code>board = game.board()</code> is the primary slowdown - but I don't see a way to get FENs directly from the game object.</p>
<pre><code># PARSE PGNS and create an opening book using a giant dictionary

import chess
import chess.pgn
import io
import datetime

#START WITH SOME DUMMY DATA FOR DEBUGGING

games = '''[Event &quot;Rated Rapid game&quot;]
[LichessURL &quot;https://lichess.org/YcFqJqyM&quot;]
[Date &quot;2020.06.01&quot;]
[Round &quot;-&quot;]
[White &quot;Poecraft94&quot;]
[Black &quot;Germanvince&quot;]
[Result &quot;1-0&quot;]
[WhiteElo &quot;2323&quot;]
[BlackElo &quot;2415&quot;]
[ECO &quot;B50&quot;]
[Opening &quot;Sicilian Defense&quot;]
[TimeControl &quot;600+0&quot;]
[UTCDate &quot;2020.06.01&quot;]
[UTCTime &quot;00:00:01&quot;]
[Termination &quot;Normal&quot;]
[WhiteRatingDiff &quot;+9&quot;]
[BlackRatingDiff &quot;-8&quot;]

1. e4 c5 2. Nf3 d6 3. Nc3 Nf6 4. g3 g6 5. Bg2 Bg7 6. O-O O-O 7. d4 cxd4 8.
Nxd4 a6 9. Re1 Nbd7 10. Bg5 Rb8 11. Nd5 e6 12. Nxf6+ Bxf6 13. Bxf6 Nxf6 14.
Qd2 Qc7 15. Rad1 Kg7 16. Nf3 Rd8 17. e5 Ne8 18. Re3 b5 19. Rd3 d5 20. Rc3
Qe7 21. Nd4 Bb7 22. Rf3 Rbc8 23. g4 h6 24. Rh3 g5 25. f4 gxf4 26. Rh5 Nc7
27. Qxf4 Rh8 28. Rf1 Rcg8 29. Rf5 Rf8 30. g5 h5 31. Bh3 b4 32. b3 Nb5 33.
Rf6 Nxd4 34. Qxd4 Bc6 35. R1f2 Bd7 36. Qa7 Rh7 37. g6 1-0

[Event &quot;Rated Blitz game&quot;]
[LichessURL &quot;https://lichess.org/AY9kSWFt&quot;]
[Date &quot;2020.06.01&quot;]
[Round &quot;-&quot;]
[White &quot;sandstorm00&quot;]
[Black &quot;yendorzerep&quot;]
[Result &quot;1/2-1/2&quot;]
[WhiteTitle &quot;NM&quot;]
[BlackTitle &quot;IM&quot;]
[WhiteElo &quot;2497&quot;]
[BlackElo &quot;2426&quot;]
[ECO &quot;B06&quot;]
[Opening &quot;Modern Defense: Standard Line&quot;]
[TimeControl &quot;180+0&quot;]
[UTCDate &quot;2020.06.01&quot;]
[UTCTime &quot;00:00:19&quot;]
[Termination &quot;Normal&quot;]
[WhiteRatingDiff &quot;-1&quot;]
[BlackRatingDiff &quot;+1&quot;]

1. e4 g6 2. d4 Bg7 3. Nc3 c6 4. Be3 d5 5. e5 f6 6. exf6 Nxf6 7. Bd3 O-O 8.
h3 Nbd7 9. Nf3 Qc7 10. Qd2 e5 11. dxe5 Nxe5 12. Nxe5 Qxe5 13. O-O-O Bf5 14.
Bxf5 Qxf5 15. g4 Qf3 16. Rhg1 Ne4 17. Nxe4 Qxe4 18. h4 a5 19. h5 a4 20. a3
Rae8 21. Rde1 Qe5 22. c3 Qd6 23. f4 Re4 24. hxg6 hxg6 25. f5 Rfe8 26. Bf2
Rxe1+ 27. Bxe1 d4 28. Bf2 Re4 29. fxg6 Qxg6 30. Bxd4 Bxd4 31. cxd4 Rxg4 32.
Rxg4 Qxg4 33. Qe3 Kf7 34. Kd2 Qe6 35. Qf4+ Ke7 36. Qc7+ Qd7 37. Qxd7+ Kxd7
38. Kc3 Kd6 39. Kb4 b5 40. b3 axb3 41. Kxb3 c5 42. dxc5+ Kxc5 43. Kc3 Kc6
44. Kb4 Kb6 45. a4 bxa4 46. Kxa4 1/2-1/2'''

# If you have the Lichess elite database, yoou can parse the entire thing by uncommenting here

#with open('/lichess_elite_2020-06.pgn') as f:
#    games = f.read()

games = games.replace(&quot;\n\n[&quot;, &quot;||||[&quot;)
pgns = games.split(&quot;||||&quot;)

import time

fens = {}
start = datetime.datetime.utcnow()

for i, pgn in enumerate(pgns):
    game = chess.pgn.read_game(io.StringIO(pgn)) #parse each pgn string
    if game.errors:
        continue

    parent_pos = None
    move_num = 0
    
    while game:
        move_num += 1
        if move_num &gt; 40: #practically, we only need to first 20 moves for our book
            break

        game=game.next()
        
        if game:
            board = game.board()
            pos = board.fen(en_passant='fen') #grab fen string of current position
            move = game.san() #what was the last move made?

            if pos not in fens:
                fens[pos]={}

            if parent_pos:
                #in the previous position, the current move was made X times
                if pos in fens[parent_pos]:
                    fens[parent_pos][move] += 1
                else:
                    fens[parent_pos][move] = 1

        parent_pos=pos

print('Run time: ', (datetime.datetime.utcnow()-start).seconds)
</code></pre>
<p>This yields a data structure with FEN strings as keys, and the most common replies as values. E.g. after parsing the 2 games, e4 is followed by g6 once and c5 once.</p>
<pre><code>{
    'rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1': {
        'c5': 1,
        'g6': 1}
...
}```
</code></pre>
 12951: <p>If you are not tied to using PythonChess and simply want FEN encodings of each position, you might like to consider using my free, open-source <a href="https://www.cs.kent.ac.uk/%7Edjb/pgn-extract/" rel="nofollow noreferrer">pgn-extract</a> program with its -Wepd option to output in EPD format. It is a compiled C program so likely to be significantly faster for this use case. For instance:</p>
<pre><code>pgn-extract --quiet -Wepd games.pgn
</code></pre>
<p>Here are the first few lines of its output for one of the games quoted in the OP:</p>
<pre><code>rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
rnbqkbnr/pppppp1p/6p1/8/4P3/8/PPPP1PPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
</code></pre>
<p>If you wish to suppress irrelevant en passant squares then there is also the --nofauxep option to do that.</p>
