{"action":"created","issue":{"url":"https://api.github.com/repos/Axelrod-Python/Axelrod/issues/520","repository_url":"https://api.github.com/repos/Axelrod-Python/Axelrod","labels_url":"https://api.github.com/repos/Axelrod-Python/Axelrod/issues/520/labels{/name}","comments_url":"https://api.github.com/repos/Axelrod-Python/Axelrod/issues/520/comments","events_url":"https://api.github.com/repos/Axelrod-Python/Axelrod/issues/520/events","html_url":"https://github.com/Axelrod-Python/Axelrod/issues/520","id":146065933,"number":520,"title":"High (too high?) memory usage when running the full tournament","user":{"login":"drvinceknight","id":2131546,"avatar_url":"https://avatars.githubusercontent.com/u/2131546?v=3","gravatar_id":"","url":"https://api.github.com/users/drvinceknight","html_url":"https://github.com/drvinceknight","followers_url":"https://api.github.com/users/drvinceknight/followers","following_url":"https://api.github.com/users/drvinceknight/following{/other_user}","gists_url":"https://api.github.com/users/drvinceknight/gists{/gist_id}","starred_url":"https://api.github.com/users/drvinceknight/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drvinceknight/subscriptions","organizations_url":"https://api.github.com/users/drvinceknight/orgs","repos_url":"https://api.github.com/users/drvinceknight/repos","events_url":"https://api.github.com/users/drvinceknight/events{/privacy}","received_events_url":"https://api.github.com/users/drvinceknight/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"marcharper","id":203345,"avatar_url":"https://avatars.githubusercontent.com/u/203345?v=3","gravatar_id":"","url":"https://api.github.com/users/marcharper","html_url":"https://github.com/marcharper","followers_url":"https://api.github.com/users/marcharper/followers","following_url":"https://api.github.com/users/marcharper/following{/other_user}","gists_url":"https://api.github.com/users/marcharper/gists{/gist_id}","starred_url":"https://api.github.com/users/marcharper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcharper/subscriptions","organizations_url":"https://api.github.com/users/marcharper/orgs","repos_url":"https://api.github.com/users/marcharper/repos","events_url":"https://api.github.com/users/marcharper/events{/privacy}","received_events_url":"https://api.github.com/users/marcharper/received_events","type":"User","site_admin":false},"milestone":null,"comments":50,"created_at":"2016-04-05T18:11:37Z","updated_at":"2016-05-01T16:18:43Z","closed_at":null,"body":"When I update the tournament results I get a crash:\r\n\r\n```\r\ntournament(gh-pages) ✗: ./update_results.sh 0 20\r\n-- Regenerating noisy tournament figures\r\n/usr/local/lib/python3.4/dist-packages/matplotlib/backends/backend_gtk3agg.py:18: UserWarning: The Gtk3Agg backend is known to not work on Python 3.x with pycairo. Try installing cairocffi.\r\n  \"The Gtk3Agg backend is known to not work on Python 3.x with pycairo. \"\r\nStarting basic_strategies tournament: Turns: 200, Repetitions: 20, Strategies: 8.\r\nFinished all basic_strategies tasks in 8.4s\r\nStarting ordinary_strategies tournament: Turns: 200, Repetitions: 20, Strategies: 113.\r\nFinished all ordinary_strategies tasks in 1370.4s\r\nStarting strategies tournament: Turns: 200, Repetitions: 20, Strategies: 123.\r\nTraceback (most recent call last):\r\n  File \"run_axelrod\", line 132, in <module>\r\n    run_tournaments(**kwargs)\r\n  File \"/home/vince/src/Axelrod/axelrod/utils.py\", line 82, in run_tournaments\r\n    manager.run_tournaments()\r\n  File \"/home/vince/src/Axelrod/axelrod/tournament_manager.py\", line 65, in run_tournaments\r\n    self._run_single_tournament(tournament)\r\n  File \"/home/vince/src/Axelrod/axelrod/tournament_manager.py\", line 96, in _run_single_tournament\r\n    self._generate_output_files(tournament, ecosystem)\r\n  File \"/home/vince/src/Axelrod/axelrod/tournament_manager.py\", line 121, in _generate_output_files\r\n    image_format=self._image_format)\r\n  File \"/home/vince/src/Axelrod/axelrod/tournament_manager.py\", line 143, in _save_plots\r\n    self._save_plot(figure, file_name)\r\n  File \"/home/vince/src/Axelrod/axelrod/tournament_manager.py\", line 164, in _save_plot\r\n    figure.savefig(file_name, bbox_inches='tight', dpi=dpi)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/figure.py\", line 1565, in savefig\r\n    self.canvas.print_figure(*args, **kwargs)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/backend_bases.py\", line 2180, in print_figure\r\n    **kwargs)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/backends/backend_svg.py\", line 1206, in print_svg\r\n    return self._print_svg(filename, svgwriter, fh_to_close, **kwargs)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/backends/backend_svg.py\", line 1234, in _print_svg\r\n    self.figure.draw(renderer)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/artist.py\", line 61, in draw_wrapper\r\n    draw(artist, renderer, *args, **kwargs)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/figure.py\", line 1159, in draw\r\n    func(*args)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/artist.py\", line 61, in draw_wrapper\r\n    draw(artist, renderer, *args, **kwargs)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/axes/_base.py\", line 2324, in draw\r\n    a.draw(renderer)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/artist.py\", line 61, in draw_wrapper\r\n    draw(artist, renderer, *args, **kwargs)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/image.py\", line 394, in draw\r\n    renderer.draw_image(gc, l, b, im)\r\n  File \"/usr/local/lib/python3.4/dist-packages/matplotlib/backends/backend_svg.py\", line 837, in draw_image\r\n    _png.write_png(np.array(im)[::-1], bytesio)\r\nMemoryError\r\n```\r\n\r\nThis is running on ubuntu 14.04 with 16GB of Ram (the same machine I've used previously). You'll see there that I'm only actually running 20 repetitions (I had similar crashes for 100 and 50 reps).\r\n\r\nI'm about to open a PR that makes improvements anyway to memory (the crash happens for me on `master` as well as on that branch although that branch shows a much lower memory consumption) but it would be good to check if this is just my machine: @marcharper could you run the results again (first on `master`?)."},"comment":{"url":"https://api.github.com/repos/Axelrod-Python/Axelrod/issues/comments/216053021","html_url":"https://github.com/Axelrod-Python/Axelrod/issues/520#issuecomment-216053021","issue_url":"https://api.github.com/repos/Axelrod-Python/Axelrod/issues/520","id":216053021,"user":{"login":"marcharper","id":203345,"avatar_url":"https://avatars.githubusercontent.com/u/203345?v=3","gravatar_id":"","url":"https://api.github.com/users/marcharper","html_url":"https://github.com/marcharper","followers_url":"https://api.github.com/users/marcharper/followers","following_url":"https://api.github.com/users/marcharper/following{/other_user}","gists_url":"https://api.github.com/users/marcharper/gists{/gist_id}","starred_url":"https://api.github.com/users/marcharper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcharper/subscriptions","organizations_url":"https://api.github.com/users/marcharper/orgs","repos_url":"https://api.github.com/users/marcharper/repos","events_url":"https://api.github.com/users/marcharper/events{/privacy}","received_events_url":"https://api.github.com/users/marcharper/received_events","type":"User","site_admin":false},"created_at":"2016-05-01T16:18:43Z","updated_at":"2016-05-01T16:18:43Z","body":"Yep, that's what I'm planning for tournaments and the \"MatchPlayer\" class that handles the parallel aspects.\r\n\r\nI actually think a class is better for match generating now, and I'm on board with your suggestions for progress bars and potentially other methods. If we decide to move to a function instead of a class we can do it in a later PR. What I've done on the parallel branch is ultimately not worth the complexity and loss of readability IMO.\r\n\r\nWe can write to disk using tempfiles so that the user won't notice any extraneous CSV files. I think in the case that the match generator doesn't report a length (or can't) that disk by default is the way to go. But we can work out the details on that once the other changes are in."}}