{"action":"closed","issue":{"url":"https://api.github.com/repos/JPaulMora/Pyrit/issues/418","labels_url":"https://api.github.com/repos/JPaulMora/Pyrit/issues/418/labels{/name}","comments_url":"https://api.github.com/repos/JPaulMora/Pyrit/issues/418/comments","events_url":"https://api.github.com/repos/JPaulMora/Pyrit/issues/418/events","html_url":"https://github.com/JPaulMora/Pyrit/issues/418","id":108036725,"number":418,"title":"import_passwords fails on large file with MySQL database","user":{"login":"GoogleCodeExporter","id":9614759,"avatar_url":"https://avatars.githubusercontent.com/u/9614759?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCodeExporter","html_url":"https://github.com/GoogleCodeExporter","followers_url":"https://api.github.com/users/GoogleCodeExporter/followers","following_url":"https://api.github.com/users/GoogleCodeExporter/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCodeExporter/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCodeExporter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCodeExporter/subscriptions","organizations_url":"https://api.github.com/users/GoogleCodeExporter/orgs","repos_url":"https://api.github.com/users/GoogleCodeExporter/repos","events_url":"https://api.github.com/users/GoogleCodeExporter/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCodeExporter/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/JPaulMora/Pyrit/labels/duplicate","name":"duplicate","color":"cccccc"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-24T01:06:21Z","updated_at":"2016-01-10T20:54:17Z","closed_at":"2016-01-10T20:54:17Z","body":"```\nHi there!\n\nI'm attempting to import a large (1.3GB) text file full of passwords into \nPyrit's MySQL database and it has failed repeatedly.\n\nThe file in question is very high-quality (UNIQPASS v11, \nhttps://dazzlepod.com/uniqpass/) so I'm confident that it's not causing the \nissue.  I'm using a MySQL database as my default storage as configured with \n~/.pyrit/config.\n\nMy system is a quad-core AMD CPU running 64-bit Ubuntu 12.10.\n\nWhen I run <pyrit -i UNIQPASS.txt import_passwords> it runs for some variable \nlength of time before crashing.  (Same issue is present in a detached screen \nsession as in a remote shell.)\n\nStrange thing is, it produces a different error and crashes after a different \nnumber of lines each time I run it (asterisks are mine for readability):\n\n****************************************************************************\nPyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com\nThis code is distributed under the GNU General Public License v3+\n\nConnecting to storage at 'mysql://localhost/pyrit1'...  connected.\n44700000 lines read (567195.0 lines/s)... Segmentation fault (core dumped)\n*****************************************************************************\nPyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com\nThis code is distributed under the GNU General Public License v3+\n\nConnecting to storage at 'mysql://localhost/pyrit1'...  connected.\n97100000 lines read (114239.9 lines/s)... Traceback (most recent call last):\n  File \"/usr/bin/pyrit\", line 6, in <module>\n    pyrit_cli.Pyrit_CLI().initFromArgv()\n  File \"/usr/lib/pymodules/python2.7/pyrit_cli.py\", line 115, in initFromArgv\n    func(self, **options)\n  File \"/usr/lib/pymodules/python2.7/pyrit_cli.py\", line 355, in import_passwords   \n    pwstore.store_password(line)\n  File \"/usr/lib/pymodules/python2.7/cpyrit/storage.py\", line 363, in store_password\n    self._flush_bucket(pw_h1, pw_bucket)\n  File \"/usr/lib/pymodules/python2.7/cpyrit/storage.py\", line 1196, in _flush_bucket\n    bucket.difference_update(db_bucket)\n  File \"/usr/lib/pymodules/python2.7/cpyrit/storage.py\", line 926, in __iter__\n    self.collection.unpack(self.collection_buffer)\n  File \"/usr/lib/pymodules/python2.7/cpyrit/storage.py\", line 287, in unpack\n    raise DigestError(\"Digest check failed\")\ncpyrit.storage.DigestError: Digest check failed\n****************************************************************************\nConnecting to storage at 'mysql://localhost/pyrit1'...  connected.\n45000000 lines read (459279.8 lines/s)... Traceback (most recent call last):\n  File \"/usr/bin/pyrit\", line 6, in <module>\n    pyrit_cli.Pyrit_CLI().initFromArgv()\n  File \"/usr/lib/pymodules/python2.7/pyrit_cli.py\", line 115, in initFromArgv\n    func(self, **options)\n  File \"/usr/lib/pymodules/python2.7/pyrit_cli.py\", line 355, in import_passwords   \n    pwstore.store_password(line)\n  File \"/usr/lib/pymodules/python2.7/cpyrit/storage.py\", line 363, in store_password\n    self._flush_bucket(pw_h1, pw_bucket)\n  File \"/usr/lib/pymodules/python2.7/cpyrit/storage.py\", line 1199, in _flush_bucket\n    session.add(PAW2_DBObject(pw_h1, bucket))\n  File \"<string>\", line 4, in __init__\n  File \"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/state.py\", line 98, in initialize_instance\n    return manager.original_init(*mixed[1:], **kwargs)\n  File \"/usr/lib/pymodules/python2.7/cpyrit/storage.py\", line 917, in __init__\n    key, collection_buffer = PAW2_Buffer(self.collection).pack()\n  File \"/usr/lib/pymodules/python2.7/cpyrit/storage.py\", line 276, in pack\n    b = zlib.compress('\\n'.join(self.collection), 1)\nTypeError: sequence item 9425: expected string,  found\n****************************************************************************\n\nConsidering that Pyrit is supposed to be a robust and scalable tool that can \nmanage massive databases, I hope this is just me! Please let me know if there \nis anything I can do to help troubleshoot or if I'm missing something.\n\nThanks for making a great tool.\nAlex\n\n```\n\nOriginal issue reported on code.google.com by `iAlex.R...@gmail.com` on 14 Feb 2013 at 7:04"}}