{"action":"created","issue":{"url":"https://api.github.com/repos/sabricot/django-elasticsearch-dsl/issues/137","repository_url":"https://api.github.com/repos/sabricot/django-elasticsearch-dsl","labels_url":"https://api.github.com/repos/sabricot/django-elasticsearch-dsl/issues/137/labels{/name}","comments_url":"https://api.github.com/repos/sabricot/django-elasticsearch-dsl/issues/137/comments","events_url":"https://api.github.com/repos/sabricot/django-elasticsearch-dsl/issues/137/events","html_url":"https://github.com/sabricot/django-elasticsearch-dsl/issues/137","id":381624886,"node_id":"MDU6SXNzdWUzODE2MjQ4ODY=","number":137,"title":"Index build memory and speed issue redux","user":{"login":"mjl","id":100283,"node_id":"MDQ6VXNlcjEwMDI4Mw==","avatar_url":"https://avatars1.githubusercontent.com/u/100283?v=4","gravatar_id":"","url":"https://api.github.com/users/mjl","html_url":"https://github.com/mjl","followers_url":"https://api.github.com/users/mjl/followers","following_url":"https://api.github.com/users/mjl/following{/other_user}","gists_url":"https://api.github.com/users/mjl/gists{/gist_id}","starred_url":"https://api.github.com/users/mjl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mjl/subscriptions","organizations_url":"https://api.github.com/users/mjl/orgs","repos_url":"https://api.github.com/users/mjl/repos","events_url":"https://api.github.com/users/mjl/events{/privacy}","received_events_url":"https://api.github.com/users/mjl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-11-16T14:37:03Z","updated_at":"2018-12-19T20:55:24Z","closed_at":null,"author_association":"NONE","body":"I'm trying to trace down that slow indexing problem that is described in issues #118, #130 and #86.\r\n\r\nAs a first data point, I have an application that uses django-haystack that I'm trying to move over to DED. Indexing the same miniature data set takes about twice as long and consumes a load of memory.\r\n\r\nHere is the indexing performance of haystack with a ~50k test data set:\r\n\r\n    $ time python manage.py rebuild_index\r\n    ...\r\n    Indexing 48479 KIT+ plastic\r\n\r\n    real\t1m16.088s\r\n    user\t0m41.056s\r\n    sys\t0m1.186s\r\n\r\nAnd here the absolute same for DED:\r\n\r\n    $ time python manage.py search_index --rebuild\r\n    ...\r\n    Indexing 48479 'Plastic' objects\r\n\r\n    real\t1m57.927s\r\n    user\t1m8.748s\r\n    sys\t0m4.084s\r\n\r\nOkay, about half as fast. Not ideal...\r\n\r\nNow, on my integration machine, I tried to index a real world dataset of about 1.8M records. I aborted the index run after about 15mins, it used up **all memory** and swap, load was hovering at 18, the machine was totally unresponsive.\r\n\r\n    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\r\n    666 myself   20   0 3649612 3.330g     16 D   6.6 89.1   2:43.09 python\r\n\r\ndjango-haystack takes about 18m for this same data set. Okay, we really have a problem.\r\n\r\n\r\nI implemented the same queryset slicing I contributed to haystack (please refer to https://github.com/django-haystack/django-haystack/issues/1077#issuecomment-277677653 for details)\r\nand added some more tweaks to disable Django's queryset caching mechanism.\r\n\r\nAgain, on the same integration machine as above, here are the results of\r\ndifferent improvements.\r\n\r\n[1] Using PagingQuerySet:\r\n\r\nThis is a proxy queryset that uses iterator() which disables the Django\r\nqueryset cache (and uses a server side cursor for fetching data on postgres),\r\nas well as slices the query into multiple parts to work around any caching\r\nin the database driver (this happening is hinted to in the Django docs).\r\n\r\nLoad stays at around 1, memory usage was virtually constant at a reasonable 60M,\r\nCPU usage is always around 75-80%.\r\n\r\n    $ time python manage.py search_index --rebuild -f\r\n    Deleting index 'cc.kitplus-20181116142151'\r\n    Creating index 'cc.kitplus-20181116142151'\r\n    Indexing 1784504 'Plastic' objects\r\n\r\n    real\t17m38.681s\r\n    user\t12m53.976s\r\n    sys\t0m14.652s\r\n\r\n\r\n[2] Using parallel_bulk:\r\n\r\nAs nothing is actually using the return value from bulk(), I replaced it\r\nwith parallel_bulk and returned a dummy value. Errors will be raised as\r\nexceptions anyway, so this does not make a difference.\r\n\r\nThis almost maxes out the CPU at 95-98%, ie. without parallelizing or optimizing\r\nthe prepare stage, this won't go any faster; it uses quite a bit more memory at 80M,\r\nbut still in a reasonable range.\r\n\r\n    $ time python manage.py search_index --rebuild -f\r\n    Deleting index 'cc.kitplus-20181116144037'\r\n    Creating index 'cc.kitplus-20181116144037'\r\n    Indexing 1784504 'Plastic' objects\r\n\r\n    real\t14m57.104s\r\n    user\t13m41.944s\r\n    sys\t0m27.508s\r\n\r\n\r\n[3] Using precomputed prepare w/ bulk:\r\n\r\nInstead of re-discovering the way to prepare each field while indexing,\r\ncompute that first and just re-use those preparation instructions.\r\nThe drawback is that you can no longer feed heterogenous data per model\r\nto the index as they all need to have the same structure for the prepare step,\r\nbut as far as I can tell this can't happen anyway.\r\n\r\nAbout the same performance improvement as using parallel_bulk, albeit with less\r\nmemory used. Exhibits the same memory and CPU behaviour as normal bulk index [1] above.\r\n\r\n    $ time python manage.py search_index --rebuild -f\r\n    Deleting index 'cc.kitplus-20181116145634'\r\n    Creating index 'cc.kitplus-20181116145634'\r\n    Indexing 1784504 'Plastic' objects\r\n\r\n    real\t14m41.170s\r\n    user\t9m59.996s\r\n    sys\t0m13.964s\r\n\r\n\r\n[4] Using precomputed prepare w/ parallel_bulk:\r\n\r\nOkay, all combined.\r\n\r\n    $ time python manage.py search_index --rebuild -f\r\n    Deleting index 'cc.kitplus-20181116151651'\r\n    Creating index 'cc.kitplus-20181116151651'\r\n    Indexing 1784504 'Plastic' objects\r\n\r\n    real\t12m1.882s\r\n    user\t10m48.872s\r\n    sys\t0m26.696s\r\n\r\n\r\nSo from totally impossible to index to 12m (django-haystack takes about 18m).\r\n\r\nI'll clean up my experiments and make them available, please stay tuned.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/sabricot/django-elasticsearch-dsl/issues/comments/448740629","html_url":"https://github.com/sabricot/django-elasticsearch-dsl/issues/137#issuecomment-448740629","issue_url":"https://api.github.com/repos/sabricot/django-elasticsearch-dsl/issues/137","id":448740629,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0ODc0MDYyOQ==","user":{"login":"fkoksal","id":380802,"node_id":"MDQ6VXNlcjM4MDgwMg==","avatar_url":"https://avatars3.githubusercontent.com/u/380802?v=4","gravatar_id":"","url":"https://api.github.com/users/fkoksal","html_url":"https://github.com/fkoksal","followers_url":"https://api.github.com/users/fkoksal/followers","following_url":"https://api.github.com/users/fkoksal/following{/other_user}","gists_url":"https://api.github.com/users/fkoksal/gists{/gist_id}","starred_url":"https://api.github.com/users/fkoksal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fkoksal/subscriptions","organizations_url":"https://api.github.com/users/fkoksal/orgs","repos_url":"https://api.github.com/users/fkoksal/repos","events_url":"https://api.github.com/users/fkoksal/events{/privacy}","received_events_url":"https://api.github.com/users/fkoksal/received_events","type":"User","site_admin":false},"created_at":"2018-12-19T20:55:24Z","updated_at":"2018-12-19T20:55:24Z","author_association":"NONE","body":"Back to the basics: I am now using a simple python script, which;\r\n- takes the input directly from postgresql\r\n- adds index lines required by elasticsearch bulk request\r\n- sends the file to elasticsearch bulk request API.\r\n\r\nFor 3M records, it reduced the indexing time from 15 hours to 6 mins.\r\n\r\nSeems that processing files is way faster than serializing objects."}}
{"actor":{"display_login":"fkoksal"}}