{"action":"created","issue":{"url":"https://api.github.com/repos/akeneo/pim-community-dev/issues/4416","repository_url":"https://api.github.com/repos/akeneo/pim-community-dev","labels_url":"https://api.github.com/repos/akeneo/pim-community-dev/issues/4416/labels{/name}","comments_url":"https://api.github.com/repos/akeneo/pim-community-dev/issues/4416/comments","events_url":"https://api.github.com/repos/akeneo/pim-community-dev/issues/4416/events","html_url":"https://github.com/akeneo/pim-community-dev/issues/4416","id":152970408,"number":4416,"title":"Memory leaks in 1.5 and ways to reduce memory usage","user":{"login":"NovikovViktor","id":8058373,"avatar_url":"https://avatars.githubusercontent.com/u/8058373?v=3","gravatar_id":"","url":"https://api.github.com/users/NovikovViktor","html_url":"https://github.com/NovikovViktor","followers_url":"https://api.github.com/users/NovikovViktor/followers","following_url":"https://api.github.com/users/NovikovViktor/following{/other_user}","gists_url":"https://api.github.com/users/NovikovViktor/gists{/gist_id}","starred_url":"https://api.github.com/users/NovikovViktor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NovikovViktor/subscriptions","organizations_url":"https://api.github.com/users/NovikovViktor/orgs","repos_url":"https://api.github.com/users/NovikovViktor/repos","events_url":"https://api.github.com/users/NovikovViktor/events{/privacy}","received_events_url":"https://api.github.com/users/NovikovViktor/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/akeneo/pim-community-dev/labels/performance","name":"performance","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-05-04T09:37:24Z","updated_at":"2016-05-09T09:36:17Z","closed_at":null,"body":"Hello everyone, I have faced with performance and memory leaks issues while importing file with about 70k of products.\r\n\r\nSo during short investigation I found that following option might speed up Akeneo import:\r\n1) Use `gc_collect_cycles` in `ProductSaver -> saveAll()`. After saving each batch of products (100) about 4k of refs were cleared.\r\n2) Remove some data from Doctrine identity map. Actually it looks like Doctrine clear it somehow by itself, but when I added this `$this->objectManager->clear('Pim\\Component\\Catalog\\Model\\Product');` to \r\n`ProductSaver -> saveAll()` it saved about 0.25 Mb of memory.\r\n3) And this one is the most interesting: method findOrCreateProduct(), searches through `product_value` table for `product identifier`, and then searches for appropriate product id in `product` table. This is quite heavy operation. So, to fix this we created `identifier__product_id _map` table, indexed by `identifier`. It holds `identifiers` and `product_id's`, so getting  `product id` became faster.\r\n\r\nIn general if to compare initial memory value which was on start - about 88 MB, and after all improvements I got 69 MB, and leak per 100 product became 0.5 MB instead of ~8 MB.\r\n\r\nWhat do you think regarding this and memory leaks in Akeneo in general?\r\n\r\n(Akeneo 1.5)\r\n\r\nRegards, Viktor.\r\n"},"comment":{"url":"https://api.github.com/repos/akeneo/pim-community-dev/issues/comments/217819220","html_url":"https://github.com/akeneo/pim-community-dev/issues/4416#issuecomment-217819220","issue_url":"https://api.github.com/repos/akeneo/pim-community-dev/issues/4416","id":217819220,"user":{"login":"ClementGautier","id":660091,"avatar_url":"https://avatars.githubusercontent.com/u/660091?v=3","gravatar_id":"","url":"https://api.github.com/users/ClementGautier","html_url":"https://github.com/ClementGautier","followers_url":"https://api.github.com/users/ClementGautier/followers","following_url":"https://api.github.com/users/ClementGautier/following{/other_user}","gists_url":"https://api.github.com/users/ClementGautier/gists{/gist_id}","starred_url":"https://api.github.com/users/ClementGautier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClementGautier/subscriptions","organizations_url":"https://api.github.com/users/ClementGautier/orgs","repos_url":"https://api.github.com/users/ClementGautier/repos","events_url":"https://api.github.com/users/ClementGautier/events{/privacy}","received_events_url":"https://api.github.com/users/ClementGautier/received_events","type":"User","site_admin":false},"created_at":"2016-05-09T09:36:17Z","updated_at":"2016-05-09T09:36:17Z","body":"Hello @NovikovViktor,\r\n\r\nFirst, thank you for your feedback :)\r\n\r\nI was wondering: What is your PHP version during your bench and if you used prod environment and/or if you optimized your composer auto-load during this benchmarks ?\r\n\r\n1) Actually I'm not sure that this is required as gc_collect_cycles will be called when the number of objects reach the gc limit. Doing it here could at most free the gc more often but not reduce the leak.\r\n\r\n2) This is interesting but I would rather do it in the import/export processes instead of the saver because it could slow down other parts of the application. Basically if you do it in the saver and need these objects after saving, doctrine will fetch them again from the database in order to hydrate new object for your needs.\r\n\r\n3) I'm not a big fan of this solution as it will change the model layer in order to bypass a storage issue. Basically in mongodb the fix will not apply. Anyway, **you are right** when you say that fetching the identifier attribute every time from the database is a waste of time and should be avoided. We could simply do a join, as this columns are indexed and save object hydration. If the identifier do not change we could even cache this attribute and save a lot of queries. I would even recommend a persistant cache for that (that we could warmup with the application cache) if it is not possible to change it from the UI.\r\n\r\nI'll ask internally at Akeneo about caching on the third point. Be pleased to contribute on this topics by creating pull requests."}}