{"action":"closed","issue":{"url":"https://api.github.com/repos/ratt-ru/CubiCal/issues/147","repository_url":"https://api.github.com/repos/ratt-ru/CubiCal","labels_url":"https://api.github.com/repos/ratt-ru/CubiCal/issues/147/labels{/name}","comments_url":"https://api.github.com/repos/ratt-ru/CubiCal/issues/147/comments","events_url":"https://api.github.com/repos/ratt-ru/CubiCal/issues/147/events","html_url":"https://github.com/ratt-ru/CubiCal/issues/147","id":313864642,"number":147,"title":"doubled memory use in Montblanc mode","user":{"login":"o-smirnov","id":6470079,"avatar_url":"https://avatars2.githubusercontent.com/u/6470079?v=4","gravatar_id":"","url":"https://api.github.com/users/o-smirnov","html_url":"https://github.com/o-smirnov","followers_url":"https://api.github.com/users/o-smirnov/followers","following_url":"https://api.github.com/users/o-smirnov/following{/other_user}","gists_url":"https://api.github.com/users/o-smirnov/gists{/gist_id}","starred_url":"https://api.github.com/users/o-smirnov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/o-smirnov/subscriptions","organizations_url":"https://api.github.com/users/o-smirnov/orgs","repos_url":"https://api.github.com/users/o-smirnov/repos","events_url":"https://api.github.com/users/o-smirnov/events{/privacy}","received_events_url":"https://api.github.com/users/o-smirnov/received_events","type":"User","site_admin":false},"labels":[{"id":345725946,"url":"https://api.github.com/repos/ratt-ru/CubiCal/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"closed","locked":false,"assignee":{"login":"sjperkins","id":3530212,"avatar_url":"https://avatars3.githubusercontent.com/u/3530212?v=4","gravatar_id":"","url":"https://api.github.com/users/sjperkins","html_url":"https://github.com/sjperkins","followers_url":"https://api.github.com/users/sjperkins/followers","following_url":"https://api.github.com/users/sjperkins/following{/other_user}","gists_url":"https://api.github.com/users/sjperkins/gists{/gist_id}","starred_url":"https://api.github.com/users/sjperkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sjperkins/subscriptions","organizations_url":"https://api.github.com/users/sjperkins/orgs","repos_url":"https://api.github.com/users/sjperkins/repos","events_url":"https://api.github.com/users/sjperkins/events{/privacy}","received_events_url":"https://api.github.com/users/sjperkins/received_events","type":"User","site_admin":false},"assignees":[{"login":"sjperkins","id":3530212,"avatar_url":"https://avatars3.githubusercontent.com/u/3530212?v=4","gravatar_id":"","url":"https://api.github.com/users/sjperkins","html_url":"https://github.com/sjperkins","followers_url":"https://api.github.com/users/sjperkins/followers","following_url":"https://api.github.com/users/sjperkins/following{/other_user}","gists_url":"https://api.github.com/users/sjperkins/gists{/gist_id}","starred_url":"https://api.github.com/users/sjperkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sjperkins/subscriptions","organizations_url":"https://api.github.com/users/sjperkins/orgs","repos_url":"https://api.github.com/users/sjperkins/repos","events_url":"https://api.github.com/users/sjperkins/events{/privacy}","received_events_url":"https://api.github.com/users/sjperkins/received_events","type":"User","site_admin":false}],"milestone":null,"comments":18,"created_at":"2018-04-12T19:51:49Z","updated_at":"2018-04-15T14:30:46Z","closed_at":"2018-04-15T14:30:46Z","author_association":"COLLABORATOR","body":"I'm seeing a lot of memory use when using Montblanc for predict. Compare these two log snippets. In the first case, I specify 10 directions via visibility columns:\r\n\r\n``` - 21:30:25 - data_handler       [0.3/0.3 2.7/2.9 0.2Gb] tile #1/4: reading MS rows 0~273596\r\n - 21:30:26 - data_handler       [2.3/2.3 16.3/16.3 1.3Gb]   reading MODEL_DATA for model 0 direction 0\r\n - 21:30:35 - data_handler       [14.4/14.4 16.8/17.3 12.8Gb] (11, 1, 273597, 64, 4)\r\n - 21:30:37 - data_handler       [15.7/15.9 19.2/19.5 13.4Gb]   17.82% input visibilities flagged and/or deselected\r\n - 21:30:50 - solver             [3.9/15.9 6.4/19.5 13.4Gb] D0T0F0 G: 9 iters, conv 100.00%, d/fl 35.71%, PGE 0.000922; dE: 1 iters, conv 100.00%, d/fl 14.29%, PGE 0.000198 0.000198 0.000198 0.000198 0.000198 0.000198 0.000198 0.000198 0.000198 0.000198 0.000198; G: 1 iters, conv 100.00%, d/fl 35.71%, PGE 0.000922, stall 0.00%, chi^2 922.6 -> 0.1406 (1.338), noise 25.5 -> 2.22\r\n - 21:30:51 - data_handler       [0.5/15.9 3.0/19.5 13.5Gb] tile #1/4: saving MS rows 0~273596\r\n - 21:30:51 - data_handler       [0.5/15.9 3.0/19.5 13.5Gb]   writing CORRECTED_DATA column\r\n - 21:30:52 - data_handler       [1.6/15.9 4.0/19.5 14.5Gb]   no new flags generated\r\n```\r\n\r\n(The numbers in the brackets are, current/max resident, current/max virtual, and shared memory use, in Gb). So you can see the memory footprint jump to ~16GB as data is read in -- and the ~13.5Gb shared memory (NB: I'm not sure if the shared memory is counted as part of resident memory?), where the 10x direction model array lives, is the biggest memory hog. The solver's own RAM impact, on top of that, is surprisingly gentle.\r\n\r\nNow a similar run using Montblanc, with a 10-direction model:\r\n\r\n```\r\n  - 21:33:53 - data_handler       [0.3/0.3 2.8/2.9 0.2Gb] tile #1/4: reading MS rows 0~273596\r\n - 21:34:02 - data_handler       [2.4/2.5 15.3/15.4 1.3Gb]   computing visibilities for 3C147-dE-apparent.lsm.html\r\n2018-04-12 21:34:04.138241: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\r\n - 21:34:34 - data_handler       [27.9/37.3 50.4/59.8 11.7Gb] (10, 1, 273597, 64, 4)\r\n - 21:34:36 - data_handler       [29.3/37.3 52.8/59.8 12.4Gb]   17.82% input visibilities flagged and/or deselected\r\n - 21:35:25 - solver             [17.8/37.3 40.3/59.8 12.4Gb] D0T0F0 G: 8 iters, conv 100.00%, d/fl 35.71%, PGE 0.000624; dE: 20 iters, conv 20.00%, d/fl 14.29%, PGE 1.23e-05 0.00891 0.0138 0.0214 0.0142 0.031 0.0539 0.0682 0.0 0.0; G: 4 iters, conv 100.00%, d/fl 35.71%, PGE 0.000624, stall 32.86%, chi^2 11.15 -> 0.117 (1.154), noise 25.5 -> 2.21\r\n - 21:35:26 - data_handler       [14.7/37.3 37.2/59.8 12.5Gb] tile #1/4: saving MS rows 0~273596\r\n - 21:35:26 - data_handler       [14.7/37.3 37.2/59.8 12.5Gb]   writing CORRECTED_DATA column\r\n - 21:35:27 - data_handler       [15.7/37.3 38.2/59.8 13.4Gb]   no new flags generated\r\n ```\r\n\r\nBoom, a ~13Gb memory hit, which is suspiciously close to the total model visibility size. So it's almost like Montblanc (or the CubiCal interface to it) is hanging on to an extra copy of the per-direction model visibilities. This should not be necessary -- after the predict is done, there should be a way to clear all internals -- so let's find it and burn it.\r\n\r\n(Important, because we're already hitting the RAM buffers with MeerKAT data...)"}}
{"actor":{"display_login":"o-smirnov"}}