{"action":"created","issue":{"url":"https://api.github.com/repos/turi-code/SFrame/issues/34","repository_url":"https://api.github.com/repos/turi-code/SFrame","labels_url":"https://api.github.com/repos/turi-code/SFrame/issues/34/labels{/name}","comments_url":"https://api.github.com/repos/turi-code/SFrame/issues/34/comments","events_url":"https://api.github.com/repos/turi-code/SFrame/issues/34/events","html_url":"https://github.com/turi-code/SFrame/issues/34","id":110541391,"number":34,"title":"Memory Size Detection","user":{"login":"ylow","id":2740522,"avatar_url":"https://avatars.githubusercontent.com/u/2740522?v=3","gravatar_id":"","url":"https://api.github.com/users/ylow","html_url":"https://github.com/ylow","followers_url":"https://api.github.com/users/ylow/followers","following_url":"https://api.github.com/users/ylow/following{/other_user}","gists_url":"https://api.github.com/users/ylow/gists{/gist_id}","starred_url":"https://api.github.com/users/ylow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ylow/subscriptions","organizations_url":"https://api.github.com/users/ylow/orgs","repos_url":"https://api.github.com/users/ylow/repos","events_url":"https://api.github.com/users/ylow/events{/privacy}","received_events_url":"https://api.github.com/users/ylow/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2015-10-08T21:08:32Z","updated_at":"2016-07-23T11:17:59Z","closed_at":"2015-10-22T20:37:28Z","body":"Currently, we try to autotune the amount of memory we use by detecting the amount of system memory. However, this is problematic when we are run inside of docker since we detect the total amount of memory on the system (via sysinfo), rather the amount of memory allocated to us. (See http://fabiokung.com/2014/03/13/memory-inside-linux-containers/, https://github.com/docker/docker/issues/12394)\r\n\r\nAs of docker 1.8 which is rather new, we should be able to look into /sys/fs/cgroups or /proc/self/cgroup or something like that. But not everyone will be on docker 1.8, so we need an API workaround as well.\r\n\r\n - Use /sys/fs/cgroups to determine our memory limit and use that\r\n - ?Alternatives?\r\n - Add an sframe.set_memory_limit function that auto-configures the memory limit based on what the user tells us. This is the ultimate fallback."},"comment":{"url":"https://api.github.com/repos/turi-code/SFrame/issues/comments/234713061","html_url":"https://github.com/turi-code/SFrame/issues/34#issuecomment-234713061","issue_url":"https://api.github.com/repos/turi-code/SFrame/issues/34","id":234713061,"user":{"login":"shardool-freightwalla","id":17472264,"avatar_url":"https://avatars.githubusercontent.com/u/17472264?v=3","gravatar_id":"","url":"https://api.github.com/users/shardool-freightwalla","html_url":"https://github.com/shardool-freightwalla","followers_url":"https://api.github.com/users/shardool-freightwalla/followers","following_url":"https://api.github.com/users/shardool-freightwalla/following{/other_user}","gists_url":"https://api.github.com/users/shardool-freightwalla/gists{/gist_id}","starred_url":"https://api.github.com/users/shardool-freightwalla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shardool-freightwalla/subscriptions","organizations_url":"https://api.github.com/users/shardool-freightwalla/orgs","repos_url":"https://api.github.com/users/shardool-freightwalla/repos","events_url":"https://api.github.com/users/shardool-freightwalla/events{/privacy}","received_events_url":"https://api.github.com/users/shardool-freightwalla/received_events","type":"User","site_admin":false},"created_at":"2016-07-23T11:17:59Z","updated_at":"2016-07-23T11:17:59Z","body":"Runtime config if that helps.\r\n\r\n     app[worker.1]: {'GRAPHLAB_FILEIO_ALTERNATIVE_SSL_CERT_DIR': '', 'GRAPHLAB_SFRAME_IO_READ_LOCK': 0, 'GRAPHLAB_FILEIO_MAXIMUM_CACHE_CAPACITY_PER_FILE': 16106591232, 'GRAPHLAB_FORCE_IPC_TO_TCP_FALLBACK': 0, 'GRAPHLAB_SGRAPH_TRIPLE_APPLY_EDGE_BATCH_SIZE': 1024, 'GRAPHLAB_SGRAPH_TRIPLE_APPLY_LOCK_ARRAY_SIZE': 1048576, 'GRAPHLAB_SGRAPH_HILBERT_CURVE_PARALLEL_FOR_NUM_THREADS': 8, 'GRAPHLAB_CACHE_FILE_HDFS_LOCATION': '', 'GRAPHLAB_FILEIO_INSECURE_SSL_CERTIFICATE_CHECKS': 0, 'GRAPHLAB_SFRAME_DEFAULT_NUM_SEGMENTS': 8, 'GRAPHLAB_SGRAPH_DEFAULT_NUM_PARTITIONS': 8, 'GRAPHLAB_SGRAPH_INGRESS_VID_BUFFER_SIZE': 1048576, 'GRAPHLAB_ODBC_BUFFER_SIZE': 3221225472, 'GRAPHLAB_LAMBDA_WORKER_CONNECTION_TIMEOUT': 60.0, 'GRAPHLAB_USE_GL_DATATYPE': 0, 'GRAPHLAB_SFRAME_MAX_LAZY_NODE_SIZE': 10000, 'GRAPHLAB_SFRAME_JOIN_BUFFER_NUM_CELLS': 125832744, 'GRAPHLAB_SFRAME_DEFAULT_BLOCK_SIZE': 65536, 'GRAPHLAB_FILEIO_ALTERNATIVE_SSL_CERT_FILE': '/app/.heroku/python/lib/python2.7/site-packages/certifi/cacert.pem', 'GRAPHLAB_SFRAME_WRITER_MAX_BUFFERED_CELLS': 33554432, 'GRAPHLAB_SFRAME_WRITER_MAX_BUFFERED_CELLS_PER_BLOCK': 262144, 'GRAPHLAB_SFRAME_READ_BATCH_SIZE': 128, 'GRAPHLAB_CACHE_FILE_LOCATIONS': '/var/tmp', 'GRAPHLAB_SFRAME_MAX_BLOCKS_IN_CACHE': 128, 'GRAPHLAB_SFRAME_CSV_PARSER_READ_SIZE': 52428800, 'GRAPHLAB_DEFAULT_NUM_GRAPH_LAMBDA_WORKERS': 16, 'GRAPHLAB_GLOBALS_PYTHON_EXECUTABLE': '/app/.heroku/python/bin/python', 'GRAPHLAB_SFRAME_GROUPBY_BUFFER_NUM_ROWS': 25166548, 'GRAPHLAB_LIBODBC_PREFIX': '', 'GRAPHLAB_DEFAULT_NUM_PYLAMBDA_WORKERS': 16, 'GRAPHLAB_SFRAME_SORT_BUFFER_SIZE': 8053295616, 'GRAPHLAB_SFRAME_SORT_PIVOT_ESTIMATION_SAMPLE_SIZE': 2000000, 'GRAPHLAB_SFRAME_SORT_MAX_SEGMENTS': 768, 'GRAPHLAB_SGRAPH_BATCH_TRIPLE_APPLY_LOCK_ARRAY_SIZE': 1048576, 'GRAPHLAB_ODBC_BUFFER_MAX_ROWS': 2000, 'GRAPHLAB_FILEIO_MAXIMUM_CACHE_CAPACITY': 16106591232, 'GRAPHLAB_SFRAME_FILE_HANDLE_POOL_SIZE': 3072}"}}
{"actor":{"display_login":"shardool-freightwalla"}}