{"push_id":2464677730,"size":7,"distinct_size":6,"ref":"refs/heads/processes-cache","head":"8fcedb65544e8ebeece8cbf9422d9769ab8b0e07","before":"173251b8b3baa342bf9b22970660daeeda1d7e61","commits":[{"sha":"1047d338e57b2e471fc5732d471a2c804338147f","author":{"name":"Jonathan Keljo","email":"1df79244be6abc71effcee5b7f583065fdc70804@users.noreply.github.com"},"message":"Fix an old regression in processes table on macOS\n\nI went looking for ways to make `processes` faster, and found what turns out to be\na regression introduced ~10 months ago.\n\n`processes` used to use a stack-allocated `char[]` for getting process command lines. A while\nback it was changed to a `vector<char>`. The `vector` constructor that takes a size initializes\nevery element in the array -- in our case, 262,144 (256K) of them. Doing that for every single\nprocess got... expensive.\n\nThis commit switches back to a stack-allocated array. Since the switch to `vector` was in an\nattempt to quiet some warnings about using the non-C++-standard variable-length-arrays feature,\nit does so with `alloca` instead.\n\nBefore:\n```\n~/osquery % sudo tools/analysis/profile.py --config test.json\nProfiling query: select total_size from processes;\n/Users/jkeljo/osquery/tools/analysis/../tests/utils.py:147: FutureWarning: memory_info_ex() is deprecated and will be removed; use memory_info() instead\n  \"memory\": p.memory_info_ex(),\n D:0  C:1  M:1  F:0  U:2  pack_processes (1/1): duration: 0.520297050476 cpu_time: 0.413083054 memory: 12574720 fds: 4 utilization: 40.75\n\n~/osquery % sudo tools/analysis/profile.py --config test.json --count 50\nProfiling query: select total_size from processes;\n/Users/jkeljo/osquery/tools/analysis/../tests/utils.py:147: FutureWarning: memory_info_ex() is deprecated and will be removed; use memory_info() instead\n  \"memory\": p.memory_info_ex(),\n D:3  C:3  M:2  F:0  U:3  pack_processes (1/1): duration: 18.6152729988 cpu_time: 18.24202144 memory: 15376384 fds: 4 utilization: 95.5263157895\n```\n\nAfter:\n```\n~/osquery % sudo tools/analysis/profile.py --config test.json\nProfiling query: select total_size from processes;\n/Users/jkeljo/osquery/tools/analysis/../tests/utils.py:147: FutureWarning: memory_info_ex() is deprecated and will be removed; use memory_info() instead\n  \"memory\": p.memory_info_ex(),\n D:0  C:0  M:1  F:0  U:0  pack_processes (1/1): duration: 0.588259935379 cpu_time: 0.07418382 memory: 12398592 fds: 4 utilization: 7.1\n\n~/osquery % sudo tools/analysis/profile.py --config test.json --count 50\nProfiling query: select total_size from processes;\n/Users/jkeljo/osquery/tools/analysis/../tests/utils.py:147: FutureWarning: memory_info_ex() is deprecated and will be removed; use memory_info() instead\n  \"memory\": p.memory_info_ex(),\n D:2  C:2  M:2  F:0  U:3  pack_processes (1/1): duration: 1.5203359127 cpu_time: 1.415935936 memory: 16990208 fds: 4 utilization: 70.475\n```","distinct":false,"url":"https://api.github.com/repos/jkeljo/osquery/commits/1047d338e57b2e471fc5732d471a2c804338147f"},{"sha":"6ac7a1e4a8fd53a6d17ba2db573c1ba52d94d13b","author":{"name":"Jonathan Keljo","email":"1df79244be6abc71effcee5b7f583065fdc70804@users.noreply.github.com"},"message":"Prefactor: Factor out helper functions for populating different parts of the process row\n\nNo behavior changes.","distinct":true,"url":"https://api.github.com/repos/jkeljo/osquery/commits/6ac7a1e4a8fd53a6d17ba2db573c1ba52d94d13b"},{"sha":"0638d2c9a48868fa2d85286c7d557e124e9f90af","author":{"name":"Jonathan Keljo","email":"1df79244be6abc71effcee5b7f583065fdc70804@users.noreply.github.com"},"message":"Prefactor: Combine proc cmdline and name/path/on_disk getters\n\nNo behavior changes.","distinct":true,"url":"https://api.github.com/repos/jkeljo/osquery/commits/0638d2c9a48868fa2d85286c7d557e124e9f90af"},{"sha":"cb7df8ab4ef5d0d77f88cf97edf1bbd22dac1d1e","author":{"name":"Jonathan Keljo","email":"1df79244be6abc71effcee5b7f583065fdc70804@users.noreply.github.com"},"message":"Cache things that don't change\n\nNo eviction yet.","distinct":true,"url":"https://api.github.com/repos/jkeljo/osquery/commits/cb7df8ab4ef5d0d77f88cf97edf1bbd22dac1d1e"},{"sha":"66409201e9de4634930ffe1ff7e971aa52885571","author":{"name":"Jonathan Keljo","email":"1df79244be6abc71effcee5b7f583065fdc70804@users.noreply.github.com"},"message":"Cache eviction: dead processes\n\nClean up entries for processes that are no longer in the list","distinct":true,"url":"https://api.github.com/repos/jkeljo/osquery/commits/66409201e9de4634930ffe1ff7e971aa52885571"},{"sha":"784dbefefc733e8bcf7fedb091a19d19f227b827","author":{"name":"Jonathan Keljo","email":"1df79244be6abc71effcee5b7f583065fdc70804@users.noreply.github.com"},"message":"Cache eviction: pid reuse\n\nUse process start time to detect pid reuse so that cache entries don't get stale.\n\nThis commit completes the feature. Performance testing results:\n\nBefore this PR:\n\n```\n~/osquery % sudo tools/analysis/profile.py --config test.json\nProfiling query: select total_size from processes;\n/Users/jkeljo/osquery/tools/analysis/../tests/utils.py:147: FutureWarning: memory_info_ex() is deprecated and will be removed; use memory_info() instead\n  \"memory\": p.memory_info_ex(),\n D:0  C:0  M:1  F:0  U:0  pack_processes (1/1): duration: 0.588259935379 cpu_time: 0.07418382 memory: 12398592 fds: 4 utilization: 7.1\n\n~/osquery % sudo tools/analysis/profile.py --config test.json --count 50\nProfiling query: select total_size from processes;\n/Users/jkeljo/osquery/tools/analysis/../tests/utils.py:147: FutureWarning: memory_info_ex() is deprecated and will be removed; use memory_info() instead\n  \"memory\": p.memory_info_ex(),\n D:2  C:2  M:2  F:0  U:3  pack_processes (1/1): duration: 1.5203359127 cpu_time: 1.415935936 memory: 16990208 fds: 4 utilization: 70.475\n```\n\nAfter:\n\n```\n~/osquery % sudo tools/analysis/profile.py --config test.json\nProfiling query: select total_size from processes;\n/Users/jkeljo/osquery/tools/analysis/../tests/utils.py:147: FutureWarning: memory_info_ex() is deprecated and will be removed; use memory_info() instead\n  \"memory\": p.memory_info_ex(),\n D:0  C:0  M:1  F:0  U:0  pack_processes (1/1): duration: 0.559644937515 cpu_time: 0.075561436 memory: 12566528 fds: 4 utilization: 7.2\n\n~/osquery % sudo tools/analysis/profile.py --config test.json --count 50\nProfiling query: select total_size from processes;\n/Users/jkeljo/osquery/tools/analysis/../tests/utils.py:147: FutureWarning: memory_info_ex() is deprecated and will be removed; use memory_info() instead\n  \"memory\": p.memory_info_ex(),\n D:2  C:1  M:2  F:0  U:2  pack_processes (1/1): duration: 1.01243519783 cpu_time: 0.736211552 memory: 18972672 fds: 4 utilization: 48.9333333333\n\n```","distinct":true,"url":"https://api.github.com/repos/jkeljo/osquery/commits/784dbefefc733e8bcf7fedb091a19d19f227b827"},{"sha":"8fcedb65544e8ebeece8cbf9422d9769ab8b0e07","author":{"name":"Jonathan Keljo","email":"1df79244be6abc71effcee5b7f583065fdc70804@users.noreply.github.com"},"message":"Fix a bug in cache eviction\n\nI was using the processes list for eviction even when the list came\nfrom a WHERE clause. That can result in a lot of false evictions if\nthere are queries with different WHERE clauses running.","distinct":true,"url":"https://api.github.com/repos/jkeljo/osquery/commits/8fcedb65544e8ebeece8cbf9422d9769ab8b0e07"}]}
{"actor":{"display_login":"jkeljo"}}