{"action":"closed","number":495,"pull_request":{"url":"https://api.github.com/repos/iovisor/bcc/pulls/495","id":67000347,"html_url":"https://github.com/iovisor/bcc/pull/495","diff_url":"https://github.com/iovisor/bcc/pull/495.diff","patch_url":"https://github.com/iovisor/bcc/pull/495.patch","issue_url":"https://api.github.com/repos/iovisor/bcc/issues/495","number":495,"state":"closed","locked":false,"title":"[RFC] Remove all dependencies on external binaries","user":{"login":"vmg","id":42793,"avatar_url":"https://avatars.githubusercontent.com/u/42793?v=3","gravatar_id":"","url":"https://api.github.com/users/vmg","html_url":"https://github.com/vmg","followers_url":"https://api.github.com/users/vmg/followers","following_url":"https://api.github.com/users/vmg/following{/other_user}","gists_url":"https://api.github.com/users/vmg/gists{/gist_id}","starred_url":"https://api.github.com/users/vmg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vmg/subscriptions","organizations_url":"https://api.github.com/users/vmg/orgs","repos_url":"https://api.github.com/users/vmg/repos","events_url":"https://api.github.com/users/vmg/events{/privacy}","received_events_url":"https://api.github.com/users/vmg/received_events","type":"User","site_admin":true},"body":"In its current implementation, BCC depends on a large set of external binaries for normal operation. Namely:\r\n\r\n- `objdump` for symbol lookup\r\n- `objdump` for load address resolution\r\n- `readelf` for USDT notes parsing\r\n- `ldconfig` for system library resolution\r\n- `file` and `magic` to separate ELF binaries and SOs\r\n- `readlink` to follow symlinks\r\n\r\nThis approach has IMO several shortcomings, amongst them:\r\n\r\n- They add a large layer of runtime dependencies for the library and bindings which are not exactly specified anywhere (and may or may not be assumed to exist on production systems).\r\n- The textual interface to every one of these tools is ill-defined. This makes parsing their output require regexps, which need to be continuously tweaked and which may break between versions of the tools.\r\n- The boilerplate to interact with these tools is cumbersome, and needs to be re-implemented by every author of bindings to `libbcc`.\r\n- They are *slow* to work with, since they are really generic utilities that process much more information than we require (particularly noticeable for `readelf` and `objdump`).\r\n\r\nThis work-in-progress RFC/Pull Request proposes an alternative take on these dependencies: they have all been replaced with native C/C++ code, inside of `libbcc` itself, which makes the APIs readily available for all existing and future binding implementations without having to re-implement any of the parsing boilerplate.\r\n\r\nThe new programmatic APIs have (presumingly) the same behavior as their binary counterparts, with none of the cruft. A new external dependency, `libelf`, has been added to libbcc, but this is a development dependency that can be linked statically and should be readily available in all major distributions.\r\n\r\nWork so far has been focused on removing the ProcessSymbols and KernelSymbols caches in both the Python and Lua wrappers, and unifying them with a native SymbolCache class, accessed through C APIs. The `check_path_symbol` API, which resolves libraries and hook addresses for the probes has also been re-implemented in C, again unifying the code between the Python and Lua wrappers.\r\n\r\nA very advanced and performant API for USDT probe resolution based on libelf has also been written (removing the dependency on `readelf` and the janky parsing), although right now it has not been wired up to either wrapper. Likewise for a `file`/`magic` replacement for shared-object detection.\r\n\r\nThere's still a bit of work to be done (particularly when testing this new implementation), but I'd like to open this PR early to gather feedback and see whether the maintainers deem this approach interesting. I'm hoping that the chance to unify large swaths of code for all `libbcc` wrappers will be as appealing to you as it is to me!\r\n\r\nAs always, all the work here has been broken up in small meaningful comments, so please check the commit messages for specifics. Thanks for your consideration!","created_at":"2016-04-19T11:14:39Z","updated_at":"2016-04-20T15:11:10Z","closed_at":"2016-04-20T15:11:10Z","merged_at":"2016-04-20T15:11:10Z","merge_commit_sha":"d516406d1be4355f009d2bd6ca79c08bf015666d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/iovisor/bcc/pulls/495/commits","review_comments_url":"https://api.github.com/repos/iovisor/bcc/pulls/495/comments","review_comment_url":"https://api.github.com/repos/iovisor/bcc/pulls/comments{/number}","comments_url":"https://api.github.com/repos/iovisor/bcc/issues/495/comments","statuses_url":"https://api.github.com/repos/iovisor/bcc/statuses/2153673aeb15d5ab6189f43c1722a8e8bec6463b","head":{"label":"vmg:vmg/elf-2","ref":"vmg/elf-2","sha":"2153673aeb15d5ab6189f43c1722a8e8bec6463b","user":{"login":"vmg","id":42793,"avatar_url":"https://avatars.githubusercontent.com/u/42793?v=3","gravatar_id":"","url":"https://api.github.com/users/vmg","html_url":"https://github.com/vmg","followers_url":"https://api.github.com/users/vmg/followers","following_url":"https://api.github.com/users/vmg/following{/other_user}","gists_url":"https://api.github.com/users/vmg/gists{/gist_id}","starred_url":"https://api.github.com/users/vmg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vmg/subscriptions","organizations_url":"https://api.github.com/users/vmg/orgs","repos_url":"https://api.github.com/users/vmg/repos","events_url":"https://api.github.com/users/vmg/events{/privacy}","received_events_url":"https://api.github.com/users/vmg/received_events","type":"User","site_admin":true},"repo":{"id":54778024,"name":"bcc","full_name":"vmg/bcc","owner":{"login":"vmg","id":42793,"avatar_url":"https://avatars.githubusercontent.com/u/42793?v=3","gravatar_id":"","url":"https://api.github.com/users/vmg","html_url":"https://github.com/vmg","followers_url":"https://api.github.com/users/vmg/followers","following_url":"https://api.github.com/users/vmg/following{/other_user}","gists_url":"https://api.github.com/users/vmg/gists{/gist_id}","starred_url":"https://api.github.com/users/vmg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vmg/subscriptions","organizations_url":"https://api.github.com/users/vmg/orgs","repos_url":"https://api.github.com/users/vmg/repos","events_url":"https://api.github.com/users/vmg/events{/privacy}","received_events_url":"https://api.github.com/users/vmg/received_events","type":"User","site_admin":true},"private":false,"html_url":"https://github.com/vmg/bcc","description":"BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more","fork":true,"url":"https://api.github.com/repos/vmg/bcc","forks_url":"https://api.github.com/repos/vmg/bcc/forks","keys_url":"https://api.github.com/repos/vmg/bcc/keys{/key_id}","collaborators_url":"https://api.github.com/repos/vmg/bcc/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/vmg/bcc/teams","hooks_url":"https://api.github.com/repos/vmg/bcc/hooks","issue_events_url":"https://api.github.com/repos/vmg/bcc/issues/events{/number}","events_url":"https://api.github.com/repos/vmg/bcc/events","assignees_url":"https://api.github.com/repos/vmg/bcc/assignees{/user}","branches_url":"https://api.github.com/repos/vmg/bcc/branches{/branch}","tags_url":"https://api.github.com/repos/vmg/bcc/tags","blobs_url":"https://api.github.com/repos/vmg/bcc/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/vmg/bcc/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/vmg/bcc/git/refs{/sha}","trees_url":"https://api.github.com/repos/vmg/bcc/git/trees{/sha}","statuses_url":"https://api.github.com/repos/vmg/bcc/statuses/{sha}","languages_url":"https://api.github.com/repos/vmg/bcc/languages","stargazers_url":"https://api.github.com/repos/vmg/bcc/stargazers","contributors_url":"https://api.github.com/repos/vmg/bcc/contributors","subscribers_url":"https://api.github.com/repos/vmg/bcc/subscribers","subscription_url":"https://api.github.com/repos/vmg/bcc/subscription","commits_url":"https://api.github.com/repos/vmg/bcc/commits{/sha}","git_commits_url":"https://api.github.com/repos/vmg/bcc/git/commits{/sha}","comments_url":"https://api.github.com/repos/vmg/bcc/comments{/number}","issue_comment_url":"https://api.github.com/repos/vmg/bcc/issues/comments{/number}","contents_url":"https://api.github.com/repos/vmg/bcc/contents/{+path}","compare_url":"https://api.github.com/repos/vmg/bcc/compare/{base}...{head}","merges_url":"https://api.github.com/repos/vmg/bcc/merges","archive_url":"https://api.github.com/repos/vmg/bcc/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/vmg/bcc/downloads","issues_url":"https://api.github.com/repos/vmg/bcc/issues{/number}","pulls_url":"https://api.github.com/repos/vmg/bcc/pulls{/number}","milestones_url":"https://api.github.com/repos/vmg/bcc/milestones{/number}","notifications_url":"https://api.github.com/repos/vmg/bcc/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/vmg/bcc/labels{/name}","releases_url":"https://api.github.com/repos/vmg/bcc/releases{/id}","deployments_url":"https://api.github.com/repos/vmg/bcc/deployments","created_at":"2016-03-26T12:27:44Z","updated_at":"2016-04-01T14:24:02Z","pushed_at":"2016-04-20T11:38:15Z","git_url":"git://github.com/vmg/bcc.git","ssh_url":"git@github.com:vmg/bcc.git","clone_url":"https://github.com/vmg/bcc.git","svn_url":"https://github.com/vmg/bcc","homepage":"","size":2524,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"iovisor:master","ref":"master","sha":"b236092d975071e5a2deb5d0fe8fc1688c718b16","user":{"login":"iovisor","id":12766186,"avatar_url":"https://avatars.githubusercontent.com/u/12766186?v=3","gravatar_id":"","url":"https://api.github.com/users/iovisor","html_url":"https://github.com/iovisor","followers_url":"https://api.github.com/users/iovisor/followers","following_url":"https://api.github.com/users/iovisor/following{/other_user}","gists_url":"https://api.github.com/users/iovisor/gists{/gist_id}","starred_url":"https://api.github.com/users/iovisor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iovisor/subscriptions","organizations_url":"https://api.github.com/users/iovisor/orgs","repos_url":"https://api.github.com/users/iovisor/repos","events_url":"https://api.github.com/users/iovisor/events{/privacy}","received_events_url":"https://api.github.com/users/iovisor/received_events","type":"Organization","site_admin":false},"repo":{"id":34921116,"name":"bcc","full_name":"iovisor/bcc","owner":{"login":"iovisor","id":12766186,"avatar_url":"https://avatars.githubusercontent.com/u/12766186?v=3","gravatar_id":"","url":"https://api.github.com/users/iovisor","html_url":"https://github.com/iovisor","followers_url":"https://api.github.com/users/iovisor/followers","following_url":"https://api.github.com/users/iovisor/following{/other_user}","gists_url":"https://api.github.com/users/iovisor/gists{/gist_id}","starred_url":"https://api.github.com/users/iovisor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iovisor/subscriptions","organizations_url":"https://api.github.com/users/iovisor/orgs","repos_url":"https://api.github.com/users/iovisor/repos","events_url":"https://api.github.com/users/iovisor/events{/privacy}","received_events_url":"https://api.github.com/users/iovisor/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/iovisor/bcc","description":"BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more","fork":false,"url":"https://api.github.com/repos/iovisor/bcc","forks_url":"https://api.github.com/repos/iovisor/bcc/forks","keys_url":"https://api.github.com/repos/iovisor/bcc/keys{/key_id}","collaborators_url":"https://api.github.com/repos/iovisor/bcc/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/iovisor/bcc/teams","hooks_url":"https://api.github.com/repos/iovisor/bcc/hooks","issue_events_url":"https://api.github.com/repos/iovisor/bcc/issues/events{/number}","events_url":"https://api.github.com/repos/iovisor/bcc/events","assignees_url":"https://api.github.com/repos/iovisor/bcc/assignees{/user}","branches_url":"https://api.github.com/repos/iovisor/bcc/branches{/branch}","tags_url":"https://api.github.com/repos/iovisor/bcc/tags","blobs_url":"https://api.github.com/repos/iovisor/bcc/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/iovisor/bcc/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/iovisor/bcc/git/refs{/sha}","trees_url":"https://api.github.com/repos/iovisor/bcc/git/trees{/sha}","statuses_url":"https://api.github.com/repos/iovisor/bcc/statuses/{sha}","languages_url":"https://api.github.com/repos/iovisor/bcc/languages","stargazers_url":"https://api.github.com/repos/iovisor/bcc/stargazers","contributors_url":"https://api.github.com/repos/iovisor/bcc/contributors","subscribers_url":"https://api.github.com/repos/iovisor/bcc/subscribers","subscription_url":"https://api.github.com/repos/iovisor/bcc/subscription","commits_url":"https://api.github.com/repos/iovisor/bcc/commits{/sha}","git_commits_url":"https://api.github.com/repos/iovisor/bcc/git/commits{/sha}","comments_url":"https://api.github.com/repos/iovisor/bcc/comments{/number}","issue_comment_url":"https://api.github.com/repos/iovisor/bcc/issues/comments{/number}","contents_url":"https://api.github.com/repos/iovisor/bcc/contents/{+path}","compare_url":"https://api.github.com/repos/iovisor/bcc/compare/{base}...{head}","merges_url":"https://api.github.com/repos/iovisor/bcc/merges","archive_url":"https://api.github.com/repos/iovisor/bcc/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/iovisor/bcc/downloads","issues_url":"https://api.github.com/repos/iovisor/bcc/issues{/number}","pulls_url":"https://api.github.com/repos/iovisor/bcc/pulls{/number}","milestones_url":"https://api.github.com/repos/iovisor/bcc/milestones{/number}","notifications_url":"https://api.github.com/repos/iovisor/bcc/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/iovisor/bcc/labels{/name}","releases_url":"https://api.github.com/repos/iovisor/bcc/releases{/id}","deployments_url":"https://api.github.com/repos/iovisor/bcc/deployments","created_at":"2015-05-01T19:52:32Z","updated_at":"2016-04-20T04:22:06Z","pushed_at":"2016-04-20T15:11:10Z","git_url":"git://github.com/iovisor/bcc.git","ssh_url":"git@github.com:iovisor/bcc.git","clone_url":"https://github.com/iovisor/bcc.git","svn_url":"https://github.com/iovisor/bcc","homepage":"","size":3353,"stargazers_count":991,"watchers_count":991,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":113,"mirror_url":null,"open_issues_count":57,"forks":113,"open_issues":57,"watchers":991,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/iovisor/bcc/pulls/495"},"html":{"href":"https://github.com/iovisor/bcc/pull/495"},"issue":{"href":"https://api.github.com/repos/iovisor/bcc/issues/495"},"comments":{"href":"https://api.github.com/repos/iovisor/bcc/issues/495/comments"},"review_comments":{"href":"https://api.github.com/repos/iovisor/bcc/pulls/495/comments"},"review_comment":{"href":"https://api.github.com/repos/iovisor/bcc/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/iovisor/bcc/pulls/495/commits"},"statuses":{"href":"https://api.github.com/repos/iovisor/bcc/statuses/2153673aeb15d5ab6189f43c1722a8e8bec6463b"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"4ast","id":12205493,"avatar_url":"https://avatars.githubusercontent.com/u/12205493?v=3","gravatar_id":"","url":"https://api.github.com/users/4ast","html_url":"https://github.com/4ast","followers_url":"https://api.github.com/users/4ast/followers","following_url":"https://api.github.com/users/4ast/following{/other_user}","gists_url":"https://api.github.com/users/4ast/gists{/gist_id}","starred_url":"https://api.github.com/users/4ast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/4ast/subscriptions","organizations_url":"https://api.github.com/users/4ast/orgs","repos_url":"https://api.github.com/users/4ast/repos","events_url":"https://api.github.com/users/4ast/events{/privacy}","received_events_url":"https://api.github.com/users/4ast/received_events","type":"User","site_admin":false},"comments":9,"review_comments":16,"commits":18,"additions":1609,"deletions":559,"changed_files":29}}