{"action":"opened","issue":{"url":"https://api.github.com/repos/clearlydefined/crawler/issues/153","repository_url":"https://api.github.com/repos/clearlydefined/crawler","labels_url":"https://api.github.com/repos/clearlydefined/crawler/issues/153/labels{/name}","comments_url":"https://api.github.com/repos/clearlydefined/crawler/issues/153/comments","events_url":"https://api.github.com/repos/clearlydefined/crawler/issues/153/events","html_url":"https://github.com/clearlydefined/crawler/issues/153","id":356215599,"node_id":"MDU6SXNzdWUzNTYyMTU1OTk=","number":153,"title":"Update PyPi support","user":{"login":"jeffmcaffer","id":10070956,"node_id":"MDQ6VXNlcjEwMDcwOTU2","avatar_url":"https://avatars2.githubusercontent.com/u/10070956?v=4","gravatar_id":"","url":"https://api.github.com/users/jeffmcaffer","html_url":"https://github.com/jeffmcaffer","followers_url":"https://api.github.com/users/jeffmcaffer/followers","following_url":"https://api.github.com/users/jeffmcaffer/following{/other_user}","gists_url":"https://api.github.com/users/jeffmcaffer/gists{/gist_id}","starred_url":"https://api.github.com/users/jeffmcaffer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeffmcaffer/subscriptions","organizations_url":"https://api.github.com/users/jeffmcaffer/orgs","repos_url":"https://api.github.com/users/jeffmcaffer/repos","events_url":"https://api.github.com/users/jeffmcaffer/events{/privacy}","received_events_url":"https://api.github.com/users/jeffmcaffer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-01T16:35:26Z","updated_at":"2018-09-01T16:35:26Z","closed_at":null,"author_association":"MEMBER","body":"Some comments from a PyPi knowledgeable person on the PyPi integration.\r\n\r\n* I think `_getRevision` should just use the top-level `version` attribute (`info.version`) returned from PyPI rather than trying to determine the latest release from the keys in the releases dictionary. The current implementation assumes a specific ordering that the real-world data does not look to be following consistently. See the entry for `pip` on https://dev.clearlydefined.io/definitions for an example where it is pulling the wrong version.\r\n* It doesn't look like the `source` field is getting populated. There are a few options for where you may be able to pull that URL from. \r\n   * One is the `info.descriptions` entry. Sometimes a link to a host repo will be embedded within there. It's free-form text though, and often there will be other links present as well, making parsing it a tricky proposition.\r\n   * Another would be to use the entry in the top-level `info.urls` list that has its `packagetype` set to `sdist` or `python_version` set to `source`. The `url` value for that entry will be a link to a tarball containing a source distribution of the package.\r\n   * Similarly, you could walk through the list in `info.releases` associated with the `version` determined in the first bullet point and get the `url` value for the entry with `packagetype: sdist` or `python_version: source`, if any.\r\n* For the license attribute, I'd consider just using the top-level `info.license` value rather than parsing it from the classifiers. Alternatively, it could fall back on the licenses parsed from the classifiers if it is missing from `info.license` or maybe be the union of the two. Also, the current parsing logic will only find licenses tagged as \"OSI Approved\" by the package maintainer. That may be too restrictive for what you're trying to accomplish, though I'm not sure. It could be made more inclusive by returning all entries that start with \"License :: \".\r\n"}}
{"actor":{"display_login":"jeffmcaffer"}}