{"action":"created","comment":{"url":"https://api.github.com/repos/python/mypy/pulls/comments/214729508","pull_request_review_id":151843799,"id":214729508,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxNDcyOTUwOA==","diff_hunk":"@@ -2363,6 +2357,34 @@ def find_module_and_diagnose(manager: BuildManager,\n             raise CompileError([\"mypy: can't find module '%s'\" % id])\n \n \n+def exist_added_packages(suppressed: List[str],\n+                        manager: BuildManager, options: Options) -> bool:\n+    \"\"\"Find if there are any newly added packages that were previously suppressed.\n+\n+    Exclude everything not in build for follow-imports=skip.\n+    \"\"\"\n+    for dep in suppressed:\n+        if dep in manager.source_set.source_modules:\n+            # We don't need to add any special logic for this. If a module\n+            # is added to build, importers will be invalidated by normal mechanism.\n+            continue\n+        path = find_module_simple(dep, manager)\n+        if not path:\n+            continue\n+        if (options.follow_imports == 'skip' and\n+                (not path.endswith('.pyi') or options.follow_imports_for_stubs)):\n+            continue\n+        if path.split(os.path.sep)[-1] in ('__init__.py', '__init__.pyi'):","path":"mypy/build.py","position":36,"original_position":36,"commit_id":"1957435cf41a0363ec161c53b3604a30f2450afb","original_commit_id":"1957435cf41a0363ec161c53b3604a30f2450afb","user":{"login":"JukkaL","id":1107911,"node_id":"MDQ6VXNlcjExMDc5MTE=","avatar_url":"https://avatars2.githubusercontent.com/u/1107911?v=4","gravatar_id":"","url":"https://api.github.com/users/JukkaL","html_url":"https://github.com/JukkaL","followers_url":"https://api.github.com/users/JukkaL/followers","following_url":"https://api.github.com/users/JukkaL/following{/other_user}","gists_url":"https://api.github.com/users/JukkaL/gists{/gist_id}","starred_url":"https://api.github.com/users/JukkaL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JukkaL/subscriptions","organizations_url":"https://api.github.com/users/JukkaL/orgs","repos_url":"https://api.github.com/users/JukkaL/repos","events_url":"https://api.github.com/users/JukkaL/events{/privacy}","received_events_url":"https://api.github.com/users/JukkaL/received_events","type":"User","site_admin":false},"body":"Using `str.split` for paths can be problematic, since Windows allows both `/` and `\\\\` as path separators, and it can be hard to ensure that the same separator is always used. On reflection, the original test seems better, if you add a comment saying that we don't rely on the test for correctness. In my previous review I didn't realize that the test can be more general than strictly necessary. \r\n\r\nThe original test was too lenient, which is better than being too strict like this test, and implementing a perfect test doesn't seem worth it.","created_at":"2018-09-03T16:07:11Z","updated_at":"2018-09-03T16:11:02Z","html_url":"https://github.com/python/mypy/pull/5544#discussion_r214729508","pull_request_url":"https://api.github.com/repos/python/mypy/pulls/5544","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/python/mypy/pulls/comments/214729508"},"html":{"href":"https://github.com/python/mypy/pull/5544#discussion_r214729508"},"pull_request":{"href":"https://api.github.com/repos/python/mypy/pulls/5544"}}},"pull_request":{"url":"https://api.github.com/repos/python/mypy/pulls/5544","id":212246742,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEyMjQ2NzQy","html_url":"https://github.com/python/mypy/pull/5544","diff_url":"https://github.com/python/mypy/pull/5544.diff","patch_url":"https://github.com/python/mypy/pull/5544.patch","issue_url":"https://api.github.com/repos/python/mypy/issues/5544","number":5544,"state":"open","locked":false,"title":"Fix performance regression caused by #5465: non-daemon part","user":{"login":"ilevkivskyi","id":12005495,"node_id":"MDQ6VXNlcjEyMDA1NDk1","avatar_url":"https://avatars1.githubusercontent.com/u/12005495?v=4","gravatar_id":"","url":"https://api.github.com/users/ilevkivskyi","html_url":"https://github.com/ilevkivskyi","followers_url":"https://api.github.com/users/ilevkivskyi/followers","following_url":"https://api.github.com/users/ilevkivskyi/following{/other_user}","gists_url":"https://api.github.com/users/ilevkivskyi/gists{/gist_id}","starred_url":"https://api.github.com/users/ilevkivskyi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilevkivskyi/subscriptions","organizations_url":"https://api.github.com/users/ilevkivskyi/orgs","repos_url":"https://api.github.com/users/ilevkivskyi/repos","events_url":"https://api.github.com/users/ilevkivskyi/events{/privacy}","received_events_url":"https://api.github.com/users/ilevkivskyi/received_events","type":"User","site_admin":false},"body":"The problem with the initial fix in #5465 is that it assumes that all modules that are in `st.suppressed` that now exist in file system are newly added, while in fact they can be not new if one uses `follow_imports = skip`.\r\n\r\nMy initial guess was that this causes a lot of files become stale. But it actually turns out that the staleness is determined correctly (all the tests I added actually pass on master). However, we still may parse a lot of modules unnecessarily.\r\n\r\nThis PR fixes the performance regression by:\r\n* Not treating modules as newly added if they were added to `st.suppressed` because of `follow_imports = skip`\r\n* ~~Never processing newly added modules on initial loading during cold daemon run (this is consistent with how it is done in other places where new modules are detected)~~ Moved to a separate PR #5556.\r\n\r\nThere still some overhead remains -- we need to clone options for module to understand why it got into suppressed dependencies. This can be improved by having to separate suppressed lists, but I think this win will be quite minor and it is not worth the complexity.","created_at":"2018-08-31T00:39:02Z","updated_at":"2018-09-03T16:11:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"74ba6eeccc2f9b8e195d5711cc525540a81e2576","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/python/mypy/pulls/5544/commits","review_comments_url":"https://api.github.com/repos/python/mypy/pulls/5544/comments","review_comment_url":"https://api.github.com/repos/python/mypy/pulls/comments{/number}","comments_url":"https://api.github.com/repos/python/mypy/issues/5544/comments","statuses_url":"https://api.github.com/repos/python/mypy/statuses/1957435cf41a0363ec161c53b3604a30f2450afb","head":{"label":"ilevkivskyi:perf-regression","ref":"perf-regression","sha":"1957435cf41a0363ec161c53b3604a30f2450afb","user":{"login":"ilevkivskyi","id":12005495,"node_id":"MDQ6VXNlcjEyMDA1NDk1","avatar_url":"https://avatars1.githubusercontent.com/u/12005495?v=4","gravatar_id":"","url":"https://api.github.com/users/ilevkivskyi","html_url":"https://github.com/ilevkivskyi","followers_url":"https://api.github.com/users/ilevkivskyi/followers","following_url":"https://api.github.com/users/ilevkivskyi/following{/other_user}","gists_url":"https://api.github.com/users/ilevkivskyi/gists{/gist_id}","starred_url":"https://api.github.com/users/ilevkivskyi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilevkivskyi/subscriptions","organizations_url":"https://api.github.com/users/ilevkivskyi/orgs","repos_url":"https://api.github.com/users/ilevkivskyi/repos","events_url":"https://api.github.com/users/ilevkivskyi/events{/privacy}","received_events_url":"https://api.github.com/users/ilevkivskyi/received_events","type":"User","site_admin":false},"repo":{"id":55439770,"node_id":"MDEwOlJlcG9zaXRvcnk1NTQzOTc3MA==","name":"mypy","full_name":"ilevkivskyi/mypy","owner":{"login":"ilevkivskyi","id":12005495,"node_id":"MDQ6VXNlcjEyMDA1NDk1","avatar_url":"https://avatars1.githubusercontent.com/u/12005495?v=4","gravatar_id":"","url":"https://api.github.com/users/ilevkivskyi","html_url":"https://github.com/ilevkivskyi","followers_url":"https://api.github.com/users/ilevkivskyi/followers","following_url":"https://api.github.com/users/ilevkivskyi/following{/other_user}","gists_url":"https://api.github.com/users/ilevkivskyi/gists{/gist_id}","starred_url":"https://api.github.com/users/ilevkivskyi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilevkivskyi/subscriptions","organizations_url":"https://api.github.com/users/ilevkivskyi/orgs","repos_url":"https://api.github.com/users/ilevkivskyi/repos","events_url":"https://api.github.com/users/ilevkivskyi/events{/privacy}","received_events_url":"https://api.github.com/users/ilevkivskyi/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ilevkivskyi/mypy","description":"Optional static typing for Python","fork":true,"url":"https://api.github.com/repos/ilevkivskyi/mypy","forks_url":"https://api.github.com/repos/ilevkivskyi/mypy/forks","keys_url":"https://api.github.com/repos/ilevkivskyi/mypy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ilevkivskyi/mypy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ilevkivskyi/mypy/teams","hooks_url":"https://api.github.com/repos/ilevkivskyi/mypy/hooks","issue_events_url":"https://api.github.com/repos/ilevkivskyi/mypy/issues/events{/number}","events_url":"https://api.github.com/repos/ilevkivskyi/mypy/events","assignees_url":"https://api.github.com/repos/ilevkivskyi/mypy/assignees{/user}","branches_url":"https://api.github.com/repos/ilevkivskyi/mypy/branches{/branch}","tags_url":"https://api.github.com/repos/ilevkivskyi/mypy/tags","blobs_url":"https://api.github.com/repos/ilevkivskyi/mypy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ilevkivskyi/mypy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ilevkivskyi/mypy/git/refs{/sha}","trees_url":"https://api.github.com/repos/ilevkivskyi/mypy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ilevkivskyi/mypy/statuses/{sha}","languages_url":"https://api.github.com/repos/ilevkivskyi/mypy/languages","stargazers_url":"https://api.github.com/repos/ilevkivskyi/mypy/stargazers","contributors_url":"https://api.github.com/repos/ilevkivskyi/mypy/contributors","subscribers_url":"https://api.github.com/repos/ilevkivskyi/mypy/subscribers","subscription_url":"https://api.github.com/repos/ilevkivskyi/mypy/subscription","commits_url":"https://api.github.com/repos/ilevkivskyi/mypy/commits{/sha}","git_commits_url":"https://api.github.com/repos/ilevkivskyi/mypy/git/commits{/sha}","comments_url":"https://api.github.com/repos/ilevkivskyi/mypy/comments{/number}","issue_comment_url":"https://api.github.com/repos/ilevkivskyi/mypy/issues/comments{/number}","contents_url":"https://api.github.com/repos/ilevkivskyi/mypy/contents/{+path}","compare_url":"https://api.github.com/repos/ilevkivskyi/mypy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ilevkivskyi/mypy/merges","archive_url":"https://api.github.com/repos/ilevkivskyi/mypy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ilevkivskyi/mypy/downloads","issues_url":"https://api.github.com/repos/ilevkivskyi/mypy/issues{/number}","pulls_url":"https://api.github.com/repos/ilevkivskyi/mypy/pulls{/number}","milestones_url":"https://api.github.com/repos/ilevkivskyi/mypy/milestones{/number}","notifications_url":"https://api.github.com/repos/ilevkivskyi/mypy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ilevkivskyi/mypy/labels{/name}","releases_url":"https://api.github.com/repos/ilevkivskyi/mypy/releases{/id}","deployments_url":"https://api.github.com/repos/ilevkivskyi/mypy/deployments","created_at":"2016-04-04T19:45:28Z","updated_at":"2018-06-09T17:35:02Z","pushed_at":"2018-09-03T14:57:28Z","git_url":"git://github.com/ilevkivskyi/mypy.git","ssh_url":"git@github.com:ilevkivskyi/mypy.git","clone_url":"https://github.com/ilevkivskyi/mypy.git","svn_url":"https://github.com/ilevkivskyi/mypy","homepage":"http://www.mypy-lang.org/","size":29000,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"python:master","ref":"master","sha":"8b0f02209e717d726b34d65d8418a0094bc00816","user":{"login":"python","id":1525981,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MjU5ODE=","avatar_url":"https://avatars3.githubusercontent.com/u/1525981?v=4","gravatar_id":"","url":"https://api.github.com/users/python","html_url":"https://github.com/python","followers_url":"https://api.github.com/users/python/followers","following_url":"https://api.github.com/users/python/following{/other_user}","gists_url":"https://api.github.com/users/python/gists{/gist_id}","starred_url":"https://api.github.com/users/python/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/python/subscriptions","organizations_url":"https://api.github.com/users/python/orgs","repos_url":"https://api.github.com/users/python/repos","events_url":"https://api.github.com/users/python/events{/privacy}","received_events_url":"https://api.github.com/users/python/received_events","type":"Organization","site_admin":false},"repo":{"id":7053637,"node_id":"MDEwOlJlcG9zaXRvcnk3MDUzNjM3","name":"mypy","full_name":"python/mypy","owner":{"login":"python","id":1525981,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MjU5ODE=","avatar_url":"https://avatars3.githubusercontent.com/u/1525981?v=4","gravatar_id":"","url":"https://api.github.com/users/python","html_url":"https://github.com/python","followers_url":"https://api.github.com/users/python/followers","following_url":"https://api.github.com/users/python/following{/other_user}","gists_url":"https://api.github.com/users/python/gists{/gist_id}","starred_url":"https://api.github.com/users/python/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/python/subscriptions","organizations_url":"https://api.github.com/users/python/orgs","repos_url":"https://api.github.com/users/python/repos","events_url":"https://api.github.com/users/python/events{/privacy}","received_events_url":"https://api.github.com/users/python/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/python/mypy","description":"Optional static typing for Python 2 and 3 (PEP 484)","fork":false,"url":"https://api.github.com/repos/python/mypy","forks_url":"https://api.github.com/repos/python/mypy/forks","keys_url":"https://api.github.com/repos/python/mypy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/python/mypy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/python/mypy/teams","hooks_url":"https://api.github.com/repos/python/mypy/hooks","issue_events_url":"https://api.github.com/repos/python/mypy/issues/events{/number}","events_url":"https://api.github.com/repos/python/mypy/events","assignees_url":"https://api.github.com/repos/python/mypy/assignees{/user}","branches_url":"https://api.github.com/repos/python/mypy/branches{/branch}","tags_url":"https://api.github.com/repos/python/mypy/tags","blobs_url":"https://api.github.com/repos/python/mypy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/python/mypy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/python/mypy/git/refs{/sha}","trees_url":"https://api.github.com/repos/python/mypy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/python/mypy/statuses/{sha}","languages_url":"https://api.github.com/repos/python/mypy/languages","stargazers_url":"https://api.github.com/repos/python/mypy/stargazers","contributors_url":"https://api.github.com/repos/python/mypy/contributors","subscribers_url":"https://api.github.com/repos/python/mypy/subscribers","subscription_url":"https://api.github.com/repos/python/mypy/subscription","commits_url":"https://api.github.com/repos/python/mypy/commits{/sha}","git_commits_url":"https://api.github.com/repos/python/mypy/git/commits{/sha}","comments_url":"https://api.github.com/repos/python/mypy/comments{/number}","issue_comment_url":"https://api.github.com/repos/python/mypy/issues/comments{/number}","contents_url":"https://api.github.com/repos/python/mypy/contents/{+path}","compare_url":"https://api.github.com/repos/python/mypy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/python/mypy/merges","archive_url":"https://api.github.com/repos/python/mypy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/python/mypy/downloads","issues_url":"https://api.github.com/repos/python/mypy/issues{/number}","pulls_url":"https://api.github.com/repos/python/mypy/pulls{/number}","milestones_url":"https://api.github.com/repos/python/mypy/milestones{/number}","notifications_url":"https://api.github.com/repos/python/mypy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/python/mypy/labels{/name}","releases_url":"https://api.github.com/repos/python/mypy/releases{/id}","deployments_url":"https://api.github.com/repos/python/mypy/deployments","created_at":"2012-12-07T13:30:23Z","updated_at":"2018-09-03T15:42:23Z","pushed_at":"2018-09-03T15:04:18Z","git_url":"git://github.com/python/mypy.git","ssh_url":"git@github.com:python/mypy.git","clone_url":"https://github.com/python/mypy.git","svn_url":"https://github.com/python/mypy","homepage":"http://www.mypy-lang.org/","size":27831,"stargazers_count":4471,"watchers_count":4471,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":614,"mirror_url":null,"archived":false,"open_issues_count":879,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":614,"open_issues":879,"watchers":4471,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/python/mypy/pulls/5544"},"html":{"href":"https://github.com/python/mypy/pull/5544"},"issue":{"href":"https://api.github.com/repos/python/mypy/issues/5544"},"comments":{"href":"https://api.github.com/repos/python/mypy/issues/5544/comments"},"review_comments":{"href":"https://api.github.com/repos/python/mypy/pulls/5544/comments"},"review_comment":{"href":"https://api.github.com/repos/python/mypy/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/python/mypy/pulls/5544/commits"},"statuses":{"href":"https://api.github.com/repos/python/mypy/statuses/1957435cf41a0363ec161c53b3604a30f2450afb"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"JukkaL"}}