{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14731","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14731/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14731/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14731/events","html_url":"https://github.com/apache/spark/pull/14731","id":172266466,"number":14731,"title":"[SPARK-17159] [streaming]: optimise check for new files in FileInputDStream","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2016-08-20T11:18:36Z","updated_at":"2016-08-26T17:42:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14731","html_url":"https://github.com/apache/spark/pull/14731","diff_url":"https://github.com/apache/spark/pull/14731.diff","patch_url":"https://github.com/apache/spark/pull/14731.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR optimises the filesystem metadata reads in `FileInputDStream`, by moving the filters used in `FileSystem.globStatus` and `FileSystem.listStatus` into filtering of the `FileStatus` instances returned in the results, so avoiding the need to create `FileStatus` instances within the `FileSystem` operation.\r\n\r\n* This doesn't add overhead to the filtering process; that's done as post-processing in the`FileSystem` glob/list operations anyway.\r\n* At worst it may result in larger lists being built up and returned.\r\n* For every glob match of a file, the code saves 1 RPC calls to the HDFS NN; 1 GET against S3\r\n* For every glob match of a directory, the code the code saves 1 RPC call and 2-3 HTTP calls to S3 for the directory check (including a slow List call whenever the directory has children as it doesn't exist as a blob any more)\r\n* for the modtime check of every file, it saves a Hadoop RPC call, against all object stores *which don't implement any client-side cache*, an HTTP GET. \r\n* By entirely eliminating all `getFileStatus()` calls in the listed files, it should reduce the risk of AWS S3 throttling the HTTP request, as it does when too many requests are made to parts of a single S3 bucket.\r\n\r\n## How was this patch tested?\r\n\r\nRunning the spark streaming tests as a regression suite. In the SPARK-7481 cloud code, I could add a test against S3 which prints to stdout the exact number of HTTP requests made to S3 before and after the patch, so as to validate speedup. (the S3A metrics in Hadoop 2.8+ are accessible at the API level, but as they are only accessible in a new API added in 2.8; it'd stop that proposed module building against Hadoop 2.7. Logging and manual assessment is the only cross-version strategy."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/242802219","html_url":"https://github.com/apache/spark/pull/14731#issuecomment-242802219","issue_url":"https://api.github.com/repos/apache/spark/issues/14731","id":242802219,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-08-26T17:42:42Z","updated_at":"2016-08-26T17:42:42Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64488/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}