{"action":"created","issue":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/635","repository_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver","labels_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/635/labels{/name}","comments_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/635/comments","events_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/635/events","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/635","id":184912315,"number":635,"title":"\"message\": \"org.apache.spark.SparkException: File ./xxx-2016-10-21T18_03_06.005Z.jar exists and does not match contents of http://:46219/jars/xxx-2016-10-21T18_03_06.005Z.jar\",","user":{"login":"cgeorge-rms","id":16123309,"avatar_url":"https://avatars.githubusercontent.com/u/16123309?v=3","gravatar_id":"","url":"https://api.github.com/users/cgeorge-rms","html_url":"https://github.com/cgeorge-rms","followers_url":"https://api.github.com/users/cgeorge-rms/followers","following_url":"https://api.github.com/users/cgeorge-rms/following{/other_user}","gists_url":"https://api.github.com/users/cgeorge-rms/gists{/gist_id}","starred_url":"https://api.github.com/users/cgeorge-rms/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgeorge-rms/subscriptions","organizations_url":"https://api.github.com/users/cgeorge-rms/orgs","repos_url":"https://api.github.com/users/cgeorge-rms/repos","events_url":"https://api.github.com/users/cgeorge-rms/events{/privacy}","received_events_url":"https://api.github.com/users/cgeorge-rms/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-24T18:03:07Z","updated_at":"2016-10-25T03:45:32Z","closed_at":null,"body":"\"message\": \"org.apache.spark.SparkException: File ./xxx-2016-10-21T18_03_06.005Z.jar exists and does not match contents of http://:46219/jars/xxx-2016-10-21T18_03_06.005Z.jar\"\r\nAfter doing some digging I've found that when two different job classpaths are being executed using the same appName I hit this exception.\r\nI found that setting the following spark conf property seems to solve the issue:\r\nspark.files.overwrite = true\r\nHowever this is kinda a hacky solution.\r\nI think it would be better that instead of always checking the latest jar date and adding it to the context, that we should maintain the cached list of jars without the date and only add a given jar once to the context. \r\nI say this because:\r\n1. since classloading prevents a class from reloading once it's already been loaded.\r\n2. if I loaded Class A and then upload a jar and then loaded Class B I would actually get class B from the newer jar... but executions from A would be from older jar... this is really confusing.\r\n\r\nSo why not just check the cache for the appName regardless of date/classpath for the context... and if it doesn't exist in the cache then go ahead and check the latest date and load the jar."},"comment":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/comments/255928632","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/635#issuecomment-255928632","issue_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/635","id":255928632,"user":{"login":"noorul","id":155376,"avatar_url":"https://avatars.githubusercontent.com/u/155376?v=3","gravatar_id":"","url":"https://api.github.com/users/noorul","html_url":"https://github.com/noorul","followers_url":"https://api.github.com/users/noorul/followers","following_url":"https://api.github.com/users/noorul/following{/other_user}","gists_url":"https://api.github.com/users/noorul/gists{/gist_id}","starred_url":"https://api.github.com/users/noorul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noorul/subscriptions","organizations_url":"https://api.github.com/users/noorul/orgs","repos_url":"https://api.github.com/users/noorul/repos","events_url":"https://api.github.com/users/noorul/events{/privacy}","received_events_url":"https://api.github.com/users/noorul/received_events","type":"User","site_admin":false},"created_at":"2016-10-25T03:45:32Z","updated_at":"2016-10-25T03:45:32Z","body":"@cgeorge-rms I think one should restart the context/adhoc-job."}}
{"actor":{"display_login":"noorul"}}