{"action":"created","issue":{"url":"https://api.github.com/repos/amplab/SparkNet/issues/63","repository_url":"https://api.github.com/repos/amplab/SparkNet","labels_url":"https://api.github.com/repos/amplab/SparkNet/issues/63/labels{/name}","comments_url":"https://api.github.com/repos/amplab/SparkNet/issues/63/comments","events_url":"https://api.github.com/repos/amplab/SparkNet/issues/63/events","html_url":"https://github.com/amplab/SparkNet/issues/63","id":133420344,"number":63,"title":"Issue with ImageNet Example","user":{"login":"rahulbhalerao001","id":4405642,"avatar_url":"https://avatars.githubusercontent.com/u/4405642?v=3","gravatar_id":"","url":"https://api.github.com/users/rahulbhalerao001","html_url":"https://github.com/rahulbhalerao001","followers_url":"https://api.github.com/users/rahulbhalerao001/followers","following_url":"https://api.github.com/users/rahulbhalerao001/following{/other_user}","gists_url":"https://api.github.com/users/rahulbhalerao001/gists{/gist_id}","starred_url":"https://api.github.com/users/rahulbhalerao001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rahulbhalerao001/subscriptions","organizations_url":"https://api.github.com/users/rahulbhalerao001/orgs","repos_url":"https://api.github.com/users/rahulbhalerao001/repos","events_url":"https://api.github.com/users/rahulbhalerao001/events{/privacy}","received_events_url":"https://api.github.com/users/rahulbhalerao001/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-02-13T10:22:23Z","updated_at":"2016-02-15T22:58:25Z","closed_at":null,"body":"Hello,\r\n\r\nI followed all the steps of the example including those in the issue, but my execution is getting stuck at a point. It is able to preprocess the training data but gets stuck in preprocessing the validation data. For the validation data, I downloaded the tar, untarred it, and repacked them into tars of around 250MB each, and then uploaded to S3. However, I am afraid I may have shuffled the order of images in this process.\r\n\r\nDue to this or some other reason, the preprocessing step for validation data remains stuck indefinitely, however preprocessing of all the train data gets completed in 20 min. I am using 2 g2.8xlarge machines as slaves, which remain at high usage during the time the execution is stuck. Any tips on debugging or resolving this issue will be greatly appreciated. I have tried with both persist() and persist(StorageLevel.MEMORY_AND_DISK).\r\n\r\nAs shown below, it gets stuck after launching 26 tasks for the 25 tars of val data.\r\n\r\n```\r\n16/02/13 09:48:27 INFO scheduler.DAGScheduler: Submitting 26 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at mapPartitions at ScaleAndConvert.scala:31)\r\n16/02/13 09:48:27 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 26 tasks\r\n16/02/13 09:48:27 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.31.16.22, PROCESS_LOCAL, 2221 bytes)\r\n16/02/13 09:48:27 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 172.31.25.24, PROCESS_LOCAL, 2228 bytes)\r\n..\r\n..\r\n16/02/13 09:48:27 INFO scheduler.TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, 172.31.25.24, PROCESS_LOCAL, 2228 bytes)\r\n16/02/13 09:48:27 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 172.31.16.22:43854 (size: 1742.0 B, free: 27.4 GB)\r\n16/02/13 09:48:27 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 172.31.25.24:38520 (size: 1742.0 B, free: 27.4 GB)\r\n16/02/13 09:48:28 INFO storage.BlockManagerInfo: Added rdd_4_0 in memory on 172.31.16.22:43854 (size: 24.0 B, free: 27.4 GB)\r\n16/02/13 09:48:28 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1503 ms on 172.31.16.22 (1/26)\r\n\r\n```\r\n\r\nAlso, on a related note I will like to point out 2-3 points that were not present in the Readme and which lead to errors on running the example.\r\n1. Creation of a /tmp/spark-events/ directory is required else it leads to an error.\r\n2. The readme states that the ~/.aws/credentials is required only on the master but not on the slaves. However, while running the example it is required on all the machines.\r\n3. After changing the S3 bucket name, the Sparknet project needs to be rebuilt using 'sbt assembly'. This was not specified in the Readme. Further, the issue states that name of the validation folders is actually test, but in the version of the project present in the AMI, the names are actually val.txt and IL..._va"},"comment":{"url":"https://api.github.com/repos/amplab/SparkNet/issues/comments/184432016","html_url":"https://github.com/amplab/SparkNet/issues/63#issuecomment-184432016","issue_url":"https://api.github.com/repos/amplab/SparkNet/issues/63","id":184432016,"user":{"login":"robertnishihara","id":249517,"avatar_url":"https://avatars.githubusercontent.com/u/249517?v=3","gravatar_id":"","url":"https://api.github.com/users/robertnishihara","html_url":"https://github.com/robertnishihara","followers_url":"https://api.github.com/users/robertnishihara/followers","following_url":"https://api.github.com/users/robertnishihara/following{/other_user}","gists_url":"https://api.github.com/users/robertnishihara/gists{/gist_id}","starred_url":"https://api.github.com/users/robertnishihara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robertnishihara/subscriptions","organizations_url":"https://api.github.com/users/robertnishihara/orgs","repos_url":"https://api.github.com/users/robertnishihara/repos","events_url":"https://api.github.com/users/robertnishihara/events{/privacy}","received_events_url":"https://api.github.com/users/robertnishihara/received_events","type":"User","site_admin":false},"created_at":"2016-02-15T22:58:25Z","updated_at":"2016-02-15T22:58:25Z","body":"Hi Rahul,\r\n\r\nA couple questions/comments:\r\n\r\n1. Have you tried the CifarApp? And if so, did that work? That one should be much easier to get running because you don't have to deal with S3. And if that doesn't work, it will be easier for us to track down the problem.\r\n2. I'd suggest deleting all of the ImageNet data from S3 and uploading it again with the script `SparkNet/scripts/put_imagenet_on_s3.py` (this will take several hours, but at least it will ensure that we are dealing with the same directory structure and file names).\r\n3. I'm not sure about the serialization error at the moment, but let me give it some thought."}}