{"action":"created","issue":{"url":"https://api.github.com/repos/cerndb/dist-keras/issues/7","repository_url":"https://api.github.com/repos/cerndb/dist-keras","labels_url":"https://api.github.com/repos/cerndb/dist-keras/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/cerndb/dist-keras/issues/7/comments","events_url":"https://api.github.com/repos/cerndb/dist-keras/issues/7/events","html_url":"https://github.com/cerndb/dist-keras/issues/7","id":198149364,"number":7,"title":"something wrong with my embedding's test","user":{"login":"sodaling","id":19169716,"avatar_url":"https://avatars.githubusercontent.com/u/19169716?v=3","gravatar_id":"","url":"https://api.github.com/users/sodaling","html_url":"https://github.com/sodaling","followers_url":"https://api.github.com/users/sodaling/followers","following_url":"https://api.github.com/users/sodaling/following{/other_user}","gists_url":"https://api.github.com/users/sodaling/gists{/gist_id}","starred_url":"https://api.github.com/users/sodaling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sodaling/subscriptions","organizations_url":"https://api.github.com/users/sodaling/orgs","repos_url":"https://api.github.com/users/sodaling/repos","events_url":"https://api.github.com/users/sodaling/events{/privacy}","received_events_url":"https://api.github.com/users/sodaling/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-30T13:47:51Z","updated_at":"2016-12-31T08:15:14Z","closed_at":"2016-12-31T08:15:14Z","body":"Here is my python runned on pyspark.\r\n```python\r\nfrom keras.layers.embeddings import Embedding\r\nfrom keras.layers.recurrent import LSTM, GRU\r\nfrom pyspark.ml.feature import VectorAssembler\r\nfrom keras.models import Sequential\r\nfrom keras.layers.core import *\r\nfrom distkeras.trainers import *\r\nfrom pyspark import SQLContext\r\n\r\n\r\nsqlContext = SQLContext(sc)\r\nreader = sqlContext\r\nraw_dataset = reader.read.format('com.databricks.spark.csv').options(header='false', inferSchema='true').load(\r\n    '/home/hpcc/test/11.csv')\r\n# raw_dataset = raw_dataset.repartition(4)\r\nfeatures = raw_dataset.columns\r\nfeatures.remove('C0')\r\nvector_assembler = VectorAssembler(inputCols=features, outputCol=\"features\")\r\ndataset = vector_assembler.transform(raw_dataset)\r\ndataset = dataset.select(\"features\", \"C0\")\r\nmodel1 = Sequential()\r\nmodel1.add(Embedding(input_dim=52965, output_dim=256))\r\nmodel1.add(LSTM(128))\r\nmodel1.add(Dropout(0.5))\r\nmodel1.add(Dense(1))\r\nmodel1.add(Activation('sigmoid'))\r\nmodel1.summary()\r\noptimizer_mlp= 'adagrad'\r\nloss_mlp = 'binary_crossentropy'\r\ndataset.cache()\r\ntrainer = DOWNPOUR(keras_model=model1, worker_optimizer=optimizer_mlp, loss=loss_mlp, num_workers=4,\r\n                   batch_size=16, communication_window=5, learning_rate=0.1, num_epoch=1,\r\n                   features_col=\"features\", label_col=\"C0\")\r\ntrainer.set_parallelism_factor(1)\r\ntrained_model = trainer.train(dataset)\r\n\r\n```\r\nBut it print error like this:\r\n```\r\n16/12/30 20:53:09 ERROR Executor: Exception in task 1.0 in stage 4.0 (TID 7)\r\norg.apache.spark.api.python.PythonException: Traceback (most recent call last):\r\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 111, in main\r\n    process()\r\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 106, in process\r\n    serializer.dump_stream(func(split_index, iterator), outfile)\r\n  File \"build/bdist.linux-x86_64/egg/distkeras/workers.py\", line 193, in train\r\n    X = np.asarray([x[self.features_column] for x in feature_iterator])\r\n  File \"/root/anaconda2/lib/python2.7/site-packages/numpy/core/numeric.py\", line 482, in asarray\r\n    return array(a, dtype, copy=False, order=order)\r\n  File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/linalg/__init__.py\", line 769, in __getitem__\r\n    raise ValueError(\"Index %d out of bounds.\" % index)\r\nValueError: Index 50 out of bounds.\r\n```\r\nMy 11.csv is like this :    \r\n```csv\r\n1,23,5211,871,1223,11,5,355,9999,1,28032,33057,1259,5575,4,1,9604,23790,83,136,18,4695,3,121,1,151,521,2,130,24677,4,1,612,947,612,1645,4534,59,29008,1,21679,1,416,3474,3,4189,1,142,5,11323,3\r\n........\r\n```\r\nIt's C0 column is label,and C1 to C50 are features.\r\nHope for your answer.Thanks a lots.   \r\n"},"comment":{"url":"https://api.github.com/repos/cerndb/dist-keras/issues/comments/269855109","html_url":"https://github.com/cerndb/dist-keras/issues/7#issuecomment-269855109","issue_url":"https://api.github.com/repos/cerndb/dist-keras/issues/7","id":269855109,"user":{"login":"sodaling","id":19169716,"avatar_url":"https://avatars.githubusercontent.com/u/19169716?v=3","gravatar_id":"","url":"https://api.github.com/users/sodaling","html_url":"https://github.com/sodaling","followers_url":"https://api.github.com/users/sodaling/followers","following_url":"https://api.github.com/users/sodaling/following{/other_user}","gists_url":"https://api.github.com/users/sodaling/gists{/gist_id}","starred_url":"https://api.github.com/users/sodaling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sodaling/subscriptions","organizations_url":"https://api.github.com/users/sodaling/orgs","repos_url":"https://api.github.com/users/sodaling/repos","events_url":"https://api.github.com/users/sodaling/events{/privacy}","received_events_url":"https://api.github.com/users/sodaling/received_events","type":"User","site_admin":false},"created_at":"2016-12-31T08:15:14Z","updated_at":"2016-12-31T08:15:14Z","body":">>> dataset.printSchema()\r\n```\r\nroot\r\n |-- features: vector (nullable = true)\r\n |-- C0: integer (nullable = true)\r\n```\r\n>>> dataset.show()\r\n```\r\n+--------------------+---+\r\n|            features| C0|\r\n+--------------------+---+\r\n|[23.0,5211.0,871....|  1|\r\n|[138.0,10992.0,4....|  1|\r\n|[906.0,182.0,13.0...|  1|\r\n|[1307.0,29.0,11.0...|  1|\r\n|[836.0,1.0,46.0,2...|  1|\r\n|[216.0,22425.0,43...|  1|\r\n|[1208.0,1.0,146.0...|  1|\r\n|[2.0,794.0,7053.0...|  1|\r\n|[930.0,307.0,2681...|  1|\r\n|[0.0,0.0,0.0,0.0,...|  1|\r\n|[452.0,59.0,6237....|  1|\r\n|[40.0,2.0,2705.0,...|  1|\r\n|[5.0,21.0,228.0,1...|  1|\r\n|[2982.0,2.0,2401....|  1|\r\n|[2.0,819.0,364.0,...|  1|\r\n|[405.0,28295.0,78...|  1|\r\n|[714.0,278.0,2653...|  1|\r\n|[21.0,129.0,5090....|  1|\r\n|[908.0,4.0,120.0,...|  1|\r\n|[321.0,944.0,1733...|  1|\r\n+--------------------+---+\r\nonly showing top 20 rows\r\n\r\n```\r\nI know this issue is not related to dist-keras.But my dissertation is build on your project........\r\nAnd i have no idea with this........So i have to ask you ........"}}
{"actor":{"display_login":"sodaling"}}