{"action":"created","issue":{"url":"https://api.github.com/repos/yieldbot/flambo/issues/103","repository_url":"https://api.github.com/repos/yieldbot/flambo","labels_url":"https://api.github.com/repos/yieldbot/flambo/issues/103/labels{/name}","comments_url":"https://api.github.com/repos/yieldbot/flambo/issues/103/comments","events_url":"https://api.github.com/repos/yieldbot/flambo/issues/103/events","html_url":"https://github.com/yieldbot/flambo/issues/103","id":190473908,"number":103,"title":"Connecting spark cluster from REPL","user":{"login":"tokomakoma123","id":840480,"avatar_url":"https://avatars.githubusercontent.com/u/840480?v=3","gravatar_id":"","url":"https://api.github.com/users/tokomakoma123","html_url":"https://github.com/tokomakoma123","followers_url":"https://api.github.com/users/tokomakoma123/followers","following_url":"https://api.github.com/users/tokomakoma123/following{/other_user}","gists_url":"https://api.github.com/users/tokomakoma123/gists{/gist_id}","starred_url":"https://api.github.com/users/tokomakoma123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tokomakoma123/subscriptions","organizations_url":"https://api.github.com/users/tokomakoma123/orgs","repos_url":"https://api.github.com/users/tokomakoma123/repos","events_url":"https://api.github.com/users/tokomakoma123/events{/privacy}","received_events_url":"https://api.github.com/users/tokomakoma123/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-11-19T03:36:42Z","updated_at":"2016-11-19T18:37:36Z","closed_at":null,"body":"I am trying to connect to a standalone spark cluster from REPL.\r\n\r\n```clj\r\n(ns core\r\n  (:require [flambo.conf :as conf]\r\n            [flambo.api :as f]\r\n            [flambo.tuple :as ft]\r\n            [clojure.string :as str]\r\n            [flambo.kryo :as kryo]\r\n            ))\r\n\r\n(def c (-> (conf/spark-conf)\r\n           (conf/master \"local[*]\")\r\n           (conf/app-name \"flame-example\")))\r\n\r\n(def sc (f/spark-context c))\r\n\r\n(def data (f/parallelize sc [[\"a\" 1] [\"b\" 2] [\"c\" 3] [\"d\" 4] [\"e\" 5]]))\r\n\r\n(-> (f/text-file sc \"data/advs.txt\")\r\n    (f/map (f/fn [s] (count s))) \r\n    (f/reduce (f/fn [x y] (+ x y))))\r\n```\r\nThe above works fine.\r\n\r\n```clj\r\n(def c (-> (conf/spark-conf)\r\n           (conf/master \"spark://phenix:7077\")\r\n           (conf/app-name \"flame-example\")))\r\n\r\n(def sc (f/spark-context c))\r\n\r\n(-> (f/text-file sc \"data/advs.txt\")\r\n    (f/map (f/fn [s] (count s))) \r\n    (f/reduce (f/fn [x y] (+ x y))))\r\n```\r\nHowever, the above fails with the following exception:\r\n\r\n```\r\nClassNotFoundException flambo.kryo.BaseFlamboRegistrator  java.net.URLClassLoader.findClass (URLClassLoader.java:381)\r\n```\r\n\r\nI've added 2 jars to the spark context and tried to run failed with the different exception.\r\n\r\n```\r\n(.addJar sc \"/home/makoto/.m2/repository/com/twitter/carbonite/1.5.0/carbonite-1.5.0.jar\")\r\n(.addJar sc \"/home/makoto/.m2/repository/yieldbot/flambo/0.8.0/flambo-0.8.0.jar\")\r\n(-> (f/text-file sc \"data/advs.txt\")\r\n    (f/map (f/fn [s] (count s))) \r\n    (f/reduce (f/fn [x y] (+ x y))))\r\n```\r\n\r\n```\r\nNoClassDefFoundError Could not initialize class carbonite.JavaBridge  flambo.kryo.BaseFlamboRegistrator.registerClasses (BaseFlamboRegistrator.java:22)\r\n```\r\n\r\nIs there anyone help me to solve these problems ?"},"comment":{"url":"https://api.github.com/repos/yieldbot/flambo/issues/comments/261731359","html_url":"https://github.com/yieldbot/flambo/issues/103#issuecomment-261731359","issue_url":"https://api.github.com/repos/yieldbot/flambo/issues/103","id":261731359,"user":{"login":"sorenmacbeth","id":130043,"avatar_url":"https://avatars.githubusercontent.com/u/130043?v=3","gravatar_id":"","url":"https://api.github.com/users/sorenmacbeth","html_url":"https://github.com/sorenmacbeth","followers_url":"https://api.github.com/users/sorenmacbeth/followers","following_url":"https://api.github.com/users/sorenmacbeth/following{/other_user}","gists_url":"https://api.github.com/users/sorenmacbeth/gists{/gist_id}","starred_url":"https://api.github.com/users/sorenmacbeth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sorenmacbeth/subscriptions","organizations_url":"https://api.github.com/users/sorenmacbeth/orgs","repos_url":"https://api.github.com/users/sorenmacbeth/repos","events_url":"https://api.github.com/users/sorenmacbeth/events{/privacy}","received_events_url":"https://api.github.com/users/sorenmacbeth/received_events","type":"User","site_admin":false},"created_at":"2016-11-19T18:37:36Z","updated_at":"2016-11-19T18:37:36Z","body":"even so, I'm not sure it will work. the recommend and tested way of running a repl on a cluster is by starting an nrepl server on a port via an uberjar started via spark-submit as stated above."}}
{"actor":{"display_login":"sorenmacbeth"}}