{"action":"created","issue":{"url":"https://api.github.com/repos/DataSystemsLab/GeoSpark/issues/334","repository_url":"https://api.github.com/repos/DataSystemsLab/GeoSpark","labels_url":"https://api.github.com/repos/DataSystemsLab/GeoSpark/issues/334/labels{/name}","comments_url":"https://api.github.com/repos/DataSystemsLab/GeoSpark/issues/334/comments","events_url":"https://api.github.com/repos/DataSystemsLab/GeoSpark/issues/334/events","html_url":"https://github.com/DataSystemsLab/GeoSpark/issues/334","id":414683907,"node_id":"MDU6SXNzdWU0MTQ2ODM5MDc=","number":334,"title":"Need GeoSpark best practices in AWS Environment","user":{"login":"viswanatharv","id":41454060,"node_id":"MDQ6VXNlcjQxNDU0MDYw","avatar_url":"https://avatars1.githubusercontent.com/u/41454060?v=4","gravatar_id":"","url":"https://api.github.com/users/viswanatharv","html_url":"https://github.com/viswanatharv","followers_url":"https://api.github.com/users/viswanatharv/followers","following_url":"https://api.github.com/users/viswanatharv/following{/other_user}","gists_url":"https://api.github.com/users/viswanatharv/gists{/gist_id}","starred_url":"https://api.github.com/users/viswanatharv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viswanatharv/subscriptions","organizations_url":"https://api.github.com/users/viswanatharv/orgs","repos_url":"https://api.github.com/users/viswanatharv/repos","events_url":"https://api.github.com/users/viswanatharv/events{/privacy}","received_events_url":"https://api.github.com/users/viswanatharv/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2019-02-26T15:54:28Z","updated_at":"2019-07-23T16:12:36Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nLooking for best practices or examples to run GeoSpark in AWS Environment. I have tried GeoSpark SQL and GeoSpark Core in EMR and seeing scalability and performance issues. \r\n\r\nI have created a below Scala program and executed in EMR. In my desktop (i7-7700 CPU, 32 GB RAM) for 1 million Points and 19k polygons, it is taking 18 minutes, where as in EMR (1 master node and 1 core node, which is double capacity than desktop) is taking 30-50 min. I tried different options in EMR but still not able achieve the best performance. Can you please let me know is there any best practices to run the GeoSpark is Cluster? \r\n\r\n```scala\r\nimport org.apache.log4j.{Level, Logger}\r\nimport org.apache.spark.serializer.KryoSerializer\r\nimport org.apache.spark.storage.StorageLevel\r\nimport org.apache.spark.{SparkConf, SparkContext}\r\nimport org.datasyslab.geospark.enums.{FileDataSplitter, GridType, IndexType}\r\nimport org.datasyslab.geospark.formatMapper.shapefileParser.ShapefileReader\r\nimport org.datasyslab.geospark.spatialOperator.{JoinQuery, KNNQuery, RangeQuery}\r\nimport org.datasyslab.geospark.spatialRDD.{CircleRDD, PointRDD, PolygonRDD}\r\nimport java.text.SimpleDateFormat\r\nimport java.util.{Calendar, Date}\r\nimport org.apache.spark.sql.SparkSession\r\nimport org.datasyslab.geospark.serde.GeoSparkKryoRegistrator\r\nimport org.datasyslab.geosparksql.utils.Adapter\r\nimport org.datasyslab.geosparksql.utils.GeoSparkSQLRegistrator\r\n\r\n  Logger.getLogger(\"org\").setLevel(Level.WARN)\r\n  Logger.getLogger(\"akka\").setLevel(Level.WARN)\r\n\r\n  var sparkSession:SparkSession = (SparkSession\r\n    .builder()\r\n    .config(\"spark.serializer\",classOf[KryoSerializer].getName)\r\n    .config(\"spark.kryo.registrator\", classOf[GeoSparkKryoRegistrator].getName)\r\n    .config(\"spark.driver.memory\", \"1g\")\r\n    .config(\"spark.executor.memory\", \"2g\")\r\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\r\n    .config(\"spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version\",true)\r\n    .config(\"spark.shuffle.service.enabled\", \"true\")\r\n    .enableHiveSupport()\r\n    .appName(\"AuctionGeoSparkTestProcess\")\r\n    .getOrCreate())\r\n\r\nGeoSparkSQLRegistrator.registerAll(spark)\r\n  \r\n  val resourceFolder = \"s3://vish-bigdata-prod-temp/data/\"\r\n\r\n  val PointRDDInputLocation = resourceFolder + \"points.tsv\"\r\n  val PointRDDSplitter = FileDataSplitter.WKT\r\n  val PointRDDIndexType = IndexType.RTREE\r\n  val PointRDDNumPartitions = 16\r\n  val PointRDDOffset = 0\r\n\r\n  val PolygonRDDInputLocation = resourceFolder + \"polygons.tsv\"\r\n  val PolygonRDDSplitter = FileDataSplitter.WKT\r\n  val PolygonRDDNumPartitions = 16\r\n  val PolygonRDDStartOffset = 0\r\n  val PolygonRDDEndOffset = -1\r\n\r\n  val joinQueryPartitioningType = GridType.QUADTREE\r\n  val eachQueryLoopTimes=5\r\n\r\nval startTime = new java.text.SimpleDateFormat(\"yyyy-MM-dd HH:mm:ss\").format(java.util.Calendar.getInstance().getTime())\r\nprintln(startTime )\r\n\r\n    val polygonRDD = new PolygonRDD(sparkSession.sparkContext, PolygonRDDInputLocation, PolygonRDDStartOffset, PolygonRDDEndOffset, PolygonRDDSplitter, true,PolygonRDDNumPartitions)\r\n    val pointRDD = new PointRDD(sparkSession.sparkContext, PointRDDInputLocation, PointRDDOffset, PointRDDSplitter, true, PointRDDNumPartitions, StorageLevel.MEMORY_ONLY)\r\n\r\n    pointRDD.spatialPartitioning(joinQueryPartitioningType)\r\n    polygonRDD.spatialPartitioning(pointRDD.getPartitioner)\r\n\r\n    pointRDD.buildIndex(PointRDDIndexType, true)\r\n\r\n    pointRDD.indexedRDD.persist(StorageLevel.MEMORY_ONLY)\r\n    polygonRDD.spatialPartitionedRDD.persist(StorageLevel.MEMORY_ONLY)\r\n\r\n    for(i <- 1 to eachQueryLoopTimes)\r\n    {\r\n      val resultRDDCount = JoinQuery.SpatialJoinQuery(pointRDD, polygonRDD, true, false).count()\r\n      println(resultRDDCount)\r\n    }\r\n\r\n  val endTime = new java.text.SimpleDateFormat(\"yyyy-MM-dd HH:mm:ss\").format(java.util.Calendar.getInstance().getTime())\r\n  println(endTime)\r\n```\r\n\r\nThanks\r\nVish"},"comment":{"url":"https://api.github.com/repos/DataSystemsLab/GeoSpark/issues/comments/514278972","html_url":"https://github.com/DataSystemsLab/GeoSpark/issues/334#issuecomment-514278972","issue_url":"https://api.github.com/repos/DataSystemsLab/GeoSpark/issues/334","id":514278972,"node_id":"MDEyOklzc3VlQ29tbWVudDUxNDI3ODk3Mg==","user":{"login":"viswanatharv","id":41454060,"node_id":"MDQ6VXNlcjQxNDU0MDYw","avatar_url":"https://avatars1.githubusercontent.com/u/41454060?v=4","gravatar_id":"","url":"https://api.github.com/users/viswanatharv","html_url":"https://github.com/viswanatharv","followers_url":"https://api.github.com/users/viswanatharv/followers","following_url":"https://api.github.com/users/viswanatharv/following{/other_user}","gists_url":"https://api.github.com/users/viswanatharv/gists{/gist_id}","starred_url":"https://api.github.com/users/viswanatharv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viswanatharv/subscriptions","organizations_url":"https://api.github.com/users/viswanatharv/orgs","repos_url":"https://api.github.com/users/viswanatharv/repos","events_url":"https://api.github.com/users/viswanatharv/events{/privacy}","received_events_url":"https://api.github.com/users/viswanatharv/received_events","type":"User","site_admin":false},"created_at":"2019-07-23T16:12:36Z","updated_at":"2019-07-23T16:12:36Z","author_association":"NONE","body":"@bpurdy1645, Below are the bootstrap details to load GeoSpark and other configuration. This needs to be go to EMR launch site under Step 1: Software and Steps / Edit software settings / Enter Configuration. Please let me know if you need more details.\r\n\r\n[  \r\n   {  \r\n      \"classification\":\"spark-defaults\",\r\n      \"properties\":{  \r\n         \"spark.executor.memory\":\"1g\",\r\n         \"spark.driver.memory\":\"8g\",\r\n         \"spark.shuffle.compress\":\"true\",\r\n         \"spark.executor.heartbeatInterval\":\"120s\",\r\n         \"spark.network.timeout\":\"240s\",\r\n         \"spark.executor.memoryOverhead\":\"1g\",\r\n         \"spark.shuffle.service.enabled\":\"true\",\r\n         \"spark.kryo.registrator\":\"org.datasyslab.geospark.serde.GeoSparkKryoRegistrator\",\r\n         \"spark.serializer\":\"org.apache.spark.serializer.KryoSerializer\",\r\n         \"spark.jars\":\"s3://vish-bigdata-prod-temp/jars/GeoSpark/geospark-1.2.0.jar,s3://vish-bigdata-prod-temp/jars/GeoSpark/geospark-sql_2.3-1.2.0.jar\"\r\n      }\r\n   }\r\n]"}}
{"actor":{"display_login":"viswanatharv"}}