title: datatype for handling big numbers in pyspark
body: <p>I am using spark with python.After uploading a csv file,I needed to parse a column in a csv file which has numbers that are 22 digits long. For parsing that column I used <strong>LongType()</strong> . I used map() function for defining column.
  Following are my commands in pyspark.</p>
  
  <pre><code>&gt;&gt;&gt; test=sc.textFile("test.csv")
  &gt;&gt;&gt; header=test.first()
  &gt;&gt;&gt; schemaString = header.replace('"','')
  &gt;&gt;&gt; testfields = [StructField(field_name, StringType(), True) for field_name in schemaString.split(',')]
  &gt;&gt;&gt; testfields[5].dataType = LongType()
  &gt;&gt;&gt; testschema = StructType(testfields)
  &gt;&gt;&gt; testHeader = test.filter(lambda l: "test_date" in l)
  &gt;&gt;&gt; testNoHeader = test.subtract(testHeader)
  &gt;&gt;&gt; test_temp = testNoHeader.map(lambda k: k.split(",")).map(lambda
  p:(p[0],p[1],p[2],p[3],p[4],***float(p[5].strip('"'))***,p[6],p[7]))
  &gt;&gt;&gt; test_temp.top(2)
  </code></pre>
  
  <p>Note: I have also tried 'long' and 'bigint' in place of 'float' in my variable <strong><em>test_temp</em></strong>, but the error in spark was 'keyword not found'
      And following is the output</p>
  
  <pre><code>[('2012-03-14', '7', '1698.00', 'XYZ02abc008793060653', 'II93', ***8.27370028700801e+21*** , 'W0W0000000000007', '879870080088815007'), ('2002-03-14', '1', '999.00', 'ABC02E000050086941', 'II93', 8.37670028702205e+21, 'A0B0080000012523', '870870080000012421')]
  </code></pre>
  
  <p>The value in my csv file  is as follows:
  8.27370028700801e+21 is <strong><em>8273700287008010012345</em></strong>
  8.37670028702205e+21 is <strong><em>8376700287022050054321</em></strong></p>
  
  <p>When I create a data frame out of it and then query it,</p>
  
  <pre><code>&gt;&gt;&gt; test_df = sqlContext.createDataFrame(test_temp, testschema)
  &gt;&gt;&gt; test_df.registerTempTable("test")
  &gt;&gt;&gt; sqlContext.sql("SELECT test_column FROM test").show()
  </code></pre>
  
  <p>the <code>test_column</code> gives value 'null' for all the records. </p>
  
  <p>So, how to solve this problem of parsing big number in spark, really appreciate your help</p>