title: Speeding up pairing of strings into objects in Python
body: <p>I'm trying to find an efficient way to pair together rows of data containing integer points, and storing them as Python objects. The data is made up of <code>X</code> and <code>Y</code> coordinate points, represented as a comma separated strings. The points have to be paired, as in <code>(x_1, y_1), (x_2, y_2), ...</code> etc. and then stored as a list of objects, where each point is an object. The function below <code>get_data</code> generates this example data:</p>
  <pre><code>def get_data(N=100000, M=10):
      import random
      data = []
      for n in range(N):
          pair = [[str(random.randint(1, 10)) for x in range(M)],
                  [str(random.randint(1, 10)) for x in range(M)]]
          row = [&quot;,&quot;.join(pair[0]),
                 &quot;,&quot;.join(pair[1])]
          data.append(row)
      return data
  </code></pre>
  <p>The parsing code I have now is:</p>
  <pre><code>class Point:
      def __init__(self, a, b):
          self.a = a
          self.b = b
      
  def test():
      import time
      data = get_data()
      all_point_sets = []
      time_start = time.time()
      for row in data:
          point_set = []
          first_points, second_points = row
          # Convert points from strings to integers
          first_points = map(int, first_points.split(&quot;,&quot;))
          second_points = map(int, second_points.split(&quot;,&quot;))
          paired_points = zip(first_points, second_points)
          curr_points = [Point(p[0], p[1]) \
                         for p in paired_points]
          all_point_sets.append(curr_points)
      time_end = time.time()
      print &quot;total time: &quot;, (time_end - time_start)
  </code></pre>
  <p>Currently, this takes nearly 7 seconds for 100,000 points, which seems very inefficient. Part of the inefficiency seems to stem from the calculation of <code>first_points</code>, <code>second_points</code> and <code>paired_points</code> - and the conversion of these into objects.</p>
  <p>Another part of the inefficiency seems to be the building up of <code>all_point_sets</code>. Taking out the <code>all_point_sets.append(...)</code> line seems to make the code go from ~7 seconds to 2 seconds!</p>
  <p>How can this be sped up?</p>
  <p><strong>FOLLOWUP</strong> Thanks for everyone's great suggestions - they were all helpful. but even with all the improvements, it's still about 3 seconds to process 100,000 entries. I'm not sure why in this case it's not just instant, and whether there's an alternative representation that would make it instant.  Would coding this in Cython change things?  Could someone offer an example of that?  thanks again.</p>