I was asking a question about Merging geodatabase previously, (link: Merge from arcpy.ListTables() producing duplicate rows?) and although the script worked, it took more than 9 hours to run when merging >3000 tables with 1 rows in each table. I was wondering whether instead of merging the tables it would be faster to write the contents of each geodatabase table into a csv file and keep appending tables to it. If so, how to do it?? I now need to merge tables of different geodatabase (it has 39 fields and varying number of rows) and I have a feeling that if I use the script from my previous question it will take even longer than 9 hours.

Basically, how to iterate through 3000 tables and write the attribute tables of each to the same csv and keep appending to it?

Here's the screenshot of the geodatabase tables I talked about above: enter image description here


With cursors and the csv module, this should go pretty quick:

import arcpy, csv, time

arcpy.env.workspace = <path to gdb>
table_list = arcpy.ListTables()

csv_out = <path to csv>

#Get name of fields from first entry
fields = [x.name for x in arcpy.ListFields(table_list[0])]   

start = time.time()
counter = 0

with open(csv_out, "wb") as f:
    wr = csv.writer(f)
    for table in table_list:
        with arcpy.da.SearchCursor(table, fields) as cursor:
            for row in cursor:
                counter += 1

end = time.time()- start

print "{0} rows from {1} tables written in {2} seconds".format(counter, len(table_list), end)

On my machine, this was the output from tables with 26 fields:

3636 rows from 101 tables written in 7.33599996567 seconds.

  • thanks so much! I no longer have access to computer arcgis and these data, but I'll try your suggestion tomorrow. would that be okay with you if I ask you again IFF something is not going to work as expected? – atoregozh Jul 10 '14 at 22:01
  • @atoregozh, sure! I updated the script a bit to show the run time. Should eat through 3000 tables in no time. – Paul Jul 10 '14 at 22:03
  • thanks again! how do get 101 pseudo tables with 26 fields? – atoregozh Jul 10 '14 at 22:08
  • I just scripted a solution to copy the same table 100 times, haha. I still iterated over every table though. – Paul Jul 10 '14 at 22:10
  • 2
    Using "with...as" disposes of the object after you are done with it. In ArcGIS, not disposing stuff sometimes creates issues dealing with memory management. Using "with...as" provides a nice clean way to use a resource and automatically get rid/release it when you are done with it. It's a cleaner method than a try/finally block: effbot.org/zone/python-with-statement.htm – Conor Jul 10 '14 at 22:22

Your Answer

