title: What is the fastest way to load a large CSV file into core data
body: <p><strong>Conclusion</strong><br>
  Problem closed, I think.<br>
  Looks like the problem had nothing to do with the methodology, but that the XCode did not clean the project correctly in between builds.<br>
  It looks like after all those tests, the sqlite file that was being used was still the very first one that wasn't indexed......<br>
  Beware of XCode 4.3.2, I have nothing but problems with Clean not cleaning, or adding files to project not automatically being added to the bundle resources...<br>
  Thanks for the different answers..</p>
  
  <p><strong>Update 3</strong><br>
  Since I invite anybody to just try the same steps to see if they get the same results, let me detail what I did:<br>
  I start with blank project <br>
  I defined a datamodel with one Entity, 3 attributes (2 strings, 1 float)<br>
  The first string is indexed<br>
  <img src="https://i.stack.imgur.com/FfLei.png" alt="enter image description here"><br><br>
  In did finishLaunchingWithOptions, I am calling:</p>
  
  <pre><code>[self performSelectorInBackground:@selector(populateDB) withObject:nil];
  </code></pre>
  
  <p>The code for populateDb is below:</p>
  
  <pre><code>-(void)populateDB{
  NSLog(@"start");
  NSPersistentStoreCoordinator *coordinator = [self persistentStoreCoordinator];
  NSManagedObjectContext *context;
  if (coordinator != nil) {
      context = [[NSManagedObjectContext alloc] init];
      [context setPersistentStoreCoordinator:coordinator];
  }
  
  NSString *filePath = [[NSBundle mainBundle] pathForResource:@"input" ofType:@"txt"];  
  if (filePath) {  
      NSString * myText = [[NSString alloc]
                                 initWithContentsOfFile:filePath
                                 encoding:NSUTF8StringEncoding
                                 error:nil];
      if (myText) {
          __block int count = 0;
  
  
          [myText enumerateLinesUsingBlock:^(NSString * line, BOOL * stop) {
              line=[line stringByReplacingOccurrencesOfString:@"\t" withString:@" "];
              NSArray *lineComponents=[line componentsSeparatedByString:@" "];
              if(lineComponents){
                  if([lineComponents count]==3){
                      float f=[[lineComponents objectAtIndex:0] floatValue];
                      NSNumber *number=[NSNumber numberWithFloat:f];
                      NSString *string1=[lineComponents objectAtIndex:1];
                      NSString *string2=[lineComponents objectAtIndex:2];
                      NSManagedObject *object=[NSEntityDescription insertNewObjectForEntityForName:@"Bigram" inManagedObjectContext:context];
                      [object setValue:number forKey:@"number"];
                      [object setValue:string1 forKey:@"string1"];
                      [object setValue:string2 forKey:@"string2"];
                      NSError *error;
                      count++;
                      if(count&gt;=1000){
                          if (![context save:&amp;error]) {
                              NSLog(@"Whoops, couldn't save: %@", [error localizedDescription]);
                          }
                          count=0;
  
                      }
                  }
              }
  
  
  
          }];
          NSLog(@"done importing");
          NSError *error;
          if (![context save:&amp;error]) {
              NSLog(@"Whoops, couldn't save: %@", [error localizedDescription]);
          }
  
      }  
  }
  NSLog(@"end");
  }
  </code></pre>
  
  <p>Everything else is default core data code, nothing added.<br>
  I run that in the simulator.<br>
  I go to ~/Library/Application Support/iPhone Simulator/5.1/Applications//Documents<br>
  There is the sqlite file that is generated<br><br>
  I take that and I copy it in my bundle<br>
  <br>
  I comment out the call to populateDb<br><br>
  I edit persistentStoreCoordinator to copy the sqlite file from bundle to documents at first run<br></p>
  
  <pre><code>- (NSPersistentStoreCoordinator *)persistentStoreCoordinator 
  {
  @synchronized (self)
  {
      if (__persistentStoreCoordinator != nil)
          return __persistentStoreCoordinator;
  
      NSString *defaultStorePath = [[NSBundle mainBundle] pathForResource:@"myProject" ofType:@"sqlite"];
      NSString *storePath = [[[self applicationDocumentsDirectory] path] stringByAppendingPathComponent: @"myProject.sqlite"];
  
      NSError *error;
      if (![[NSFileManager defaultManager] fileExistsAtPath:storePath]) 
      {
          if ([[NSFileManager defaultManager] copyItemAtPath:defaultStorePath toPath:storePath error:&amp;error])
              NSLog(@"Copied starting data to %@", storePath);
          else 
              NSLog(@"Error copying default DB to %@ (%@)", storePath, error);
      }
  
      NSURL *storeURL = [NSURL fileURLWithPath:storePath];
  
      __persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];
  
      NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                               [NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
                               [NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
  
      if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeURL options:options error:&amp;error]) 
      {
  
          NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
          abort();
      }    
  
      return __persistentStoreCoordinator;
  }    
  }
  </code></pre>
  
  <p><br>
  I remove the app from the simulator, I check that ~/Library/Application Support/iPhone Simulator/5.1/Applications/ is now removed<br>I rebuild and launch again<br>
  As expected, the sqlite file is copied over to ~/Library/Application Support/iPhone Simulator/5.1/Applications//Documents<br></p>
  
  <p><strong>However the size of the file is smaller than in the bundle, significantly!</strong>
  <strong>Also, doing a simple query with a predicate like this predicate = [NSPredicate predicateWithFormat:@"string1 == %@", string1];   clearly shows that string1 is not indexed anymore</strong>
  <br>
  <br>
  Following that, I create a new version of the datamodel, with a meaningless update, just to force a lightweight migration<br>
  If run on the simulator, the migration takes a few seconds, the database doubles in size and the same query now takes less than a second to return instead of minutes.<br>
  This would solve my problem, force a migration, but that same migration takes 3 minutes on the iPad and happens in the foreground.<br>
  So hat's where I am at right now, the best solution for me would still be to prevent the indexes to be removed, any other importing solution at launch time just takes too much time.<br>
  Let me know if you need more clarifications...</p>
  
  <p><strong>Update 2</strong><br>
  So the best result I have had so far is to seed the core data database with the sqlite file produced from a quick tool with similar data model, but without the indexes set when producing the sqlite file. Then, I import this sqlite file in the core data app with the indexes set, and allowing for a lightweight migration. For 2 millions record on the new iPad, this migration stills take 3 minutes. The final app should have 5 times this number of records, so we're still looking at a long long processing time. 
  If I go that route, the new question would be: can a lightweight migration be performed in the background?</p>
  
  <p><strong>Update</strong><br>
  My question is NOT how to create a tool to populate a Core Data database, and then import the sqlite file into my app.<br> I know how to do this, I have done it countless times.<br> But until now, I had not realized that such method could have some side effect: in my case, an indexed attribute in the resulting database clearly got 'unindexed' when importing the sqlite file that way.<br>
  If you were able to verify that any indexed data is still indexed after such transfer, I am interested to know how you proceed, or otherwise what would be the best strategy to seed such database efficiently.
  <br><br>
  <strong>Original</strong></p>
  
  <p>I have a large CSV file (millions of lines) with 4 columns, strings and floats.
  This is for an iOS app.<br><br>
  I need this to be loaded into core data the first time the app is loaded.<br><br>
  The app is pretty much non functional until the data is available, so loading time matters, as a first time user obviously does not want the app to take 20 minutes to load before being able to run it.<br><br>
  Right now, my current code takes 20 min on the new iPad to process a 2 millions line csv file.<br><br>
  I am using a background context to not lock the UI, and save the context every 1,000 records<br><br>
  The first idea I had was to generate the database on the simulator, then to copy/paste it in the document folder at first launch, as this is the common non official way of seeding a large database. Unfortunately, the indexes don't seem to survive such a transfer, and although the database was available after just a few seconds, performance is terrible because my indexes were lost. I posted a question about the indexes already, but there doesn't seem to be a good answer to that.<br>
  <br>
  So what I am looking for, either:</p>
  
  <ul>
  <li>a way to improve performance on loading millions of records in core data</li>
  <li>if the database is pre-loaded and moved at first startup, a way to keep my indexes</li>
  <li>best practices for handling this kind of scenario. I don't remember using any app that requires me to wait for x minutes before first use (but maybe The Daily, and that was a terrible experience).</li>
  <li>Any creative way to make the user wait without him realizing it: background import while going through tutorial, etc...</li>
  <li>Not Using Core Data? </li>
  <li>...</li>
  </ul>