{"push_id":926860551,"size":27,"distinct_size":26,"ref":"refs/heads/profiling_and_indexing_improvements","head":"96f82dc2cc70cc24bd8f492576d595c41f12566c","before":"a8b034f8fca802bd2a707767fe207ca5dedc8270","commits":[{"sha":"33fd2395a12df8ffc36442799503efd109d6253f","author":{"email":"c3f0b7d839d46cb8e9acd325562384ec01839d26@stanford.edu","name":"Darren L. Weber, Ph.D"},"message":"gitignore my local development files","distinct":false,"url":"https://api.github.com/repos/sul-dlss/argo/commits/33fd2395a12df8ffc36442799503efd109d6253f"},{"sha":"692b89465379897894f44d9414f3c8b0a4365673","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"profiler refactoring and cleanup, move indexing code to class in lib, improved indexing tools\n\n* include the ruby-prof gem all the time, might want to profile in any env\n* move DevelopmentProfiler to Argo::Profiler, give it an output dir, let it output callgrind\n* min_percent passed to CallTreePrinter to keep callgrind output size somewhat reasonable\n\n* old reindex_all code is deprecated\n* method to query fedora for pids by model type\n* reindex_pid_list and reindex_pid_list_with_profiling, with supporting methods for reindexing individual objects (with logging and fault tolerance).  repeats and slightly modifies code from dor_controller.rb, should probably find ways to generalize the new lib code and call it from dor_controller.\n\n* simple test job (SimpleLogJob) for testing delayed_job behavior.  initially used to see how strictly delayed_job adhered to the job priority setting.\n* a reindexing ActiveJob subclass, which takes a list of pids.  optionally commits to solr and does performance profiling.\n* Set ActiveJobs to use delayed_jobs, needed for Rails 4.2 https://github.com/collectiveidea/delayed_job#rails-42\n\nthis commit is the result of squashing a bunch of work many times.  thanks to @mejackreed for the many structural improvements and refactorings along the way.","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/692b89465379897894f44d9414f3c8b0a4365673"},{"sha":"d09b5d733d7ee6dd69cbc99348d28b1c4231aa6f","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"* add self.get_all_pids and self.get_pid_lists_for_full_reindex\n* better output file name for self.reindex_pid_list_with_profiling, logging statement tying pids to output files\n* add Argo::BulkReindexer class\n* rake argo:reindex_all calls Argo::BulkReindexer.reindex_all","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/d09b5d733d7ee6dd69cbc99348d28b1c4231aa6f"},{"sha":"7ca7560d55395ef60d0c455e978f50c91c2678fd","author":{"email":"48bfd871604a23ea4e07553f9e1703d235b10062@stanford.edu","name":"Jack Reed"},"message":"Move pid gathering functionality to its own class. Methods in a class instance memoize the called pid lists, so that these calls are not repeated.","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/7ca7560d55395ef60d0c455e978f50c91c2678fd"},{"sha":"03e42c48485ad7bdefd2169a7e52e05a2a9ec6c4","author":{"email":"48bfd871604a23ea4e07553f9e1703d235b10062@stanford.edu","name":"Jack Reed"},"message":"minor refactor of BulkReindexer to use instance of class","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/03e42c48485ad7bdefd2169a7e52e05a2a9ec6c4"},{"sha":"2c77fe0682a836291b9e86eb469a923bdfaf4d64","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"drop legacy reindexing code","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/2c77fe0682a836291b9e86eb469a923bdfaf4d64"},{"sha":"b30b6551b1fbb384db2def69b967656d22b6535c","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"set number of delayed_job workers and allocate them to pools for queues","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/b30b6551b1fbb384db2def69b967656d22b6535c"},{"sha":"c4dcde39f43fc288e8ec50ae08f506e931d0cedc","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"rename prioritize_pid_lists to queue_prioritized_pid_lists_for_reindexing","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/c4dcde39f43fc288e8ec50ae08f506e931d0cedc"},{"sha":"5ac6a87f34576dcbd374d096bcded02a92fe27f5","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"add logging, default to stdout, with the option to use the same logger as Argo::Indexer","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/5ac6a87f34576dcbd374d096bcded02a92fe27f5"},{"sha":"8f6035ee6f28a776572f82e74f3ef87d97231d0b","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"try a smaller pool of delayed_job workers","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/8f6035ee6f28a776572f82e74f3ef87d97231d0b"},{"sha":"2da85aecb1579915202d10a1676ccfb888db17f8","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"refactor Argo::Profiler\n* it's now something that's instantiated, and stores results in an instance var.\n* it has methods that are explicitly called on the profiler instance to print.  no longer automatically prints all formats.\n* optional min_percent level parameter for callgrind printer (print_results_call_tree), puts that info in the filename, defaults to 1.\n* reindex_pid_list_with_profiling prints callgrind with min_percent=3 (and is otherwise fixed for the new Argo::Profiler style).","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/2da85aecb1579915202d10a1676ccfb888db17f8"},{"sha":"2624ae87ab2a4403f67f3360b4a8c1f32f2825b3","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"default should_profile to false, allow setting should_commit and should_profile via params to argo:reindex_all rake task, remove override of min_percent on invocation of profiler.print_results_call_tree","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/2624ae87ab2a4403f67f3360b4a8c1f32f2825b3"},{"sha":"cce885fe078f87859865f5c3a3ce07e7dfcca1dc","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"longer Delayed::Worker.max_run_time","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/cce885fe078f87859865f5c3a3ce07e7dfcca1dc"},{"sha":"47e315fd58f9378edc7dc7f9004f166d61beeb55","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"turn the following into settings: reindexing batch size, commit completed bulk reindex batches, enable bulk reindexing profiling, calltree printer (callgrind) min percent, profiling output dir\n\n* settings.yml:  add sections for BULK_REINDEXER, INDEXING_JOB, and PROFILER.\n* indexing_job.rb:  replace should_commit and should_profile with Settings.INDEXING_JOB.SHOULD_COMMIT_BATCHES and Settings.INDEXING_JOB.SHOULD_PROFILE.\n* bulk_reindexer.rb:  no longer taking should_commit or should_profile, since they don't need to be passed on to IndexingJob.  get batch size from Settings.BULK_REINDEXER.BATCH_SIZE.\n* profiler.rb:  print_results_call_tree gets min_percent from Settings.  all print_* methods get output dir from Settings.\n* argo.rake:  don't take or send should_commit and should_profile, since those are handled by Settings now.","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/47e315fd58f9378edc7dc7f9004f166d61beeb55"},{"sha":"ff4432400c78fc67a2f8ff741fba32f989a23021","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"control RubyProf.measure_mode via Settings.PROFILER.MEASURE_MODE","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/ff4432400c78fc67a2f8ff741fba32f989a23021"},{"sha":"2eaa6fce55098c84f69243399575c4a09cc7bb09","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"catch, log, and re-raise SystemStackError in reindex_pid","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/2eaa6fce55098c84f69243399575c4a09cc7bb09"},{"sha":"82aedaa7da1b96593249c98a029a6d8e5ce73031","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"new methods for finding what's indexed in solr, and what's not.  usage comment on pid_lists_for_full_reindex.","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/82aedaa7da1b96593249c98a029a6d8e5ce73031"},{"sha":"c7a5618bebea6fc836b24cd9055615f8a262d223","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"PidGatherer query improvements, minor style fixes\n\nPidGatherer:\n* default to filtering invalid druids\n* new method for getting prioritized lists of druids that haven't been indexed from fedora to solr\n* fix tests to account for new filtering behavior and new fixtures\n* tighten up query logging\n * only mention that fedora and solr are being queried when fedora and solr are actually being queried\n * clarify counts by mentioning that they're based on unfiltered results\n\nalso:\n* fix hound complaints\n* style cleanup on block for each in reindex_pid_list","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/c7a5618bebea6fc836b24cd9055615f8a262d223"},{"sha":"d8b974bfd4f31504b8678f1f0ec70a8bd863c1ad","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"add method for getting objects of type Dor_Item, remove limit from risearch call in pids_for_model_type","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/d8b974bfd4f31504b8678f1f0ec70a8bd863c1ad"},{"sha":"ab9d5a6cc8175f1708fa3d9a96a9ae3b52d950f8","author":{"email":"39cf4b53b9e0fc78379b09ad6dc0a104f0f66c6c@stanford.edu","name":"Johnathan Martin"},"message":"allow solr_pids to take query definition, default to all","distinct":true,"url":"https://api.github.com/repos/sul-dlss/argo/commits/ab9d5a6cc8175f1708fa3d9a96a9ae3b52d950f8"}]}