{"action":"created","issue":{"url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/issues/445","repository_url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore","labels_url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/issues/445/labels{/name}","comments_url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/issues/445/comments","events_url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/issues/445/events","html_url":"https://github.com/aic-collections/aicdams-lakeshore/pull/445","id":293268308,"number":445,"title":"Bug #1686: Improve duplicate checking on Lakeshore UI ingestion","user":{"login":"RudyOnRails","id":1021154,"avatar_url":"https://avatars0.githubusercontent.com/u/1021154?v=4","gravatar_id":"","url":"https://api.github.com/users/RudyOnRails","html_url":"https://github.com/RudyOnRails","followers_url":"https://api.github.com/users/RudyOnRails/followers","following_url":"https://api.github.com/users/RudyOnRails/following{/other_user}","gists_url":"https://api.github.com/users/RudyOnRails/gists{/gist_id}","starred_url":"https://api.github.com/users/RudyOnRails/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RudyOnRails/subscriptions","organizations_url":"https://api.github.com/users/RudyOnRails/orgs","repos_url":"https://api.github.com/users/RudyOnRails/repos","events_url":"https://api.github.com/users/RudyOnRails/events{/privacy}","received_events_url":"https://api.github.com/users/RudyOnRails/received_events","type":"User","site_admin":false},"labels":[{"id":828036183,"url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/labels/DB%20migration","name":"DB migration","color":"d9db57","default":false},{"id":814140698,"url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/labels/WIP","name":"WIP","color":"7b21b7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/milestones/20","html_url":"https://github.com/aic-collections/aicdams-lakeshore/milestone/20","labels_url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/milestones/20/labels","id":3243581,"number":20,"title":"1.7.3","description":null,"creator":{"login":"RudyOnRails","id":1021154,"avatar_url":"https://avatars0.githubusercontent.com/u/1021154?v=4","gravatar_id":"","url":"https://api.github.com/users/RudyOnRails","html_url":"https://github.com/RudyOnRails","followers_url":"https://api.github.com/users/RudyOnRails/followers","following_url":"https://api.github.com/users/RudyOnRails/following{/other_user}","gists_url":"https://api.github.com/users/RudyOnRails/gists{/gist_id}","starred_url":"https://api.github.com/users/RudyOnRails/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RudyOnRails/subscriptions","organizations_url":"https://api.github.com/users/RudyOnRails/orgs","repos_url":"https://api.github.com/users/RudyOnRails/repos","events_url":"https://api.github.com/users/RudyOnRails/events{/privacy}","received_events_url":"https://api.github.com/users/RudyOnRails/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":0,"state":"open","created_at":"2018-04-04T22:11:57Z","updated_at":"2018-04-04T22:12:15Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2018-01-31T18:42:32Z","updated_at":"2018-04-10T17:32:22Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/pulls/445","html_url":"https://github.com/aic-collections/aicdams-lakeshore/pull/445","diff_url":"https://github.com/aic-collections/aicdams-lakeshore/pull/445.diff","patch_url":"https://github.com/aic-collections/aicdams-lakeshore/pull/445.patch"},"body":"# https://cits.artic.edu/redmine/issues/1686\r\n\r\n## QUESTIONS\r\n- [x] if we just add the postgres db location to the DuplicateUploadVerificationService will it cause problems? Yes b/c if the job checks the PG table for the checksum and it exists, the jobs won't perform.\r\n- [x] Where will we calculate checksums on new files? We should probably calculate the checksum on each of the uploaded_files inside some model.\r\n- [x] Where will we save the checksum string? We will save it in the `uploaded_files` table, under a new column called `checksum`.\r\n\r\n## SUPPORTING INFO/FACTS\r\n* currently, DuplicateUploadVerificationService is used in the following places: `CreateAssetJob`, `Sufia::UploadsController`, and `Lakeshore::IngestController`. The Job should check for a checksum in solr, and the UI and API controllers should check both Solr and the DB\r\n*  inside the batch_uploads controller (possibly a callback inside a AR model), calculate checksums on each uploaded_file and store in Postgres.\r\n* split into separate methods, calling the `.solr` class method from the Jobs, and calling the `.solr_and_db` from the controllers\r\n* calculate checksum as soon as UploadsController receives successful upload, and store in tmp so when batch is submitted, those checksums can be stored in DB, per Stefano, this way the web process won't take forever to calculate checksums on large batches and respond to the web client\r\n\r\n## Acceptance Criteria:\r\n- [x] When a user uploads a file that has already has begun ingestion (but *not* indexed in Solr with a checksum), they should receive a warning just like when a file has already been indexed.\r\n- [x] When a user uploads a file that does not exist in Solr, the status of the file should be \"submitted\"\r\n- [x] When an uploaded file is finally indexed in Solr with a checksum, that checksum should be cleared from the DB table.\r\n\r\n## TO DO:\r\n- [x] add a new column, of type :[string](https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb#L76), to the uploaded_files table\r\n- [x] on XHR upload, add a [callback](http://guides.rubyonrails.org/v4.0/active_record_callbacks.html#available-callbacks) to store checksum as [Digest::SHA1](http://ruby-doc.org/stdlib-2.4.0/libdoc/digest/rdoc/Digest/SHA1.html) the same string that ends up in Fedora, and Solr\r\n"},"comment":{"url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/issues/comments/380184119","html_url":"https://github.com/aic-collections/aicdams-lakeshore/pull/445#issuecomment-380184119","issue_url":"https://api.github.com/repos/aic-collections/aicdams-lakeshore/issues/445","id":380184119,"user":{"login":"RudyOnRails","id":1021154,"avatar_url":"https://avatars0.githubusercontent.com/u/1021154?v=4","gravatar_id":"","url":"https://api.github.com/users/RudyOnRails","html_url":"https://github.com/RudyOnRails","followers_url":"https://api.github.com/users/RudyOnRails/followers","following_url":"https://api.github.com/users/RudyOnRails/following{/other_user}","gists_url":"https://api.github.com/users/RudyOnRails/gists{/gist_id}","starred_url":"https://api.github.com/users/RudyOnRails/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RudyOnRails/subscriptions","organizations_url":"https://api.github.com/users/RudyOnRails/orgs","repos_url":"https://api.github.com/users/RudyOnRails/repos","events_url":"https://api.github.com/users/RudyOnRails/events{/privacy}","received_events_url":"https://api.github.com/users/RudyOnRails/received_events","type":"User","site_admin":false},"created_at":"2018-04-10T17:32:22Z","updated_at":"2018-04-10T17:32:22Z","author_association":"CONTRIBUTOR","body":"unit builds (PR and PUSH) were failing. grabbed travis log and restarted. here are failures so far:\r\n```\r\n\u001B[31m1535 examples, 15 failures, 6 pending\u001B[0m\r\n\r\nFailed examples:\r\n\r\n\u001B[31mrspec ./spec/controllers/sufia/uploads_controller_spec.rb:72\u001B[0m \u001B[36m# Sufia::UploadsController updating the use uri use_uri should eq #<RDF::Vocabulary::Term:0x41ff1a4 URI:http://definitions.artic.edu/ontology/1.0/type/IntermediateFileSet>\u001B[0m\r\n\u001B[31mrspec ./spec/jobs/attach_files_to_work_job_spec.rb:15\u001B[0m \u001B[36m# AttachFilesToWorkJob with each of the different use files attaches the files to the work and updates them\u001B[0m\r\n\u001B[31mrspec ./spec/actors/replace_file_actor_spec.rb:20\u001B[0m \u001B[36m# ReplaceFileActor an intermediate file when it is present replaces the content of the existing file set\u001B[0m\r\n\u001B[31mrspec ./spec/presenters/asset_presenter_spec.rb:101\u001B[0m \u001B[36m# AssetPresenter#publishable when false publishable should equal \"No\"\u001B[0m\r\n\u001B[31mrspec ./spec/presenters/asset_presenter_spec.rb:106\u001B[0m \u001B[36m# AssetPresenter#publishable when true publishable should equal \"Yes\"\u001B[0m\r\n\u001B[31mrspec ./spec/presenters/asset_presenter_spec.rb:94\u001B[0m \u001B[36m# AssetPresenter#public_domain when true public_domain should equal \"Yes\"\u001B[0m\r\n\u001B[31mrspec ./spec/presenters/asset_presenter_spec.rb:89\u001B[0m \u001B[36m# AssetPresenter#public_domain by default public_domain should equal \"No\"\u001B[0m\r\n\u001B[31mrspec ./spec/controllers/lakeshore/ingest_controller_spec.rb:263\u001B[0m \u001B[36m# Lakeshore::IngestController#update with no duplicates with an original file when one already exists replaces the file\u001B[0m\r\n\u001B[31mrspec ./spec/controllers/lakeshore/ingest_controller_spec.rb:252\u001B[0m \u001B[36m# Lakeshore::IngestController#update with no duplicates with an original file when none exists title \u001B[0m\r\n\u001B[31mrspec ./spec/controllers/lakeshore/ingest_controller_spec.rb:280\u001B[0m \u001B[36m# Lakeshore::IngestController#update with no duplicates with a preservation master file when none exists title \u001B[0m\r\n\u001B[31mrspec ./spec/controllers/lakeshore/ingest_controller_spec.rb:291\u001B[0m \u001B[36m# Lakeshore::IngestController#update with no duplicates with a preservation master file when one already exists replaces the file\u001B[0m\r\n\u001B[31mrspec ./spec/controllers/lakeshore/ingest_controller_spec.rb:183\u001B[0m \u001B[36m# Lakeshore::IngestController#update with no duplicates with an intermediate file when one does not exist title \u001B[0m\r\n\u001B[31mrspec ./spec/controllers/lakeshore/ingest_controller_spec.rb:209\u001B[0m \u001B[36m# Lakeshore::IngestController#update with no duplicates with an intermediate file when one already exists when the depositor is another user in the asset depositor's department changes the file_set's depositor\u001B[0m\r\n\u001B[31mrspec ./spec/controllers/lakeshore/ingest_controller_spec.rb:319\u001B[0m \u001B[36m# Lakeshore::IngestController#update with no duplicates with a legacy file when one already exists replaces the file\u001B[0m\r\n\u001B[31mrspec ./spec/controllers/lakeshore/ingest_controller_spec.rb:308\u001B[0m \u001B[36m# Lakeshore::IngestController#update with no duplicates with a legacy file when none exists title \u001B[0m\r\n```\r\n\r\nWill run locally. Travis failing and tests passing locally is nothing new, it's just been a pain and I'd like to document this for when we throw time at it and resolve it."}}
{"actor":{"display_login":"RudyOnRails"}}