{"action":"created","issue":{"url":"https://api.github.com/repos/Sp2000/colplus-backend/issues/61","repository_url":"https://api.github.com/repos/Sp2000/colplus-backend","labels_url":"https://api.github.com/repos/Sp2000/colplus-backend/issues/61/labels{/name}","comments_url":"https://api.github.com/repos/Sp2000/colplus-backend/issues/61/comments","events_url":"https://api.github.com/repos/Sp2000/colplus-backend/issues/61/events","html_url":"https://github.com/Sp2000/colplus-backend/issues/61","id":303202433,"node_id":"MDU6SXNzdWUzMDMyMDI0MzM=","number":61,"title":"Use Elastic Search for searches","user":{"login":"mdoering","id":327505,"node_id":"MDQ6VXNlcjMyNzUwNQ==","avatar_url":"https://avatars0.githubusercontent.com/u/327505?v=4","gravatar_id":"","url":"https://api.github.com/users/mdoering","html_url":"https://github.com/mdoering","followers_url":"https://api.github.com/users/mdoering/followers","following_url":"https://api.github.com/users/mdoering/following{/other_user}","gists_url":"https://api.github.com/users/mdoering/gists{/gist_id}","starred_url":"https://api.github.com/users/mdoering/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdoering/subscriptions","organizations_url":"https://api.github.com/users/mdoering/orgs","repos_url":"https://api.github.com/users/mdoering/repos","events_url":"https://api.github.com/users/mdoering/events{/privacy}","received_events_url":"https://api.github.com/users/mdoering/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"ayco-at-naturalis","id":7249727,"node_id":"MDQ6VXNlcjcyNDk3Mjc=","avatar_url":"https://avatars3.githubusercontent.com/u/7249727?v=4","gravatar_id":"","url":"https://api.github.com/users/ayco-at-naturalis","html_url":"https://github.com/ayco-at-naturalis","followers_url":"https://api.github.com/users/ayco-at-naturalis/followers","following_url":"https://api.github.com/users/ayco-at-naturalis/following{/other_user}","gists_url":"https://api.github.com/users/ayco-at-naturalis/gists{/gist_id}","starred_url":"https://api.github.com/users/ayco-at-naturalis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ayco-at-naturalis/subscriptions","organizations_url":"https://api.github.com/users/ayco-at-naturalis/orgs","repos_url":"https://api.github.com/users/ayco-at-naturalis/repos","events_url":"https://api.github.com/users/ayco-at-naturalis/events{/privacy}","received_events_url":"https://api.github.com/users/ayco-at-naturalis/received_events","type":"User","site_admin":false},"assignees":[{"login":"ayco-at-naturalis","id":7249727,"node_id":"MDQ6VXNlcjcyNDk3Mjc=","avatar_url":"https://avatars3.githubusercontent.com/u/7249727?v=4","gravatar_id":"","url":"https://api.github.com/users/ayco-at-naturalis","html_url":"https://github.com/ayco-at-naturalis","followers_url":"https://api.github.com/users/ayco-at-naturalis/followers","following_url":"https://api.github.com/users/ayco-at-naturalis/following{/other_user}","gists_url":"https://api.github.com/users/ayco-at-naturalis/gists{/gist_id}","starred_url":"https://api.github.com/users/ayco-at-naturalis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ayco-at-naturalis/subscriptions","organizations_url":"https://api.github.com/users/ayco-at-naturalis/orgs","repos_url":"https://api.github.com/users/ayco-at-naturalis/repos","events_url":"https://api.github.com/users/ayco-at-naturalis/events{/privacy}","received_events_url":"https://api.github.com/users/ayco-at-naturalis/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2018-03-07T18:05:14Z","updated_at":"2018-09-05T07:56:54Z","closed_at":null,"author_association":"COLLABORATOR","body":"Use Elastic search for searching, removing the performance bottleneck #57 .\r\nConsider how to best expose facets and counts. Should we expose ES directly or wrap it in our API?\r\n\r\nThe importer needs to update ES for each indexed dataset. But there should also be a backfill method that regenerates the entire index for desaster recovery and updating an out of sync index  e.g. because of network outages. A DW admin command that reindexes the entire database or just a single dataset is needed to manage ES data.\r\n\r\nFor testing an embedded ES should be used when testing populating and querying the index. When running other tests not directly using searches it should be doable to swap out the implementation for a mock that doesn't do anything. A new ElasticConfig would be good that allows to specify the ES location (if null use a mock) and any other needed settings. "},"comment":{"url":"https://api.github.com/repos/Sp2000/colplus-backend/issues/comments/418635794","html_url":"https://github.com/Sp2000/colplus-backend/issues/61#issuecomment-418635794","issue_url":"https://api.github.com/repos/Sp2000/colplus-backend/issues/61","id":418635794,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODYzNTc5NA==","user":{"login":"mdoering","id":327505,"node_id":"MDQ6VXNlcjMyNzUwNQ==","avatar_url":"https://avatars0.githubusercontent.com/u/327505?v=4","gravatar_id":"","url":"https://api.github.com/users/mdoering","html_url":"https://github.com/mdoering","followers_url":"https://api.github.com/users/mdoering/followers","following_url":"https://api.github.com/users/mdoering/following{/other_user}","gists_url":"https://api.github.com/users/mdoering/gists{/gist_id}","starred_url":"https://api.github.com/users/mdoering/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdoering/subscriptions","organizations_url":"https://api.github.com/users/mdoering/orgs","repos_url":"https://api.github.com/users/mdoering/repos","events_url":"https://api.github.com/users/mdoering/events{/privacy}","received_events_url":"https://api.github.com/users/mdoering/received_events","type":"User","site_admin":false},"created_at":"2018-09-05T07:56:54Z","updated_at":"2018-09-05T07:56:54Z","author_association":"COLLABORATOR","body":"the neo4j store is not read-only, but I would go for postgres as the source:\r\n\r\n1. mybatis mapper for NameUsage instances exists already and its not trivial to map these complex objects\r\n2. surrogate keys dont exist in neo yet (can be changed though)\r\n3. rebuilding the entire ES index would require the copies matching the current sql of neo4j to persist and opening/closing of a store per dataset. When we start a new import the old neo4j store gets removed\r\n4. unclear how neo4j handles parallel, multithreaded reads and transactional locks\r\n5. postgres is partitioned and we can read the specific partition tables directly not interferring with the aggregated table. The partition can even be attached to the main table after ES has been build so its not yet visible to the public, full queries\r\n\r\nBuilding the ES index sequentially will obviously stretch the entire import process considerably, but we can parallize entire imports more to counter that. Removing the full text search index from the names table will also slightly speed up the sql import. \r\n"}}
{"actor":{"display_login":"mdoering"}}