{"action":"created","issue":{"url":"https://api.github.com/repos/blevesearch/bleve/issues/1052","repository_url":"https://api.github.com/repos/blevesearch/bleve","labels_url":"https://api.github.com/repos/blevesearch/bleve/issues/1052/labels{/name}","comments_url":"https://api.github.com/repos/blevesearch/bleve/issues/1052/comments","events_url":"https://api.github.com/repos/blevesearch/bleve/issues/1052/events","html_url":"https://github.com/blevesearch/bleve/issues/1052","id":383310610,"node_id":"MDU6SXNzdWUzODMzMTA2MTA=","number":1052,"title":"Provide example on how to force a manual merge to flatten an index","user":{"login":"fproulx-eoscanada","id":42417098,"node_id":"MDQ6VXNlcjQyNDE3MDk4","avatar_url":"https://avatars3.githubusercontent.com/u/42417098?v=4","gravatar_id":"","url":"https://api.github.com/users/fproulx-eoscanada","html_url":"https://github.com/fproulx-eoscanada","followers_url":"https://api.github.com/users/fproulx-eoscanada/followers","following_url":"https://api.github.com/users/fproulx-eoscanada/following{/other_user}","gists_url":"https://api.github.com/users/fproulx-eoscanada/gists{/gist_id}","starred_url":"https://api.github.com/users/fproulx-eoscanada/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fproulx-eoscanada/subscriptions","organizations_url":"https://api.github.com/users/fproulx-eoscanada/orgs","repos_url":"https://api.github.com/users/fproulx-eoscanada/repos","events_url":"https://api.github.com/users/fproulx-eoscanada/events{/privacy}","received_events_url":"https://api.github.com/users/fproulx-eoscanada/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-21T22:16:37Z","updated_at":"2018-11-24T14:03:28Z","closed_at":null,"author_association":"NONE","body":"Hi Blevers,\r\n\r\nFirst of all thank you for this amazing library.\r\n\r\nWe've been experimenting with Bleve to index large amount of data (we don't need to store it, we have an efficient raw storage), we create small documents with attributes to do conjunctive searches. We have on the order of 1 million document per hour, each with about a dozen attributes to index on.\r\n\r\nWe index by batches covering about 1 hour worth of documents and in some cases we get 20 GB index on disk, other hours may be much much smaller. Anyhow, once we close an hour-worth index, we re-open in read-only and offer for search. \r\n\r\nI would appreciate some guidance on merging those, or at least force optimizing a given range (for instance it seems excessive to have 600 zap files, the vast majority being less than 50 KB, other being like 8 GB)\r\n\r\nAlso, it appears that when opening such large index there is a lag (dozens of seconds) before the first query returns, after it is warmed up it's much much faster (microseconds) to do the same or similar, but not identical queries. Any advice on improving the cold start without necessarily issuing a bogus test query.\r\n\r\nI have tried to setup a custom `scorchMergePlanOptions` when opening the index, but even though I confirmed that it loaded and validated it it does not perform any further merging (I'm doing that on the read-only, immutable index).  Do you have any recommendation of `scorchMergePlanOptions` values. We can trade off using more memory and/or disk during indexing, as long as the final index (immutable for read-only use) is fast and compact.\r\n\r\nThanks for your hard work,"},"comment":{"url":"https://api.github.com/repos/blevesearch/bleve/issues/comments/441370127","html_url":"https://github.com/blevesearch/bleve/issues/1052#issuecomment-441370127","issue_url":"https://api.github.com/repos/blevesearch/bleve/issues/1052","id":441370127,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTM3MDEyNw==","user":{"login":"sreekanth-cb","id":25240115,"node_id":"MDQ6VXNlcjI1MjQwMTE1","avatar_url":"https://avatars1.githubusercontent.com/u/25240115?v=4","gravatar_id":"","url":"https://api.github.com/users/sreekanth-cb","html_url":"https://github.com/sreekanth-cb","followers_url":"https://api.github.com/users/sreekanth-cb/followers","following_url":"https://api.github.com/users/sreekanth-cb/following{/other_user}","gists_url":"https://api.github.com/users/sreekanth-cb/gists{/gist_id}","starred_url":"https://api.github.com/users/sreekanth-cb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sreekanth-cb/subscriptions","organizations_url":"https://api.github.com/users/sreekanth-cb/orgs","repos_url":"https://api.github.com/users/sreekanth-cb/repos","events_url":"https://api.github.com/users/sreekanth-cb/events{/privacy}","received_events_url":"https://api.github.com/users/sreekanth-cb/received_events","type":"User","site_admin":false},"created_at":"2018-11-24T14:03:28Z","updated_at":"2018-11-24T14:03:28Z","author_association":"CONTRIBUTOR","body":"Hi @fproulx-eoscanada ,\r\n\r\nTo answer this query, there are many areas to touch upon. Not sure how effective I will be able to cover that.\r\nLet me start with the number of files wrinkle,\r\n\r\nDuring indexing, the number of files on disk is controlled by the merge policy as you \r\nfigured out already. There are a few test cases which verifies the number of files expected for a \r\ngiven number of documents. \r\nYou may refer here: https://github.com/blevesearch/bleve/blob/scorch/index/scorch/mergeplan/merge_plan_test.go\r\n\r\nIf you play around with that test to feed it with the number of documents at hand, \r\nyou should be able to find the expected number of files on disk for a given number of documents, once the\r\nindexing is complete.\r\n\r\nFor eg: 1 M document will result in 24 files on disk.\r\n\r\n\r\nFrom a higher level, the persister routine in the system keeps writing segment files to disk and the merger keep merging those files to lesser number of compact and bigger files. \r\nAnd the current progress of those routines can be tracked over the stats like LastPersistedEpoch Vs LastMergedEpoch. \r\nThe bigger the difference between those two stats, the higher would be number of files on disk \r\nthan that is defined by the merge policy. Usually once the indexing is over, the merger will slowly catch up with the persister to result in ideal number of files on disk.\r\n\r\nAs you mentioned that, bleve isn't used as the primary data store in your case,\r\nthere is another possibility here.\r\n\r\nBleve has an option of unsafeBatch as here: \r\nhttps://github.com/blevesearch/bleve/blob/695c0fd368f8d0a5eaeb3400b06c7cce78ef2a70/index/scorch/scorch.go#L52\r\n\r\nThis would let the batch execute calls to return before the persistence of the data. Certainly this batch data would be searchable, just that its not yet made to the disk.\r\n\r\nYou could parameterise this:\r\nhttps://github.com/blevesearch/bleve/blob/695c0fd368f8d0a5eaeb3400b06c7cce78ef2a70/index/scorch/scorch.go#L111\r\n\r\nThis option gives scorch more opportunity to perform in memory merging of segments/(batches) and would result in lesser number of files getting introduced to disk. \r\n\r\nAnd you may also tune these two options here\r\nhttps://github.com/blevesearch/bleve/blob/695c0fd368f8d0a5eaeb3400b06c7cce78ef2a70/index/scorch/persister.go#L50\r\nhttps://github.com/blevesearch/bleve/blob/695c0fd368f8d0a5eaeb3400b06c7cce78ef2a70/index/scorch/persister.go#L60\r\n\r\nto control the persister merger dancing better to control the overall system behaviour. ;-) \r\n(You may experiment with these configs even in the default safe batch mode as well)\r\n\r\n\r\nBut the above mentioned unsafeBatch option inherently has higher memory requirement and you may just start stressing your system in that direction. (RAM).  This demands the application to control the indexing flow/batch execution depending on the memory usage of bleve.\r\n\r\nThe theory here is that, if we stop/pause indexing, then all the already indexed (non persisted) memory segments will go to disk and memory will become available for further indexing. \r\nSo, if bleve/scorch is using too much memory, one thing we can do is stop/pause indexing from application's side.\r\n\r\nTo do this we can use a callback in scorch, which fires for key events. Setting this up is a bit weird in bleve because our configuration data is persisted, but callback functions are ephemeral at runtime, so we use a registry with string identifiers. \r\n\r\nYou can see how this is processed here:\r\nhttps://github.com/blevesearch/bleve/blob/master/index/scorch/scorch.go#L113\r\n\r\nAnd you can see how an application uses it by registering a name/callback here:\r\nhttps://github.com/couchbase/cbft/blob/1bcec5832c6d1ad1e73a83acc5be3f548479a4a0/cmd/cbft/init_bleve.go#L81\r\n\r\nAnd configuring bleve to use it here:\r\nhttps://github.com/couchbase/cbft/blob/a33ad7b7000a9d8d237ba273c47cc100401a0fb0/pindex_bleve.go#L526\r\n\r\nSo, how does this callback function work? Basically there is an event fired at the start of every batch that is about to be indexed.  Inside this event, we can check how much memory bleve is using and if necessary block at this point. \r\nHow long should we block? Fortunately, there are other scorch events which can tell us when the index persister has made progress (wrote data to disk) which is a useful signal that we may be using less memory now and should check again.\r\n\r\nHow do we know much memory Bleve is using? It's a tedious job to count this manually. One option is start accounting the heap usage from Go runtime. You may play around with the Go runtime stats to figure this out.\r\nAnd if you are interested in exploring this option further, please check the above cbft package which follows the unsafeBatch approach and all the above mentioned indexing flow controls based on bleve callbacks.\r\n\r\n-Once the number of files are within the defined range, I guess the cold start problem should also alleviate?\r\n\r\n-There are no other direct manual way to merge/compact the already existing zap files on disk.\r\n\r\n-Playing around with the size of batches also helps in certain cases. Whats the average batch (no of documents) in your case.\r\n\r\ncheers!"}}
{"actor":{"display_login":"sreekanth-cb"}}