{"action":"created","issue":{"url":"https://api.github.com/repos/Zygo/bees/issues/44","repository_url":"https://api.github.com/repos/Zygo/bees","labels_url":"https://api.github.com/repos/Zygo/bees/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/Zygo/bees/issues/44/comments","events_url":"https://api.github.com/repos/Zygo/bees/issues/44/events","html_url":"https://github.com/Zygo/bees/issues/44","id":287577770,"node_id":"MDU6SXNzdWUyODc1Nzc3NzA=","number":44,"title":"Problems with handle beeshash.dat unaligned by 16MiB","user":{"login":"Nefelim4ag","id":3943460,"node_id":"MDQ6VXNlcjM5NDM0NjA=","avatar_url":"https://avatars1.githubusercontent.com/u/3943460?v=4","gravatar_id":"","url":"https://api.github.com/users/Nefelim4ag","html_url":"https://github.com/Nefelim4ag","followers_url":"https://api.github.com/users/Nefelim4ag/followers","following_url":"https://api.github.com/users/Nefelim4ag/following{/other_user}","gists_url":"https://api.github.com/users/Nefelim4ag/gists{/gist_id}","starred_url":"https://api.github.com/users/Nefelim4ag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Nefelim4ag/subscriptions","organizations_url":"https://api.github.com/users/Nefelim4ag/orgs","repos_url":"https://api.github.com/users/Nefelim4ag/repos","events_url":"https://api.github.com/users/Nefelim4ag/events{/privacy}","received_events_url":"https://api.github.com/users/Nefelim4ag/received_events","type":"User","site_admin":false},"labels":[{"id":949508776,"node_id":"MDU6TGFiZWw5NDk1MDg3NzY=","url":"https://api.github.com/repos/Zygo/bees/labels/roadmap","name":"roadmap","color":"c5ea6e","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-01-10T21:08:23Z","updated_at":"2018-09-27T15:44:12Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Hi, i've try try test how bees behave if db size are unaligned (256MiB -> 257MiB),\r\nand if i understood correctly the code, bees must die/crash on that.\r\n\r\n(Sorry i'm have a problems with C++ magic, that happens in bees code).\r\n\r\nBut i see, so i think something broken,\r\nat least that it lie that problem happens with beesstats.txt:\r\n```\r\nbees version v0.5-11-gd5bcdae\r\n2018-01-11 00:01:54 22494.22494 main: set_root_path /mnt/\r\n2018-01-11 00:01:54 22494.22494 main: set_root_fd /mnt\r\n2018-01-11 00:01:54 22494.22494 main: Filesystem UUID is 00000000-0000-0000-0000-000000004321\r\n2018-01-11 00:01:54 22494.22494 main: BeesStringFile /mnt/.beeshome/beescrawl.dat max size 1M\r\n2018-01-11 00:01:54 22494.22494 main: BeesThread exec crawl\r\n2018-01-11 00:01:54 22494.22494 main: returning from set_root_fd in /mnt\r\n2018-01-11 00:01:54 22494.22494 main: BeesThread construct status\r\n2018-01-11 00:01:54 22494.22494 main: BeesThread exec status\r\n2018-01-11 00:01:54 22494.22495 bees: Starting thread crawl\r\n2018-01-11 00:01:54 22494.22495 crawl: loading crawl state\r\n2018-01-11 00:01:54 22494.22496 bees: Starting thread status\r\n2018-01-11 00:01:54 22494.22496 status: Exiting thread status, 0.001 sec\r\n2018-01-11 00:01:54 22494.22495 crawl: Read line: root 5 objectid 18446744073709551605 offset 1 min_transid 685382 max_transid 709570 started 1515618040 start_ts 2018-01-11-00-00-40\r\n2018-01-11 00:01:54 22494.22495 crawl: loaded_state BeesCrawlState 5:18446744073709551605 offset 0x1 transid 685382..709570 started 2018-01-11-00-00-40 (74s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Read line: root 1092 objectid 48569 offset 1 min_transid 679441 max_transid 681131 started 1515520058 start_ts 2018-01-09-20-47-38\r\n2018-01-11 00:01:54 22494.22495 crawl: loaded_state BeesCrawlState 1092:48569 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Read line: root 1093 objectid 324440 offset 1192 min_transid 585545 max_transid 674792 started 1515518238 start_ts 2018-01-09-20-17-18\r\n2018-01-11 00:01:54 22494.22495 crawl: loaded_state BeesCrawlState 1093:324440 offset 0x4a8 transid 585545..674792 started 2018-01-09-20-17-18 (99876s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Read line: root 1094 objectid 118686 offset 73138177 min_transid 0 max_transid 585545 started 1515392468 start_ts 2018-01-08-09-21-08\r\n2018-01-11 00:01:54 22494.22495 crawl: loaded_state BeesCrawlState 1094:118686 offset 0x45c0001 transid 0..585545 started 2018-01-08-09-21-08 (225646s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Read line: root 1140 objectid 293 offset 1 min_transid 679441 max_transid 681131 started 1515520058 start_ts 2018-01-09-20-47-38\r\n2018-01-11 00:01:54 22494.22495 crawl: loaded_state BeesCrawlState 1140:293 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: BeesThread exec crawl_writeback\r\n2018-01-11 00:01:54 22494.22497 bees: Starting thread crawl_writeback\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 5:18446744073709551605 offset 0x1 transid 685382..709570 started 2018-01-11-00-00-40 (74s ago)\r\n2018-01-11 00:01:54 22494.22497 crawl_writeback: open_root_nocache 1092: /mnt/@ccache\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawl empty BeesCrawlState 5:18446744073709551605 offset 0x1 transid 685382..709570 started 2018-01-11-00-00-40 (74s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Deferring next transid in BeesCrawlState 5:18446744073709551605 offset 0x1 transid 685382..709570 started 2018-01-11-00-00-40 (74s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1092:48569 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22497 crawl_writeback: Saving crawl state\r\n2018-01-11 00:01:54 22494.22497 crawl_writeback: Saved crawl state in 0.001s\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawl empty BeesCrawlState 1092:48569 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Next transid in BeesCrawlState 1092:48569 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Restarting crawl BeesCrawlState 1092:48569 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1092:0 offset 0x0 transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling 3473 results from BeesCrawlState 1092:0 offset 0x0 transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawled inline 37 data 0 other 3436 unknown 0 gen_low 37 gen_high 0 BeesCrawlState 1092:440 offset 0x5c transid 681131..709598 started 2018-01-11-00-01-54 (0s ago) in 0.006s\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1092:440 offset 0x5c transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling 3116 results from BeesCrawlState 1092:440 offset 0x5c transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawled inline 30 data 0 other 3086 unknown 0 gen_low 30 gen_high 0 BeesCrawlState 1092:591 offset 0x13348cb8 transid 681131..709598 started 2018-01-11-00-01-54 (0s ago) in 0.005s\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1092:591 offset 0x13348cb8 transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling 3285 results from BeesCrawlState 1092:591 offset 0x13348cb8 transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawled inline 34 data 0 other 3251 unknown 0 gen_low 34 gen_high 0 BeesCrawlState 1092:795 offset 0xff7462b transid 681131..709598 started 2018-01-11-00-01-54 (0s ago) in 0.005s\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1092:795 offset 0xff7462b transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling 3069 results from BeesCrawlState 1092:795 offset 0xff7462b transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawled inline 37 data 5 other 3027 unknown 0 gen_low 42 gen_high 0 BeesCrawlState 1092:1049 offset 0x57c350f transid 681131..709598 started 2018-01-11-00-01-54 (0s ago) in 0.004s\r\n2018-01-11 00:01:54 22494.22495 crawl: BeesStringFile /mnt/.beeshome/beesstats.txt max size 1M\r\n2018-01-11 00:01:54 22494.22495 crawl: Thread hash_prefetch no thread ptr\r\n2018-01-11 00:01:54 22494.22495 crawl: Thread hash_writeback no thread ptr\r\n2018-01-11 00:01:54 22494.22495 crawl: scan extent Extent { begin = 0x0, end = 0x148e, physical = 0x43a3bba000, flags = FIEMAP_EXTENT_LAST, physical_len = 0x2000, logical_len = 0x2000 }\r\n2018-01-11 00:01:54 22494.22495 crawl: \r\n2018-01-11 00:01:54 22494.22495 crawl: \r\n2018-01-11 00:01:54 22494.22495 crawl: *** EXCEPTION ***\r\n2018-01-11 00:01:54 22494.22495 crawl:  exception type std::invalid_argument: new_size = 270532608 failed constraint check ((new_size % BLOCK_SIZE_HASHTAB_EXTENT) == 0)\r\n2018-01-11 00:01:54 22494.22495 crawl: ***\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1093:324440 offset 0x4a8 transid 585545..674792 started 2018-01-09-20-17-18 (99876s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling 2796 results from BeesCrawlState 1093:324440 offset 0x4a8 transid 585545..674792 started 2018-01-09-20-17-18 (99876s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawled inline 220 data 882 other 1694 unknown 0 gen_low 1098 gen_high 0 BeesCrawlState 1093:330874 offset 0x50abd transid 585545..674792 started 2018-01-09-20-17-18 (99876s ago) in 0.011s\r\n2018-01-11 00:01:54 22494.22495 crawl: BeesStringFile /mnt/.beeshome/beesstats.txt max size 1M\r\n2018-01-11 00:01:54 22494.22495 crawl: Thread hash_prefetch no thread ptr\r\n2018-01-11 00:01:54 22494.22495 crawl: Thread hash_writeback no thread ptr\r\n2018-01-11 00:01:54 22494.22495 crawl: scan extent Extent { begin = 0x0, end = 0xd12, physical = 0x5c3a604000, flags = FIEMAP_EXTENT_LAST, physical_len = 0x1000, logical_len = 0x1000 }\r\n2018-01-11 00:01:54 22494.22495 crawl: \r\n2018-01-11 00:01:54 22494.22495 crawl: \r\n2018-01-11 00:01:54 22494.22495 crawl: *** EXCEPTION ***\r\n2018-01-11 00:01:54 22494.22495 crawl:  exception type std::invalid_argument: new_size = 270532608 failed constraint check ((new_size % BLOCK_SIZE_HASHTAB_EXTENT) == 0)\r\n2018-01-11 00:01:54 22494.22495 crawl: ***\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1094:118686 offset 0x45c0001 transid 0..585545 started 2018-01-08-09-21-08 (225646s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling 4092 results from BeesCrawlState 1094:118686 offset 0x45c0001 transid 0..585545 started 2018-01-08-09-21-08 (225646s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawled inline 0 data 4084 other 8 unknown 0 gen_low 0 gen_high 0 BeesCrawlState 1094:118692 offset 0x4ce0001 transid 0..585545 started 2018-01-08-09-21-08 (225646s ago) in 0.008s\r\n2018-01-11 00:01:54 22494.22495 crawl: BeesStringFile /mnt/.beeshome/beesstats.txt max size 1M\r\n2018-01-11 00:01:54 22494.22495 crawl: Thread hash_prefetch no thread ptr\r\n2018-01-11 00:01:54 22494.22495 crawl: Thread hash_writeback no thread ptr\r\n2018-01-11 00:01:54 22494.22495 crawl: scan extent Extent { begin = 0xec0000, end = 0xee0000, physical = 0x5439992000, flags = FIEMAP_EXTENT_ENCODED, physical_len = 0x20000, logical_len = 0x20000 }\r\n2018-01-11 00:01:54 22494.22495 crawl: \r\n2018-01-11 00:01:54 22494.22495 crawl: \r\n2018-01-11 00:01:54 22494.22495 crawl: *** EXCEPTION ***\r\n2018-01-11 00:01:54 22494.22495 crawl:  exception type std::invalid_argument: new_size = 270532608 failed constraint check ((new_size % BLOCK_SIZE_HASHTAB_EXTENT) == 0)\r\n2018-01-11 00:01:54 22494.22495 crawl: ***\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1140:293 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawl empty BeesCrawlState 1140:293 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Next transid in BeesCrawlState 1140:293 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Restarting crawl BeesCrawlState 1140:293 offset 0x1 transid 679441..681131 started 2018-01-09-20-47-38 (98056s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling BeesCrawlState 1140:0 offset 0x0 transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawling 1863 results from BeesCrawlState 1140:0 offset 0x0 transid 681131..709598 started 2018-01-11-00-01-54 (0s ago)\r\n2018-01-11 00:01:54 22494.22495 crawl: Crawled inline 0 data 1863 other 0 unknown 0 gen_low 0 gen_high 0 BeesCrawlState 1140:257 offset 0xf480001 transid 681131..709598 started 2018-01-11-00-01-54 (0s ago) in 0.003s\r\n2018-01-11 00:01:54 22494.22495 crawl: BeesStringFile /mnt/.beeshome/beesstats.txt max size 1M\r\n2018-01-11 00:01:54 22494.22495 crawl: Thread hash_prefetch no thread ptr\r\n2018-01-11 00:01:54 22494.22495 crawl: Thread hash_writeback no thread ptr\r\n2018-01-11 00:01:54 22494.22495 crawl: scan extent Extent { begin = 0xbc0000, end = 0xbe0000, physical = 0x45c7061000, flags = FIEMAP_EXTENT_ENCODED, physical_len = 0x20000, logical_len = 0x20000 }\r\n\r\n*Many many similar lines*\r\n```"},"comment":{"url":"https://api.github.com/repos/Zygo/bees/issues/comments/425142347","html_url":"https://github.com/Zygo/bees/issues/44#issuecomment-425142347","issue_url":"https://api.github.com/repos/Zygo/bees/issues/44","id":425142347,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTE0MjM0Nw==","user":{"login":"Zygo","id":1883298,"node_id":"MDQ6VXNlcjE4ODMyOTg=","avatar_url":"https://avatars0.githubusercontent.com/u/1883298?v=4","gravatar_id":"","url":"https://api.github.com/users/Zygo","html_url":"https://github.com/Zygo","followers_url":"https://api.github.com/users/Zygo/followers","following_url":"https://api.github.com/users/Zygo/following{/other_user}","gists_url":"https://api.github.com/users/Zygo/gists{/gist_id}","starred_url":"https://api.github.com/users/Zygo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zygo/subscriptions","organizations_url":"https://api.github.com/users/Zygo/orgs","repos_url":"https://api.github.com/users/Zygo/repos","events_url":"https://api.github.com/users/Zygo/events{/privacy}","received_events_url":"https://api.github.com/users/Zygo/received_events","type":"User","site_admin":false},"created_at":"2018-09-27T15:44:12Z","updated_at":"2018-09-27T15:44:12Z","author_association":"OWNER","body":"It seems we will need to support incompatible hash table format changes (see https://github.com/Zygo/bees/issues/33#issuecomment-425136474).  Changes to hash table bucket size (assuming the same hash algorithm) can be achieved the same way as a hash table resize.\r\n\r\nAdd to the above:  \r\n\r\n  * when a new hash table is written due to a size, algorithm, or layout change, store some format information in the table (e.g. store `0x4245455300000000 | format_info_bits` in the first cell's hash field, which is not a possible value for that field in the current format).\r\n\r\n  * change the size alignment to 128KB instead of 16M, or make it configurable.  128KB is the largest useful size for hash table storage on compressed filesystems.  The hash table is usually compressible because adjacent hash values are similar (20+ bits in common) and there are a lot of zeros in block numbers."}}
{"actor":{"display_login":"Zygo"}}