{"action":"created","issue":{"url":"https://api.github.com/repos/yandex/ClickHouse/issues/3614","repository_url":"https://api.github.com/repos/yandex/ClickHouse","labels_url":"https://api.github.com/repos/yandex/ClickHouse/issues/3614/labels{/name}","comments_url":"https://api.github.com/repos/yandex/ClickHouse/issues/3614/comments","events_url":"https://api.github.com/repos/yandex/ClickHouse/issues/3614/events","html_url":"https://github.com/yandex/ClickHouse/issues/3614","id":382483240,"node_id":"MDU6SXNzdWUzODI0ODMyNDA=","number":3614,"title":"DB::Exception: Allocator: Cannot mmap 8.00 EiB. (Distributed / LowCardinality / Nullable)","user":{"login":"champtar","id":3536755,"node_id":"MDQ6VXNlcjM1MzY3NTU=","avatar_url":"https://avatars1.githubusercontent.com/u/3536755?v=4","gravatar_id":"","url":"https://api.github.com/users/champtar","html_url":"https://github.com/champtar","followers_url":"https://api.github.com/users/champtar/followers","following_url":"https://api.github.com/users/champtar/following{/other_user}","gists_url":"https://api.github.com/users/champtar/gists{/gist_id}","starred_url":"https://api.github.com/users/champtar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/champtar/subscriptions","organizations_url":"https://api.github.com/users/champtar/orgs","repos_url":"https://api.github.com/users/champtar/repos","events_url":"https://api.github.com/users/champtar/events{/privacy}","received_events_url":"https://api.github.com/users/champtar/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"KochetovNicolai","id":4092911,"node_id":"MDQ6VXNlcjQwOTI5MTE=","avatar_url":"https://avatars1.githubusercontent.com/u/4092911?v=4","gravatar_id":"","url":"https://api.github.com/users/KochetovNicolai","html_url":"https://github.com/KochetovNicolai","followers_url":"https://api.github.com/users/KochetovNicolai/followers","following_url":"https://api.github.com/users/KochetovNicolai/following{/other_user}","gists_url":"https://api.github.com/users/KochetovNicolai/gists{/gist_id}","starred_url":"https://api.github.com/users/KochetovNicolai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KochetovNicolai/subscriptions","organizations_url":"https://api.github.com/users/KochetovNicolai/orgs","repos_url":"https://api.github.com/users/KochetovNicolai/repos","events_url":"https://api.github.com/users/KochetovNicolai/events{/privacy}","received_events_url":"https://api.github.com/users/KochetovNicolai/received_events","type":"User","site_admin":false},"assignees":[{"login":"KochetovNicolai","id":4092911,"node_id":"MDQ6VXNlcjQwOTI5MTE=","avatar_url":"https://avatars1.githubusercontent.com/u/4092911?v=4","gravatar_id":"","url":"https://api.github.com/users/KochetovNicolai","html_url":"https://github.com/KochetovNicolai","followers_url":"https://api.github.com/users/KochetovNicolai/followers","following_url":"https://api.github.com/users/KochetovNicolai/following{/other_user}","gists_url":"https://api.github.com/users/KochetovNicolai/gists{/gist_id}","starred_url":"https://api.github.com/users/KochetovNicolai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KochetovNicolai/subscriptions","organizations_url":"https://api.github.com/users/KochetovNicolai/orgs","repos_url":"https://api.github.com/users/KochetovNicolai/repos","events_url":"https://api.github.com/users/KochetovNicolai/events{/privacy}","received_events_url":"https://api.github.com/users/KochetovNicolai/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-11-20T03:20:22Z","updated_at":"2018-11-22T20:17:04Z","closed_at":"2018-11-22T20:17:04Z","author_association":"CONTRIBUTOR","body":"Hi Clickhouse Team,\r\n\r\nI'm running e3b0cecdec24d63f7abf5db5b847b33fdcf3452d, which include #3487 \r\n\r\n```\r\nSELECT \r\n    geoip_latitude AS latitude, \r\n    geoip_longitude AS longitude, \r\n    dictGetString('geoname', 'city_name', toUInt64(geoip_geoname_id)) AS city, \r\n    uniq(clientip) AS uniqueip\r\nFROM analytics.vme1 \r\nGROUP BY \r\n    latitude, \r\n    longitude, \r\n    city\r\n\r\n↖ Progress: 1.45 billion rows, 33.17 GB (75.32 million rows/s., 1.72 GB/s.) ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ 98%\r\nReceived exception from server (version 18.14.51):\r\nCode: 0. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Allocator: Cannot mmap 8.00 EiB., errno: 12, strerror: Cannot allocate memory. \r\n```\r\n\r\n```\r\n2018.11.20 03:13:35.679001 [ 47 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> InterpreterSelectQuery: WithMergeableState -> Complete\r\n2018.11.20 03:13:35.679191 [ 47 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Debug> executeQuery: Query pipeline:\r\nExpression\r\n Expression\r\n  MergingAggregated\r\n   Union\r\n    Remote × 3\r\n\r\n2018.11.20 03:13:35.679595 [ 58 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Aggregator: Reading blocks of partially aggregated data.\r\n2018.11.20 03:13:35.679977 [ 60 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Connection (172.27.209.172:9000): Connecting. Database: analytics_shard_2. User: distributed\r\n2018.11.20 03:13:35.679973 [ 61 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Connection (172.27.209.173:9000): Connecting. Database: analytics_shard_3. User: distributed\r\n2018.11.20 03:13:35.679996 [ 59 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Connection (172.27.209.171:9000): Connecting. Database: analytics_shard_1. User: distributed\r\n2018.11.20 03:13:35.680550 [ 48 ] {} <Trace> TCPHandlerFactory: TCP Request. Address: [::ffff:172.27.209.173]:42138\r\n2018.11.20 03:13:35.680726 [ 48 ] {} <Debug> TCPHandler: Connected ClickHouse server version 18.14.0, revision: 54409, database: analytics_shard_3, user: distributed.\r\n2018.11.20 03:13:35.681003 [ 61 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Connection (172.27.209.173:9000): Connected to ClickHouse server version 18.14.51.\r\n2018.11.20 03:13:35.681774 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Debug> executeQuery: (from [::ffff:172.27.209.173]:42138, user: distributed, initial_query_id: fd14e035-b9b7-4bff-8a0a-72cdca6f9d47) SELECT geoip_latitude AS latitude, geoip_longitude AS longitude, dictGetString('geoname', 'city_name', toUInt64(geoip_geoname_id)) AS city, uniq(clientip) AS uniqueip FROM vme1  GROUP BY latitude, longitude, city\r\n2018.11.20 03:13:35.683014 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Debug> analytics_shard_3.vme1 (SelectExecutor): Key condition: unknown\r\n2018.11.20 03:13:35.683107 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Debug> analytics_shard_3.vme1 (SelectExecutor): MinMax index condition: unknown\r\n2018.11.20 03:13:35.683158 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Debug> analytics_shard_3.vme1 (SelectExecutor): Selected 19 parts by date, 19 parts by key, 59189 marks to read from 19 ranges\r\n2018.11.20 03:13:35.685235 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> analytics_shard_3.vme1 (SelectExecutor): Reading approx. 484876288 rows with 8 streams\r\n2018.11.20 03:13:35.685488 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> InterpreterSelectQuery: FetchColumns -> WithMergeableState\r\n2018.11.20 03:13:35.685944 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Debug> executeQuery: Query pipeline:\r\nParallelAggregating\r\n Expression × 8\r\n  MergeTreeThread\r\n\r\n2018.11.20 03:13:35.686205 [ 59 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Connection (172.27.209.171:9000): Connected to ClickHouse server version 18.14.51.\r\n2018.11.20 03:13:35.686461 [ 60 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Connection (172.27.209.172:9000): Connected to ClickHouse server version 18.14.51.\r\n2018.11.20 03:13:35.686673 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregating\r\n2018.11.20 03:13:35.729611 [ 67 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Aggregation method: serialized\r\n2018.11.20 03:13:35.733235 [ 63 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Aggregation method: serialized\r\n2018.11.20 03:13:35.734032 [ 66 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Aggregation method: serialized\r\n2018.11.20 03:13:35.737261 [ 64 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Aggregation method: serialized\r\n2018.11.20 03:13:35.762267 [ 67 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Converting aggregation data to two-level.\r\n2018.11.20 03:13:35.762645 [ 66 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Converting aggregation data to two-level.\r\n2018.11.20 03:13:35.762816 [ 64 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Converting aggregation data to two-level.\r\n2018.11.20 03:13:35.763115 [ 63 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Converting aggregation data to two-level.\r\n2018.11.20 03:13:35.767598 [ 70 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Aggregation method: serialized\r\n2018.11.20 03:13:35.772766 [ 70 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Converting aggregation data to two-level.\r\n2018.11.20 03:13:35.773156 [ 68 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Aggregation method: serialized\r\n2018.11.20 03:13:35.775783 [ 69 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Aggregation method: serialized\r\n2018.11.20 03:13:35.776401 [ 65 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Aggregation method: serialized\r\n2018.11.20 03:13:35.782188 [ 65 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Converting aggregation data to two-level.\r\n2018.11.20 03:13:35.783568 [ 68 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Converting aggregation data to two-level.\r\n2018.11.20 03:13:35.787821 [ 69 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Converting aggregation data to two-level.\r\n2018.11.20 03:13:53.511451 [ 63 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 63 exited\r\n2018.11.20 03:13:53.512486 [ 65 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 65 exited\r\n2018.11.20 03:13:53.515355 [ 70 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 70 exited\r\n2018.11.20 03:13:53.515539 [ 69 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 69 exited\r\n2018.11.20 03:13:53.517699 [ 68 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 68 exited\r\n2018.11.20 03:13:53.518333 [ 64 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 64 exited\r\n2018.11.20 03:13:53.526868 [ 66 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 66 exited\r\n2018.11.20 03:13:53.537051 [ 67 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 67 exited\r\n2018.11.20 03:13:53.537275 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregated. 60755965 to 1296 rows (from 1287.876 MiB) in 17.851 sec. (3403590.174 rows/sec., 72.148 MiB/sec.)\r\n2018.11.20 03:13:53.537334 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregated. 60986337 to 1294 rows (from 1293.523 MiB) in 17.851 sec. (3416495.769 rows/sec., 72.464 MiB/sec.)\r\n2018.11.20 03:13:53.537353 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregated. 59865413 to 1294 rows (from 1269.541 MiB) in 17.851 sec. (3353700.849 rows/sec., 71.121 MiB/sec.)\r\n2018.11.20 03:13:53.537375 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregated. 61147619 to 1305 rows (from 1296.128 MiB) in 17.851 sec. (3425530.895 rows/sec., 72.610 MiB/sec.)\r\n2018.11.20 03:13:53.537392 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregated. 62460616 to 1303 rows (from 1323.072 MiB) in 17.851 sec. (3499085.873 rows/sec., 74.119 MiB/sec.)\r\n2018.11.20 03:13:53.537408 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregated. 61620224 to 1306 rows (from 1306.419 MiB) in 17.851 sec. (3452006.546 rows/sec., 73.186 MiB/sec.)\r\n2018.11.20 03:13:53.537422 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregated. 60667324 to 1298 rows (from 1286.050 MiB) in 17.851 sec. (3398624.445 rows/sec., 72.045 MiB/sec.)\r\n2018.11.20 03:13:53.537436 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Aggregated. 57302138 to 1286 rows (from 1215.060 MiB) in 17.851 sec. (3210104.453 rows/sec., 68.068 MiB/sec.)\r\n2018.11.20 03:13:53.537449 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ParallelAggregatingBlockInputStream: Total aggregated. 484805636 rows (from 10277.669 MiB) in 17.851 sec. (27159139.004 rows/sec., 575.762 MiB/sec.)\r\n2018.11.20 03:13:53.537459 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> Aggregator: Merging aggregated data\r\n2018.11.20 03:13:53.598635 [ 62 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 62 exited\r\n2018.11.20 03:13:53.598814 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Information> executeQuery: Read 484805636 rows, 10.30 GiB in 17.917 sec., 27058515 rows/sec., 588.82 MiB/sec.\r\n2018.11.20 03:13:53.598850 [ 48 ] {43d31689-d879-4c66-a600-b226863ea15d} <Debug> MemoryTracker: Peak memory usage (for query): 142.12 MiB.\r\n2018.11.20 03:13:53.598914 [ 48 ] {} <Trace> virtual DB::MergingAndConvertingBlockInputStream::~MergingAndConvertingBlockInputStream(): Waiting for threads to finish\r\n2018.11.20 03:13:53.599009 [ 72 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 72 exited\r\n2018.11.20 03:13:53.599046 [ 73 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 73 exited\r\n2018.11.20 03:13:53.599109 [ 78 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 78 exited\r\n2018.11.20 03:13:53.599141 [ 75 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 75 exited\r\n2018.11.20 03:13:53.599164 [ 76 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 76 exited\r\n2018.11.20 03:13:53.599171 [ 77 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 77 exited\r\n2018.11.20 03:13:53.599201 [ 74 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 74 exited\r\n2018.11.20 03:13:53.599663 [ 71 ] {43d31689-d879-4c66-a600-b226863ea15d} <Trace> ThreadStatus: Thread 71 exited\r\n2018.11.20 03:13:53.600209 [ 48 ] {} <Information> TCPHandler: Processed in 17.919 sec.\r\n2018.11.20 03:13:53.609377 [ 61 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> ThreadStatus: Thread 61 exited\r\n2018.11.20 03:13:54.427269 [ 60 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> ThreadStatus: Thread 60 exited\r\n2018.11.20 03:13:54.468361 [ 58 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Aggregator: Read 767 blocks of partially aggregated data, total 4547 rows.\r\n2018.11.20 03:13:54.468376 [ 59 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> ThreadStatus: Thread 59 exited\r\n2018.11.20 03:13:54.469474 [ 58 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Aggregator: Merging partially aggregated two-level data.\r\n2018.11.20 03:13:54.504718 [ 58 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Aggregator: Merged partially aggregated two-level data.\r\n2018.11.20 03:13:54.505005 [ 58 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Aggregator: Converting aggregated data to blocks\r\n2018.11.20 03:13:54.514064 [ 58 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> Aggregator: Destroying aggregate states\r\n2018.11.20 03:13:54.514522 [ 58 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Trace> ThreadStatus: Thread 58 exited\r\n2018.11.20 03:13:54.591167 [ 47 ] {fd14e035-b9b7-4bff-8a0a-72cdca6f9d47} <Error> executeQuery: Code: 0, e.displayText() = DB::Exception: Allocator: Cannot mmap 8.00 EiB., errno: 12, strerror: Cannot allocate memory, e.what() = DB::Exception (from [::1]:50624) (in query: SELECT      geoip_latitude AS latitude,      geoip_longitude AS longitude,      dictGetString('geoname', 'city_name', toUInt64(geoip_geoname_id)) AS city,      uniq(clientip) AS uniqueip FROM analytics.vme1  GROUP BY      latitude,      longitude,      city), Stack trace:\r\n\r\n0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x5c8f116]\r\n1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::string const&, int)+0x1f) [0x29b1e9f]\r\n2. /usr/bin/clickhouse-server(DB::throwFromErrno(std::string const&, int, int)+0x128) [0x5c76d98]\r\n3. /usr/bin/clickhouse-server(Allocator<false>::alloc(unsigned long, unsigned long)+0x303) [0x5c6e753]\r\n4. /usr/bin/clickhouse-server(Allocator<false>::realloc(void*, unsigned long, unsigned long, unsigned long)+0x55) [0x5c6ea95]\r\n5. /usr/bin/clickhouse-server() [0x55be509]\r\n6. /usr/bin/clickhouse-server(DB::ColumnString::deserializeAndInsertFromArena(char const*)+0x84) [0x55bfd54]\r\n7. /usr/bin/clickhouse-server(DB::ColumnUnique<DB::ColumnString>::uniqueDeserializeAndInsertFromArena(char const*, char const*&)+0x63) [0x4a81283]\r\n8. /usr/bin/clickhouse-server(DB::ColumnLowCardinality::deserializeAndInsertFromArena(char const*)+0x30) [0x55b22e0]\r\n9. /usr/bin/clickhouse-server(void DB::Aggregator::convertToBlockImplFinal<DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >, HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >(DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >&, HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> >&, std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&) const+0x74) [0x59648d4]\r\n10. /usr/bin/clickhouse-server(void DB::Aggregator::convertToBlockImpl<DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >, HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >(DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >&, HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> >&, std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, std::vector<DB::PODArray<char*, 4096ul, Allocator<false>, 15ul>*, std::allocator<DB::PODArray<char*, 4096ul, Allocator<false>, 15ul>*> >&, std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, bool) const+0x78) [0x5964a48]\r\n11. /usr/bin/clickhouse-server(DB::Block DB::Aggregator::prepareBlockAndFill<DB::Block DB::Aggregator::convertOneBucketToBlock<DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > > >(DB::AggregatedDataVariants&, DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >&, bool, unsigned long) const::{lambda(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, std::vector<DB::PODArray<char*, 4096ul, Allocator<false>, 15ul>*, std::allocator<DB::PODArray<char*, 4096ul, Allocator<false>, 15ul>*> >&, std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, bool)#1}>(DB::AggregatedDataVariants&, bool, unsigned long, DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >&&) const+0x3d7) [0x5964eb7]\r\n12. /usr/bin/clickhouse-server(std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<DB::Block>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::_Bind<std::list<DB::Block, std::allocator<DB::Block> > DB::Aggregator::prepareBlocksAndFillTwoLevelImpl<DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > > >(DB::AggregatedDataVariants&, DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >&, bool, ThreadPool*) const::{lambda(unsigned long, std::shared_ptr<DB::ThreadGroupStatus>)#1} (unsigned long, std::shared_ptr<DB::ThreadGroupStatus>)>, std::allocator<int>, DB::Block ()>::_M_run()::{lambda()#1}, DB::Block> >::_M_invoke(std::_Any_data const&)+0xa8) [0x5965558]\r\n13. /usr/bin/clickhouse-server(std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)+0x1f) [0x591699f]\r\n14. /lib64/libpthread.so.0(pthread_once+0x50) [0x7f1fbcaabe70]\r\n15. /usr/bin/clickhouse-server() [0x58ff69e]\r\n16. /usr/bin/clickhouse-server(std::__future_base::_Task_state<std::_Bind<std::list<DB::Block, std::allocator<DB::Block> > DB::Aggregator::prepareBlocksAndFillTwoLevelImpl<DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > > >(DB::AggregatedDataVariants&, DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >&, bool, ThreadPool*) const::{lambda(unsigned long, std::shared_ptr<DB::ThreadGroupStatus>)#1} (unsigned long, std::shared_ptr<DB::ThreadGroupStatus>)>, std::allocator<int>, DB::Block ()>::_M_run()+0x43) [0x5917e03]\r\n17. /usr/bin/clickhouse-server(std::list<DB::Block, std::allocator<DB::Block> > DB::Aggregator::prepareBlocksAndFillTwoLevelImpl<DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > > >(DB::AggregatedDataVariants&, DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true> > >&, bool, ThreadPool*) const+0x40b) [0x594f4ab]\r\n18. /usr/bin/clickhouse-server(DB::Aggregator::prepareBlocksAndFillTwoLevel(DB::AggregatedDataVariants&, bool, ThreadPool*) const+0x1ae) [0x5904ace]\r\n19. /usr/bin/clickhouse-server(DB::Aggregator::convertToBlocks(DB::AggregatedDataVariants&, bool, unsigned long) const+0x44d) [0x590f86d]\r\n20. /usr/bin/clickhouse-server(DB::MergingAggregatedBlockInputStream::readImpl()+0x3ec) [0x58a217c]\r\n21. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x21f) [0x4a117ff]\r\n22. /usr/bin/clickhouse-server(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x5882bfb]\r\n23. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x21f) [0x4a117ff]\r\n24. /usr/bin/clickhouse-server(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x5882bfb]\r\n25. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x21f) [0x4a117ff]\r\n26. /usr/bin/clickhouse-server(DB::AsynchronousBlockInputStream::calculate()+0x44) [0x4a08284]\r\n27. /usr/bin/clickhouse-server() [0x4a0877a]\r\n28. /usr/bin/clickhouse-server(ThreadPool::worker()+0x167) [0x5e95097]\r\n29. /usr/bin/clickhouse-server() [0x656ceff]\r\n30. /lib64/libpthread.so.0(+0x7e25) [0x7f1fbcaa6e25]\r\n31. /lib64/libc.so.6(clone+0x6d) [0x7f1fbc2c6bad]\r\n```\r\n\r\nanalytics.vme1 is a distributed table\r\nIf I query the shard directly, everything work fine\r\nto add some more fun, i'm using circular replication\r\n\r\nWill try to provide small repro case soon"},"comment":{"url":"https://api.github.com/repos/yandex/ClickHouse/issues/comments/441114358","html_url":"https://github.com/yandex/ClickHouse/issues/3614#issuecomment-441114358","issue_url":"https://api.github.com/repos/yandex/ClickHouse/issues/3614","id":441114358,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTExNDM1OA==","user":{"login":"champtar","id":3536755,"node_id":"MDQ6VXNlcjM1MzY3NTU=","avatar_url":"https://avatars1.githubusercontent.com/u/3536755?v=4","gravatar_id":"","url":"https://api.github.com/users/champtar","html_url":"https://github.com/champtar","followers_url":"https://api.github.com/users/champtar/followers","following_url":"https://api.github.com/users/champtar/following{/other_user}","gists_url":"https://api.github.com/users/champtar/gists{/gist_id}","starred_url":"https://api.github.com/users/champtar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/champtar/subscriptions","organizations_url":"https://api.github.com/users/champtar/orgs","repos_url":"https://api.github.com/users/champtar/repos","events_url":"https://api.github.com/users/champtar/events{/privacy}","received_events_url":"https://api.github.com/users/champtar/received_events","type":"User","site_admin":false},"created_at":"2018-11-22T20:17:04Z","updated_at":"2018-11-22T20:17:04Z","author_association":"CONTRIBUTOR","body":"can't reproduce on master + #3632 (https://github.com/champtar/ClickHouse/commit/2a8b78d8ef6e492fca8be2c78def7fe4c1aa0934)\r\nThanks"}}
{"actor":{"display_login":"champtar"}}