{"action":"created","issue":{"url":"https://api.github.com/repos/yandex/ClickHouse/issues/3174","repository_url":"https://api.github.com/repos/yandex/ClickHouse","labels_url":"https://api.github.com/repos/yandex/ClickHouse/issues/3174/labels{/name}","comments_url":"https://api.github.com/repos/yandex/ClickHouse/issues/3174/comments","events_url":"https://api.github.com/repos/yandex/ClickHouse/issues/3174/events","html_url":"https://github.com/yandex/ClickHouse/issues/3174","id":361834223,"node_id":"MDU6SXNzdWUzNjE4MzQyMjM=","number":3174,"title":"DB::Exception: Too many parts (600). Merges are processing significantly slower than inserts","user":{"login":"q0987","id":16629186,"node_id":"MDQ6VXNlcjE2NjI5MTg2","avatar_url":"https://avatars3.githubusercontent.com/u/16629186?v=4","gravatar_id":"","url":"https://api.github.com/users/q0987","html_url":"https://github.com/q0987","followers_url":"https://api.github.com/users/q0987/followers","following_url":"https://api.github.com/users/q0987/following{/other_user}","gists_url":"https://api.github.com/users/q0987/gists{/gist_id}","starred_url":"https://api.github.com/users/q0987/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/q0987/subscriptions","organizations_url":"https://api.github.com/users/q0987/orgs","repos_url":"https://api.github.com/users/q0987/repos","events_url":"https://api.github.com/users/q0987/events{/privacy}","received_events_url":"https://api.github.com/users/q0987/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-19T16:42:51Z","updated_at":"2018-09-20T12:27:32Z","closed_at":null,"author_association":"NONE","body":"ClickHouse client version 18.6.0.\r\nConnected to ClickHouse server version 18.6.0 revision 54401.\r\n\r\nHello all,\r\nI have been working on this issue for almost a week and still see the problem coming back.\r\n\r\nThe following changes have been made on config.xml:\r\n\r\n    <merge_tree>\r\n        <parts_to_delay_insert>300</parts_to_delay_insert>\r\n        <parts_to_throw_insert>600</parts_to_throw_insert>\r\n        <max_delay_to_insert>2</max_delay_to_insert>\r\n        <max_suspicious_broken_parts>5</max_suspicious_broken_parts>\r\n    </merge_tree>\r\n\r\nThe input data size: 177GB\r\nThe total number of input files: 2050\r\nAverage size of each file: 110MB\r\nCommand used to import the data:\r\ncat *.clickhouse | clickhouse-client --query=\"INSERT INTO default.MyTable(....\")...\r\nQuestion 1> Based on the above command,  how many insert statement generated?\r\nIs this a bulk insertion?\r\n\r\nAfter the change, the system still reports the same errors as below.\r\nQuestion 2> What else should I try? Increase the size again?\r\n\r\nThank you very much\r\n\r\n2018.09.19 10:25:02.114885 [ 49 ] <Error> executeQuery: Code: 252, e.displayText() = DB::Exception: Too many parts (600). Merges are processing significantly slower than inserts., e.what() = DB::Exception (from 127.0.0.1:46902) (in query: INSERT INTO default.MyTable(timestamp, ...) FORMAT RowBinary), Stack trace:\r\n\r\n0. clickhouse-server(StackTrace::StackTrace()+0x16) [0x48abed6]\r\n1. clickhouse-server(DB::Exception::Exception(std::string const&, int)+0x1f) [0x2682dcf]\r\n2. clickhouse-server(DB::MergeTreeData::delayInsertOrThrowIfNeeded(Poco::Event*) const+0x3a7) [0x431ccc7]\r\n3. clickhouse-server(DB::MergeTreeBlockOutputStream::write(DB::Block const&)+0x2c) [0x4314cfc]\r\n4. clickhouse-server(DB::PushingToViewsBlockOutputStream::write(DB::Block const&)+0x31) [0x44b38a1]\r\n5. clickhouse-server(DB::AddingDefaultBlockOutputStream::write(DB::Block const&)+0x9a1) [0x44442c1]\r\n6. clickhouse-server(DB::SquashingBlockOutputStream::write(DB::Block const&)+0x363) [0x44b91a3]\r\n7. clickhouse-server(DB::CountingBlockOutputStream::write(DB::Block const&)+0x1d) [0x4468a3d]\r\n8. clickhouse-server(DB::TCPHandler::receiveData()+0x8d) [0x268b63d]\r\n9. clickhouse-server(DB::TCPHandler::receivePacket()+0x85) [0x268c435]\r\n10. clickhouse-server(DB::TCPHandler::readData(DB::Settings const&)+0x19b) [0x268c8eb]\r\n11. clickhouse-server(DB::TCPHandler::processInsertQuery(DB::Settings const&)+0x1d6) [0x268cc86]\r\n12. clickhouse-server(DB::TCPHandler::runImpl()+0x43c) [0x268d24c]\r\n13. clickhouse-server(DB::TCPHandler::run()+0x1c) [0x268e0ac]\r\n14. clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x49eefaf]\r\n15. clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x110) [0x49ef610]\r\n16. clickhouse-server(Poco::PooledThread::run()+0x77) [0x4a4df27]\r\n17. clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x4a4b488]\r\n18. clickhouse-server() [0x4d3fe1f]\r\n19. /lib64/libpthread.so.0(+0x7e25) [0x7f912655ae25]\r\n20. /lib64/libc.so.6(clone+0x6d) [0x7f9125d8234d]\r\n\r\n2018.09.19 10:25:02.162124 [ 49 ] <Error> ServerErrorHandler: Code: 99, e.displayText() = DB::Exception: Unknown packet 44 from client, e.what() = DB::Exception, Stack trace:\r\n\r\n0. clickhouse-server(StackTrace::StackTrace()+0x16) [0x48abed6]\r\n1. clickhouse-server(DB::Exception::Exception(std::string const&, int)+0x1f) [0x2682dcf]\r\n2. clickhouse-server(DB::TCPHandler::receivePacket()+0x2e0) [0x268c690]\r\n3. clickhouse-server(DB::TCPHandler::runImpl()+0x2b0) [0x268d0c0]\r\n4. clickhouse-server(DB::TCPHandler::run()+0x1c) [0x268e0ac]\r\n5. clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x49eefaf]\r\n6. clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x110) [0x49ef610]\r\n7. clickhouse-server(Poco::PooledThread::run()+0x77) [0x4a4df27]\r\n8. clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x4a4b488]\r\n9. clickhouse-server() [0x4d3fe1f]\r\n10. /lib64/libpthread.so.0(+0x7e25) [0x7f912655ae25]\r\n11. /lib64/libc.so.6(clone+0x6d) [0x7f9125d8234d]"},"comment":{"url":"https://api.github.com/repos/yandex/ClickHouse/issues/comments/423164272","html_url":"https://github.com/yandex/ClickHouse/issues/3174#issuecomment-423164272","issue_url":"https://api.github.com/repos/yandex/ClickHouse/issues/3174","id":423164272,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzE2NDI3Mg==","user":{"login":"q0987","id":16629186,"node_id":"MDQ6VXNlcjE2NjI5MTg2","avatar_url":"https://avatars3.githubusercontent.com/u/16629186?v=4","gravatar_id":"","url":"https://api.github.com/users/q0987","html_url":"https://github.com/q0987","followers_url":"https://api.github.com/users/q0987/followers","following_url":"https://api.github.com/users/q0987/following{/other_user}","gists_url":"https://api.github.com/users/q0987/gists{/gist_id}","starred_url":"https://api.github.com/users/q0987/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/q0987/subscriptions","organizations_url":"https://api.github.com/users/q0987/orgs","repos_url":"https://api.github.com/users/q0987/repos","events_url":"https://api.github.com/users/q0987/events{/privacy}","received_events_url":"https://api.github.com/users/q0987/received_events","type":"User","site_admin":false},"created_at":"2018-09-20T12:27:32Z","updated_at":"2018-09-20T12:27:32Z","author_association":"NONE","body":"Hello filimonov,\r\n\r\nI use the following command to insert the data into ClickHouse.\r\n\r\ncat *.clickhouse | clickhouse-client --query=\"INSERT INTO default.MyTable(....\")...\r\n\r\nSince the number of files are large, I don't think many insert statement can be triggered.\r\n\r\nQuestion> Based on the above statement, how many insert statement has been issues?\r\n\r\nThank you\r\n"}}
{"actor":{"display_login":"q0987"}}