{"action":"created","issue":{"url":"https://api.github.com/repos/Qihoo360/Atlas/issues/174","repository_url":"https://api.github.com/repos/Qihoo360/Atlas","labels_url":"https://api.github.com/repos/Qihoo360/Atlas/issues/174/labels{/name}","comments_url":"https://api.github.com/repos/Qihoo360/Atlas/issues/174/comments","events_url":"https://api.github.com/repos/Qihoo360/Atlas/issues/174/events","html_url":"https://github.com/Qihoo360/Atlas/issues/174","id":305431365,"number":174,"title":"使用Atalas作为代理层mysql 导致mysql5.6奔溃","user":{"login":"lilien1010","id":3814966,"avatar_url":"https://avatars1.githubusercontent.com/u/3814966?v=4","gravatar_id":"","url":"https://api.github.com/users/lilien1010","html_url":"https://github.com/lilien1010","followers_url":"https://api.github.com/users/lilien1010/followers","following_url":"https://api.github.com/users/lilien1010/following{/other_user}","gists_url":"https://api.github.com/users/lilien1010/gists{/gist_id}","starred_url":"https://api.github.com/users/lilien1010/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lilien1010/subscriptions","organizations_url":"https://api.github.com/users/lilien1010/orgs","repos_url":"https://api.github.com/users/lilien1010/repos","events_url":"https://api.github.com/users/lilien1010/events{/privacy}","received_events_url":"https://api.github.com/users/lilien1010/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-15T06:36:48Z","updated_at":"2018-03-15T06:41:16Z","closed_at":null,"author_association":"NONE","body":"mysql 版本：mysql  Ver 14.14 Distrib 5.6.36, for Linux (x86_64) using  EditLine wrapper\r\n\r\n使用模式\r\nClient -> Atlas -> Mysql\r\n\r\n原来是使用方式是客户端直连Mysql，但是后面想在连接上以及做读写分离 就用了Atlas，但是发现通过Atalas 代理基本的读写业务之后，Mysql 会不定期奔溃（几乎每天都会奔溃），并且自动重启。\r\n\r\n后面我们客户端改回去直连Atalas，mysql 就没有出现奔溃的现象。\r\n\r\n下面我贴出的是coredump记录，以及奔溃日志\r\n\r\n\r\n**COREDUMP 文件**\r\n通过对比源码发现是parser_state 从 thd 解析原始报文出来之后，数据报文里面的长度和sql语句对不上，解析出来的sql语句是 ***pdate HT_USER_PROPERTY set PROFILEVERSION = 1520829348 where USERID = 6059889;*** \r\n sql语句前面少了一个u，对后面的语法解析的时候直接就奔溃了\r\n\r\n\r\n下面是奔溃Log ，mysql 每日奔溃一次，然后自动重启，影响业务在5秒内。\r\n```\r\nThis could be because you hit a bug. It is also possible that this binary\r\nor one of the libraries it was linked against is corrupt, improperly built,\r\nor misconfigured. This error can also be caused by malfunctioning hardware.\r\nWe will try our best to scrape up some info that will hopefully help\r\ndiagnose the problem, but since we have already crashed, \r\nsomething is definitely wrong and this may fail.\r\n\r\nkey_buffer_size=134217728\r\nread_buffer_size=2097152\r\nmax_used_connections=197\r\nmax_threads=1000\r\nthread_count=182\r\nconnection_count=181\r\nIt is possible that mysqld could use up to \r\nkey_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4239845 K  bytes of memory\r\nHope that's ok; if not, decrease some variables in the equation.\r\n\r\nThread pointer: 0x281d0770\r\nAttempting backtrace. You can use the following information to find out\r\nwhere mysqld died. If you see no messages after this, something went\r\nterribly wrong...\r\nstack_bottom = 7f5184ccbea0 thread_stack 0x100000\r\n/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0x9009dc]\r\n/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x357)[0x6729c7]\r\n/lib64/libpthread.so.0(+0xf5e0)[0x7f578bbd65e0]\r\n/usr/local/mysql/bin/mysqld(_Z16digest_add_tokenP16sql_digest_statejP7YYSTYPE+0x71)[0x6cb691]\r\n/usr/local/mysql/bin/mysqld(_Z8MYSQLlexP7YYSTYPEP3THD+0x60)[0x6e98c0]\r\n/usr/local/mysql/bin/mysqld(_Z10MYSQLparseP3THD+0x7f6)[0x79c126]\r\n/usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x130)[0x6fc740]\r\n/usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1418)[0x6fe468]\r\n/usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x1fd)[0x6c568d]\r\n/usr/local/mysql/bin/mysqld(handle_one_connection+0x39)[0x6c56d9]\r\n/usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x143)[0xb31923]\r\n/lib64/libpthread.so.0(+0x7e25)[0x7f578bbcee25]\r\n/lib64/libc.so.6(clone+0x6d)[0x7f578aca134d]\r\n\r\nTrying to get some variables.\r\nSome pointers may be invalid and cause the dump to abort.\r\nQuery (7f510442302f): is an invalid pointer\r\nConnection ID (thread ID): 6554\r\nStatus: NOT_KILLED\r\n\r\nThe manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains\r\ninformation that should help you find out what is causing the crash.\r\nWriting a core file\r\n2018-03-12 12:36:29 12176 [Note] Plugin 'FEDERATED' is disabled.\r\n2018-03-12 12:36:29 12176 [Note] InnoDB: Using atomics to ref count buffer pool pages\r\n2018-03-12 12:36:29 12176 [Note] InnoDB: The InnoDB memory heap is disabled\r\n2018-03-12 12:36:29 12176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins\r\n2018-03-12 12:36:29 12176 [Note] InnoDB: Memory barrier is not used\r\n2018-03-12 12:36:29 12176 [Note] InnoDB: Compressed tables use zlib 1.2.3\r\n2018-03-12 12:36:29 12176 [Note] InnoDB: Using CPU crc32 instructions\r\n2018-03-12 12:36:29 12176 [Note] InnoDB: Initializing buffer pool, size = 22.0G\r\n2018-03-12 12:36:30 12176 [Note] InnoDB: Completed initialization of buffer pool\r\n2018-03-12 12:36:31 12176 [Note] InnoDB: Highest supported file format is Barracuda.\r\n2018-03-12 12:36:31 12176 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3412483943928\r\n2018-03-12 12:36:31 12176 [Note] InnoDB: Database was not shutdown normally!\r\n2018-03-12 12:36:31 12176 [Note] InnoDB: Starting crash recovery.\r\n2018-03-12 12:36:31 12176 [Note] InnoDB: Reading tablespace information from the .ibd files...\r\n2018-03-12 12:36:32 12176 [Note] InnoDB: Restoring possible half-written data pages \r\n2018-03-12 12:36:32 12176 [Note] InnoDB: from the doublewrite buffer...\r\nInnoDB: Doing recovery: scanned up to log sequence number 3412489186304\r\nInnoDB: Doing recovery: scanned up to log sequence number 3412492675710\r\n2018-03-12 12:36:32 12176 [Note] InnoDB: Starting an apply batch of log records to the database...\r\nInnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 \r\nInnoDB: Apply batch completed\r\nInnoDB: Last MySQL binlog file position 0 728658151, file name mysql-bin.000279\r\n2018-03-12 12:36:37 12176 [Note] InnoDB: 128 rollback segment(s) are active.\r\n2018-03-12 12:36:37 12176 [Note] InnoDB: Waiting for purge to start\r\n2018-03-12 12:36:37 12176 [Note] InnoDB: 5.6.36 started; log sequence number 3412492675710\r\n2018-03-12 12:36:37 12176 [Note] Recovering after a crash using mysql-bin\r\n2018-03-12 12:36:40 12176 [Note] Starting crash recovery...\r\n2018-03-12 12:36:40 12176 [Note] Crash recovery finished.\r\n2018-03-12 12:36:42 12176 [Note] Server hostname (bind-address): '*'; port: 3306\r\n2018-03-12 12:36:42 12176 [Note] IPv6 is available.\r\n2018-03-12 12:36:42 12176 [Note]   - '::' resolves to '::';\r\n2018-03-12 12:36:42 12176 [Note] Server socket created on IP: '::'.\r\n2018-03-12 12:36:42 12176 [Warning] 'user' entry 'root@ht6' ignored in --skip-name-resolve mode.\r\n2018-03-12 12:36:42 12176 [Warning] 'user' entry '@ht6' ignored in --skip-name-resolve mode.\r\n2018-03-12 12:36:42 12176 [Warning] 'proxies_priv' entry '@ root@HT6' ignored in --skip-name-resolve mode.\r\n2018-03-12 12:36:42 12176 [Note] Event Scheduler: Loaded 3 events\r\n2018-03-12 12:36:42 12176 [Note] /usr/local/mysql/bin/mysqld: ready for connections.\r\nVersion: '5.6.36-log'  socket: '/htdata/mysql/mysql.sock'  port: 3306  Source distribution\r\n2018-03-12 12:36:42 12176 [Note] Event Scheduler: scheduler thread started with id 1\r\n```\r\n**COREDUMP 文件**\r\n \r\n```\r\n(gdb) bt\r\n#0  0x00007f578bbd39b1 in pthread_kill () from /lib64/libpthread.so.0\r\n#1  0x000000000067298d in handle_fatal_signal (sig=11) at /usr/local/src/mysql-5.6.36/sql/signal_handler.cc:230\r\n#2  <signal handler called>\r\n#3  read_token (tok=<synthetic pointer>, index=4294967294, digest_storage=0x281d20c8) at /usr/local/src/mysql-5.6.36/sql/sql_digest.cc:70\r\n#4  digest_add_token (state=0x281d20c0, token=token@entry=0, yylval=0x7f5184cca920) at /usr/local/src/mysql-5.6.36/sql/sql_digest.cc:603\r\n#5  0x00000000006e98c0 in add_digest_token (yylval=<optimized out>, token=<optimized out>, this=<optimized out>) at /usr/local/src/mysql-5.6.36/sql/sql_lex.cc:391\r\n#6  MYSQLlex (yylval=yylval@entry=0x7f5184cca920, thd=thd@entry=0x281d0770) at /usr/local/src/mysql-5.6.36/sql/sql_lex.cc:980\r\n#7  0x000000000079c126 in MYSQLparse (YYTHD=YYTHD@entry=0x281d0770) at /export/home2/pb2/build/sb_2-22757651-1489819423.63/dist_GPL/sql/sql_yacc.cc:18502\r\n#8  0x00000000006fc740 in parse_sql (creation_ctx=0x0, parser_state=0x7f5184ccb740, thd=0x281d0770) at /usr/local/src/mysql-5.6.36/sql/sql_parse.cc:8408\r\n#9  mysql_parse (thd=thd@entry=0x281d0770, rawbuf=rawbuf@entry=0x7f510442302f \"\", length=length@entry=4017, parser_state=parser_state@entry=0x7f5184ccb740)\r\n    at /usr/local/src/mysql-5.6.36/sql/sql_parse.cc:6340\r\n#10 0x00000000006fe468 in dispatch_command (command=<optimized out>, thd=0x281d0770, packet=<optimized out>, packet_length=<optimized out>) at /usr/local/src/mysql-5.6.36/sql/sql_parse.cc:1451\r\n#11 0x00000000006ffca4 in do_command (thd=<optimized out>) at /usr/local/src/mysql-5.6.36/sql/sql_parse.cc:1039\r\n#12 0x00000000006c568d in do_handle_one_connection (thd_arg=thd_arg@entry=0x28260da0) at /usr/local/src/mysql-5.6.36/sql/sql_connect.cc:982\r\n#13 0x00000000006c56d9 in handle_one_connection (arg=arg@entry=0x28260da0) at /usr/local/src/mysql-5.6.36/sql/sql_connect.cc:899\r\n#14 0x0000000000b31923 in pfs_spawn_thread (arg=0x275c2c30) at /usr/local/src/mysql-5.6.36/storage/perfschema/pfs.cc:1860\r\n#15 0x00007f578bbcee25 in start_thread () from /lib64/libpthread.so.0\r\n#16 0x00007f578aca134d in clone () from /lib64/libc.so.6\r\n(gdb) bt full\r\n parser_state = {m_input = {m_compute_digest = false}, m_lip = {m_thd = 0x281d0770, yylineno = 1, yytoklen = 0, yylval = 0x7f5184cca920, lookahead_token = -1, lookahead_yylval = 0x0, \r\n            m_ptr = 0x7f5104423030 \"\", m_tok_start = 0x7f510442302f \"\", m_tok_end = 0x7f510442302f \"\", m_end_of_query = 0x7f5104423fe0 \"\", m_tok_start_prev = 0x0, m_buf = 0x7f510442302f \"\", \r\n            m_buf_length = 4017, m_echo = true, \r\nm_echo_saved = false, m_cpp_buf = 0x7f5104408a90 \"\", \r\n            m_cpp_ptr = 0x7f5104408a91 \"pdate HT_USER_PROPERTY set PROFILEVERSION = 1520829348 where USERID = 6059889;\", m_cpp_tok_start = 0x7f5104408a90 \"\", m_cpp_tok_start_prev = 0x0, \r\n            m_cpp_tok_end = 0x7f5104408a90 \"\", m_body_utf8 = 0x0, m_body_utf8_ptr = 0x0, m_cpp_utf8_processed_ptr = 0x0, next_state = MY_LEX_START, found_semicolon = 0x0, tok_bitmap = 127 '\\177', \r\n            ignore_space = false, stmt_prepare_mode = false, multi_statements = true, in_comment = NO_COMMENT, in_comment_saved = NO_COMMENT, m_cpp_text_start = 0x7f5104408ad7 \"6059889;\", \r\n            m_cpp_text_end = 0x7f5104408ade \";\", m_underscore_cs = 0x0, m_digest = 0x281d20c0}, m_yacc = {yacc_yyss = 0x0, yacc_yyvs = 0x0, m_set_signal_info = {m_item = {0x0 <repeats 12 times>}}, \r\n            m_lock_type = TL_READ_DEFAULT, m_mdl_type = MDL_SHARED_READ, m_ha_rkey_mode = HA_READ_KEY_EXACT}, m_digest_psi = 0x281d2108}\r\n        net = 0x281d0980\r\n        error = false\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/Qihoo360/Atlas/issues/comments/373276297","html_url":"https://github.com/Qihoo360/Atlas/issues/174#issuecomment-373276297","issue_url":"https://api.github.com/repos/Qihoo360/Atlas/issues/174","id":373276297,"user":{"login":"lilien1010","id":3814966,"avatar_url":"https://avatars1.githubusercontent.com/u/3814966?v=4","gravatar_id":"","url":"https://api.github.com/users/lilien1010","html_url":"https://github.com/lilien1010","followers_url":"https://api.github.com/users/lilien1010/followers","following_url":"https://api.github.com/users/lilien1010/following{/other_user}","gists_url":"https://api.github.com/users/lilien1010/gists{/gist_id}","starred_url":"https://api.github.com/users/lilien1010/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lilien1010/subscriptions","organizations_url":"https://api.github.com/users/lilien1010/orgs","repos_url":"https://api.github.com/users/lilien1010/repos","events_url":"https://api.github.com/users/lilien1010/events{/privacy}","received_events_url":"https://api.github.com/users/lilien1010/received_events","type":"User","site_admin":false},"created_at":"2018-03-15T06:41:16Z","updated_at":"2018-03-15T06:41:16Z","author_association":"NONE","body":"Atlas版本\r\nAtlas-sharding_1.0.1-el6.x86_64.rpm\r\n\r\nmysql-proxy 0.8.2\r\n  chassis: mysql-proxy 0.8.2\r\n  glib2: 2.32.4\r\n  libevent: 2.0.20-stable"}}
{"actor":{"display_login":"lilien1010"}}