{"action":"created","issue":{"url":"https://api.github.com/repos/SpiderClub/weibospider/issues/124","repository_url":"https://api.github.com/repos/SpiderClub/weibospider","labels_url":"https://api.github.com/repos/SpiderClub/weibospider/issues/124/labels{/name}","comments_url":"https://api.github.com/repos/SpiderClub/weibospider/issues/124/comments","events_url":"https://api.github.com/repos/SpiderClub/weibospider/issues/124/events","html_url":"https://github.com/SpiderClub/weibospider/issues/124","id":357980810,"node_id":"MDU6SXNzdWUzNTc5ODA4MTA=","number":124,"title":"使用master分支，sea","user":{"login":"mrvx666","id":43062346,"node_id":"MDQ6VXNlcjQzMDYyMzQ2","avatar_url":"https://avatars2.githubusercontent.com/u/43062346?v=4","gravatar_id":"","url":"https://api.github.com/users/mrvx666","html_url":"https://github.com/mrvx666","followers_url":"https://api.github.com/users/mrvx666/followers","following_url":"https://api.github.com/users/mrvx666/following{/other_user}","gists_url":"https://api.github.com/users/mrvx666/gists{/gist_id}","starred_url":"https://api.github.com/users/mrvx666/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrvx666/subscriptions","organizations_url":"https://api.github.com/users/mrvx666/orgs","repos_url":"https://api.github.com/users/mrvx666/repos","events_url":"https://api.github.com/users/mrvx666/events{/privacy}","received_events_url":"https://api.github.com/users/mrvx666/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-07T09:01:30Z","updated_at":"2018-09-07T13:01:32Z","closed_at":null,"author_association":"NONE","body":"大大你好，非常喜欢你的项目，最近想从1.7.2切换到master分支，搜索任务无法运行，恳请大大解答\r\n\r\n1.你是怎么操作的？\r\ngit clone master分支到本地，然后直接\r\ncelery -A tasks.workers -Q login_queue,search_crawler worker -l info --concurrency=1 -Ofair\r\n运行，控制台运行正常，然后执行登陆\r\npython3 first_task_execution/login_first.py，到此为止一切正常\r\n然后运行搜索任务python3 first_task_execution/search_first.py，开始提示\r\nThe account has been expired，然后开始大面积报错。\r\n\r\n2.你期望的结果是什么？\r\nsearch_first任务正常执行，并成功写入数据库。\r\n\r\n3.实际上你得到的结果是什么？\r\n\r\n无法搜索任务，数据库为空。\r\n\r\n4.你使用的是哪个版本的WeiboSpider? 你的操作系统是什么？是否有读本项目的[常见问题])？\r\n前段时间用的都是1.7.2，但是无奈稳定版太久不更新了，而且出现了爬取数据比预期数据少很多的情况，甚至一页只能获取到四五个数据，操作系统使用的是ubuntu 16.04 LTS，已经阅读过相关的Issues，没有查到类似情况的解决方案。\r\n\r\nmrvx@mrvx:~/weibospider$ celery -A tasks.workers -Q login_queue,search_crawler worker -l info --concurrency=1 -Ofair\r\n \r\n -------------- celery@mrvx v4.1.0 (latentcall)\r\n---- **** ----- \r\n--- * ***  * -- Linux-4.4.0-85-generic-i686-with-Ubuntu-16.04-xenial 2018-09-07 16:48:09\r\n-- * - **** --- \r\n- ** ---------- [config]\r\n- ** ---------- .> app:         weibo_task:0xb62b9b6c\r\n- ** ---------- .> transport:   redis://:**@127.0.0.1:6379/5\r\n- ** ---------- .> results:     disabled://\r\n- *** --- * --- .> concurrency: 1 (prefork)\r\n-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)\r\n--- ***** ----- \r\n -------------- [queues]\r\n                .> login_queue      exchange=login_queue(direct) key=for_login\r\n                .> search_crawler   exchange=search_crawler(direct) key=for_search_info\r\n\r\n[tasks]\r\n  . tasks.comment.crawl_comment_by_page\r\n  . tasks.comment.crawl_comment_page\r\n  . tasks.comment.execute_comment_task\r\n  . tasks.dialogue.crawl_dialogue\r\n  . tasks.dialogue.crawl_dialogue_by_comment_id\r\n  . tasks.dialogue.crawl_dialogue_by_comment_page\r\n  . tasks.dialogue.execute_dialogue_task\r\n  . tasks.downloader.download_img_task\r\n  . tasks.home.crawl_ajax_page\r\n  . tasks.home.crawl_weibo_datas\r\n  . tasks.home.execute_home_task\r\n  . tasks.login.execute_login_task\r\n  . tasks.login.login_task\r\n  . tasks.praise.crawl_praise_by_page\r\n  . tasks.praise.crawl_praise_page\r\n  . tasks.praise.execute_praise_task\r\n  . tasks.repost.crawl_repost_by_page\r\n  . tasks.repost.crawl_repost_page\r\n  . tasks.repost.execute_repost_task\r\n  . tasks.search.execute_search_task\r\n  . tasks.search.search_keyword\r\n  . tasks.user.crawl_follower_fans\r\n  . tasks.user.crawl_person_infos\r\n  . tasks.user.crawl_person_infos_not_in_seed_ids\r\n  . tasks.user.execute_user_task\r\n\r\n[2018-09-07 16:48:10,047: INFO/MainProcess] Connected to redis://:**@127.0.0.1:6379/5\r\n[2018-09-07 16:48:10,058: INFO/MainProcess] mingle: searching for neighbors\r\n[2018-09-07 16:48:11,080: INFO/MainProcess] mingle: all alone\r\n[2018-09-07 16:48:11,089: INFO/MainProcess] celery@mrvx ready.\r\n[2018-09-07 16:48:14,534: INFO/MainProcess] Received task: tasks.login.login_task[8949bc11-55b1-4dbd-9be4-6ebfb3e2f0c1]  \r\n[2018-09-07 16:48:24,551: INFO/MainProcess] Received task: tasks.login.login_task[b1d4521b-b529-410b-a254-4a85edf9220d]  \r\n[2018-09-07 16:48:34,563: INFO/MainProcess] Received task: tasks.login.login_task[6aea175a-0247-41bc-b575-18af7c869514]  \r\n2018-09-07 16:48:38 - other - INFO - Login successful! The login account is 13528612875\r\n[2018-09-07 16:48:38,753: INFO/ForkPoolWorker-1] Login successful! The login account is 13528612875\r\n[2018-09-07 16:48:38,756: INFO/ForkPoolWorker-1] Task tasks.login.login_task[8949bc11-55b1-4dbd-9be4-6ebfb3e2f0c1] succeeded in 24.221189369000058s: None\r\n[2018-09-07 16:48:57,880: INFO/MainProcess] Received task: tasks.search.search_keyword[d19f3335-15d2-4287-b4d2-72c93afdd85f]  \r\n2018-09-07 16:49:02 - other - INFO - Login successful! The login account is 13620883760\r\n[2018-09-07 16:49:02,267: INFO/ForkPoolWorker-1] Login successful! The login account is 13620883760\r\n[2018-09-07 16:49:02,272: INFO/ForkPoolWorker-1] Task tasks.login.login_task[b1d4521b-b529-410b-a254-4a85edf9220d] succeeded in 23.512898163000045s: None\r\n2018-09-07 16:49:16 - other - INFO - Login successful! The login account is 17827463305\r\n[2018-09-07 16:49:16,348: INFO/ForkPoolWorker-1] Login successful! The login account is 17827463305\r\n[2018-09-07 16:49:16,351: INFO/ForkPoolWorker-1] Task tasks.login.login_task[6aea175a-0247-41bc-b575-18af7c869514] succeeded in 14.0774650379999s: None\r\n2018-09-07 16:49:16 - crawler - INFO - We are searching keyword \"地震\"\r\n[2018-09-07 16:49:16,353: INFO/ForkPoolWorker-1] We are searching keyword \"地震\"\r\n2018-09-07 16:49:16 - crawler - INFO - the crawling url is http://s.weibo.com/weibo/%E5%9C%B0%E9%9C%87&region=custom:22:1000&scope=ori&suball=1&timescope=custom:2018-01-01-00:2018-01-15-23&page=1\r\n[2018-09-07 16:49:16,354: INFO/ForkPoolWorker-1] the crawling url is http://s.weibo.com/weibo/%E5%9C%B0%E9%9C%87&region=custom:22:1000&scope=ori&suball=1&timescope=custom:2018-01-01-00:2018-01-15-23&page=1\r\n2018-09-07 16:49:16 - crawler - WARNING - The account has been expired\r\n[2018-09-07 16:49:16,356: WARNING/ForkPoolWorker-1] The account has been expired\r\n[2018-09-07 16:49:34,518: WARNING/ForkPoolWorker-1] /home/mrvx/.local/lib/python3.5/site-packages/pymysql/cursors.py:166: Warning: (1287, \"'@@tx_isolation' is deprecated and will be removed in a future release. Please use '@@transaction_isolation' instead\")\r\n  result = self._query(query)\r\n[2018-09-07 16:49:34,528: ERROR/ForkPoolWorker-1] Task tasks.search.search_keyword[d19f3335-15d2-4287-b4d2-72c93afdd85f] raised unexpected: InternalError('(pymysql.err.InternalError) (1054, \"Unknown column \\'weibo_data.weibo_img_path\\' in \\'field list\\'\")',)\r\nTraceback (most recent call last):\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1182, in _execute_context\r\n    context)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py\", line 470, in do_execute\r\n    cursor.execute(statement, parameters)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/cursors.py\", line 166, in execute\r\n    result = self._query(query)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/cursors.py\", line 322, in _query\r\n    conn.query(q)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 856, in query\r\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 1057, in _read_query_result\r\n    result.read()\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 1340, in read\r\n    first_packet = self.connection._read_packet()\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 1014, in _read_packet\r\n    packet.check_error()\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 393, in check_error\r\n    err.raise_mysql_exception(self._data)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/err.py\", line 107, in raise_mysql_exception\r\n    raise errorclass(errno, errval)\r\npymysql.err.InternalError: (1054, \"Unknown column 'weibo_data.weibo_img_path' in 'field list'\")\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/celery/app/trace.py\", line 374, in trace_task\r\n    R = retval = fun(*args, **kwargs)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/celery/app/trace.py\", line 629, in __protected_call__\r\n    return self.run(*args, **kwargs)\r\n  File \"/home/mrvx/weibospider/tasks/search.py\", line 50, in search_keyword\r\n    rs = WbDataOper.get_wb_by_mid(wb_data.weibo_id)\r\n  File \"/home/mrvx/weibospider/db/dao.py\", line 174, in get_wb_by_mid\r\n    return db_session.query(WeiboData).filter(WeiboData.weibo_id == mid).first()\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/orm/query.py\", line 2755, in first\r\n    ret = list(self[0:1])\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/orm/query.py\", line 2547, in __getitem__\r\n    return list(res)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/orm/query.py\", line 2855, in __iter__\r\n    return self._execute_and_instances(context)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/orm/query.py\", line 2878, in _execute_and_instances\r\n    result = conn.execute(querycontext.statement, self._params)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 945, in execute\r\n    return meth(self, multiparams, params)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/sql/elements.py\", line 263, in _execute_on_connection\r\n    return connection._execute_clauseelement(self, multiparams, params)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1053, in _execute_clauseelement\r\n    compiled_sql, distilled_params\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1189, in _execute_context\r\n    context)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1402, in _handle_dbapi_exception\r\n    exc_info\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause\r\n    reraise(type(exception), exception, tb=exc_tb, cause=cause)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py\", line 186, in reraise\r\n    raise value.with_traceback(tb)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1182, in _execute_context\r\n    context)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py\", line 470, in do_execute\r\n    cursor.execute(statement, parameters)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/cursors.py\", line 166, in execute\r\n    result = self._query(query)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/cursors.py\", line 322, in _query\r\n    conn.query(q)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 856, in query\r\n    self._affected_rows = self._read_query_result(unbuffered=unbuffered)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 1057, in _read_query_result\r\n    result.read()\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 1340, in read\r\n    first_packet = self.connection._read_packet()\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 1014, in _read_packet\r\n    packet.check_error()\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/connections.py\", line 393, in check_error\r\n    err.raise_mysql_exception(self._data)\r\n  File \"/home/mrvx/.local/lib/python3.5/site-packages/pymysql/err.py\", line 107, in raise_mysql_exception\r\n    raise errorclass(errno, errval)\r\nsqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1054, \"Unknown column 'weibo_data.weibo_img_path' in 'field list'\") [SQL: 'SELECT weibo_data.id AS weibo_data_id, weibo_data.weibo_id AS weibo_data_weibo_id, weibo_data.weibo_cont AS weibo_data_weibo_cont, weibo_data.weibo_img AS weibo_data_weibo_img, weibo_data.weibo_img_path AS weibo_data_weibo_img_path, weibo_data.weibo_video AS weibo_data_weibo_video, weibo_data.repost_num AS weibo_data_repost_num, weibo_data.comment_num AS weibo_data_comment_num, weibo_data.praise_num AS weibo_data_praise_num, weibo_data.uid AS weibo_data_uid, weibo_data.is_origin AS weibo_data_is_origin, weibo_data.device AS weibo_data_device, weibo_data.weibo_url AS weibo_data_weibo_url, weibo_data.create_time AS weibo_data_create_time, weibo_data.comment_crawled AS weibo_data_comment_crawled, weibo_data.repost_crawled AS weibo_data_repost_crawled, weibo_data.dialogue_crawled AS weibo_data_dialogue_crawled, weibo_data.praise_crawled AS weibo_data_praise_crawled \\nFROM weibo_data \\nWHERE weibo_data.weibo_id = %(weibo_id_1)s \\n LIMIT %(param_1)s'] [parameters: {'weibo_id_1': '4196603482885035', 'param_1': 1}]\r\n^C\r\nworker: Hitting Ctrl+C again will terminate all running tasks!\r\n\r\nworker: Warm shutdown (MainProcess)"},"comment":{"url":"https://api.github.com/repos/SpiderClub/weibospider/issues/comments/419432041","html_url":"https://github.com/SpiderClub/weibospider/issues/124#issuecomment-419432041","issue_url":"https://api.github.com/repos/SpiderClub/weibospider/issues/124","id":419432041,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTQzMjA0MQ==","user":{"login":"thekingofcity","id":3353040,"node_id":"MDQ6VXNlcjMzNTMwNDA=","avatar_url":"https://avatars3.githubusercontent.com/u/3353040?v=4","gravatar_id":"","url":"https://api.github.com/users/thekingofcity","html_url":"https://github.com/thekingofcity","followers_url":"https://api.github.com/users/thekingofcity/followers","following_url":"https://api.github.com/users/thekingofcity/following{/other_user}","gists_url":"https://api.github.com/users/thekingofcity/gists{/gist_id}","starred_url":"https://api.github.com/users/thekingofcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thekingofcity/subscriptions","organizations_url":"https://api.github.com/users/thekingofcity/orgs","repos_url":"https://api.github.com/users/thekingofcity/repos","events_url":"https://api.github.com/users/thekingofcity/events{/privacy}","received_events_url":"https://api.github.com/users/thekingofcity/received_events","type":"User","site_admin":false},"created_at":"2018-09-07T13:01:32Z","updated_at":"2018-09-07T13:01:32Z","author_association":"MEMBER","body":"先检查下数据库weibo_data表中有无weibo_img_path字段\r\nPS. 虽然项目的表中有这个字段, 但是没有代码用到, 也没有任何一行是有内容的"}}
{"actor":{"display_login":"thekingofcity"}}