{"action":"opened","issue":{"url":"https://api.github.com/repos/slashdoom/cdr_parser/issues/3","repository_url":"https://api.github.com/repos/slashdoom/cdr_parser","labels_url":"https://api.github.com/repos/slashdoom/cdr_parser/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/slashdoom/cdr_parser/issues/3/comments","events_url":"https://api.github.com/repos/slashdoom/cdr_parser/issues/3/events","html_url":"https://github.com/slashdoom/cdr_parser/issues/3","id":170921337,"number":3,"title":"Bulk Indexing will issue timeout from ES","user":{"login":"dhyeem","id":18233818,"avatar_url":"https://avatars.githubusercontent.com/u/18233818?v=3","gravatar_id":"","url":"https://api.github.com/users/dhyeem","html_url":"https://github.com/dhyeem","followers_url":"https://api.github.com/users/dhyeem/followers","following_url":"https://api.github.com/users/dhyeem/following{/other_user}","gists_url":"https://api.github.com/users/dhyeem/gists{/gist_id}","starred_url":"https://api.github.com/users/dhyeem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhyeem/subscriptions","organizations_url":"https://api.github.com/users/dhyeem/orgs","repos_url":"https://api.github.com/users/dhyeem/repos","events_url":"https://api.github.com/users/dhyeem/events{/privacy}","received_events_url":"https://api.github.com/users/dhyeem/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-12T17:36:12Z","updated_at":"2016-08-12T17:36:12Z","closed_at":null,"body":"good Day Slashdoom, \r\ni have test the script with the new function you added, it is great but i face some issue i want you to know about. i have test your script with almost 10,000 CDR and CMR, i face two issue: \r\n\r\n- High CPU and memory after reaching 3000 file, the CPU raised to be 100%.\r\n- After a while the ES will issue timeout below you will find the log : \r\n\r\n\r\n\r\n> root@dcpelksap21:/home/dhyeem# curl localhost:9200\r\n{\r\n  \"name\" : \"Baron Zemo\",\r\n  \"cluster_name\" : \"elasticsearch\",\r\n  \"version\" : {\r\n    \"number\" : \"2.3.3\",\r\n    \"build_hash\" : \"218bdf10790eef486ff2c41a3df5cfa32dadcfde\",\r\n    \"build_timestamp\" : \"2016-05-17T15:40:04Z\",\r\n    \"build_snapshot\" : false,\r\n    \"lucene_version\" : \"5.5.0\"\r\n  },\r\n  \"tagline\" : \"You Know, for Search\"\r\n}\r\nroot@dcpelksap21:/home/dhyeem# /opt/cdr_parser/init.d/cdr_parser start\r\nStarting server\r\nroot@dcpelksap21:/home/dhyeem# started with pid 2411\r\n2016-08-12 14:41:38,431 - cdr_parser started\r\n2016-08-12 14:41:38,561 - src_file = /home/dhyeem/logs/cmr_StandAloneCluster_03_201602180657_65425\r\n2016-08-12 14:41:38,561 - dest_file = /home/dhyeem/logs/archive/cmr_StandAloneCluster_03_201602180657_65425\r\n2016-08-12 14:41:38,561 - es_type = cmr_StandAloneCluster_03_201602180657_65425\r\n2016-08-12 14:41:38,846 - 0 documents found\r\n2016-08-12 14:41:39,347 - CDR successfully moved to archive.\r\n2016-08-12 14:41:39,348 - built es_index: cmr-2016.02.18\r\nTraceback (most recent call last):\r\n  File \"/opt/cdr_parser/cdr_parser.py\", line 63, in <module>\r\n    daemon_runner.do_action()\r\n  File \"/usr/local/lib/python2.7/dist-packages/daemon/runner.py\", line 267, in do_action\r\n    func(self)\r\n  File \"/usr/local/lib/python2.7/dist-packages/daemon/runner.py\", line 186, in _start\r\n    self.app.run()\r\n  File \"/opt/cdr_parser/cdr_parser.py\", line 39, in run\r\n    do_main_program()\r\n  File \"/opt/cdr_parser/mod_cdr_main.py\", line 257, in do_main_program\r\n    logger.debug(es.index(index=es_index,doc_type=es_type,body=es_body))\r\n  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py\", line 69, in _wrapped\r\n    return func(*args, params=params, **kwargs)\r\n  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py\", line 279, in index\r\n    _make_path(index, doc_type, id), params=params, body=body)\r\n  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py\", line 329, in perform_request\r\n    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)\r\n  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py\", line 102, in perform_request\r\n    raise ConnectionTimeout('TIMEOUT', str(e), e)\r\nelasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='localhost', port='9200'): Read timed out. (read timeout=10))\r\n^C\r\nroot@dcpelksap21:/home/dhyeem# /opt/cdr_parser/init.d/cdr_parser stop\r\nStopping server\r\nTraceback (most recent call last):\r\n  File \"/opt/cdr_parser/cdr_parser.py\", line 63, in <module>\r\n    daemon_runner.do_action()\r\n  File \"/usr/local/lib/python2.7/dist-packages/daemon/runner.py\", line 267, in do_action\r\n    func(self)\r\n  File \"/usr/local/lib/python2.7/dist-packages/daemon/runner.py\", line 217, in _stop\r\n    raise error\r\ndaemon.runner.DaemonRunnerStopFailureError: PID file '/var/run/cdr_parser.pid' not locked\r\n\r\nServer info : \r\nOperating system: \tUbuntu Linux 16.04\r\nKernel and CPU: \tLinux 4.4.0-31-generic on x86_64\r\nProcessor information: \tIntel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz, 2 cores\r\nRAM : 4GB \r\n\r\nRegards,"}}
{"actor":{"display_login":"dhyeem"}}