{"action":"created","issue":{"url":"https://api.github.com/repos/rsyslog/rsyslog/issues/1782","repository_url":"https://api.github.com/repos/rsyslog/rsyslog","labels_url":"https://api.github.com/repos/rsyslog/rsyslog/issues/1782/labels{/name}","comments_url":"https://api.github.com/repos/rsyslog/rsyslog/issues/1782/comments","events_url":"https://api.github.com/repos/rsyslog/rsyslog/issues/1782/events","html_url":"https://github.com/rsyslog/rsyslog/issues/1782","id":259043482,"number":1782,"title":"Messages are stuck in rsyslog queue when using RELP delivery","user":{"login":"jay7x","id":11234613,"avatar_url":"https://avatars2.githubusercontent.com/u/11234613?v=4","gravatar_id":"","url":"https://api.github.com/users/jay7x","html_url":"https://github.com/jay7x","followers_url":"https://api.github.com/users/jay7x/followers","following_url":"https://api.github.com/users/jay7x/following{/other_user}","gists_url":"https://api.github.com/users/jay7x/gists{/gist_id}","starred_url":"https://api.github.com/users/jay7x/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jay7x/subscriptions","organizations_url":"https://api.github.com/users/jay7x/orgs","repos_url":"https://api.github.com/users/jay7x/repos","events_url":"https://api.github.com/users/jay7x/events{/privacy}","received_events_url":"https://api.github.com/users/jay7x/received_events","type":"User","site_admin":false},"labels":[{"id":778725387,"url":"https://api.github.com/repos/rsyslog/rsyslog/labels/worked-on","name":"worked-on","color":"6bc43a","default":false}],"state":"open","locked":false,"assignee":{"login":"rgerhards","id":1482123,"avatar_url":"https://avatars3.githubusercontent.com/u/1482123?v=4","gravatar_id":"","url":"https://api.github.com/users/rgerhards","html_url":"https://github.com/rgerhards","followers_url":"https://api.github.com/users/rgerhards/followers","following_url":"https://api.github.com/users/rgerhards/following{/other_user}","gists_url":"https://api.github.com/users/rgerhards/gists{/gist_id}","starred_url":"https://api.github.com/users/rgerhards/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rgerhards/subscriptions","organizations_url":"https://api.github.com/users/rgerhards/orgs","repos_url":"https://api.github.com/users/rgerhards/repos","events_url":"https://api.github.com/users/rgerhards/events{/privacy}","received_events_url":"https://api.github.com/users/rgerhards/received_events","type":"User","site_admin":false},"assignees":[{"login":"rgerhards","id":1482123,"avatar_url":"https://avatars3.githubusercontent.com/u/1482123?v=4","gravatar_id":"","url":"https://api.github.com/users/rgerhards","html_url":"https://github.com/rgerhards","followers_url":"https://api.github.com/users/rgerhards/followers","following_url":"https://api.github.com/users/rgerhards/following{/other_user}","gists_url":"https://api.github.com/users/rgerhards/gists{/gist_id}","starred_url":"https://api.github.com/users/rgerhards/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rgerhards/subscriptions","organizations_url":"https://api.github.com/users/rgerhards/orgs","repos_url":"https://api.github.com/users/rgerhards/repos","events_url":"https://api.github.com/users/rgerhards/events{/privacy}","received_events_url":"https://api.github.com/users/rgerhards/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rsyslog/rsyslog/milestones/26","html_url":"https://github.com/rsyslog/rsyslog/milestone/26","labels_url":"https://api.github.com/repos/rsyslog/rsyslog/milestones/26/labels","id":2936552,"number":26,"title":"v8.35","description":"","creator":{"login":"rgerhards","id":1482123,"avatar_url":"https://avatars3.githubusercontent.com/u/1482123?v=4","gravatar_id":"","url":"https://api.github.com/users/rgerhards","html_url":"https://github.com/rgerhards","followers_url":"https://api.github.com/users/rgerhards/followers","following_url":"https://api.github.com/users/rgerhards/following{/other_user}","gists_url":"https://api.github.com/users/rgerhards/gists{/gist_id}","starred_url":"https://api.github.com/users/rgerhards/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rgerhards/subscriptions","organizations_url":"https://api.github.com/users/rgerhards/orgs","repos_url":"https://api.github.com/users/rgerhards/repos","events_url":"https://api.github.com/users/rgerhards/events{/privacy}","received_events_url":"https://api.github.com/users/rgerhards/received_events","type":"User","site_admin":false},"open_issues":89,"closed_issues":12,"state":"open","created_at":"2017-11-25T09:50:15Z","updated_at":"2018-04-11T06:27:03Z","due_on":"2018-05-11T07:00:00Z","closed_at":null},"comments":30,"created_at":"2017-09-20T05:25:12Z","updated_at":"2018-04-12T09:08:46Z","closed_at":null,"author_association":"NONE","body":"I have following action configured in ruleset on API server. This ruleset receive messages from PHP process (`fwrite()` to `stream_socket_client()`-connected socket) via `imuxsock` input.\r\n\r\n```\r\n  action(type=\"omrelp\" name=\"XXX_sock_relay\"\r\n    target=\"10.X.X.X\"\r\n    port=\"2510\"\r\n    tls=\"off\"\r\n    tls.compression=\"off\"\r\n    template=\"XXX_relay_msg\"\r\n    action.reportSuspensionContinuation=\"on\"\r\n    action.resumeRetryCount=\"-1\"\r\n    action.resumeInterval=\"10\"\r\n    queue.type=\"fixedArray\"\r\n    queue.size=\"250000\"\r\n    queue.dequeueBatchSize=\"1024\"\r\n    queue.workerThreads=\"4\"\r\n    queue.workerThreadMinimumMessages=\"50000\"\r\n    queue.spoolDirectory=\"/shop/logs/rsyslog\"\r\n    queue.fileName=\"XXX_sock\"\r\n    queue.maxFileSize=\"16m\"\r\n    queue.maxDiskSpace=\"2G\"\r\n    queue.highWaterMark=\"200000\"\r\n    queue.lowWaterMark=\"100000\"\r\n    queue.checkpointInterval=\"30000\"\r\n    queue.saveOnShutdown=\"on\"\r\n    queue.timeoutEnqueue=\"1\"\r\n  )\r\n```\r\n\r\nFew days ago I've started to investigate why this queue `XXX_sock_relay` always have 1000-4000 messages in it when other API's queues have < 10 messages usually.\r\n\r\nI've realised this queue on every API server have few old messages stuck in it. Every few mins those messages are delivered to log collector. But those messages seems are not marked as delivered after! So they are remaining in queue to be delivered again and again. Delivery attempt happens every 10s.\r\n\r\nI have saved tcpdump for this but it contains private info I don't like to put in public. So if you're interested in investigation of this case please let's contact directly.\r\n\r\nI've switched partially to TCP delivery (omfwd/imptcp). There is no issues at all. So it's exactly about `omrelp`/`imrelp`.\r\n\r\nCentOS 7 x86_64\r\nrsyslog 8.27\r\nlibrelp 1.2.14"},"comment":{"url":"https://api.github.com/repos/rsyslog/rsyslog/issues/comments/380733061","html_url":"https://github.com/rsyslog/rsyslog/issues/1782#issuecomment-380733061","issue_url":"https://api.github.com/repos/rsyslog/rsyslog/issues/1782","id":380733061,"user":{"login":"davidelang","id":2208335,"avatar_url":"https://avatars1.githubusercontent.com/u/2208335?v=4","gravatar_id":"","url":"https://api.github.com/users/davidelang","html_url":"https://github.com/davidelang","followers_url":"https://api.github.com/users/davidelang/followers","following_url":"https://api.github.com/users/davidelang/following{/other_user}","gists_url":"https://api.github.com/users/davidelang/gists{/gist_id}","starred_url":"https://api.github.com/users/davidelang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidelang/subscriptions","organizations_url":"https://api.github.com/users/davidelang/orgs","repos_url":"https://api.github.com/users/davidelang/repos","events_url":"https://api.github.com/users/davidelang/events{/privacy}","received_events_url":"https://api.github.com/users/davidelang/received_events","type":"User","site_admin":false},"created_at":"2018-04-12T09:08:46Z","updated_at":"2018-04-12T09:08:46Z","author_association":"CONTRIBUTOR","body":"On Thu, 12 Apr 2018, Rainer Gerhards wrote:\n\n> just as an update: a root cause of the problem is that imfile did not properly \n> obey maxmessage size - and maybe some other parts of the core engine. I also \n> need to think on a method how we can really enforcee maxmessage size - we \n> should not permit any message larger than the configured max to be able to \n> float through the engine.\n\nThe problem is what to do when you hit a size limit. Especially if it's a matter \nof the output of a template being too large as a result of including a JSON tree \nin it.\n\nAs an admin, I want the ability to figure out that the message is too large to \nsend remotely, adn write it to a local file and modify the message to send a \nsmaller version to the remote destination\n"}}
{"actor":{"display_login":"davidelang"}}