{"action":"created","issue":{"url":"https://api.github.com/repos/rsyslog/rsyslog/issues/2655","repository_url":"https://api.github.com/repos/rsyslog/rsyslog","labels_url":"https://api.github.com/repos/rsyslog/rsyslog/issues/2655/labels{/name}","comments_url":"https://api.github.com/repos/rsyslog/rsyslog/issues/2655/comments","events_url":"https://api.github.com/repos/rsyslog/rsyslog/issues/2655/events","html_url":"https://github.com/rsyslog/rsyslog/issues/2655","id":315500793,"number":2655,"title":"rsyslogd creates corrupted queue files","user":{"login":"sgiessl","id":253190,"avatar_url":"https://avatars2.githubusercontent.com/u/253190?v=4","gravatar_id":"","url":"https://api.github.com/users/sgiessl","html_url":"https://github.com/sgiessl","followers_url":"https://api.github.com/users/sgiessl/followers","following_url":"https://api.github.com/users/sgiessl/following{/other_user}","gists_url":"https://api.github.com/users/sgiessl/gists{/gist_id}","starred_url":"https://api.github.com/users/sgiessl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sgiessl/subscriptions","organizations_url":"https://api.github.com/users/sgiessl/orgs","repos_url":"https://api.github.com/users/sgiessl/repos","events_url":"https://api.github.com/users/sgiessl/events{/privacy}","received_events_url":"https://api.github.com/users/sgiessl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-18T14:19:41Z","updated_at":"2018-04-18T19:35:10Z","closed_at":null,"author_association":"NONE","body":"### Expected behavior\r\n`rsyslogd` creates queue files in `/var/spool/rsyslog` in a custom ASCII based file format. The persistent queue is used to replay messages that could not be delivered while the machine was offline, even throughout machine reboots.\r\n\r\n### Actual behavior\r\nRunning this in larger scale seems to lead to corrupted queue files sporadically (leading to issues described in https://github.com/rsyslog/rsyslog/issues/2654 ). Some examples follow:\r\n\r\nNotice duplicated attributes such as `tRcvdAt` etc.:\r\n```\r\n<Obj:1:msg:1:\r\n+iProtocolVersion:2:1:0:\r\n+iSeverity:2:1:6:\r\n+iFacility:2:1:3:\r\n+msgFlags:2:2:36:\r\n+ttGenTime:2:10:1523292412:\r\n+tRcvdAt:3:34:2:2018:4:9:18:46:52:133567:6:+:2:0:\r\n+tTIMESTAMP:3:34:2:2018:4:9:18:46:52:133567:6:+:2:0:\r\n+pszTAG:1:17:ma<Obj:1:msg:1:\r\n+iProtocolVersion:2:1:0:\r\n+iSeverity:2:1:6:\r\n+iFacility:2:1:3:\r\n+msgFlags:2:2:36:\r\n+ttGenTime:2:10:1523292412:\r\n+tRcvdAt:3:34:2:2018:4:9:18:46:52:522920:6:+:2:0:\r\n+tTIMESTAMP:3:34:2:2018:4:9:18:46:52:522920:6:+:2:0:\r\n+pszTAG:1:17:aaaaaaaaaaa[228]::\r\n+pszRawMsg:1:147:<30>Apr  9 18:46:52 xxxxxxxxxxx[228]: [2018-04-09 16:46:50 1523292410000]   DEBUG:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:\r\n+pszInputName:1:8:imuxsock:\r\n+pszRcvFrom:1:19:0000-0000-0001-1224:\r\n+pszRcvFromIP:1:9:127.0.0.1:\r\n+offMSG:2:2:37:\r\n>End\r\n.\r\n```\r\n\r\nNotice the `<Obj` tag opening within an incomplete object definition's `pszTAG` attribute:\r\n```\r\n<Obj:1:msg:1:\r\n+iProtocolVersion:2:1:0:\r\n+iSeverity:2:1:6:\r\n+iFacility:2:1:3:\r\n+msgFlags:2:2:36:\r\n+ttGenTime:2:10:1523889804:\r\n+tRcvdAt:3:35:2:2018:4:16:16:43:24:102767:6:+:2:0:\r\n+tTIMESTAMP:3:35:2:2018:4:16:16:43:24:102767:6:+:2:0:\r\n+pszTAG:1:17:<Obj:1:msg:1:\r\n+iProtocolVersion:2:1:0:\r\n+iSeverity:2:1:6:\r\n+iFacility:2:1:3:\r\n+msgFlags:2:2:36:\r\n+ttGenTime:2:10:1523889740:\r\n+tRcvdAt:3:35:2:2018:4:16:16:42:20:294647:6:+:2:0:\r\n+tTIMESTAMP:3:35:2:2018:4:16:16:42:20:294647:6:+:2:0:\r\n+pszTAG:1:17:machine-gui[216]::\r\n+pszRawMsg:1:137:<30>Apr 16 16:42:20 xxxxxxxxxxx[216]: [2018-04-16 14:42:20 1523889740234]   DEBUG:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:\r\n+pszInputName:1:8:imuxsock:\r\n+pszRcvFrom:1:19:0000-0000-0001-2205:\r\n+pszRcvFromIP:1:9:127.0.0.1:\r\n+offMSG:2:2:37:\r\n>End\r\n.\r\n```\r\n\r\n### Steps to reproduce the behavior\r\n\r\nActually this is probably not easy to reproduce - we are running this configuration on a few hundret of embedded devices and within a period of about 2 months, about 40% were suffering symptoms that would indicate this cause. Please let me know how I can provide you with more relevant information on this, beyond the environment information below.\r\n\r\n### Environment\r\n- rsyslog version: 8.33.0\r\n- platform: Yocto embedded linux, based on Boot2Qt v5.9.0 (which is based on mortz-16.0.1). Rsyslog packages are customized to get the more recent version 8.33.\r\n- system configuration: Ext4 file system (journal_data mount option)\r\n- see more files below\r\n\r\nrsyslogd.conf:\r\n```\r\n#\r\n# General setup\r\n#\r\n$ModLoad immark\r\n$ModLoad imuxsock\r\n$ModLoad imklog\r\n\r\n$FileOwner root\r\n$FileGroup adm\r\n$FileCreateMode 0640\r\n$DirCreateMode 0755\r\n$Umask 0022\r\n\r\n#\r\n# User message emergency logging\r\n#\r\n*.emerg :omusrmsg:*\r\n\r\n#\r\n# Remote logging to Shizuoka\r\n#\r\n$WorkDirectory /var/spool/rsyslog\r\n$ActionQueueFileName uniqName\r\n$ActionQueueMaxDiskSpace 500m\r\n$ActionQueueSaveOnShutdown on\r\n$ActionQueueType LinkedList\r\n$ActionResumeRetryCount -1\r\n*.* @@10.8.1.10:514\r\n```"},"comment":{"url":"https://api.github.com/repos/rsyslog/rsyslog/issues/comments/382503744","html_url":"https://github.com/rsyslog/rsyslog/issues/2655#issuecomment-382503744","issue_url":"https://api.github.com/repos/rsyslog/rsyslog/issues/2655","id":382503744,"user":{"login":"davidelang","id":2208335,"avatar_url":"https://avatars1.githubusercontent.com/u/2208335?v=4","gravatar_id":"","url":"https://api.github.com/users/davidelang","html_url":"https://github.com/davidelang","followers_url":"https://api.github.com/users/davidelang/followers","following_url":"https://api.github.com/users/davidelang/following{/other_user}","gists_url":"https://api.github.com/users/davidelang/gists{/gist_id}","starred_url":"https://api.github.com/users/davidelang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidelang/subscriptions","organizations_url":"https://api.github.com/users/davidelang/orgs","repos_url":"https://api.github.com/users/davidelang/repos","events_url":"https://api.github.com/users/davidelang/events{/privacy}","received_events_url":"https://api.github.com/users/davidelang/received_events","type":"User","site_admin":false},"created_at":"2018-04-18T19:35:10Z","updated_at":"2018-04-18T19:35:10Z","author_association":"CONTRIBUTOR","body":"was this from a system crash, rsyslog being killed, or normal rsyslog exit?\n"}}
{"actor":{"display_login":"davidelang"}}