{"action":"created","issue":{"url":"https://api.github.com/repos/sympa-community/sympa/issues/305","repository_url":"https://api.github.com/repos/sympa-community/sympa","labels_url":"https://api.github.com/repos/sympa-community/sympa/issues/305/labels{/name}","comments_url":"https://api.github.com/repos/sympa-community/sympa/issues/305/comments","events_url":"https://api.github.com/repos/sympa-community/sympa/issues/305/events","html_url":"https://github.com/sympa-community/sympa/issues/305","id":323229914,"number":305,"title":"Encoding issue","user":{"login":"ldidry","id":922350,"avatar_url":"https://avatars0.githubusercontent.com/u/922350?v=4","gravatar_id":"","url":"https://api.github.com/users/ldidry","html_url":"https://github.com/ldidry","followers_url":"https://api.github.com/users/ldidry/followers","following_url":"https://api.github.com/users/ldidry/following{/other_user}","gists_url":"https://api.github.com/users/ldidry/gists{/gist_id}","starred_url":"https://api.github.com/users/ldidry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ldidry/subscriptions","organizations_url":"https://api.github.com/users/ldidry/orgs","repos_url":"https://api.github.com/users/ldidry/repos","events_url":"https://api.github.com/users/ldidry/events{/privacy}","received_events_url":"https://api.github.com/users/ldidry/received_events","type":"User","site_admin":false},"labels":[{"id":521819377,"url":"https://api.github.com/repos/sympa-community/sympa/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/sympa-community/sympa/milestones/5","html_url":"https://github.com/sympa-community/sympa/milestone/5","labels_url":"https://api.github.com/repos/sympa-community/sympa/milestones/5/labels","id":3050926,"number":5,"title":"6.2.34","description":"New release of Sympa \"legacy\" version. Day of next stable release will be at the date of northern solstice, Thursday, 21st June UTC. Debug phase will start about seven weeks before, around Thursday, 3rd May.","creator":{"login":"ikedas","id":5743546,"avatar_url":"https://avatars2.githubusercontent.com/u/5743546?v=4","gravatar_id":"","url":"https://api.github.com/users/ikedas","html_url":"https://github.com/ikedas","followers_url":"https://api.github.com/users/ikedas/followers","following_url":"https://api.github.com/users/ikedas/following{/other_user}","gists_url":"https://api.github.com/users/ikedas/gists{/gist_id}","starred_url":"https://api.github.com/users/ikedas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ikedas/subscriptions","organizations_url":"https://api.github.com/users/ikedas/orgs","repos_url":"https://api.github.com/users/ikedas/repos","events_url":"https://api.github.com/users/ikedas/events{/privacy}","received_events_url":"https://api.github.com/users/ikedas/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":13,"state":"open","created_at":"2018-01-21T03:40:22Z","updated_at":"2018-05-18T09:57:58Z","due_on":"2018-06-21T07:00:00Z","closed_at":null},"comments":14,"created_at":"2018-05-15T13:57:20Z","updated_at":"2018-05-23T01:26:40Z","closed_at":null,"author_association":"NONE","body":"**Disclaimer** I provided the problematic mail to some Sympa developers but will not provide it here since it contains personal informations. And I anonymized the informations in this bug report\r\n\r\nFor the record, here's a message I posted on sympa-fr:\r\nhttps://listes.renater.fr/sympa/arc/sympa-fr/2017-10/msg00015.html\r\n\r\nMy configuration:\r\nDebian 9, PostgreSQL 9.6, Sympa 6.2.16\r\n\r\nLocales configuration:\r\n```\r\n# locale\r\nLANG=fr_FR.UTF-8\r\nLANGUAGE=\r\nLC_CTYPE=\"fr_FR.UTF-8\"\r\nLC_NUMERIC=\"fr_FR.UTF-8\"\r\nLC_TIME=\"fr_FR.UTF-8\"\r\nLC_COLLATE=\"fr_FR.UTF-8\"\r\nLC_MONETARY=\"fr_FR.UTF-8\"\r\nLC_MESSAGES=\"fr_FR.UTF-8\"\r\nLC_PAPER=\"fr_FR.UTF-8\"\r\nLC_NAME=\"fr_FR.UTF-8\"\r\nLC_ADDRESS=\"fr_FR.UTF-8\"\r\nLC_TELEPHONE=\"fr_FR.UTF-8\"\r\nLC_MEASUREMENT=\"fr_FR.UTF-8\"\r\nLC_IDENTIFICATION=\"fr_FR.UTF-8\"\r\nLC_ALL=\r\n```\r\n\r\nFor the DB:\r\n```\r\n    Nom    | Propriétaire | Encodage | Collationnement | Type caract.\r\n-----------+--------------+----------+-----------------+-------------\r\n sympa     | sympa        | UTF8     | fr_FR.UTF-8     | fr_FR.UTF-8\r\n```\r\n\r\nSome characters crash sympa_msg: `’` or `—`. By deleting them, there's\r\nno more crash.\r\n\r\nSome characters are badly encoded in the database, even simple `é`\r\n(which are current in french first name).\r\n\r\nHere's a crash log:\r\n```\r\nDIED: Cannot decode string with wide characters at /usr/lib/x86_64-linux-gnu/perl/5.24/Encode.pm line 243.\r\n  at /usr/lib/x86_64-linux-gnu/perl/5.24/Encode.pm line 243.\r\n         Encode::decode_utf8('XXXXXX Fran\\x{fffd}ois') called at /home/sympa/bin/Sympa/DatabaseDriver/PostgreSQL.pm line 93\r\n         \r\nSympa::DatabaseDriver::PostgreSQL::do_prepared_query(Sympa::DatabaseDriver::PostgreSQL <db_host=localhost;db_name=XXX;db_port=5432;db_user=XXX>, 'INSERT INTO subscriber_table\\x{a}                      (user_subs...', 'XXX@XXXX.net', 'XXXX Fran\\x{fffd}ois', 'list_name', 'framalistes.org', 'mail', undef, ...) called at /home/sympa/bin/Sympa/List.pm line 3982\r\n         Sympa::List::add_list_member(Sympa::List <list_name@framalistes.org>, HASH(0x558ee2091a70)) called at /home/sympa/bin/Sympa/Request/Handler/add.pm line 76\r\n         \r\nSympa::Request::Handler::add::_twist(Sympa::Spindle::ProcessMessage=HASH(0x558ee241a6c8), Sympa::Request <action=add;context=list_name@framalistes.org;email=XXX@XXX.net>) called at /home/sympa/bin/Sympa/Spindle.pm line 92\r\n         \r\nSympa::Spindle::spin(Sympa::Spindle::ProcessMessage=HASH(0x558ee241a6c8)) called at /home/sympa/bin/Sympa/Spindle/DoCommand.pm line 117\r\n         \r\nSympa::Spindle::DoCommand::_twist(Sympa::Spindle::ProcessIncoming=HASH(0x558ee244b0b0), Sympa::Message <sympa@framalistes.org.1515358089.11027>) called at /home/sympa/bin/Sympa/Spindle.pm line 92\r\n         \r\nSympa::Spindle::spin(Sympa::Spindle::ProcessIncoming=HASH(0x558ee244b0b0)) called at /home/sympa/bin/sympa_msg.pl line 240\r\n```\r\n\r\nI digged a little with a small perl script. This one don't work:\r\n```perl\r\n#!/usr/bin/perl\r\nuse strict;\r\nuse warnings;\r\nuse 5.10.0;\r\nuse Encode qw(decode_utf8);\r\n\r\nopen my $fh, '<', 'sympa@framalistes.org.1515533067.24289';\r\nwhile(defined(my $foo = <$fh>)) {\r\n     chomp $foo;\r\n     $foo = decode_utf8($foo);\r\n     say $foo if $foo =~ m/XXX.jean-rene/;\r\n}\r\n```\r\n\r\n```\r\n% ./test.pl\r\nQUIET ADD list_name XXX.jean-rene@XXX.fr Jean-Ren�\r\n```\r\n\r\nBut it works with binmode:\r\n```perl\r\n#!/usr/bin/perl\r\nuse strict;\r\nuse warnings;\r\nuse 5.10.0;\r\n\r\nopen my $fh, '<', 'sympa@framalistes.org.1515533067.24289';\r\nwhile(defined(my $foo = <$fh>)) {\r\n     binmode(STDIN,  \":utf8\");\r\n     chomp $foo;\r\n     say $foo if $foo =~ m/XXX.jean-rene/;\r\n}\r\n```\r\n\r\n```\r\n% ./test.pl\r\nQUIET ADD list_name XXX.jean-rene@XXX.fr Jean-René\r\n```\r\n\r\nThat's all that I have and found.\r\n"},"comment":{"url":"https://api.github.com/repos/sympa-community/sympa/issues/comments/391191091","html_url":"https://github.com/sympa-community/sympa/issues/305#issuecomment-391191091","issue_url":"https://api.github.com/repos/sympa-community/sympa/issues/305","id":391191091,"user":{"login":"ikedas","id":5743546,"avatar_url":"https://avatars2.githubusercontent.com/u/5743546?v=4","gravatar_id":"","url":"https://api.github.com/users/ikedas","html_url":"https://github.com/ikedas","followers_url":"https://api.github.com/users/ikedas/followers","following_url":"https://api.github.com/users/ikedas/following{/other_user}","gists_url":"https://api.github.com/users/ikedas/gists{/gist_id}","starred_url":"https://api.github.com/users/ikedas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ikedas/subscriptions","organizations_url":"https://api.github.com/users/ikedas/orgs","repos_url":"https://api.github.com/users/ikedas/repos","events_url":"https://api.github.com/users/ikedas/events{/privacy}","received_events_url":"https://api.github.com/users/ikedas/received_events","type":"User","site_admin":false},"created_at":"2018-05-23T01:26:40Z","updated_at":"2018-05-23T01:26:40Z","author_association":"MEMBER","body":"Ok, if both 3.6.x and 3.7.4 crashes, could you please apply [this additional patch](https://github.com/sympa-community/sympa/commit/bcc5789) and check if problem will be solved? Thanks.\r\n"}}
{"actor":{"display_login":"ikedas"}}