{"action":"created","issue":{"url":"https://api.github.com/repos/elastacloud/parquet-dotnet/issues/322","repository_url":"https://api.github.com/repos/elastacloud/parquet-dotnet","labels_url":"https://api.github.com/repos/elastacloud/parquet-dotnet/issues/322/labels{/name}","comments_url":"https://api.github.com/repos/elastacloud/parquet-dotnet/issues/322/comments","events_url":"https://api.github.com/repos/elastacloud/parquet-dotnet/issues/322/events","html_url":"https://github.com/elastacloud/parquet-dotnet/issues/322","id":348899616,"node_id":"MDU6SXNzdWUzNDg4OTk2MTY=","number":322,"title":"Tests that rely on CSV file comparison fails in locales with commas reversed","user":{"login":"slogen","id":1450856,"node_id":"MDQ6VXNlcjE0NTA4NTY=","avatar_url":"https://avatars2.githubusercontent.com/u/1450856?v=4","gravatar_id":"","url":"https://api.github.com/users/slogen","html_url":"https://github.com/slogen","followers_url":"https://api.github.com/users/slogen/followers","following_url":"https://api.github.com/users/slogen/following{/other_user}","gists_url":"https://api.github.com/users/slogen/gists{/gist_id}","starred_url":"https://api.github.com/users/slogen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slogen/subscriptions","organizations_url":"https://api.github.com/users/slogen/orgs","repos_url":"https://api.github.com/users/slogen/repos","events_url":"https://api.github.com/users/slogen/events{/privacy}","received_events_url":"https://api.github.com/users/slogen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-08T21:31:23Z","updated_at":"2018-08-08T22:17:58Z","closed_at":null,"author_association":"NONE","body":"<!--\r\n   PLEASE NOTE THAT WE WILL CLOSE THE ISSUE IMMEDIATELY IF THERE IS NOT ENOUGH INFORMATION.\r\n\r\n   Please Note that Elastacloud (https://elastacloud.com) provides commercial support for Parquet.Net, therefore if you need any professional advise or speedy development of new features and bugfixes please write to mailto:parquetsupport@elastacloud.com.\r\n\r\n-->\r\n\r\n**Version:** Parquet.Net v2.1 (and git, master 0a153b092040ccaa7de001daf2cb2ed1b09e2d0d)\r\n\r\n**Runtime Version:** N/A\r\n\r\n**OS:** N/A\r\n\r\n#### Expected behavior\r\n\r\nI expected all Unit tests to pass. \r\n\r\n#### Actual behavior\r\n\r\nAll 7 Unit tests in  Parquet.Test.Reader.TestDataTest all fail due to the same locale-dependent CSV parsing issue.\r\n\r\nThe applied parsing is not valid in all locales, as some locales have inverted the meaning of \".\" and \",\", as compared to en-US. This specifically happens in my locale (da-DK).\r\n\r\nThe CSV parser is not immediately fixable (uses Netbox and has no options for specifying the locale to parse). \r\n\r\nSix tests fails with the same message as `Alltypes_dictionary_gzipped`:\r\n\r\n```\r\nTest Name:\tParquet.Test.Reader.TestDataTest.Alltypes_dictionary_gzipped\r\nTest FullName:\tParquet.Test.Reader.TestDataTest.Alltypes_dictionary_gzipped\r\nTest Source:\tC:\\Users\\hej\\source\\repos\\parquet-dotnet\\src\\Parquet.Test\\Reader\\TestDataTest.cs : line 125\r\nTest Outcome:\tFailed\r\nTest Duration:\t0:00:00,159\r\n\r\nResult StackTrace:\t\r\nat Parquet.Test.Reader.ParquetCsvComparison.Compare(DataColumn[] parquet, DataColumn[] csv, Type[] columnTypes) in C:\\Users\\hej\\source\\repos\\parquet-dotnet\\src\\Parquet.Test\\Reader\\ParquetCsvComparison.cs:line 67\r\n   at Parquet.Test.Reader.ParquetCsvComparison.CompareFiles(String baseName, String encoding, Type[] columnTypes) in C:\\Users\\hej\\source\\repos\\parquet-dotnet\\src\\Parquet.Test\\Reader\\ParquetCsvComparison.cs:line 23\r\n   at Parquet.Test.Reader.TestDataTest.Alltypes_dictionary_gzipped() in C:\\Users\\hej\\source\\repos\\parquet-dotnet\\src\\Parquet.Test\\Reader\\TestDataTest.cs:line 127\r\nResult Message:\t\r\nexpected 11 but was 1,1 in column float_col, value #1\r\nExpected: True\r\nActual:   False\r\n```\r\n`Parquet.Test.Reader.TestDataTest.Postcodes_sample_no_compression` fails with\r\n\r\n```\r\nTest Name:\tParquet.Test.Reader.TestDataTest.Postcodes_sample_no_compression\r\nTest FullName:\tParquet.Test.Reader.TestDataTest.Postcodes_sample_no_compression\r\nTest Source:\tC:\\Users\\hej\\source\\repos\\parquet-dotnet\\src\\Parquet.Test\\Reader\\TestDataTest.cs : line 143\r\nTest Outcome:\tFailed\r\nTest Duration:\t0:00:00,009\r\n\r\nResult StackTrace:\t\r\nat Parquet.Test.Reader.ParquetCsvComparison.Compare(DataColumn[] parquet, DataColumn[] csv, Type[] columnTypes) in C:\\Users\\hej\\source\\repos\\parquet-dotnet\\src\\Parquet.Test\\Reader\\ParquetCsvComparison.cs:line 67\r\n   at Parquet.Test.Reader.ParquetCsvComparison.CompareFiles(String baseName, String encoding, Type[] columnTypes) in C:\\Users\\hej\\source\\repos\\parquet-dotnet\\src\\Parquet.Test\\Reader\\ParquetCsvComparison.cs:line 23\r\n   at Parquet.Test.Reader.TestDataTest.Postcodes_sample_no_compression() in C:\\Users\\hej\\source\\repos\\parquet-dotnet\\src\\Parquet.Test\\Reader\\TestDataTest.cs:line 145\r\nResult Message:\t\r\nexpected 57119458 but was 57,119458 in column Latitude, value #0\r\nExpected: True\r\nActual:   False\r\n```\r\n\r\n### Steps to reproduce the behavior\r\n\r\n1. Checkout code & compile\r\n2. Run unit-tests in locale da-DK\r\n\r\nThis can be a bit hard to do :) So I have forked and created a demo for it at https://github.com/slogen/parquet-dotnet/tree/bug/locale_dependent_csv_parsing\r\n\r\n#### Code snippet reproducing the behavior\r\n\r\n\r\nhttps://github.com/slogen/parquet-dotnet/commit/72b434b3a562eb87da8b38003c93fa7c849587ea\r\n\r\nThe demo utilizes a workaround, which is quite ugly, from https://github.com/xunit/samples.xunit/blob/master/UseCulture/UseCultureAttribute.cs.\r\n\r\n- [ ] I'm attaching a sample .parquet file containing the problem.\r\n- [x] I am happy to contribute to this repository by fixing this issue and raising a Pull Request (see [contribution guide](CONTRIBUTING.md))"},"comment":{"url":"https://api.github.com/repos/elastacloud/parquet-dotnet/issues/comments/411571167","html_url":"https://github.com/elastacloud/parquet-dotnet/issues/322#issuecomment-411571167","issue_url":"https://api.github.com/repos/elastacloud/parquet-dotnet/issues/322","id":411571167,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTU3MTE2Nw==","user":{"login":"aloneguid","id":3155189,"node_id":"MDQ6VXNlcjMxNTUxODk=","avatar_url":"https://avatars3.githubusercontent.com/u/3155189?v=4","gravatar_id":"","url":"https://api.github.com/users/aloneguid","html_url":"https://github.com/aloneguid","followers_url":"https://api.github.com/users/aloneguid/followers","following_url":"https://api.github.com/users/aloneguid/following{/other_user}","gists_url":"https://api.github.com/users/aloneguid/gists{/gist_id}","starred_url":"https://api.github.com/users/aloneguid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aloneguid/subscriptions","organizations_url":"https://api.github.com/users/aloneguid/orgs","repos_url":"https://api.github.com/users/aloneguid/repos","events_url":"https://api.github.com/users/aloneguid/events{/privacy}","received_events_url":"https://api.github.com/users/aloneguid/received_events","type":"User","site_admin":false},"created_at":"2018-08-08T22:17:58Z","updated_at":"2018-08-08T22:17:58Z","author_association":"COLLABORATOR","body":"@slogen thanks for reporting this. I agree the parsing could be better, either better parsing or culture workaround should be added. Are you happy to raise a PR?"}}
{"actor":{"display_login":"aloneguid"}}