{"action":"created","issue":{"url":"https://api.github.com/repos/axboe/fio/issues/673","repository_url":"https://api.github.com/repos/axboe/fio","labels_url":"https://api.github.com/repos/axboe/fio/issues/673/labels{/name}","comments_url":"https://api.github.com/repos/axboe/fio/issues/673/comments","events_url":"https://api.github.com/repos/axboe/fio/issues/673/events","html_url":"https://github.com/axboe/fio/issues/673","id":357625436,"node_id":"MDU6SXNzdWUzNTc2MjU0MzY=","number":673,"title":"fio output intermittently scrambled when multiple --client options specified","user":{"login":"jml9904","id":8404469,"node_id":"MDQ6VXNlcjg0MDQ0Njk=","avatar_url":"https://avatars1.githubusercontent.com/u/8404469?v=4","gravatar_id":"","url":"https://api.github.com/users/jml9904","html_url":"https://github.com/jml9904","followers_url":"https://api.github.com/users/jml9904/followers","following_url":"https://api.github.com/users/jml9904/following{/other_user}","gists_url":"https://api.github.com/users/jml9904/gists{/gist_id}","starred_url":"https://api.github.com/users/jml9904/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jml9904/subscriptions","organizations_url":"https://api.github.com/users/jml9904/orgs","repos_url":"https://api.github.com/users/jml9904/repos","events_url":"https://api.github.com/users/jml9904/events{/privacy}","received_events_url":"https://api.github.com/users/jml9904/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2018-09-06T12:08:30Z","updated_at":"2018-09-07T18:11:34Z","closed_at":null,"author_association":"NONE","body":"Running into an issue where output is interleaved among client/server runs. This is somewhat intermittent but has occurred across multiple fio versions. For example, starting two fio servers on a host (this is from a single host, but typically we see the problem when running servers on two separate hosts, and the client on a third):\r\n\r\nfio --server, 8765\r\nfio --server, 8766\r\n\r\nand then running:\r\n\r\n./fio --minimal --eta=never --client=xx.xx.xx.xx,8765 ./00test1.fio --client=xx.xx.xx.xx,8766 ./00test1.fio \r\n\r\nyields the following output:\r\n\r\nhostname=gorgonzola, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.5-59-g7c54, flags=1\r\nhostname=gorgonzola, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-3.5-59-g7c54, flags=1\r\n<gorgonzola> 00-test1: (g=0): rw=randrw, <gorgonzola> 00-test1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64\r\nioengine=libaio, iodepth=64\r\n<gorgonzola> Starting <gorgonzola> Starting 1 process1 process\r\n\r\n3;fio-3.5-59-g7c54;00-test1;0;0;3208676;320835;80208;10001;2;71;3.275114;1.275014;8;7206;242.193155;305.997537;1.000000%=115;5.000000%=134;10.000000%=144;20.000000%=158;30.000000%=168;40.000000%\r\n<... stats deleted for clarity ...>\r\n\r\nDisk stats (read/write):\r\n;nvme0n1;1584561;1583508;0;0;309780;781008;1100100;99.38%<gorgonzola> \r\n3;fio-3.5-59-g7c54;00-test1;0;0;3203416;320245;80061;10003;2;39;3.275121;1.244327;21;7556;241.026224;306.086266;1.000000%=114;5.000000%=132;10.000000%=144;20.000000%=156;30.000000%=168;40.000000%=179;50.000000%=189;60.000000%=199;70.000000%=214;80.000000%=232;90.000000%=26\r\n<... stats deleted for clarity ...>\r\n\r\nclients;0;0;6412092;641016;160254;10003;2;71;3.275121;0.879511;8;7556;241.026224;216.346815;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%\r\n<... stats deleted for clarity ...>\r\n\r\nDisk stats (read/write):\r\n;nvme0n1;1584570;1583514;0;0;309780;781008;1100100;99.38%<gorgonzola> \r\n\r\nThe information at the head of the output is scrambled, and sometimes we get corruption within the body too, as in:\r\n\r\n3;fio-3.5-45-g69b98f;All clients;0;0;280711168;20338441;4965;13802;99;2882701;16072.372432;73958.321310;2947;9798\r\n<... stats deleted for clarity ...>\r\n;5.5%;89.2%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.10%;0.57%;3.06%;3.03%;12.69%;12.53%;10.44%;9.88%;0.00%;0.00%\r\n;nvme2n1;55808;22944;0;0;5167415;2646201;7832048;64.34%;nvme16n1;54080;24672;0;0;13073942;6934592;20772754;67.79%\r\nDisk stats (read/write):\r\n;nvme19n1;53561;24219;0;0;106317193;49912036;186159849;97.82%;nvme13n1;53984;24768;0;0;7390954;4233313;11808161;66.83%;nvme8n1;53108;22066;0;0;10262887;4664063;15155532;61.38%;nvme14n1;55904;22848;0;0;7006908;3108602;10194615;64.81%;nvme10n1;52904;22270;0;0;5889436;2925676;8899389;60.74%<FNM00181400121-A> \r\n\r\nThat ;nvme2n1 line is from client B, and doesn't belong where it is.\r\n\r\nI think the best solution might be to add an option that would open a separate output file for each server stream (and possibly concatenate them into a single file at the termination of the run), although there might be other ways to accomplish it. Is this a known issue, and has anyone else reported it?\r\n\r\nThanks!\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/axboe/fio/issues/comments/419522236","html_url":"https://github.com/axboe/fio/issues/673#issuecomment-419522236","issue_url":"https://api.github.com/repos/axboe/fio/issues/673","id":419522236,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTUyMjIzNg==","user":{"login":"jml9904","id":8404469,"node_id":"MDQ6VXNlcjg0MDQ0Njk=","avatar_url":"https://avatars1.githubusercontent.com/u/8404469?v=4","gravatar_id":"","url":"https://api.github.com/users/jml9904","html_url":"https://github.com/jml9904","followers_url":"https://api.github.com/users/jml9904/followers","following_url":"https://api.github.com/users/jml9904/following{/other_user}","gists_url":"https://api.github.com/users/jml9904/gists{/gist_id}","starred_url":"https://api.github.com/users/jml9904/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jml9904/subscriptions","organizations_url":"https://api.github.com/users/jml9904/orgs","repos_url":"https://api.github.com/users/jml9904/repos","events_url":"https://api.github.com/users/jml9904/events{/privacy}","received_events_url":"https://api.github.com/users/jml9904/received_events","type":"User","site_admin":false},"created_at":"2018-09-07T18:11:34Z","updated_at":"2018-09-07T18:11:34Z","author_association":"NONE","body":"Looks like it'll help. As soon as I can get out from under the current testing (probably Monday) I'll give it a whirl. Thanks!"}}
{"actor":{"display_login":"jml9904"}}