{"action":"opened","issue":{"url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/317","repository_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet","labels_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/317/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/317/comments","events_url":"https://api.github.com/repos/dotnet/BenchmarkDotNet/issues/317/events","html_url":"https://github.com/dotnet/BenchmarkDotNet/issues/317","id":193341151,"number":317,"title":"[Raw idea]: Share pseudo-random seed for same run benchmarks?","user":{"login":"ig-sinicyn","id":11225241,"avatar_url":"https://avatars.githubusercontent.com/u/11225241?v=3","gravatar_id":"","url":"https://api.github.com/users/ig-sinicyn","html_url":"https://github.com/ig-sinicyn","followers_url":"https://api.github.com/users/ig-sinicyn/followers","following_url":"https://api.github.com/users/ig-sinicyn/following{/other_user}","gists_url":"https://api.github.com/users/ig-sinicyn/gists{/gist_id}","starred_url":"https://api.github.com/users/ig-sinicyn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ig-sinicyn/subscriptions","organizations_url":"https://api.github.com/users/ig-sinicyn/orgs","repos_url":"https://api.github.com/users/ig-sinicyn/repos","events_url":"https://api.github.com/users/ig-sinicyn/events{/privacy}","received_events_url":"https://api.github.com/users/ig-sinicyn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-04T09:25:27Z","updated_at":"2016-12-04T09:25:27Z","closed_at":null,"body":"_DISCLAIMER_\r\nI'm not sure the thing is required at all and I have no good thoughs on how to implement it.\r\nSo, I just leave it there:)\r\n\r\nOkay,  back to topic. We have some benchmarks we run constantly and we want to ensure that results are accurate for wide range of data.\r\n\r\nWe're doing it via generating random seed, logging it to test output and then creating a new random to be used across rest of the code. Something like this:\r\n```c#\r\n\t\t[Setup]\r\n\t\tpublic void Setup()\r\n\t\t{\r\n\t\t\t// Rnd seed to get repeatable results\r\n\t\t\tvar rndSeed = new Random().Next();\r\n\t\t\tConsole.WriteLine($\"// ! Rnd seed: {rndSeed}.\");\r\n\r\n\t\t\tvar rnd = new Random(rndSeed); // using the seed\r\n\t\t\t_arrayA = Enumerable.Range(0, 128).Select(i => (byte)rnd.Next()).ToArray();\r\n\r\n\t\t\t// ...\r\n\t\t}\r\n```\r\nThe issue is, each benchmark method uses its own random seed and it's very tricky to reproduce run with same seeds.\r\n* You have to read large portion of the log to restore mapping from logged rnd seed to the target method (there's `IOrderProvider` so execution order and summary order may differ).\r\n* There's no API to determine target method the setup method is called for. This means you have to create separate set of fields per each method in benchmark, and to init each set using its own rnd seed.\r\n\r\nThe proposal is, there should be a way to share same pseudo-random rnd seed for all methods in benchmark.\r\n\r\n**Simplest possible solution**\r\nJust add a static field to be filled with seed during codegen.\r\nThe code will use it like this:\r\n```c#\r\n\t\t[Setup]\r\n\t\tpublic void Setup()\r\n\t\t{\r\n\t\t\t// Rnd seed to get repeatable results\r\n\t\t\tvar rndSeed = BenchmarkContext.RndSeed; // name is preliminary\r\n\t\t\tvar rnd = new Random(rndSeed); // using the seed\r\n\t\t\t_arrayA = Enumerable.Range(0, 128).Select(i => (byte)rnd.Next()).ToArray();\r\n\r\n\t\t\t// ...\r\n\t\t}\r\n```\r\n\r\n**Robust solution**\r\n* Add an API to obtain current Job (and current target method), these should be stored as AsyncLocal.\r\n* Add an characteristic to Job to store the rnd seed (have no idea where to put it exactly. `Job.Run.RndSeed`, maybe?).\r\n* Init the characteristic in `BenchmarkConverter.GetFullConfig()` if it was not set earlier.\r\n\r\nand the result will be:\r\n```c#\r\n\t\t[Setup]\r\n\t\tpublic void Setup()\r\n\t\t{\r\n\t\t\t// Rnd seed to get repeatable results\r\n\t\t\tvar rndSeed = BenchmarkContext.CurrentJob.Run.RndSeed; // name is preliminary\r\n\t\t\tvar rnd = new Random(rndSeed); // using the seed\r\n\t\t\t_arrayA = Enumerable.Range(0, 128).Select(i => (byte)rnd.Next()).ToArray();\r\n\r\n\t\t\t// ...\r\n\t\t}\r\n```"}}
{"actor":{"display_login":"ig-sinicyn"}}