{"action":"created","issue":{"url":"https://api.github.com/repos/clash-lang/clash-compiler/issues/176","repository_url":"https://api.github.com/repos/clash-lang/clash-compiler","labels_url":"https://api.github.com/repos/clash-lang/clash-compiler/issues/176/labels{/name}","comments_url":"https://api.github.com/repos/clash-lang/clash-compiler/issues/176/comments","events_url":"https://api.github.com/repos/clash-lang/clash-compiler/issues/176/events","html_url":"https://github.com/clash-lang/clash-compiler/issues/176","id":177610255,"number":176,"title":"RFC: Improvements to the test bench interface","user":{"login":"thoughtpolice","id":3416,"avatar_url":"https://avatars.githubusercontent.com/u/3416?v=3","gravatar_id":"","url":"https://api.github.com/users/thoughtpolice","html_url":"https://github.com/thoughtpolice","followers_url":"https://api.github.com/users/thoughtpolice/followers","following_url":"https://api.github.com/users/thoughtpolice/following{/other_user}","gists_url":"https://api.github.com/users/thoughtpolice/gists{/gist_id}","starred_url":"https://api.github.com/users/thoughtpolice/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thoughtpolice/subscriptions","organizations_url":"https://api.github.com/users/thoughtpolice/orgs","repos_url":"https://api.github.com/users/thoughtpolice/repos","events_url":"https://api.github.com/users/thoughtpolice/events{/privacy}","received_events_url":"https://api.github.com/users/thoughtpolice/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-09-17T21:28:23Z","updated_at":"2016-09-18T01:10:48Z","closed_at":null,"body":"Here's an example of a test bench I have for something I'm working on. This is actually only a single test bench, stressing one part of the design; I have other tests that stress other things. It simply tests that some examples of using the instruction decoder work.\r\n\r\nMy intent is to be able to test this:\r\n\r\n- In CLaSH, i.e. as a Haskell simulation.\r\n- In _simulated_ Verilog, i.e. by simply using `iverilog` or `verilator` on the results of the CLaSH compiler. This works pretty great today with `iverilog`.\r\n- In a _synthesized_ or a _placed-and-routed_ netlist. With Yosys/Arachne-pnr/IceStorm, I can resynthesize a Verilog description of my circuit after Yosys performs synthesis for my board, or even later after place-and-routing that synthesized design with `arachne-pnr`, resulting in a description of the synthesized netlist. This is mostly a toolchain issue (.e.g getting the full netlist back, so you can plug it up to your test).\r\n- In hardware. This requires hooking up a clock source, generally to some onboard PLL, and writing some code specific to your board to exercise the test functionality (e.g. blink LED0 on port J3 every second when a failure occurs). Needless to say, you can basically get away with this by writing a new module with a new `topEntity` , specific to that board/hardware/test. CLaSH probably can't automate away anything more than it already does, here.\r\n\r\nTogether these all provide relatively high assurance that the design works on all fronts. This ticket is mostly about the first one, which is my primary line of testing as it's far less cumbersome and much more interactive.\r\n\r\nOK, so here's my example.\r\n\r\n```haskell\r\nmodule AAP.Test.Decoder16\r\n  ( topEntity      -- :: Signal (BitVector 16) -> Signal Instr\r\n  , testInput      -- :: Signal (BitVector 16)\r\n  , expectedOutput -- :: Signal Instr -> Signal Bool\r\n  , fullTest       -- :: [Bool]\r\n  ) where\r\nimport CLaSH.Prelude\r\n\r\nimport AAP.Decoder\r\n\r\n-- | Trivial circuit for testing the 16 bit instruction decoder.\r\ntopEntity :: Signal (BitVector 16) -> Signal Instr\r\ntopEntity = fmap decode16\r\n\r\n-- | Input test values for @'topEntity'@.\r\ntestInput :: Signal (BitVector 16)\r\ntestInput = stimuliGenerator vals where\r\n  vals = 0b0000000000000010\r\n      :> 0b1110001100000101\r\n      :> 0b1111100000000110\r\n      :> Nil\r\n\r\n-- | Output verifier: make sure that the results of @'topEntity'@ are\r\n-- legitimate. The signal output is @'True'@ when all of the expected values\r\n-- have been tested.\r\nexpectedOutput :: Signal Instr -> Signal Bool\r\nexpectedOutput = outputVerifier (map Instr16 vals) where\r\n  vals = ADD16 0 0 0\r\n      :> SUB16 7 4 3\r\n      :> AND16 3 7 0\r\n      :> Nil\r\n\r\nfullSimulationTest :: [Bool]\r\nfullSimulationTest = Prelude.takeWhile (== False) vals where\r\n  vals = sample $ expectedOutput (topEntity testInput)\r\n```\r\n\r\nSo I have this, and now there's the question.\r\n\r\n## How do I test it with CLaSH?\r\n\r\nThe CLaSH simulation can be tested, roughly, like this (with `clash --interactive`):\r\n\r\n```\r\n> fullSimulationTest\r\n[False, False, False]\r\n```\r\n\r\nThis output is expected, although it is unintuitive. `expectedOutput` returns `False` while it still has samples to verify, and returns `True` once it has verified the last sample. Thus, stopping at the first `True` value gives a list, the length of which is the length of the `testInput` signal (.e. `length (toList testInput)` == the number of tests in this test bench, basically).\r\n\r\nHowever, if the test *fails*, this results in something like this:\r\n\r\n```\r\n[False,False\r\ncycle(system1000): 2, outputVerifier\r\nexpected value: Instr16 (AND16 (Reg 000) (Reg 111) (Reg 000)), not equal to actual value: Instr16 (AND16 (Reg 011) (Reg 111) (Reg 000))\r\n,False\r\ncycle(system1000): 3, outputVerifier\r\nexpected value: Instr16 (AND16 (Reg 000) (Reg 111) (Reg 000)), not equal to actual value: Instr16 (AND16 (Reg 011) (Reg 111) (Reg 000))\r\n]\r\n```\r\n\r\nThis isn't really great. Essentially, there's some `unsafePerformIO` going on somewhere that throws an exception when some particular sampled result from `topEntity` isn't the expected result.\r\n\r\nThe *reason* for this interface is somewhat obvious, though: The CLaSH simulation can call `trace` or `error`, or `$print` or whatever in the iverilog test bench when something goes wrong, but that's *an implementation detail of the test bench*. That has nothing to do with the result of the compiler, or whether or not the actual functions `testInput` and `expectedOutput` can be synthesized. It is not exposed to the person writing the tests. The test itself being synthesizable in this way is desirable.\r\n\r\nThe problem is this makes running the CLaSH simulation tests is really weird, IMO, with this interface. Remember, I have *multiple* test benches like this one, each in a different module/file stressing multiple parts of the design (or intend to): instruction fetch, 32-bit decode, the register file, etc. So this is one among many.\r\n\r\nMy build system *can*, relatively trivially, simply find all these files and do its business. Let's say it runs `clash --verilog path/to/Decode16.hs` on each one (to generate the verilog), and then it could run something like `clash -e AAP.Test.Decode16.fullSimulationTest path/to/Decode16.hs` to run the simulation in the interpreter.\r\n\r\nIf the result of that command outputs `[False, False, False]` it passes, otherwise it fails. But this isn't great, because I have to do some weird hacky grep-like thing to determine test failure. Maybe I check that the exact string is output (but this is brittle). Or maybe I just check for any error messages (maybe less brittle). Or maybe I write some wrapper around the `[Bool]` output that does some crazy stuff to silently gobble results/exceptions from `trace` using some weird ancient unix voodoo or something, and then determine if evaluating the spine of the result gave out any impure behavior (god please no).\r\n\r\nBut this is Haskell! There should just be a type specifying what happened.\r\n\r\n## Hairbrained idea\r\n\r\nHere's my suggestion: let the test bench *for simulations* have some kind of interface like this:\r\n\r\n```haskell\r\ndata SimulationTestResult\r\n  = OK  -- Test passed\r\n  | Done -- No more tests\r\n  | Failed String -- Oops, one of the signal inputs didn't have the expected output\r\n```\r\n\r\nThe problem is that `String` cannot be synthesized (re: implementation detail of the simulation). But that's fine!\r\n\r\nWhile I originally thought that maybe this needs some hacking in the compiler, perhaps it does not. This interface can just be an extension of the current interface, and not break anything. For example, given the scheme in 0.6, this could be the exposed interface from my module\r\n\r\n```haskell\r\n------------------------------\r\n-- My functions\r\n\r\n-- Circuit to test\r\ntopEntity :: Signal a -> Signal b\r\n\r\n-- Test input\r\ntestInput :: Signal a\r\n\r\n-- Output verifier function\r\nexpectedOutput :: Signal b -> Signal Bool -- Synthesizable\r\n\r\n-- Output verifier function, run only for simulation.\r\nsimulatedOutput :: Signal b -> Signal SimulationTestResult -- Not synthesizable\r\n\r\n------------------------------\r\n-- CLaSH functions\r\n\r\noutputVerifier ::  ...         -> Signal b -> Signal Bool\r\nsimulatedOutputVerifier :: ... -> Signal b -> Signal SimulationTestResult\r\n```\r\n\r\n`simulatedOutput` needs no special care from the compiler, it can just be completely ignored - and that's OK, because we only wanted to test it with simulation anyway!\r\n\r\nThe only change this would imply is the intrudction of `simulatedOutputVerifier` and `SimulationTestResult`.\r\n\r\nIf I had `simulatedOutput` exported from my module, then having a higher-level module export it would let me do something like:\r\n\r\n```haskell\r\n-- drop the initial sample, just as an example, because maybe you're using blockRam# or something\r\ndecoderSamples = Prelude.drop 1 $ sample $ simulatedOutput (topEntity testInput)\r\n\r\n-- the simulatedOutputVerifier outputs only a single 'Done' value, once it's tested every input\r\nallTests = Prelude.takeWhile (!= Done) decoderSamples\r\n\r\nnumberOfTests = Prelude.length allTests\r\npassedTests = length $ Prelude.filter (== OK) allTests\r\n\r\nwhen ((numberOfTests - passedTests) > 0) $ do\r\n  putStrLn $ \"omg, some tests failed: \" ++ ... some very informative message here ...\r\n```\r\n\r\nThis would allow me to hook up this interface or abstract over it in a number of ways to a bigger, more complete test driver that tests multiple components.\r\n\r\nAnyway, that's my basic complaint. Any thoughts?"},"comment":{"url":"https://api.github.com/repos/clash-lang/clash-compiler/issues/comments/247817598","html_url":"https://github.com/clash-lang/clash-compiler/issues/176#issuecomment-247817598","issue_url":"https://api.github.com/repos/clash-lang/clash-compiler/issues/176","id":247817598,"user":{"login":"thoughtpolice","id":3416,"avatar_url":"https://avatars.githubusercontent.com/u/3416?v=3","gravatar_id":"","url":"https://api.github.com/users/thoughtpolice","html_url":"https://github.com/thoughtpolice","followers_url":"https://api.github.com/users/thoughtpolice/followers","following_url":"https://api.github.com/users/thoughtpolice/following{/other_user}","gists_url":"https://api.github.com/users/thoughtpolice/gists{/gist_id}","starred_url":"https://api.github.com/users/thoughtpolice/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thoughtpolice/subscriptions","organizations_url":"https://api.github.com/users/thoughtpolice/orgs","repos_url":"https://api.github.com/users/thoughtpolice/repos","events_url":"https://api.github.com/users/thoughtpolice/events{/privacy}","received_events_url":"https://api.github.com/users/thoughtpolice/received_events","type":"User","site_admin":false},"created_at":"2016-09-18T01:10:48Z","updated_at":"2016-09-18T01:10:48Z","body":"I've managed to quickly implement this myself and I'm pretty happy with the results!\r\n\r\nFirst, the harness:\r\n\r\n```haskell\r\n{-# LANGUAGE ScopedTypeVariables #-}\r\n\r\nmodule AAP.Test.Fancy\r\n  ( SimulatedTestResult(..) -- :: *\r\n  , simulationTest          -- :: ...\r\n  ) where\r\n\r\nimport GHC.TypeLits          (KnownNat)\r\nimport Prelude               hiding ((!!))\r\n\r\nimport CLaSH.Signal          (Signal, fromList, sampleN)\r\nimport CLaSH.Signal.Explicit (Signal', SClock, register', systemClock)\r\nimport CLaSH.Signal.Bundle   (unbundle')\r\nimport CLaSH.Sized.Index     (Index)\r\nimport CLaSH.Sized.Vector    (Vec, (!!), maxIndex)\r\n\r\nimport CLaSH.Promoted.Nat    (SNat, snat, snatToInteger)\r\n\r\nimport CLaSH.Prelude.Testbench (assert')\r\n\r\ndata SimulatedTestResult\r\n  = TestOK            -- ^ Test passed\r\n  | TestFailed String -- ^ Test failed\r\n  | TestsDone         -- ^ No more tests\r\n  deriving (Eq, Show)\r\n\r\nsimulatedOutputVerifier :: forall l a . (KnownNat l, Eq a, Show a)\r\n                        => Vec l a\r\n                        -- ^ Samples to compare with\r\n                        -> Signal a\r\n                        -- ^ Signal to verify\r\n                        -> Signal SimulatedTestResult\r\n                        -- ^ Output signal indicating test result\r\nsimulatedOutputVerifier = simulatedOutputVerifier' systemClock\r\n\r\nsimulatedOutputVerifier' :: forall l clk a . (KnownNat l, Eq a, Show a)\r\n                => SClock clk\r\n                -- ^ Clock to which the input signal is synchronized to\r\n                -> Vec l a\r\n                -- ^ Samples to compare with\r\n                -> Signal' clk a\r\n                -- ^ Signal to verify\r\n                -> Signal' clk SimulatedTestResult\r\n                -- ^ Indicator that all samples are verified\r\nsimulatedOutputVerifier' clk samples i =\r\n    let (s,o) = unbundle' clk (genT <$> register' clk 0 s)\r\n        (e,f) = unbundle' clk o\r\n    in simulatedAssert' clk \"outputVerifier\" i e (register' clk TestOK f)\r\n  where\r\n    genT :: Index l -> (Index l,(a, SimulatedTestResult))\r\n    genT s = (s',(samples !! s,finished))\r\n      where\r\n        maxI = toEnum (maxIndex samples)\r\n\r\n        s' = if s < maxI\r\n                then s + 1\r\n                else s\r\n\r\n        finished = if s == maxI then TestsDone else TestOK\r\n\r\nsimulatedAssert' :: (Eq a,Show a)\r\n        => SClock t\r\n        -> String      -- ^ Additional message\r\n        -> Signal' t a -- ^ Checked value\r\n        -> Signal' t a -- ^ Expected value\r\n        -> Signal' t SimulatedTestResult\r\n        -> Signal' t SimulatedTestResult\r\nsimulatedAssert' clk msg checked expected returned =\r\n  (\\c e cnt r ->\r\n      if c == e\r\n       then r\r\n       else TestFailed (concat [ \"\\ncycle(\" ++ show clk ++ \"): \"\r\n                               , show cnt\r\n                               , \", \"\r\n                               , msg\r\n                               , \"\\nexpected value: \"\r\n                               , show e\r\n                               , \", not equal to actual value: \"\r\n                               , show c\r\n                               ]))\r\n  <$> checked <*> expected <*> fromList [(0::Integer)..] <*> returned\r\n\r\n-- | Run a test in simulation mode, and give a friendly report of the results.\r\n--\r\n-- __NB__: This function CANNOT be synthesized; it is merely a more\r\n-- user-friendly test interface, meant so you can easily run your tests inside\r\n-- of CLaSH. To create test benches which can be synthesized, see\r\n-- @'CLaSH.Prelude.Testbench'@.\r\nsimulationTest :: forall a n. (Eq a, Show a, KnownNat n)\r\n               => Vec n a  -- ^ Expected list of outputs from a given signal.\r\n               -> Signal a -- ^ Input signal\r\n               -> [SimulatedTestResult]\r\nsimulationTest out inp = sampleN n (simulatedOutputVerifier out inp)\r\n  where n = fromIntegral (snatToInteger (snat :: SNat n))\r\n```\r\n\r\nNow my new tests can look like this:\r\n\r\n```haskell\r\nmodule AAP.Test.Decoder16\r\n  ( -- * Synthesis testing.\r\n    topEntity        -- :: Signal (BitVector 16) -> Signal Instr\r\n  , testInput        -- :: Signal (BitVector 16)\r\n  , expectedOutput   -- :: Signal Instr -> Signal Bool\r\n\r\n    -- * Simulation testing.\r\n  , simulationOutput -- :: [SimulationTestResult]\r\n  ) where\r\nimport CLaSH.Prelude\r\n\r\nimport AAP.Decoder\r\nimport AAP.Test.Fancy\r\n\r\n--------------------------------------------------------------------------------\r\n-- Test description\r\n\r\n-- | Trivial circuit for testing the 16 bit instruction decoder.\r\ntopEntity :: Signal (BitVector 16) -> Signal Instr\r\ntopEntity = fmap decode16\r\n\r\n-- | The number of tests in this test bench.\r\ntype NumTests = (3 :: Nat)\r\n\r\n-- | Inputs into the test bench; these are run through @'topEntity'@ and checked\r\n-- against @'output'@.\r\ninput :: Vec NumTests (BitVector 16)\r\ninput = 0b0000000000000010\r\n     :> 0b1110001100000101\r\n     :> 0b1111100000000110\r\n     :> Nil\r\n\r\n-- | Expected outputs from running @'topEntity'@ on the test @'input'@.\r\noutput :: Vec NumTests Instr\r\noutput = map Instr16 vals where\r\n  vals = ADD16 0 0 0\r\n      :> SUB16 7 4 3\r\n      :> AND16 3 7 0\r\n      :> Nil\r\n\r\n--------------------------------------------------------------------------------\r\n-- Synthesis test harness.\r\n\r\n-- | Input value signal for synthesis tests.\r\ntestInput :: Signal (BitVector 16)\r\ntestInput = stimuliGenerator input\r\n\r\n-- | Output verifier for synthesis tests.\r\nexpectedOutput :: Signal Instr -> Signal Bool\r\nexpectedOutput = outputVerifier output\r\n\r\n--------------------------------------------------------------------------------\r\n-- Simulation test harness\r\n\r\n-- | Results of running all the tests. __NB__: This is only meant for running\r\n-- CLaSH simulation tests.\r\nsimulationOutput :: [SimulatedTestResult]\r\nsimulationOutput = simulationTest output (topEntity testInput)\r\n```\r\n\r\nWhich I think is pretty good! I mean, the *actual* implementation could be simplified probably, but I just copy/pasted to make it as fast as possible."}}
{"actor":{"display_login":"thoughtpolice"}}