{"action":"created","issue":{"url":"https://api.github.com/repos/behave/behave/issues/504","repository_url":"https://api.github.com/repos/behave/behave","labels_url":"https://api.github.com/repos/behave/behave/issues/504/labels{/name}","comments_url":"https://api.github.com/repos/behave/behave/issues/504/comments","events_url":"https://api.github.com/repos/behave/behave/issues/504/events","html_url":"https://github.com/behave/behave/issues/504","id":185946183,"number":504,"title":"The reporting and error logging need to be improved","user":{"login":"fredizzimo","id":9946255,"avatar_url":"https://avatars.githubusercontent.com/u/9946255?v=3","gravatar_id":"","url":"https://api.github.com/users/fredizzimo","html_url":"https://github.com/fredizzimo","followers_url":"https://api.github.com/users/fredizzimo/followers","following_url":"https://api.github.com/users/fredizzimo/following{/other_user}","gists_url":"https://api.github.com/users/fredizzimo/gists{/gist_id}","starred_url":"https://api.github.com/users/fredizzimo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredizzimo/subscriptions","organizations_url":"https://api.github.com/users/fredizzimo/orgs","repos_url":"https://api.github.com/users/fredizzimo/repos","events_url":"https://api.github.com/users/fredizzimo/events{/privacy}","received_events_url":"https://api.github.com/users/fredizzimo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-28T15:02:40Z","updated_at":"2016-11-29T21:33:28Z","closed_at":null,"body":"I think one of the biggest limitations of Behave is how the error messages are reported. It works ok, when you run the tests locally and use console output with debugging. But if you move the tests to a continous integration server that runs them automatically, you want to be able to find the issues by just reading the log files. \n\nThere are several different issues, so let's look at them one at a time.\n1. **The callstacks are ignored for the asserts.** This is fine if you always have good asserts messages, or if you break into the debugger as soon as there's an assert. However it's not easy to write the asserts with enough context to debug the issue, and many time something important might have been left out.\n2. Related to the above, the actual error messages are formatted in Step class from the models module, which doesn't seem to be the correct place for doing something like this. IMO, we should only catch the assert, and let the formatter or reporter take care of the exact format of the error message. In that case, if we wanted to display full asserts messages, we could just write a custom reporter of formatter, based on the stored exception information. Technically this is probably already possible if the reporter just ignores the error message given, and always uses the stored exception instead.\n3. There doesn't seem to be any good way to capture output unless the step fails. One way is to capture all logs by adding another logging device. But by doing this we loose the context of in which step the logged line actually happened. \n   \n   Another approach is described in #500, but doesn't allow the reporters to decide what to do with the logs. It might be possible to store it in the model as an extra field, and read that in the formatter\n   \n   Another potential option would be to let the reporter read the context, and the captured output from there. However I'm not sure if the formatter has access to that information.\n   \n   In any case, both of these solutions seems like hacks, it would be much better if the captured output was always included in the model. \n4. Reporting errors from hooks is a mess, since hooks don't have proper models and support in the formatters. So for hooks I think you should include models that works almost exactly like steps. There should also be new callbacks for the hooks added to the formatters. \n   \n   Hooks also suffers from the same formatting of error messages as described in 2. Additionally from reading the code, it looks like there's a bug; if a step fails and then immediately afterwards the hook for the same step fails, then the error message of the step would be overwritten by the one of the hook.\n5. Generally, I think the interface for the formatters is quite hard to work with, as not all needed parameters are passed to all of the functions. It's also missing things like `end_scenario`. I think it should be something more similar to the hooks. But on the other hand if the models are good enough and contain all the information most of the formatters could be written as much more simple reporters instead, and only things that really need realtime output would use formatters.\n\nThere are probably more things that I have forgot, but I think you get the idea. The formatters/reporters should be passed all the needed information, and those should be the only ones that format error messages. But of course there could be some common functions for formatting exeptions for example.\n"},"comment":{"url":"https://api.github.com/repos/behave/behave/issues/comments/263705904","html_url":"https://github.com/behave/behave/issues/504#issuecomment-263705904","issue_url":"https://api.github.com/repos/behave/behave/issues/504","id":263705904,"user":{"login":"jenisys","id":371945,"avatar_url":"https://avatars.githubusercontent.com/u/371945?v=3","gravatar_id":"","url":"https://api.github.com/users/jenisys","html_url":"https://github.com/jenisys","followers_url":"https://api.github.com/users/jenisys/followers","following_url":"https://api.github.com/users/jenisys/following{/other_user}","gists_url":"https://api.github.com/users/jenisys/gists{/gist_id}","starred_url":"https://api.github.com/users/jenisys/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jenisys/subscriptions","organizations_url":"https://api.github.com/users/jenisys/orgs","repos_url":"https://api.github.com/users/jenisys/repos","events_url":"https://api.github.com/users/jenisys/events{/privacy}","received_events_url":"https://api.github.com/users/jenisys/received_events","type":"User","site_admin":false},"created_at":"2016-11-29T21:33:28Z","updated_at":"2016-11-29T21:33:28Z","body":"@fredizzimo \r\nThe problem that I have with this issue is that it is not an issue. Instead, it is a collection of issues (with points that you do not like). \r\n\r\nGuidelines for pull-requests to be acceptable:\r\n* provide a pull-request that addresses one problem at a time (and not a JUMBO pull-request). \r\n\r\nCharacteristics of non-acceptable pull-requests:\r\n* doing the same thing in another way without adding additional value (\"using another flavour\"). "}}
{"actor":{"display_login":"jenisys"}}