{"action":"opened","issue":{"url":"https://api.github.com/repos/tomav/docker-mailserver/issues/324","repository_url":"https://api.github.com/repos/tomav/docker-mailserver","labels_url":"https://api.github.com/repos/tomav/docker-mailserver/issues/324/labels{/name}","comments_url":"https://api.github.com/repos/tomav/docker-mailserver/issues/324/comments","events_url":"https://api.github.com/repos/tomav/docker-mailserver/issues/324/events","html_url":"https://github.com/tomav/docker-mailserver/issues/324","id":176431090,"number":324,"title":"Test suite deficiencies","user":{"login":"KyleOndy","id":1640900,"avatar_url":"https://avatars.githubusercontent.com/u/1640900?v=3","gravatar_id":"","url":"https://api.github.com/users/KyleOndy","html_url":"https://github.com/KyleOndy","followers_url":"https://api.github.com/users/KyleOndy/followers","following_url":"https://api.github.com/users/KyleOndy/following{/other_user}","gists_url":"https://api.github.com/users/KyleOndy/gists{/gist_id}","starred_url":"https://api.github.com/users/KyleOndy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KyleOndy/subscriptions","organizations_url":"https://api.github.com/users/KyleOndy/orgs","repos_url":"https://api.github.com/users/KyleOndy/repos","events_url":"https://api.github.com/users/KyleOndy/events{/privacy}","received_events_url":"https://api.github.com/users/KyleOndy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-09-12T17:12:26Z","updated_at":"2016-09-12T17:12:26Z","closed_at":null,"body":"As I started looking back into #256 and #250 some maintainability issues have become apparent within the test suite. Given the current set of tests, coverage is low since we are not testing full functionality when testing single configurations.\r\n\r\nFor instance, in #250, the test passes because the service is not running, however the ability to send an email is never tested, allowing the bug to be introduced.\r\n\r\nAs I see it, an ideal solution that would avoid the bugs above is to run through the basic functions of the mail server for each option that is configured.\r\n\r\n-----\r\n\r\n# Proposal\r\n*Possible implementation would need to be explored*\r\n\r\n### A set of core tests that are common across all configurations\r\nThis is a mail server, so always test SMTP and any other core services are always correctly working no matter what combination of environmental variables are set.\r\n\r\n### Each environmental variable would have a set of core tests associated with each possible value.\r\n**example:** `SSL_TYPE` would have 5 sets of tests associated with it.\r\n\r\n1. `SSL_TYPE=` => verify no ssl is configured\r\n2. `SSL_TYPE=letsencrypt` => verify Let's Encrypt certificates\r\n3. `SSL_TYPE=custom` => verify custom certificates\r\n4. `SSL_TYPE=manual` => verify custom locations of your SSL certificates for non-standard cases\r\n5. `SSL_TYPE=self-signed` => verify self-signed certificate work\r\n\r\n### Some kind of preprocessorfor BATS that generates all tests cases\r\nOnce this information is defined, it would be possible to generate complete test coverage for every configuration (all combinations of environmental variables) by using the associated set of tests for each option being tested.\r\n\r\n## Possible Issues\r\n\r\n1. Compexity in creating / maintaing a BATS preprocessorfor.\r\n2. The number of combinations grows exponentially, making testing all combinations infeasible.\r\n\r\n-------------------\r\n\r\nThe more I think about this particular solution, the messier it seems. Ultimately, I still think there are coverage issues that should be solved.\r\n\r\nThoughts?\r\n"}}
{"actor":{"display_login":"KyleOndy"}}