{"action":"created","issue":{"url":"https://api.github.com/repos/piwik/piwik/issues/8695","labels_url":"https://api.github.com/repos/piwik/piwik/issues/8695/labels{/name}","comments_url":"https://api.github.com/repos/piwik/piwik/issues/8695/comments","events_url":"https://api.github.com/repos/piwik/piwik/issues/8695/events","html_url":"https://github.com/piwik/piwik/issues/8695","id":104425128,"number":8695,"title":"Automatically run test against minimum/maximum supported Piwik version defined in plugin.json","user":{"login":"tsteur","id":273120,"avatar_url":"https://avatars.githubusercontent.com/u/273120?v=3","gravatar_id":"","url":"https://api.github.com/users/tsteur","html_url":"https://github.com/tsteur","followers_url":"https://api.github.com/users/tsteur/followers","following_url":"https://api.github.com/users/tsteur/following{/other_user}","gists_url":"https://api.github.com/users/tsteur/gists{/gist_id}","starred_url":"https://api.github.com/users/tsteur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsteur/subscriptions","organizations_url":"https://api.github.com/users/tsteur/orgs","repos_url":"https://api.github.com/users/tsteur/repos","events_url":"https://api.github.com/users/tsteur/events{/privacy}","received_events_url":"https://api.github.com/users/tsteur/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/piwik/piwik/labels/c:%20Tests%20&%20QA","name":"c: Tests & QA","color":"bfdadc"}],"state":"open","locked":false,"assignee":{"login":"tsteur","id":273120,"avatar_url":"https://avatars.githubusercontent.com/u/273120?v=3","gravatar_id":"","url":"https://api.github.com/users/tsteur","html_url":"https://github.com/tsteur","followers_url":"https://api.github.com/users/tsteur/followers","following_url":"https://api.github.com/users/tsteur/following{/other_user}","gists_url":"https://api.github.com/users/tsteur/gists{/gist_id}","starred_url":"https://api.github.com/users/tsteur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsteur/subscriptions","organizations_url":"https://api.github.com/users/tsteur/orgs","repos_url":"https://api.github.com/users/tsteur/repos","events_url":"https://api.github.com/users/tsteur/events{/privacy}","received_events_url":"https://api.github.com/users/tsteur/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/piwik/piwik/milestones/89","html_url":"https://github.com/piwik/piwik/milestones/2.16.0","labels_url":"https://api.github.com/repos/piwik/piwik/milestones/89/labels","id":1257878,"number":89,"title":"2.16.0","description":"","creator":{"login":"mattab","id":466765,"avatar_url":"https://avatars.githubusercontent.com/u/466765?v=3","gravatar_id":"","url":"https://api.github.com/users/mattab","html_url":"https://github.com/mattab","followers_url":"https://api.github.com/users/mattab/followers","following_url":"https://api.github.com/users/mattab/following{/other_user}","gists_url":"https://api.github.com/users/mattab/gists{/gist_id}","starred_url":"https://api.github.com/users/mattab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattab/subscriptions","organizations_url":"https://api.github.com/users/mattab/orgs","repos_url":"https://api.github.com/users/mattab/repos","events_url":"https://api.github.com/users/mattab/events{/privacy}","received_events_url":"https://api.github.com/users/mattab/received_events","type":"User","site_admin":false},"open_issues":11,"closed_issues":293,"state":"open","created_at":"2015-08-17T09:20:14Z","updated_at":"2016-01-27T01:17:54Z","due_on":"2016-01-27T11:00:00Z","closed_at":null},"comments":8,"created_at":"2015-09-02T08:17:48Z","updated_at":"2016-01-27T01:38:38Z","closed_at":null,"body":"Follow up on https://github.com/piwik/piwik/issues/8660\r\n\r\n## Background: Defining supported Piwik version\r\n\r\nIn `plugin.json` a developer can specify supported Piwik version. Minimum supported Piwik version defaults to `2.0` and maximum supported Piwik versions defaults to the latest stable. \r\n\r\nThis is described in http://developer.piwik.org/guides/distributing-your-plugin#pluginjson-file . Versions can be specified like \r\n\r\n```json\r\n\"require\": {\r\n    \"piwik\": \">=2.0.3\" // requires at least Piwik 2.0.3\r\n} \r\n```\r\n\r\nor \r\n\r\n```\r\n\"require\": {\r\n    \"piwik\": \">=2.0.0,<=2.2.0\" // requires a Piwik version 2.0.0 up to 2.2.0\r\n}\r\n```\r\n\r\nA full list of possible version definitions can be seen here: https://getcomposer.org/doc/articles/versions.md\r\n\r\n## Problem: Tests on Travis\r\n\r\nThey currently run against the minimum defined version in `plugin.json` automatically and against the maximum supported version (currently named `PIWIK_LATEST_STABLE_TEST_TARGET`). The maximum supported version in `.travis.yml` is however not automatically updated but it should. This means, currently, a plugin developer has to always update `.travis.yml` automatically which can be easily forgotten and plugin developers might not be even aware that they have to do this. Having to specify such things twice (`plugin.json` and `.travis.yml`) is not good experience and is fragile. Especially since the name suggests that it always runs against latest stable Piwik version. This can end in broken Piwik installations for thousands of users.\r\n\r\n## Expected solution (and more on the problem):\r\n\r\nTests on travis should automatically read the defined version constraints in `plugin.json` and run one build against minimum supported version and one build against maximum supported version. This has to be automatic. One should not have to define it in `plugin.json` and `.travis.yml`. The whole point of having tests run on Travis is to get notified when something doesn't work anymore. It's expected that a build fails if we release a new version and the plugin is not compatible with that version anymore. This will help the developer to maintain the supported versions in `plugin.json`. If a build fails because we release a new version the developer has to get notified and take action: Either change the required Piwik version in `plugin.json` to not support the latest stable version anymore (by specifying eg `<2.X.X`) or to make the plugin compatible. Afterwards a plugin developer needs to release a new version otherwise users that use this plugin end up in having a broken Piwik that might no longer work at all. \r\n\r\nAll this part about what is the correct required Piwik version is a very critical part see eg the issue I created in #6149 since it's basically not possible to know for a developer in which versions a plugin will work (maybe will be a bit better once we do #8547 ) and people easily forget to update the required Piwik version when they work on their plugin. \r\n\r\n## More about this:\r\n\r\nCan be found in the discussion of https://github.com/piwik/piwik/issues/8660\r\n\r\nI do not think this is too complicated to implement but in case there are maybe some more steps in between...\r\n\r\n* We should maybe rename PIWIK_LATEST_STABLE_TEST_TARGET. The name is why I thought it would be always the latest stable automatically. In Travis etc one often doesn't see the full variable but only PIWIK_LATEST_STABLE or even if one can see the full variable in travis it still kinda suggests it will be tested against latest stable. Maybe a better name could be `PIWIK_TARGET_VERSION`.\r\n\r\n* A simple first step could be also to email plugin developers (at least the once we know or a mailing list etc) to check their plugins re compatibility with latest beta eg 5 days before a new release and we could mention in this email that they have to update `.travis.yml` to do this.\r\n\r\n* Piwik PRO developers vs Community / 3rd party developers:\r\n\r\nFrom what I understand the expected solution is not implemented yet because it's possibly not working for Piwik PRO developers. I think this is not the right way to go as it ends in thousands of broken Piwik installations and I think if we find the right solution then it is good for both. Eg Piwik PRO developers can still define a minimum/maximum supported version via `plugin.json`. By default no maximum version is defined but Piwik PRO developers could always specify a max supported version and increase it manually and it should work for them too. For community plugin developers no maximum supported Piwik version should be specified for now as they will be usually still compatible when a new Piwik version is released and if it breaks, we need to make sure they get notified when we run a test. \r\n\r\nIf we don't do this because it just doesn't work for Piwik PRO developers we should maybe remove the `generate:travis-yml` command from Piwik core and either provide it via a separate \"ProDevelopment\" plugin or rename the command in core to `pro:generate:travis-yml` or `generate:travis-yml-piwikpro`. This way it is clear it is not intended to be used by the community. We can then maybe at some point add a second command that generates a different travis-yml that will automatically update versions. "},"comment":{"url":"https://api.github.com/repos/piwik/piwik/issues/comments/175333738","html_url":"https://github.com/piwik/piwik/issues/8695#issuecomment-175333738","issue_url":"https://api.github.com/repos/piwik/piwik/issues/8695","id":175333738,"user":{"login":"mattab","id":466765,"avatar_url":"https://avatars.githubusercontent.com/u/466765?v=3","gravatar_id":"","url":"https://api.github.com/users/mattab","html_url":"https://github.com/mattab","followers_url":"https://api.github.com/users/mattab/followers","following_url":"https://api.github.com/users/mattab/following{/other_user}","gists_url":"https://api.github.com/users/mattab/gists{/gist_id}","starred_url":"https://api.github.com/users/mattab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattab/subscriptions","organizations_url":"https://api.github.com/users/mattab/orgs","repos_url":"https://api.github.com/users/mattab/repos","events_url":"https://api.github.com/users/mattab/events{/privacy}","received_events_url":"https://api.github.com/users/mattab/received_events","type":"User","site_admin":false},"created_at":"2016-01-27T01:38:38Z","updated_at":"2016-01-27T01:38:38Z","body":"> Also it means if a developer supports all Piwik versions from a certain version (eg >=2.14.0) one does no longer always have to adjust the .travis.yml when a new Piwik version is released as the maximium supported version is automatically fetched.\r\n\r\nThis sounds really useful and would save time and effort :+1: We'll be testing over next few days/weeks that this work as expected."}}