{"action":"created","issue":{"url":"https://api.github.com/repos/gocd/gocd/issues/2121","repository_url":"https://api.github.com/repos/gocd/gocd","labels_url":"https://api.github.com/repos/gocd/gocd/issues/2121/labels{/name}","comments_url":"https://api.github.com/repos/gocd/gocd/issues/2121/comments","events_url":"https://api.github.com/repos/gocd/gocd/issues/2121/events","html_url":"https://github.com/gocd/gocd/issues/2121","id":149476162,"number":2121,"title":"Proposal to partially drop xml data store","user":{"login":"tomzo","id":5710970,"avatar_url":"https://avatars.githubusercontent.com/u/5710970?v=3","gravatar_id":"","url":"https://api.github.com/users/tomzo","html_url":"https://github.com/tomzo","followers_url":"https://api.github.com/users/tomzo/followers","following_url":"https://api.github.com/users/tomzo/following{/other_user}","gists_url":"https://api.github.com/users/tomzo/gists{/gist_id}","starred_url":"https://api.github.com/users/tomzo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomzo/subscriptions","organizations_url":"https://api.github.com/users/tomzo/orgs","repos_url":"https://api.github.com/users/tomzo/repos","events_url":"https://api.github.com/users/tomzo/events{/privacy}","received_events_url":"https://api.github.com/users/tomzo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-19T14:24:18Z","updated_at":"2016-04-20T14:17:06Z","closed_at":null,"body":"Apparently most (all?) gocd developers agree that **storing** pipeline configuration\r\nin xml is bad. Secondly we also agree that moving away from existing xml-based\r\nconfiguration system is **way too much work** to even touch it.\r\nThis is a proposal on how we can *realisticly* move away from xml.\r\n\r\nI would say that theoretically **pipelines and environments and agents** are\r\ncandidates that should NOT be stored in xml. IMHO this is because these entities\r\nare **user data/input**, not **server configuration**.\r\n\r\nAfter working on #1133 and discussing it with many of you I had some ideas on\r\nhow we can move away from storing pipelines, (optionally) environments and agents.\r\n\r\n## Brief overview\r\n\r\n * continue to use `cruise-config.xml` as datastore for **server configuration** only. (`<server>`, `<repositories>`, `<config-repos>`, ...)\r\n * **eventually** stop storing user data entities as xml (`<pipelines>`, `<agents>`, `<environments>`) in git config repository.\r\n * **initially** add new, **second** data store for pipelines, agents and environments. (`ConfigOrigin=db`)\r\n * store above entities as JSON in database. Means we don't use sql schema for pipelines.\r\n * define JSON schemas, at least for pipelines.\r\n * use something like [Jolt transformations](https://github.com/bazaarvoice/jolt) to handle configuration migrations instead of `xsl`.\r\n * drop use of git merges to handle concurrent edits of configuration (pipelines only). Merge changes at object level instead.\r\n * in UI add new Config JSON tab to allow editing pipelines and environments in json format.\r\n\r\nFor rather desired backward compatibility with xml:\r\n\r\n * in general we still allow XML to be used for *transport*. I mean, XML as old API\r\n for users that depend on XML to change configuration.\r\n * keep the Config XML tab with `<pipelines>` and `<environments>` rendered,\r\n but their actual source is db. We advertise that this tab is obsolete.\r\n * similarly `/etc/go/cruise-config.xml` can still contain `<pipelines>`, `<environments>`,\r\n but that file is rendered lazily from database, it is not blocking actual\r\n configuration-changing operations.\r\n\r\n## Benefits and why\r\n\r\n * performance - #838 (still a problem right?).\r\n * elastic agents will only add more load in the `agents` section. It needs to be moved.\r\n * since #1133 now there is more than one **source of pipeline configuration**.\r\n There can more such sources. Not necessarily SCM-based, e.g. server fetches\r\n pipeline config from URL.\r\n * Moreover we can standardize such source to reuse it in other tools. There can be\r\n web UI to edit such source (like one in #1404), or there can be a desktop application\r\n for editing configuration in local files. Both would use same format for storage\r\n  and protocol (the pipeline config API?) for editing. Then SPA from #1404 can be\r\n  used to build standalone application (out of gocd server) for editing pipelines.\r\n * no one likes magical *too-much-reflective* xml loader and writer. Hard to debug, hard to maintain.\r\n * JSON is the new standard\r\n\r\n### Why is this realistic?\r\n\r\nIn #838 there were some suggestions to drop xml and more to something else, e.g. database. But main problem was that we need backwards compatibility with XML. \r\nIn #1133 we faced similar problem - there are other sources than XML which provide configuration to server. Apparently we can merge all these sources to provide consistent configuration service for Go server.\r\nPresuming that we want to keep pipeline configuration in database, we can add new configuration source `db` and handle it similarly to the partials in config repos. The key is that **`db` source can exist simultanously with the current xml data store**, thus allowing to phase-out the old xml config. E.g. new users/servers would not have xml pipelines at all, because by default adding a pipeline would be *routed* to db.\r\n\r\n## Final notes\r\n\r\n * So far I heard, the plan is to move agents from xml. But moving it to another\r\n xml file would be making the same error again.\r\n * To me, current xml data store looks like a growing tech debt. Yes it might have been working once,\r\n few years ago, when there were less pipelines, they were less dynamic, there was no API,\r\n the XML was the standard.\r\n * From user perspective this issue is not that important. I think it is more\r\n a question to gocd developers - do you want to drag this limited, slow,\r\n hard to maintain XML subsystem? Or would you rather work on top of new JSON-oriented\r\n system, designed for current/future needs?\r\n * I don't feel strongly about moving away environments from xml, since they are\r\n rather small and not so dynamic. However for the sake of argument, they are\r\n same *configuration category* as pipelines.\r\n\r\n I can elaborate on this some more if you're interested in moving into this direction.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/gocd/gocd/issues/comments/212444070","html_url":"https://github.com/gocd/gocd/issues/2121#issuecomment-212444070","issue_url":"https://api.github.com/repos/gocd/gocd/issues/2121","id":212444070,"user":{"login":"tomzo","id":5710970,"avatar_url":"https://avatars.githubusercontent.com/u/5710970?v=3","gravatar_id":"","url":"https://api.github.com/users/tomzo","html_url":"https://github.com/tomzo","followers_url":"https://api.github.com/users/tomzo/followers","following_url":"https://api.github.com/users/tomzo/following{/other_user}","gists_url":"https://api.github.com/users/tomzo/gists{/gist_id}","starred_url":"https://api.github.com/users/tomzo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomzo/subscriptions","organizations_url":"https://api.github.com/users/tomzo/orgs","repos_url":"https://api.github.com/users/tomzo/repos","events_url":"https://api.github.com/users/tomzo/events{/privacy}","received_events_url":"https://api.github.com/users/tomzo/received_events","type":"User","site_admin":false},"created_at":"2016-04-20T14:17:05Z","updated_at":"2016-04-20T14:17:05Z","body":"@zabil \r\n\r\n> I'd like to approach it like how we did with the new pipeline config api on which the new config ui #1404 is built.\r\n\r\nYou mean to build JSON API on top of *some* new config storage, regardless of its format (JSON or not)? I agree \r\n\r\nMy main point here is to create second config storage next to current XML.\r\n\r\n> The performance is much better in the new api.\r\n\r\nBTW, the reason for that is because there is less update-config-commands sent from UI to config store, right? I understand that now there will be just one submit-new-pipeline-config when user finished updating single pipeline. But the underlying handling of xml did not change much yet."}}