{"action":"created","issue":{"url":"https://api.github.com/repos/NixOS/nixpkgs/issues/13819","repository_url":"https://api.github.com/repos/NixOS/nixpkgs","labels_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/13819/labels{/name}","comments_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/13819/comments","events_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/13819/events","html_url":"https://github.com/NixOS/nixpkgs/pull/13819","id":139963995,"number":13819,"title":"goPackages: make it easier to automate updates","user":{"login":"cstrahan","id":143982,"avatar_url":"https://avatars.githubusercontent.com/u/143982?v=3","gravatar_id":"","url":"https://api.github.com/users/cstrahan","html_url":"https://github.com/cstrahan","followers_url":"https://api.github.com/users/cstrahan/followers","following_url":"https://api.github.com/users/cstrahan/following{/other_user}","gists_url":"https://api.github.com/users/cstrahan/gists{/gist_id}","starred_url":"https://api.github.com/users/cstrahan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cstrahan/subscriptions","organizations_url":"https://api.github.com/users/cstrahan/orgs","repos_url":"https://api.github.com/users/cstrahan/repos","events_url":"https://api.github.com/users/cstrahan/events{/privacy}","received_events_url":"https://api.github.com/users/cstrahan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-03-10T17:52:46Z","updated_at":"2016-03-10T20:01:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/NixOS/nixpkgs/pulls/13819","html_url":"https://github.com/NixOS/nixpkgs/pull/13819","diff_url":"https://github.com/NixOS/nixpkgs/pull/13819.diff","patch_url":"https://github.com/NixOS/nixpkgs/pull/13819.patch"},"body":"This is a new PR based on the old #13701, which merged in ee3b295 and reverted in 7760d6e (see there for commentary re: process).\r\n\r\nTo restate what this does:\r\n\r\n#### What this does\r\n\r\nThis moves revisions and checksums to a separate, machine readable (specifically, JSON) file.\r\n\r\n#### Why?\r\n\r\nPresently, we have to go chasing each dependency of each go package in order to update revisions. This requires opening a browser, going to each GitHub repo, looking at the commits and tags, think long and hard about which revision to choose, open a terminal and get the sha256, then plug the revision and sha256 back into the `go-packages.nix` file. I tried updating the `serf` package by hand, and that took almost 3 hours.\r\n\r\nIdeally, a tool would provide a central way to pull the last couple commits and tags from each GitHub repo, and either completely automatically update the package set (or a given subset, like `serf` and it's closure), _or_ it would provide a UI for users that presents the revisions from upstream so you can make intelligent decisions about which revision to use (e.g. there are 4 new commits on master, but version 1.0.0 was tagged just a couple days ago... I think I'll go with tag).\r\n\r\nHowever, this commit doesn't place a dependency on such tooling (in fact, none exists as of yet), so we can consider the design later.\r\n\r\n#### Why JSON, and not Nix?\r\n\r\nThe Node and Haskell package sets are good examples of where we can update the whole package by regenerating the Nix expressions from scratch. That works for those ecosystems, because they have a culture of tagging releases. They also explicitly state their dependencies and the version bounds thereof. So, with minimal tweaks a separate file (e.g. `pkgs/development/haskell-modules/configuration-common.nix`) can extend the few troublesome packages with patches and such.\r\n\r\nHowever, that doesn't work for Go packages. Go projects don't explicitly list their dependencies and version bounds. Go projects don't tag releases[1]. If we want to have any chance of updating Go packages in an assisted or automatic way, we need to know what revisions we currently package.\r\n\r\n#### What this doesn't do\r\n\r\n* This does not place a dependency on a particular auto-packaging tool.\r\n* This does not make it any harder to update packages by hand.\r\n* This does not change the currently packaged versions.\r\n* This does not introduce massive rebuilds.\r\n* This does not change any interfaces in a backwards incompatible way.\r\n* This does not break any existing packages.\r\n\r\n--------\r\n\r\n[1]: If you're unfamiliar with the Go ecosystem, I repeat: they don't tag releases. _They don't have any sort of package repository_, so they all pull from each other's  master and hope everything works out."},"comment":{"url":"https://api.github.com/repos/NixOS/nixpkgs/issues/comments/195023398","html_url":"https://github.com/NixOS/nixpkgs/pull/13819#issuecomment-195023398","issue_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/13819","id":195023398,"user":{"login":"cstrahan","id":143982,"avatar_url":"https://avatars.githubusercontent.com/u/143982?v=3","gravatar_id":"","url":"https://api.github.com/users/cstrahan","html_url":"https://github.com/cstrahan","followers_url":"https://api.github.com/users/cstrahan/followers","following_url":"https://api.github.com/users/cstrahan/following{/other_user}","gists_url":"https://api.github.com/users/cstrahan/gists{/gist_id}","starred_url":"https://api.github.com/users/cstrahan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cstrahan/subscriptions","organizations_url":"https://api.github.com/users/cstrahan/orgs","repos_url":"https://api.github.com/users/cstrahan/repos","events_url":"https://api.github.com/users/cstrahan/events{/privacy}","received_events_url":"https://api.github.com/users/cstrahan/received_events","type":"User","site_admin":false},"created_at":"2016-03-10T20:01:00Z","updated_at":"2016-03-10T20:01:00Z","body":"@edolstra \r\n\r\n> It introduces yet another auto-update mechanism, and we already have a lot of those, most of them only used by the person who introduced them. (E.g. gnupdate, the .upstream files, language-specific mechanisms...) This too raises the burden for someone else to modify a package because they have to figure out how the particular auto-updater for that package works. \r\n\r\nWith what I propose, we won't lose the ability to continue updating packages as we currently do, that is, manually. When a tool is created, I would definitely want to see it documented, so people could benefit from it. Unlike the Haskell case, however, we _could_ continue to update packages by hand without having them clobbered and rolled back upon running an update tool (which is something I've had happen long ago when I first tried contributing something Haskell related).\r\n\r\n> It introduces the notion of writing packages in JSON format rather than as Nix expressions.\r\n\r\nI could make the format Nix, and just parse (or evaluate) the file in the (as of yet hypothetical) tool I suggest using to update package revisions.\r\n\r\n> This too raises the burden for someone else to modify a package because they have to figure out how the particular auto-updater for that package works. We should really come up with a universal auto-update tool, or at least a standard for what they should do (e.g. update Nix expressions in place or overwrite an external version file?).\r\n\r\nIf it's possible to pull off a single, universal updater, then I agree -- that sounds great. However, I think that's a monumental effort, vs creating a few specialized tools, and providing really good documentation on how to use them (which is precisely what I intend to do once I've written such a tool, but that comes next). I would argue that the burden of needing to know how to use a tool for automated or assisted updates far outweighs the present burden of updating Go packages.\r\n\r\n> It further entrenches go-packages.nix, which I already objected to when it was introduced. We don't have a giant c-packages.nix for everything written in C either.\r\n\r\nSpeaking of burden, I think there's an important difference to keep in mind between Go and C packages. The release cycle for C libs usually spans something between months and years; the go release cycle is effectively each commit -- if you don't keep up with changes, your stuff is broken. One change in a single library causes a cascade of required version bumps. C libs have versioned releases; Go packages don't.\r\n\r\nTrying to keep up with changes by hand is going to become an intractable ordeal as we package more Go packages. I wanted to update `serf`, `consul` and `vault` for a presentation I planned to give at work at the beginning of this week (totally missed that goal), and updating `serf` alone took over 2 hours of my time. As is evidenced by my previous work updating other C libs and apps, the C ecosystem usually requires just a couple minutes of effort to find the latest version, update the version in the nix expression (which is usually sufficient to update the url), do a quick `nix-prefetch-url -A somePackage.src` and plug the resulting sha256 back into the expression. In the go world, I would literally draw a pentagram on the ground and shower myself in goat's blood if it would somehow let me update Go packages that quickly."}}