{"push_id":2421982238,"size":3,"distinct_size":3,"ref":"refs/heads/fnichol/install-offline","head":"9b7aa5c92f02d2d749fbd257ffb2fc67d2771d34","before":"1d556ffe1b42a4ec4a1a1a9db4cf6d7e33aec621","commits":[{"sha":"ec848da4368da14cea0e5f96cc4f165ec4cf503a","author":{"name":"Fletcher Nichol","email":"e5b43649bd990e696b89f2058fbf099adeaa2d7c@nichol.ca"},"message":"[common] Support an \"offline\" mode for the install logic.\n\nThis change introduces a new mode for the install logic to use in\nsituations where contacting a Builder API is not possible or not\nallowed. As this code is internal business logic, no external behavior\nhas (yet) been exposed, which will happen in future work. For now, the\ngroundwork is laid and is ready to be experimented with, mostly likely\nusing feature toggles in the `hab` CLI and the Supervisor.\n\nRationale\n---------\n\nThere are circumstances and situations where a healthy internet\nconnection isn't available and others where it is not permitted. For\nexample:\n\n* devices operating in environments where and internet connection\nis supported over a cellular or mesh network\n* demonstrating an aspect of Habitat in a hotel, conference, or\ninternet-less cafe\n* in a completely air-gapped environment with no ingress or egress\ncommunication allowed\n* a completely offline build environment\n\nAn implementation of the install logic that cannot talk to a Builder API\nwould have to derive its own solution for a \"latest\" package and have\ndifferent failure scenarios, therefore it seemed reasonable to make the\n\"offline\" behavior explicit and opt-in. That way, any tools or components\nusing this new logic could offer the end user an explicit opt-in to this\nbehavior so it's understood by all why certain errors must result in an\ninstall failure, etc.\n\nNetwork Calls in Online Mode\n----------------------------\n\nIn an effort to understand what an offline implementation would look\nlike, we should first describe all points at which the normal \"online\"\nlogic contacts Builder API endpoints over the network. Then for an\n\"offline\" behavior,  it's a matter of determining what is the reasonable\nequivalent and when we must loudly fail.\n\nCurrently, the following network calls are made (in order) to a single\nBuilder API instance (sometimes referred to as the `HAB_BLDR_URL`):\n\n1. If the given package identifier is not fully qualified (referred\nhereafter as a \"fuzzy\" package identifier), we call the `/latest`\nendpoint (for example `GET depot/channels/acme/stable/pkgs/foo/latest`\nin the case of trying to get the latest for an `acme/foo` package in the\n`stable` channel) in order to get back a fully-qualified package\nidentifier. Note that if the given identifier is fully-qualified we skip\nthe network call as there can only be *one* possible match for a\nfully-qualified identifier.\n2. If the target package is not already installed (that is, an extracted\npackage living in a `/hab/pkgs/<origin>/<name>/<version>/<release>`\npath) and if the target artifact (that is, a file ending in `.hart`) is\nnot already present in the artifact cache, then we call the package\ndownload endpoint (for example `GET\ndepot/pkgs/acme/foo/1.2.3/20180316084443/download` in the case of trying\nto download the `acme/foo/1.2.3/20180316084443` release) in order to get\nthe artifact into the artifact cache. Note that if the package is either\ninstalled or the artifact is cached then we skip this network call since\nartifacts and packages are immutable and will never change (therefore\nartifacts are infinitely cacheable).\n3. If the target package is not installed, the `TDEPS` metadata file\nwill be read and used to ensure every corresponding artifact is cached\nor installed. For each of these fully-qualified package identifiers, the\nsame determination is done as in step 2 above.\n4. For each package dependency in the target package's `TDEPS` metadata\nfile, and for the target package itself, the origin's public signing key\nis checked to make sure it's cached in the key cache. Each package is\nsigned with an origin key with a specific revision and so for any\nmissing public key-with-revision, we call the origin key download\nendpoint (for example `GET depot/origins/acme/keys/20180316084443` in\nthe case of trying to download the `acme-20180316084443.pub` public\norigin key) in order to get the public key into the key cache. Note that\nif the public key is already in the cache then we skip this network call\nsince a key-with-revision is also immutable and will never change.\n\nOffline Mode Alternatives to Network Calls\n------------------------------------------\n\nGiven the network calls above, here is what the offline strategy does,\nusing the same case numbering for clarity:\n\n1. (resolve fuzzy identifiers) We can only look at what is present on\ndisk, so there are 3 steps: i) find the latest locally installed\npackage, if any (no matching packages may be found) and record the\nfully-qualified package identifier, ii) find the latest cached artifact,\nif any (again, no matching artifacts may be found) and record the\nfully-qualified package identifier, and iii) take the latest winner\nbetween the latest installed package and the latest cached artifact. If\nwe can determine a winner, we can return that fully-qualified package\nidentifier and continue the logic. If no installed package or cached\nartifact can be found, we must return an error reporting that in offline\nmode we can't resolve this fuzzy package identifier as no local\ncandidates were found (offline failure 1a).\n2. (download target package) Now that we have a fully-qualified package\nidentifier, we need to find either this package installed locally or the\ncorresponding cached artifact. If either are found, the logic continues\nas normal. If neither are found, we must return an error reporting that\nin offline mode we don't have the package locally and can't continue\n(offline failure 2a).\n3. (download TDEPS) The same determination must be made for each\nruntime-dependent package which is the contents of the `TDEPS` metadata\nfile in the target package. For each package, step 2 is followed with\nthe same offline failure 2a returned if any dependency package cannot be\nfound locally.\n4. (download origin key-with-revision) Finally, we need to ensure that\nevery related package artifact has its public origin key present in the\nkey cache. As in step 2, if the key is not already present in the key\ncache, we must return an error reporting that in offline mode we don't\nhave the specific origin key and can't continue (offline failure 4a).\n\nClarifying \"Locally Installed Wins\"\n-----------------------------------\n\nIn working on this feature, it became clear that an always-present\nunderlying assumption has been at play with Habitat packages that was at\nodds with the flow of this install logic before this change set. It's\nthe idea that (to date), the concept of what is the \"latest\" package\nincludes looking at locally installed packages on disk. This means that\neven if the \"latest\" version is downloaded from a Builder API, it might\nbe older than a version already installed on a system. Most of the\nHabitat tooling and software will favor this newer version and\nshadow/mask out the older version--all assuming we're talking about a\nfuzzy package identifier.\n\nAn example might help make this more concrete. We'll assume we have a\nhost system running Habitat that has a version of an `acme/foo` package\nalready installed: `acme/foo/5.0/20180310120105`. We'll also assume that\nthis package is a service (i.e. we can run `hab start acme/foo`) and\nthat it contains a standalone binary program called `foobar`. So on\nthis system when we run `hab start acme/foo`, `hab pkg exec acme/foo\nfoobar`, or `hab pkg binlink acme/foo foobar`, we know this will resolve\nto the `acme/foo/5.0/20180310120105` release of `acme/foo`.\n\nIf we were to install an older version of `acme/foo`, let's say\n`acme/foo/2.0/20160614130203`, we know that `hab start acme/foo`, `hab\npkg exec acme/foo foobar`, and `hab pkg binlink acme/foo foobar` would\nall still resolve to the `acme/foo/5.0/20180310120105` release of\n`acme/foo`. In other words, `acme/foo/2.0/20160614130203` would be\n\"masked\" or \"shadowed\" by `acme/foo/5.0/20180310120105`. If we truly\ndesired to target the older version of `acme/foo`, we'd need to either\nfurther constrain our fully package identifier (i.e., use `acme/foo/2.0`\ninstead of `acme/foo`), or provide the fully qualified package\nidentifier to be 100% explicit.\n\nNow let's say that this local release `acme/foo/5.0/20180310120105` is\npresently installed because it is a yet unpublished artifact that is\nbeing tested or that it is a newly built version that is being used to\nbuild another package. Additionally, let's assume that our associated\nBuilder API has `acme/foo/3.0/20171113025501` as the latest package in\nits \"stable\" channel. The current behavior of the build program would\nstill pick the local version as \"latest\", despite downloading\n`acme/foo/3.0/20171113025501` and then immediately not use it (again,\nthis version is older and gets masked out by the newer locally installed\n`acme/foo/5.0/20180310120105`).\n\n(Sidenote) While this \"locally install wins\" behavior might seem strange\nin the context on building on first glance, it helped when bootstrapping\nthe Habitat package system, helps in iterating builds in development,\nquickly testing dependencies when regression testing, and has been the\nmechanism by which every package has been built in Habitat to date.\nThere's a good case to be made for altering the concept of \"latest\" in\nthe build case, although more work would need to happen to make this\nfully realized. However, the work in this changeset shouldn't interfere\nwith some future \"Builder API always wins\" semantics. Rather, more\ntooling is required like a standalone `hab pkg latest`-style command\nwhich would return a fully qualified package identifier given optional\nconstraints such as: factoring in locally installed packages (or not),\nfactoring in cached artifacts (or not), exclusively only using the\nBuilder API result, etc. To make a long story short, this distinction is\non our radar and may be in future work.\n\nIn order to make the \"locally install wins\" a bit more consistent, there\nare 2 edge case updates made to the install logic in the default/online\ncase (again, this assumes we are installing from a fuzzy package\nidentifier that needs to be resolved):\n\n1. The latest version of an installed package matching the identifier is\ndetermined (if any) before calling out to the Builder API endpoint for\n\"/latest\". If the locally installed version is newer than the latest\navailable in Builder (again, this request is always scoped to a release\nchannel), then a status is reported to the user that a newer version was\nfound locally and will be used. Finally, this is considered a successful\ninstallation in terms of the library logic and this would result in a\n\"clean install\" from `hab pkg install`. That is, an exit code of zero\nwould result on Unix-derived systems. This is a change from the previous\nbehavior where the known-older version would be downloaded, installed\nand then not used as the older version would be masked out. This change\nis made partly to save on download bandwidth and to inform the end user\nthat the \"latest\" is already present locally and did not come from the\nBuilder API.\n2. The second case relates to the first. In this case we still have a\nlocally installed package matching the identifier, but now there is no\npackage present in Builder for the specified release channel. Prior to\nthis change the user would get a warning with a list of package/channel\nmatches that a user could opt to try instead. This change will use the\nlocally installed package version (if found) and will report to the user\nthat it found a version of the package locally and could not find one\nremotely. As before, if a locally installed package is not present, the\nuser would still be presented with the warning and package/channel\nrecommendations.\n\nAdditional Changes\n------------------\n\nIn the course of supporting this mode, the following\nchanges/updates/fixes were made:\n\n* A `UI.begin()` message is now displayed to a user when installing a\npackage from an artifact file. It will look like `Installing\n./results/core-wget-static-1.19.1-20180316084443-x86_64-linux.hart` to\nmake it clearer that we are install from a local artifact and\ndownloading an artifact remotely.\n\nSigned-off-by: Fletcher Nichol <fnichol@nichol.ca>","distinct":true,"url":"https://api.github.com/repos/habitat-sh/habitat/commits/ec848da4368da14cea0e5f96cc4f165ec4cf503a"},{"sha":"d769b51b7b2090541df648cd04b9a329476822f8","author":{"name":"Fletcher Nichol","email":"e5b43649bd990e696b89f2058fbf099adeaa2d7c@nichol.ca"},"message":"[common] Use token if present when installing from archive.\n\nThis is a reasonably simple fix to the install-from-hart logic to\ninclude an optional Builder API token if provided, allowing any private\ndependent packages to be queried and downloaded.\n\nSigned-off-by: Fletcher Nichol <fnichol@nichol.ca>","distinct":true,"url":"https://api.github.com/repos/habitat-sh/habitat/commits/d769b51b7b2090541df648cd04b9a329476822f8"},{"sha":"9b7aa5c92f02d2d749fbd257ffb2fc67d2771d34","author":{"name":"Fletcher Nichol","email":"e5b43649bd990e696b89f2058fbf099adeaa2d7c@nichol.ca"},"message":"[common] Canonicalize paths when copying artifact to cache.\n\nThis change is intended to close any remaining edge conditions occurring\nwhen a user tries to install a `.hart` file directly from the artifact\ncache directory. For example:\n\n```\nhab pkg install /hab/cache/artifacts/core-redis-*.hart\n```\n\nWithout a sufficient guard, the install logic will effectively try to do\nthe following:\n\n```\ncp /hab/cache/artifacts/artifact.hart /hab/cache/artifacts/artifact.hart\n```\n\nThat is, the source file will be copied over and onto itself, usually\nresulting in a zero-sized file--obviously not what was expected.\n\nThere is currently a check for this in the install logic, however both\nthe source and target paths are not canonicalized (dereferencing\nsymlinks, relative paths, etc.). This change fixes that and adds a debug\nline in both the `true` and `false` arms of the check for debugging\ncompleteness.\n\nSigned-off-by: Fletcher Nichol <fnichol@nichol.ca>","distinct":true,"url":"https://api.github.com/repos/habitat-sh/habitat/commits/9b7aa5c92f02d2d749fbd257ffb2fc67d2771d34"}]}
{"actor":{"display_login":"fnichol"}}