{"action":"created","issue":{"url":"https://api.github.com/repos/blockstack/blockstack.js/issues/699","repository_url":"https://api.github.com/repos/blockstack/blockstack.js","labels_url":"https://api.github.com/repos/blockstack/blockstack.js/issues/699/labels{/name}","comments_url":"https://api.github.com/repos/blockstack/blockstack.js/issues/699/comments","events_url":"https://api.github.com/repos/blockstack/blockstack.js/issues/699/events","html_url":"https://github.com/blockstack/blockstack.js/issues/699","id":482452098,"node_id":"MDU6SXNzdWU0ODI0NTIwOTg=","number":699,"title":"Proposal: Decouple Profile Resolution from Zonefile URLs","user":{"login":"jcnelson","id":459947,"node_id":"MDQ6VXNlcjQ1OTk0Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/459947?v=4","gravatar_id":"","url":"https://api.github.com/users/jcnelson","html_url":"https://github.com/jcnelson","followers_url":"https://api.github.com/users/jcnelson/followers","following_url":"https://api.github.com/users/jcnelson/following{/other_user}","gists_url":"https://api.github.com/users/jcnelson/gists{/gist_id}","starred_url":"https://api.github.com/users/jcnelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcnelson/subscriptions","organizations_url":"https://api.github.com/users/jcnelson/orgs","repos_url":"https://api.github.com/users/jcnelson/repos","events_url":"https://api.github.com/users/jcnelson/events{/privacy}","received_events_url":"https://api.github.com/users/jcnelson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"jcnelson","id":459947,"node_id":"MDQ6VXNlcjQ1OTk0Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/459947?v=4","gravatar_id":"","url":"https://api.github.com/users/jcnelson","html_url":"https://github.com/jcnelson","followers_url":"https://api.github.com/users/jcnelson/followers","following_url":"https://api.github.com/users/jcnelson/following{/other_user}","gists_url":"https://api.github.com/users/jcnelson/gists{/gist_id}","starred_url":"https://api.github.com/users/jcnelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcnelson/subscriptions","organizations_url":"https://api.github.com/users/jcnelson/orgs","repos_url":"https://api.github.com/users/jcnelson/repos","events_url":"https://api.github.com/users/jcnelson/events{/privacy}","received_events_url":"https://api.github.com/users/jcnelson/received_events","type":"User","site_admin":false},"assignees":[{"login":"jcnelson","id":459947,"node_id":"MDQ6VXNlcjQ1OTk0Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/459947?v=4","gravatar_id":"","url":"https://api.github.com/users/jcnelson","html_url":"https://github.com/jcnelson","followers_url":"https://api.github.com/users/jcnelson/followers","following_url":"https://api.github.com/users/jcnelson/following{/other_user}","gists_url":"https://api.github.com/users/jcnelson/gists{/gist_id}","starred_url":"https://api.github.com/users/jcnelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcnelson/subscriptions","organizations_url":"https://api.github.com/users/jcnelson/orgs","repos_url":"https://api.github.com/users/jcnelson/repos","events_url":"https://api.github.com/users/jcnelson/events{/privacy}","received_events_url":"https://api.github.com/users/jcnelson/received_events","type":"User","site_admin":false},{"login":"kantai","id":1153552,"node_id":"MDQ6VXNlcjExNTM1NTI=","avatar_url":"https://avatars1.githubusercontent.com/u/1153552?v=4","gravatar_id":"","url":"https://api.github.com/users/kantai","html_url":"https://github.com/kantai","followers_url":"https://api.github.com/users/kantai/followers","following_url":"https://api.github.com/users/kantai/following{/other_user}","gists_url":"https://api.github.com/users/kantai/gists{/gist_id}","starred_url":"https://api.github.com/users/kantai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kantai/subscriptions","organizations_url":"https://api.github.com/users/kantai/orgs","repos_url":"https://api.github.com/users/kantai/repos","events_url":"https://api.github.com/users/kantai/events{/privacy}","received_events_url":"https://api.github.com/users/kantai/received_events","type":"User","site_admin":false},{"login":"zone117x","id":1447546,"node_id":"MDQ6VXNlcjE0NDc1NDY=","avatar_url":"https://avatars1.githubusercontent.com/u/1447546?v=4","gravatar_id":"","url":"https://api.github.com/users/zone117x","html_url":"https://github.com/zone117x","followers_url":"https://api.github.com/users/zone117x/followers","following_url":"https://api.github.com/users/zone117x/following{/other_user}","gists_url":"https://api.github.com/users/zone117x/gists{/gist_id}","starred_url":"https://api.github.com/users/zone117x/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zone117x/subscriptions","organizations_url":"https://api.github.com/users/zone117x/orgs","repos_url":"https://api.github.com/users/zone117x/repos","events_url":"https://api.github.com/users/zone117x/events{/privacy}","received_events_url":"https://api.github.com/users/zone117x/received_events","type":"User","site_admin":false},{"login":"yknl","id":29081231,"node_id":"MDQ6VXNlcjI5MDgxMjMx","avatar_url":"https://avatars3.githubusercontent.com/u/29081231?v=4","gravatar_id":"","url":"https://api.github.com/users/yknl","html_url":"https://github.com/yknl","followers_url":"https://api.github.com/users/yknl/followers","following_url":"https://api.github.com/users/yknl/following{/other_user}","gists_url":"https://api.github.com/users/yknl/gists{/gist_id}","starred_url":"https://api.github.com/users/yknl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yknl/subscriptions","organizations_url":"https://api.github.com/users/yknl/orgs","repos_url":"https://api.github.com/users/yknl/repos","events_url":"https://api.github.com/users/yknl/events{/privacy}","received_events_url":"https://api.github.com/users/yknl/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2019-08-19T18:06:52Z","updated_at":"2019-09-05T23:53:26Z","closed_at":null,"author_association":"MEMBER","body":"# Proposal\r\n\r\nUser profiles are increasingly used to host control-plane state in BNS.  What this means is that we need to do a better job of ensuring that the control-plane information remains as available as possible.  In particular, the inability to resolve a user's profile means that a user's multiplayer data remains inaccessible, even if it is online.\r\n\r\nIn this proposal, I recommend adding a field in the user's profile called `api.bns.replicas`.  This field maps to some state that includes a URL that points to the primary replica of the user's profile (as well as any authoritative secondary replicas), if different from that in the zone file.  The point of this field is to (a) make it easy for users to change where their latest profile is hosted, if different from the zone file, and (b) make it possible for profiles to be stored and resolved by many different non-authoritative nodes called _profile resolvers_.\r\n\r\nThe ultimate purpose of adding this field is to make it possible to remove the need for a zone file URL to remain valid forever.  To achieve this, the proposal for `blockstack.js` documented here is coupled with another proposal to build up a network of profile resolvers, just as we currently do with zone files.  Each profile resolver node can host non-authoritative replicas of profiles on behalf of _other users_ who want to resolve the profile.  If the user's profile's `api` struct is sufficiently fresh on them, then any such node can direct other clients towards the user's authoritative replicas.\r\n\r\nThis issue was touched on briefly [here](https://github.com/blockstack/blockstack-browser/issues/1910).\r\n\r\n## Background\r\n\r\nResolving a name to a profile is a two-step process today:  (1) fetch and authenticate the name's zone file, and (2) use the `URI` record to fetch and authenticate the name's associated profile.  The original thinking in the BNS system design was that the user should always be able to control where their profile could be found.  At the time, it was thought that the user's profile and zone file would be tightly coupled -- the zone file information would always be followed, and the information it contained always need to be chain-anchored in order to render it as difficult as possible to censor profile URL updates.  The current design does this by ensuring that all BNS nodes have a copy of the latest `URI` record (by means of having the latest zone file data for all BNS names), and if they don't, they can at least inform clients that the zone file they need is missing.\r\n\r\nHowever, there are several limitations to this approach:\r\n\r\n* Changing a profile URL is expensive and time-consuming -- someone has to send a blockchain transaction (either a subdomain update batch, or an on-chain `NAME_UPDATE`).\r\n\r\n* The benefits of requiring a blockchain transaction each time the user changes their profile URL are questionable in practice.  In particular, if a popular Gaia hub goes offline permanently, the act of mass-migrating its many users away from it will make the transaction-generation process slower and more expensive and will upset many users (who may have their data knocked offline for some time -- at least, until the transactions confirm).\r\n\r\n* Related to what happens if a Gaia hub crashes and goes offline, right now the BNS system has no way to allow users to specify alternative replicas of their profile, and offers no way to propagate knowledge of other authoritative replicas to Blockstack Core nodes.  This is problematic because it makes it possible for a large Gaia hub failure to prevent off-chain data from being resolved -- even if the user runs their own Gaia hub for app-specific data!\r\n\r\n* When the user registers a BNS name, they usually do not appreciate the need to run their own Gaia hub or choose one besides the default.  If they discover the need to do so later, they need to go through the difficult process of updating their BNS zone file.\r\n\r\nWhat I'm suggesting we do is decouple a BNS name's profile from its zone file URL. We could remove the need to send a zone file update altogether by treating the zone file URL as the _initial replica URL_ instead of the _primary replica URL_, and use some state in the profile itself to help clients determine where the primary replica can be found.  In the design below, we would be able to make it so a user's profile could be hosted on many different Gaia hubs (including stale copies), but the user would be able to designate which replica should be treated as fresh.\r\n\r\n## Design\r\n\r\nBy far the most important pieces of control-plane information obtained from the user's profile are:\r\n\r\n* The user's owner public key\r\n* The URLs to each multiplayer application's Gaia hubs\r\n* The Gaia hub configuration\r\n\r\nAll of this information must be present and available to clients in order to interact with other users' data.  Right now, we don't do a good job of ensuring that this data is highly available -- it is only as available as the user's initially-chosen Gaia hub (which they almost certainly did _not_ choose themselves).\r\n\r\nWhat I recommend is making it so profiles contain an additional piece of control-plane state to point to the user's authoritative profile replicas.  That is, the user can decide that the profile pointed to by the zone file is _not_ the profile's primary replica, and can do so completely off-chain.  While this adds one extra step to resolving the user's profile, the upsides are that (a) users now have a way to migrate their profiles off of Gaia hubs without a transaction, (b) users can ensure their control-plane data is available from many sources, and (c) clients can use non-authoritative resolvers to direct them towards authoritative profile replicas.\r\n\r\nTo achieve all this, the user's profile would store the following extra fields:\r\n\r\n* `api.bns.replicas` -- this contains an entry called `urls` that is a list of `url` and `ttl` fields:\r\n   * `api.bns.replicas.urls[].url` -- this is a URL to the user's profile replica.\r\n   * `api.bns.replicas.urls[].ttl` -- this is the time-to-live for caching the profile replica URL, and is used by downstream and non-authoritative profile resolvers to help clients find an authoritative replica.\r\n* `api.bns.replicas.sequence` -- this is a monotonically-incrementing sequence number that the profile owner increments whenever they replicate new profile data.  The sequence number is used to invalidate older cached data in the profile resolver network.\r\n\r\nThe profile resolution algorithms would work as follows.  The default slow-path case would be:\r\n\r\n```\r\n(1) given a BNS name, look up the zone file on a Core node and authenticate it\r\n(2) given the profile URL, resolve it to the initial profile replica and authenticate it\r\n(3) get the api.bns.replicas[0].url field if present.  If not, return the profile.\r\n(4) if the api.bns.replicas[0].url is different than the URL in (2), fetch the profile it points to\r\n(5) verify that the profile is signed by the same key as the initial profile replica\r\n(6) if the new profile has a different api.bns.replicas[0].url field, go to (2)\r\n(7) otherwise, return the profile\r\n```\r\n\r\nStep (2) can fail of the primary replica URL is set, but the data unavailable.  If this happens, then `api.bns.replicas[1].url` is tried, and so on.\r\n\r\nStep (1) can fail if the initial replica URL does not resolve to valid profile data.  If this happens, then the user would fall back to a non-authoritative profile resolver (see below) and use it to find an authoritative replica.  The client would need to select one or more non-authoritative resolvers _a priori_.\r\n\r\nTo resolve a profile from a non-authoritative profile resolver, the client would:\r\n\r\n```\r\n(1) given a BNS name and a non-authoritative resolver, query the cached api.bns.replicas state\r\n(2) verify that the state was signed by the BNS name owner (querying the key from a Core node)\r\n(3) for each entry in api.bns.replicas, try to fetch the profile\r\n(4) on success, verify that the profile was signed by the BNS name owner.  If not, go to (3)\r\n(5) verify that the successfully-fetched URL is present in the profile's api.bns.replicas.  If not, go to (3)\r\n(6) return the first profile to succeed\r\n```\r\n\r\nClients should limit the number of profile resolution attempts made, discussed in the implementation section below.\r\n\r\nClients would learn a particular BNS name owner's preferred profile resolvers via a `_profile_resolver` URI entry in either their name's zone file, or the on-chain name's zone file.  The non-authoritative resolvers should be tried in the following order:\r\n\r\n* The URL in the user's name's zone file's `_profile_resolver` URI record, if given\r\n* If the name is off-chain, then the `profile_resolver` URI record in the corresponding on-chain name's zone file may be considered as well.\r\n* The set of public resolver nodes, described [here](https://github.com/blockstack/blockstack-core/issues/1095).\r\n\r\n## Profile Updates\r\n\r\nIf for whatever reason a profile is rendered unresolvable, the user now has two courses of action:\r\n\r\n* Change the `api.bns.replicas` URLs and re-upload the profile to both the _initial_ and _new_ locations, if possible.  Also, upload the new profile to as many non-authoritative resolvers as possible.\r\n* Send a blockchain transaction that changes the initial replica URL to the desired primary replica URL.\r\n\r\nThe first option is preferred if possible because it can be done near-instantaneously and does not require a transaction.  Profile resolvers would need to allow the user to easily update their cached copies of the `api.bns.replicas` state.\r\n\r\nThe second option is slower and more expensive, but can be done far more cheaply for off-chain names via any off-chain subdomain registrar (recall that by design, any subdomain registrar can send an off-chain `NAME_UPDATE` for any other subdomain).\r\n\r\n## Profile Resolver Network\r\n\r\nThe initial replica URL may stop resolving to fresh profile data for a multitude of reasons.  If the user cannot or does not want to issue a new initial replica URL via a transaction, then the user would instead (a) replicate their profile to new locations and (b) replicate the new URLs to a network of non-authoritative profile resolvers.\r\n\r\nIf we're going to completely do away with the need for a zone file update, then we need to build out a network of profile resolvers that build up a cached mapping between a BNS name and its latest `api.bns.replica` control-plane state.  Anyone can run a profile resolver -- there is no coordination required between them, since a profile resolver only needs to be able to access the BNS name state to authenticate new profile information.  Profile resolvers would be designed to be able to serve as \"downstream replicas\" of other profile resolvers, and the client software would be aware of a set of \"well-known\" resolver nodes.\r\n\r\nIn fact, I'd like to ultimately make it an add-on service for Blockstack Core nodes, so any such Blockstack Core node can resolve a BNS name to its latest profile replica this way.  Moreover, personal-use Blockstack Core nodes that run on users' computers could be configured to lazily cache `api.bns.replicas` state on name lookups, so that a user that frequently interacts with other users can quickly learn their latest profile replica URLs.\r\n\r\nProfile resolvers are not part of the trusted computing base, nor are they part of the consensus code.  They simply host downstream replicas of the user's profile in order to help clients find the user's authoritative replicas.  The worst they can do to clients is serve stale profiles, but this can be fixed by the user simply by either (1) re-keying their name, or (2) changing their initial replica in the zone file to point to their primary replica in their profile.\r\n\r\nThe proposed design of a profile resolver node is described [here](https://github.com/blockstack/blockstack-core/issues/1095).  The availability of such a network would serve to allow users to update their profile URLs in a way that is still decentralized but without requiring a blockchain transaction.\r\n\r\n## Implementation\r\n\r\nImplementing this proposal can be done in a backwards-compatible way.  All that needs to happen in `blockstack.js` is the `lookupProfile` method needs to be aware of `api.bns` state and handle it according to the above algorithms.\r\n\r\n### Validation and Caveats\r\n\r\n* The `api.bns` structure must be well-formed per the rules described above.  The `replicas` field must be a list, and each item must be an object with a `url` field that is a valid HTTP(S) URL, and an optional `ttl` field that is a non-negative integer.\r\n\r\n* The `lookupProfile` method should not attempt to make more than N sequential requests and M total requests to resolve a profile, in order to not keep the user waiting or suck up all of their bandwidth.  M and N can be something like 5 by default, but a caller may choose to override it.\r\n\r\n* The `lookupProfile` method needs to take a URL to at least one profile resolver as an optional parameter.  The RESTful interface to this resolver is documented with the profile resolve network \r\nissue [here](https://github.com/blockstack/blockstack-core/issues/1095)\r\n\r\n## Conclusion\r\n\r\nThe need to make profile resolution work without zone files will be necessary for the long-term success of this project. This proposal adds an entry to the user's profile that tells clients where to go and look for the _current_ profile, and treats the zone file URL as a URL to a stale profile that only needs to contain this pointer.  A sibling issue describes how to build a network of profile resolvers that allow the user to disseminate the location of their latest profile URLs to a wide variety of hosts, similar to how we do now with zone files.\r\n\r\nThoughts and feedback are welcome!\r\n"},"comment":{"url":"https://api.github.com/repos/blockstack/blockstack.js/issues/comments/528650755","html_url":"https://github.com/blockstack/blockstack.js/issues/699#issuecomment-528650755","issue_url":"https://api.github.com/repos/blockstack/blockstack.js/issues/699","id":528650755,"node_id":"MDEyOklzc3VlQ29tbWVudDUyODY1MDc1NQ==","user":{"login":"jcnelson","id":459947,"node_id":"MDQ6VXNlcjQ1OTk0Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/459947?v=4","gravatar_id":"","url":"https://api.github.com/users/jcnelson","html_url":"https://github.com/jcnelson","followers_url":"https://api.github.com/users/jcnelson/followers","following_url":"https://api.github.com/users/jcnelson/following{/other_user}","gists_url":"https://api.github.com/users/jcnelson/gists{/gist_id}","starred_url":"https://api.github.com/users/jcnelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcnelson/subscriptions","organizations_url":"https://api.github.com/users/jcnelson/orgs","repos_url":"https://api.github.com/users/jcnelson/repos","events_url":"https://api.github.com/users/jcnelson/events{/privacy}","received_events_url":"https://api.github.com/users/jcnelson/received_events","type":"User","site_admin":false},"created_at":"2019-09-05T23:53:26Z","updated_at":"2019-09-05T23:53:26Z","author_association":"MEMBER","body":"> First, designating profile_resolver or _profile_resolver entries in the zonefile doesn't seem to add much benefit -- it just puts a user in the same relationship with a profile resolver that they currently have with their primary hub. However, it complicates the proposal by requiring a new field be added to registrations, be parsed by blockstack.js, and handled as an additional control path in lookups. I think we can just eliminate it.\r\n\r\nI'm fine with omitting this for now.  We can always add it in later if finding a working resolver proves problematic.\r\n\r\n> Second -- I think that the use of the term non-authoritative with respect to profile resolvers is somewhat misleading. They cannot forge profiles, but they would have more or less the same power as the authoritative profile URL from a user's zone file (namely, the power to replay old profiles). The difference is that they will never be the \"first\" in a lookup path.\r\n\r\nI think they can be much more prone to serve stale profiles by accident, especially if the user does not always upload new profiles to them.  In particular, a profile resolver may be serving a profile that the user did not upload themselves, but that someone else did on their behalf.\r\n\r\n> Third -- I don't think the sequence number adds much for clients, and profile resolvers could instead use the standard cache-invalidation headers supplied by HTTP, which have the advantage of being present in HEAD requests.\r\n\r\nIt's used for two other purposes:  (1) to show clients how old the profile actually is according to the user who signed it (the HTTP headers may indicate a different timestamp -- i.e. how old the replica is on this resolver), an (2) to allow correct profile resolvers to prevent other people from uploading stale profiles.  I think we should rename the \"sequence\" number to be the \"epoch timestamp\" if it better serves these purposes and more clearly indicates these intentions.\r\n\r\n> Allow a recursive lookup in profiles to allow a profile to point to an ordered list of other profiles.\r\n\r\nWhy recursive?  I'd prefer iterative, since it's less intense on the resolver.  If you want the resolver to assist with the resolution, then the resolver should probably just return HTTP 302 for the next replica to try (so \"vanilla\" Web clients can fall over to fetching profiles from different resolvers automatically).\r\n\r\n> In the event that a profile URL is unavailable, allow lookups to initiate from a public profile cache.\r\n\r\nI think the client needs to explicitly switch to querying a public profile cache, so that the client logic can inform the caller which resolver was used (and whether or not it was user-sanctioned).\r\n\r\n> but (2) is also a source of changes in the user threat model --- it's no longer only user-designated Gaia hubs that can perform rollback attacks, but also public resolvers. \r\n\r\nI think stale data can happen by accident with far more frequency in this model, simply by virtue of the fact that profile replication is now asynchronous and eventually-consistent.  I think that as long as the client knows (1) the user-signed timestamp of the profile, and (2) which resolver(s) were tried (and whether or not they were user-sanctioned), then the client can make an informed decision as to what to do with the profile it eventually gets."}}
{"actor":{"display_login":"jcnelson"}}