{"action":"created","issue":{"url":"https://api.github.com/repos/CenterForOpenScience/osf.io/issues/4745","labels_url":"https://api.github.com/repos/CenterForOpenScience/osf.io/issues/4745/labels{/name}","comments_url":"https://api.github.com/repos/CenterForOpenScience/osf.io/issues/4745/comments","events_url":"https://api.github.com/repos/CenterForOpenScience/osf.io/issues/4745/events","html_url":"https://github.com/CenterForOpenScience/osf.io/pull/4745","id":122318744,"number":4745,"title":"[READY FOR REVIEW] [feature] Add Edge-Side-Include functionality to API #4669","user":{"login":"cwisecarver","id":3709,"avatar_url":"https://avatars.githubusercontent.com/u/3709?v=3","gravatar_id":"","url":"https://api.github.com/users/cwisecarver","html_url":"https://github.com/cwisecarver","followers_url":"https://api.github.com/users/cwisecarver/followers","following_url":"https://api.github.com/users/cwisecarver/following{/other_user}","gists_url":"https://api.github.com/users/cwisecarver/gists{/gist_id}","starred_url":"https://api.github.com/users/cwisecarver/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cwisecarver/subscriptions","organizations_url":"https://api.github.com/users/cwisecarver/orgs","repos_url":"https://api.github.com/users/cwisecarver/repos","events_url":"https://api.github.com/users/cwisecarver/events{/privacy}","received_events_url":"https://api.github.com/users/cwisecarver/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":133,"created_at":"2015-12-15T17:02:40Z","updated_at":"2016-01-05T16:35:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/CenterForOpenScience/osf.io/pulls/4745","html_url":"https://github.com/CenterForOpenScience/osf.io/pull/4745","diff_url":"https://github.com/CenterForOpenScience/osf.io/pull/4745.diff","patch_url":"https://github.com/CenterForOpenScience/osf.io/pull/4745.patch"},"body":"## Description\r\n\r\n- **Purpose**\r\n    - [Edge-Side-Includes](https://en.wikipedia.org/wiki/Edge_Side_Includes) allow Varnish to replace specific xml tags with the contents of the url specified at the href property.\r\n        - Example: `<esi:include href=\"http://www.google.com\"/>` would be replaced with the contents of the google homepage.\r\n    - The embed functionality in the API makes this very useful since a user embedded into a node is the same as a user requested individually. So varnish can cache `/v2/users/1234z/` and anytime that user is embedded in a request it can use the cached version replacing the `<esi:include href=\"/v2/users/1234z/\"/>` with the full json representation of the user.\r\n\r\n- **Changes**\r\n    - There is a `to_esi_representation` method that was added on certain serializers. This method takes the place of the `to_representation` when the `ENABLE_ESI` setting is `True` and a resource is embedded.\r\n    - Instead of python embedding the full content of the user as it does with to_representation it responds with `<esi:include href=\"{{ self.get_absolute_url() }}\" />`. When the response is processed by Varnish it will do a request for the specified href and replace the xml tag with the response from that request and caching the response. The next time that url is included in an esi tag it will be pulled from cache.\r\n    - Blinker events were added to GuidStoredObject that fire upon model save. When these events fire they're caught by a listener that fires off a celery task to send a `BAN` HTTP request to the varnish server in the pattern of `BAN {{ self.get_absolute_url }}.*`. This is processed as a regex and should account for all possible urls that represent that object. Since objects are stored by varnish individually, even when they're embedded in another object, this pattern should remove all instances, even those that are embedded, from cache.\r\n        - `BAN /v2/users/1234z/.*`, to varnish, means add this pattern to the list of invalid url patterns. When someone requests a URL that matches an invalid url pattern make sure to request it from the backend server instead of serving it from cache. BANning instead of PURGEing means that we can avoid the thundering herd problem that commonly occurs with cache invalidation and it also means that we're not actually emptying the cache until we need to.\r\n\r\n- **Side Effects**\r\n\t- URLs are supposed to represent unique data. Since we combine URLs and user authentication information to represent unique data this complicates caching. Varnish's configuration is VARYing on osf cookie, Authorization HTTP header, and URL. This **should** account for different users hitting the same url and getting a different result based on their individual permissions. There is always a chance that I've missed something. My hope is that the only side-effect of this is a low cache hit-rate. Meaning that since we're caching things per user our cache won't be utilized as well as it could for non-anonymous users.\r\n\t- Embed code is somewhat in flux right now. I've been basing my changes off of the two PRs that Dawn has been working on related to embeds but they have been changing while I've been working. I've tried to keep this branch as up to date with her stuff as I possibly could but there is a chance that a merge has gone sideways or I've missed a PR that was related to the embed work.\r\n\t- Since everything is behind a setting, unless you've switched ENABLE_ESI to true there should be no change from this PR.\r\n    - The JSONAPIRenderer has a bit of code in it that I would really like to refactor but I couldn't find a better way to do it. Right now, because ESI is XML and all our stuff is JSON, DRF is quoting the value of the `to_esi_representation` method. Varnish doesn't expect ESI tags to be in quotes so `\"<esi:include href=\"something\"/>\"` is ignored. The JSONAPIRenderer.render method has a regex that removes the double-quotes from the outside of the xml tag. This causes invalid JSON to be passed to varnish and varnish replaces the ESI tag prior to the client seeing it making it valid JSON again. While I love regex it can be brittle. Advice and alternate approaches are welcomed.\r\n\r\n\t\r\n## Test Setup\r\n\r\nFor testing/CR, you'll need to install and setup varnish. A sample VCL file is attached to this PR. VCL is a domain-specific language designed to be used to define request handling and document caching policies for Varnish Cache. It is compiled when varnish starts up. The final VCL, very similar to the attached one, is included in the private docker repo.\r\n\r\n- **Assumptions**\r\n\t- Your API server is running on port 8000\r\n\t- You want Varnish to listen on port 8193\r\n\t- You've got 2 gigs of free ram for varnish to use for caching (change `-s malloc,2G` in the varnish startup command to something else, `-s malloc,256M` for instance)\r\n- **Steps**\r\n\t- `brew update; brew install varnish`\r\n\t- Download the attached VCL file and copy it to `/usr/local/etc/varnish/default.vcl` or wherever your varnish installation is. [default.vcl.txt](https://github.com/CenterForOpenScience/osf.io/files/48741/default.vcl.txt)\r\n\t- Startup varnish with the following command `sudo /usr/local/opt/varnish/sbin/varnishd -n /usr/local/var/varnish -f /usr/local/etc/varnish/default.vcl -s malloc,2G -T 127.0.0.1:2000 -a 127.0.0.1:8193 -p feature=+esi_disable_xml_check -p workspace_backend=262144 -p workspace_client=262144`\r\n\t- Change the `ENABLE_ESI` setting in your `api/base/settings/local.py` to `True`\r\n\t- Change the `ENABLE_VARNISH` setting in your `api/base/settings/local.py` to `True`\r\n\t- Set the `VARNISH_SERVERS` setting in your `api/base/settings/local.py` to `['127.0.0.1:8193']`\r\n\t- Navigate to `http://localhost:8193/v2/nodes/?format=jsonapi&embed=contributors&page[size]=100`. In your `inv apiserver` console you should see the initial request followed by several other requests for individual contributors. This is varnish replacing the ESI tags with the individual responses. If you refresh this page you should see your pageload time decrease significantly and you should see no requests in your `inv apiserver` console.\r\n\t- Now update a contributor or node that was included in your previous request. If celery is running in your development environment the next time you request the list of nodes you should see the node and or the contributor you updated be requested from the API server but the other components should be pulled from the varnish cache. This means that the signal fired upon save, the celery task banned that content from varnish, and varnish re-requested the banned content the next time you needed it.\r\n\t- Making the same list-of-nodes request with either a non-logged-in user or logged in as a different user should cause requests to the python apiserver. But repeated requests as a non-logged in user or repeated requests as the same logged in user should be drawn from cache instead of making a request to the python apiserver.\r\n\r\n## Random informations\r\n\r\n- Varnish is **ONLY** configured to cache GET and HEAD requests where the URL starts with /v2/. Anything that does not meet that criteria will be passed directly to the python api server.\r\n- If you open the developer console in your browser and have a look at the response headers you should see `X-Cache: HIT` (it was a cache hit) and `X-Cache-Hits: #` (there have been # hits on this version of the cached object) and `Age: ##` (the cached object is ## seconds old) and `X-beresp-ttl: 120.000` (the time-to-live of this cached object is 120.000 seconds). When `Age` == `X-beresp-ttl` the object is removed from the cache.\r\n- We can send headers back from DRF to control the TTL based on whatever we'd like. Individual objects could be cached for days or seconds. This is not implemented yet.\r\n"},"comment":{"url":"https://api.github.com/repos/CenterForOpenScience/osf.io/issues/comments/169053011","html_url":"https://github.com/CenterForOpenScience/osf.io/pull/4745#issuecomment-169053011","issue_url":"https://api.github.com/repos/CenterForOpenScience/osf.io/issues/4745","id":169053011,"user":{"login":"whit537","id":134455,"avatar_url":"https://avatars.githubusercontent.com/u/134455?v=3","gravatar_id":"","url":"https://api.github.com/users/whit537","html_url":"https://github.com/whit537","followers_url":"https://api.github.com/users/whit537/followers","following_url":"https://api.github.com/users/whit537/following{/other_user}","gists_url":"https://api.github.com/users/whit537/gists{/gist_id}","starred_url":"https://api.github.com/users/whit537/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whit537/subscriptions","organizations_url":"https://api.github.com/users/whit537/orgs","repos_url":"https://api.github.com/users/whit537/repos","events_url":"https://api.github.com/users/whit537/events{/privacy}","received_events_url":"https://api.github.com/users/whit537/received_events","type":"User","site_admin":false},"created_at":"2016-01-05T16:35:09Z","updated_at":"2016-01-05T16:35:09Z","body":"Blam. :-)\r\n\r\n![screen shot 2016-01-05 at 11 34 56 am](https://cloud.githubusercontent.com/assets/134455/12120797/5ecd95f8-b3a0-11e5-94ce-4be9f2b94396.png)\r\n"}}