{"action":"created","issue":{"url":"https://api.github.com/repos/pyvec/naucse.python.cz/issues/362","repository_url":"https://api.github.com/repos/pyvec/naucse.python.cz","labels_url":"https://api.github.com/repos/pyvec/naucse.python.cz/issues/362/labels{/name}","comments_url":"https://api.github.com/repos/pyvec/naucse.python.cz/issues/362/comments","events_url":"https://api.github.com/repos/pyvec/naucse.python.cz/issues/362/events","html_url":"https://github.com/pyvec/naucse.python.cz/pull/362","id":310584247,"number":362,"title":"WIP: Rendering courses from forks and content fragments caching","user":{"login":"mikicz","id":5583319,"avatar_url":"https://avatars3.githubusercontent.com/u/5583319?v=4","gravatar_id":"","url":"https://api.github.com/users/mikicz","html_url":"https://github.com/mikicz","followers_url":"https://api.github.com/users/mikicz/followers","following_url":"https://api.github.com/users/mikicz/following{/other_user}","gists_url":"https://api.github.com/users/mikicz/gists{/gist_id}","starred_url":"https://api.github.com/users/mikicz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikicz/subscriptions","organizations_url":"https://api.github.com/users/mikicz/orgs","repos_url":"https://api.github.com/users/mikicz/repos","events_url":"https://api.github.com/users/mikicz/events{/privacy}","received_events_url":"https://api.github.com/users/mikicz/received_events","type":"User","site_admin":false},"labels":[{"id":620011240,"url":"https://api.github.com/repos/pyvec/naucse.python.cz/labels/sprint-idea","name":"sprint-idea","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-04-02T19:34:36Z","updated_at":"2018-04-07T17:00:48Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/pyvec/naucse.python.cz/pulls/362","html_url":"https://github.com/pyvec/naucse.python.cz/pull/362","diff_url":"https://github.com/pyvec/naucse.python.cz/pull/362.diff","patch_url":"https://github.com/pyvec/naucse.python.cz/pull/362.patch"},"body":"So, I guess it's about time to submit this... I'm submitting now so you guys can get acquainted with the implementation before the sprint. (I'll come Friday afternoon.). Feel free to question anything and everything.\r\n\r\nHow courses from forks work:\r\n============================\r\n\r\nCourses and runs in the base repo are defined by ``info.yml``, the forked courses by ``link.yml``. That file can contain two values, but only ``repo`` is required (https link to a git repo), the default for ``branch`` is ``master``.\r\n\r\nThe forks are serviced by model ``CourseLink``. The model has just a selection of attributes, which are required for a list of courses/runs, for a list of ongoing runs for base-courses and for cache determination.\r\n\r\nOtherwise, contents of course pages are rendered in the fork. (So even the formats of the pages can be updated). By content I mean: things which are in the template folder ``templates/content`` or rendered lessons (``Page.render_html``). These templates are also used in rendering things from the base repository, via inclusion.\r\n\r\nThe content rendered in the fork is run through a HTML whitelist (``naucse.utils.routes:AllowedElementsParser``) and then put into templates from ``templates/link``.\r\n\r\nIn the fork, I'm using the logger which gets urls from ``url_from``, which are returned with the content and metadata and are put in a queue of absolute urls to be rendered, so the urls from the forks are frozen without parsing. Only urls leading to pages inside the course are added to the queue.\r\n\r\nThere's a new file in the root of the repository, ``config.yml``, which contains info about the current repo and branch. This file should be modified in the fork with the info about the fork, so edit links lead to the right direction. Alternatively, if the fork is not on GitHub, the edit link can be redefined completely in ``naucse.utils.forks:get_edit_page_name``, ``naucse.utils.forks:get_edit_icon`` and ``naucse.templates:edit_link``.\r\n\r\nHow caching works:\r\n==================\r\n\r\nSo the first thing that needs to be said about caching is that Arca has its own caching. If cache is set, every result is stored under the commit hash of the repo at the time of execution and the task definition. So if the same task is executed and nothing changes in the repo, the result comes straight from cache. This is useful for forks, since the tasks and the repos won't change often. (Arca cache is persistent on Travis, see bellow.)\r\n\r\nBut that's not the only caching I implemented. I also did the content fragments caching as requested in #175. I'm reusing the arca dogpile.cache region for it.\r\n\r\nSince absolute urls are generated in lessons for almost anything (subpages, solutions, static), the first change I did was generating relative urls instead. Since the url generators were already injected in the rendering, I just modified the functions to generate relative urls. The functions are generated in ``naucse.routes:relative_url_functions``.\r\n\r\nThe next thing needed was the list of urls generated in a specific fragment, so it doesn't have to be parsed to get urls to freeze. I used the logger from Frozen-Flask again, this time in a custom context manager way (``naucse.freezer:temporary_url_for_logger``).\r\n\r\nSo that's the value - a content fragment with relative urls, so it can be reused and with it the list of the relative urls that have to be frozen when used.\r\n\r\nThe key is bit more complicated. I created names which can share individual fragments. The namespace is the commit hash of the commit which last modified anything inside of the folder ``naucse``. The rest of the keys is hash of the following information:\r\n\r\n  * the lesson/subpage/solution identificators\r\n  * course variables (or ``None`` for canonical lessons)\r\n  * the commit which last modified the folder the lesson is defined in\r\n\r\nSo when the value is retrieved from cache, absolute urls are generated from the current url and the relative urls and they're added to the queue of urls to be frozen.\r\n\r\nNow only sharing the cache with forks is left to explain:\r\n\r\nThe base repository creates a key for the content fragment for the specific page from the fork on the same principles as described as above. The cache is checked if the value is present. If the value is present, the key is provided to the fork and the fork can decide if it wants to use it. If so, ``None`` is returned instead of content and the base repository uses the value from cache.\r\n\r\nIf the value is not present in cache, the key is not provided and the fork has to render the content on its own. The value the fork returns is then used to populate the cache for next time.\r\n\r\nHow are errors handled:\r\n=======================\r\n\r\nBy default, all errors in forks are silenced. This can be overridden by setting the environment variable RAISE_FORK_ERRORS to ``true``.\r\n\r\nOf course, the forks can be malfunctioning in a number of ways.\r\n\r\n * If a fork can't be pulled (repo or branch doesn't exist),\r\n   the course doesn't exist in the fork or the function that's supposed return basic info about the course\r\n   (title, description, start/end dates etc) doesn't do that, the course is ignored completely (doesn't appear in /courses/ or /runs/).\r\n * If a non-lesson page fails to render a message is rendered with a warning.\r\n   * Course page: http://naucse.poul.me/course/broken-course-page/\r\n   * Session front: http://naucse.poul.me/course/broken-course-subpages/sessions/install/\r\n   * Session back: http://naucse.poul.me/course/broken-course-subpages/sessions/install/back/\r\n   * Calendar: http://naucse.poul.me/2018/broken-run-subpages/calendar/\r\n * If a lesson page (lesson, subpage or solution) fails to render:\r\n   * And the material exists in the base repository, it's rendered instead with a warning: http://naucse.poul.me/course/broken-course-subpages/beginners/cmdline/\r\n   * Otherwise only an error is displayed: http://naucse.poul.me/course/broken-course-subpages/custom/new-lesson/\r\n   * When replacing a failing page with the base version, the footer links are generated in the fork, if that fails,\r\n     they're ignored and they're missing.\r\n * Since a static content is a separate route, it's handling is separate as well.\r\n   If a page from fork contains a static file and the file is missing:\r\n   * If the lesson is canonical (exists in the base repo) and it has a static file with exact same name, it's used instead.\r\n   * Otherwise 404 is raised.\r\n\r\nTravis build:\r\n=============\r\n\r\nThe [Docker](http://arca.readthedocs.io/en/latest/backends.html#docker) backend is used in Travis. To speed things up, the built images (with installed requirements) are pushed to a registry, so next time they can be pulled instead of building. While implementing the solution I was using https://hub.docker.com/r/mikicz/naucse/tags/, but I think the best solution would be to create a new account for naucse.\r\n\r\nUnfortunately, the build takes some extra time compared to the current version once forks are introduced, this is mainly because of the extra boot time (sudo is required for docker) and pulling/building images.\r\n\r\nThe first thing I've done to improve the situation is persistent arca cache. (Meaning if there are no new commits in a fork, the whole thing can be retrieved from cache.)\r\n\r\nIf a really bad combination comes about (e.g. clean cache and new requirements), the freeze itself can be killed by Travis. (It kills builds that don't print anything in 10 minutes (and freezing doesn't)).\r\n\r\nSo the second thing is a new cli command which lists all courses and runs with info about them (slug and title, dates for runs, repo/branch for forked). Even courses which can't be pulled or don't return basic info are printed (only slug, repo/branch and a warning), so it serves like a nice overview what's actually being rendered. But the practical effect is that all docker images are pulled or built during this command, which also prints stuff regularly so it won't be killed and the freezing itself doesn't take that much time.\r\n\r\nFinally, travis prints out debug information about what failed (the exact url is always mentioned so debugging is easier...). The ``sleep 1`` command is necessary so Travis can read the output before the machine is shut down.\r\n\r\nWebhooks:\r\n=========\r\n\r\nI implemented the webhook for triggering builds in a separate repository at [mikicz/naucse-hooks](https://github.com/mikicz/naucse-hooks). Currently, the hook has to be installed manually, I'll code the automatic creation later.\r\n\r\nIt's a simple Flask app which listens at `/hooks/push/` for events from GitHub. It only reacts to ``ping`` (to verify the hook was installed correctly) and to ``push``. Before a build is triggered, the app checks that the pushed repository and branch is actually used in Naucse (all the ``link.yml`` files are checked, Naucse is pulled using Arca).\r\n\r\nFor my own [naucse](http://naucse.poul.me) I've deployed the hook at [naucse_hooks.poul.me](http://naucse_hooks.poul.me/). I'm willing to keep hosting the hooks on my VPS (vpsfree.cz), however, I'd rather not use the ``poul.me`` domain. (Maybe a new domain, something like ``naucsepython.cz``?) If you don't like me hosting the app I'll figure out something else (like PythonAnywhere or some reasonable serverless?).\r\n\r\nTesting:\r\n========\r\n\r\nNaucse uses the [Current Environment](http://arca.readthedocs.io/en/latest/backends.html#current-environment) backend when launched locally (as mentioned in #175). Caching is enabled. So pages from forks are cached and so are content fragments in lessons from the base repository. The cache for content fragments in the base repository is disabled automatically if your local version is modified (so changes you're making are always visible)\r\n\r\nIn addition to testing stuff manually I've written a couple of tests. They make a local fork from the current state of the repo (even with changes), however, they only test the basic stuff, usually just the structure of the response and couple of values. Furthermore, they test how naucse behaves if there's an error in a fork  - that the page returns 200 with a warning.\r\n\r\n\r\nWhat should to be finalized:\r\n============================\r\n\r\n * [ ] Release Arca on PyPI so it's not installed from my GitHub\r\n * [ ] Where to push docker images (see above)\r\n * [ ] Deploy production hook somewhere (see above)\r\n\r\nWhat's not in this PR:\r\n======================\r\n\r\n*I will submit these later in further pull requests.*\r\n\r\n * Automatic webhook installation (to ``mikicz/naucse-hooks``)\r\n * The meta course with a description of the process (to this repository)\r\n"},"comment":{"url":"https://api.github.com/repos/pyvec/naucse.python.cz/issues/comments/379484019","html_url":"https://github.com/pyvec/naucse.python.cz/pull/362#issuecomment-379484019","issue_url":"https://api.github.com/repos/pyvec/naucse.python.cz/issues/362","id":379484019,"user":{"login":"encukou","id":302922,"avatar_url":"https://avatars1.githubusercontent.com/u/302922?v=4","gravatar_id":"","url":"https://api.github.com/users/encukou","html_url":"https://github.com/encukou","followers_url":"https://api.github.com/users/encukou/followers","following_url":"https://api.github.com/users/encukou/following{/other_user}","gists_url":"https://api.github.com/users/encukou/gists{/gist_id}","starred_url":"https://api.github.com/users/encukou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/encukou/subscriptions","organizations_url":"https://api.github.com/users/encukou/orgs","repos_url":"https://api.github.com/users/encukou/repos","events_url":"https://api.github.com/users/encukou/events{/privacy}","received_events_url":"https://api.github.com/users/encukou/received_events","type":"User","site_admin":false},"created_at":"2018-04-07T17:00:48Z","updated_at":"2018-04-07T17:00:48Z","author_association":"OWNER","body":"- [ ] Authors of branched courses should be able to build their branch locally, without Docker/Vagrant, and safely (no externaly linked courses)"}}
{"actor":{"display_login":"encukou"}}