{"action":"created","comment":{"url":"https://api.github.com/repos/WeAreWizards/blog/pulls/comments/59838468","id":59838468,"diff_hunk":"@@ -0,0 +1,136 @@\n+Title: Introducing Tera, a template engine in Rust\n+Date: 2016-04-14\n+Short_summary: Talking about a template engine in Rust called Tera, inspired by Jinja2 and Django\n+Category: Dev\n+Authors: Vincent\n+\n+\n+*Reading time: ~15 minutes.*\n+\n+Back in October 2015, I tried [Rust for web services](https://blog.wearewizards.io/trying-rust-for-web-services) and found the ecosystem lacking at the time. That's why I've been working on porting some of the tools we use in [Proppy](https://proppy.io/) to Rust: [jwt](https://crates.io/crates/jsonwebtoken), [bcrypt](https://crates.io/crates/bcrypt) (granted that Argon2 seems superior) and a [migration tool](https://crates.io/crates/dbmigrate). While I mostly do SPAs these days and don't write many templates in the backend, I still need one for some occasions. Being a Python, I like [Jinja2](http://jinja.pocoo.org/docs/dev/) and [Django templates](https://docs.djangoproject.com/en/1.9/topics/templates/#the-django-template-language). Here's how I attempted to port them to Rust and the result is [Tera](https://github.com/Keats/tera/).\n+<!-- PELICAN_END_SUMMARY --> \n+\n+## Goals and philosophy\n+As mentioned before, the inspiration comes from both Jinja2 and django templates. As you might know, those two have similar syntax but different philosophies: Django templates are for presentation only and don't have a lot of logic allowed while Jinja2 is way more powerful.\n+I side with Django on this one as complex logic is better put in code than in a template but Django goes a bit too far by even not supporting something like `{{ count + 1 }}`.\n+\n+So here are some of the things I want:\n+\n+- math operations in templates\n+- no macros or other complex logic in the template\n+- beautiful html output out of the box (ie no need for the `{{-` tags)\n+- simple inheritance\n+- simple to use filters\n+- able to register new tags easily like the `{% url ... %}` in Django\n+\n+While new tags are definitely logic added in the template, that logic would have to be written in Rust and not in a template. That limits reusability but is simpler to understand in the end.\n+\n+Filters should be kept simple and be limited in scope: variable in, modifier function with optional argument and return a string. The easiest to think of would be uppercase, lowercase, capitalize and more importantly time formatting. Here are some examples of how it should look:\n+```\n+{{ name | uppercase }}\n+{{ birthday | time:\"YYYY-MM-dd\" }}\n+```\n+Users should be able to add their own filters as well.\n+\n+In terms of error handling, we cannot do anything if a template cannot be parsed so panicking when encountering an error is fine and compilation should be done at compile time to ensure everything works perfectly (more on that on the README.md).\n+On the API side, using Tera should be trivial:\n+\n+- give a glob that will select all the templates\n+- add tag/filter to Tera\n+- parse all templates\n+- create a context easily (not as simple as a dict obviously though)\n+- call a render method that returns a `Result`\n+\n+Template compilation should only happen once by using something like [lazy_static](https://crates.io/crates/lazy_static).\n+\n+Let's see how it's built now!\n+\n+## How it's made\n+I actually thought of making a template engine after watching the \"Lexical Scanning in Go\" video by Rob Pike ([youtube link](https://www.youtube.com/watch?v=HxaD_trXwRE)). \n+\n+### Lexer/Parser\n+This talk explains how the lexing in the [template package in the Go standard library](https://golang.org/pkg/text/template/) is implemented. I thought it was pretty cool and implemented something similar last summer.\n+For those that don't want to watch the video, here's a quick summary.\n+\n+The lexer can be in a few states: inside a block, text, space, number, identifier, string for Tera currently and there are actions that represents what we do and result in a new state. In short, that means we have a state function that takes the lexer as an argument and returns a state function. While this is easy to do in Go, you cannot do reference the type while declaring it but it's ok to do so for a struct.\n+```rust\n+// working\n+struct StateFn(Option<fn(&mut Lexer) -> StateFn>);\n+// not working\n+type StateFn = fn(&mut Lexer) -> Option<StateFn>;\n+```\n+Thanks for the help on IRC for that one.\n+The lexer just runs the state function until we reach EOF which in our case is represented by returning `None` as a state function. Here's the whole run method of the lexer:\n+\n+```rust\n+pub fn run(&mut self) {\n+    loop {\n+        let StateFn(state_fn) = self.state;\n+        if state_fn.is_none() {\n+            break;\n+        }\n+        self.state = state_fn.unwrap()(self);\n+    }\n+}\n+```\n+The actions read the next character and know what to do for each kind of character, eg. finding a number in a variable block will return the `lex_number` state function. The lexer ouput is a vector of tokens that ends with either a EOF or and error.\n+\n+You can read the whole lexer [on github](https://github.com/Keats/tera/blob/fddb8a0b82cba7374bd0552fed1cf831b8943395/src/lexer.rs), it is actually quite simple and readable.\n+\n+\n+The parser is quite simple as well. Since we are either in text, in a variable block or in a tag block, we just handle those cases in a loop and use EOF to break. Each \"state\" knows how to parse itself so the main logic is actually only:\n+\n+```rust\n+pub fn parse(&mut self) {\n+    loop {\n+        match self.peek().kind {\n+            TokenType::TagStart => self.parse_tag_block(),\n+            TokenType::VariableStart => self.parse_variable_block(),\n+            TokenType::Text => self.parse_text(),\n+            TokenType::Eof => break,\n+            _ => unreachable!()\n+        };\n+    }\n+}\n+```\n+\n+The trickiest bit was handling precedence in blocks so that something like `{{1 / 2 + 3 * 2 + 42}}` would parse as expected. This is done by assigning precedence values to each kind of token and looking forward to see if something with higer precedence is coming. I would be surprised if there was not a bug in there.","path":"content/articles/tera-template-engine.md","position":97,"original_position":97,"commit_id":"b7da1f460f9cfb06584bee4ad858802e524d8bb8","original_commit_id":"b7da1f460f9cfb06584bee4ad858802e524d8bb8","user":{"login":"teh","id":139251,"avatar_url":"https://avatars.githubusercontent.com/u/139251?v=3","gravatar_id":"","url":"https://api.github.com/users/teh","html_url":"https://github.com/teh","followers_url":"https://api.github.com/users/teh/followers","following_url":"https://api.github.com/users/teh/following{/other_user}","gists_url":"https://api.github.com/users/teh/gists{/gist_id}","starred_url":"https://api.github.com/users/teh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teh/subscriptions","organizations_url":"https://api.github.com/users/teh/orgs","repos_url":"https://api.github.com/users/teh/repos","events_url":"https://api.github.com/users/teh/events{/privacy}","received_events_url":"https://api.github.com/users/teh/received_events","type":"User","site_admin":false},"body":"I would not be surprised","created_at":"2016-04-15T07:54:36Z","updated_at":"2016-04-15T07:54:36Z","html_url":"https://github.com/WeAreWizards/blog/pull/38#discussion_r59838468","pull_request_url":"https://api.github.com/repos/WeAreWizards/blog/pulls/38","_links":{"self":{"href":"https://api.github.com/repos/WeAreWizards/blog/pulls/comments/59838468"},"html":{"href":"https://github.com/WeAreWizards/blog/pull/38#discussion_r59838468"},"pull_request":{"href":"https://api.github.com/repos/WeAreWizards/blog/pulls/38"}}},"pull_request":{"url":"https://api.github.com/repos/WeAreWizards/blog/pulls/38","id":66391505,"html_url":"https://github.com/WeAreWizards/blog/pull/38","diff_url":"https://github.com/WeAreWizards/blog/pull/38.diff","patch_url":"https://github.com/WeAreWizards/blog/pull/38.patch","issue_url":"https://api.github.com/repos/WeAreWizards/blog/issues/38","number":38,"state":"open","locked":false,"title":"Add tera article","user":{"login":"Keats","id":680355,"avatar_url":"https://avatars.githubusercontent.com/u/680355?v=3","gravatar_id":"","url":"https://api.github.com/users/Keats","html_url":"https://github.com/Keats","followers_url":"https://api.github.com/users/Keats/followers","following_url":"https://api.github.com/users/Keats/following{/other_user}","gists_url":"https://api.github.com/users/Keats/gists{/gist_id}","starred_url":"https://api.github.com/users/Keats/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Keats/subscriptions","organizations_url":"https://api.github.com/users/Keats/orgs","repos_url":"https://api.github.com/users/Keats/repos","events_url":"https://api.github.com/users/Keats/events{/privacy}","received_events_url":"https://api.github.com/users/Keats/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-04-13T21:05:59Z","updated_at":"2016-04-15T07:54:36Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0bf8da98e50d57eb7abc4f0359a60f23c5f59167","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/WeAreWizards/blog/pulls/38/commits","review_comments_url":"https://api.github.com/repos/WeAreWizards/blog/pulls/38/comments","review_comment_url":"https://api.github.com/repos/WeAreWizards/blog/pulls/comments{/number}","comments_url":"https://api.github.com/repos/WeAreWizards/blog/issues/38/comments","statuses_url":"https://api.github.com/repos/WeAreWizards/blog/statuses/b7da1f460f9cfb06584bee4ad858802e524d8bb8","head":{"label":"WeAreWizards:tera","ref":"tera","sha":"b7da1f460f9cfb06584bee4ad858802e524d8bb8","user":{"login":"WeAreWizards","id":8532150,"avatar_url":"https://avatars.githubusercontent.com/u/8532150?v=3","gravatar_id":"","url":"https://api.github.com/users/WeAreWizards","html_url":"https://github.com/WeAreWizards","followers_url":"https://api.github.com/users/WeAreWizards/followers","following_url":"https://api.github.com/users/WeAreWizards/following{/other_user}","gists_url":"https://api.github.com/users/WeAreWizards/gists{/gist_id}","starred_url":"https://api.github.com/users/WeAreWizards/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/WeAreWizards/subscriptions","organizations_url":"https://api.github.com/users/WeAreWizards/orgs","repos_url":"https://api.github.com/users/WeAreWizards/repos","events_url":"https://api.github.com/users/WeAreWizards/events{/privacy}","received_events_url":"https://api.github.com/users/WeAreWizards/received_events","type":"Organization","site_admin":false},"repo":{"id":24109771,"name":"blog","full_name":"WeAreWizards/blog","owner":{"login":"WeAreWizards","id":8532150,"avatar_url":"https://avatars.githubusercontent.com/u/8532150?v=3","gravatar_id":"","url":"https://api.github.com/users/WeAreWizards","html_url":"https://github.com/WeAreWizards","followers_url":"https://api.github.com/users/WeAreWizards/followers","following_url":"https://api.github.com/users/WeAreWizards/following{/other_user}","gists_url":"https://api.github.com/users/WeAreWizards/gists{/gist_id}","starred_url":"https://api.github.com/users/WeAreWizards/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/WeAreWizards/subscriptions","organizations_url":"https://api.github.com/users/WeAreWizards/orgs","repos_url":"https://api.github.com/users/WeAreWizards/repos","events_url":"https://api.github.com/users/WeAreWizards/events{/privacy}","received_events_url":"https://api.github.com/users/WeAreWizards/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/WeAreWizards/blog","description":"We Are Wizards Blog","fork":false,"url":"https://api.github.com/repos/WeAreWizards/blog","forks_url":"https://api.github.com/repos/WeAreWizards/blog/forks","keys_url":"https://api.github.com/repos/WeAreWizards/blog/keys{/key_id}","collaborators_url":"https://api.github.com/repos/WeAreWizards/blog/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/WeAreWizards/blog/teams","hooks_url":"https://api.github.com/repos/WeAreWizards/blog/hooks","issue_events_url":"https://api.github.com/repos/WeAreWizards/blog/issues/events{/number}","events_url":"https://api.github.com/repos/WeAreWizards/blog/events","assignees_url":"https://api.github.com/repos/WeAreWizards/blog/assignees{/user}","branches_url":"https://api.github.com/repos/WeAreWizards/blog/branches{/branch}","tags_url":"https://api.github.com/repos/WeAreWizards/blog/tags","blobs_url":"https://api.github.com/repos/WeAreWizards/blog/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/WeAreWizards/blog/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/WeAreWizards/blog/git/refs{/sha}","trees_url":"https://api.github.com/repos/WeAreWizards/blog/git/trees{/sha}","statuses_url":"https://api.github.com/repos/WeAreWizards/blog/statuses/{sha}","languages_url":"https://api.github.com/repos/WeAreWizards/blog/languages","stargazers_url":"https://api.github.com/repos/WeAreWizards/blog/stargazers","contributors_url":"https://api.github.com/repos/WeAreWizards/blog/contributors","subscribers_url":"https://api.github.com/repos/WeAreWizards/blog/subscribers","subscription_url":"https://api.github.com/repos/WeAreWizards/blog/subscription","commits_url":"https://api.github.com/repos/WeAreWizards/blog/commits{/sha}","git_commits_url":"https://api.github.com/repos/WeAreWizards/blog/git/commits{/sha}","comments_url":"https://api.github.com/repos/WeAreWizards/blog/comments{/number}","issue_comment_url":"https://api.github.com/repos/WeAreWizards/blog/issues/comments{/number}","contents_url":"https://api.github.com/repos/WeAreWizards/blog/contents/{+path}","compare_url":"https://api.github.com/repos/WeAreWizards/blog/compare/{base}...{head}","merges_url":"https://api.github.com/repos/WeAreWizards/blog/merges","archive_url":"https://api.github.com/repos/WeAreWizards/blog/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/WeAreWizards/blog/downloads","issues_url":"https://api.github.com/repos/WeAreWizards/blog/issues{/number}","pulls_url":"https://api.github.com/repos/WeAreWizards/blog/pulls{/number}","milestones_url":"https://api.github.com/repos/WeAreWizards/blog/milestones{/number}","notifications_url":"https://api.github.com/repos/WeAreWizards/blog/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/WeAreWizards/blog/labels{/name}","releases_url":"https://api.github.com/repos/WeAreWizards/blog/releases{/id}","deployments_url":"https://api.github.com/repos/WeAreWizards/blog/deployments","created_at":"2014-09-16T17:23:36Z","updated_at":"2016-03-25T16:20:20Z","pushed_at":"2016-04-14T13:42:52Z","git_url":"git://github.com/WeAreWizards/blog.git","ssh_url":"git@github.com:WeAreWizards/blog.git","clone_url":"https://github.com/WeAreWizards/blog.git","svn_url":"https://github.com/WeAreWizards/blog","homepage":"https://blog.wearewizards.io/","size":1710,"stargazers_count":1,"watchers_count":1,"language":"Jupyter Notebook","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":1,"forks":2,"open_issues":1,"watchers":1,"default_branch":"master"}},"base":{"label":"WeAreWizards:master","ref":"master","sha":"c30b14d36ff671e15c3398306f26e77707964e09","user":{"login":"WeAreWizards","id":8532150,"avatar_url":"https://avatars.githubusercontent.com/u/8532150?v=3","gravatar_id":"","url":"https://api.github.com/users/WeAreWizards","html_url":"https://github.com/WeAreWizards","followers_url":"https://api.github.com/users/WeAreWizards/followers","following_url":"https://api.github.com/users/WeAreWizards/following{/other_user}","gists_url":"https://api.github.com/users/WeAreWizards/gists{/gist_id}","starred_url":"https://api.github.com/users/WeAreWizards/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/WeAreWizards/subscriptions","organizations_url":"https://api.github.com/users/WeAreWizards/orgs","repos_url":"https://api.github.com/users/WeAreWizards/repos","events_url":"https://api.github.com/users/WeAreWizards/events{/privacy}","received_events_url":"https://api.github.com/users/WeAreWizards/received_events","type":"Organization","site_admin":false},"repo":{"id":24109771,"name":"blog","full_name":"WeAreWizards/blog","owner":{"login":"WeAreWizards","id":8532150,"avatar_url":"https://avatars.githubusercontent.com/u/8532150?v=3","gravatar_id":"","url":"https://api.github.com/users/WeAreWizards","html_url":"https://github.com/WeAreWizards","followers_url":"https://api.github.com/users/WeAreWizards/followers","following_url":"https://api.github.com/users/WeAreWizards/following{/other_user}","gists_url":"https://api.github.com/users/WeAreWizards/gists{/gist_id}","starred_url":"https://api.github.com/users/WeAreWizards/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/WeAreWizards/subscriptions","organizations_url":"https://api.github.com/users/WeAreWizards/orgs","repos_url":"https://api.github.com/users/WeAreWizards/repos","events_url":"https://api.github.com/users/WeAreWizards/events{/privacy}","received_events_url":"https://api.github.com/users/WeAreWizards/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/WeAreWizards/blog","description":"We Are Wizards Blog","fork":false,"url":"https://api.github.com/repos/WeAreWizards/blog","forks_url":"https://api.github.com/repos/WeAreWizards/blog/forks","keys_url":"https://api.github.com/repos/WeAreWizards/blog/keys{/key_id}","collaborators_url":"https://api.github.com/repos/WeAreWizards/blog/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/WeAreWizards/blog/teams","hooks_url":"https://api.github.com/repos/WeAreWizards/blog/hooks","issue_events_url":"https://api.github.com/repos/WeAreWizards/blog/issues/events{/number}","events_url":"https://api.github.com/repos/WeAreWizards/blog/events","assignees_url":"https://api.github.com/repos/WeAreWizards/blog/assignees{/user}","branches_url":"https://api.github.com/repos/WeAreWizards/blog/branches{/branch}","tags_url":"https://api.github.com/repos/WeAreWizards/blog/tags","blobs_url":"https://api.github.com/repos/WeAreWizards/blog/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/WeAreWizards/blog/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/WeAreWizards/blog/git/refs{/sha}","trees_url":"https://api.github.com/repos/WeAreWizards/blog/git/trees{/sha}","statuses_url":"https://api.github.com/repos/WeAreWizards/blog/statuses/{sha}","languages_url":"https://api.github.com/repos/WeAreWizards/blog/languages","stargazers_url":"https://api.github.com/repos/WeAreWizards/blog/stargazers","contributors_url":"https://api.github.com/repos/WeAreWizards/blog/contributors","subscribers_url":"https://api.github.com/repos/WeAreWizards/blog/subscribers","subscription_url":"https://api.github.com/repos/WeAreWizards/blog/subscription","commits_url":"https://api.github.com/repos/WeAreWizards/blog/commits{/sha}","git_commits_url":"https://api.github.com/repos/WeAreWizards/blog/git/commits{/sha}","comments_url":"https://api.github.com/repos/WeAreWizards/blog/comments{/number}","issue_comment_url":"https://api.github.com/repos/WeAreWizards/blog/issues/comments{/number}","contents_url":"https://api.github.com/repos/WeAreWizards/blog/contents/{+path}","compare_url":"https://api.github.com/repos/WeAreWizards/blog/compare/{base}...{head}","merges_url":"https://api.github.com/repos/WeAreWizards/blog/merges","archive_url":"https://api.github.com/repos/WeAreWizards/blog/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/WeAreWizards/blog/downloads","issues_url":"https://api.github.com/repos/WeAreWizards/blog/issues{/number}","pulls_url":"https://api.github.com/repos/WeAreWizards/blog/pulls{/number}","milestones_url":"https://api.github.com/repos/WeAreWizards/blog/milestones{/number}","notifications_url":"https://api.github.com/repos/WeAreWizards/blog/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/WeAreWizards/blog/labels{/name}","releases_url":"https://api.github.com/repos/WeAreWizards/blog/releases{/id}","deployments_url":"https://api.github.com/repos/WeAreWizards/blog/deployments","created_at":"2014-09-16T17:23:36Z","updated_at":"2016-03-25T16:20:20Z","pushed_at":"2016-04-14T13:42:52Z","git_url":"git://github.com/WeAreWizards/blog.git","ssh_url":"git@github.com:WeAreWizards/blog.git","clone_url":"https://github.com/WeAreWizards/blog.git","svn_url":"https://github.com/WeAreWizards/blog","homepage":"https://blog.wearewizards.io/","size":1710,"stargazers_count":1,"watchers_count":1,"language":"Jupyter Notebook","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":1,"forks":2,"open_issues":1,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/WeAreWizards/blog/pulls/38"},"html":{"href":"https://github.com/WeAreWizards/blog/pull/38"},"issue":{"href":"https://api.github.com/repos/WeAreWizards/blog/issues/38"},"comments":{"href":"https://api.github.com/repos/WeAreWizards/blog/issues/38/comments"},"review_comments":{"href":"https://api.github.com/repos/WeAreWizards/blog/pulls/38/comments"},"review_comment":{"href":"https://api.github.com/repos/WeAreWizards/blog/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/WeAreWizards/blog/pulls/38/commits"},"statuses":{"href":"https://api.github.com/repos/WeAreWizards/blog/statuses/b7da1f460f9cfb06584bee4ad858802e524d8bb8"}}}}