{"action":"created","issue":{"url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/issues/10873","repository_url":"https://api.github.com/repos/zephyrproject-rtos/zephyr","labels_url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/issues/10873/labels{/name}","comments_url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/issues/10873/comments","events_url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/issues/10873/events","html_url":"https://github.com/zephyrproject-rtos/zephyr/pull/10873","id":374347551,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI2MDY5Mzky","number":10873,"title":"[RFC]Template prototype with updated edts","user":{"login":"evenl","id":12154119,"node_id":"MDQ6VXNlcjEyMTU0MTE5","avatar_url":"https://avatars1.githubusercontent.com/u/12154119?v=4","gravatar_id":"","url":"https://api.github.com/users/evenl","html_url":"https://github.com/evenl","followers_url":"https://api.github.com/users/evenl/followers","following_url":"https://api.github.com/users/evenl/following{/other_user}","gists_url":"https://api.github.com/users/evenl/gists{/gist_id}","starred_url":"https://api.github.com/users/evenl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evenl/subscriptions","organizations_url":"https://api.github.com/users/evenl/orgs","repos_url":"https://api.github.com/users/evenl/repos","events_url":"https://api.github.com/users/evenl/events{/privacy}","received_events_url":"https://api.github.com/users/evenl/received_events","type":"User","site_admin":false},"labels":[{"id":807706246,"node_id":"MDU6TGFiZWw4MDc3MDYyNDY=","url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/labels/DNM","name":"DNM","color":"b60205","default":false},{"id":724796876,"node_id":"MDU6TGFiZWw3MjQ3OTY4NzY=","url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/labels/RFC","name":"RFC","color":"d8af5d","default":false},{"id":700909632,"node_id":"MDU6TGFiZWw3MDA5MDk2MzI=","url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/labels/area:%20Device%20Tree","name":"area: Device Tree","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-10-26T11:44:30Z","updated_at":"2018-10-29T13:35:03Z","closed_at":null,"author_association":"COLLABORATOR","pull_request":{"url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/pulls/10873","html_url":"https://github.com/zephyrproject-rtos/zephyr/pull/10873","diff_url":"https://github.com/zephyrproject-rtos/zephyr/pull/10873.diff","patch_url":"https://github.com/zephyrproject-rtos/zephyr/pull/10873.patch"},"body":"**Goal**\r\n\r\nTo propose the jinja2 template engine as an alternative to the COG\r\nbased solution proposed for code generation.\r\n\r\n**Rationale**\r\n\r\nThe version of COG proposed to be used in Zephyr is a highly modified\r\nversion of the original COG implementation. Which means that we can\r\nnot leverage future improvements made to COG or contribute our own\r\nmodifications back to COG.\r\n\r\nThis commit https://github.com/evenl/zephyr/commit/d8c986c9bfc2f207adc31d7e16b91e32e753ea92\r\nshows what code can be removed by using a jinja instead of a COG base solution\r\n\r\nThe Jinja2 engine however can be used as a pip package dependency\r\nwithout modifications.\r\n\r\nCOG is more or less straight forward python which means that a generator\r\nmust more or less be written for each case. Which means that even the\r\nsimplest things will need a lot of boiler code just to output something.\r\n\r\n**Example:**\r\n\r\nData in is “dev_name”\r\n\r\nCog:\r\n```\r\n/**\r\n * @code{.codegen}\r\n *\r\n *     codegen.outl(“#ifndef {}_INCLUDE_H”.format(dev_name))\r\n *     codegen.outl(“#define {}_INCLUDE_H”.format(dev_name))\r\n *\r\n * @endcode{.codegen}\r\n */\r\n```\r\n\r\nJinja2:\r\n```\r\n    #ifndef {{ dev_name }}_INCLUDE_H\r\n    #define {{ dev_name }}_INCLUDE_H\r\n```\r\n\r\n**TODO:**\r\n\r\nThe Jinja2 implementation is on par with what is currently implemented using the COG\r\nsolution. But because the COG solution give the user the ability to directly write\r\npython code directly in the template there may be a need to develop some custom filters/\r\nhelper functions for the jinja2 solution.\r\n\r\nThis code is based on this code https://github.com/galak/zephyr/tree/EDTS which is not \r\nyet in master. \r\n\r\n**Other advantages:**\r\n\r\n        * Big active community\r\n        * Standardized and documented (http://jinja.pocoo.org)\r\n        * Has been in development for a long time with multiple stable releases\r\n        * Extensive enough to be a one-stop-shop for all code generation in the Zephyr project"},"comment":{"url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/issues/comments/433911686","html_url":"https://github.com/zephyrproject-rtos/zephyr/pull/10873#issuecomment-433911686","issue_url":"https://api.github.com/repos/zephyrproject-rtos/zephyr/issues/10873","id":433911686,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzkxMTY4Ng==","user":{"login":"evenl","id":12154119,"node_id":"MDQ6VXNlcjEyMTU0MTE5","avatar_url":"https://avatars1.githubusercontent.com/u/12154119?v=4","gravatar_id":"","url":"https://api.github.com/users/evenl","html_url":"https://github.com/evenl","followers_url":"https://api.github.com/users/evenl/followers","following_url":"https://api.github.com/users/evenl/following{/other_user}","gists_url":"https://api.github.com/users/evenl/gists{/gist_id}","starred_url":"https://api.github.com/users/evenl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evenl/subscriptions","organizations_url":"https://api.github.com/users/evenl/orgs","repos_url":"https://api.github.com/users/evenl/repos","events_url":"https://api.github.com/users/evenl/events{/privacy}","received_events_url":"https://api.github.com/users/evenl/received_events","type":"User","site_admin":false},"created_at":"2018-10-29T13:35:03Z","updated_at":"2018-10-29T13:35:03Z","author_association":"COLLABORATOR","body":"Just to make sure we are on the same page, I assume a data driven code generation scheme for zephyr. Based on the fact that there is already a lot of metadata available that can be used for code generation. So my choices in this proposal are based on that assumption. \r\n\r\n> Would be nice if you could use a different name (e.g. 'codejinja'/'gen_jinja' instead of codegen). By \r\n> this the code generation could be used in parallel and my OOT work does not break.\r\n\r\nI though the word \"codegen\" was so ambiguous so I though It was represented the feature and not the implementation, but for sure, I can rename the files. I am not sure how much sense it makes to use multiple code generators though. \r\n\r\n> I evaluated Jinja when I started codegen. I like it and use it in my project.\r\n\r\nOk, I was not aware of this, I am fairly new to the Zephyr project, and there was already so many comments in the codegen related pull requests that I probably have missed many comments and discussions. \r\n\r\n> There was a general request to only have pure Python for code generation. Even complex C macros \r\n> were rejected because they are too complex for a driver developer. Ninja templates require to learn \r\n> another (simple) scripting 'language' addionally to Python.\r\n\r\nThis argument just does not make sense at all when there are multiple perfectly good engines out there for generating content. It is important always pick the right tool for the right job. But of course, Python has it's place in the zephyr project, but not for generating code directly. \r\n\r\n> A driver writer that needs data that is not already precomputed has to extend Ninja by some form of \r\n> Python function that may e.g. be called by find_undeclared_variables(). To do this some \r\n> infrastructure where the driver can add the data generation has to be added.\r\n\r\nThe thing is that by using an already feature full template engine, we can start attacking such problems directly. We do not need to spend time implementing and debugging the basics first, but if there is a need, the infrastructure that you have made for codegen can easily be made available for jinja2 templates. It's is however important to identify where the missing feature are missing, is it in the templates/template engine, in the data model or somewhere else. \r\n\r\n> Having this all in one file and getting it automatically executed looked less complex to me than \r\n> having to teach a driver writer about how to do it with Jinja.\r\n\r\nThe process of generating a jinja template and codegen template is the same. Codegen templates still have to be identified in the CMakeLists.txt files, and you still have to process the files with the codegen tool. It's just a matter of how you organize the files, I think it's cleaner to identify which files are templates and not. If you already know python, jinja2 should be a breeze, and if you don't, jinja2 should be a quicker learn than python. \r\n\r\n> Please don´t call it COG. COG is Ned´s tool. Codegen is something different.\r\n\r\nOk, sorry for that, but I see it as an extension of COG. It still works the way COG was intended, but you have added many features to it. This argument validates my argument that codegen (as implemented) will add a lot of extra development/maintenance work just to have a feature that we can get almost for free through engines like Jinja2. I am actually not arguing that it _MUST_ be jinja, but that it should be an externally developed engine that we can get maintained/developed almost for free.\r\n\r\nJinja2 comes with good documentation and a big community which should make it easier to learn and get help.\r\n\r\n> For simple placeholder replacements you can really use any template engine. The hard part is to get \r\n> the \"custom filters/helper functions\" and the template syntax in a way that is acceptable to driver \r\n> writers and other users.\r\n\r\nA template should preferably be kept simple and only contain simple logic and string replacement. This is the beauty with many template engine, they enforces a much stricter border between the view and the model. If driver A and driver B shares commonalities it will be easier to avoid re-inventing the wheel or in worst case model divergence with a template engine that forces you to update a common script or data model.\r\n\r\n> Jinja can be used as is but the driver infrastructure to use Jinja still has to be in Zephyr. This also will \r\n> most probably never be upstreamed as it is in this case not the target of Jinja.\r\n\r\nWe don't need to upstream anything to the Jinja2 project, the engine supports one template to import or extend another template (much like I do it in this pull request). But the engine it self is very also customization/extendable, either through custom filters or by making python function available in the templates.\r\n\r\nLike this:\r\n\r\nCustom filters:\r\nhttps://stackoverflow.com/questions/22401959/how-to-create-jinja2-custom-filters-with-django-jinja-app\r\n\r\nPython functions:\r\nYou could for instance use a data object called api, which you populate with python functions, this\r\nwould look something like this in jinja2 syntax\r\n\r\npython code:\r\nBoth of these can easily be done straight in jinja, so meant as an example:\r\n\r\n``` python\r\ndef print_hello_world():\r\n    return \"Hello World!\"\r\n\r\ndef get_list_of_devices(compat_list):\r\n    return data['devicetree']['compatibles'][compa_list]\r\n\r\napi['print_hello_world'] = print_hello_world\r\napi['get_list_of_devices'] = get_list_of_devices\r\n\r\nenv = Environment(loader=FileSystemLoader('/path/to/templates'))\r\ntemplate_handle = env.get_template(template_file)\r\nrendered_content = template_handle.render(api=api)\r\n```\r\n\r\nJinja2:\r\n\r\n``` c\r\n{{ api.print_hello_world() }}\r\n\r\n{{ api.get_list_of_devices(['st,stm32-i2c-v1','st,stm32-i2c-v2']) }}\r\n```\r\n\r\nor I could populate the env.globals in the python script  to avoid the 'api' namespace. \r\n\r\n\r\n``` python\r\nenv.globals['print_hello_world'] = print_hello_world\r\nenv.globals['get_list_of_devices'] = get_list_of_devices\r\n```\r\n\r\njinja2 template:\r\n\r\n``` c\r\n{{ print_hello_world() }}\r\n\r\n{{ get_list_of_devices(['st,stm32-i2c-v1','st,stm32-i2c-v2']) }}\r\n```\r\n\r\n\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"evenl"}}