LSP for Salt States
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Čermák, Dan
The Language Server Protocol A language server is a piece of software that speaks a JSON RPC protocol (called the Language Server Protocol, abbreviated LSP) to provide text editors with code completion, diagnostics, documentation, etc. There are several editors and numerous language servers already implementing this protocol. The advantage of the LSP this is, that each language server works independently of the used text editor/IDE and thereby makes all implemented features available to a wider audience. Salt States SaltStack is a configuration management software like Ansible or Puppet which allows you to configure your machines via so-called salt states. Salt states are YAML documents with support for Jinja2 templates: mysql: pkg.installed: - name: mysql service.running: - name: mysql web_server: pkg.installed: {% if grains['os_family'] == 'RedHat' %} - name: {% elif grains['os_family'] == 'Debian' %} - name: apache2 {% endif %} The Salt States Language Server During this year's hackweek #20 Cédric Bosdonnat and Dan Čermák built an initial prototype of a language server for salt states. It already supports rudimentary completion, go to definition, document symbols and it can show the documentation of salt modules. This talk will give a brief overview over the current state of the language server, how we got there and which challenges and surprises we encountered along the way.
 Hello everyone, welcome to my presentation about building a language server for Sold State. Part of the OpenSuser Virtual Conference 2021. My name is Dan, I am a software developer, part of the developer engagement program at SUSE. I am essentially responsible for building tools for other developers, for example this one. This was just a hack with Project Blitzel. So besides my day job at SUSE, I am also part of the OpenSuser community, where I package maintain a few packages. I am also quite active in Fedora, where I have been recently elected into Fesco and I am also there part of the i3 special interest group, where we shipped the i3 spin for Fedora 34. I am also package maintainer Dan, contribute to a few upstream projects here and there. So but without further ado, let's take a look at today's outline. So first I'd like to cover what is actually the language server protocol, in case you have never heard about that. Then I also cover what is SoldStack, the why and when, so why did we do this, when did we do this, what were the circumstances around that, the architecture of this server. And then we'll have a brief demo, what it currently can do. And finally I'd like to showcase a few challenges that we faced and provide a brief outlook. And with that let's take a look at what is the language server protocol actually. So as you might have guessed, that's it's a protocol, actually a JSON RPC protocol and it addresses this old problem. So you have a whole ton of programming languages and you have a whole ton of different editors and they are all written in different programming languages, have different APIs if they have APIs at all. Now if you have programming language A and you want to give your users, so other developers, you want to give them code completion, documentation showing etc. Pp in five different editors, you have to write a plugin for five different editors and that's a whole lot of work and every single person implementing something like that for their programming language has to do that. And so there's a lot of duplication going on. The language server protocol tries to address this in the following way. So it defines a common protocol for all these kinds of stuff, for all these things that you want to have in an editor when working with programming languages like auto completion, diagnostics, documentation showing, code formatting for instance, jump to definition, jump to references, refactoring etc. And so the language server protocol defines how a language server which is some kind of backend program that analyzes your source code and the editor simply talks to this language server and says it hey I'm at this position in this file, what can I auto complete now or is this thing correct or what's the current symbol at this point, does it have a documentation, where is it used and so on. And so that's defined by the language server protocol. And the cool thing about this is you as a developer of a programming language, you only have to write this backend server and you can talk to all, you can provide all these nice cities to your users independently of the editor that they're using provided that it talks the language server protocol. On the other hand, if you're developing a new editor and you want your users to be able to have access to all these nice things, you just have to implement the language server protocol and you can talk to all these backend servers and you have access to a whole ecosystem. That's pretty great. So that's why we looked into it to improve the editor integration for Solstack. So what's Solstack? Solstack is a configuration management software. It's quite comparable to Ansible in the regard that it runs in agentless mode, but it also provides a server pool mode like you would know it from Puppet. It's I would say it's more closely related to Ansible since it's also written in Python. It also uses Jinger 2 and YAML for your main files. And so what's also notable about Solstack is the configuration management software behind Uuni, which is the upstream project of SUSE manager. So what you usually write in Solstack when using Solstack are so-called Solstates. And these are essentially files that describe the desired state of a system. So how does this look like? So that's a mix of Jinger 2 for templating and YAML files. And here you can see an example. So here, for instance, we describe something that should be a web server and that in this case it means we want a package to be installed, in this case Apache. And here we use Jinger templates to distinguish the different package names for different distributions. So for your Red Hat variants, the package is called Httpd and on your Debian and derivatives it's called Apache 2. Please keep in mind that in contrast to Ansible, the Jinger templating is applied before feeding it into YAML, which has the advantage that you can really leverage the full power of Jinger templates. In Ansible, you can only use them really in YAML strings, which has its upsides and its downsides. So why did we do this and when? So the why is we found editor support for configuration management to be rather lacking. So there is extensions for Solstack for various editors. Same goes for Ansible. But unfortunately, these are all tied to certain editors and they are not really too powerful. So usually they provide good completion. Excuse me. So they provide good completion. They provide use in certain places, diagnostics and stuff like that, but it's not really context where it's limited to one editor. And configuration management is really becoming increasingly complex and especially if you have really powerful tools like Solstack, Ansible, it's becoming more like writing code. But the tools that you have as someone writing this stuff, they are not on par with your powerful IDEs. And we wanted to take a look if we can really write a prototype of a language server that shows that it's possible to provide you with more. And so we, that's Cedric, Bosná and myself, we sat down during this year's Hack Week 20 in March and we developed a prototype of this. So and what did we do? So we developed a language server and we wrote it in Python. We chose Python A because we are both familiar with it and B because Solstack itself is written in Python and that means we have some easy interoperability with it. For the language server protocol itself, we leveraged the PyGLS library. So that allows us to actually to really focus on just providing the data and PyGLS then takes care of all the actual protocol talking itself. We don't have to take care of that. We then use PyYAML for the YAML parsing. Initially, we really used PyYAML's YAML parser then we briefly switched to RUAML since it provided more features then switched back again to PyYAML. And now we just use its scanner and have a custom state machine that Cedric implemented so that we can parse broken YAML, which is quite important in case the user is currently typing stuff. In the future, we'd also like to use the JINJA 2 for JINJA templating, but unfortunately we are not there yet. Really currently, tiny component is the front end. So essentially the VS code and the Emacs extension, but these are really, they don't do a whole ton. They just launch a long language server and tell the editor, hey, if you edit SaltStack files, talk to this language server and that's about it. And with that, let's take a look how this currently looks like, what it can do. So what I have in here is VS code. I have an example SaltStack file open. And so first let's take a look at what we can do with in terms of all code to completion. So one of the things that SaltStack has is these includes. You can tell Salt to include the definitions from other files in here. And what the language server will provide you is potential other files that you can include. So these are all the ones that this repository currently has. And so you can just, you can just bring a completion here and let it include whatever you like. So that's it. That's one thing. Another thing that it will complete is the submodule names where you can essentially, it will take, it will find out what's the current module that you have here before the dot and it will provide you with the completion of the correct submodules. So if you're familiar with this, you'll see it will only give you those that belong to the file module. If I, for instance, replace this with the Git submodule, it will only show you stuff of the Git submodule or for PKG and so on. Yeah. Good. So that's what's supported in terms of auto completion. Then what you can also see, I hope at least in the recording are the breadcrumbs in here. So these are also provided as these are so called document symbols in the backend. And you can use these to jump around. And essentially means that the language server is kind of, is aware of the structure of this document and jump around in that. You also saw some, some documentation is shown up here. So that could be also shown here in places. And the last thing that we, that's also supported is jump to definition. So what Solstack has is these so called requires. It means that this, if you want to, that this state requires this other one. And what you can do, you just go right mouse button, go to definition and it will jump to the correct place. It works from other places. Oh, here we have other ones. Okay. So here I require this jam jackal and I can jump in there. So that should be about it for the demo and a few challenges that we faced. So one of the, one of the challenging parts is if you are currently typing in your editor, then you don't really have valid YAML. So we need to be, we need to be able to parse broken YAML. That's also why we only use YAMLs, the pyaml scanner and have this custom state machine that Cedric implemented so that we can really do that stuff. Yeah, meaningful testing is as usual, as you might have guessed, pretty challenging, especially if you have different editors in place. And the most challenging part at this point is really the Ginger 2 interpretation. The really nasty part here is that the templating is applied before the YAML parsing, which means extracting stuff from files like this is, can be really tough because you can go really, really crazy with the Ginger templating. And so this is yet an unsolved problem. Let's take a brief look into the future. So unfortunately, this is really just a side project for us and we don't have a whole ton of time to invest into it. So unfortunately, progress has been a little bit slow. But the biggest thing that needs to be implemented in this is the, as I already said, the Ginger 2 parsing. And then there's a few other stuff and things that we could implement. For instance, show the documentation of various elements, integration with Saltlin to provide you with some linting of your document, auto completion in more places. And then as I showed you, you can jump to these requisite nodes where they are defined. It would be nice to be able to do it the other way around, so to jump to where they're referenced. Here you can find a few links. So to the source code, to the extension in the VS Code marketplace, a blog post summary of the Hack Week, the link to the slides are also on GitHub and the LSP specifications. The obligatory legal slides. And with that, I'd like to thank you for your attention. Have a nice day and bye.
 Hello everyone, welcome to my presentation about building a language server for Sold  State.  Part of the OpenSuser Virtual Conference 2021.  My name is Dan, I am a software developer, part of the developer engagement program at  SUSE.  I am essentially responsible for building tools for other developers, for example this  one.  This was just a hack with Project Blitzel.  So besides my day job at SUSE, I am also part of the OpenSuser community, where I package  maintain a few packages.  I am also quite active in Fedora, where I have been recently elected into Fesco and  I am also there part of the i3 special interest group, where we shipped the i3 spin for Fedora  34.  I am also package maintainer Dan, contribute to a few upstream projects here and there.  So but without further ado, let's take a look at today's outline.  So first I'd like to cover what is actually the language server protocol, in case you  have never heard about that.  Then I also cover what is SoldStack, the why and when, so why did we do this, when did  we do this, what were the circumstances around that, the architecture of this server.  And then we'll have a brief demo, what it currently can do.  And finally I'd like to showcase a few challenges that we faced and provide a brief outlook.  And with that let's take a look at what is the language server protocol actually.  So as you might have guessed, that's it's a protocol, actually a JSON RPC protocol and  it addresses this old problem.  So you have a whole ton of programming languages and you have a whole ton of different editors  and they are all written in different programming languages, have different APIs if they have  APIs at all.  Now if you have programming language A and you want to give your users, so other developers,  you want to give them code completion, documentation showing etc.  Pp in five different editors, you have to write a plugin for five different editors and that's  a whole lot of work and every single person implementing something like that for their  programming language has to do that.  And so there's a lot of duplication going on.  The language server protocol tries to address this in the following way.  So it defines a common protocol for all these kinds of stuff, for all these things that  you want to have in an editor when working with programming languages like auto completion,  diagnostics, documentation showing, code formatting for instance, jump to definition, jump to  references, refactoring etc.  And so the language server protocol defines how a language server which is some kind of  backend program that analyzes your source code and the editor simply talks to this language  server and says it hey I'm at this position in this file, what can I auto complete now  or is this thing correct or what's the current symbol at this point, does it have a documentation,  where is it used and so on.  And so that's defined by the language server protocol.  And the cool thing about this is you as a developer of a programming language, you only  have to write this backend server and you can talk to all, you can provide all these nice  cities to your users independently of the editor that they're using provided that it  talks the language server protocol.  On the other hand, if you're developing a new editor and you want your users to be able  to have access to all these nice things, you just have to implement the language server  protocol and you can talk to all these backend servers and you have access to a whole ecosystem.  That's pretty great.  So that's why we looked into it to improve the editor integration for Solstack.  So what's Solstack?  Solstack is a configuration management software.  It's quite comparable to Ansible in the regard that it runs in agentless mode, but it also  provides a server pool mode like you would know it from Puppet.  It's I would say it's more closely related to Ansible since it's also written in Python.  It also uses Jinger 2 and YAML for your main files.  And so what's also notable about Solstack is the configuration management software behind  Uuni, which is the upstream project of SUSE manager.  So what you usually write in Solstack when using Solstack are so-called Solstates.  And these are essentially files that describe the desired state of a system.  So how does this look like?  So that's a mix of Jinger 2 for templating and YAML files.  And here you can see an example.  So here, for instance, we describe something that should be a web server and that in this  case it means we want a package to be installed, in this case Apache.  And here we use Jinger templates to distinguish the different package names for different  distributions.  So for your Red Hat variants, the package is called Httpd and on your Debian and derivatives  it's called Apache 2.  Please keep in mind that in contrast to Ansible, the Jinger templating is applied before feeding  it into YAML, which has the advantage that you can really leverage the full power of  Jinger templates.  In Ansible, you can only use them really in YAML strings, which has its upsides and its  downsides.  So why did we do this and when?  So the why is we found editor support for configuration management to be rather lacking.  So there is extensions for Solstack for various editors.  Same goes for Ansible.  But unfortunately, these are all tied to certain editors and they are not really too powerful.  So usually they provide good completion.  Excuse me.  So they provide good completion.  They provide use in certain places, diagnostics and stuff like that, but it's not really context  where it's limited to one editor.  And configuration management is really becoming increasingly complex and especially if you  have really powerful tools like Solstack, Ansible, it's becoming more like writing code.  But the tools that you have as someone writing this stuff, they are not on par with your  powerful IDEs.  And we wanted to take a look if we can really write a prototype of a language server that  shows that it's possible to provide you with more.  And so we, that's Cedric, Bosná and myself, we sat down during this year's Hack Week 20  in March and we developed a prototype of this.  So and what did we do?  So we developed a language server and we wrote it in Python.  We chose Python A because we are both familiar with it and B because Solstack itself is written  in Python and that means we have some easy interoperability with it.  For the language server protocol itself, we leveraged the PyGLS library.  So that allows us to actually to really focus on just providing the data and PyGLS then  takes care of all the actual protocol talking itself.  We don't have to take care of that.  We then use PyYAML for the YAML parsing.  Initially, we really used PyYAML's YAML parser then we briefly switched to RUAML since it  provided more features then switched back again to PyYAML.  And now we just use its scanner and have a custom state machine that Cedric implemented  so that we can parse broken YAML, which is quite important in case the user is currently  typing stuff.  In the future, we'd also like to use the JINJA 2 for JINJA templating, but unfortunately  we are not there yet.  Really currently, tiny component is the front end.  So essentially the VS code and the Emacs extension, but these are really, they don't  do a whole ton.  They just launch a long language server and tell the editor, hey, if you edit SaltStack  files, talk to this language server and that's about it.  And with that, let's take a look how this currently looks like, what it can do.  So what I have in here is VS code.  I have an example SaltStack file open.  And so first let's take a look at what we can do with in terms of all code to completion.  So one of the things that SaltStack has is these includes.  You can tell Salt to include the definitions from other files in here.  And what the language server will provide you is potential other files that you can  include.  So these are all the ones that this repository currently has.  And so you can just, you can just bring a completion here and let it include whatever  you like.  So that's it.  That's one thing.  Another thing that it will complete is the submodule names where you can essentially,  it will take, it will find out what's the current module that you have here before the  dot and it will provide you with the completion of the correct submodules.  So if you're familiar with this, you'll see it will only give you those that belong to  the file module.  If I, for instance, replace this with the Git submodule, it will only show you stuff  of the Git submodule or for PKG and so on.  Yeah.  Good.  So that's what's supported in terms of auto completion.  Then what you can also see, I hope at least in the recording are the breadcrumbs in here.  So these are also provided as these are so called document symbols in the backend.  And you can use these to jump around.  And essentially means that the language server is kind of, is aware of the structure of this  document and jump around in that.  You also saw some, some documentation is shown up here.  So that could be also shown here in places.  And the last thing that we, that's also supported is jump to definition.  So what Solstack has is these so called requires.  It means that this, if you want to, that this state requires this other one.  And what you can do, you just go right mouse button, go to definition and it will jump  to the correct place.  It works from other places.  Oh, here we have other ones.  Okay.  So here I require this jam jackal and I can jump in there.  So that should be about it for the demo and a few challenges that we faced.  So one of the, one of the challenging parts is if you are currently typing in your editor,  then you don't really have valid YAML.  So we need to be, we need to be able to parse broken YAML.  That's also why we only use YAMLs, the pyaml scanner and have this custom state machine  that Cedric implemented so that we can really do that stuff.  Yeah, meaningful testing is as usual, as you might have guessed, pretty challenging, especially  if you have different editors in place.  And the most challenging part at this point is really the Ginger 2 interpretation.  The really nasty part here is that the templating is applied before the YAML parsing, which  means extracting stuff from files like this is, can be really tough because you can go  really, really crazy with the Ginger templating.  And so this is yet an unsolved problem.  Let's take a brief look into the future.  So unfortunately, this is really just a side project for us and we don't have a whole  ton of time to invest into it.  So unfortunately, progress has been a little bit slow.  But the biggest thing that needs to be implemented in this is the, as I already said, the Ginger  2 parsing.  And then there's a few other stuff and things that we could implement.  For instance, show the documentation of various elements, integration with Saltlin to provide  you with some linting of your document, auto completion in more places.  And then as I showed you, you can jump to these requisite nodes where they are defined.  It would be nice to be able to do it the other way around, so to jump to where they're referenced.  Here you can find a few links.  So to the source code, to the extension in the VS Code marketplace, a blog post summary  of the Hack Week, the link to the slides are also on GitHub and the LSP specifications.  The obligatory legal slides.  And with that, I'd like to thank you for your attention.  Have a nice day and bye.
Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_0_347.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_348_2555.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_2556_7115.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_7116_10283.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_10284_12695.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_12696_13163.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_13164_13667.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_13668_15143.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_15144_15227.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_15228_15347.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_15348_19079.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_19080_19163.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_19164_19559.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_19560_21215.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_21216_21287.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_21288_21623.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_21624_22631.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_22632_23015.png Building_a_Language_Server_for_Salt_States__Llz3Md4gkE_23016_23203.png