I'm Luvin' It
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Muraus, Tomaz
Short introduction to Luvit and virgo project. Luvit is an open-source project which uses libuv and exposes Node.js API on top of Lua(jit). Virgo is an open-source framework for building different host agents (e.g. monitoring / metrics collection agent, etc.) built on top of Luvit.
 Okay, so good morning everyone. I hope everyone is doing okay even though it's still pretty early. My name is Tomasz Muroz and today I'm gonna present the Love It and Virgo project. I know you're probably thinking what the hell is that because both of the projects have kind of a cryptic name but I'll dive in soon and you'll soon find out. So here's the agenda. And who am I? I'm Tomasz Muroz. I work at RecSpace. I used to work on cloud monitoring which is a REST-based fully distributed monitoring system and currently I work on service registry which is a REST-based API for registering your services and your processes. I'm also a project chair of a battery deep cloud project which is a project which deals with working with different cloud APIs and also like open source and open standards. So what is Love It? Love It is a framework or platform for building different network applications written in Lua. Love It exposes Node.js API but on top of Lua. I don't know how familiar you are with Node.js but Node.js is basically a server-side JavaScript. Basically you can write your networking applications in JavaScript and Love It aims to do the same thing but instead of writing your apps in JavaScript you can do it in Lua. So Love It stands, there basically, Love It name is composed of three words. First one is, first letter is L which stands for Lua which is the language you write your apps in. Second abbreviation is UV which stands for the LibuV. LibuV is the actual C library which deals with the cross-platform and the event loop stuff which is also used by Node.js. That's one of the biggest users and also used by Love It and the third part stands for LuaDriot which is a just-in-time compiler for Lua. So Love It is an open source project licensed under Apache 2.0. So here's a quick example to get you excited. If you're familiar with Node you can probably already see what the example does but basically it starts an HTTP server and when you hit it you get a back response which says welcome to FuzzDem. So the code in JavaScript would actually look pretty similar. There's some syntactic differences because Lua doesn't have an object but it has tables and meta tables and there's some minor differences there but otherwise the API is pretty similar. So some history, Love It project was originally developed by Tim Caswell. Tim Caswell is a pretty cool guy. You should probably go check out his GitHub page and follow him there because he has a bunch of cool and crazy projects. I think at the moment he's working in his own language so basically he has a bunch of cool stuff but Love It is a pretty young project. First version was released in 2011 and soon after the first version was released my team at WrexPACE decided to help with the project because basically we decided to use it for our monitoring agent which I'll describe more later. So yeah, like the slide says we contributed a lot to the project and we're still contributing so if nothing else there are also a bunch of other independent contributors but there are also a bunch of a couple of paid people working on it. So yeah, what should you care? First thing I guess is Lua is cool. I personally like Lua more than JavaScript although it does have some bad parts which are similar to JavaScript. For example, JavaScript doesn't have a 64-bit integers. It has this messy number type which makes a lot of things difficult and there's pretty much the same problem in Lua but in Lua you have other things such as coroutines and other stuff. So Lua was primarily developed by some academics from Brazilian university and it's used a lot as a secondary slash scripting language in different apps and games and so on. I guess a couple of popular uses are World of Warcraft game where I think the graphical interface and AI is built in Lua and FSMVM which is a tiling window manager. And one of the strong points I guess or one of the many reasons Lua exists is because it's easy to embed it. It provides a simple C-based stack API which means you push a value on a stack and you can pop it off and that's basically how you integrate with other languages. So here's an example of the C-based stack API. As you can see you push an integer on a stack and you push integers and then you call the function. You previously saved the reference to and you get a result back. So it's basically pretty easy to embed it. So what is Virgo? Virgo is a monitoring agent we built at RecSpace for running on many different servers. So when we decided to build the agent we had some requirements in mind and one of the requirements was I guess decent performance and low memory footprint because this agent is going to run on your server pretty much all the time and you probably don't want this agent to affect other services like web server or whatever running on your servers so that's why we decided to go with it. So yeah basically Virgo is not just a RecSpace Cloud Monitoring agent but is a platform for building other host agents. Host agents is basically pretty much any kind of long running process which performs some kind of action on host operating system. In our case this is collecting metrics and sending them back to their endpoint but you could also use it for example to execute commands on your system and so on. So yeah why did we pick Lovett? One of the goals was previously mentioned was low memory footprint and we basically wanted to speed up the development cycle. If the agent would be written in C this would probably probably get even lower memory usage but this means we need to deal with the manual garbage collection and so on and it will also the whole development cycle would be pretty slow so we basically have core which bootstraps the process which is written in C and the core is pretty much responsible for handing the option parsing and logging but the rest of the stuff is outsourced to the Lovett layer so pretty much all of the business logic is written in Lua which allows us to also besides the faster development cycle it also allows us to basically if you want to ship the update we can just ship the updated Lua code and we don't need to restart the whole process we can just reload the Lua code and without restarting the whole process we can reload the business logic. That's stuff I previously already mentioned and one of the things I also wanted to add is we also wanted to make it really secure because there are a bunch of other monitoring agents written in Ruby, Python and so on and some of those agents don't even use HTTPS or a secure connection so our goal was to only use a secure connection on all the bundles. Bundle is basically a tar package which contains a Lua code is signed and also the binaries are signed the actual signing depends on the platform we use the Davian specific stuff and Davian systems and on Windows we also use the Microsoft thing for signing the executable so we pretty much made it really secure. Another thing we wanted to do when we build agent is keep the protocol simple so we decided to go with the new line the limited JSON. Here's an example of the outgoing message in this case we send a heartbeat which is kind of like a ping so we have this semi-structured JSON some of the fields like ID and version and so on are required and other stuff like prams are optional and depend on the actual method and here's the response from the server for example. Here's a graph which shows a memory usage of agents written in different programming languages there's agent written C++ Ruby and so on and there's our Virgo which only uses around like five megabytes of memory we could actually go even lower but because of we also wanted to make the whole agent really cross-platform and portable so we decided to statically link all the libraries which does increase memory usage a bit especially because we link against LibOpenSSL which is known to take a lot of memory and here's a quick diagram of the our endpoint architecture basically agent connects to multiple endpoints that's for high availability reasons so if one of our data centers goes down for example your agent can still send metrics to us and we can process them and send an alert for example basically agent establishes three persistent long-running connections but it actually only sends data which is metrics and other system information over one connection and if this connection dies other connection other secondary connection is promoted to a primary connection so I guess that's it I hope you I got you interested and if you have a similar problem for example if you want to embed something or you need a low memory footprint and fast execution you should probably check love it and Virgo and also want to add that yeah if you're interested in this kind of problems my company rex space is looking for new people to join us so let me know go to this website or send me an email or find me around the around this room after the talk I guess thank you oh yeah and also have some Apache lipcloth Rex with God monitoring and Rex based service registry stickers I'll leave them on the desk and you can go grab it.
 Okay, so good morning everyone. I hope everyone is doing okay even though it's still pretty  early. My name is Tomasz Muroz and today I'm gonna present the Love It and Virgo project.  I know you're probably thinking what the hell is that because both of the projects have  kind of a cryptic name but I'll dive in soon and you'll soon find out. So here's the agenda.  And who am I? I'm Tomasz Muroz. I work at RecSpace. I used to work on cloud monitoring which  is a REST-based fully distributed monitoring system and currently I work on service registry  which is a REST-based API for registering your services and your processes. I'm also  a project chair of a battery deep cloud project which is a project which deals with working  with different cloud APIs and also like open source and open standards. So what is Love  It? Love It is a framework or platform for building different network applications written  in Lua. Love It exposes Node.js API but on top of Lua. I don't know how familiar you are  with Node.js but Node.js is basically a server-side JavaScript. Basically you can write your networking  applications in JavaScript and Love It aims to do the same thing but instead of writing  your apps in JavaScript you can do it in Lua. So Love It stands, there basically, Love It  name is composed of three words. First one is, first letter is L which stands for Lua  which is the language you write your apps in. Second abbreviation is UV which stands for  the LibuV. LibuV is the actual C library which deals with the cross-platform and the event  loop stuff which is also used by Node.js. That's one of the biggest users and also used by  Love It and the third part stands for LuaDriot which is a just-in-time compiler for Lua. So  Love It is an open source project licensed under Apache 2.0. So here's a quick example  to get you excited. If you're familiar with Node you can probably already see what the  example does but basically it starts an HTTP server and when you hit it you get a back response  which says welcome to FuzzDem. So the code in JavaScript would actually look pretty similar.  There's some syntactic differences because Lua doesn't have an object but it has tables  and meta tables and there's some minor differences there but otherwise the API is pretty similar.  So some history, Love It project was originally developed by Tim Caswell. Tim Caswell is a  pretty cool guy. You should probably go check out his GitHub page and follow him there because  he has a bunch of cool and crazy projects. I think at the moment he's working in his  own language so basically he has a bunch of cool stuff but Love It is a pretty young project.  First version was released in 2011 and soon after the first version was released my team  at WrexPACE decided to help with the project because basically we decided to use it for  our monitoring agent which I'll describe more later. So yeah, like the slide says we contributed  a lot to the project and we're still contributing so if nothing else there are also a bunch  of other independent contributors but there are also a bunch of a couple of paid people  working on it. So yeah, what should you care? First thing I guess is Lua is cool. I personally  like Lua more than JavaScript although it does have some bad parts which are similar  to JavaScript. For example, JavaScript doesn't have a 64-bit integers. It has this messy  number type which makes a lot of things difficult and there's pretty much the same problem in  Lua but in Lua you have other things such as coroutines and other stuff. So Lua was  primarily developed by some academics from Brazilian university and it's used a lot as  a secondary slash scripting language in different apps and games and so on. I guess a couple  of popular uses are World of Warcraft game where I think the graphical interface and  AI is built in Lua and FSMVM which is a tiling window manager. And one of the strong points  I guess or one of the many reasons Lua exists is because it's easy to embed it. It provides  a simple C-based stack API which means you push a value on a stack and you can pop it  off and that's basically how you integrate with other languages. So here's an example  of the C-based stack API. As you can see you push an integer on a stack and you push integers  and then you call the function. You previously saved the reference to and you get a result  back. So it's basically pretty easy to embed it. So what is Virgo? Virgo is a monitoring  agent we built at RecSpace for running on many different servers. So when we decided  to build the agent we had some requirements in mind and one of the requirements was I  guess decent performance and low memory footprint because this agent is going to run on your  server pretty much all the time and you probably don't want this agent to affect other services  like web server or whatever running on your servers so that's why we decided to go with  it. So yeah basically Virgo is not just a RecSpace Cloud Monitoring agent but is a platform  for building other host agents. Host agents is basically pretty much any kind of long  running process which performs some kind of action on host operating system. In our case  this is collecting metrics and sending them back to their endpoint but you could also  use it for example to execute commands on your system and so on. So yeah why did we  pick Lovett? One of the goals was previously mentioned was low memory footprint and we  basically wanted to speed up the development cycle. If the agent would be written in C  this would probably probably get even lower memory usage but this means we need to deal  with the manual garbage collection and so on and it will also the whole development  cycle would be pretty slow so we basically have core which bootstraps the process which  is written in C and the core is pretty much responsible for handing the option parsing  and logging but the rest of the stuff is outsourced to the Lovett layer so pretty much all of  the business logic is written in Lua which allows us to also besides the faster development  cycle it also allows us to basically if you want to ship the update we can just ship the  updated Lua code and we don't need to restart the whole process we can just reload the Lua  code and without restarting the whole process we can reload the business logic. That's stuff  I previously already mentioned and one of the things I also wanted to add is we also  wanted to make it really secure because there are a bunch of other monitoring agents written  in Ruby, Python and so on and some of those agents don't even use HTTPS or a secure connection  so our goal was to only use a secure connection on all the bundles. Bundle is basically a  tar package which contains a Lua code is signed and also the binaries are signed the actual  signing depends on the platform we use the Davian specific stuff and Davian systems and  on Windows we also use the Microsoft thing for signing the executable so we pretty much  made it really secure. Another thing we wanted to do when we build agent is keep the protocol  simple so we decided to go with the new line the limited JSON. Here's an example of the  outgoing message in this case we send a heartbeat which is kind of like a ping so we have this  semi-structured JSON some of the fields like ID and version and so on are required and  other stuff like prams are optional and depend on the actual method and here's the response  from the server for example. Here's a graph which shows a memory usage of agents written  in different programming languages there's agent written C++ Ruby and so on and there's  our Virgo which only uses around like five megabytes of memory we could actually go even  lower but because of we also wanted to make the whole agent really cross-platform and  portable so we decided to statically link all the libraries which does increase memory  usage a bit especially because we link against LibOpenSSL which is known to take a lot of  memory and here's a quick diagram of the our endpoint architecture basically agent connects  to multiple endpoints that's for high availability reasons so if one of our data centers goes  down for example your agent can still send metrics to us and we can process them and  send an alert for example basically agent establishes three persistent long-running  connections but it actually only sends data which is metrics and other system information  over one connection and if this connection dies other connection other secondary connection  is promoted to a primary connection so I guess that's it I hope you I got you interested  and if you have a similar problem for example if you want to embed something or you need  a low memory footprint and fast execution you should probably check love it and Virgo  and also want to add that yeah if you're interested in this kind of problems my company  rex space is looking for new people to join us so let me know go to this website or send  me an email or find me around the around this room after the talk I guess thank you oh yeah  and also have some Apache lipcloth Rex with God monitoring and Rex based service registry  stickers I'll leave them on the desk and you can go grab it.
I_m_Luvin_It_iIzKQ--iYyU_2_0_71.png I_m_Luvin_It_iIzKQ--iYyU_2_72_683.png I_m_Luvin_It_iIzKQ--iYyU_2_684_1487.png I_m_Luvin_It_iIzKQ--iYyU_2_1488_3227.png I_m_Luvin_It_iIzKQ--iYyU_2_3228_3623.png I_m_Luvin_It_iIzKQ--iYyU_2_3624_4403.png I_m_Luvin_It_iIzKQ--iYyU_2_4404_5219.png I_m_Luvin_It_iIzKQ--iYyU_2_5220_7415.png I_m_Luvin_It_iIzKQ--iYyU_2_7416_7775.png I_m_Luvin_It_iIzKQ--iYyU_2_7776_11771.png I_m_Luvin_It_iIzKQ--iYyU_2_11772_13019.png I_m_Luvin_It_iIzKQ--iYyU_2_13020_13667.png I_m_Luvin_It_iIzKQ--iYyU_2_13668_14531.png I_m_Luvin_It_iIzKQ--iYyU_2_14532_15527.png I_m_Luvin_It_iIzKQ--iYyU_2_15528_15707.png I_m_Luvin_It_iIzKQ--iYyU_2_15708_16331.png I_m_Luvin_It_iIzKQ--iYyU_2_16332_16677.png