Hermes Message Dispatching
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Freitag, Klaas
Hermes is a message digesting and distribution system which is aimed to give back control to the user. Many already existing systems produce a lot of information for users, mostly by flooding with email. That was a very practical and quick solution in the past, but nowadays there are more elegant methods of notifying. Hermes lets the user control, which information approaches him and how. It collects information from connected sources, manages user subscriptions to the various notification types and provides the user with a selection of the messages he wants to get, as mails, feeds or jabber messages and more. This talk is an introduction to Hermes and the way the openSUSE project uses it for all kinds of notifications about the distribution.
 Anyway, our first speaker today is class phytarch. Sorry, but don't get to remember your name. He's going to talk about Hermes message dispatching, as you can see here, which, as I understood, is a system meant to have several store-bought each other, so I mean, very interesting, and which you guys use it too. So please, everybody, welcome class phytarch. Thank you. Thank you. Thank you. Thank you. Thanks a lot. So yeah, my name is class phytarch. I work for SUSE in our more than 10 years, and over the time, I was doing a lot of projects in the tools area that we use at SUSE, how to build and databases we used for stuff to build packages and distributions. And now I'm part of the so-called open SUSE boosters team. We try to do activity to enable the community better to work with open SUSE, and use open SUSE as a platform for the upstream and projects activities. Today, I'd like to talk about Hermes. Hermes is a project I was setting up, I think, three or four years ago. We use it in the open SUSE build service, because there are problems I'd like to talk about in the first. And the first part of this talk, the second part of the talk, will be the solution, which is called Hermes in this case. And the third part, I give you some technical details about this system. And in the fourth part, you're invited to ask questions if you have some. So what's the problem? The problem, I'm talking about problems with emails or notifications at all. So in the beginning, most of you probably remember that. You said to people, send me a quick email, and I can sort it out and do stuff. You got important information with email. Email was fun and was a new thing. And your work organization used a lot of email over the time, and it got very, very efficient. And then, machines started to send email. With machines, I mean stuff like build systems, like bug trackers, like maybe feature trackers, and stuff like that. And I send out emails to you. If your package doesn't build on this and this platform, it doesn't build for this distribution, which is three years old, and stuff like that. So you end up with a lot of emails. And all these emails look similar. You got too much of them. You're not interested in stuff like this one week ago. So with this machine sending email, everything just turns upside down. You got tons of emails. You lose important information, because if you have 1,500 emails in your inbox, you're not able to sort out which are really important and pushing off. It's not fun anymore, and your work organization probably breaks. So what we were thinking about was the possibility to get back control. So we set up hammers with the intention to give the decision back to you which information you want and which you don't want. So don't let the machine, the system, the build system, the system send email to you. But you go to Hermes and say, I'm interested in this particular message from the build system. So it's not any more longer that you've got 10 emails from the build system, because one of your package fails. Because it fails in 10 different distributions, while you're only interested in one at this moment. So with Hermes, you can configure very detailed which information you want to have. So that works in the way that the client system, in case, in the example, the build system, doesn't deliver the message directly to you as a user. But it delivers its notification of an event to the Hermes system, and you configure and decide in Hermes which notification you are interested in. And it's a conscious decision, of course. So you don't get flooded with unwanted stuff, but you decide consciously with what you want to have. The messages are, of course, kept in a system which stores it securely, and it's predictable and visible, which messages were sent out. And also the message itself is configurable by the Hermes admin and yourself. So you can decide how the message should look like. Because the style of the message is no longer part of the implementation of the build system, which simply dumps a text file to a setmate. But it's part of the Hermes system where you have a configuration to configure it as you need it. We have various deliveries. Also I'm talking here about mail a lot. We have other deliveries such as RSA feeds or Jabber or Twittering, which is interesting because what I said up recently was that Opens with the Build Service now fills a Twitter channel with information about package updates in the factory project. So of course, not every individual sets up a Twitter thing that's more a general approach to set up interesting sources of information for people. But also it's possible as an HTTP post that gives the possibility to send information to other systems like on a push approach. Whenever a package, for example, changes in the Opens with the Build Service, we can do an HTTP post to another system which can do stuff with information. So this is something that goes away from a personal messaging approach. It's more an interdisciplinar communication style. There's a web client where you can read messages and there are tons of other possibilities which could be implemented. The idea is that every user can pick for himself in which way a message should be delivered. So it's up to the user to decide if he wants package build failure messages, for example, as S feeds or as a Jabber notification. Another feature is our digest. This is an example for an example mail that is sent off by Hermes for, I think it's a submit request in the Build Service. So the actual information is this part. This is just a footer. And for some messages, you will get a lot of these messages which just have a little bit of information. And you get them every two minutes. So what digesting means is you can combine them. You tell Hermes, please don't send this kind of message every two minutes to me, but collect them and send them hourly digest, for example. And then all the interesting parts are collected together, combined with a header and footer. And then it's sent out on a timely basis. So we can send the digest once a minute hourly daily or weekly. Let's come to some technical details. Hermes consists of four parts at the moment, four important parts. This is Terminator. It's a web server-based little script that receives a notification from the clients, for example, a build system. The generator, which connects the incoming notifications and the receivers. The worker, which actually sends out messages. And Starship is a configuration and user web application. So why did we split that up? We started over with one big tool that did most of the stuff. It turned out that, for example, the actual sending of the messages takes too long, takes time, as sometimes there are issues, stuff like that. So it's better if it runs isolated. The same is for the web application. This is where we have rates up, which is better isolated. And in the future, we could, for example, combine Herminator and Starship again. But it's nothing really important. As a data storage, we use a MySQL database at the moment. There's a pretty optimized schema over time. Of course, as most of you might have noticed, you have to pay attention to the database schema if you want performance and reliability. So I think most of the attention went into the database schema. All the applications work on the database. It's pretty performant. Let's take a deeper look into each of them. The Herminator thing is a policy-GI application-based script that runs on lite or Apache. I think we run it in lite at the moment on the build service. The client interface is nothing but an overall where the clients, for example, the build service, can post a notification. And the notification is basically just a string. So one string identifies the kind of notification that is happening. For example, package build fail. This is a complete notification. And because you need more information about this event, for example, the package name, the platform, the architecture, what else, this can come as key value pairs as much as you want. And the system works that way that you don't have to set up notification types of stuff. You just send them once, and then it's there. And after that, since your client often or frequently sends the same notification type, they come in more and more messages from the same type. So just send one, and it's there. Later, there's an admin interface where stuff can be administered. The notification type, the string, identifies the only required parameter. Together with Terminator comes an admin interface where you can see which notification types were sent up by just sending them in. And also, as an admin, can administer the text template that is sent out for mail and for all the other things. This is publicly available under notifyopenscruiser.org. But you have to have an admin account, of course. The generator is just a false script that connects incoming notifications to the subscriptions. Subscriptions are people have to subscribe to the notifications. So you have to say, I'm interested in package space, or I'm interested in feature changes, or I'm interested in whatever. This is called a subscription. So the connection between a message type or a notification type and the user as a subscription. And with the subscription, you also state in which way you want to receive the message. And when, if you want an immediate view or if it should come as a digest thing. The worker is a person that actually sends out the messages. It does mail sending, connects to send mail, and sends it out, or does the RSS stuff and twittering, these kind of things, jabber. It runs in an infinite loop on the server with breaks, so that it doesn't cause too much load on the machine. And whenever something gets generated from the generator, it sends out afterwards. This does it. It's the Ruby on Rails application. It's a harmless user interface for users, where you maintain your subscriptions, where you can read actual messages, where you can configure RSS feeds you want to see, and you want to fill with your messages. It has a user mode, which is pretty simple. It's set up as an abstraction layer where people just can say, I'm interested in feature changes, for example. And in the background, there's subscription with subscription filters, and some more complex stuff is set up for the people. And if you want to go one step beyond and look into the details, you can go to the expert interface and set up stuff more in detail. Again, an overview diagram. Here's how Minata, which is just a server-based web application. The client system posts the raw notifications in there. They are stored in the notifications and parameters tables in the database. The generator frequently checks if there are new raw notifications, and combines it with the subscriptions and the filters, and generates the messages that should go out to the people based on the settings here and the subscriptions. The messages are stored in this part of the database, and the worker all the time checks if there's something in and sends it out as mail or RSS. And Starship is editing subscriptions and the filters, and you can read messages and view stuff and so on. So far, I have run some production for our OpenSuser project for OBS and OpenFate. Currently, we are filling up and connect more and more stuff to it, since we have now fixed all the stuff that comes in the beginning. So when you start over with such a project, you run into a lot of tiny problems. Currently, we are processing more than 50,000 notifications a day. Of course, there's stuff like cleanup scripts in the background, which again clean up the database and stuff like that. There's a project page in the OpenSuser Biki, if you are interested. And it's free software, of course. It's independent from the OpenSuser Build service, so you can run Hermes for any other purpose than the OpenSuser Build service. It's not connected. It's not using the Build service code base or something. So it's just an independent project. But of course, it's developed along the Build service, so it's pretty part of that. So I'm finished. I'm probably we have plenty of time left, right? Yeah, 10 to 15 minutes. Are there any questions from your side? You said clients send a message to the Herminator. So there's some kind of age-eater in the client? Or is this an email, actually? No, it's just an HTTP post. So you need an HTTP connection between the client and the Herminator. And of course, you have to make sure that you do your network configuration right, because otherwise, if you, for example, open up your Herminator to the internet, you're, of course, the perfect spammer. Because people could send notifications, and if everything is set up accordingly, Hermes would send all messages. You have to be careful with that. For example, for the OpenSuser stuff, we have it configured that way that only internal network can reach the Herminator. So what exactly needs to be stored in the client's systems? It's simple. On the client's systems, you can either use a Perl script, which provides another Perl script, a Perl module that provides one function to send out a notification. I think it takes as parameters the notification name or a string, and then hash reference, which contains the parameters. So if you use other languages, you would have to implement an HTTP post, which is easy for most systems, I guess. Other questions? All right, then thank you very much, Dupar. Thank you very much. Thank you.
 Anyway, our first speaker today is class phytarch.  Sorry, but don't get to remember your name.  He's going to talk about Hermes message dispatching,  as you can see here, which, as I understood,  is a system meant to have several store-bought  each other, so I mean, very interesting,  and which you guys use it too.  So please, everybody, welcome class phytarch.  Thank you.  Thank you.  Thank you.  Thank you.  Thanks a lot.  So yeah, my name is class phytarch.  I work for SUSE in our more than 10 years,  and over the time, I was doing a lot of projects  in the tools area that we use at SUSE,  how to build and databases we used for stuff  to build packages and distributions.  And now I'm part of the so-called open SUSE boosters team.  We try to do activity to enable the community better  to work with open SUSE, and use open SUSE as a platform  for the upstream and projects activities.  Today, I'd like to talk about Hermes.  Hermes is a project I was setting up, I think,  three or four years ago.  We use it in the open SUSE build service,  because there are problems I'd like to talk about in the first.  And the first part of this talk, the second part of the talk,  will be the solution, which is called Hermes in this case.  And the third part, I give you some technical details  about this system.  And in the fourth part, you're invited to ask questions  if you have some.  So what's the problem?  The problem, I'm talking about problems  with emails or notifications at all.  So in the beginning, most of you probably remember that.  You said to people, send me a quick email,  and I can sort it out and do stuff.  You got important information with email.  Email was fun and was a new thing.  And your work organization used a lot of email over the time,  and it got very, very efficient.  And then, machines started to send email.  With machines, I mean stuff like build systems, like bug  trackers, like maybe feature trackers, and stuff like that.  And I send out emails to you.  If your package doesn't build on this and this platform,  it doesn't build for this distribution, which is three  years old, and stuff like that.  So you end up with a lot of emails.  And all these emails look similar.  You got too much of them.  You're not interested in stuff like this one week ago.  So with this machine sending email, everything just turns  upside down.  You got tons of emails.  You lose important information, because if you have 1,500  emails in your inbox, you're not able to sort out which are  really important and pushing off.  It's not fun anymore, and your work organization  probably breaks.  So what we were thinking about was the possibility to get  back control.  So we set up hammers with the intention to give the decision  back to you which information you want and which you don't  want.  So don't let the machine, the system, the build system, the  system send email to you.  But you go to Hermes and say, I'm interested in this  particular message from the build system.  So it's not any more longer that you've got 10 emails from  the build system, because one of your package fails.  Because it fails in 10 different distributions,  while you're only interested in one at this moment.  So with Hermes, you can configure very detailed  which information you want to have.  So that works in the way that the client system, in case,  in the example, the build system, doesn't deliver the  message directly to you as a user.  But it delivers its notification of an event to the  Hermes system, and you configure and decide in  Hermes which notification you are interested in.  And it's a conscious decision, of course.  So you don't get flooded with unwanted stuff, but you decide  consciously with what you want to have.  The messages are, of course, kept in a system which stores  it securely, and it's predictable and visible,  which messages were sent out.  And also the message itself is configurable by the Hermes  admin and yourself.  So you can decide how the message should look like.  Because the style of the message is no longer part of the  implementation of the build system, which simply dumps a  text file to a setmate.  But it's part of the Hermes system where you have a  configuration to configure it as you need it.  We have various deliveries.  Also I'm talking here about mail a lot.  We have other deliveries such as RSA feeds or Jabber or  Twittering, which is interesting because what I  said up recently was that Opens with the Build Service now  fills a Twitter channel with information about package  updates in the factory project.  So of course, not every individual sets up a Twitter  thing that's more a general approach to set up interesting  sources of information for people.  But also it's possible as an HTTP post that gives the  possibility to send information to other systems like on a  push approach.  Whenever a package, for example, changes in the  Opens with the Build Service, we can do an HTTP post to  another system which can do stuff with information.  So this is something that goes away from a personal  messaging approach.  It's more an interdisciplinar communication style.  There's a web client where you can read messages and there  are tons of other possibilities which could be  implemented.  The idea is that every user can pick for himself in which way  a message should be delivered.  So it's up to the user to decide if he wants package build  failure messages, for example, as S feeds or as a Jabber  notification.  Another feature is our digest.  This is an example for an example mail that is sent  off by Hermes for, I think it's a submit request in the  Build Service.  So the actual information is this part.  This is just a footer.  And for some messages, you will get a lot of these  messages which just have a little bit of information.  And you get them every two minutes.  So what digesting means is you can combine them.  You tell Hermes, please don't send this kind of message  every two minutes to me, but collect them and send them  hourly digest, for example.  And then all the interesting parts are collected  together, combined with a header and footer.  And then it's sent out on a timely basis.  So we can send the digest once a minute hourly daily or  weekly.  Let's come to some technical details.  Hermes consists of four parts at the moment, four  important parts.  This is Terminator.  It's a web server-based little script that receives a  notification from the clients, for example, a build  system.  The generator, which connects the incoming  notifications and the receivers.  The worker, which actually sends out messages.  And Starship is a configuration and user  web application.  So why did we split that up?  We started over with one big tool that did most of the  stuff.  It turned out that, for example, the actual sending of  the messages takes too long, takes time, as sometimes  there are issues, stuff like that.  So it's better if it runs isolated.  The same is for the web application.  This is where we have rates up, which is better isolated.  And in the future, we could, for example, combine Herminator  and Starship again.  But it's nothing really important.  As a data storage, we use a MySQL database at the moment.  There's a pretty optimized schema over time.  Of course, as most of you might have noticed, you have to  pay attention to the database schema if you want  performance and reliability.  So I think most of the attention went into the  database schema.  All the applications work on the database.  It's pretty performant.  Let's take a deeper look into each of them.  The Herminator thing is a policy-GI application-based  script that runs on lite or Apache.  I think we run it in lite at the moment on the build  service.  The client interface is nothing but an  overall where the clients, for example, the build service,  can post a notification.  And the notification is basically just a string.  So one string identifies the kind of notification that  is happening.  For example, package build fail.  This is a complete notification.  And because you need more information about this event,  for example, the package name, the platform, the  architecture, what else, this can come as key value pairs  as much as you want.  And the system works that way that you don't have to set  up notification types of stuff.  You just send them once, and then it's there.  And after that, since your client often or frequently  sends the same notification type, they come in more and  more messages from the same type.  So just send one, and it's there.  Later, there's an admin interface where stuff can be  administered.  The notification type, the string, identifies the  only required parameter.  Together with Terminator comes an admin interface where you  can see which notification types were sent up by just  sending them in.  And also, as an admin, can administer the text template  that is sent out for mail and for all the other things.  This is publicly available under notifyopenscruiser.org.  But you have to have an admin account, of course.  The generator is just a false script that connects  incoming notifications to the subscriptions.  Subscriptions are people have to subscribe to the  notifications.  So you have to say, I'm interested in package space, or  I'm interested in feature changes, or  I'm interested in whatever.  This is called a subscription.  So the connection between a message type or a notification  type and the user as a subscription.  And with the subscription, you also state in which way you  want to receive the message.  And when, if you want an immediate view or if it  should come as a digest thing.  The worker is a person that actually sends out the  messages.  It does mail sending, connects to send mail, and sends it  out, or does the RSS stuff and twittering, these kind of  things, jabber.  It runs in an infinite loop on the server with breaks, so  that it doesn't cause too much load on the machine.  And whenever something gets generated from the generator,  it sends out afterwards.  This does it.  It's the Ruby on Rails application.  It's a harmless user interface for users, where you  maintain your subscriptions, where you can read actual  messages, where you can configure RSS feeds you want  to see, and you want to fill with your messages.  It has a user mode, which is pretty simple.  It's set up as an abstraction layer where people just can  say, I'm interested in feature changes, for example.  And in the background, there's subscription with  subscription filters, and some more complex stuff is set  up for the people.  And if you want to go one step beyond and look into the  details, you can go to the expert interface and set up  stuff more in detail.  Again, an overview diagram.  Here's how Minata, which is just a server-based  web application.  The client system posts the raw notifications in there.  They are stored in the notifications and parameters  tables in the database.  The generator frequently checks if there are new raw  notifications, and combines it with the subscriptions and  the filters, and generates the messages that should go out  to the people based on the settings here and the  subscriptions.  The messages are stored in this part of the database, and  the worker all the time checks if there's something in and  sends it out as mail or RSS.  And Starship is editing subscriptions and the filters,  and you can read messages and view stuff and so on.  So far, I have run some production for our OpenSuser  project for OBS and OpenFate.  Currently, we are filling up and connect more and more stuff  to it, since we have now fixed all the stuff that  comes in the beginning.  So when you start over with such a project, you run into a lot  of tiny problems.  Currently, we are processing more than 50,000  notifications a day.  Of course, there's stuff like cleanup scripts in the  background, which again clean up the database and stuff like  that.  There's a project page in the OpenSuser Biki, if you are  interested.  And it's free software, of course.  It's independent from the OpenSuser Build service, so you  can run Hermes for any other purpose than the OpenSuser  Build service.  It's not connected.  It's not using the Build service code base or something.  So it's just an independent project.  But of course, it's developed along the Build service, so  it's pretty part of that.  So I'm finished.  I'm probably we have plenty of time left, right?  Yeah, 10 to 15 minutes.  Are there any questions from your side?  You said clients send a message to the Herminator.  So there's some kind of age-eater in the client?  Or is this an email, actually?  No, it's just an HTTP post.  So you need an HTTP connection between the client and the  Herminator.  And of course, you have to make sure that you do your  network configuration right, because otherwise, if you,  for example, open up your Herminator to the internet,  you're, of course, the perfect spammer.  Because people could send notifications, and if everything  is set up accordingly, Hermes would send  all messages.  You have to be careful with that.  For example, for the OpenSuser stuff, we have it  configured that way that only internal network can reach  the Herminator.  So what exactly needs to be stored in the client's  systems?  It's simple.  On the client's systems, you can either use a Perl script,  which provides another Perl script, a Perl module that  provides one function to send out a notification.  I think it takes as parameters the notification name or a  string, and then hash reference, which  contains the parameters.  So if you use other languages, you would have to  implement an HTTP post, which is easy for most systems, I  guess.  Other questions?  All right, then thank you very much, Dupar.  Thank you very much.  Thank you.
Hermes_Message_Dispatching_gzyPp8OoqaQ_0_47.png Hermes_Message_Dispatching_gzyPp8OoqaQ_48_95.png Hermes_Message_Dispatching_gzyPp8OoqaQ_96_143.png Hermes_Message_Dispatching_gzyPp8OoqaQ_144_191.png Hermes_Message_Dispatching_gzyPp8OoqaQ_192_239.png Hermes_Message_Dispatching_gzyPp8OoqaQ_240_335.png Hermes_Message_Dispatching_gzyPp8OoqaQ_336_443.png Hermes_Message_Dispatching_gzyPp8OoqaQ_444_491.png Hermes_Message_Dispatching_gzyPp8OoqaQ_492_563.png Hermes_Message_Dispatching_gzyPp8OoqaQ_564_695.png Hermes_Message_Dispatching_gzyPp8OoqaQ_696_2807.png Hermes_Message_Dispatching_gzyPp8OoqaQ_2808_3707.png Hermes_Message_Dispatching_gzyPp8OoqaQ_3708_4775.png Hermes_Message_Dispatching_gzyPp8OoqaQ_4776_5339.png Hermes_Message_Dispatching_gzyPp8OoqaQ_5340_9095.png Hermes_Message_Dispatching_gzyPp8OoqaQ_9096_11879.png Hermes_Message_Dispatching_gzyPp8OoqaQ_11880_13919.png Hermes_Message_Dispatching_gzyPp8OoqaQ_13920_15923.png Hermes_Message_Dispatching_gzyPp8OoqaQ_15924_20447.png Hermes_Message_Dispatching_gzyPp8OoqaQ_20448_21743.png Hermes_Message_Dispatching_gzyPp8OoqaQ_21744_24047.png Hermes_Message_Dispatching_gzyPp8OoqaQ_24048_25427.png Hermes_Message_Dispatching_gzyPp8OoqaQ_25428_25475.png Hermes_Message_Dispatching_gzyPp8OoqaQ_25476_25571.png Hermes_Message_Dispatching_gzyPp8OoqaQ_25572_25631.png Hermes_Message_Dispatching_gzyPp8OoqaQ_25632_25691.png Hermes_Message_Dispatching_gzyPp8OoqaQ_25692_31324.png