Anatomy of GNOME Calls
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Tzaras, Evangelos Ribeiro (devrtz)
In this talk we will take a look at the anatomy of GNOME Calls. We will cover libraries used and how Calls interacts with them to provide call functionality and other things you'd expect from a dialer application.
 Welcome to my first ever FOSTEM Talk. Today I want to talk about GNOME calls, all the building blocks that it is made of and briefly explain what it takes to place and receive phone calls. So who am I? My name is Evangeles and I've been a long time Linux user and free software enthusiast and what got me started contributing to free software was basically receiving my pine phone shortly after FOSTEM 2020 where I got involved with the Mobian project. Today I work as a free software developer for Purism on GNOME calls. So what is GNOME calls? I hear you ask. It is a dialer application so that means you can place and receive calls with it. It is written in C using GTK and G-Object and you can use it both for regular telephony calls and voice over IP calls using the SIP protocol which is a relatively new addition. Now let's have a broad overview of all the different things that are involved. So we obviously need to have some kind of user interface. There we use GTK3 and Lippenvi and also LippcallUI which I will get to in a few seconds. Then we obviously need to talk to our modem for that we use modem manager. In the SIP case we went with the Zofia SIP library and obviously we want to be able to actually hear something when we are doing a call and that is where a call audio and whiz come into play in the cellular case and the G-streamer in the SIP case. So how does it look like? On the left side you can see the recent call history which is pretty standard stuff. On the right side there is a dialpad which you can use to call arbitrary numbers as long as they are valid. I've mentioned LippcallUI. This is basically a library that was born out of the necessity to share some UI widgets between calls and the phone shell. Since you usually want to be able to hang up on an ongoing call on the lock screen, this is basically where it all came from. And it's a private library which both calls and phone are sharing. Now let's go to the modem manager side. Basically what it does is it has a daemon which talks to the modem hardware and exposes a number of objects over debuts. On the application side we use libmm-glip which provides a convenient API to talk to the modem manager daemon and it gives us G objects for all the different things like our modems and calls and so on. And how it's roughly used is we watch for the org-free desktop modem manager 1 object path and look at all the objects that are exported under that path and if an object implements the modem manager 1 voice interface, that is basically a voice capable modem. And then we can go on. So we have this, like the corresponding G object would be an mm-modem-voice for a call. We have an mm call and for example if we want to start an outgoing call we would invoke something like mm-modem-voice-create-call which returns an mm call. In the outgoing case we would then call on the mm call the mm call start method or for an incoming call the accept method hang up when we're done talking, send DTMF tones and so on. Now when since modem manager doesn't deal with any audio we need some more stuff. We are using call audio which is a daemon talking to pulse audio and exposing some controls on Dbass. It allows us to switch audio profiles, enable speaker, mute microphone and from an application point of view we're using lib call audio which gives us easy to use API to talk with the daemon and these are some of the methods that are in the public API. Then there's also WIS. WIS uses loopback devices to set up audio routing between the host and modem. So when your modem exposes its own audio interface in the system that is why it's not used on the original Pinephone because the audio routing is done in hardware and how WIS basically works is that it also watches for any new calls on Dbass and once then an active state let's say this setup of the loopback devices will be done and your audio will be flowing from your modem to your microphone to your modem and from your modem to the speaker let's say. Now let's look at the zip side. So Fia zip is basically a lot of things or brings a lot of things. It's a library for dealing with a ton of zip related stuff so you have a parser and a module for zip offer answer but what we're looking at from a high level perspective is it's a user agent and does the signaling. How we're using it inside of calls is basically we provide some callbacks for different event types for example for incoming invites so for an incoming call or as a response to an outgoing invite we provide certain functions that we want to have called and also Fia zip comes with an additional helper library which provides a g-source for easy glib main loop integration. How it basically works is when you want to register to a server you call the new register function give it some arguments like your username and where to send or which zip server to register to and then you get some zip messages sent. This is cut short a bit just for brevity's sake and next up let's look at how it would look for placing a call. Again you have an invite call tell it who you want to call and in the SDP string you give it the session description so which type of audio codecs you speak and you get a message which looks something like this in the lower part you have the body which in this case for example tells your peer where you can be reached and what kind of codecs you speak. The actual audio will be done using gstreamer so in the zip case you have basically RTP so the real time protocol for sending audio over the network also works for video but we are only doing audio as of yet. Basically we have two pipelines they are obviously written in our C code but if you are familiar with the GST launch command this is roughly what it takes to set up the sending pipeline so where our microphone input gets encoded put into RTP packets and then send off over the network to some host and the reverse for the receive pipeline where the receiving audio getting it out of our RTP packets decoding it and then playing it back. I've mentioned earlier that there are some or I've showed earlier that there are some other libraries that we are using and first of all no dialer application is worth its name without some sort of contact integration for that we are using libfolks so that gives us basically from the evolution data server backend a list of all known contacts we can use it to query the contacts by phone number. And there is libfeedback which is also very important to have it provides us an easy way to have feedback hence the name it gives us audio feedback so ringing, haptic like the vibration motor and visual feedback so your LED turns on. And there's also libsecrets which is only used in the zip case and is basically used to store and retrieve credentials. And last but not least we have libpiece which provides us with a plugin system that we are using basically all the different backends that we can use are written as plugins which are DL loaded, opened. And what brings the future there's a couple of things which are currently missing in the cellular case we don't yet support supplementary call services like call holding, call waiting, transferring calls and so on we also don't support conference calls yet we don't have a convenient way to access our voicemail and in the voice over IP case we currently only do unencrypted media and we are also lacking video calls. And if you want to learn more check out the repository over at the GNOME good lab instance and look at the documentation and if you have any questions feel free to get in contact via email, matrix, all the fatty verus. And lastly I want to thank Arnaud for creating Mobian, Henri Nicolas for helping me with the ambient packaging, Guido for all the code reviews, Purism for building the Fosh ecosystem, GNOME for generously hosting the project and being a helpful community and all of you who are listening. Thank you. We are live now. So first of all thank you very much for the interesting talk. We have four questions at this time and people in the room please don't be shy, ask away. Start from the first one. Will Sofia Sip be used to or can use it to help support VoLTE? While it's definitely true that VoiceOver LTE is using Sip that is actually done inside of the modem so calls doesn't explicitly use Sip in that case. Yes, in my understanding there is no way to bypass so to speak the modem and have direct access to the SIP server. Yes, that's correct. I mean maybe for, I'm not sure how this relates to say something like the open firmware from Big York on the Pine phone if that needs to be explicitly handled in that case but from my understanding all of this stuff needs to be done by the modem itself. Thank you. Next question is would it be possible to use Sopya Sip to bridge calls between modem, manager and matrix? No, not as such but what we can do is there are some parts of the code mainly the G streamer pipeline for handling all the audio and potentially video. That part could certainly be reused when we are at some point supporting matrix but the Sopya Sip is primarily about the signaling and that is very SIP specific but of course you have all these things like the session description which are common for these sort of voice over IP sessions. Okay, thank you. The next question is from me, can you provide some more details on what the auxiliary libraries do? Yes, sure. So LibFolks, I think most of the things that I'm using it for I've covered in the talk itself so from a high level perspective it just is able to query evolution data server and it aggregates all your contacts into so-called folk individuals and that is basically what I am given if I ask it to give me all the contacts and I use that to then fill up the contact box and the same or a similar thing applies when we are querying folks for any number so we give it a number and we want to get out the contact that is associated with that number. Then maybe one thing that I forgot actually to mention is we are also using a GOM. GOM basically handles all the details when dealing with our SQL or SQLite I should rather say a database where we store all the previous calls that we have so that is basically where the history comes from. Thank you. Moving on, does the N8XX support the SIP codes? So an XX I am assuming that is basically all the different Nokia N8 something devices. I am afraid that I can't really answer that. The best answer I can give is I would assume so because the Sofia SIP library was originally written by Nokia people and so I would be very much surprised if they didn't use that on their devices. Thanks. Moving on, is there any issue when using Pipewire with calls? I have not actually run this. I think the Manjaro people are actually using Pipewire and seeing as you usually have all this compatibility layers like Pipewire dash pulse, those should basically be able to handle this sort of thing. Now if we are looking at something like call audio, I am not entirely sure if all these features like switching to the speaker and so on if they really do work. But I would guess if you are only looking at the actual call audio that should be working. I hope someone correct me if I am wrong. The Wildpac wire is very new so it is still rough around the edges. I think we have 30 seconds for a last quick question. If there is matric support on the roadmap for non-calls? I definitely think that this is something that we want to have but when exactly that might happen. I am not sure, since Chatti is using matrix and there will probably have to be some sort of, we need to flash out the interactions there. But at some point it will come. Okay, the live broadcast is finishing here. Thank you very much for your time and we can stay here in the room.
 Welcome to my first ever FOSTEM Talk.  Today I want to talk about GNOME calls, all the building blocks that it is made of and  briefly explain what it takes to place and receive phone calls.  So who am I?  My name is Evangeles and I've been a long time Linux user and free software enthusiast  and what got me started contributing to free software was basically receiving my pine phone  shortly after FOSTEM 2020 where I got involved with the Mobian project.  Today I work as a free software developer for Purism on GNOME calls.  So what is GNOME calls?  I hear you ask.  It is a dialer application so that means you can place and receive calls with it.  It is written in C using GTK and G-Object and you can use it both for regular telephony  calls and voice over IP calls using the SIP protocol which is a relatively new addition.  Now let's have a broad overview of all the different things that are involved.  So we obviously need to have some kind of user interface.  There we use GTK3 and Lippenvi and also LippcallUI which I will get to in a few seconds.  Then we obviously need to talk to our modem for that we use modem manager.  In the SIP case we went with the Zofia SIP library and obviously we want to be able to  actually hear something when we are doing a call and that is where a call audio and  whiz come into play in the cellular case and the G-streamer in the SIP case.  So how does it look like?  On the left side you can see the recent call history which is pretty standard stuff.  On the right side there is a dialpad which you can use to call arbitrary numbers as long  as they are valid.  I've mentioned LippcallUI.  This is basically a library that was born out of the necessity to share some UI widgets  between calls and the phone shell.  Since you usually want to be able to hang up on an ongoing call on the lock screen, this  is basically where it all came from.  And it's a private library which both calls and phone are sharing.  Now let's go to the modem manager side.  Basically what it does is it has a daemon which talks to the modem hardware and exposes  a number of objects over debuts.  On the application side we use libmm-glip which provides a convenient API to talk to  the modem manager daemon and it gives us G objects for all the different things like  our modems and calls and so on.  And how it's roughly used is we watch for the org-free desktop modem manager 1 object  path and look at all the objects that are exported under that path and if an object  implements the modem manager 1 voice interface, that is basically a voice capable modem.  And then we can go on.  So we have this, like the corresponding G object would be an mm-modem-voice for a call.  We have an mm call and for example if we want to start an outgoing call we would invoke  something like mm-modem-voice-create-call which returns an mm call.  In the outgoing case we would then call on the mm call the mm call start method or for  an incoming call the accept method hang up when we're done talking, send DTMF tones and  so on.  Now when since modem manager doesn't deal with any audio we need some more stuff.  We are using call audio which is a daemon talking to pulse audio and exposing some controls  on Dbass.  It allows us to switch audio profiles, enable speaker, mute microphone and from an application  point of view we're using lib call audio which gives us easy to use API to talk with the  daemon and these are some of the methods that are in the public API.  Then there's also WIS.  WIS uses loopback devices to set up audio routing between the host and modem.  So when your modem exposes its own audio interface in the system that is why it's not used on  the original Pinephone because the audio routing is done in hardware and how WIS basically  works is that it also watches for any new calls on Dbass and once then an active state  let's say this setup of the loopback devices will be done and your audio will be flowing  from your modem to your microphone to your modem and from your modem to the speaker let's  say.  Now let's look at the zip side.  So Fia zip is basically a lot of things or brings a lot of things.  It's a library for dealing with a ton of zip related stuff so you have a parser and a module  for zip offer answer but what we're looking at from a high level perspective is it's a  user agent and does the signaling.  How we're using it inside of calls is basically we provide some callbacks for different event  types for example for incoming invites so for an incoming call or as a response to an  outgoing invite we provide certain functions that we want to have called and also Fia zip  comes with an additional helper library which provides a g-source for easy glib main loop  integration.  How it basically works is when you want to register to a server you call the new register  function give it some arguments like your username and where to send or which zip server  to register to and then you get some zip messages sent.  This is cut short a bit just for brevity's sake and next up let's look at how it would  look for placing a call.  Again you have an invite call tell it who you want to call and in the SDP string you  give it the session description so which type of audio codecs you speak and you get a message  which looks something like this in the lower part you have the body which in this case  for example tells your peer where you can be reached and what kind of codecs you speak.  The actual audio will be done using gstreamer so in the zip case you have basically RTP so  the real time protocol for sending audio over the network also works for video but we are  only doing audio as of yet.  Basically we have two pipelines they are obviously written in our C code but if you are familiar  with the GST launch command this is roughly what it takes to set up the sending pipeline  so where our microphone input gets encoded put into RTP packets and then send off over  the network to some host and the reverse for the receive pipeline where the receiving  audio getting it out of our RTP packets decoding it and then playing it back.  I've mentioned earlier that there are some or I've showed earlier that there are some  other libraries that we are using and first of all no dialer application is worth its  name without some sort of contact integration for that we are using libfolks so that gives  us basically from the evolution data server backend a list of all known contacts we can  use it to query the contacts by phone number.  And there is libfeedback which is also very important to have it provides us an easy way  to have feedback hence the name it gives us audio feedback so ringing, haptic like the  vibration motor and visual feedback so your LED turns on.  And there's also libsecrets which is only used in the zip case and is basically used  to store and retrieve credentials.  And last but not least we have libpiece which provides us with a plugin system that we are  using basically all the different backends that we can use are written as plugins which  are DL loaded, opened.  And what brings the future there's a couple of things which are currently missing in the  cellular case we don't yet support supplementary call services like call holding, call waiting,  transferring calls and so on we also don't support conference calls yet we don't have  a convenient way to access our voicemail and in the voice over IP case we currently only  do unencrypted media and we are also lacking video calls.  And if you want to learn more check out the repository over at the GNOME good lab instance  and look at the documentation and if you have any questions feel free to get in contact  via email, matrix, all the fatty verus.  And lastly I want to thank Arnaud for creating Mobian, Henri Nicolas for helping me with  the ambient packaging, Guido for all the code reviews, Purism for building the Fosh  ecosystem, GNOME for generously hosting the project and being a helpful community and  all of you who are listening.  Thank you.  We are live now.  So first of all thank you very much for the interesting talk.  We have four questions at this time and people in the room please don't be shy, ask away.  Start from the first one.  Will Sofia Sip be used to or can use it to help support VoLTE?  While it's definitely true that VoiceOver LTE is using Sip that is actually done inside  of the modem so calls doesn't explicitly use Sip in that case.  Yes, in my understanding there is no way to bypass so to speak the modem and have direct  access to the SIP server.  Yes, that's correct.  I mean maybe for, I'm not sure how this relates to say something like the open firmware from  Big York on the Pine phone if that needs to be explicitly handled in that case but from  my understanding all of this stuff needs to be done by the modem itself.  Thank you.  Next question is would it be possible to use Sopya Sip to bridge calls between modem,  manager and matrix?  No, not as such but what we can do is there are some parts of the code mainly the G streamer  pipeline for handling all the audio and potentially video.  That part could certainly be reused when we are at some point supporting matrix but the  Sopya Sip is primarily about the signaling and that is very SIP specific but of course  you have all these things like the session description which are common for these sort  of voice over IP sessions.  Okay, thank you.  The next question is from me, can you provide some more details on what the auxiliary libraries  do?  Yes, sure.  So LibFolks, I think most of the things that I'm using it for I've covered in the talk  itself so from a high level perspective it just is able to query evolution data server  and it aggregates all your contacts into so-called folk individuals and that is basically what  I am given if I ask it to give me all the contacts and I use that to then fill up the  contact box and the same or a similar thing applies when we are querying folks for any  number so we give it a number and we want to get out the contact that is associated  with that number.  Then maybe one thing that I forgot actually to mention is we are also using a GOM.  GOM basically handles all the details when dealing with our SQL or SQLite I should rather  say a database where we store all the previous calls that we have so that is basically where  the history comes from.  Thank you.  Moving on, does the N8XX support the SIP codes?  So an XX I am assuming that is basically all the different Nokia N8 something devices.  I am afraid that I can't really answer that.  The best answer I can give is I would assume so because the Sofia SIP library was originally  written by Nokia people and so I would be very much surprised if they didn't use that  on their devices.  Thanks.  Moving on, is there any issue when using Pipewire with calls?  I have not actually run this.  I think the Manjaro people are actually using Pipewire and seeing as you usually have all  this compatibility layers like Pipewire dash pulse, those should basically be able to handle  this sort of thing.  Now if we are looking at something like call audio, I am not entirely sure if all these  features like switching to the speaker and so on if they really do work.  But I would guess if you are only looking at the actual call audio that should be working.  I hope someone correct me if I am wrong.  The Wildpac wire is very new so it is still rough around the edges.  I think we have 30 seconds for a last quick question.  If there is matric support on the roadmap for non-calls?  I definitely think that this is something that we want to have but when exactly that  might happen.  I am not sure, since Chatti is using matrix and there will probably have to be some sort  of, we need to flash out the interactions there.  But at some point it will come.  Okay, the live broadcast is finishing here.  Thank you very much for your time and we can stay here in the room.
mobile_calls_0_623.png mobile_calls_624_2351.png mobile_calls_2352_6239.png mobile_calls_6240_13355.png mobile_calls_13356_14579.png mobile_calls_14580_16055.png mobile_calls_16056_18779.png mobile_calls_18780_20735.png mobile_calls_20736_21923.png mobile_calls_21924_25703.png mobile_calls_25704_25883.png mobile_calls_25884_26447.png mobile_calls_26448_26963.png mobile_calls_26964_27311.png mobile_calls_27312_27875.png mobile_calls_27876_29183.png mobile_calls_29184_29711.png mobile_calls_29712_30839.png mobile_calls_30840_32039.png mobile_calls_32040_32771.png mobile_calls_32772_34487.png mobile_calls_34488_35903.png mobile_calls_35904_36023.png