The road to End-to-End Encryption in Jitsi Meet
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Corretgé, Saúl Ibarra (saghul)
With the advent of Zoom's questionable use of the term "End to End Encryption" many turned their eyes to FLOSS solutions. Thanks to the insertable streams feature which shipped in Chrome at just the right time, we were able to ship a working E2EE implementation in a reasonably short amount of time. In this presentation we'll walk through all that was needed, what our plans for the future are and how others can leverage what we learned to add E2EE to their WebRTC applications.
 Hey everybody, thanks for joining. Saul here from the GCC project. Well, today I'm going to talk about our road or a way to end to an encryption, how we implemented end to an encryption in GCC Meet, what we did, what tools we used, where we ended up and how you can use it yourself in your project. Let's jump right into it. First let's do a quick review of the WebRTC security model to be able to frame this conversation a bit better. WebRTC restricts its availability in current browsers to safe origins, which means that by definition you cannot have a WebRTC call where you're sending data over HTTP. So at least we do have that secure signaling path established. Also when it comes to the media, so the actual audio and video, DTLS, SRTP is mandatory. So WebRTC is encrypted by design. The typical use case is for two peers to find each other in some form of signaling server and then exchange media between them that is as I said encrypted with DTLS, SRTP. However, when it comes to scaling up, so creating conferences with tens, hundreds of people, this model doesn't work. So we switched to using something called selective forwarding unit. And in a selective forwarding unit, the peer connections are established between each participant and the server. And then the server will route each participant's video to all the other participants. In this scenario, because the server is the terminator of these connections, it has access to this media. Now what does what kind of media processing do these kinds of servers do and why is access to this necessary? So selective forwarding units as the name implies, they selectively forward, which means they decide what to forward, what layer, do I forward high definition for this user, middle definition, low definition, you can only switch between layers at certain times, for example, when a keyframe comes, so they need to be able to detect this information. And this information is available in the packet header, for instance. So that's why it was it's needed for these entities to access the media. Now you may wonder, wait a second, so I thought that WebRTC was encrypted. That's what everybody told me. And the answer is a little bit sort of, it's kind of complicated. Not very really. So when the media is flowing peer to peer, as I mentioned in the previous example, then yes, in that case, your your your media would be anti encrypted. But the moment you introduce a selective forwarding unit, so that you can scale up and have multiple, you know, tens, hundreds of participants, this does not work that way. Because again, the connection is terminated on the server side. And in addition, the browsers don't really provide any user experience for for validating that there is no money in the middle, that the remote participant, the key that the remote participant has is the same that you received, and so on. And there's also no indication if the web application running the JavaScript code decides to start sending the media elsewhere. What do I mean by by these last two things? This is an SDP that you may receive with all the information on how to establish a media session. And the highlighted part in bold there is the fingerprint. You will need to compare this fingerprint. You will need to read it out loud to the other participant to know that, you know, there was what they have is what you received in terms of certificates. And as you can imagine, it's not really user friendly. And here's another example. Here is a conference, a call that I'm having with myself to participants, as you can see the IP address is local is a connection is peer to peer. And that's all good and when encrypted. But the moment we had a third participant to the conference, not much happens. Suddenly our layout, in this case, because we're using GTSimmy changes to this. So from the user's perspective, pretty much nothing has changed, except that if we now look where our media is being sent, we're now sending it to the server. So what happened here? Well, we for three participants, we run media through the bridge as is pretty common with these kinds of architectures, but the browser didn't tell you anything. So you thought that that you were sending media peer to peer, but now you're sending through a server and you don't know this happened. And this is where into an encryption can help you for the purposes of these talk. I'm going to kind of focus on one instance or one example of where into an encryption helps, which is eliminate the trust that we put on the selective forwarding unit on the servers. But as a broader exploration of this topic, I really encourage you to take a look at the Millivob's end to an encryption beyond buzzwords talk that he gave at CrankyGeek. It's very elucidating. So how did we go about it? Well, it wasn't possible with existing tools, so new tools had to be invented. And this is where insertable streams comes in. It's the API that unlocked this possibility. It's a new API that allows you for manipulating full frames, full audio and video frames. The data that you're allowed to touch is actually the data is mangled before transport encryption. So before is encrypted with DTLS, SRTP, you can do whatever you want to it. You can encrypt it, you can change it, whatever. You can use web crypto APIs for this process. You can do this in a worker, so it's friendly. You don't need to do expensive operations on the main thread. And as of today, it's only available in the Chromium based browsers. But hopefully these changes in the future. The processing pipeline looks like this. So after encoding the frames, you get access to them in the insertable streams API. You can do pretty much whatever you want. I'm going to focus on one specific implementation or use case of this API, which is mangled the data in the form of encrypting it. But you could do something else. Now, if you look at this processing chain, you think, well, so if I encrypt the data in the insertable stream and on the way out and decrypted on the way in, but then there's also DTLS, are we encrypting twice? Exactly. So we're encrypting once and two and another time hop by hop. So when it arrives at the server, it knows how to decrypt the hop by hop encryption, but it doesn't know how to decrypt the end to end encryption because it doesn't have access to the key. How does this look code wise? Well, when we create our peer connection, we need to pass this new parameter encoded media streams. And then for the sending part for each sender, we need to apply a transform, as you can see there in the code sample. And the transform takes the form of a simple function that takes two parameters, a chunk, which is a full audio or video frame and a controller, which is a kind of a magical object that you can call and key on it to say, hey, this frame is ready. So send it to the next element in the chain. If we look at the opposite direction, so decrypting, we would need to attach a handler for all the remote tracks. When we receive a remote track, we will want to attach a transform to it. And the transform will decrypt in this case. As you can see right there, the decrypt function looks very similar to encrypt one. It takes the chunk and the controller with the same meaning. In this case, it would be an encrypted chunk. And you can after you've decrypted it, you could enqueue it in the controller. Now you probably heard this sentence very often in general. I don't roll your own crypto. So of course, you have all the web crypto APIs available to do whatever you want. But there is this upcoming standard called S frame or secure frames that actually specifies how to use this in this specific scenario. So for implementing end to end encryption and authentication in a web conferencing, in a multi-party video conferencing scenario, it's still early stages, but it is what we use for implementation. There is an ITF working group already formed in Apple is experimenting with the idea of providing this S frame natively in the browser, which would be pretty interesting actually. So with S frame, you get like a payload or a layout definition, how you put the bytes on the wire after you have done your encryption. But of course, it leaves one of the very important aspects out of the specification, which is key management. Like it will tell you what to do with multiple keys, but it won't tell you where you get those keys from and how to secure the exchange and whatnot. And for that, we can use things like MLS or signal or leave all as we did or something else. So what did we do? Well, for encryption, we use what S frame mandates or suggest at the moment, which is ASCTR with 256 keys and a truncated hashes. For signing, we couldn't use what S frame suggests because those curves are not available in the browser. So we use ECDSA with that curve over up there. So we call this slide variation. We have some other slide variations that we will discuss with the S frame working group. We call this J frame. And all of our encryption happens in a worker, which means it's actually and we, for example, we don't minify this worker, so it's easy to inspect when you are in a video call. The net result of this is that when the call is unencrypted, you get the video just fine, like the bunny right there. But when it is and when encrypted, you get that garbage result over there. I lead you into a secret. The reason why you see that garbage is because we leave just a couple of bytes unencrypted to trick the decoder into thinking this is a valid video frame. And then you get this beautiful effect. We'll probably remove this in the future. There's nothing in those bytes, but there's also no reason why you wouldn't encrypt it other than to get like, you know, a beautiful effect, I guess. Now, as I said, important missing piece, key management. Where do keys come from? What do we do with them? We have implemented two modes over time with transitions from one to another. And I'm going to talk about those two. So first is what we call unmanaged shared passphrase based key management. So which means there is no management at all. Users need to type a key that they have agreed upon out of ban. However they want it, they agree that they're going to use this key. If everybody types the same key, then boom, you get video. If you don't type the right key, you get garbled audio and video. We derive a key from this passphrase using PBKDF2 and the key never leaves the user machine, of course. So this is the first iteration that we did on this. But then we wanted to transition to something a little bit more elaborate or that simplified the usage of these passphrases. Here's where we call it the Managed version. Here we started using matrixes, leave all. Leave all by the matrix project is an end to end encryption. Well, it's the double ratchet implementation. Pretty much like what the single protocol is. And what we do is we establish an end to end encrypted channel of communication between each two participants. And we exchange randomly generated per participant keys over these channels. We do automatic key rotation and ratcheting. So if a participant leaves, we rotate the key so this participant is not able to decrypt any future data with the keys that they had, even if they were somehow wiretapping the whole thing. And if a new participant joins, we ratchet the key forward so that this new participant is not able to decrypt past data, but it is able to decrypt future data. And this way we also save ourselves some operations there. As I said, the keys are exchanged over these old channels, given as a secure way to exchange them. So if they are not used for the audio and video encryption, then they are disposed. And you can verify users using a short authentication string mechanism, which is by comparing some numbers or some emoji. So this is as opposed to a big long hex string. And the result of this is that now our user interface just has an on off switch instead of having to type a shared passphrase, which we think is more user friendly. Implementation wise, this is around 1000 lines of code. It's pretty self-contained, which is kind of a very good thing. It's not invasive because you do the encryption focused on this insertable string part, and that's most of it. You can find it in the libjitsimid project in the address up there. Now we already have this implemented and in production for months, but we still label it as experimental because we're still working on it. So we like to finish the short authentication string validation. It's almost there. Then as I said, we started with unmanaged, and then we transitioned to a managed key management scheme. But we want to actually have both because we think there is value in having these two approaches depending on how your product or how you're going to use this. So we plan to bring that back and make it configurable for your own deployment. Of course, we will continue collaborating with the ITF to make sure we are using the latest developments in the S-Rain working group and contributor feedback. We still think there is UI and user experience polish that could be applied here. We feel pretty proud of being the first video conferencing application in the open source domain to deploy this at scale. But we didn't do this alone. And there's some people and teams we'd like to thank for their helping us get where we are. Philip Hankey worked with us in doing the actual implementation of our user insertable streams, reporting bugs to Google, and pretty much it was instrumental to our success. Fx helped us, we met with them a number of times and chatted about LibOwl and how to use it properly for our use case. A really good collaboration there. Of course, Google for not only championing the insertable streams effort, but also reacting to feedback very quickly. And this improved significantly across Chrome versions. So it was also really nice to see how fast they reacted to... My last point, which is our community. The community wanted this. And when we launched it, we received tons of good feedback, which we felt really happy about. And yeah, it was a great effort to be able to bring this forward. But it warms our hearts to get this reward. That's all I got for you today. I hope this helped you understand how end-to-end encryption can be achieved in a web RTC environment and how you can take suggestions from our implementation and apply yourself. You can reach out to us via Twitter or our community. And I'll take a few questions now. If you're watching this not live, yeah, those are our contact details. Thank you.
 Hey everybody, thanks for joining.  Saul here from the GCC project.  Well, today I'm going to talk about our road or a way to end to an encryption, how we implemented  end to an encryption in GCC Meet, what we did, what tools we used, where we ended up  and how you can use it yourself in your project.  Let's jump right into it.  First let's do a quick review of the WebRTC security model to be able to frame this conversation  a bit better.  WebRTC restricts its availability in current browsers to safe origins, which means that  by definition you cannot have a WebRTC call where you're sending data over HTTP.  So at least we do have that secure signaling path established.  Also when it comes to the media, so the actual audio and video, DTLS, SRTP is mandatory.  So WebRTC is encrypted by design.  The typical use case is for two peers to find each other in some form of signaling server  and then exchange media between them that is as I said encrypted with DTLS, SRTP.  However, when it comes to scaling up, so creating conferences with tens, hundreds of people,  this model doesn't work.  So we switched to using something called selective forwarding unit.  And in a selective forwarding unit, the peer connections are established between each participant  and the server.  And then the server will route each participant's video to all the other participants.  In this scenario, because the server is the terminator of these connections, it has access  to this media.  Now what does what kind of media processing do these kinds of servers do and why is access  to this necessary?  So selective forwarding units as the name implies, they selectively forward, which means they  decide what to forward, what layer, do I forward high definition for this user, middle definition,  low definition, you can only switch between layers at certain times, for example, when  a keyframe comes, so they need to be able to detect this information.  And this information is available in the packet header, for instance.  So that's why it was it's needed for these entities to access the media.  Now you may wonder, wait a second, so I thought that WebRTC was encrypted.  That's what everybody told me.  And the answer is a little bit sort of, it's kind of complicated.  Not very really.  So when the media is flowing peer to peer, as I mentioned in the previous example, then  yes, in that case, your your your media would be anti encrypted.  But the moment you introduce a selective forwarding unit, so that you can scale up and have multiple,  you know, tens, hundreds of participants, this does not work that way.  Because again, the connection is terminated on the server side.  And in addition, the browsers don't really provide any user experience for for validating  that there is no money in the middle, that the remote participant, the key that the remote  participant has is the same that you received, and so on.  And there's also no indication if the web application running the JavaScript code decides  to start sending the media elsewhere.  What do I mean by by these last two things?  This is an SDP that you may receive with all the information on how to establish a media  session.  And the highlighted part in bold there is the fingerprint.  You will need to compare this fingerprint.  You will need to read it out loud to the other participant to know that, you know, there  was what they have is what you received in terms of certificates.  And as you can imagine, it's not really user friendly.  And here's another example.  Here is a conference, a call that I'm having with myself to participants, as you can see  the IP address is local is a connection is peer to peer.  And that's all good and when encrypted.  But the moment we had a third participant to the conference, not much happens.  Suddenly our layout, in this case, because we're using GTSimmy changes to this.  So from the user's perspective, pretty much nothing has changed, except that if we now  look where our media is being sent, we're now sending it to the server.  So what happened here?  Well, we for three participants, we run media through the bridge as is pretty common with  these kinds of architectures, but the browser didn't tell you anything.  So you thought that that you were sending media peer to peer, but now you're sending  through a server and you don't know this happened.  And this is where into an encryption can help you for the purposes of these talk.  I'm going to kind of focus on one instance or one example of where into an encryption  helps, which is eliminate the trust that we put on the selective forwarding unit on the  servers.  But as a broader exploration of this topic, I really encourage you to take a look at the  Millivob's end to an encryption beyond buzzwords talk that he gave at CrankyGeek.  It's very elucidating.  So how did we go about it?  Well, it wasn't possible with existing tools, so new tools had to be invented.  And this is where insertable streams comes in.  It's the API that unlocked this possibility.  It's a new API that allows you for manipulating full frames, full audio and video frames.  The data that you're allowed to touch is actually the data is mangled before transport encryption.  So before is encrypted with DTLS, SRTP, you can do whatever you want to it.  You can encrypt it, you can change it, whatever.  You can use web crypto APIs for this process.  You can do this in a worker, so it's friendly.  You don't need to do expensive operations on the main thread.  And as of today, it's only available in the Chromium based browsers.  But hopefully these changes in the future.  The processing pipeline looks like this.  So after encoding the frames, you get access to them in the insertable streams API.  You can do pretty much whatever you want.  I'm going to focus on one specific implementation or use case of this API, which is mangled  the data in the form of encrypting it.  But you could do something else.  Now, if you look at this processing chain, you think, well, so if I encrypt the data  in the insertable stream and on the way out and decrypted on the way in, but then there's  also DTLS, are we encrypting twice?  Exactly.  So we're encrypting once and two and another time hop by hop.  So when it arrives at the server, it knows how to decrypt the hop by hop encryption,  but it doesn't know how to decrypt the end to end encryption because it doesn't have  access to the key.  How does this look code wise?  Well, when we create our peer connection, we need to pass this new parameter encoded  media streams.  And then for the sending part for each sender, we need to apply a transform, as you can see  there in the code sample.  And the transform takes the form of a simple function that takes two parameters, a chunk,  which is a full audio or video frame and a controller, which is a kind of a magical  object that you can call and key on it to say, hey, this frame is ready.  So send it to the next element in the chain.  If we look at the opposite direction, so decrypting, we would need to attach a handler for all the  remote tracks.  When we receive a remote track, we will want to attach a transform to it.  And the transform will decrypt in this case.  As you can see right there, the decrypt function looks very similar to encrypt one.  It takes the chunk and the controller with the same meaning.  In this case, it would be an encrypted chunk.  And you can after you've decrypted it, you could enqueue it in the controller.  Now you probably heard this sentence very often in general.  I don't roll your own crypto.  So of course, you have all the web crypto APIs available to do whatever you want.  But there is this upcoming standard called S frame or secure frames that actually specifies  how to use this in this specific scenario.  So for implementing end to end encryption and authentication in a web conferencing,  in a multi-party video conferencing scenario, it's still early stages, but it is what we  use for implementation.  There is an ITF working group already formed in Apple is experimenting with the idea of  providing this S frame natively in the browser, which would be pretty interesting actually.  So with S frame, you get like a payload or a layout definition, how you put the bytes  on the wire after you have done your encryption.  But of course, it leaves one of the very important aspects out of the specification, which is  key management.  Like it will tell you what to do with multiple keys, but it won't tell you where you get  those keys from and how to secure the exchange and whatnot.  And for that, we can use things like MLS or signal or leave all as we did or something  else.  So what did we do?  Well, for encryption, we use what S frame mandates or suggest at the moment, which is  ASCTR with 256 keys and a truncated hashes.  For signing, we couldn't use what S frame suggests because those curves are not available  in the browser.  So we use ECDSA with that curve over up there.  So we call this slide variation.  We have some other slide variations that we will discuss with the S frame working group.  We call this J frame.  And all of our encryption happens in a worker, which means it's actually and we, for example,  we don't minify this worker, so it's easy to inspect when you are in a video call.  The net result of this is that when the call is unencrypted, you get the video just fine,  like the bunny right there.  But when it is and when encrypted, you get that garbage result over there.  I lead you into a secret.  The reason why you see that garbage is because we leave just a couple of bytes unencrypted  to trick the decoder into thinking this is a valid video frame.  And then you get this beautiful effect.  We'll probably remove this in the future.  There's nothing in those bytes, but there's also no reason why you wouldn't encrypt it  other than to get like, you know, a beautiful effect, I guess.  Now, as I said, important missing piece, key management.  Where do keys come from?  What do we do with them?  We have implemented two modes over time with transitions from one to another.  And I'm going to talk about those two.  So first is what we call unmanaged shared passphrase based key management.  So which means there is no management at all.  Users need to type a key that they have agreed upon out of ban.  However they want it, they agree that they're going to use this key.  If everybody types the same key, then boom, you get video.  If you don't type the right key, you get garbled audio and video.  We derive a key from this passphrase using PBKDF2 and the key never leaves the user machine,  of course.  So this is the first iteration that we did on this.  But then we wanted to transition to something a little bit more elaborate or that simplified  the usage of these passphrases.  Here's where we call it the Managed version.  Here we started using matrixes, leave all.  Leave all by the matrix project is an end to end encryption.  Well, it's the double ratchet implementation.  Pretty much like what the single protocol is.  And what we do is we establish an end to end encrypted channel of communication between  each two participants.  And we exchange randomly generated per participant keys over these channels.  We do automatic key rotation and ratcheting.  So if a participant leaves, we rotate the key so this participant is not able to decrypt  any future data with the keys that they had, even if they were somehow wiretapping the whole  thing.  And if a new participant joins, we ratchet the key forward so that this new participant  is not able to decrypt past data, but it is able to decrypt future data.  And this way we also save ourselves some operations there.  As I said, the keys are exchanged over these old channels, given as a secure way to exchange  them.  So if they are not used for the audio and video encryption, then they are disposed.  And you can verify users using a short authentication string mechanism, which is by comparing some  numbers or some emoji.  So this is as opposed to a big long hex string.  And the result of this is that now our user interface just has an on off switch instead  of having to type a shared passphrase, which we think is more user friendly.  Implementation wise, this is around 1000 lines of code.  It's pretty self-contained, which is kind of a very good thing.  It's not invasive because you do the encryption focused on this insertable string part, and  that's most of it.  You can find it in the libjitsimid project in the address up there.  Now we already have this implemented and in production for months, but we still label  it as experimental because we're still working on it.  So we like to finish the short authentication string validation.  It's almost there.  Then as I said, we started with unmanaged, and then we transitioned to a managed key  management scheme.  But we want to actually have both because we think there is value in having these two  approaches depending on how your product or how you're going to use this.  So we plan to bring that back and make it configurable for your own deployment.  Of course, we will continue collaborating with the ITF to make sure we are using the  latest developments in the S-Rain working group and contributor feedback.  We still think there is UI and user experience polish that could be applied here.  We feel pretty proud of being the first video conferencing application in the open source  domain to deploy this at scale.  But we didn't do this alone.  And there's some people and teams we'd like to thank for their helping us get where we  are.  Philip Hankey worked with us in doing the actual implementation of our user insertable streams,  reporting bugs to Google, and pretty much it was instrumental to our success.  Fx helped us, we met with them a number of times and chatted about LibOwl and how to  use it properly for our use case.  A really good collaboration there.  Of course, Google for not only championing the insertable streams effort, but also reacting  to feedback very quickly.  And this improved significantly across Chrome versions.  So it was also really nice to see how fast they reacted to...  My last point, which is our community.  The community wanted this.  And when we launched it, we received tons of good feedback, which we felt really happy  about.  And yeah, it was a great effort to be able to bring this forward.  But it warms our hearts to get this reward.  That's all I got for you today.  I hope this helped you understand how end-to-end encryption can be achieved in a web RTC environment  and how you can take suggestions from our implementation and apply yourself.  You can reach out to us via Twitter or our community.  And I'll take a few questions now.  If you're watching this not live, yeah, those are our contact details.  Thank you.
e2ee_0_179.png e2ee_180_263.png e2ee_264_443.png e2ee_444_611.png e2ee_612_731.png e2ee_732_815.png e2ee_816_1019.png e2ee_1020_3251.png e2ee_3252_4343.png e2ee_4344_6755.png e2ee_6756_7103.png e2ee_7104_7367.png e2ee_7368_7439.png e2ee_7440_8255.png e2ee_8256_9155.png e2ee_9156_10451.png e2ee_10452_13523.png e2ee_13524_15815.png e2ee_15816_17027.png e2ee_17028_18011.png e2ee_18012_23003.png e2ee_23004_23291.png e2ee_23292_24527.png e2ee_24528_26807.png e2ee_26808_27897.png