gRPC, Protobufs and Go... OH MY!
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Griffith, John
gRPC is an open source remote procedure call framework built on HTTP/2 that provides an easy way to create high performance, scalable APIs in a multitude of programming languages. In this talk we’ll describe what grPC is, how we it came to be, and compare it to some other alternatives. We’ll talk about the pros and cons and hopefully convince you to take a closer look at it and add it to your toolbox. Best of all we’ll walk through an example of how to build a gRPC based application using gRPC, Protobufs and Go! We’ll even take a look at how easy it is to add a client using another programming language.
 My talk is GRPC, Protobufs and Go, oh my. So how many of you have already used GRPC and Protobufs? Okay, so a bunch of you. This might be a review for you. This is kind of like an intro, a sort of getting started and so we're going to go through. I'm John Griffith. There's my Twitter, there's my GitHub. Reach out to me after this if you guys want to chat, if anybody has any questions or wants to tell me I did something wrong or whatever or just wants to meet me and talk. I work at Red Hat and I'm mostly around the Kubernetes space these days. So RPC basically we're just talking about machines, talking to machines. The whole GRPC thing and everything, it's not necessarily a groundbreaking new idea. This technique and these kind of ideas have been around for a long time. I'm old enough to have worked with most of these unfortunately. So GRPC is kind of the newest take on this idea. Simple idea is basically you have some clients that you want to be able to actually affect actions on a remote server. Usually that transport that you're going to use is TCP. One of the nice things about GRPC is that it's going to use HTTP2 specifically which gives us a lot of flexibility and a lot of capabilities and much better performance and efficiency. So my talk really is actually just to tell you that GRPC is really the only option out of all the different things that are out there and it's the only thing that you should use. So that's really why I'm here. Okay, not really. Actually on the opposite, there's a lot of really good options out there. There's a lot of really good technologies, a lot of different things you can do. But GRPC is actually really cool. It's got a ton of super, super good features, things like that. So what I always tell people is just please always investigate new things and look at other options out there when you're doing your research. Don't just listen to what somebody said at FOSM or anywhere else, right, or what they wrote on their blog. Check it out for yourself and see what you think. So some of the things that I think are really nice about GRPC is you get pretty much a complete solution out of the box with GRPC. So I lump Protobufs into GRPC because they're hidden when you're actually trying to do an implementation or build this. So these Protobufs, it's got some really cool advanced features that we might get into today, hopefully, like streaming. So you can even do bi-directional streaming, so you can have a client in the server streaming back and forth open a channel, blocking and unblocking options, flow control, all the sorts of things you would expect. And one of the really cool things is the ability to build in with Protocy a reverse proxy for REST. So if you're hung up on REST and you need a REST compliant API, you can actually still do that. You can actually use things like Protocy to build that. So that's really cool, in my opinion. But the main thing, the reason I love GRPC and I started using it a while back is because it's super easy and I'm lazy. So basically building a service with GRPC distills down to just three steps. So the first thing we're going to do is we're going to create a interface definition. So that's our protofile. And that's going to just be saying what it is that we're going to serialize back and forth across the wire. The second step is to actually implement the server. And then the third step is to implement the client. Now I did leave one important thing out of here, right? You have to have an API of something that you're building to do something useful. So that's something you've probably already done. So my argument is that it's so easy that we could actually just build a GRPC service right here and now. And not just the usual ping, pong, you know, anything like that. So what we're going to do today is we're going to take this simple LVM package that I have, the simple LVM API. And we're going to build a GRPC service for it right here on the fly in this session. So it's going to be really interesting because this is not my laptop. And it has a UK keyboard. So you guys should be entertained if nothing else. All right. So what we're doing is, you know, you should always kind of have an idea of what you're trying to do. We're going to basically build our own sand device. So I'm doing a lot of stuff with Kubernetes and containers and all these different things in the storage space. And I don't want to go out and like buy a storage box from somebody or something like that because I left those startups and I don't do that anymore. So what I want to do is I want to take LVM and I want to put LVM on a server that I have with a bunch of disks and then use iSCSI and just do exports and just build my own sand, basically. I actually started to do this as a project for Kubernetes for iSCSI D-driver that's in there and found that all of the code that's there is actually deprecated and obsolete. So I'm rewriting it and this is what I'm doing. So, okay. So I've got this package. It's called GoLVM and we're going to start off really simple. It just creates logical volumes, lists logical volumes and deletes them. So everybody knows what LVM is, I'm assuming. All right. So step one, I'm going to walk through the steps again real quick and just give you a preview of what we're going to do. So we're going to define our interface. So that's going to be our proto file. And then we're going to implement a server real quick. And then we're going to implement an actual client and then we're going to deploy this and run it and check it out. So we'll start off and we'll just make this really simple. We'll just do like create volumes. So I'll try not to talk too much while I'm... Well, that's not going to work. Who am I kidding? Yeah, okay. I can talk really loud. So if I can find my mouse or my cursor. All right. So this is going to be good. Let's minimize this and get this over here. All right. So I've got a couple of Fedora boxes running on VMs that I'm running on this machine. So what I'm going to do is I'm going to go onto those and let me actually mirror this because, well, we'll just try it. Okay. So the first thing I need to do is I am in my go path. I'm in my get directory. So if you look PWD, you can see there's JGryphith. Can you guys see that or should I make it bigger? All right. So we'll do that. So I'm going to set up my directory structure real quick and the way I'm going to do that is I'm just going to say I'm going to create a directory for my proto files, a directory for my server and a directory for my client. Okay. We're quiet. Oh, see, and there's the UK keyboard right there. Oh, no. All right. Relax. Don't panic. There we go. We can recover. Okay, so let's go into the proto file. The proto file is actually, this is really kind of the nuts and bolts, right? This is the main thing. And what we're going to do inside of here is we're going to create this file and we're going to call it LVM.proto. And what this is going to do is this is going to define what we want our service to be, right? So let me see here. So we start off and we just say... Okay, so syntax. We're going to use proto3 as our syntax. So this just tells the protocol, the protocol compiler what protocol we're using in this definition, right? So this is what we're looking for. The next thing we need to do is we need to give this a name, right? So we use kind of a Go-ish syntax here and we say package and we're going to call it LVM. And if I'm typing things or spelling them wrong, holler at me because I can't see. So the first thing we want to define is this thing called a message, right? So a message is the actual thing that we're going to serialize and send over the wire, right? So anything that you want to serialize and send over the wire, it's basically called a message. So we're going to create a message and we're going to call it a volume. And... I hit caps lock instead of uppercase. So we're going to call it volume and we're going to give it a string called name. The next thing you'll see is this one. One is the parameter. So what you do is you order the parameters that you have. So it identifies that parameter inside of the message. That's what we have there. The next thing we want is our actual create volume. That's not even close. Hey, I could get used to this keyboard. Okay. Create volume request. So string, we're going to give it... To do a create, we're going to need a name and we're going to need... If you know anything about LVM, you're going to want a volume group. And in our case, we're going to use thin provisioning. So we're going to want a pool. We're going to want to specify a pool. And then we're going to specify a size. Numb, numb, numb, numb. All right. So we've got that piece. The next thing we need to do is we need to have a request. So we do a message create volume request. I'm sorry. Yeah. Holy cow. Off the rails. All right. So we're going to give this another minute and then if things don't get better and people keep leaving. Not cool by the way. Then what I'll do is I'll just go ahead and bring up my code and we can just kind of walk through it instead. Okay. So there we have a response. We're going to call it... It's just message. So the last thing we need to do is to find the actual service. So we'll do... Like that and then we want to do the same thing. I almost got it. That's not terrible. All right. So here's what we're going to do because we're going to run out of time. What I'm going to do instead is I'm going to pull the code because I already wrote it. And we're going to look at it. It's so easy to do if you have a non-UK keyboard when you're a stupid American who doesn't have his laptop, their laptop. OK. I'll just use magic. Voila. Right? Okay. If you look back, remember our prototype that we were building? Oh, stop that. And he fell apart. So that's the prototype that we were trying to build. Those are the last two lines I was trying to enter there. So the next step that we needed was we needed a server, right? So if you look here, I have a pretty simple server that I put together. Just standard go. We have a main.go here. And the last thing that we needed was we went ahead and we defined our function for create volume. So you can see there we take a context. All of our gRPC stuff is going to pass a context back and forth. And then there's that create volume request that we had. So what we did with that proto file is we actually ran the proto compiler, the proto C compiler, and we compiled a pb.go file. And that's what I'm referencing there, that pb.create volume request. Right? So that's where that's coming from. So you can see we have the create volume request. We have the create volume response. And then I just go ahead and the server is actually where LVM is installed and everything. So I'm just executing the LVM commands on that server when it gets this command. So that's how it's working. One of the things that I say that's super nice about gRPC and one of the reasons why I like it so much is in terms of setting up the server and get everything wired together, this is basically it. Unless you wanted to do some more advanced things, which you can. But this will get you a basic running server and everything will work. Right? So it's pretty simple. All right. So we've got that. And then we would need a client. And the client is even easier. We just put basically the calls into our client. We're going to just do this directly right now. We're going to go ahead and wire in our IP address for the server. And we're going to go ahead and just use this hard coded create volume request. Okay? So if we look here, so if we do a sudo LVS, we look. So right now you can see we've got the system partitions and we've got that thin pool for LVM thin provisioning. And that's it. So what I'm going to do right now is I'm going to kick off this server. Okay? So now the server, the gRPC services up and running and listening on this server. Right? So we can go over here. I have this other box called FETI 2. And what we're going to do over there is we're going to run the client. Oops. And I'm already there, I think. I am. Okay. So this is the same client. It's all the same code and everything else. I just copied the whole thing over here. So what we do is we do a client. We run that. Uh-oh. Did I not sudo that when I started it? No. Uh-uh. So very important. Okay. So we get our message as it was created successfully. So we'll go over here and we'll do... And you can see we have a new LV called test. And that's the demo that I was trying to code on the fly. I have a much greater appreciation for just for fun now. I'm sorry, what? Oh, yes. How's that? Is that better? Yep. So about that time, but real quick, anybody have any questions? All right. Thank you. Thank you.
 My talk is GRPC, Protobufs and Go, oh my.  So how many of you have already used GRPC and Protobufs?  Okay, so a bunch of you.  This might be a review for you.  This is kind of like an intro, a sort of getting started and so we're going to go through.  I'm John Griffith.  There's my Twitter, there's my GitHub.  Reach out to me after this if you guys want to chat, if anybody has any questions or wants  to tell me I did something wrong or whatever or just wants to meet me and talk.  I work at Red Hat and I'm mostly around the Kubernetes space these days.  So RPC basically we're just talking about machines, talking to machines.  The whole GRPC thing and everything, it's not necessarily a groundbreaking new idea.  This technique and these kind of ideas have been around for a long time.  I'm old enough to have worked with most of these unfortunately.  So GRPC is kind of the newest take on this idea.  Simple idea is basically you have some clients that you want to be able to actually affect  actions on a remote server.  Usually that transport that you're going to use is TCP.  One of the nice things about GRPC is that it's going to use HTTP2 specifically which  gives us a lot of flexibility and a lot of capabilities and much better performance and  efficiency.  So my talk really is actually just to tell you that GRPC is really the only option out  of all the different things that are out there and it's the only thing that you should use.  So that's really why I'm here.  Okay, not really.  Actually on the opposite, there's a lot of really good options out there.  There's a lot of really good technologies, a lot of different things you can do.  But GRPC is actually really cool.  It's got a ton of super, super good features, things like that.  So what I always tell people is just please always investigate new things and look at  other options out there when you're doing your research.  Don't just listen to what somebody said at FOSM or anywhere else, right, or what they  wrote on their blog.  Check it out for yourself and see what you think.  So some of the things that I think are really nice about GRPC is you get pretty much a complete  solution out of the box with GRPC.  So I lump Protobufs into GRPC because they're hidden when you're actually trying to do an  implementation or build this.  So these Protobufs, it's got some really cool advanced features that we might get into today,  hopefully, like streaming.  So you can even do bi-directional streaming, so you can have a client in the server streaming  back and forth open a channel, blocking and unblocking options, flow control, all the  sorts of things you would expect.  And one of the really cool things is the ability to build in with Protocy a reverse proxy for  REST.  So if you're hung up on REST and you need a REST compliant API, you can actually still  do that.  You can actually use things like Protocy to build that.  So that's really cool, in my opinion.  But the main thing, the reason I love GRPC and I started using it a while back is because  it's super easy and I'm lazy.  So basically building a service with GRPC distills down to just three steps.  So the first thing we're going to do is we're going to create a interface definition.  So that's our protofile.  And that's going to just be saying what it is that we're going to serialize back and  forth across the wire.  The second step is to actually implement the server.  And then the third step is to implement the client.  Now I did leave one important thing out of here, right?  You have to have an API of something that you're building to do something useful.  So that's something you've probably already done.  So my argument is that it's so easy that we could actually just build a GRPC service right  here and now.  And not just the usual ping, pong, you know, anything like that.  So what we're going to do today is we're going to take this simple LVM package that I have,  the simple LVM API.  And we're going to build a GRPC service for it right here on the fly in this session.  So it's going to be really interesting because this is not my laptop.  And it has a UK keyboard.  So you guys should be entertained if nothing else.  All right.  So what we're doing is, you know, you should always kind of have an idea of what you're  trying to do.  We're going to basically build our own sand device.  So I'm doing a lot of stuff with Kubernetes and containers and all these different things  in the storage space.  And I don't want to go out and like buy a storage box from somebody or something like  that because I left those startups and I don't do that anymore.  So what I want to do is I want to take LVM and I want to put LVM on a server that I have  with a bunch of disks and then use iSCSI and just do exports and just build my own sand,  basically.  I actually started to do this as a project for Kubernetes for iSCSI D-driver that's in  there and found that all of the code that's there is actually deprecated and obsolete.  So I'm rewriting it and this is what I'm doing.  So, okay.  So I've got this package.  It's called GoLVM and we're going to start off really simple.  It just creates logical volumes, lists logical volumes and deletes them.  So everybody knows what LVM is, I'm assuming.  All right.  So step one, I'm going to walk through the steps again real quick and just give you a  preview of what we're going to do.  So we're going to define our interface.  So that's going to be our proto file.  And then we're going to implement a server real quick.  And then we're going to implement an actual client and then we're going to deploy this  and run it and check it out.  So we'll start off and we'll just make this really simple.  We'll just do like create volumes.  So I'll try not to talk too much while I'm...  Well, that's not going to work.  Who am I kidding?  Yeah, okay.  I can talk really loud.  So if I can find my mouse or my cursor.  All right.  So this is going to be good.  Let's minimize this and get this over here.  All right.  So I've got a couple of Fedora boxes running on VMs that I'm running on this machine.  So what I'm going to do is I'm going to go onto those and let me actually mirror this  because, well, we'll just try it.  Okay.  So the first thing I need to do is I am in my go path.  I'm in my get directory.  So if you look PWD, you can see there's JGryphith.  Can you guys see that or should I make it bigger?  All right.  So we'll do that.  So I'm going to set up my directory structure real quick and the way I'm going to do that  is I'm just going to say I'm going to create a directory for my proto files, a directory  for my server and a directory for my client.  Okay.  We're quiet.  Oh, see, and there's the UK keyboard right there.  Oh, no.  All right.  Relax.  Don't panic.  There we go.  We can recover.  Okay, so let's go into the proto file.  The proto file is actually, this is really kind of the nuts and bolts, right?  This is the main thing.  And what we're going to do inside of here is we're going to create this file and we're  going to call it LVM.proto.  And what this is going to do is this is going to define what we want our service to be,  right?  So let me see here.  So we start off and we just say...  Okay, so syntax.  We're going to use proto3 as our syntax.  So this just tells the protocol, the protocol compiler what protocol we're using in this  definition, right?  So this is what we're looking for.  The next thing we need to do is we need to give this a name, right?  So we use kind of a Go-ish syntax here and we say package and we're going to call it  LVM.  And if I'm typing things or spelling them wrong, holler at me because I can't see.  So the first thing we want to define is this thing called a message, right?  So a message is the actual thing that we're going to serialize and send over the wire,  right?  So anything that you want to serialize and send over the wire, it's basically called  a message.  So we're going to create a message and we're going to call it a volume.  And...  I hit caps lock instead of uppercase.  So we're going to call it volume and we're going to give it a string called name.  The next thing you'll see is this one.  One is the parameter.  So what you do is you order the parameters that you have.  So it identifies that parameter inside of the message.  That's what we have there.  The next thing we want is our actual create volume.  That's not even close.  Hey, I could get used to this keyboard.  Okay.  Create volume request.  So string, we're going to give it...  To do a create, we're going to need a name and we're going to need...  If you know anything about LVM, you're going to want a volume group.  And in our case, we're going to use thin provisioning.  So we're going to want a pool.  We're going to want to specify a pool.  And then we're going to specify a size.  Numb, numb, numb, numb.  All right.  So we've got that piece.  The next thing we need to do is we need to have a request.  So we do a message create volume request.  I'm sorry.  Yeah.  Holy cow.  Off the rails.  All right.  So we're going to give this another minute and then if things don't get better and people  keep leaving.  Not cool by the way.  Then what I'll do is I'll just go ahead and bring up my code and we can just kind of walk  through it instead.  Okay.  So there we have a response.  We're going to call it...  It's just message.  So the last thing we need to do is to find the actual service.  So we'll do...  Like that and then we want to do the same thing.  I almost got it.  That's not terrible.  All right.  So here's what we're going to do because we're going to run out of time.  What I'm going to do instead is I'm going to pull the code because I already wrote it.  And we're going to look at it.  It's so easy to do if you have a non-UK keyboard when you're a stupid American who doesn't  have his laptop, their laptop.  OK.  I'll just use magic.  Voila.  Right?  Okay.  If you look back, remember our prototype that we were building?  Oh, stop that.  And he fell apart.  So that's the prototype that we were trying to build.  Those are the last two lines I was trying to enter there.  So the next step that we needed was we needed a server, right?  So if you look here, I have a pretty simple server that I put together.  Just standard go.  We have a main.go here.  And the last thing that we needed was we went ahead and we defined our function for create volume.  So you can see there we take a context.  All of our gRPC stuff is going to pass a context back and forth.  And then there's that create volume request that we had.  So what we did with that proto file is we actually ran the proto compiler, the proto C compiler,  and we compiled a pb.go file.  And that's what I'm referencing there, that pb.create volume request.  Right?  So that's where that's coming from.  So you can see we have the create volume request.  We have the create volume response.  And then I just go ahead and the server is actually where LVM is installed and everything.  So I'm just executing the LVM commands on that server when it gets this command.  So that's how it's working.  One of the things that I say that's super nice about gRPC and one of the reasons why I like it so much  is in terms of setting up the server and get everything wired together, this is basically it.  Unless you wanted to do some more advanced things, which you can.  But this will get you a basic running server and everything will work.  Right?  So it's pretty simple.  All right.  So we've got that.  And then we would need a client.  And the client is even easier.  We just put basically the calls into our client.  We're going to just do this directly right now.  We're going to go ahead and wire in our IP address for the server.  And we're going to go ahead and just use this hard coded create volume request.  Okay?  So if we look here, so if we do a sudo LVS, we look.  So right now you can see we've got the system partitions and we've got that thin pool for LVM thin provisioning.  And that's it.  So what I'm going to do right now is I'm going to kick off this server.  Okay?  So now the server, the gRPC services up and running and listening on this server.  Right?  So we can go over here.  I have this other box called FETI 2.  And what we're going to do over there is we're going to run the client.  Oops.  And I'm already there, I think.  I am.  Okay.  So this is the same client.  It's all the same code and everything else.  I just copied the whole thing over here.  So what we do is we do a client.  We run that.  Uh-oh.  Did I not sudo that when I started it?  No.  Uh-uh.  So very important.  Okay.  So we get our message as it was created successfully.  So we'll go over here and we'll do...  And you can see we have a new LV called test.  And that's the demo that I was trying to code on the fly.  I have a much greater appreciation for just for fun now.  I'm sorry, what?  Oh, yes.  How's that?  Is that better?  Yep.  So about that time, but real quick, anybody have any questions?  All right.  Thank you.  Thank you.
gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_0_179.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_180_8423.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_8424_8711.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_8712_9791.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_9792_9935.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_9936_10103.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_10104_10223.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_10224_10811.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_10812_10871.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_10872_13955.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_13956_15887.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_15888_16271.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_16272_16403.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_16404_16775.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_16776_16895.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_16896_19079.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_19080_19847.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_19848_20483.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_20484_20999.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_21000_23423.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_23424_24323.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_24324_24707.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_24708_25955.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_25956_26699.png gRPC__Protobufs_and_Go____OH_MY__An_introduction_to_building_client_server_systems_with_gRPC_2d9qCGD_26700_27364.png