Technology of the Characterisation Virtual Laboratory (C-DEVL project) #2 - July 18
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Wilson, Lance
06.07.18 - Lance Wilson Characterisation DEVL (Data enhanced Virtual Lab). In June & July's TechTalk events, representatives from each of the DEVLs will introduce their projects from a developers' perspective: the problem the project is trying to solve, the tech stacks deployed and to be developed, the approaches of their software development and community engagement while developing tools and applications. The demonstration/discussion of 8 projects are scheduled as follows: DEVL #1 June (1st): Astronomy (Robert Shen), Marine Science (Roger Proctor), Terrestrial Ecology (Gerhard Weis), and Climate (Clare Richards). DEVL #2 July (6th), we welcome speakers from: GeoScience (Carsten Friedrich), Characterisation (Lance Wilson), BioScience (Jeff Christiansen), and Culture and Communities (Sarah Nisbet, Alexis Tindall).
 So, there goes our seatbelt. So there's a few things that we'd like to present today, but more than anything else, this is going to be a technical presentation on the technology that we're using to support Federation and also the technology we're using to support desktops and identity. But before we do that, we just want to define the characterization of virtual laboratory for those who don't have necessarily the background. Fundamentally, it is a program of work with the characterization virtual laboratory, which is a program of work to connect Australian characterization instruments with data management environments, tools and announced pipelines on Australian research. The achievements of the program today are that we've connected over 100 Australian instruments, which are funded by Encriss, of a significant value to Australian researchers, both financial and in the data they've produced. And we've had over 2,700 users of IPHCVL or the software that we've developed under the CDL, because a lot of the software that we've developed is reused across HPC facilities in Australian international. From the researcher's point of view, it's really just a place where a researcher can access their characterization data and many of the softwares and tools that they might need to process and analyse them. The C-Devil itself is doing a number of things. It's connecting instruments, it's continuing to create a rich online environment, and it's undertaking a training and fair data program as well. The part that we're going to talk about specifically today is the rich online environment, and it's really just one segment of what we do, but we can't necessarily present the entire technology step of all the program. And one of the things that we're focused on is the federation of the characterization VL across multiple nodes. So the characterization VL desktop is the desktop environment that we provide to our research user community. Largely, it's been run across two nodes, an instantonetic cloud and an instantonetic massive, and we're now federating that out so that it's run across a significantly larger cohort of nodes. And so the nodes that we're developing and deploying under this sort of prototype fashion at UWA, on the Porsi Research Cloud, at UQ as well, and a prototype in Sydney of a Windows CDL instance, and also deploying some software at University of Wollongong, which is part of the CDL as well. It's important to actually understand that many of the instances here are actually quite different, and different partners in the project require different things. So we have gone through a bit of a process to try to work out what federation and what CDL nodes across Australia might look like. As an example, UQ wants to integrate with an existing partner from the computer, whereas UWA, the instance there, will be an instance very similar to what we run at Monash, and it will run basically from source. So they're very different instances. But what we have done is we've developed a set of federation principles that are consistent across the entire network and work of CDL nodes, or that will be consistent across the entire network. And basically this is what it means to be a CDL node. It's a single user portal, a single sign-up and registration mechanism through AEF, a single help desk, a single shared software step in as much as possible, considering the technical and licensing challenges, and a single desktop user interface and UI branding menus, background software module. In a sense, what we're saying is a consistent user experience across all of the deployments that we're undertaking. There's a number of things that aren't in scope just yet, such as the ability to capture data at a number of instruments that's inconsistent, and it probably should be because all instruments are different. And so there's different technology stacks that we're deploying, and it's largely these two architectures here. One is to deploy from source on OpenStack using our, really, our scripts, our Ansible scripts, that we use to deploy cluster and Crystal talk about that. And the other one is to integrate with an existing high-performance computing system and basically provide a new layer over the top, which makes that system consistent with the CDL model. I won't go into this in too much detail because we don't have a lot of time, but those are the two models, the two technical models that they're deploying from there over to Crystal. All right. So please stop me if I get too excited. I'm actually quite pleased with the technology stack we've developed for doing all this stuff, because for me it's actually really simple, basic bits of technology that we've hung together to make something really useful. So as I'm presenting this, you might find I'm a little bit disdainful as not much to see here, but on the other hand, it's actually a really cool way of doing things together. Now, as Woytek mentioned, part of what we've got is a technology stack that's deploying what amounts to a cluster. And it might not be a really high-performance cluster, but it's a cluster model. You have a head node, you have a set of batch nodes that you can share with or you have a shared file system. And we've placed all of this around OpenStack because it was free and connected in one job of putting it in front of us. Ansible and good. Now, I'm going to talk about that first and a little bit after that. I'll talk about the desktop delivery technology stuff that actually faces the user. Now, Ansible, if you don't know, it's a configuration management tool, much like Puppet and Chef, but it's in Python. You define your configuration and you let Ansible get it there. So you define the end state you want to be in. We end up using this for almost everything. You can see the advances provided as a lovely example here, probing all the nodes in a class to find out the uptime. So it actually replaces tools like PDSH as well if you wanted to. And it's done as bricks. It's basically wrapping SSH. Anything you can do with SSH, you can do with Ansible, but it lays on a bit of Python good. It means you can write out, write and write in shell script with SSH, write a YAML file, define what you want. It's got loads of support around the world, so you can hang together ideas from other people. Now, Solve would have been just as good. Honestly, there's nothing special about Ansible. Move along. If you want to learn Ansible, I think that's great. And I think it's one of those things that you learn by doing the last one. The other really important part is Git. Now, we've got a slightly weird Git setup with our definition of a cluster because we've actually found there's loads of bits that are common to every cluster I've ever touched, because I think they're a good idea. And I think most people are familiar with these. Most people know that if you're going to run a cluster, you're probably going to put an LDAP on it. If you're going to run a cluster, you're probably going to make an NSFine to connect to user local to get your software off. So we've tried to separate all of our Ansible components into stuff which can get reused and stuff which is the discrete variables which vary between the clusters. And most importantly, passwords. As a result, we've got effectively two Git repositories, one of which we're completely open and anyone can go and have a look at. The other which has all my passwords in it and we would be horrified if they'd ever got out to anybody else. And we actually have a different password based repository, a different repository each of the clusters we run. Now, we do use Git sub modules and this gets confusing as heck for most people. Because Git defines this as a mechanism for bringing in dependencies. So it's actually really useful to say we've got this set of roles that are common things to do on all of our clusters. And each cluster depends on that. And it allows you to do this sort of QA where the upstream roles, the most recent version might change but you might want a few A-values being ready for your cluster. But for anybody who's used Git, they know that Git's confusing enough on its own and now I'm telling you that you've got a second Git commit in there and you have to make sure which one you're committing and update and to keep everything in set. So it takes a little bit of a learning curve even for highly technical people to get over it. But it is the exact solution that you want for this scenario where you've got a shared piece of code and a non-shared piece of code. A really important component is also how we make changes. We follow a very standard software developer principle in that one person is going to actually include the work and submit a request and another person is going to review it. At that point we base where a little bit really easy in that anyone can go ahead and deploy it. We don't have a big CI button for our clusters because it turns out it's really hard to figure out what on Earth a unit test is for a compute service. But at that point we can go ahead and break it. And we have one more. If you make a mistake that the user's noticed, you bring in a cake to apologise for your mistake. And I have to have a little bit of documentation up there at isocape.reba docs. I encourage everyone to support this as a principle and self-improvement. It really leads to trying hard not to break things when you know the penalty is apologising to everyone by making. So that's how we build our clusters really is. The other really interesting bit of, in my opinion of what we do, is this trick of federating across Linux-based POSIX systems. It's really long it's been around the traps for a few years. There's been multiple attempts within the Australian community to say, hey, let's all have one Linux user name. Let's have one password even. So all of our Raja accounts are the same as our Pausier accounts and the same as our Massive accounts. You can log in anywhere and you don't have to remember a part of passwords. This is really nice. It would be lovely to be able to authenticate once and access more clusters or even just have the identical authentication keys. But it turns out nobody wants to really sign up on this. There's a lot of technical work and political work necessary to be able to make that happen. So we actually decided to work around it. Now we have gone, I've got the point here that our clusters, we try to give them identical environments to try to make sure our software is in store everywhere. And one of the things we're leveraging at that for that moment is Stingularity containers. We're finding this is a really awesome technology for effectively non-root privileges on the same things which Docker does. The nuts and bolts end for our Federation. When you access a desktop through the web browser, what happens is the application, which is all actually mispasses Java and AngularJS, the application goes and asks for access to a cluster. You've been logged in by AAF and because all our clusters do have different usernames, we've configured so the AAF service has to look up your username for a given cluster. You then in your web browser still provide an OAuth 2 permission to allow the screwed up web web service to access the cluster on your behalf. And that web service never sees your password for the cluster and only gets a short-lived credential which is going to expire after a while. This is my way of saying, really, you can trust this guy to have him told she can't spell it and I'm going to lose access in an hour or so with the credentials buyers anyway. This is really one of the ways we're trying to get buy-in from everyone for a really low barrier to entry from the security perspective to say, look, it's a secure service. You don't have to do a full year-long order to figure out this is going to be okay. We really felt that was important for getting buy-in from building up the Federation is to allow each cluster to keep control of their username and group. And to guarantee that just because you signed up to the CDL and we've perhaps given you resources on M3 to be able to do some basic CDL work, you don't necessarily have resources to do work in any other part of the country on any other cluster. That's up to you talking to those cluster administrators to ask for resources. Now, in the future, we might get enough political buy-in to be able to say, yes, you signed up to CDL. You are clearly an Australian characterization researcher. But keep your gaps used anyway. And our next generation Federation, I'm actually thinking about developing this right now. We're actually only to have our web application asked for access to all of the clusters at once and we'll hold around and see which clusters you're able to access. The result of this is hopefully in our new web UI, we'll have a little option to copy between Massive and CDL at EWA or copy between Massive and Rajan. There won't be any extra passwords to M3 to. We've already got a short list of credentials. We'll just make that happen for you. So that's our technology staff. Thank you.
 So, there goes our seatbelt.  So there's a few things that we'd like to present today, but more than anything else,  this is going to be a technical presentation on the technology that we're using to support  Federation and also the technology we're using to support desktops and identity.  But before we do that, we just want to define the characterization of virtual laboratory  for those who don't have necessarily the background.  Fundamentally, it is a program of work with the characterization virtual laboratory,  which is a program of work to connect Australian characterization instruments with data management environments,  tools and announced pipelines on Australian research.  The achievements of the program today are that we've connected over 100 Australian instruments,  which are funded by Encriss, of a significant value to Australian researchers,  both financial and in the data they've produced.  And we've had over 2,700 users of IPHCVL or the software that we've developed under the CDL,  because a lot of the software that we've developed is reused across HPC facilities in Australian international.  From the researcher's point of view, it's really just a place where a researcher can access their characterization data  and many of the softwares and tools that they might need to process and analyse them.  The C-Devil itself is doing a number of things.  It's connecting instruments, it's continuing to create a rich online environment,  and it's undertaking a training and fair data program as well.  The part that we're going to talk about specifically today is the rich online environment,  and it's really just one segment of what we do, but we can't necessarily present the entire technology step of all the program.  And one of the things that we're focused on is the federation of the characterization VL across multiple nodes.  So the characterization VL desktop is the desktop environment that we provide to our research user community.  Largely, it's been run across two nodes, an instantonetic cloud and an instantonetic massive,  and we're now federating that out so that it's run across a significantly larger cohort of nodes.  And so the nodes that we're developing and deploying under this sort of prototype fashion at UWA,  on the Porsi Research Cloud, at UQ as well, and a prototype in Sydney of a Windows CDL instance,  and also deploying some software at University of Wollongong, which is part of the CDL as well.  It's important to actually understand that many of the instances here are actually quite different,  and different partners in the project require different things.  So we have gone through a bit of a process to try to work out what federation and what CDL nodes across Australia might look like.  As an example, UQ wants to integrate with an existing partner from the computer, whereas UWA, the instance there,  will be an instance very similar to what we run at Monash, and it will run basically from source.  So they're very different instances.  But what we have done is we've developed a set of federation principles that are consistent across the entire network  and work of CDL nodes, or that will be consistent across the entire network.  And basically this is what it means to be a CDL node.  It's a single user portal, a single sign-up and registration mechanism through AEF, a single help desk,  a single shared software step in as much as possible,  considering the technical and licensing challenges,  and a single desktop user interface and UI branding menus, background software module.  In a sense, what we're saying is a consistent user experience across all of the deployments that we're undertaking.  There's a number of things that aren't in scope just yet, such as the ability to capture data at a number of instruments  that's inconsistent, and it probably should be because all instruments are different.  And so there's different technology stacks that we're deploying, and it's largely these two architectures here.  One is to deploy from source on OpenStack using our, really, our scripts, our Ansible scripts,  that we use to deploy cluster and Crystal talk about that.  And the other one is to integrate with an existing high-performance computing system  and basically provide a new layer over the top, which makes that system consistent with the CDL model.  I won't go into this in too much detail because we don't have a lot of time,  but those are the two models, the two technical models that they're deploying from there over to Crystal.  All right. So please stop me if I get too excited.  I'm actually quite pleased with the technology stack we've developed for doing all this stuff,  because for me it's actually really simple, basic bits of technology that we've hung together to make something really useful.  So as I'm presenting this, you might find I'm a little bit disdainful as not much to see here,  but on the other hand, it's actually a really cool way of doing things together.  Now, as Woytek mentioned, part of what we've got is a technology stack that's deploying what amounts to a cluster.  And it might not be a really high-performance cluster, but it's a cluster model.  You have a head node, you have a set of batch nodes that you can share with or you have a shared file system.  And we've placed all of this around OpenStack because it was free and connected in one job of putting it in front of us.  Ansible and good.  Now, I'm going to talk about that first and a little bit after that. I'll talk about the desktop delivery technology stuff that actually faces the user.  Now, Ansible, if you don't know, it's a configuration management tool, much like Puppet and Chef, but it's in Python.  You define your configuration and you let Ansible get it there.  So you define the end state you want to be in.  We end up using this for almost everything. You can see the advances provided as a lovely example here, probing all the nodes in a class to find out the uptime.  So it actually replaces tools like PDSH as well if you wanted to.  And it's done as bricks. It's basically wrapping SSH.  Anything you can do with SSH, you can do with Ansible, but it lays on a bit of Python good.  It means you can write out, write and write in shell script with SSH, write a YAML file, define what you want.  It's got loads of support around the world, so you can hang together ideas from other people.  Now, Solve would have been just as good. Honestly, there's nothing special about Ansible.  Move along. If you want to learn Ansible, I think that's great.  And I think it's one of those things that you learn by doing the last one.  The other really important part is Git.  Now, we've got a slightly weird Git setup with our definition of a cluster because we've actually found there's loads of bits that are common to every cluster I've ever touched,  because I think they're a good idea. And I think most people are familiar with these.  Most people know that if you're going to run a cluster, you're probably going to put an LDAP on it.  If you're going to run a cluster, you're probably going to make an NSFine to connect to user local to get your software off.  So we've tried to separate all of our Ansible components into stuff which can get reused and stuff which is the discrete variables which vary between the clusters.  And most importantly, passwords. As a result, we've got effectively two Git repositories, one of which we're completely open and anyone can go and have a look at.  The other which has all my passwords in it and we would be horrified if they'd ever got out to anybody else.  And we actually have a different password based repository, a different repository each of the clusters we run.  Now, we do use Git sub modules and this gets confusing as heck for most people.  Because Git defines this as a mechanism for bringing in dependencies. So it's actually really useful to say we've got this set of roles that are common things to do on all of our clusters.  And each cluster depends on that. And it allows you to do this sort of QA where the upstream roles, the most recent version might change but you might want a few A-values being ready for your cluster.  But for anybody who's used Git, they know that Git's confusing enough on its own and now I'm telling you that you've got a second Git commit in there and you have to make sure which one you're committing and update and to keep everything in set.  So it takes a little bit of a learning curve even for highly technical people to get over it.  But it is the exact solution that you want for this scenario where you've got a shared piece of code and a non-shared piece of code.  A really important component is also how we make changes. We follow a very standard software developer principle in that one person is going to actually include the work and submit a request and another person is going to review it.  At that point we base where a little bit really easy in that anyone can go ahead and deploy it. We don't have a big CI button for our clusters because it turns out it's really hard to figure out what on Earth a unit test is for a compute service.  But at that point we can go ahead and break it. And we have one more. If you make a mistake that the user's noticed, you bring in a cake to apologise for your mistake.  And I have to have a little bit of documentation up there at isocape.reba docs.  I encourage everyone to support this as a principle and self-improvement. It really leads to trying hard not to break things when you know the penalty is apologising to everyone by making.  So that's how we build our clusters really is. The other really interesting bit of, in my opinion of what we do, is this trick of federating across Linux-based POSIX systems.  It's really long it's been around the traps for a few years. There's been multiple attempts within the Australian community to say, hey, let's all have one Linux user name. Let's have one password even. So all of our Raja accounts are the same as our Pausier accounts and the same as our Massive accounts.  You can log in anywhere and you don't have to remember a part of passwords. This is really nice. It would be lovely to be able to authenticate once and access more clusters or even just have the identical authentication keys.  But it turns out nobody wants to really sign up on this. There's a lot of technical work and political work necessary to be able to make that happen. So we actually decided to work around it.  Now we have gone, I've got the point here that our clusters, we try to give them identical environments to try to make sure our software is in store everywhere.  And one of the things we're leveraging at that for that moment is Stingularity containers. We're finding this is a really awesome technology for effectively non-root privileges on the same things which Docker does.  The nuts and bolts end for our Federation. When you access a desktop through the web browser, what happens is the application, which is all actually mispasses Java and AngularJS, the application goes and asks for access to a cluster.  You've been logged in by AAF and because all our clusters do have different usernames, we've configured so the AAF service has to look up your username for a given cluster.  You then in your web browser still provide an OAuth 2 permission to allow the screwed up web web service to access the cluster on your behalf. And that web service never sees your password for the cluster and only gets a short-lived credential which is going to expire after a while.  This is my way of saying, really, you can trust this guy to have him told she can't spell it and I'm going to lose access in an hour or so with the credentials buyers anyway. This is really one of the ways we're trying to get buy-in from everyone for a really low barrier to entry from the security perspective to say,  look, it's a secure service. You don't have to do a full year-long order to figure out this is going to be okay. We really felt that was important for getting buy-in from building up the Federation is to allow each cluster to keep control of their username and group.  And to guarantee that just because you signed up to the CDL and we've perhaps given you resources on M3 to be able to do some basic CDL work, you don't necessarily have resources to do work in any other part of the country on any other cluster.  That's up to you talking to those cluster administrators to ask for resources. Now, in the future, we might get enough political buy-in to be able to say, yes, you signed up to CDL. You are clearly an Australian characterization researcher.  But keep your gaps used anyway. And our next generation Federation, I'm actually thinking about developing this right now. We're actually only to have our web application asked for access to all of the clusters at once and we'll hold around and see which clusters you're able to access.  The result of this is hopefully in our new web UI, we'll have a little option to copy between Massive and CDL at EWA or copy between Massive and Rajan. There won't be any extra passwords to M3 to. We've already got a short list of credentials. We'll just make that happen for you.  So that's our technology staff.  Thank you.
Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_0_83.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_84_167.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_168_215.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_216_707.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_708_755.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_756_1943.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_1944_2399.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_2400_3887.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_3888_5579.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_5580_7235.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_7236_8339.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_8340_9899.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_9900_10115.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_10116_10247.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_10248_11231.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_11232_11507.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_11508_13211.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_13212_14591.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_14592_16127.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_16128_18107.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_18108_20003.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_20004_21191.png Characterisation_DEVL_-_Tech_Talk_-_July_18_cbLy8uCTbZ0_21192_22161.png