Handshakes, Citizen Science and Evolution
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Knight, Vince
In the 1980s Robert Axelrod invited submissions for a computer tournament in which people submitted strategies for the Iterated Prisoner's dilemma. This is a game in which individuals choose to either cooperate or defect with each other. Since then similar work has been used to understand the evolution of cooperative behaviour in an evolutionary setting. Like a large number of early (and sadly ongoing) research code, the code from Robert Axelrod's work was lost. Similar ongoing work is often done with poor sustainable practice for the software involved. In 2015, a Python library aiming to reproduce Axelrod's work was put on github (the Axelrod library). It has since accumulated more than 200 strategies with contributions from Academics and hobbyists alike. This vast OPEN treasure trove of game theoretic tools is now being used to undertake a number of research projects. Including one that looks at an evolutionary process called a Moran Process which is a model of a population in which the makeup of future generations of the population depend on how well individuals of the current generation perform. In 2012 a piece of research claimed that there was no advantage to having long memory of interactions. The work this talk will describe demonstrates how that's not true in evolutionary dynamics. Indeed: complex strategies have been trained using reinforcement learning and the huge number of strategies available through the Axelrod library to perform particularly well. Interesting behavioural aspects also emerge: without external input, the strong strategies evolve "handshakes". These handshakes allow them to recognise friend or foe in the population and act accordingly. This work not only has implications at a game theoretic and reinforcement learning level but can also help understand how and why complex behaviour can emerge in evolutionary settings.
 Okay. Welcome back everyone. We're going to be starting in a minute or so as people are trickling back in from the coffee break. So Vince kindly agreed to swap slots with Valerio, but otherwise the program will proceed as announced. And he's going to be talking about the game theory project. Similar to Nicoleta's project. Yes. Right, we worked together. So welcome Vince and thanks for helping out the organizers. No problem. Thanks for having me. I know how hard organizing these things can be and how much work it is. So thank you for putting on this great event. My name's Vince. I've already tweeted out links to this talk as well as links to the pre-print that I'm actually going to be talking about. So this talk is going to be kind of following this archive pre-print here and explaining some of the ideas and how we've used Python to do that. I'm a mathematician at Cardiff University in Wales in the UK, still in Europe. I'm also a software sustainable institute fellow that's been mentioned a little bit. So best practice in terms of software as used in research is important to me. And finally, I'm a game theorist. I'm one of the core developers for the Axelrod Library, which is Nicoleta's poster is about it. And that's what this logo is about. And the work I'm going to talk to you about today is not just mine. There's five authors on the paper, four authors on the paper, pardon me. And also a huge number of contributors to the Axelrod Library itself. And so that's why the title of this talk, if you've got the program in front of you, has the word citizen science in it, because I really feel it's a nice scientific effort. So this is a tweet. I don't know how well you can see it at the back. This is a tweet I saw a little while ago by a postdoc from Penn State called Kirstie McCloyde. And she tweets, sets up flawless heat competition trial. Lizards will fight over hot podium. There can only be one winner. So the idea is, I understand by my naive understanding, is there's this pen, there's two lizards in the pen, and there's one podium that's warm in the pen. And lizards are cold-blooded, so they like the warm podium. And so she's going to be able to look at how they fight over this podium. All right? Perhaps, and here I'm extrapolating from a tweet, perhaps to study over a longitudinal time if it's always the same lizard that wins the podium, perhaps. Okay? So that's interesting. Certainly nothing I know anything about. And this is the picture that came up. Again, I don't know how well you can see it at the back, but what the picture is showing is the two lizards lying on top of each other on top of the podium. So everything was put in place for these two lizards to fight, I was going to say to the death, but I don't think that's true, to fight for the podium. And the two lizards just said, hey, let's just chill, let's just cooperate, right? Let's just work together, we can both enjoy the podium, and let's just all get along. So they didn't defect, and I'm going to use the word defect and cooperate in this talk, defect meaning not cooperate. So in this instance, fight, scratch, whatever it is lizards do. Instead they cooperate it. And they just said, let's just work it out. And this tweet kind of encompasses everything that game theory tries to do. Apologies to the cameraman, I realize I walk around, so just give up. But thank you. So this tweet is really nice because it kind of encompasses everything about game theory in that a lot of modeling we do is we set out rules from above and we let that dictate the environment. So we define the rules and we measure the environment. But game theory does the opposite. It creates the environment and then it finds out what the behavior is from it. So in this particular instance, Kirstie created the environment, had an expectation of what the behavior would be, right? The lizards would fight and was wrong. The lizards cooperate it. And this is a really nice kind of example of game theory where you can try to understand what behavior is going to be given an environment. And in particular, a lot of what game theory is interested in is why are we nice to each other, right? Why do we cooperate? Why do people work really, really hard to put on these types of conferences, right? Why do people do that? One tool game theory has to study this is called a Moran process. So a Moran process is a very simple birth-death process that puts behavior in an evolutionary setting. And the keynote yesterday, Julia actually spoke about a really cool paper where essentially they were putting scientific behavior in this kind of setting. I haven't read that paper just yet, but I'm really looking forward to it. What I said here is we have a population, and so perhaps blue are the lizards that are cooperative and red are the lizards that are not cooperative. And we select one of the lizards, so one of the behaviors, and that selection can be based on fitness of how well they do in their environment. We reproduce that. So we have this birth step. So we have another one. And then we choose another individual completely at random, not necessarily dependent on their fitness. And that individual is removed. So the Moran process is a model of an evolutionary process in which the population numbers stay the same. There's no one enters, no one leaves. You can write a Moran process, basically evolution, in 30 lines of Python if you include a dock string. I don't expect you to read that. I'm going to zoom in on the lines 14 to 26. And this is just me repeating what I just said. So the idea is we go through every player in our population, and we start building up a score, and that score comes out of playing a game. Every player plays against every other player in the population, and we add up that score. The game is, I'll talk about it a little bit, just a NumPy array. And then this is the selection part, where our fitness just informs an overall probability distribution based on how good we were. So that does allow weak people to continue and to have birth, but also allows, but is more favorable towards strong in terms of fitness. And then finally, we simply get rid of a random individual. This is the game we're going to talk about. So I've mentioned that we're talking about cooperation, and one of the famous games for the study of cooperation is the Prisoner's Dilemma. So that's a two-player game. Some of you might already be aware about the Prisoner's Dilemma from your own knowledge, but a couple of months ago, something went viral. Nicky Case put a wonderful web game together that went around where you could look at cooperation stuff. That's basically what I'm talking about, and that's a wonderful, wonderful tool. I recommend playing around with it. So this is the game, and we have two players, a row player. Let's call them lizards, a row lizard, and a column lizard. And the row lizard decides which column, which row we're in, and the column lizard decides which column we're in. And the first matrix is the utilities to the row lizard, and the second matrix is the utilities to the column lizard. They have two strategies to cooperate or defect, so to just lie there at the top of the podium or to be ready to fight. We'll call lying there at the top of the podium their first strategy, so cooperation is their first strategy. They both cooperate, they both get a score of three, which is pretty good. But if I see you lying there at the top of the podium, basically ready to share this podium with me, and I just come in and start fighting you, you're not ready for the fight, so I'll immediately get a utility of five, because I'll have the whole podium myself, I'll feel real good about pushing you off the podium, and you'll get a utility of zero, because you'll be sad. So if you put us in the first row, I should defect, I should put us in the second column. And that also holds, if you're standing at the top of the podium in a very aggressive way ready for a fight, then that means that if I just walk up, you're going to throw me off. So I'm going to come prepared for a fight, we'll both get very damaged, maybe not even make it to the top of the podium, who knows, right? And so when we just look at the prisoners dilemma by itself, we have what's called a Nash equilibria, which is that we both end up in this bottom left, the bottom right corner of just defecting. So then we come back to the question of why are we nice to each other, right? Because game theory says we shouldn't, because something like this is more or less valid of any interaction. Robert Axelrod is a game theorist who in the 1980s really started a bunch of research about the prisoners dilemma. He invited individuals to submit strategies for the prisoners dilemma, but the strategy is being what happens if we play it multiple times. So we don't just play it once, because if we just play it once, we should defect. There's nothing subtle about that. However, if we start playing it multiple times and our reputation becomes important, and it's all about how do we invest in our reputation, that becomes interesting. So Robert Axelrod invited academics in the 1980s to submit computer code that would play this game against each other and would see what happens. And this leads me to a Python library called the Axelrod Python Library. It's almost three years old, and it allows you to do this type of research. So for example, we can create tit for tat as a strategy, and we can play it cooperator, and we can get them to play against each other for five turns. When we see cooperator versus tit for tat, they both cooperated for the five turns. So they just got along from the start. Two lizards from the start jumped on top of the podium and got along. Tit for tat versus defector, defector just blindly looks for a fight, takes advantage of a cooperation. So what happens is we start, cooperator starts out by cooperating, sorry, tit for tat starts by cooperating, defector defects, and then tit for tat realizes what's happening and starts defecting against defector. And then finally, we've got tit for tat versus alternator. Alternator starts by cooperating and then just alternates. It's a good name. And what we see here is we can kind of see what tit for tat is doing. Tit for tat is in fact just doing whatever alternator did in the previous turn. And Axelrod's tournaments kind of spurned a whole bunch of research because the fact that tit for tat won both these tournaments and won these tournaments against much more complicated strategies, people were like, well, that's kind of an inclination to begin to understanding why we cooperate with each other because tit for tat is fundamentally a strategy that will cooperate. And we can look at that in an evolutionary setting. So putting those strategies in a Moran process. And so what we're seeing here is a Moran process that's created with the Axelrod library, which has lots of plotting capabilities built in. And we see if we start off with a population of defectors and we throw one cooperator in there, over time the cooperator gets pushed back out. So the defectors have resisted the invasion of the cooperator. But the opposite is also possible. We can throw one cooperator in and after time the whole population could change. So that's kind of what we're interested in is why over time do we evolve this cooperative behavior? And these strategies are very simple, cooperate and defect are not subtle. They're in fact rather dumb, right? They just always do the same thing. They don't gain any information from who they're playing. They don't build up an understanding of what's going on. So this work that I'm talking to you about today made use of a structure of a strategy. This is called a finite state machine, which is a well studied mathematical object. Right in a prisoner's dilemma setting, basically we have the history of both plays and we have these states that change over time and they represent how we feel. Cool. We can train these with what's called a dojo. So actually we train these strategies using a genetic algorithm. And they look like this. So all the genetic algorithm is doing is basically moving arrows around, changing actions and building up a strategy. So what we're doing is we're using a genetic algorithm to train strategies to perform in a genetic process. We put all these strategies that we had, some of them trained, some of them pre-programmed, in a big tournament, but this tournament wasn't just a match, it was an evolutionary process. And here are some of the results. So we have the top invaders, we have some strategies that are able to invade 20% of the time and we have the top resistors. And I want to talk about the top resistors, TF1, TF2 and TF3 that you see in bold there are these trained finite state machines. And they are in fact basically uninvatable. What's really cool about them, looking at this plot, this shows what these TF1 strategy does over multiple turns and against multiple strategies. Every row is another strategy and then this are all these turns. And what we're seeing is TF1 always starts with two cooperations and a defection. We didn't tell it to do that, the algorithm trained it to do that. And what that means is that TF1 in fact has a secret handshake. Where if we follow this line against the player that's playing the same finite state machine, they'll both cooperate twice, then defect. And then as long as they have that final defection, they'll end up in this state here where they'll just be in a loop of cooperations. So that's why they're so good at resisting invasion because they basically have a secret handshake. They walk around and go like, oh, you're me, cool, let's just get along. And they do that very, very quickly. And so that is a really neat finding because it came out of the algorithm to go back to those lizards. We didn't say it, we got it out of it. And so it kind of points towards why and how evolutionary processes can evolve self-recognition mechanisms. The reason this was possible is because we had a huge diversity of strategies with which we could train. At the time of writing the paper, we had 164 strategies. In fact, have more than 211 now. And that's all possible because writing a strategy for the library is a simple PR on GitHub. And we've had contributors from all walks of life. This is my last slide, kind of. There's lots of information if you want a photo, if any of this is interesting. This is the slide to take a photo of. Nicoletta, you've already seen, is one of the collaborators on this. And Owen and Mark are the other two core developers of the library. The actual information for the library is there. If you're interested in game theory, my course is there. And that's actually built up on a bunch of Jupyter notebooks where I teach game theory through our first years. The final thing I'll finish on is Julie is one of these collaborators who, as part of our course, had to contribute to an open source project. And we've had contributors from game theorists, computer scientists, but actually a lot of students that just come and submit the strategy. Julie, in fact, did way more than submit a strategy. She made fundamental changes to the library itself. That greatly improved it. But that's been a really great contribution where I say citizen science, we don't just have people lending us some cores on a computer they're not using. People are actually writing strategies that we're using. So that's fantastic. The final thing that I finish on, I'm an organizer of Pycon UK. If you're interested in Pycon UK, come see me. But accommodation is already super expensive, so good luck. And yeah, that's all I have to say.
 Okay.  Welcome back everyone.  We're going to be starting in a minute or so as people are trickling back in from the  coffee break.  So Vince kindly agreed to swap slots with Valerio, but otherwise the program will proceed  as announced.  And he's going to be talking about the game theory project.  Similar to Nicoleta's project.  Yes.  Right, we worked together.  So welcome Vince and thanks for helping out the organizers.  No problem.  Thanks for having me.  I know how hard organizing these things can be and how much work it is.  So thank you for putting on this great event.  My name's Vince.  I've already tweeted out links to this talk as well as links to the pre-print that I'm  actually going to be talking about.  So this talk is going to be kind of following this archive pre-print here and explaining  some of the ideas and how we've used Python to do that.  I'm a mathematician at Cardiff University in Wales in the UK, still in Europe.  I'm also a software sustainable institute fellow that's been mentioned a little bit.  So best practice in terms of software as used in research is important to me.  And finally, I'm a game theorist.  I'm one of the core developers for the Axelrod Library, which is Nicoleta's poster is about  it.  And that's what this logo is about.  And the work I'm going to talk to you about today is not just mine.  There's five authors on the paper, four authors on the paper, pardon me.  And also a huge number of contributors to the Axelrod Library itself.  And so that's why the title of this talk, if you've got the program in front of you,  has the word citizen science in it, because I really feel it's a nice scientific effort.  So this is a tweet.  I don't know how well you can see it at the back.  This is a tweet I saw a little while ago by a postdoc from Penn State called Kirstie  McCloyde.  And she tweets, sets up flawless heat competition trial.  Lizards will fight over hot podium.  There can only be one winner.  So the idea is, I understand by my naive understanding, is there's this pen, there's  two lizards in the pen, and there's one podium that's warm in the pen.  And lizards are cold-blooded, so they like the warm podium.  And so she's going to be able to look at how they fight over this podium.  All right?  Perhaps, and here I'm extrapolating from a tweet, perhaps to study over a longitudinal  time if it's always the same lizard that wins the podium, perhaps.  Okay?  So that's interesting.  Certainly nothing I know anything about.  And this is the picture that came up.  Again, I don't know how well you can see it at the back, but what the picture is showing  is the two lizards lying on top of each other on top of the podium.  So everything was put in place for these two lizards to fight, I was going to say to the  death, but I don't think that's true, to fight for the podium.  And the two lizards just said, hey, let's just chill, let's just cooperate, right?  Let's just work together, we can both enjoy the podium, and let's just all get along.  So they didn't defect, and I'm going to use the word defect and cooperate in this talk,  defect meaning not cooperate.  So in this instance, fight, scratch, whatever it is lizards do.  Instead they cooperate it.  And they just said, let's just work it out.  And this tweet kind of encompasses everything that game theory tries to do.  Apologies to the cameraman, I realize I walk around, so just give up.  But thank you.  So this tweet is really nice because it kind of encompasses everything about game theory  in that a lot of modeling we do is we set out rules from above and we let that dictate  the environment.  So we define the rules and we measure the environment.  But game theory does the opposite.  It creates the environment and then it finds out what the behavior is from it.  So in this particular instance, Kirstie created the environment, had an expectation of what  the behavior would be, right?  The lizards would fight and was wrong.  The lizards cooperate it.  And this is a really nice kind of example of game theory where you can try to understand  what behavior is going to be given an environment.  And in particular, a lot of what game theory is interested in is why are we nice to each  other, right?  Why do we cooperate?  Why do people work really, really hard to put on these types of conferences, right?  Why do people do that?  One tool game theory has to study this is called a Moran process.  So a Moran process is a very simple birth-death process that puts behavior in an evolutionary  setting.  And the keynote yesterday, Julia actually spoke about a really cool paper where essentially  they were putting scientific behavior in this kind of setting.  I haven't read that paper just yet, but I'm really looking forward to it.  What I said here is we have a population, and so perhaps blue are the lizards that are  cooperative and red are the lizards that are not cooperative.  And we select one of the lizards, so one of the behaviors, and that selection can be based  on fitness of how well they do in their environment.  We reproduce that.  So we have this birth step.  So we have another one.  And then we choose another individual completely at random, not necessarily dependent on their  fitness.  And that individual is removed.  So the Moran process is a model of an evolutionary process in which the population numbers stay  the same.  There's no one enters, no one leaves.  You can write a Moran process, basically evolution, in 30 lines of Python if you include a dock  string.  I don't expect you to read that.  I'm going to zoom in on the lines 14 to 26.  And this is just me repeating what I just said.  So the idea is we go through every player in our population, and we start building up  a score, and that score comes out of playing a game.  Every player plays against every other player in the population, and we add up that score.  The game is, I'll talk about it a little bit, just a NumPy array.  And then this is the selection part, where our fitness just informs an overall probability  distribution based on how good we were.  So that does allow weak people to continue and to have birth, but also allows, but is  more favorable towards strong in terms of fitness.  And then finally, we simply get rid of a random individual.  This is the game we're going to talk about.  So I've mentioned that we're talking about cooperation, and one of the famous games for  the study of cooperation is the Prisoner's Dilemma.  So that's a two-player game.  Some of you might already be aware about the Prisoner's Dilemma from your own knowledge,  but a couple of months ago, something went viral.  Nicky Case put a wonderful web game together that went around where you could look at cooperation  stuff.  That's basically what I'm talking about, and that's a wonderful, wonderful tool.  I recommend playing around with it.  So this is the game, and we have two players, a row player.  Let's call them lizards, a row lizard, and a column lizard.  And the row lizard decides which column, which row we're in, and the column lizard decides  which column we're in.  And the first matrix is the utilities to the row lizard, and the second matrix is the utilities  to the column lizard.  They have two strategies to cooperate or defect, so to just lie there at the top of the podium  or to be ready to fight.  We'll call lying there at the top of the podium their first strategy, so cooperation is their  first strategy.  They both cooperate, they both get a score of three, which is pretty good.  But if I see you lying there at the top of the podium, basically ready to share this  podium with me, and I just come in and start fighting you, you're not ready for the fight,  so I'll immediately get a utility of five, because I'll have the whole podium myself,  I'll feel real good about pushing you off the podium, and you'll get a utility of zero,  because you'll be sad.  So if you put us in the first row, I should defect, I should put us in the second column.  And that also holds, if you're standing at the top of the podium in a very aggressive  way ready for a fight, then that means that if I just walk up, you're going to throw me  off.  So I'm going to come prepared for a fight, we'll both get very damaged, maybe not even  make it to the top of the podium, who knows, right?  And so when we just look at the prisoners dilemma by itself, we have what's called a Nash  equilibria, which is that we both end up in this bottom left, the bottom right corner  of just defecting.  So then we come back to the question of why are we nice to each other, right?  Because game theory says we shouldn't, because something like this is more or less valid  of any interaction.  Robert Axelrod is a game theorist who in the 1980s really started a bunch of research  about the prisoners dilemma.  He invited individuals to submit strategies for the prisoners dilemma, but the strategy  is being what happens if we play it multiple times.  So we don't just play it once, because if we just play it once, we should defect.  There's nothing subtle about that.  However, if we start playing it multiple times and our reputation becomes important, and  it's all about how do we invest in our reputation, that becomes interesting.  So Robert Axelrod invited academics in the 1980s to submit computer code that would play  this game against each other and would see what happens.  And this leads me to a Python library called the Axelrod Python Library.  It's almost three years old, and it allows you to do this type of research.  So for example, we can create tit for tat as a strategy, and we can play it cooperator,  and we can get them to play against each other for five turns.  When we see cooperator versus tit for tat, they both cooperated for the five turns.  So they just got along from the start.  Two lizards from the start jumped on top of the podium and got along.  Tit for tat versus defector, defector just blindly looks for a fight, takes advantage  of a cooperation.  So what happens is we start, cooperator starts out by cooperating, sorry, tit for tat starts  by cooperating, defector defects, and then tit for tat realizes what's happening and  starts defecting against defector.  And then finally, we've got tit for tat versus alternator.  Alternator starts by cooperating and then just alternates.  It's a good name.  And what we see here is we can kind of see what tit for tat is doing.  Tit for tat is in fact just doing whatever alternator did in the previous turn.  And Axelrod's tournaments kind of spurned a whole bunch of research because the fact  that tit for tat won both these tournaments and won these tournaments against much more  complicated strategies, people were like, well, that's kind of an inclination to begin  to understanding why we cooperate with each other because tit for tat is fundamentally  a strategy that will cooperate.  And we can look at that in an evolutionary setting.  So putting those strategies in a Moran process.  And so what we're seeing here is a Moran process that's created with the Axelrod library,  which has lots of plotting capabilities built in.  And we see if we start off with a population of defectors and we throw one cooperator in  there, over time the cooperator gets pushed back out.  So the defectors have resisted the invasion of the cooperator.  But the opposite is also possible.  We can throw one cooperator in and after time the whole population could change.  So that's kind of what we're interested in is why over time do we evolve this cooperative  behavior?  And these strategies are very simple, cooperate and defect are not subtle.  They're in fact rather dumb, right?  They just always do the same thing.  They don't gain any information from who they're playing.  They don't build up an understanding of what's going on.  So this work that I'm talking to you about today made use of a structure of a strategy.  This is called a finite state machine, which is a well studied mathematical object.  Right in a prisoner's dilemma setting, basically we have the history of both plays and we have  these states that change over time and they represent how we feel.  Cool.  We can train these with what's called a dojo.  So actually we train these strategies using a genetic algorithm.  And they look like this.  So all the genetic algorithm is doing is basically moving arrows around, changing actions and  building up a strategy.  So what we're doing is we're using a genetic algorithm to train strategies to perform in  a genetic process.  We put all these strategies that we had, some of them trained, some of them pre-programmed,  in a big tournament, but this tournament wasn't just a match, it was an evolutionary process.  And here are some of the results.  So we have the top invaders, we have some strategies that are able to invade 20% of  the time and we have the top resistors.  And I want to talk about the top resistors, TF1, TF2 and TF3 that you see in bold there  are these trained finite state machines.  And they are in fact basically uninvatable.  What's really cool about them, looking at this plot, this shows what these TF1 strategy  does over multiple turns and against multiple strategies.  Every row is another strategy and then this are all these turns.  And what we're seeing is TF1 always starts with two cooperations and a defection.  We didn't tell it to do that, the algorithm trained it to do that.  And what that means is that TF1 in fact has a secret handshake.  Where if we follow this line against the player that's playing the same finite state machine,  they'll both cooperate twice, then defect.  And then as long as they have that final defection, they'll end up in this state here where they'll  just be in a loop of cooperations.  So that's why they're so good at resisting invasion because they basically have a secret  handshake.  They walk around and go like, oh, you're me, cool, let's just get along.  And they do that very, very quickly.  And so that is a really neat finding because it came out of the algorithm to go back to  those lizards.  We didn't say it, we got it out of it.  And so it kind of points towards why and how evolutionary processes can evolve self-recognition  mechanisms.  The reason this was possible is because we had a huge diversity of strategies with which  we could train.  At the time of writing the paper, we had 164 strategies.  In fact, have more than 211 now.  And that's all possible because writing a strategy for the library is a simple PR on  GitHub.  And we've had contributors from all walks of life.  This is my last slide, kind of.  There's lots of information if you want a photo, if any of this is interesting.  This is the slide to take a photo of.  Nicoletta, you've already seen, is one of the collaborators on this.  And Owen and Mark are the other two core developers of the library.  The actual information for the library is there.  If you're interested in game theory, my course is there.  And that's actually built up on a bunch of Jupyter notebooks where I teach game theory  through our first years.  The final thing I'll finish on is Julie is one of these collaborators who, as part of  our course, had to contribute to an open source project.  And we've had contributors from game theorists, computer scientists, but actually a lot of  students that just come and submit the strategy.  Julie, in fact, did way more than submit a strategy.  She made fundamental changes to the library itself.  That greatly improved it.  But that's been a really great contribution where I say citizen science, we don't just  have people lending us some cores on a computer they're not using.  People are actually writing strategies that we're using.  So that's fantastic.  The final thing that I finish on, I'm an organizer of Pycon UK.  If you're interested in Pycon UK, come see me.  But accommodation is already super expensive, so good luck.  And yeah, that's all I have to say.
EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_0_155.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_156_1895.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_1896_3707.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_3708_5459.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_5460_5819.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_5820_5975.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_5976_6335.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_6336_6599.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_6600_8435.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_8436_10307.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_10308_11975.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_11976_12731.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_12732_15743.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_15744_18515.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_18516_20027.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_20028_20075.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_20076_20123.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_20124_20651.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_20652_24155.png EuroSciPy_2017__Handshakes__Citizen_Science_and_Evolution_p4t9dMKxZAo_24156_26485.png