Python's Role in Big Data Analytics: Past, Present, and Future
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Oliphant, Travis
Travis Oliphant - Python's Role in Big Data Analytics: Past, Present, and Future Python has had a long history in Scientific Computing which means it has had the fundamental building blocks necessary for doing Data Analysis for many years. As a result, Python has long played a role in scientific problems with the largest data sets. Lately, it has also grown in traction as a tool for doing rapid Data Analysis. As a result, Python is the center of an emerging trend that is unifying traditional High Performance Computing with "Big Data" applications. In this talk I will discuss the features of Python and its popular libraries that have promoted its use in data analytics. I will also discuss the features that are still missing to enable Python to remain competitive and useful for data scientists and other domain experts. Finally, will describe open source projects that are currently occupying my attention which can assist in keeping Python relevant and even essential in Data Analytics for many years to come.
.. Good morning, everybody. It's a big honor to announce our next keynote speaker, Travis Elephant. He's a good friend of mine. If you don't know him, he's also of Numpy. Numpy is a scientific Python package. Pretty much everybody who's in the scientific era in Python uses Numpy and he's the one who wrote it. And he's also the CEO of Continuum Analytics, the company behind Anaconda and a lot of other open source packages, though he is really supporting the Python community. And he's also a founding member of NumFocus. NumFocus is a non-profit behind PyData. Speaking about PyData, that's a great question. I'm going to talk about this in the afternoon with tutorials. If you're interested in Python and PyData, this is a topic Travis is going to talk about. There will be a lot of talks and tutorials. We have much more depth about it. So I encourage you to join the PyData conference. There are still tickets available. So please go ahead and buy tickets. And now I would like to give the word to Travis talking about some content and future of this field. Please give a big hand to Travis. APPLAUSE Thank you very much. I think we're having a little technical difficulty here. So it seems to be a common pattern. Sorry. So, can you show two displays? It does. So, do you have a picture here? Do you have any picture? We are now connected to VGA. We are VGA, namely MIRROR. We are now connected to VGA. There we are. You can see a very messy desktop and a bouncing beach ball. Excellent. Maybe you will come back to me soon. I don't know. I'm still not live. I don't have the control of my screen system. I don't have it. It's a bouncing beach ball. I don't know why you were swapping things out while I was doing things. I don't know why you were swapping things out while I was doing things. Hopefully that will work. Are we there? Thank you. You all had time to catch up on your email. Now you can give me your full attention. Last minute email you got a chance to send off. I understand. I like to check my email there in talks too. I will try to keep your attention so you don't get too bored. I want to talk a little about me. Not too much. I want to talk about the technologies I have been involved with. Just to give you a little background. I'm familiar with NumPy stack and science. I'm a scientist by training. My roots are in satellite scatterometry. I used to measure wind speed over the ocean. That got me into large scale data analysis. You had backscatter from the ocean surface and satellites. They came on big tape drives and used a VMS machine. It was awesome. It had a floating point format that was different than IEEE 54. That's where I started. We used to make really nice pictures. I did some Perl. I did some Matlab. I used a lot of C in order to produce these kind of slides. When I did my PhD program at the Mayo Clinic, I got into a different kind of wave. It was waves I was making inside of people. Basically we'd wiggle people with speakers. We'd start to shake them. When you shake people, waves propagate inside. Then with either MRI or ultrasound, you can actually see those waves propagating. From that, you end up with a big kind of an inverse problem. It's that equation there which I like to scare people with. I scared my committee with it too. It was fun. It's not that hard of an equation. It's just a simple linear equation. Many of you here are pretty good at it. My goal was to invert that. I had to find the derivatives of five-dimensional data. Here I was with a very large cube of data. It was too big to fit in memory for Matlab. The Matlab double. There wasn't a Matlab float at the time. I really liked working at a high level. I could program in C, but when I was thinking about my data problem, I didn't want to be programming in C because I had to think about pointers and arithmetic and figure out where my memory leaks were. I really liked that high level. I searched around and I found Python. The rest is history. Basically I found Python started to do a lot with Python. I did finish my PhD, although it was somewhat delayed. Just to give you kind of a context for Python's origins and where I started to use Python. Guido released it. There's arguably maybe a little before this, but I know that 0.9.0 came out in February 1991. I was not a Python user then. I really came to the scene in 97. So you can see that 96 is kind of the... I used version 1.4 of Python. It's a great version, actually. In fact, if you'd like to try the 1.0 series, you can make a new environment in Conda and install the 1.0 version of Python just for fun. You can see kind of how it worked and do a test environment of Python 1. I started using it in 97. I have that little gold highlight. Keep that in mind because that's an important date, 94, with respect to data analysis in Python. I'd like to show this slide because this is really what got me going in writing C extensions for Python. If I've done anything else, I've written a whole bunch of C extensions for Python. Much to the chagrin of the PyPy crowd. Probably they're the number one enemy because I've written so many C extensions that make their job harder in moving people off of the large platform. But I'm not the only one that does it. There's a lot of people that write C extensions to Python because it's so easy to and because it's so extensible. But I show this slide because of Mike Miller, actually very close to my friend Mike Miller, but Mike Miller, he released a package called TableIO, and that's how I learned to program C extensions. I grabbed his package, studied the source, and then also read an essay by Guido about reference counting. You've got to figure out how reference counting, if you write extensions, fortunately if you write Scython and other things, you don't have to do that anymore. But if you really want to get to the root of things, you have to do a reference counting. So this opened my eyes to the power of open source. I could basically look at the code, understand it, learn something. I learned a tremendous amount by reading this code. And then I started to experiment with my own modules. And in 1998 was my first extension module for Python. It's called NumPyIO. I've been embedded in other parts of the stack these days. And in fact, there's better ways to do it. But that was my very first module in 1998. And I kind of got hooked. And sort of that's when I, my career as a scientist, sort of pivoted, let's say pivoted, that's the right word. Pivoted into tools for scientists. So in 1998, in 1999, I started to get really addicted, got really caught by that bug. I think there is a chemical compound that is called Addiction to Open Source. It's somewhat related to our addiction to Facebook. I think it's connected. But back in the time, I started releasing wrappers in 1998. First FFTW, then C-Feeze modules, then stats, which was from help from Gary Stringman. He put out something in 1998. And in 1999, I went back one year and looked at the mailing list. It's really nice to go back and look at the history of what you said in the past and cringe a little bit at how stupid you were. But, you know, we all make mistakes. But go back and kind of, and also see kind of how motivated you are and how excited you were about something new. And I was very excited about Python in 1999. I said, hey, we could use Python to build a data analysis environment. We could do all of our calculation code, all the things I loved using a high level language like Matlab about. I could do that in Python. So that year, I went back and looked at the mailing list and sort of every month, I was sort of saying, hey, here's a new package I just made. And of course, it wasn't very pretty. In the web page for it was very ugly. I'm still not a very good web designer. I'm not a good content but not really good pretty pictures. So if you look at those, back in the day, nobody else had pretty pictures either, so it was okay. I just had a really Spartan website. And I made a bunch of releases. Then a guy named Perot Peterson came along and said, hey, this is really stupid of you, handwriting Fortran wrappers to multi-pack on all the libraries on Netlib. I'm going to write a tool to do that. That's when I first learned the difference between me and a real computer scientist. Right? Me, I'm like, I'll do this manually because I just want to get it done. And a real computer scientist goes, we have to automate this and make it. And in fact, you know you're a real computer scientist, you'll spend more time automating than it would have taken to do it manually. Right? And now that's not the case. F2Py was a tremendous tool and we worked together basically through the last part of June, 99 to that year. So 1999, that was sort of multi-pack, it was called the time. And that was my, I put it on the web and people started to download it. I got started to get to know people from all over the world. Piero Peterson, the gentleman who wrote F2Py is from Stonia. I don't know if he's here, he could be here, but he's kind of, yeah, I know he likes to water ski in the lakes in Estonia. That was just a tremendous rush to be able to coordinate and communicate with people all over the world and see them use your stuff and contribute back and make it better. It was an amazing thing. And it was my first taste of open source community. And I've just seen that grow and grow and grow ever since. You could use the Python for data analysis way back in 2000. Whereas I wrote NumPy, NumPy came from a tradition, it came from a history that had started in 1994. And here in 2000, I could use it to publish pictures in my thesis. Now this is using a Python interface with something called Dyslin. Anybody here use Dyslin? Anybody? Dyslin is actually a pretty sophisticated tool, although a little bit tedious to use. But I was able to do it, publish all my pictures in my thesis. I think there was 200 different images in my thesis, all with Dyslin and with Python. Now in 2001, that's when Eric Jones contacted me and together with Piero Peterson, we kind of pulled stuff together and spent a whole bunch of time building for Windows. A whole bunch of time essentially going through the bug, building a package. And we came out with what we called the SciPy library, but really was the SciPy distribution. And I didn't realize at the time, but that's really what it was. And it was a collection of all these tools together kind of with a single installer, so you could get everything up and running quickly. There were a lot of people who contributed to getting SciPy out the door. But it was a lot of work, just on pulling everything together and doing a lot of... I was not a Windows program at the time. I've since learned to make my piece with the Windows platform, and it's fine. There's some interesting things about it. Actually in the parallel space, there's some really interesting things about it. But as I said before, NumPy really inherited a long legacy of great minds that had gone and tried to build numeric array object for Python. Jim Fulton, he's embarrassed when I show this because it was just a Python matrix object, but it was real. It was part of the early discussion, and it caused Jim Hugenin as a graduate student at MIT to get really excited and to sort of procrastinate his graduation in order to write numeric, which became the foundation of... And that's why I came to Python because numeric existed, so I'm really grateful for that. Then in 2001, some features were desired for numeric and Perry Greenfield, Rick White, Todd Miller at the Space Science Telescope, the folks that put out Hubble and processed the Hubble images. They needed some changes, most particularly, they needed to be able to memory map to more diverse data, and so they were writing numeric. And then at that time, I kind of said, well, it turned out actually I didn't have a class to teach. It was really kind of a nice, confluence event. I didn't have a class to teach because the one that was scheduled, nobody signed up for, it was sort of a build an MRI, and only one person signed up. I guess they might have been too intimidating. It was smart, actually, not to sign up for that class. But I ended up without a class to teach, and I probably should have been publishing papers in order to keep my position as an academic professor. But at the time, I saw numerae, I saw sci-pi, which is built on numeric, and I saw modules getting built for numerae, and I saw this split happening in the community. It was already so nascent, and people were still struggling just to support one thing. And I felt like I had to do something about it. It was just a really strong feeling that somebody has to do something about this, and I don't know who's going to, because nobody else knows the code base very well. And I had time, so I did. I just did it and said, okay, throw caution to the wind and dive in. And basically spent about what I thought would be about a three-month project, turn it into an 18-month project, in order to kind of put the first version of NumPy out there. Since about 2007, the community, it took a while for the community to get excited about it and get more contributors, but 2007 came and it really started to take off, and lots more people joined. There's still room for people, especially at the low level. The number of people that can understand the CAPI of Python and then help maintain the NumPy code base, which is in C, is shrinking, right? And that becomes a challenge. But there is a NumPy and SciPy both now are a very impressive community effort, and I'm really grateful for that, because they wouldn't be what they are without all the people that are contributing and making it possible. I don't know exactly how many NumPy users there are. I estimate maybe 3 million. That's on the basis of hits to a web page, download numbers, but it's always hard to tell, because nobody writes home, nobody sends a postcard, so you never really know how many users of NumPy you have. I wanted to kind of just pause a little bit and maybe to motivate some of you who I know are building your own software packages and communities, talk a little about the things that I've learned, about what it takes to do this, because if Python is going to have a role in big data analytics, it's going to take the work and effort of a lot of people, just as it has to date. So one of the things that's most important, I think, is to recognize that it is hard work initially. It's not easy. And usually it's quite lonely, actually, when you start off on a new venture. Initially, nobody really believes in your idea except you, and that's really the way it should be. Others will need some proof that this is actually going to work before they dive in. When I said, hey, I'm going to merge NumRay and NumRoc and do it on the NumRoc code base, a lot of people said, that's great. Go do that. It'll be fine. And it was really until they start seeing results, and they say, oh, this is actually going to work. Okay, now we'll dive in. And very gratefully do that as well. But that's going to be how it works. You're just going to have to dive in and do some things, maybe with a small team, maybe you and another person. And in fact, the more complicated what you're doing is, the more lonely it's going to be, because the fewer people that will understand enough to be able to make the trade-offs and help you. So SciPy was another example. I mean, I started SciPy, and it took a while before we got more and more help. And Piaro joined basically the end of the year. I procrastinated my PhD by at least a year to create the beginnings of SciPy. And don't tell my wife, right, she's in the room, so I shouldn't have said that. I was a poor struggling graduate student with three kids, and we were making 18,000 a year in Minnesota. And, you know, we did great. We did great. But I know she was thinking, when are you going to finish, honey, so we can actually have a job. Piaro Peterson put in tremendous work to create F2Py and SciPy Lin-Alge. I was just flabbergasted by the amount of work he put in. In fact, one of my great anecdotes from Piaro is when I sent out my multi-pack, and he basically submitted a make file. It's still an incomprehensible make file. I have no idea what it said. But it was amazing. It sort of built everything, including, I think, made the coffee in the morning. People will do that. And he put in tremendous amount of work. It makes that kind of work to get things off the ground. It really does. You can't just sort of show up and hope that things happen. You really have to put in the work. The other thing I think is important is you do what's right. In other words, you've got to put aside kind of thoughts of, oh, I'm going to make a lot of money, or I'm going to be able to do something really cool. Because doing what's right means you have your information, you have your knowledge, you have your semantic environment, the things you know about, and those things come together for you to feel, hey, this is what I think is right. And you're the only one that has that feeling, and you need to own it and do something that's right. Timing is everything. Sometimes you're the right person for the job. Sometimes it's the right time for that to happen. And that's hard to know when is the right time. But when the time is right, you have to have an urgency. It can't wait. It won't wait. People are impatient. If things don't move in a certain direction, they'll wander off and do something else after a little bit of time. You have to strive for excellence. And it really takes work. And you have to give the best you have. It'll never be enough. But give your best anyway. I love this quote from Mother Teresa. There's actually a whole series of these aphorisms. The good you do today will often be forgotten. Do good anyway. It's absolutely true. And it's essential. It's one of those life qualities that I think I would love to grow into someday. Another key to a success is while it's initially lonely and you're doing things on your own, you have to build a community if it's going to have long-term success. You have to get others involved. And that means interacting with people. And that means having your ideas be put up and shut down. It basically means you'll need to help other people. Other people have different ideas than you. You have to give up some of your hard-won ideas. You have to give up some of your ego. Someone will point out how bad you suck. And you probably do in some places. And it's okay. Listen to that. And that's the only way to make progress in the community is for people to come together and have some empathy towards each other. It does expose you. You know, to treat other people the right way, you have to care about them. And that does expose you because you can't get hurt. But you do it anyway. And then you keep moving forward. And it's a great thing. Much more we said on this topic. I think the key to building a community, there's a lot of lessons to be learned from all kinds of projects. I'd love to talk to some of you about your lessons of how you build communities. Then patience and a little bit of luck also. Good things take time. It doesn't happen right away. NumPy took a long time to get real adoption. 2007, you know, six is sort of two years afterwards. It really was 2009 before I started seeing a lot of adoption of NumPy. So it's four years later. And the right factors have to come together. One of those factors, for example, in the NumPy sci-fi communities was GitHub. For a long time, the number of contributors just sort of creeped along. And then the ease in which you could contribute jumped. And all of a sudden, we got more contributors. All right. So that's a little bit of history of what I did in the past with NumPy and sci-fi. So what is this NumPy anyway? Perhaps a show of hands. Who uses NumPy or knows what it is? Okay, a fair number of you. Excellent. You know, the Europeans are always smarter than the... I asked that question in the U.S. and I don't quite get the same response. There's maybe half of the room. But here, everybody at least has seen it. Now, you all rejected it, of course, and do something else. But that's another beauty of the European. But what NumPy is essentially an array-oriented extension. It's got an array object and fast operations on an array object. It's fairly simple, actually, it's core. Here's just simple examples. You can build a two-dimensional array and do some computations on it. And you can sum along different axes. Or you can have a three-dimensional array. You can have an n-dimensional array. And that's the idea of organizing data together in a way that can be operated over quickly. And by quickly, we mean you hand it over to a precompiled loop or a precompiled engine that does the computation. And so that's not happening in Python. So when you work with NumPy arrays, you can release the GIL. You are releasing that global interpreter lock. And you don't have the same problems that happen if you're not using a NumPy array. So here's just a diagram of what it looks like as a Python object. Each element of the array has to be exactly the same type. That's one restriction of a NumPy array. So it has to be the same number of bytes as its basic restriction. But that bytes can be a lot of different things. It can be a pointer to a Python object. It can be a structure with an int, a float, and 10 bytes of strings. It can be Unicode. UTF32 is always what Unicode is in NumPy. And then there's these array scalar things. It'll also creep up and bite you on occasion if you're trying to understand what comes out of a NumPy array. So fairly straightforward. Once about three years ago, four or five years ago now, somebody asked me to come up with a Zen of NumPy. If you don't import this, you all know Tim Peter's aphorisms that have come up. And very, very nice. And this is definitely not at that same caliber, but it gives a little bit of a flavor of how I think about NumPy. Strided better than scattered. Contiguous better than strided. Descriptive is better than imperative. Array-oriented is better than object-oriented. Perhaps a little debatable. We might...it's a fun one. Broadcasting is a great idea. And vectorize is better in explicit loop, unless it's too complicated. And then you can use scython or numba. And think in higher dimensions to solve your problems. Real quick kind of example of array-oriented computing. I like to show this. It's a little bit of a cheesy example, but the Fibonacci numbers are so common in Python, we like to show them. Of course, here's the Python implementation. One bad one, one better one in terms of performance. Now, if you compare the Python approach to Fibonacci versus, okay, I'm an array-oriented guy. I use the NumPy stack. What do I do? Well, of course, I reach the solution and find the roots of the discrete difference equation and just compute them for any n. So that's a vectorized computation here, Fib2a. I just generate a vector of numbers n, and then I calculate with the root r1 and r2. I can use the roots command to take the roots of a polynomial, calculate the power of those roots, subtract. Those are array expressions happening. I don't see any for loops there, but for loops are happening under the covers. That's the concept of array-oriented computing, is gathering your data, doing high-level computations on them all at once. And then, if you're really clever, you understand that it's basically the output of an unstable filter, and I can use the linear filter tool in SciPy to generate the output of at least the first part of the Fibonacci sequence. Those who have done this in the room will also understand that I'll have overflow if I use the floating point of the machine like I'm doing with SciPy and NumPy. But you can get faster performance. That's one of the benefits of array-oriented computing, is you immediately typically get faster performance. And so that's usually what people reach for, and why they reach for array-oriented computing, is to get the fast performance they're looking for. There are other reasons to do it, however. APL really was the father of array-oriented languages. It's been around since 1964, but the hieroglyphics of APL are still trying to be decoded. We have not found the Rosetta Stone yet to understand what people actually said in all these wonderful array-oriented codes. Just kidding, actually. I know some people can read APL. And there were other English versions of that same concept brought in. Had a lot of the same ideas, and NumPy is a descendant of APL. All right, so another simple idea of array-oriented computing is to gather your data together, whereas a lot of object-oriented approaches end up scattering your memory all over the place, objects versus attributes. If you gather that all together and make objects essentially rows and a table of attributes, you're going to be able to do column-oriented processing. Your data is all together, and your modern processors can scream through this. Array-oriented computing is perfectly suited and matched to the vector computers of today, the multi-core, the multi-CPU. So whenever you can do that, you get the added benefit of actually being able to take advantage of that hardware that's otherwise not really exposed very well in the languages of today. So I've talked about those benefits. I'll move on. I'm going to skip this example, put it up briefly just so you get a feel for the kind of code. Once it was awesome, somebody tweeted, here's a problem, and I said, oh, I think I can solve that rather than spend time on company stuff. So I had a good time just to play with this problem, and this is what I came up with, to basically find a circle out of this roughly circle-like image. So that's the kind of thing you can do. Now NumPy has had a story in data analytics for a long time because of these structured arrays. I said briefly before that every element of a NumPy array can be an arbitrary structure. It can be integer, float, whatever. So you can think of a one-dimensional NumPy array as a table, as an Excel table, and it's a really nice mapping. However, it's an array of structures, which sometimes isn't the optimal data structure when you're trying to say add new columns quickly or do computations down the columns. So even though it works, it's not as flexible as we'd like, and PAN has emerged as basically this generic structure of arrays where it basically has pointers to different arrays under the covers. So it sits on top of NumPy, and it has a lot more user-friendly tools for people doing data analysis. So whereas behind the past, when you were using NumPy to do data analysis, you might have to write five to ten lines of code with PAN as its one, or a method call, and it's quite a bit simpler. So a lot of people have come to the Python data community because of PAN as this list basically comes from a user of PAN as who says, this is why I love PAN as a few of these reasons, and then I modified it slightly. So currently, today, in big data analytics, Python, this is the basic key libraries, and I might be missing a few here, but these are sort of the basic ones. NumPy, PsiPy, PAN as MapLotLib, Ipython. The list goes on, it becomes quite a stack, so when you're sitting there, I want to use Python for data analysis, you have to get a bit of stuff together to make that happen. That's really why we created Anaconda and Kanda. Some of you may be interested to note that whereas a lot of people have been using R for data science, Python is rapidly creeping as an equal footing for data analytics with R. A lot of people, this is a recent survey done at O'Reilly, and they surveyed the people of Tendon Strata in 2012 and 2013. This is a revelation to O'Reilly as well, I think, because they've been really searching for people to write books on Python for data analysis. I think also Wes McKinney's book was successful, and that really opened the floodgates as well. They're really, really, as a market here, we should get books. I've been asked to write a lot of books, I have no time to write any books, I'm a little bit more, I handed that to a few other people, maybe we'll get a book out of Continuum. We also see articles like this. I don't want to get into language wars, I think actually we can work together with the R community, but it just goes to show that it becomes kind of a choice. You can do everything you need to do in Python pretty much, and occasionally you need to call out to R and you can do that. Python is growing as the top language in schools. Many of you have seen this, but I like to show it. We're in a Python conference. We should kind of celebrate the fact that Python is being used forever, lots of places. This is US schools, top universities. Python is the number one introductory class being taught. Now some will say that's how languages go to die, so maybe it's not such good news. Right now we're going to get a whole lot of people using Python who have no idea what they're doing, but I trust that our community is vibrant enough and robust enough to welcome them in, train them up, actually unlearn the things they might have learned wrong in school and help move the community forward. So I have here, I know I'm running out of time, I've had plenty of slides, plenty of things to talk about, but I wanted to talk a little bit about why I think Python is fantastic for technical computing. As I said, I was a domain expert, a data scientist, a scientist coming to Python, and I had reasons for it. Some of those were the same as they are today. One, syntax. It gets out of your way. You don't have to learn jargon and concepts. It basically leverages my English language data centers. Perhaps there will be a language that leverages Mandarin data centers in the future. I won't benefit from it, but others will. This leverages those Latin character centers. White space. I love white space, the fact that it conveys intention. I'll tell you why, because my field of view is limited. I have limited horizontal and limited vertical real estate that I can see, and I have to understand something in that limited space. If I'm using that up with braces and brackets and things that are unnecessary, it's just waste for me. It's also why if I have long, long paths, long, long dot paths for things and my variable name takes up the whole screen, I'm in trouble. I'm not a big fan of that either. Complex numbers were built in early. Overloadable operas were built in early. This is a mistake Java made for the scientists. Scientists need complex numbers. The FFT is the reason that electrical engineers have jobs, and it immediately has the FFT. You've got to have a complex number, or you'll have 200 of them, and nobody will be able to agree on what it should be. Just enough language support for arrays, which is the brackets, the ability to have commas go immediately to tuples, so you don't have to have that funny indexing. These things were added actually at a critical time. I definitely have to thank the Conrad Hinsons, the Paul de Bois, Jim Huguenin, who worked with the Python devs, Guido and others, to make sure these were added in language in an early time. It's been fantastic. Occasional programmers can understand it. Occasional programmers are the people like I was, who don't have to solve this five-dimensional differential equation and don't want to spend time chasing pointers, but I need to be able to see code and read it and understand it. And Haskell or Clojure is too much to put in my head. And remember, so Python works perfectly in that space. And I used to say that packaging was a problem with Python. I no longer say that because packaging is awesome with Kanda. Kanda makes your packaging problem go away. It's fantastic. We get that feedback from users all the time. So I'm not just saying it because we put it out. I'm saying it because we get that feedback from people. And I use it and I love it. It solves exactly the problem I've seen with not being able to get everything installed easily and quickly. So lots of great things about Python, simple, extensible implementation. It's general built. You can build a system, general purpose, supports multiple programming styles, all these things you know. But a critical one is that it does have critical mass, because you could have the ideal language, but I don't have people using it and you'd be stuck. You could not build a community. That's the hard thing. And that's a bit of a chaotic question about when that will happen. I can't give you answers. It's sort of one of those emergent phenomena. Now, there's things I don't like about Python. And we could all probably rag on it together. But some of these are being addressed. I would love to see anonymous blocks. I'd love to see the ability to have anonymous chunks of code that you could then send around places. Really for deferred evaluation is the most common use case for me. I'd love to be able to do slice syntax outside of the brackets. Please just let a slice syntax be able to create the slice operator. I use that a lot as a ray-oriented programming guy. To see Python run time to gil, global variables inside, lack of dynamic compilation, there's some work to be done there. That's a really hard one. I would love to see some language extension other than import hooks. I've seen a lot of very creative uses of import hooks to have kind of DSLs that can be imported into Python. It's kind of cool actually. You can extend Python as you like with the import statement. And it can be hard using a general purpose language because the devs of that language don't necessarily understand your use case. It's a ray-oriented programmer. And I'll have a story about that in a little bit about PEP 3.1.1.8. So NumPy, I like it. It's good. It's got a lot of good things. But there's got a lot of problems with it too. The D-type system, the data-type system, which essentially allows the structure to raise, it's too limiting and difficult to extend. It's really, it's more, it grew out of Numeric's data descriptor that was there at the beginning. And it kind of extended it just far enough, but it needs to be kind of overhauled. The immediate mode, creating huge temporaries all the time when you have an equation to evaluate. It's almost an in-memory database, really close. If you're using SQLite, you can also use NumPy for the same purpose. It's going to be faster actually, but it's not quite. It doesn't really evaluate some of the operations that you would like. Lots of unoptimized parts. Lots of embarrassingly unoptimized parts, actually. If you start giving the code, you'll say, who's the idiot that wrote this? Hopefully the blame's not me anymore, because somebody's changed the comments or something. The code base is organic and hard to extend. I think, as I reflect on back to the history, I think one of the most important pieces of work that I did in 2005-2006 was actually sit down with Guido. I remember flying to San Mateo where he was working, and I said, I'm going to have lunch with Guido and get NumPy into Python. I saw this little professor and think, I can go do this. He's a very nice guy, very accommodating. I went there with Paul DeWaughn. We sat and we had lunch and we talked about what can we do to get NumPy into numeric or the new numeric at the time when we were calling it into Python. He cautioned about, well, you know, if you get into Python and you have an 18-month release cycle, it won't be able to be updated very quickly. There's some downsides too. That was enough to go, yeah, we probably ought to not do that. We want kind of the structure of the NumPy rate in Python. And so I spent time writing this PEP, PEP 3118, to really extend the buffer protocol. Now raise your hands if you know what PEP 3118 is or know what the buffer protocol is. I should see a fewer hands because this is one of those sort of underbellies of Python, where the ability, it's kind of on the lowest level, but it's really the ability for arbitrary objects to share data. And that's what the buffer protocol allowed that initially, but it only allowed you to share a single point of data about the data. You couldn't share that it was really an array. It had this kind of data type in it. So the extended buffer protocol was really all about getting more metadata around the pointer to memory that was being shared. It really makes possible a heterogeneous world, a powerful array like objects. So it isn't really necessary for there to be only one NumPy. There really could be a lot of array like objects that share memory and they could really operate independently and coexist. I think adding multiple dispatch to language would actually improve that better. A multiple dispatch library would actually make that a heaven. At that point, you just need to pep through an eight multiple dispatch. We don't need a single array library like NumPy. So that's what I think of the future of NumPy and the future of the world. I think of a heterogeneous world, a world with a lot of things working. The buffer protocol exposes this idea and I don't have time to talk about it today. Someday I'll maybe give a talk about this. I don't feel always qualified to do this. I'm a scientist by training and I'm always kind of an approaching scientist, software developer by effort and trade and learning from other people. But there's something real about this dual, the dual of encapsulation, the buffer protocol exposes. These data types, they sort of invert that idea of having data and having methods attached to that data. The data is exposed and you talk about it in full terms like a schema and then you throw code past it. I'll talk a little bit about that, about I think what it allows for us to do in the future. So what of the future? We've talked about today, we've talked about a little bit of the past. What's going to happen in the future? Well, I've watched the Star Trek episodes so I think I know the future and I'm going to be able to tell you what's going to happen. I'll just describe what I'd like to see and some of the principles that I think that will guide the future but we can't ever tell. We can't ever tell what's going to happen. One of the things that's very real and is based in physics is the idea that data has mass. Data is growing faster than speed of light can carry that data from one point to another. Therefore that means you're going to want to have data sit where it is and that's true whether it's in a GPU or it's in a memory cache or whether it's on a cluster somewhere. You don't want to be pulling data. So what that means is a lot of our systems that were built around the idea of encapsulation and serialization are actually wrong. They don't work very well when we do that. And so we have to kind of think differently about how we're going to manage this. There's a blog about this as well. It's a well-known observation. Data gravity some guys even invented. You can even talk about the formula for data and how it attracts each other. I don't know, maybe that's useful. I think it is useful to think differently in a relativistic sense. Normally we think about ourselves on a platform while data moves past us. We kind of serialize data into our objects and then we do our little computation and then we serialize them on their way. But when data has mass that's really expensive. It turns out our computations are really simple. Our machines can rapidly do the computation and wait all day long for the data to pipe through. So how do we invert our thinking about this and think about it from a data-centric perspective where the code comes to the data and flows through the data? So that's one thing that I think about. And part of what's in PEP 318, that buffer protocol, has some of the answers I think. At least some of what I've been able to see. Some of you I'm sure will be able to think even more deeply and better about that. And I'd love to get your feedback and your inspiration. I think fundamentally the future of big data in Python is going to be heterogeneous. While as before we've had this notion of here's NumPy and everybody uses NumPy as a single kind of channel. NumPy really is just a description. It's a protocol. I love the talk on Tuesday by Pieter, the zero and Q author, who talked about the decentralized role in the future and contracts being the most important thing. PEP 318 is an example of that kind of protocol or contract between objects. And it's a beautiful thing. It's an important thing. And I think that's what the future is going to be like as well. Much more of that rather than here's a single library that everything sits around. And then what Python, its role is going to be doing what it's always done really well. And that's playing this tremendous glue, this tremendous ability to just stick things together very, very quickly. That's the advantage of not having static types is you can pull things together from all sorts of places in an agile fashion, in an iterative fashion, fail quickly, find solutions, and then move forward. So at Continuum, we're going to basically, okay, what I learned from NumPy and SciPy and watching it get deployed, what would I do differently? What kind of things would I do differently? And some of that is expressed in what I described before about data having mass. So the projects that basically encompass that reality is really three projects, Conda, Numba, and Blaze. SciPy really was a distribution, not a library. And so to do a distribution, you got to have a package. So we have to come up with a cross-platform, complete Python independent package manager called Conda. So that's why we've done that. And all of these are open source. NumPy is about making code as fast as possible. So it's just a little blurb on Conda. I wasn't going to mention it, but yesterday after my talk, somebody said you've got to talk about Conda. Keep talking about Conda because we love it. It's awesome. So if you're not using Anaconda or haven't tried Conda, download MiniConda. You don't have to get Anaconda. You can actually just do pip install Conda, Conda knit, if you want. And now you can do a Conda-based management of your packages without even downloading anything from our site. So it's completely open source, completely free. We communicate with the Python packaging authority with Nick and others to try to help understand how we integrate this even better. Blaze and Numba are our two open source projects. And really, they have a lot of, they have some dependencies, especially Numba has this LVM dependency. You want to be able to get it installed. So the idea around this is Blaze is motivated by generalizing this PEP-218 to all languages and datasets. Really creating this Python Glue 2.0 that glues things together in a marvelous way that makes what you, currently when you do data processing, if you store your data in SQL or you store it in HDFS or you store it in Postgres, that defines how you query it. And your query becomes some kind of convoluted version of the query language they've created for you. And that's how you have to use it. And a lot of us in data analysis, we love to use NumPy or Panus expressions because we like the way they feel. They fit our brain. They match the way we want to think about the problem. We like to use those expressions. Currently, the only way to do that is to pull the data to us in order to use those expressions. And so Blaze is about inverting that and creating expressions that then move to the data in multiple ways. Numba is motivated by the desire to not have to make people write extensions anymore and be able to write high-level code that is as fast or can be as fast as Fortran. And if that exists, then array-oriented computing can be done at full speed on modern processors with very little effort. And so that's the goal. So I'm not going to be able to cover all these slides. They're here. They'll be posted online. You can see them. But Blaze, its goal is to deal with data pain. Its architecture is divided up into an API. Deferred expressions are at the heart. And it's got data adapters and compute interpreters, basically. Compute interpreters to run on different backends. That's its architecture. And it uses a flexible architecture so you can easily add new ones, new data adapters, and new compute backends. The data descriptor is the data format approach. It allows you to have a uniform array-oriented interface to whatever data, to directories of CSV files, to a SQL database, HDFI files, to just JSON sitting on disk, directories of JSON files. Then the compute allows you to have a uniform interface to Dyn, which is our next generation NumPy. It's a C++ library and a Python interface to it. Pandas, even just Python, actually. You can run a compute on just Python lists of lists, or lists of dicks, or lists of tuples, just to see if everything is working right. Spark, PyTables, we do have support for Spark. Spark is a member of the Hadoop family. It allows you to run in memory on multiple machines. I said a lot of things about Hadoop. I don't like Hadoop normally, but Spark and Impala, I'm finally warming up to. They've saved the Hadoop ecosystem from my perspective. Blaze expressions, these are deferred evaluations. You basically create an expression, as I'll show an example, and that builds up a DAG, a directory-seq-with-graph, that describes that expression. Then the arrays in that graph can be referred to various data adapters. Then that gets sent to a compute, all separated. We've separated out compute from data, from code, and you reuse those components independently and then bring them together for an actual computation. Here's a simple example of counting web links. At the heart is what was missing from PEP 3.18, which is a really good data description language. In fact, I remember those discussions back and forth. We argued whether it would be NumPyD types or C types, specifications for data, or whether, and then Guido finally said enough, it'll just be a string, like the struct syntax. Boom. That's the data declaration language in the buffer protocol. Not quite good enough. The data shape we've created, we spent a lot of time on trying to figure out a data shape that can encompass all kinds of data. We'd love your feedback on this. It's a separate project. It's independent. It can be downloaded, installed independently of anything else. It's basically got some parsers for the data shape language. You can interpret it in various backends. You construct a table symbol, which is in this case a simple two-column table in the name and a node ID, and they have different types. String is actually Unicode. And then you have these two table objects. And then here's an expression. I'm joining these two tables together as a deferred evaluation and doing a group buy and counting. The load data is what distinguishes where I get the data from. There's different versions of load data depending on where the data is. But when I compute, notice I just compute on a dictionary that I've mapped. My data I got back from where I loaded it from to the actual variables that will be expressed in that dictionary, in that compute expression. And then that's when the computation happens. That's when you brought together the data and the code on a compute context. And the load data would be different depending on whether your data is in Spark and HDFS, or maybe it's in Pandas and a local disk. So you write that to load your data, but then your expression is completely separated. And you can have a very complex expression that looks Pandas-like, but then it is mapped over wherever your data is stored. So no longer do you have to write differently in order to have your data stored differently. And our goal is to end data silos, allow your data to be where it's optimally fits, where you can get the most performance and not have to change your code so much in order to use that best performance data set. Just write it back into Blaze and it will work beautifully. So Blaze has an ecosystem around it. We've been a lot of experimentation, a lot of exploring, trying to understand what we mean by this space. Currently, Dying, LiveDying and DataShape are the key pieces. And then the Blaze library itself, which has its different components. DataShape is that general data description language that I think was missing from PEP through 1.8. I'm very excited about this. I think it should have been what NumPy D types were. You can use it now. We use it in Blaze. Dying uses it as its data description language. Dying is a Python wrapper to a C++ equivalent of NumPy. The nice thing about that is you can bind that to Ruby, you can bind that to JavaScript, you can bind that C++ library to wherever you like, and have that multi-dimensional array concept across the board. So it can help with the gluing, again, for Python 2.0. All right, and I think I'm out of time. So I'm going to skip NumBa. I've talked about NumBa quite a bit, but NumBa is awesome. I love it. It's still growing. It's still pre-1.0. We still need help. We're looking for people to help us with it. I will just show you that CUDA Python works in NumBa. So you can actually, with NumBa today, target the GPU. If you have a GPU, very, very easily. So CUDA Python comes in NumBa. We're working on making interfaces to that that are more Blaze-like to make it much easier and less CUDA-specific. All right, so Python is a long and fruitful history in data analytics. It will have a long and bright future with your help. Join the PyTedic community and help make the world a better place. I want to dedicate my talk to Amy Olyphant, my wife. I don't know if she's here. She made it over. Thank you for all you've done. Nothing I've ever done would be possible if it weren't for you. Thank you very much. Thank you. Thank you very much. You're a little bit over time, but I think we have time for two questions. Please take the microphones. Thank you. Is it working? Yes. Thank you for the very enlightening talk. I had a question. I'm sorry about it, by PyPy and NumPy. You mentioned PyPy. My question is they are trying really hard to reimplement NumPyPy. But did they contribute? You mentioned that some rusty cards in NumPy weren't very optimized. Like you said, very controversial. Did they contribute back in order to be able to optimize these parts into NumPy? It's really hard because the stacks are so different. The code they write is quite different than what you write to make NumPy work. I am really excited about our number of ray objects. Our number of ray object writing has a lot in common with NumPyPy. I finally see a way to collaborate with them. I'm really excited by that because I always love to collaborate where I can. But sometimes it's challenging. Thank you. Hello. Thanks for the awesome talk and the insights about the history of NumPy and the ecosystem all around. My question is regarding the packaging. Because I looked up the PyPy index. NumPy still don't provide Python wheels for the wheel packages for Windows, which would be great because on Windows system it's always hard. The compiling is not as convenient as on Linux. Are there any plans to provide pre-compiled packages for Windows? I think I've heard of plans like that. Yes, I think people are talking about doing that. My attention is on... I mean, Konda install NumPy solves the problem. So I'm less motivated myself to worry about that. But I think there are some people that are trying to produce wheels. Yeah. So great. I'll try it out. Yeah, excellent. Thank you very much again. Thank you.
. .  Good morning, everybody. It's a big honor to announce our next  keynote speaker, Travis Elephant. He's a good friend of  mine. If you don't know him, he's also of Numpy. Numpy is  a scientific Python package. Pretty much everybody who's in  the scientific era in Python uses Numpy and he's the one who  wrote it. And he's also the CEO of Continuum Analytics, the  company behind Anaconda and a lot of other open source  packages, though he is really supporting the Python community.  And he's also a founding member of NumFocus. NumFocus is a  non-profit behind PyData. Speaking about PyData, that's a  great question. I'm going to talk about this in the afternoon  with tutorials. If you're interested in Python and PyData,  this is a topic Travis is going to talk about. There will be a lot  of talks and tutorials. We have much more depth about it.  So I encourage you to join the PyData conference. There are still  tickets available. So please go ahead and buy tickets. And now I  would like to give the word to Travis talking about some content  and future of this field. Please give a big hand to Travis.  APPLAUSE  Thank you very much. I think we're having a little technical difficulty here. So it  seems to be a common pattern.  Sorry.  So, can you show two displays?  It does.  So, do you have a picture here?  Do you have any picture?  We are now connected to VGA.  We are VGA, namely MIRROR.  We are now connected to VGA.  There we are.  You can see a very messy desktop and a bouncing beach ball.  Excellent. Maybe you will come back to me soon.  I don't know.  I'm still not live.  I don't have the control of my screen system.  I don't have it. It's a bouncing beach ball.  I don't know why you were swapping things out while I was doing things.  I don't know why you were swapping things out while I was doing things.  Hopefully that will work.  Are we there?  Thank you.  You all had time to catch up on your email.  Now you can give me your full attention.  Last minute email you got a chance to send off.  I understand. I like to check my email there in talks too.  I will try to keep your attention so you don't get too bored.  I want to talk a little about me.  Not too much.  I want to talk about the technologies I have been involved with.  Just to give you a little background.  I'm familiar with NumPy stack and science.  I'm a scientist by training.  My roots are in satellite scatterometry.  I used to measure wind speed over the ocean.  That got me into large scale data analysis.  You had backscatter from the ocean surface and satellites.  They came on big tape drives and used a VMS machine.  It was awesome.  It had a floating point format that was different than IEEE 54.  That's where I started.  We used to make really nice pictures.  I did some Perl.  I did some Matlab.  I used a lot of C in order to produce these kind of slides.  When I did my PhD program at the Mayo Clinic,  I got into a different kind of wave.  It was waves I was making inside of people.  Basically we'd wiggle people with speakers.  We'd start to shake them.  When you shake people, waves propagate inside.  Then with either MRI or ultrasound,  you can actually see those waves propagating.  From that, you end up with a big kind of an inverse problem.  It's that equation there which I like to scare people with.  I scared my committee with it too. It was fun.  It's not that hard of an equation.  It's just a simple linear equation.  Many of you here are pretty good at it.  My goal was to invert that.  I had to find the derivatives of five-dimensional data.  Here I was with a very large cube of data.  It was too big to fit in memory for Matlab.  The Matlab double.  There wasn't a Matlab float at the time.  I really liked working at a high level.  I could program in C,  but when I was thinking about my data problem,  I didn't want to be programming in C  because I had to think about pointers and arithmetic  and figure out where my memory leaks were.  I really liked that high level.  I searched around and I found Python.  The rest is history.  Basically I found Python started to do a lot with Python.  I did finish my PhD, although it was somewhat delayed.  Just to give you kind of a context  for Python's origins and where I started to use Python.  Guido released it.  There's arguably maybe a little before this,  but I know that 0.9.0 came out in February 1991.  I was not a Python user then.  I really came to the scene in 97.  So you can see that 96 is kind of the...  I used version 1.4 of Python.  It's a great version, actually.  In fact, if you'd like to try the 1.0 series,  you can make a new environment in Conda  and install the 1.0 version of Python just for fun.  You can see kind of how it worked  and do a test environment of Python 1.  I started using it in 97.  I have that little gold highlight.  Keep that in mind because that's an important date, 94,  with respect to data analysis in Python.  I'd like to show this slide because this is really  what got me going in writing C extensions for Python.  If I've done anything else,  I've written a whole bunch of C extensions for Python.  Much to the chagrin of the PyPy crowd.  Probably they're the number one enemy  because I've written so many C extensions  that make their job harder  in moving people off of the large platform.  But I'm not the only one that does it.  There's a lot of people that write C extensions to Python  because it's so easy to and because it's so extensible.  But I show this slide because of Mike Miller,  actually very close to my friend Mike Miller,  but Mike Miller, he released a package called TableIO,  and that's how I learned to program C extensions.  I grabbed his package, studied the source,  and then also read an essay by Guido about reference counting.  You've got to figure out how reference counting,  if you write extensions, fortunately if you write  Scython and other things, you don't have to do that anymore.  But if you really want to get to the root of things,  you have to do a reference counting.  So this opened my eyes to the power of open source.  I could basically look at the code,  understand it, learn something.  I learned a tremendous amount by reading this code.  And then I started to experiment with my own modules.  And in 1998 was my first extension module for Python.  It's called NumPyIO.  I've been embedded in other parts of the stack these days.  And in fact, there's better ways to do it.  But that was my very first module in 1998.  And I kind of got hooked.  And sort of that's when I, my career as a scientist,  sort of pivoted, let's say pivoted, that's the right word.  Pivoted into tools for scientists.  So in 1998, in 1999, I started to get really addicted,  got really caught by that bug.  I think there is a chemical compound that is called  Addiction to Open Source.  It's somewhat related to our addiction to Facebook.  I think it's connected.  But back in the time, I started releasing wrappers in 1998.  First FFTW, then C-Feeze modules, then stats,  which was from help from Gary Stringman.  He put out something in 1998.  And in 1999, I went back one year and looked at the mailing list.  It's really nice to go back and look at the history of what you said  in the past and cringe a little bit at how stupid you were.  But, you know, we all make mistakes.  But go back and kind of, and also see kind of how motivated you are  and how excited you were about something new.  And I was very excited about Python in 1999.  I said, hey, we could use Python to build a data analysis environment.  We could do all of our calculation code,  all the things I loved using a high level language like Matlab about.  I could do that in Python.  So that year, I went back and looked at the mailing list  and sort of every month, I was sort of saying,  hey, here's a new package I just made.  And of course, it wasn't very pretty.  In the web page for it was very ugly.  I'm still not a very good web designer.  I'm not a good content but not really good pretty pictures.  So if you look at those, back in the day,  nobody else had pretty pictures either, so it was okay.  I just had a really Spartan website.  And I made a bunch of releases.  Then a guy named Perot Peterson came along and said, hey,  this is really stupid of you,  handwriting Fortran wrappers to multi-pack on all the libraries on Netlib.  I'm going to write a tool to do that.  That's when I first learned the difference between me and a real computer scientist.  Right? Me, I'm like, I'll do this manually  because I just want to get it done.  And a real computer scientist goes, we have to automate this and make it.  And in fact, you know you're a real computer scientist,  you'll spend more time automating than it would have taken to do it manually.  Right?  And now that's not the case.  F2Py was a tremendous tool and we worked together  basically through the last part of June, 99 to that year.  So 1999, that was sort of multi-pack, it was called the time.  And that was my, I put it on the web and people started to download it.  I got started to get to know people from all over the world.  Piero Peterson, the gentleman who wrote F2Py is from Stonia.  I don't know if he's here, he could be here, but he's kind of,  yeah, I know he likes to water ski in the lakes in Estonia.  That was just a tremendous rush to be able to coordinate and communicate with people all over the world  and see them use your stuff and contribute back and make it better.  It was an amazing thing.  And it was my first taste of open source community.  And I've just seen that grow and grow and grow ever since.  You could use the Python for data analysis way back in 2000.  Whereas I wrote NumPy, NumPy came from a tradition,  it came from a history that had started in 1994.  And here in 2000, I could use it to publish pictures in my thesis.  Now this is using a Python interface with something called Dyslin.  Anybody here use Dyslin? Anybody?  Dyslin is actually a pretty sophisticated tool, although a little bit tedious to use.  But I was able to do it, publish all my pictures in my thesis.  I think there was 200 different images in my thesis, all with Dyslin and with Python.  Now in 2001, that's when Eric Jones contacted me and together with Piero Peterson,  we kind of pulled stuff together and spent a whole bunch of time building for Windows.  A whole bunch of time essentially going through the bug, building a package.  And we came out with what we called the SciPy library, but really was the SciPy distribution.  And I didn't realize at the time, but that's really what it was.  And it was a collection of all these tools together kind of with a single installer,  so you could get everything up and running quickly.  There were a lot of people who contributed to getting SciPy out the door.  But it was a lot of work, just on pulling everything together and doing a lot of...  I was not a Windows program at the time.  I've since learned to make my piece with the Windows platform, and it's fine.  There's some interesting things about it.  Actually in the parallel space, there's some really interesting things about it.  But as I said before, NumPy really inherited a long legacy of great minds that had gone  and tried to build numeric array object for Python.  Jim Fulton, he's embarrassed when I show this because it was just a Python matrix object,  but it was real.  It was part of the early discussion, and it caused Jim Hugenin as a graduate student at MIT  to get really excited and to sort of procrastinate his graduation in order to write numeric,  which became the foundation of...  And that's why I came to Python because numeric existed, so I'm really grateful for that.  Then in 2001, some features were desired for numeric and Perry Greenfield,  Rick White, Todd Miller at the Space Science Telescope, the folks that put out Hubble  and processed the Hubble images.  They needed some changes, most particularly, they needed to be able to memory map  to more diverse data, and so they were writing numeric.  And then at that time, I kind of said, well, it turned out actually I didn't have a class to teach.  It was really kind of a nice, confluence event.  I didn't have a class to teach because the one that was scheduled, nobody signed up for,  it was sort of a build an MRI, and only one person signed up.  I guess they might have been too intimidating.  It was smart, actually, not to sign up for that class.  But I ended up without a class to teach, and I probably should have been publishing papers  in order to keep my position as an academic professor.  But at the time, I saw numerae, I saw sci-pi, which is built on numeric,  and I saw modules getting built for numerae, and I saw this split happening in the community.  It was already so nascent, and people were still struggling just to support one thing.  And I felt like I had to do something about it.  It was just a really strong feeling that somebody has to do something about this,  and I don't know who's going to, because nobody else knows the code base very well.  And I had time, so I did.  I just did it and said, okay, throw caution to the wind and dive in.  And basically spent about what I thought would be about a three-month project,  turn it into an 18-month project, in order to kind of put the first version of NumPy out there.  Since about 2007, the community, it took a while for the community to get excited about it  and get more contributors, but 2007 came and it really started to take off,  and lots more people joined.  There's still room for people, especially at the low level.  The number of people that can understand the CAPI of Python  and then help maintain the NumPy code base, which is in C, is shrinking, right?  And that becomes a challenge.  But there is a NumPy and SciPy both now are a very impressive community effort,  and I'm really grateful for that, because they wouldn't be what they are  without all the people that are contributing and making it possible.  I don't know exactly how many NumPy users there are.  I estimate maybe 3 million.  That's on the basis of hits to a web page, download numbers,  but it's always hard to tell, because nobody writes home, nobody sends a postcard,  so you never really know how many users of NumPy you have.  I wanted to kind of just pause a little bit and maybe to motivate some of you  who I know are building your own software packages and communities,  talk a little about the things that I've learned, about what it takes to do this,  because if Python is going to have a role in big data analytics,  it's going to take the work and effort of a lot of people, just as it has to date.  So one of the things that's most important, I think, is to recognize that it is hard work initially.  It's not easy. And usually it's quite lonely, actually, when you start off on a new venture.  Initially, nobody really believes in your idea except you, and that's really the way it should be.  Others will need some proof that this is actually going to work before they dive in.  When I said, hey, I'm going to merge NumRay and NumRoc and do it on the NumRoc code base,  a lot of people said, that's great. Go do that. It'll be fine.  And it was really until they start seeing results, and they say, oh, this is actually going to work.  Okay, now we'll dive in. And very gratefully do that as well.  But that's going to be how it works. You're just going to have to dive in and do some things,  maybe with a small team, maybe you and another person.  And in fact, the more complicated what you're doing is, the more lonely it's going to be,  because the fewer people that will understand enough to be able to make the trade-offs and help you.  So SciPy was another example. I mean, I started SciPy,  and it took a while before we got more and more help.  And Piaro joined basically the end of the year.  I procrastinated my PhD by at least a year to create the beginnings of SciPy.  And don't tell my wife, right, she's in the room, so I shouldn't have said that.  I was a poor struggling graduate student with three kids, and we were making 18,000 a year in Minnesota.  And, you know, we did great. We did great.  But I know she was thinking, when are you going to finish, honey, so we can actually have a job.  Piaro Peterson put in tremendous work to create F2Py and SciPy Lin-Alge.  I was just flabbergasted by the amount of work he put in.  In fact, one of my great anecdotes from Piaro is when I sent out my multi-pack,  and he basically submitted a make file.  It's still an incomprehensible make file. I have no idea what it said.  But it was amazing. It sort of built everything, including, I think, made the coffee in the morning.  People will do that. And he put in tremendous amount of work.  It makes that kind of work to get things off the ground. It really does.  You can't just sort of show up and hope that things happen. You really have to put in the work.  The other thing I think is important is you do what's right.  In other words, you've got to put aside kind of thoughts of, oh, I'm going to make a lot of money,  or I'm going to be able to do something really cool.  Because doing what's right means you have your information, you have your knowledge,  you have your semantic environment, the things you know about,  and those things come together for you to feel, hey, this is what I think is right.  And you're the only one that has that feeling, and you need to own it and do something that's right.  Timing is everything. Sometimes you're the right person for the job.  Sometimes it's the right time for that to happen.  And that's hard to know when is the right time.  But when the time is right, you have to have an urgency. It can't wait. It won't wait.  People are impatient. If things don't move in a certain direction,  they'll wander off and do something else after a little bit of time.  You have to strive for excellence. And it really takes work.  And you have to give the best you have. It'll never be enough.  But give your best anyway. I love this quote from Mother Teresa.  There's actually a whole series of these aphorisms.  The good you do today will often be forgotten. Do good anyway.  It's absolutely true. And it's essential. It's one of those life qualities that I think I would love to grow into someday.  Another key to a success is while it's initially lonely and you're doing things on your own,  you have to build a community if it's going to have long-term success.  You have to get others involved. And that means interacting with people.  And that means having your ideas be put up and shut down.  It basically means you'll need to help other people. Other people have different ideas than you.  You have to give up some of your hard-won ideas. You have to give up some of your ego.  Someone will point out how bad you suck. And you probably do in some places.  And it's okay. Listen to that.  And that's the only way to make progress in the community is for people to come together  and have some empathy towards each other.  It does expose you. You know, to treat other people the right way, you have to care about them.  And that does expose you because you can't get hurt. But you do it anyway.  And then you keep moving forward. And it's a great thing.  Much more we said on this topic. I think the key to building a community,  there's a lot of lessons to be learned from all kinds of projects.  I'd love to talk to some of you about your lessons of how you build communities.  Then patience and a little bit of luck also. Good things take time.  It doesn't happen right away. NumPy took a long time to get real adoption.  2007, you know, six is sort of two years afterwards.  It really was 2009 before I started seeing a lot of adoption of NumPy.  So it's four years later. And the right factors have to come together.  One of those factors, for example, in the NumPy sci-fi communities was GitHub.  For a long time, the number of contributors just sort of creeped along.  And then the ease in which you could contribute jumped.  And all of a sudden, we got more contributors.  All right. So that's a little bit of history of what I did in the past with NumPy and sci-fi.  So what is this NumPy anyway? Perhaps a show of hands.  Who uses NumPy or knows what it is?  Okay, a fair number of you. Excellent.  You know, the Europeans are always smarter than the...  I asked that question in the U.S. and I don't quite get the same response.  There's maybe half of the room. But here, everybody at least has seen it.  Now, you all rejected it, of course, and do something else.  But that's another beauty of the European.  But what NumPy is essentially an array-oriented extension.  It's got an array object and fast operations on an array object.  It's fairly simple, actually, it's core.  Here's just simple examples.  You can build a two-dimensional array and do some computations on it.  And you can sum along different axes.  Or you can have a three-dimensional array.  You can have an n-dimensional array.  And that's the idea of organizing data together in a way that can be operated over quickly.  And by quickly, we mean you hand it over to a precompiled loop  or a precompiled engine that does the computation.  And so that's not happening in Python.  So when you work with NumPy arrays, you can release the GIL.  You are releasing that global interpreter lock.  And you don't have the same problems that happen if you're not using a NumPy array.  So here's just a diagram of what it looks like as a Python object.  Each element of the array has to be exactly the same type.  That's one restriction of a NumPy array.  So it has to be the same number of bytes as its basic restriction.  But that bytes can be a lot of different things.  It can be a pointer to a Python object.  It can be a structure with an int, a float, and 10 bytes of strings.  It can be Unicode.  UTF32 is always what Unicode is in NumPy.  And then there's these array scalar things.  It'll also creep up and bite you on occasion  if you're trying to understand what comes out of a NumPy array.  So fairly straightforward.  Once about three years ago, four or five years ago now,  somebody asked me to come up with a Zen of NumPy.  If you don't import this, you all know Tim Peter's aphorisms that have come up.  And very, very nice.  And this is definitely not at that same caliber,  but it gives a little bit of a flavor of how I think about NumPy.  Strided better than scattered. Contiguous better than strided.  Descriptive is better than imperative.  Array-oriented is better than object-oriented.  Perhaps a little debatable. We might...it's a fun one.  Broadcasting is a great idea.  And vectorize is better in explicit loop,  unless it's too complicated.  And then you can use scython or numba.  And think in higher dimensions to solve your problems.  Real quick kind of example of array-oriented computing.  I like to show this. It's a little bit of a cheesy example,  but the Fibonacci numbers are so common in Python, we like to show them.  Of course, here's the Python implementation.  One bad one, one better one in terms of performance.  Now, if you compare the Python approach to Fibonacci  versus, okay, I'm an array-oriented guy.  I use the NumPy stack. What do I do?  Well, of course, I reach the solution  and find the roots of the discrete difference equation  and just compute them for any n.  So that's a vectorized computation here, Fib2a.  I just generate a vector of numbers n,  and then I calculate with the root r1 and r2.  I can use the roots command to take the roots of a polynomial,  calculate the power of those roots, subtract.  Those are array expressions happening.  I don't see any for loops there,  but for loops are happening under the covers.  That's the concept of array-oriented computing,  is gathering your data, doing high-level computations on them all at once.  And then, if you're really clever,  you understand that it's basically the output of an unstable filter,  and I can use the linear filter tool in SciPy  to generate the output of at least the first part of the Fibonacci sequence.  Those who have done this in the room will also understand  that I'll have overflow if I use the floating point of the machine  like I'm doing with SciPy and NumPy.  But you can get faster performance.  That's one of the benefits of array-oriented computing,  is you immediately typically get faster performance.  And so that's usually what people reach for,  and why they reach for array-oriented computing,  is to get the fast performance they're looking for.  There are other reasons to do it, however.  APL really was the father of array-oriented languages.  It's been around since 1964, but the hieroglyphics of APL  are still trying to be decoded.  We have not found the Rosetta Stone yet  to understand what people actually said  in all these wonderful array-oriented codes.  Just kidding, actually.  I know some people can read APL.  And there were other English versions of that same concept brought in.  Had a lot of the same ideas, and NumPy is a descendant of APL.  All right, so another simple idea of array-oriented computing  is to gather your data together,  whereas a lot of object-oriented approaches end up scattering  your memory all over the place, objects versus attributes.  If you gather that all together and make objects essentially rows  and a table of attributes,  you're going to be able to do column-oriented processing.  Your data is all together, and your modern processors  can scream through this.  Array-oriented computing is perfectly suited  and matched to the vector computers of today,  the multi-core, the multi-CPU.  So whenever you can do that, you get the added benefit  of actually being able to take advantage of that hardware  that's otherwise not really exposed very well  in the languages of today.  So I've talked about those benefits.  I'll move on. I'm going to skip this example,  put it up briefly just so you get a feel for the kind of code.  Once it was awesome, somebody tweeted,  here's a problem, and I said, oh, I think I can solve that  rather than spend time on company stuff.  So I had a good time just to play with this problem,  and this is what I came up with, to basically find a circle  out of this roughly circle-like image.  So that's the kind of thing you can do.  Now NumPy has had a story in data analytics for a long time  because of these structured arrays.  I said briefly before that every element of a NumPy array  can be an arbitrary structure.  It can be integer, float, whatever.  So you can think of a one-dimensional NumPy array  as a table, as an Excel table, and it's a really nice mapping.  However, it's an array of structures,  which sometimes isn't the optimal data structure  when you're trying to say add new columns quickly  or do computations down the columns.  So even though it works, it's not as flexible as we'd like,  and PAN has emerged as basically this generic structure of arrays  where it basically has pointers to different arrays  under the covers.  So it sits on top of NumPy,  and it has a lot more user-friendly tools  for people doing data analysis.  So whereas behind the past, when you were using NumPy  to do data analysis, you might have to write  five to ten lines of code with PAN as its one,  or a method call, and it's quite a bit simpler.  So a lot of people have come to the Python data community  because of PAN as this list basically comes from a user of PAN  as who says, this is why I love PAN as a few of these reasons,  and then I modified it slightly.  So currently, today, in big data analytics,  Python, this is the basic key libraries,  and I might be missing a few here,  but these are sort of the basic ones.  NumPy, PsiPy, PAN as MapLotLib, Ipython.  The list goes on, it becomes quite a stack,  so when you're sitting there, I want to use Python  for data analysis, you have to get a bit of stuff together  to make that happen.  That's really why we created Anaconda and Kanda.  Some of you may be interested to note  that whereas a lot of people have been using R  for data science, Python is rapidly creeping  as an equal footing for data analytics with R.  A lot of people, this is a recent survey done at O'Reilly,  and they surveyed the people of Tendon Strata in 2012 and 2013.  This is a revelation to O'Reilly as well, I think,  because they've been really searching  for people to write books on Python for data analysis.  I think also Wes McKinney's book was successful,  and that really opened the floodgates as well.  They're really, really, as a market here, we should get books.  I've been asked to write a lot of books,  I have no time to write any books,  I'm a little bit more, I handed that to a few other people,  maybe we'll get a book out of Continuum.  We also see articles like this.  I don't want to get into language wars,  I think actually we can work together with the R community,  but it just goes to show that it becomes kind of a choice.  You can do everything you need to do in Python pretty much,  and occasionally you need to call out to R and you can do that.  Python is growing as the top language in schools.  Many of you have seen this, but I like to show it.  We're in a Python conference.  We should kind of celebrate the fact that Python is being used forever,  lots of places.  This is US schools, top universities.  Python is the number one introductory class being taught.  Now some will say that's how languages go to die,  so maybe it's not such good news.  Right now we're going to get a whole lot of people using Python  who have no idea what they're doing,  but I trust that our community is vibrant enough and robust enough  to welcome them in, train them up,  actually unlearn the things they might have learned wrong in school  and help move the community forward.  So I have here, I know I'm running out of time,  I've had plenty of slides, plenty of things to talk about,  but I wanted to talk a little bit about why I think Python is fantastic  for technical computing.  As I said, I was a domain expert, a data scientist,  a scientist coming to Python, and I had reasons for it.  Some of those were the same as they are today.  One, syntax. It gets out of your way.  You don't have to learn jargon and concepts.  It basically leverages my English language data centers.  Perhaps there will be a language that leverages Mandarin data centers  in the future. I won't benefit from it, but others will.  This leverages those Latin character centers.  White space. I love white space, the fact that it conveys intention.  I'll tell you why, because my field of view is limited.  I have limited horizontal and limited vertical real estate that I can see,  and I have to understand something in that limited space.  If I'm using that up with braces and brackets and things that are unnecessary,  it's just waste for me.  It's also why if I have long, long paths, long, long dot paths for things  and my variable name takes up the whole screen, I'm in trouble.  I'm not a big fan of that either.  Complex numbers were built in early.  Overloadable operas were built in early.  This is a mistake Java made for the scientists.  Scientists need complex numbers.  The FFT is the reason that electrical engineers have jobs,  and it immediately has the FFT.  You've got to have a complex number, or you'll have 200 of them,  and nobody will be able to agree on what it should be.  Just enough language support for arrays, which is the brackets,  the ability to have commas go immediately to tuples,  so you don't have to have that funny indexing.  These things were added actually at a critical time.  I definitely have to thank the Conrad Hinsons, the Paul de Bois, Jim Huguenin,  who worked with the Python devs, Guido and others,  to make sure these were added in language in an early time.  It's been fantastic.  Occasional programmers can understand it.  Occasional programmers are the people like I was,  who don't have to solve this five-dimensional differential equation  and don't want to spend time chasing pointers,  but I need to be able to see code and read it and understand it.  And Haskell or Clojure is too much to put in my head.  And remember, so Python works perfectly in that space.  And I used to say that packaging was a problem with Python.  I no longer say that because packaging is awesome with Kanda.  Kanda makes your packaging problem go away.  It's fantastic.  We get that feedback from users all the time.  So I'm not just saying it because we put it out.  I'm saying it because we get that feedback from people.  And I use it and I love it.  It solves exactly the problem I've seen  with not being able to get everything installed easily and quickly.  So lots of great things about Python, simple, extensible implementation.  It's general built.  You can build a system, general purpose,  supports multiple programming styles, all these things you know.  But a critical one is that it does have critical mass,  because you could have the ideal language,  but I don't have people using it and you'd be stuck.  You could not build a community.  That's the hard thing.  And that's a bit of a chaotic question about when that will happen.  I can't give you answers.  It's sort of one of those emergent phenomena.  Now, there's things I don't like about Python.  And we could all probably rag on it together.  But some of these are being addressed.  I would love to see anonymous blocks.  I'd love to see the ability to have anonymous chunks of code  that you could then send around places.  Really for deferred evaluation is the most common use case for me.  I'd love to be able to do slice syntax outside of the brackets.  Please just let a slice syntax be able to create the slice operator.  I use that a lot as a ray-oriented programming guy.  To see Python run time to gil, global variables inside,  lack of dynamic compilation, there's some work to be done there.  That's a really hard one.  I would love to see some language extension other than import hooks.  I've seen a lot of very creative uses of import hooks  to have kind of DSLs that can be imported into Python.  It's kind of cool actually.  You can extend Python as you like with the import statement.  And it can be hard using a general purpose language  because the devs of that language don't necessarily understand your use case.  It's a ray-oriented programmer.  And I'll have a story about that in a little bit about PEP 3.1.1.8.  So NumPy, I like it.  It's good. It's got a lot of good things.  But there's got a lot of problems with it too.  The D-type system, the data-type system,  which essentially allows the structure to raise,  it's too limiting and difficult to extend.  It's really, it's more, it grew out of Numeric's data descriptor  that was there at the beginning.  And it kind of extended it just far enough,  but it needs to be kind of overhauled.  The immediate mode, creating huge temporaries all the time  when you have an equation to evaluate.  It's almost an in-memory database, really close.  If you're using SQLite, you can also use NumPy for the same purpose.  It's going to be faster actually, but it's not quite.  It doesn't really evaluate some of the operations that you would like.  Lots of unoptimized parts.  Lots of embarrassingly unoptimized parts, actually.  If you start giving the code, you'll say,  who's the idiot that wrote this?  Hopefully the blame's not me anymore,  because somebody's changed the comments or something.  The code base is organic and hard to extend.  I think, as I reflect on back to the history,  I think one of the most important pieces of work that I did in 2005-2006  was actually sit down with Guido.  I remember flying to San Mateo where he was working,  and I said, I'm going to have lunch with Guido  and get NumPy into Python.  I saw this little professor and think, I can go do this.  He's a very nice guy, very accommodating.  I went there with Paul DeWaughn.  We sat and we had lunch and we talked about what can we do  to get NumPy into numeric or the new numeric at the time  when we were calling it into Python.  He cautioned about, well, you know,  if you get into Python and you have an 18-month release cycle,  it won't be able to be updated very quickly.  There's some downsides too.  That was enough to go, yeah, we probably ought to not do that.  We want kind of the structure of the NumPy rate in Python.  And so I spent time writing this PEP, PEP 3118,  to really extend the buffer protocol.  Now raise your hands if you know what PEP 3118 is  or know what the buffer protocol is.  I should see a fewer hands because this is one of those  sort of underbellies of Python, where the ability,  it's kind of on the lowest level,  but it's really the ability for arbitrary objects to share data.  And that's what the buffer protocol allowed that initially,  but it only allowed you to share a single point of data  about the data. You couldn't share that it was really an array.  It had this kind of data type in it.  So the extended buffer protocol was really all about getting  more metadata around the pointer to memory that was being shared.  It really makes possible a heterogeneous world,  a powerful array like objects.  So it isn't really necessary for there to be only one NumPy.  There really could be a lot of array like objects  that share memory and they could really operate independently  and coexist.  I think adding multiple dispatch to language  would actually improve that better.  A multiple dispatch library would actually make that a heaven.  At that point, you just need to pep through an eight multiple dispatch.  We don't need a single array library like NumPy.  So that's what I think of the future of NumPy  and the future of the world.  I think of a heterogeneous world, a world with a lot of things working.  The buffer protocol exposes this idea  and I don't have time to talk about it today.  Someday I'll maybe give a talk about this.  I don't feel always qualified to do this.  I'm a scientist by training and I'm always kind of an approaching  scientist, software developer by effort and trade  and learning from other people.  But there's something real about this dual,  the dual of encapsulation, the buffer protocol exposes.  These data types, they sort of invert that idea  of having data and having methods attached to that data.  The data is exposed and you talk about it in full terms  like a schema and then you throw code past it.  I'll talk a little bit about that,  about I think what it allows for us to do in the future.  So what of the future?  We've talked about today, we've talked about a little bit of the past.  What's going to happen in the future?  Well, I've watched the Star Trek episodes  so I think I know the future and I'm going to be able to tell you what's going to happen.  I'll just describe what I'd like to see  and some of the principles that I think that will guide the future  but we can't ever tell.  We can't ever tell what's going to happen.  One of the things that's very real and is based in physics  is the idea that data has mass.  Data is growing faster than speed of light  can carry that data from one point to another.  Therefore that means you're going to want to have data sit where it is  and that's true whether it's in a GPU  or it's in a memory cache or whether it's on a cluster somewhere.  You don't want to be pulling data.  So what that means is a lot of our systems that were built around the idea  of encapsulation and serialization are actually wrong.  They don't work very well when we do that.  And so we have to kind of think differently about how we're going to manage this.  There's a blog about this as well.  It's a well-known observation.  Data gravity some guys even invented.  You can even talk about the formula for data  and how it attracts each other.  I don't know, maybe that's useful.  I think it is useful to think differently in a relativistic sense.  Normally we think about ourselves on a platform while data moves past us.  We kind of serialize data into our objects  and then we do our little computation and then we serialize them on their way.  But when data has mass that's really expensive.  It turns out our computations are really simple.  Our machines can rapidly do the computation  and wait all day long for the data to pipe through.  So how do we invert our thinking about this  and think about it from a data-centric perspective  where the code comes to the data and flows through the data?  So that's one thing that I think about.  And part of what's in PEP 318, that buffer protocol,  has some of the answers I think.  At least some of what I've been able to see.  Some of you I'm sure will be able to think even more deeply and better about that.  And I'd love to get your feedback and your inspiration.  I think fundamentally the future of big data in Python is going to be heterogeneous.  While as before we've had this notion of here's NumPy  and everybody uses NumPy as a single kind of channel.  NumPy really is just a description. It's a protocol.  I love the talk on Tuesday by Pieter, the zero and Q author,  who talked about the decentralized role in the future  and contracts being the most important thing.  PEP 318 is an example of that kind of protocol or contract between objects.  And it's a beautiful thing. It's an important thing.  And I think that's what the future is going to be like as well.  Much more of that rather than here's a single library that everything sits around.  And then what Python, its role is going to be doing what it's always done really well.  And that's playing this tremendous glue,  this tremendous ability to just stick things together very, very quickly.  That's the advantage of not having static types is you can pull things together  from all sorts of places in an agile fashion, in an iterative fashion,  fail quickly, find solutions, and then move forward.  So at Continuum, we're going to basically,  okay, what I learned from NumPy and SciPy and watching it get deployed,  what would I do differently? What kind of things would I do differently?  And some of that is expressed in what I described before about data having mass.  So the projects that basically encompass that reality is really three projects,  Conda, Numba, and Blaze.  SciPy really was a distribution, not a library.  And so to do a distribution, you got to have a package.  So we have to come up with a cross-platform, complete Python independent package manager called Conda.  So that's why we've done that.  And all of these are open source.  NumPy is about making code as fast as possible.  So it's just a little blurb on Conda.  I wasn't going to mention it, but yesterday after my talk,  somebody said you've got to talk about Conda.  Keep talking about Conda because we love it. It's awesome.  So if you're not using Anaconda or haven't tried Conda, download MiniConda.  You don't have to get Anaconda.  You can actually just do pip install Conda, Conda knit, if you want.  And now you can do a Conda-based management of your packages  without even downloading anything from our site.  So it's completely open source, completely free.  We communicate with the Python packaging authority with Nick and others  to try to help understand how we integrate this even better.  Blaze and Numba are our two open source projects.  And really, they have a lot of, they have some dependencies,  especially Numba has this LVM dependency.  You want to be able to get it installed.  So the idea around this is Blaze is motivated by generalizing this PEP-218  to all languages and datasets.  Really creating this Python Glue 2.0 that glues things together  in a marvelous way that makes what you, currently when you do data processing,  if you store your data in SQL or you store it in HDFS or you store it in Postgres,  that defines how you query it.  And your query becomes some kind of convoluted version of the query language  they've created for you.  And that's how you have to use it.  And a lot of us in data analysis, we love to use NumPy or Panus expressions  because we like the way they feel.  They fit our brain.  They match the way we want to think about the problem.  We like to use those expressions.  Currently, the only way to do that is to pull the data to us  in order to use those expressions.  And so Blaze is about inverting that and creating expressions  that then move to the data in multiple ways.  Numba is motivated by the desire to not have to make people write extensions anymore  and be able to write high-level code that is as fast or can be as fast as Fortran.  And if that exists, then array-oriented computing can be done at full speed  on modern processors with very little effort.  And so that's the goal.  So I'm not going to be able to cover all these slides.  They're here.  They'll be posted online.  You can see them.  But Blaze, its goal is to deal with data pain.  Its architecture is divided up into an API.  Deferred expressions are at the heart.  And it's got data adapters and compute interpreters, basically.  Compute interpreters to run on different backends.  That's its architecture.  And it uses a flexible architecture so you can easily add new ones,  new data adapters, and new compute backends.  The data descriptor is the data format approach.  It allows you to have a uniform array-oriented interface to whatever data,  to directories of CSV files, to a SQL database, HDFI files, to just JSON sitting on disk,  directories of JSON files.  Then the compute allows you to have a uniform interface to Dyn,  which is our next generation NumPy.  It's a C++ library and a Python interface to it.  Pandas, even just Python, actually.  You can run a compute on just Python lists of lists, or lists of dicks, or lists of tuples,  just to see if everything is working right.  Spark, PyTables, we do have support for Spark.  Spark is a member of the Hadoop family.  It allows you to run in memory on multiple machines.  I said a lot of things about Hadoop.  I don't like Hadoop normally, but Spark and Impala, I'm finally warming up to.  They've saved the Hadoop ecosystem from my perspective.  Blaze expressions, these are deferred evaluations.  You basically create an expression, as I'll show an example,  and that builds up a DAG, a directory-seq-with-graph, that describes that expression.  Then the arrays in that graph can be referred to various data adapters.  Then that gets sent to a compute, all separated.  We've separated out compute from data, from code,  and you reuse those components independently and then bring them together for an actual computation.  Here's a simple example of counting web links.  At the heart is what was missing from PEP 3.18, which is a really good data description language.  In fact, I remember those discussions back and forth.  We argued whether it would be NumPyD types or C types, specifications for data,  or whether, and then Guido finally said enough, it'll just be a string, like the struct syntax.  Boom.  That's the data declaration language in the buffer protocol.  Not quite good enough.  The data shape we've created, we spent a lot of time on trying to figure out a data shape  that can encompass all kinds of data.  We'd love your feedback on this.  It's a separate project. It's independent.  It can be downloaded, installed independently of anything else.  It's basically got some parsers for the data shape language.  You can interpret it in various backends.  You construct a table symbol, which is in this case a simple two-column table  in the name and a node ID, and they have different types.  String is actually Unicode.  And then you have these two table objects.  And then here's an expression.  I'm joining these two tables together as a deferred evaluation and doing a group buy and counting.  The load data is what distinguishes where I get the data from.  There's different versions of load data depending on where the data is.  But when I compute, notice I just compute on a dictionary that I've mapped.  My data I got back from where I loaded it from to the actual variables that will be expressed  in that dictionary, in that compute expression.  And then that's when the computation happens.  That's when you brought together the data and the code on a compute context.  And the load data would be different depending on whether your data is in Spark and HDFS,  or maybe it's in Pandas and a local disk.  So you write that to load your data, but then your expression is completely separated.  And you can have a very complex expression that looks Pandas-like,  but then it is mapped over wherever your data is stored.  So no longer do you have to write differently in order to have your data stored differently.  And our goal is to end data silos, allow your data to be where it's optimally fits,  where you can get the most performance and not have to change your code so much  in order to use that best performance data set.  Just write it back into Blaze and it will work beautifully.  So Blaze has an ecosystem around it.  We've been a lot of experimentation, a lot of exploring,  trying to understand what we mean by this space.  Currently, Dying, LiveDying and DataShape are the key pieces.  And then the Blaze library itself, which has its different components.  DataShape is that general data description language  that I think was missing from PEP through 1.8.  I'm very excited about this.  I think it should have been what NumPy D types were.  You can use it now. We use it in Blaze.  Dying uses it as its data description language.  Dying is a Python wrapper to a C++ equivalent of NumPy.  The nice thing about that is you can bind that to Ruby,  you can bind that to JavaScript, you can bind that C++ library to wherever you like,  and have that multi-dimensional array concept across the board.  So it can help with the gluing, again, for Python 2.0.  All right, and I think I'm out of time.  So I'm going to skip NumBa.  I've talked about NumBa quite a bit, but NumBa is awesome.  I love it. It's still growing. It's still pre-1.0.  We still need help. We're looking for people to help us with it.  I will just show you that CUDA Python works in NumBa.  So you can actually, with NumBa today, target the GPU.  If you have a GPU, very, very easily.  So CUDA Python comes in NumBa.  We're working on making interfaces to that that are more Blaze-like  to make it much easier and less CUDA-specific.  All right, so Python is a long and fruitful history in data analytics.  It will have a long and bright future with your help.  Join the PyTedic community and help make the world a better place.  I want to dedicate my talk to Amy Olyphant, my wife.  I don't know if she's here. She made it over.  Thank you for all you've done.  Nothing I've ever done would be possible if it weren't for you.  Thank you very much.  Thank you.  Thank you very much.  You're a little bit over time, but I think we have time for two questions.  Please take the microphones.  Thank you. Is it working?  Yes. Thank you for the very enlightening talk.  I had a question. I'm sorry about it, by PyPy and NumPy.  You mentioned PyPy.  My question is they are trying really hard to reimplement NumPyPy.  But did they contribute?  You mentioned that some rusty cards in NumPy weren't very optimized.  Like you said, very controversial.  Did they contribute back in order to be able to optimize these parts into NumPy?  It's really hard because the stacks are so different.  The code they write is quite different than what you write to make NumPy work.  I am really excited about our number of ray objects.  Our number of ray object writing has a lot in common with NumPyPy.  I finally see a way to collaborate with them.  I'm really excited by that because I always love to collaborate where I can.  But sometimes it's challenging.  Thank you.  Hello. Thanks for the awesome talk and the insights about the history of NumPy and the ecosystem all around.  My question is regarding the packaging.  Because I looked up the PyPy index.  NumPy still don't provide Python wheels for the wheel packages for Windows,  which would be great because on Windows system it's always hard.  The compiling is not as convenient as on Linux.  Are there any plans to provide pre-compiled packages for Windows?  I think I've heard of plans like that.  Yes, I think people are talking about doing that.  My attention is on...  I mean, Konda install NumPy solves the problem.  So I'm less motivated myself to worry about that.  But I think there are some people that are trying to produce wheels.  Yeah. So great. I'll try it out.  Yeah, excellent.  Thank you very much again.  Thank you.
Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_0_383.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_384_2507.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_2508_2987.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_2988_3047.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_3048_5723.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_5724_6335.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_6336_6563.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_6564_9071.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_9072_9527.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_9528_9731.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_9732_9851.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_9852_12455.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_12456_21839.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_21840_25835.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_25836_30947.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_30948_32363.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_32364_36755.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_36756_37451.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_37452_39875.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_39876_39971.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_39972_43475.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_43476_45131.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_45132_46595.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_46596_46703.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_46704_48011.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_48012_49295.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_49296_50423.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_50424_50699.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_50700_55607.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_55608_55907.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_55908_56315.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_56316_57431.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_57432_58367.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_58368_59795.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_59796_61667.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_61668_62063.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_62064_66119.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_66120_69563.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_69564_70835.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_70836_71459.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_71460_72383.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_72384_72839.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_72840_74423.png Travis_Oliphant_-_Python_s_Role_in_Big_Data_Analytics__Past__Present__and_Future_74424_74611.png