Getting more out of Matplotlib with GR
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Heinen, Josef
Josef Heinen - Getting more out of Matplotlib with GR Python is well established in software development departments of research and industry, not least because of the proliferation of libraries such as _SciPy_ and _Matplotlib_. However, when processing large amounts of data, in particular in combination with GUI toolkits (_Qt_) or three-dimensional visualizations (_OpenGL_), Python as an interpretative programming language seems to be reaching its limits. In particular, large amounts of data or the visualization of three- dimensional scenes may overwhelm the system. This presentation shows how visualization applications with special performance requirements can be designed on the basis of _Matplotlib_ and _GR_, a high-performance visualization library for Linux, OS X and Windows. The lecture focuses on the development of a new graphics backend for _Matplotlib_ based on the _GR_ framework. By combining the power of those libraries the responsiveness of animated visualization applications and their resulting frame rates can be improved significantly. This in turn allows the use of _Matplotlib_ in real- time environments, for example in the area of signal processing. Using concrete examples, the presentation will demonstrate the benefits of the [GR framework] as a companion module for _Matplotlib_, both in _Python_ and _Julia_. Based on selected applications, the suitability of the _GR framework_ will be highlighted especially in environments where time is critical. The system’s performance capabilities will be illustrated using demanding live applications. In addition, the special abilities of the _GR framework_ are emphasized in terms of interoperability with graphical user interfaces (_Qt/PySide_) and _OpenGL_, which opens up new possibilities for existing _Matplotlib_ applications.
 Okay, good afternoon. Thank you for coming to my talk. I'm going to tell you something about how you can try to get the plot lip a little bit faster. Maybe you remember my last year's talks about the GR framework, where I promised that I had the intention to write a backend for my plot lip for our software. And I made a promise to use our software in iPython notebooks. And these are the things which I want to introduce to you this afternoon. So I don't have to talk about visualization needs. I think most of us have the same needs when it comes to the point that you want to make plots or diagrams. But for our scientific world, we have some more requirements. For example, we need real-time graphics. And we want to visualize very large data sets. And that might be challenging. So we began to think about another solution. And there are a lot of visualization solutions in the Python world. The most popular ones are mentioned here on this slide. And I think the basic workhorse, which is also part of the scientific stack of Python, is met plot lip. And it has some solutions where you can use met plot lip output in the browser. Maybe you have heard this morning about the BookA software or plotly, which are browser solutions which allow to redirect the met plot lip output to a browser window. There are even other tools which make 3D graphics. They are very powerful, like MyAvi, VDK, or VisP. And they are very fast. But the problem is they are low-level. And there are other problems. And for example, in principle, you have three different things which have to be mentioned. We need interoperability, speeds, and quality. And you can't have all of these three things. So we tried to get another solution. We wanted to combine the 2D and the 3D, the hardware-accelerated 3D world. And we wanted to create a back-end which can not only produce figures, but also can stream data. So this was our primary attention. And at that point, I started to write a back-end format plot lip. Because I didn't know another method to get all these graphics stuff faster. Scython didn't help. It doesn't make sense to improve or speed up the numerical code segments. And hardware acceleration is a very nice feature. But in most cases, it cannot be applied to visualization software. Because you have your existing codes, and you want probably to mix it with other code snippets. So there was the idea, whether it would be possible to write a back-end format plot lip, which would be faster. And as our software is completely written in C, it is capable of presenting continuous data streams. And with a module called GR3, which has been written by a colleague of mine, which is also here in the audience, Florian Reem, we can also mix 2D and 3D graphics into one canvas or one window. There's one important point that the software right now is also interoperability, that there's interoperability with the graphical user interfaces or web frameworks like iPython or in the meantime called Jupyter. So what can we do with our framework? You can combine the power of Matplotlib and GR, so you can mix the output. For example, output where you create real-time plots of signal analysis or whatever you can think of. And you can, for example, produce video content on the fly. You don't have to put your frames into a PNG and then put them together and render them as an MPEG file. You can do it on the fly. And one important point is that you can really mix 2D and 3D graphics elements. So how does this all work? I don't want to go into the detail because I want to show you some demos later. But due to the layer structure of GR, which contains logical device drivers for nearly every technique we have today, we are not there. There's no dependency on third-party software, so we can really mix components together. And we have a very good 3D software which is capable of producing HTML5 outputs and all these things. So this is the output of the Matplotlib GR backend. You can see that there's no difference to the original output. The only advantage is that it's a little bit faster. And here you can see the GR framework in action. You can really produce a very fast output. For example, you can take an audio signal, calculate Fourier transform of this audio signal and display it in real time. This is done in the middle section here. In the right section you could see that you can produce a molecule, that you can visualize a molecule sequence while on the right side creating a histogram with Matplotlib. So they can be combined in one plot and you don't have to change any line of code in your Matplotlib examples. So another feature I mentioned, right now you can use the GR framework in notebooks, for example in Jupyter, which is a follower of Ipython. It can be used both with Python and Julia. I was very excited about the performance, but I had expectations and when I saw the first results I was a little bit disappointed. You can see the left two bars that when using the GR backend the performance improvement is only factor two or three. I had expected much more because in the right bar you can see that with a GR framework as a standalone software in Jupyter you get much higher results. So we have to explain what's the reason for that. You can see in those codes, in those log files here which I have produced with the Python profiler that Matplotlib is wasting too much time in Python itself. It doesn't send enough output to the graphics backend but it's organizing plot data and data files. So at this point I was a little bit disappointed. But then we had another new feature, interoperability. And on this side maybe it's a little bit too small, you can see how you can mix different code segments. I did not change any line in the Matplotlib code in this example and I did not change any line in the GR3 code and I simply put them one after the other and this can then be displayed as a sequence in one canvas or in one web canvas. There's another advantage, if you have such a sequence you can create an MPEG file for example on the fly without adding any animation code. If you know Matplotlib you always have to define some animation functions and then create a loop and put your scenes together. That's not required with this GR framework so you have a big advantage here. Also it's possible to use inline graphics both with Matplotlib and GR. It's not a problem to produce inline figures with Matplotlib but if it comes to the point that you want to generate streams it's much easier with a GR framework and I will show you later how this works. Although I have to say that with the GR software itself it's again ten times faster than Matplotlib. At this point I would like to show you some demos. Let's start with the animation example. Can you see? Okay. Should be big enough. Okay, first of all we define some numpy array. We tell Matplotlib to use inline graphics and we create a figure which right now is empty and then we have to do an animation loop. We have to define animation function. Then we start the loop and in the final step we have to save this animation and this takes a lot of time and once this is done Matplotlib will give us an mpack file which can be displayed here in the browser by this HTML macro. Hopefully it will. Okay, here it is. But we have seen it's a little bit complicated because we have to write this callback function. We have to write, we trigger this animation loop and all these things are not very comfortable because we have to write extra code. So let's try this with the Matplotlib backend. Unfortunately I have to restart the kernel at this time because Matplotlib only checks once whether there is an external backend available. So I have to restart the kernel. I also have to redefine my numpy arrays and now I can tell Matplotlib to use a gfrain work and generate a movie on the fly. Again I import Matplotlib. I create the plot and I can immediately show it. He has to render it like Matplotlib has to do it too. But we have seen we didn't, there was no need to write a callback function, an animation function. You could make the animation on the fly without any change of the Matplotlib code. So finally let me show you how this works with the gfrain work. You have a very simple loop. I also already told the gfrain work to generate inline graphics and as you can see here there are only three lines of code and you have the same animation here. The speed is the same because the frame rate is set to the exact same value. So let's take another example. I want to show you that there are advantages concerning interoperability. In this example I use a Matplotlib package to draw a histogram of some angles. I use a GR3 module called Mocly for molecular dynamics visualizations to read and visualize a sequence of molecules. Then I add a 2D plot as shown in one of the earlier slides using GR. And so let's look whether this works. Again I use a Matplotlib package. I use a 3D package Mocly to read the data file which contains the coordinates of the atoms. Then I use a GR framework until the GR framework to generate the movie on the fly and to generate the output. That is important because Matplotlib should not create the output too early. So now we start the loop. This should take some time because we have to render about 100 scenes. Quite slow this afternoon. Now we are done. And finally we show what has been rendered. And here you can see we have both GR output. We have Matplotlib output and we have output from GR framework. I do it again so you can see what happens. And we have this line output on the bottom here. And all this is done in real time. It has been rendered in real time when you can produce it in your browser. So there's one nice feature which I also want to show. You can export this scene from our GR3 software and then even rotate it in the browser. Create something which Florian has written and which we will probably make available for tool de-graphics in the next release. So let's open the next one. I talked about inline graphics so I have to speed up a little bit. Again we read some data here and you see that inline graphics with Matplotlib is a little bit slow and it's flickering and there's a function I never knew about. It's called clear output which is part of IPython and you can use this function to re-create or re-draw your plot. This is very useful for our scientists to generate sequences of plots in the browser. So I make the same now with GR and you can see it's much faster and it's less code. So we have a speedup of 10 here and you can even do this in JavaScript. So with the next version we can create this output as a JavaScript, HTML file which can then be displayed in real time which will be much faster than our own version. Right now we have finished writing a JavaScript and with the next version we will make it available in the open source. So let's take the last example. This is an example taken from a sci-fi talk from Matplotlib guy which has produced this graph here and I just want to show that with our software. You can visualize this in real time in 3D. It's flickering at this time but I have to mention that all this data has to be in-transferred from the kernel to the browser and back again so that's much traffic here. Maybe this demo makes no sense. I only want to show you that there's a lot of room for performance. Okay, that's for the demos. So let's finalize. So I already mentioned that we are planning to make a JavaScript, that we have written a JavaScript logical device driver for our software which can then be used for example to embed JavaScript code in your UP browser or you could write your own JavaScript code and take the and visualize a display list which has been generated by our software and then fill it with your own JavaScript code. Here on the right side you can see that they are nearly the same commands or that are the same commands which are used in C or in Python or in Julia. Okay, what can GRLs be used for? Here are some examples. PyMolden is a new development of a colleague of mine and which will be made available for the open source very soon. And then we have Nicos, an instrument control system which has been written by two colleagues of mine which will demonstrate it in the poster session tomorrow. I don't know. And so what are the conclusions? Well, you can use the Map.Lip.Lar backend as a GR logical device driver. But the speedups are not as expected. It's mostly factor times two or three faster. But I think the second feature that you can mix 2D and 3D graphics and that you can create movies on the fly is still something which could be interesting for most of you especially for scientists. And you can produce plots and figures much faster with a GR framework. And that's the reason why we are really planning to write a complete Map.Lip implementation, PyLap implementation in C because I think that's the future. And well, these are our plans for the future and I hope I can fulfill my promises next year and show you what we have done. So thank you for your attention. We have five minutes for questions if you have any. I have one. No questions. So. Quick question. This seems quite popular in the scientific community as you showed the PyMol version. Where else do you see this for non-scientific community applications? There are not so much users. For example, with Map.Lip you can't compare those software packages. The package itself is very old because it's C code. It has already written years ago. But now we have written both for backends, both for Julia and for Python, Map.Lip and I think that it will be more popular in the near future because especially with Julia you can get even more performance than I showed today. As Julia can call Python modules, that might be very challenging. I hope that there will be more users in the future. Any other questions from the crowd? Okay. Well, let's thank Joseph again for his presentation. Okay. Okay. Thank you.
 Okay, good afternoon.  Thank you for coming to my talk.  I'm going to tell you something about how you can try to get the plot lip a little bit  faster.  Maybe you remember my last year's talks about the GR framework, where I promised that I  had the intention to write a backend for my plot lip for our software.  And I made a promise to use our software in iPython notebooks.  And these are the things which I want to introduce to you this afternoon.  So I don't have to talk about visualization needs.  I think most of us have the same needs when it comes to the point that you want to make  plots or diagrams.  But for our scientific world, we have some more requirements.  For example, we need real-time graphics.  And we want to visualize very large data sets.  And that might be challenging.  So we began to think about another solution.  And there are a lot of visualization solutions in the Python world.  The most popular ones are mentioned here on this slide.  And I think the basic workhorse, which is also part of the scientific stack of Python,  is met plot lip.  And it has some solutions where you can use met plot lip output in the browser.  Maybe you have heard this morning about the BookA software or plotly, which are browser  solutions which allow to redirect the met plot lip output to a browser window.  There are even other tools which make 3D graphics.  They are very powerful, like MyAvi, VDK, or VisP.  And they are very fast.  But the problem is they are low-level.  And there are other problems.  And for example, in principle, you have three different things which have to be mentioned.  We need interoperability, speeds, and quality.  And you can't have all of these three things.  So we tried to get another solution.  We wanted to combine the 2D and the 3D, the hardware-accelerated 3D world.  And we wanted to create a back-end which can not only produce figures, but also can stream  data.  So this was our primary attention.  And at that point, I started to write a back-end format plot lip.  Because I didn't know another method to get all these graphics stuff faster.  Scython didn't help.  It doesn't make sense to improve or speed up the numerical code segments.  And hardware acceleration is a very nice feature.  But in most cases, it cannot be applied to visualization software.  Because you have your existing codes, and you want probably to mix it with other code  snippets.  So there was the idea, whether it would be possible to write a back-end format plot lip,  which would be faster.  And as our software is completely written in C, it is capable of presenting continuous  data streams.  And with a module called GR3, which has been written by a colleague of mine, which is also  here in the audience, Florian Reem, we can also mix 2D and 3D graphics into one canvas  or one window.  There's one important point that the software right now is also interoperability, that there's  interoperability with the graphical user interfaces or web frameworks like iPython  or in the meantime called Jupyter.  So what can we do with our framework?  You can combine the power of Matplotlib and GR, so you can mix the output.  For example, output where you create real-time plots of signal analysis or whatever you can  think of.  And you can, for example, produce video content on the fly.  You don't have to put your frames into a PNG and then put them together and render them  as an MPEG file.  You can do it on the fly.  And one important point is that you can really mix 2D and 3D graphics elements.  So how does this all work?  I don't want to go into the detail because I want to show you some demos later.  But due to the layer structure of GR, which contains logical device drivers for nearly  every technique we have today, we are not there.  There's no dependency on third-party software, so we can really mix components together.  And we have a very good 3D software which is capable of producing HTML5 outputs and  all these things.  So this is the output of the Matplotlib GR backend.  You can see that there's no difference to the original output.  The only advantage is that it's a little bit faster.  And here you can see the GR framework in action.  You can really produce a very fast output.  For example, you can take an audio signal, calculate Fourier transform of this audio  signal and display it in real time.  This is done in the middle section here.  In the right section you could see that you can produce a molecule, that you can visualize  a molecule sequence while on the right side creating a histogram with Matplotlib.  So they can be combined in one plot and you don't have to change any line of code in  your Matplotlib examples.  So another feature I mentioned, right now you can use the GR framework in notebooks,  for example in Jupyter, which is a follower of Ipython.  It can be used both with Python and Julia.  I was very excited about the performance, but I had expectations and when I saw the  first results I was a little bit disappointed.  You can see the left two bars that when using the GR backend the performance improvement  is only factor two or three.  I had expected much more because in the right bar you can see that with a GR framework as  a standalone software in Jupyter you get much higher results.  So we have to explain what's the reason for that.  You can see in those codes, in those log files here which I have produced with the Python  profiler that Matplotlib is wasting too much time in Python itself.  It doesn't send enough output to the graphics backend but it's organizing plot data and  data files.  So at this point I was a little bit disappointed.  But then we had another new feature, interoperability.  And on this side maybe it's a little bit too small, you can see how you can mix different  code segments.  I did not change any line in the Matplotlib code in this example and I did not change  any line in the GR3 code and I simply put them one after the other and this can then  be displayed as a sequence in one canvas or in one web canvas.  There's another advantage, if you have such a sequence you can create an MPEG file for  example on the fly without adding any animation code.  If you know Matplotlib you always have to define some animation functions and then create  a loop and put your scenes together.  That's not required with this GR framework so you have a big advantage here.  Also it's possible to use inline graphics both with Matplotlib and GR.  It's not a problem to produce inline figures with Matplotlib but if it comes to the point  that you want to generate streams it's much easier with a GR framework and I will show  you later how this works.  Although I have to say that with the GR software itself it's again ten times faster than Matplotlib.  At this point I would like to show you some demos.  Let's start with the animation example.  Can you see?  Okay.  Should be big enough.  Okay, first of all we define some numpy array.  We tell Matplotlib to use inline graphics and we create a figure which right now is empty  and then we have to do an animation loop.  We have to define animation function.  Then we start the loop and in the final step we have to save this animation and this takes  a lot of time and once this is done Matplotlib will give us an mpack file which can be displayed  here in the browser by this HTML macro.  Hopefully it will.  Okay, here it is.  But we have seen it's a little bit complicated because we have to write this callback function.  We have to write, we trigger this animation loop and all these things are not very comfortable  because we have to write extra code.  So let's try this with the Matplotlib backend.  Unfortunately I have to restart the kernel at this time because Matplotlib only checks  once whether there is an external backend available.  So I have to restart the kernel.  I also have to redefine my numpy arrays and now I can tell Matplotlib to use a gfrain  work and generate a movie on the fly.  Again I import Matplotlib.  I create the plot and I can immediately show it.  He has to render it like Matplotlib has to do it too.  But we have seen we didn't, there was no need to write a callback function, an animation  function.  You could make the animation on the fly without any change of the Matplotlib code.  So finally let me show you how this works with the gfrain work.  You have a very simple loop.  I also already told the gfrain work to generate inline graphics and as you can see here there  are only three lines of code and you have the same animation here.  The speed is the same because the frame rate is set to the exact same value.  So let's take another example.  I want to show you that there are advantages concerning interoperability.  In this example I use a Matplotlib package to draw a histogram of some angles.  I use a GR3 module called Mocly for molecular dynamics visualizations to read and visualize  a sequence of molecules.  Then I add a 2D plot as shown in one of the earlier slides using GR.  And so let's look whether this works.  Again I use a Matplotlib package.  I use a 3D package Mocly to read the data file which contains the coordinates of the  atoms.  Then I use a GR framework until the GR framework to generate the movie on the fly and to generate  the output.  That is important because Matplotlib should not create the output too early.  So now we start the loop.  This should take some time because we have to render about 100 scenes.  Quite slow this afternoon.  Now we are done.  And finally we show what has been rendered.  And here you can see we have both GR output.  We have Matplotlib output and we have output from GR framework.  I do it again so you can see what happens.  And we have this line output on the bottom here.  And all this is done in real time.  It has been rendered in real time when you can produce it in your browser.  So there's one nice feature which I also want to show.  You can export this scene from our GR3 software and then even rotate it in the browser.  Create something which Florian has written and which we will probably make available  for tool de-graphics in the next release.  So let's open the next one.  I talked about inline graphics so I have to speed up a little bit.  Again we read some data here and you see that inline graphics with Matplotlib is a little  bit slow and it's flickering and there's a function I never knew about.  It's called clear output which is part of IPython and you can use this function to re-create  or re-draw your plot.  This is very useful for our scientists to generate sequences of plots in the browser.  So I make the same now with GR and you can see it's much faster and it's less code.  So we have a speedup of 10 here and you can even do this in JavaScript.  So with the next version we can create this output as a JavaScript, HTML file which can  then be displayed in real time which will be much faster than our own version.  Right now we have finished writing a JavaScript and with the next version we will make it  available in the open source.  So let's take the last example.  This is an example taken from a sci-fi talk from Matplotlib guy which has produced this  graph here and I just want to show that with our software.  You can visualize this in real time in 3D.  It's flickering at this time but I have to mention that all this data has to be in-transferred  from the kernel to the browser and back again so that's much traffic here.  Maybe this demo makes no sense.  I only want to show you that there's a lot of room for performance.  Okay, that's for the demos.  So let's finalize.  So I already mentioned that we are planning to make a JavaScript, that we have written  a JavaScript logical device driver for our software which can then be used for example  to embed JavaScript code in your UP browser or you could write your own JavaScript code  and take the and visualize a display list which has been generated by our software and then  fill it with your own JavaScript code.  Here on the right side you can see that they are nearly the same commands or that are the  same commands which are used in C or in Python or in Julia.  Okay, what can GRLs be used for?  Here are some examples.  PyMolden is a new development of a colleague of mine and which will be made available for  the open source very soon.  And then we have Nicos, an instrument control system which has been written by two colleagues  of mine which will demonstrate it in the poster session tomorrow.  I don't know.  And so what are the conclusions?  Well, you can use the Map.Lip.Lar backend as a GR logical device driver.  But the speedups are not as expected.  It's mostly factor times two or three faster.  But I think the second feature that you can mix 2D and 3D graphics and that you can create  movies on the fly is still something which could be interesting for most of you especially  for scientists.  And you can produce plots and figures much faster with a GR framework.  And that's the reason why we are really planning to write a complete Map.Lip implementation,  PyLap implementation in C because I think that's the future.  And well, these are our plans for the future and I hope I can fulfill my promises next  year and show you what we have done.  So thank you for your attention.  We have five minutes for questions if you have any.  I have one.  No questions.  So.  Quick question.  This seems quite popular in the scientific community as you showed the PyMol version.  Where else do you see this for non-scientific community applications?  There are not so much users.  For example, with Map.Lip you can't compare those software packages.  The package itself is very old because it's C code.  It has already written years ago.  But now we have written both for backends, both for Julia and for Python, Map.Lip and  I think that it will be more popular in the near future because especially with Julia  you can get even more performance than I showed today.  As Julia can call Python modules, that might be very challenging.  I hope that there will be more users in the future.  Any other questions from the crowd?  Okay.  Well, let's thank Joseph again for his presentation.  Okay.  Okay.  Thank you.
Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_0_299.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_300_383.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_384_563.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_564_767.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_768_1883.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_1884_9011.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_9012_11507.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_11508_20447.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_20448_20711.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_20712_20951.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_20952_21071.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_21072_21155.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_21156_21575.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_21576_21635.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_21636_23375.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_23376_23975.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_23976_25259.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_25260_25595.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_25596_25919.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_25920_26015.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_26016_27251.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_27252_27863.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_27864_29567.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_29568_29771.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_29772_30251.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_30252_30299.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_30300_30515.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_30516_30599.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_30600_31355.png Josef_Heinen_-_Getting_more_out_of_Matplotlib_with_GR_31356_31575.png