Panel Discussion: Exascale and Beyond: Challenges in Productive and Sustainable Software
CC Attribution - NonCommercial - NoDerivatives 2.5 Switzerland:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Bauer, Peter
Fu, Haohuan
Matsuoka, Satoshi
Taufer, Michela
While we tend to think of Exascale as a hardware scaling problem, the required monumental exercise of software co-design between multiple teams tends to get overlooked. Software—the key crosscutting technology by which teams collaborate toward predictive science—is dramatically increasing in complexity due to disruptive architectural changes, multiphysics and multiscale modeling, the coupling of simulations and data analytics, and the demand for greater reproducibility and sustainability. In this session, Lois Curfman McInnes from Argonne National Laboratory and Rich Brueckner from insideHPC will moderate a panel discussion with thought leaders focused on software challenges for Exascale and beyond – mixing “big picture” and technical discussions. McInnes will bring her unique perspective on emerging Exascale software ecosystems to the table, while Brueckner will illustrate the benefits of Exascale to world-wide audiences.
 Hello, everyone. Welcome to our panel on challenges in productive and sustainable software for Exascale and Beyond. My name is Lois Curfin McKinnis, and I am co-moderating this panel with my partner, Rich Bookner. I'm a computational scientist at Argonne National Lab, and my background is scalable numerical software and issues in interoperability among software developed by different teams, as well as issues in software productivity and sustainability. Today we will have a conversation with our international experts on various aspects of software, and we'll begin by each one of them introducing themselves. So first, Satoshi Matsuoka. So hi, I'm Satoshi Matsuoka. My current affiliation is Rican, Rican's Center for Computational Science. I'm the director there, as well as I'm a professor at Tokenshove Technology, and I have a very long career in this business, but I'm basically a systems guy. I've been building, you know, built in software and hardware. I've been building systems like Zbami series of supercomputers to our GPU, pioneering GPU systems, but now I'm responsible for the computer, and then the next generation, Fugaku, the post-K Fugaku supercomputer, which will be one of the first XFKL systems in the world, and we'll have more than 150,000 nodes, or about 8 million GPU cores. Thanks, Satoshi. Next we have Mikaela Toffer. Thank you. I'm very excited to be here. My name is Mikaela Toffer. I'm a professor in electrical engineering and computer science at the University of Tennessee Knoxville. I come from the computational science simulations of molecular dynamics, and I move through the process that many of us are moving through from compute intensive to data intensive applications aspect of in-situ analytics and non-determinism. Thanks, Mikaela. Peter Bauer. Thanks very much. My name is Peter Bauer. I'm with an international organization that sits in the United Kingdom in Reading. It's called the European Center for Medium-Ranged Weather Forecasts. I'm obviously representing weather and climate prediction as an application here. We have a big supercomputer as well because we operate under very tight schedules. At ECMWF, I'm the deputy director of the research department, which is our biggest department. We need to do a lot of research to produce accurate forecasts. I'm leading a very large effort on scalability, as we call it, which is basically all the computational science around our application, the computational aspects, the data aspects, and the workflow aspects. Thank you. How are you? Right. I guess you should still remember me from the morning. How are you? I'm also from Chihua University and the National Supercomputing Center in Wuxi. My job at the center is the deputy director in charge of research and development. Basically, writing software for some way type of light. My background, most of my education is in computer science, but I'm a professor in the Department of Earth System Science. A lot of my efforts are actually trying to use HPC to resolve Earth science-related problems. Thank you. Rich. Well, I'm very humbled to be here. I don't know if you folks know me, but I'm a reporter for Inside HPC, and I've been coming here for a number of years. But if you're wondering why there's two moderators, I'm here to get stories. As a storyteller, you're all going to get to participate today at each of these seats, every other seat, or so there's a microphone. So we have some prepared questions that Lois is going to ask, and at about the 30-minute mark, we're going to ask you to participate, and we'll raise your hand. You hit a button on there, it turns red, that means your mic is live, and we want you to tell us your stories, ask the panel what's on your mind, and at the very end, we're going to share some logistics about the evening dinner event. So with that, Lois, I think we should get started. Thank you. So as we've already been discussing at this conference, software is a key cross-cutting technology by which we do computational science. It's the way in which we collaborate and encapsulate expertise across mathematics, computer science, and various domains in science and engineering. Our challenges in software are getting more and more complex as we're dealing with disruptive architectural changes, increasing demands of our software to tackle multi-scale, multi-physics problems, to incorporate both simulations and data analytics, and likewise to have simulations that are reproducible as well as sustainable. So today, our various panelists are going to speak about some of their experiences in this space. So we'll begin by asking each panelist, what do you consider to be the most pressing challenge in the space of software productivity and sustainability both now and in the future, exascale and beyond? Satoshi? Okay. So, you know, that topic has been actually one of the central topics during the career, you know, on the systems person, both from software and hardware perspective. And the challenge has always been how to balance the ecosystem. The important, you know, ecosystem is very important. That's in software, of course, everywhere. And that's not just with HPC, but very how to sustain the very broad software ecosystem versus trying to pursue the bleeding edge. And, you know, try, and if you do too much later, what will end up is you just die. So, you know, there's a very interesting webpage, you know, that supercomputing vendor page if you search Google. And, you know, you get a list of whole, this whole list of vendors that have passed away over the years. Because, and the reason they passed away is predominantly because, you know, the software ecosystem, they can create the software ecosystem to sustain the overall business. But if you do that too much, what happens is you get very underperforming system because you'll be just buying into this very vanilla thing. And, and thus, you know, you no longer are in the supercomputing realm. Because that's the whole purpose of supercomputing to achieve performance. So how do you, so this reality of how do you have this really very rich ecosystem versus how do you pursue the bleeding edge? And so, you know, my, all my, a lot of my efforts in the past have been to how to realize both at the same time. And I can give more exemplars, but I'll just pass it on for the interest of time. I think we need to sort of have a 360 degree vision of what is that our software does. We started 10, 20 years ago, think about software as a simulation of a phenomenon. We were sort of focused on one single aspect, which was the simulation itself. And now we look at what our workflows are becoming. And it's such an heterogeneous path towards science production and discovery. And so it's no longer just a single architecture, a single package, but we integrate, we move from compute intensive to data intensive stages. And we have to plug them together to work. And it looks easy. It looks quite trivial. But then when we do that, and when we share that with other, that is a big challenge. And I think that if you start looking at what the workflows are becoming and how we need to integrate them, we need also to change our state of mind towards what is that we are doing with our software. So I think that is a big challenge. I probably have to give an answer slightly from an application point of view. I think in the past we have been spoiled by general purpose chips, general purpose architectures, general purpose compilers. Everything worked for and programming standards. Everything worked for us. We could just afford to run our application with 5% sustained efficiency on computers and still buy them and run them and fulfill our scientific needs. So while we have a very good idea of where we want to go science wise in the next 5 or 10 years, we don't know where the technological evolution is going to get us and whether it's actually supporting our scientific ambitions. And there were two sessions today about weather and climate. And I'd say at least 80% of the contents of the talks were actually about how to overcome this and take things in our own hand. So on this domain specific language, for example, buzzword, we only started to think about 5 years ago is now in everybody's talk and there's actually living examples of how to do this. But that implies as well that there's actually a load now, a development load on us to take care of ourselves rather than rely on external factors just delivering this to us. You know, the number I can give is at the center of the ECMWF, we have great staff, excellent staff, but we probably had like two, three, four people max taking care of code optimization and adaptation. Today with the work we're investing in developing our own, taking care of ourselves, it's at least 20 people, 25 people, you know, and that is reflected in many other places as well. So you have a shift of responsibility because of this lack of natural evolution of standards and common technology. Right. I didn't know we speak in the sequence of the seats. Otherwise... Yeah, I would follow up on the application side actually. So the first sort of piece of code that we tried to put onto some way type of light is the CESM, this Earth System model with, you know, over one million lines of code. And so after that, you can imagine it's a very frustrating process. We have roughly 20 students to go on the projects and after three months, I think only one steer would like to continue because I, because I force him, he's my student. So actually after this, I was thinking the toughest challenge is actually we don't have students or say engineers with a multidisciplinary set of knowledge and skills. Because when they look at this one million lines of code, they don't understand, I mean, the computer can understand this. It's about cloud, not that cloud, it's the real cloud. Or, you know, Typhoon or interesting phenomenon that you want to study. But for computer science students, after three months, they will come to you and say that deep learning is very sexy. I want that. So for me, I think the talents, the talents that can really understand both science and engineering is probably one of the challenge that we're going to face. Absolutely. Yeah. Do you have follow-ups at this point? Well, real quick, I wanted to revisit this idea of ecosystems because Satoshi, the K computer that you're going to supplant with the Fugaku machine, was kind of a one-off, right? It was a spark machine with Linux. And it might have been the only one in the world. There are lots of Linux in your previous company. Well, at least for HPC, right? And now you're starting with ARM, which is not exactly a big presence on the top 500, but you're going to have a machine of incredible scale, but has its own ecosystem outside of HPC. Right. What is the challenge there? Well, the challenge, of course, the first challenge was convincing everyone, right, that ecosystem was of great importance because the Japanese supercomputing vendors have had a long history of not necessarily the computer vendors themselves because if you look at the online history of Japanese computers, they've always created IBM compatibles, right? And then they got sued. So there were a lot of stories there. So despite the fact that mainstream computing that Japanese vendors have been creating basically copies of IBM and stuff and also microprocessors, for HPC, for some reason, they had always committed to building some proprietary stuff. And this continued for many years. So it took a lot of effort to initially try to convince a partner like Fujitsu to say we go ARM to the heated discussion. And now that was done. Then, of course, the second challenge was to how to come up with enhancements of ARM, ISA, and the micro architecture in order to support HPC workloads. And there were a lot of initial trials to try to adapt mobile processors in the ARM space to be used in HPC, for example, Mont Blanc effort in Europe. And we had all the efforts and there were several other efforts in the world. And, of course, the foregone conclusion was that was not going to happen because inherently it was just for various reasons, it was just too hard because these mobile devices for various reasons were not tailored to run at scale. So then we had to go through all the exercises of how to add these features to come up with a brand new micro architecture, but brand new instruction sets which were turned into SVE. And then also have HP enhancements like, you know, putting HPM2 on a stack just like GPUs. So that's the second challenge. The third challenge, of course, despite this very broad ARM ecosystem and the other in this space, in the mobile space, there are some tools which still needed to be ported over, over to HPC space. There are some unique requirements like, you know, MPI or parallel file systems or some of the debugging tools and also SV itself, you know, how to come up with the optimized SV library. So there are a lot of challenges there. But overall, I think once these challenges are overcome, we'll be at the very tip of this very humongous ARM ecosystem. And it's nowadays because the security reasons become very important that the code would run ubiquitously from the edge all the way to the cloud. So this very same code will run from the edge to the cloud, which will be inherent advantage in the ARM ecosystem. So I think we have a pretty good solution actually in our system, in our upcoming system. Thanks. So already our panelists have talked about multidisciplinary collaboration, ecosystems, teamwork. I wonder if you could comment about whether some of the aspects of the culture and the reward structure in high-performance computing and computational science either help or hinder some of those challenges that we're encountering. Why don't we start at the other end? I think it's S-word. S-word is unexpected. Yeah, actually, so I think we have some of the culture issues. Like I think in each university, the president of the university would say, you have to collaborate with different departments. But then after you write the paper, the paper only can go through the credit of one single department. So after you become, let's say, after you write a good paper and then the department has started to fight and then they become angry with you. So that's the part. Currently, not working very well. We were proposing to the president, you should just give double credit to all the departments related. But currently it's not like that. And also there's one thing. So Satoshi was talking about actually the basic software, like the compiler, the operating system, is actually a problem in China now. If you look at the universities, there used to be good professors doing research on that. But it's very difficult to write papers for compilers or operating systems. So nowadays we don't have many professors doing that. So that's also another part of the problem. How do you evaluate your research in terms of HPC-related software? Yes, those are challenges. Peter? Well again, application-oriented response. I think in our case, a lot of the reward comes from what science can actually implement. That applies equally to the applied scientist, the person developing the cloud scheme, as to the computational scientist. Because if you happen to improve the forecast by a certain factor, you can see the result every day. It saves lives actually. If you happen to be able to prepare a weather prediction system for the next generation of supercomputers that looks entirely different than today, and you can do that on time, you save money and lives again. So I think that is why we're actually seeing, maybe opposite what you said in your first statement, is actually that I think we can actually see at the interface between our science, the applied science, and the computational science, a new generation of people coming from aerospace engineering or other disciplines that have backgrounds in fluid dynamics, but at the same time, a very good computational scientist coming into our discipline, because they understand that weather and climate prediction has great societal impact, and they have a conscience that on Google, they come to us, and they really want to improve the system. And apparently, working in that environment is very rewarding. So I can see actually in our field that there's obviously enough incentive and reward to draw people in and build up that interface that didn't exist before. I think we are in a sort of critical time in which we need probably to pause a moment and think about how we evaluate the contribution of computational scientists when the research is in between computer science and sciences. I think that what several of us maybe experienced that we submit paper and we get back reviewed that tell us, oh, this is engineering, or you are not really HPC, or you are not really computational chemistry. We need really to pause and understand that putting together workflows, generating real science, it's not trivial. And describing how to achieve this process is a contribution. So several of us who review papers or assess the quality of our colleagues should be more careful in assessing work as trivial because there is not a mathematical formula. And I think it's a transition that we need to do because otherwise we will lose this opportunity of really on a long term collaborate. Yeah. Okay. So when I became director at Recon CCS, so I try to restructure our directions a bit because we were a little bit too focused on the simulation outcomes that are not broad enough to be a more broader HPC, part of a broader HPC ecosystem. So my question was what does Recon Center for Computational Science mean? So it's that, well, you know, we're studying computing, obviously, and we're doing science about or related to computing because our key element is computing. So I said, well, we should really strive in pursuing science of computing, meaning that we, especially in high performance because that's what we're doing. That's a subject matter of science. So we should really stress that. But also we say science by computing because that's the more traditional computational science where the outcomes of studies of computing will actually drive other sciences. And science for computing, there are other scientific disciplines which would contribute to computing like material science, when your device is, and so forth. And so we should collaborate with those. So together, science of computing by computing for computing, which is kind of stealing a little bit of a quote from a more, when US presidents were very honorable and not computing first, something like that. So that's the direction. Now, as for the reward structure, if you keep that in mind, I think the reward structure I think has become quite enriched in the recent years because that really tells you that we're involved in a very multidisciplinary area. And actually there's a very high demand for people of those skill sets. And whether they become scientists or they start working at Google, they have a very flexible career path. And as such, and very high demand, as such very high reward would be at financial reward or some reward in some prestigious scientific institutions or whatnot. I think there's a very, the reward themselves are quite enriched as compared to the previous years. And this is, so the interest in computing has, I think, really had changed the value systems of the young people at large because in Japan, there's a recent poll. We asked the junior, like 13 year olds, what do you want to become when you become an adult? So for a long time, people said, well, you know, we become a medical doctor. We want to become a baseball player or soccer player for that matter. Or, you know, some things like that. But now the top number one ranking professional is IT professional. So that, by the way, number six is a YouTuber. So you have to take it with a little grain of salt. But you know, actually that will accumulate talent. Basically, you know, when people aspire to be, the best talents aspire to go into the field. Then you get very rich sets of talents. So if we sustain a proper reward system such that no matter what they do, being engulfed and being exposed to this multidisciplinary area to the extent that they have a very rich career path, I think we will have a very promising future. We just have to make sure that all the people go work for Google. We have to retain a little talent for ourselves. I would like to agree and disagree with you, Satoshi. Disagree. I agree with you that working across discipline is one of the most exciting jobs that a person can have. I fully agree with you. I disagree with you that there is a community that welcomed that. I think that today still we struggle to get ten years when we do internet research. We still struggle to get jobs when we do interdiction. And probably we make jobs in position like industry. But if you go in academia, it's still hard to make a career if you work across disciplines. Well, you know, so becoming a director at Recon is an interesting one. Well, we have 15 centers at Recon. And it's a very interesting experience career. It's much more interesting to some extent than being a professor at this point, although I'm still a professor. The reason is there are 15 other centers that have something related to biology. So you get to basically converse with a lot of these directors, some of who are like Nobel Prize candidates, very esteemed people. But they complain a lot about their center. And one of the complaints they have is the, you know, one of the things I found out is that biology has a completely different type of value system as opposed to physics or computer science or whatnot. In fact, in biology, a lot of people are very narrow. I don't know if you have biologists. I'm sorry if I'm not. You're around somebody who's watching the rest. They're very narrow skill sets. So it's very hard for them to change their career, especially when you experiment. So compared to that, I think we have a very broad, you know, people's skill sets are broad. So I don't know why they're having such problems finding good jobs. But it's not the problem in finding good jobs. It's sometimes a challenge in moving forward in careers, especially in environments. You know, not everybody has the luck, the fortune to work in a very dynamic environment. A lot of institutions still are small and have traditional criteria for assessing the success of our students, the success of our faculty. And there, even the success of paper. So we need to learn to be more open mind and revaluate what is success in these careers. It's not an H index, which sometimes is not as high as people that work in a single field. It's not a single paper, but it's this bringing vision and how do you measure visions in a community that is interdisciplinary? That's particularly challenging, I think, as we think about software and many contributions we need to have to make effective software, as Peter talked about. I mean, you work at the top university in China, so do you have similar problems? Yeah, I think it's especially... So for me, I kind of resolved my issue because I demonstrated that I can use HPC to solve problems that they cannot solve. But here, when I go to the next level, it's like Michal said, I don't know whether I should compete with people from computer science, they will say you are from Earth Systems Science Department. If I compete with climate scientists, they will say you are a programmer, you are not a scientist. So it's... But why do we need to compete? Isn't it more like that you have new disciplines coming up at exactly the interfaces I mentioned before? You don't need to invade a computational scientist or climate scientist area. There's new areas coming up at the interface that didn't exist at all before. It's a new type of expertise and therefore there's new dynamics and new incentives and new career paths. I definitely agree with that, as Michal also mentioned. If I give up and I start up a company to do all these things that I can do, it would be quite rewarding. But in universities, they make very clear categories. You have to fall into one specific category. And there's funding. I mean, in the end, funding is a big issue. If you can supply the funding for what you want to do, it doesn't matter what you want to do. And there's big funding programs. So that's one of the issues. But I want to go back to the software aspect and describe a small personal experience. I report in my CV some of the software that with my group we developed. And I have a department chair asking me, why do you report the software in your CV during an annual evaluation, which is because it's a product that... So there is the culture of software is not as a product. Few of us are able to associate a DOI to our software and distribute that. And that is something that we should do more. You moved to a new university, right? Yes. Precisely for that reason. Same for data sets. Or data sets, exactly. Can we put DOI? Can we assess them? But that exists now. So the infrastructure is there? I think there is... Because I'm on these government committees and all that. And I think at the very top level there is a growing recognition for example, need for open science and open data sets. And of course we advocate for them. And people at the very senior level also bureaucrats and so forth, they do understand the needs. So maybe it's people who are kind of stuck in the middle, the old people who have very solved mindsets who will probably retire pretty soon. And then we have a very brave new world where I think these multidisciplinary work values and the new world and the new realm will be well evaluated. And I think that's... And we can... Sort of being very forward looking institution as opposed to universities. National lab some more than the epic. I'm seeing... I think... Seeing this tremendous change in the culture. All right, folks. Well, we're at the halfway point for this session and I'm a reporter, so I have a million questions. We want to make this participatory. So if we have questions from the audience, please raise your hand. There's going to be a microphone. We ask you to use that so that the camera can hear you. So first... Who has a question for our panel today? Don't be shy right here. Red is on. Okay, now it's red. Now, actually the subject of the discussion was the scale and beyond. Then we just went to the careers and these careers of numerical scientists is not very estimated. It's not very nicely founded. And I'm coming from fusion community. It's exactly the same problem we have. Exactly. So physicists have careers, they have publications, but people who are doing codes or optimizing codes. And now what I see already what's going on in Europe because you're presenting many countries. We used to think in Europe that in Japan it's good. In China it's very nice. In a way that there are people and there is funding to adapt existing, for example, codes for applications to exascale. And architectures are developing together with codes and applications. And now you didn't mention it at all. I wonder what do you think? For the moment we're just looking at the tendencies that people are doing exascale based on GPUs or something. But nobody is listening to us to use this. Because for example, if it scales badly, what we can do and stuff like this, I would like to hear opinion of somebody. Can you comment on it? Because you're talking about applications. But now we're doing architectures kind of independently going to exascale. That's not what's completely untrue. That's completely untrue. It's completely untrue maybe in Japan, but it's more or less like this in Europe. And we worry about it. I'm in place scientific steering committee. And we worry about it. And scientists will just keep talking about this. And so what do you think about it? Is it because of funding? Is it because of organization? What kind of efforts we could do to join development of high performance computing and applications? I think I understood what you want to say. But please correct me if I didn't once I start. I think partly what you're saying is true. And we had a very big discussion in Europe over the past five years in the context of where large European infrastructures that are eventually were in part managed by praise, for example, should go in the future. And how do we get the applications along? And are these architectures actually at all designed to fulfill what the individual applications need? And again, I think the level at which that started was again in the old school thinking of we have a general purpose machine. It's flop oriented as long as it's high enough in the top 500 list, it should be all right. And then we make a separate efforts probably underfunded to get the applications on it. And they just achieve the same performances as ever. I think that's how it started. I think that's slightly changing now. I think there are increasingly there's an increasing in awareness, for example, at the European Commission level, but also at national level, national level, because for those of you who don't know, it's a co funded effort. So there's a big European as United's investment and that is complemented by individual national funding, which is a complicated thing. But this democracy, democracy. So in any case, I think there is a step or movement towards getting increasingly increasingly domain specific to recognize where the individual domains suffer to maybe really invest in co design and not just try to optimize and adapt applications for pre designed HPC systems, but to help them along at least at system level, if not at component level. So I think it's actually coming along. And I hope that maybe not for the pre access scale systems, but for the ones after or even the ones after we will actually will be in a different place where we have not just four or five machines of the same type, but four or five machines that serve specifically specific application groups that share the same problems. So you have different flavors of technology that is available and maybe in part actually co designed to serve applications better groups of applications better than today with the general purpose machines. I think it's moving and I can add something related to the United States. I've been asked to advise some university on what to do when they try to improve their HPC system. And the attitude is to start with the big machine. But as the university go through surveys that they reach out to their scientists, who are the ones that are listening in this case, it come clearly out that is not a single machine that is better to invest in more domain specific machine and embrace a group of scientists or domains. So there is some movement in that direction. Of course, there is still the old fashioned tradition of the big top 500. And I think there is a shifting also in that takes some time. So you know, one, we are close game was developed. So basically the first thing that was dished from the objective was to do well on the top 500. That was the first objective that was thrown away. And the whole project started around 2009 even before K, the K computer. So there was initial studies of the possible architectures in 2020 era, time frame, and you know, possible alternatives. And then the next phase that happened in 2012 was to basically start a set of what we call the feasibility study project by which we involved all the Japanese application community like in its entirety. Everyone basically, well, everyone who is using big machines. Not, you know, not 100,000 people, but at least people who are using supercomputers. And to get them involved in the co-design process. And there were four teams and I was responsible for one of the application teams from the system perspective. And what we did was to basically group all the people, all the stakeholders in a single room. And then in a short version, we forced them to write a 600 page report. What would happen? What are the science, what are the science they project 10 years from now? 5 years from now, 10 years from now. And what are the computational capacity requirements? What are the algorithms they'll use and so forth. So if you look at the report and the English summary, 60 page English summary, 600 page Japanese version. And you know, you get the whole chart of the applications and the computational requirements and the scientific outcome. And then based on those, the Fosuke Fugaku project was officially launched in 2014. And out of, but at the same time, it wasn't just a machine app development project. The nine priority areas, priority sub-party areas were selected. And each of these areas were asked to come up with one representative application code design targets. So nine codes, and there were nine areas, so nine code design targets were chosen. And the whole design process was basically to achieve up to 100 times performance boost given the capacity requirements, 100 times performance boost when the average of about 40 over K for each of the areas. And it's projected that, you know, we know, in fact, we know now because we have the world machine, we will chip and everything, we will achieve this goal. So, you know, it was a very orchestrated process, code design process by which we developed Fugaku post-K. It was not to hit the top 500. It was not to, you know, those plot numbers. It was not to take off the shelf components. All the components are new. But especially designed for optimized for HPC and listening to all the application requirements. Well, Lois, you manage a huge team of people preparing for these exascale machines. Are you seeing this kind of frustration out there, or are you working to fix that? What would you say? Well, so I work in the exascale computing project, in particular, coordinate work in math libraries, where the math library's teams all collaborate with scientific applications. And I would say that we've been very pleased and happy with the multidisciplinary collaborations that we're seeing. And we're making some good steps forward in code design. I will say that we're still challenged, though I believe in terms of collaborative software, to develop appropriate career paths for everyone who are participants in our broad teams. Research software engineers are incredibly important for the work that we do. And I believe as an international community, there's lots of opportunity for all of us to work together to make sure that we can fully support our broad skills that we need in order to do collaborative computational science. Okay. Did that answer your question? Okay. Yeah. We have another right there. Got the mic. Red is on. So I do experimental high energy physics. And from our point of view in the community, the next generation and the generation after that probably of high HPCs is an utter and complete disaster. Intel has told us, don't worry, we'll help you port your code to A21. NVIDIA has said the same for Perlmutter. I'm sure AMD will tell us the same for Frontier. We have millions of lines of code. And we cannot afford to port to specific machines every two years. Do you see any hope of convergence of technologies either on the hardware side or on the software side that will spare us from this? It's just an impossible burden, frankly. I have an answer later. I think I mentioned it maybe earlier. Where I said that kind of in our community, we're trying to take things on our own hand. I don't think there's technological convergence. I'm not sure you want that, actually. I think you want actually heterogeneous architectures because at least in our application, if we go through the big components that kind of cost the most in our system, they have very different patterns in terms of how they stress memory or what memory bandwidth they require or what data they produce and these kind of things or what level of parallelism they actually allow or vectorization and these things. So I don't think you want actually one size fits all for that. So I think heterogeneous architectures is probably good. But the different difficulties, of course, programming the different types of hardware and that's where I said earlier, I think we at least try to take things in our own hand and implement what's been mentioned many times before in many past conferences, the so-called separation of concerns where we hide away or abstract away all these different dimensions of adaptation and optimization away from the science code so that your actual native science code becomes independent from the hardware and below you develop a big middle layer that's the part we have to develop ourselves and just underneath at the bottom end you have back ends that work for specific components on specific hardware and that's exchangeable and ideally fully flexible. So whatever you're facing in five years or ten years, been taken care of and you don't have to touch your top level code anymore. But where of course not naive, we know this is very intrusive, it takes different thinking in terms of computational science. You have to reeducate your scientists but they will have less opportunities to break the code because they will never see the bottom layer actually. So I think there's solutions to that but it's not going to be easy and it's probably not something you can do incrementally. It's not like in the past where you do little changes piece by piece every little year to your code. I think it's one big jump. It's really a quantum leap in the real sense. There's nothing in between. And I would like to sort of ask a question back which is what are our measures of success, what are our metrics of success? If it is flops, yes we are constantly running and rewriting and adapting. But if it is science, I think that it changes a lot the perspective of how to reengineer software. It is not something that every time that there is a new processor you have to go and run and rewrite everything. I think we need to change also our metrics of success and it's in terms of science that we can achieve. If we look in that criteria and that condition, the requirement to rewrite in the software is so urgent every time we have a new platform. Success for us means publishing a paper and the DOE has told us that we cannot use these machines if we don't use the GPUs. So we cannot publish, we cannot produce science. But the GPU is probably one of these major jump and then the question is do you really need the last GPU? We go back also to the fact of what... If we need a GPU, we need a computer. Well, let's speak about accelerators in general. Well, get the Fujitsu CPU and... Accelerate. No, I mean, it's a CPU but it has... Well, seriously. There's always... I mean, there are... There are... I want to think... When you kind of structure the problem in various ways, you know, most of the HPC codes are bandwidth bound, memory bandwidth bound. There are a few that are exceptions like first principle, look at orbitals and some are looking... And also look at dynamics and maybe like the DFT, maybe the other three instances that are playoffs bound. But most of the other HPC codes are bandwidth bound. So how do you drive the bandwidth? Well, architecturally, there are like three options. One is YCyMD, which is like the now sort of rest in peace as the on-fi. And then the proper version of this, which is the A64FX, which is done right, by the way. The ARM people say they got the instructions right too, but that's a different issue. Okay. And then there's the SIMTI style dynamic threading plus vector lengths and that's the GPU. So whether being at NVIDIA or NVIDIA or now Intel, GPUs basically have the same SIMR structure. And then there is the more of a data flow like approach, which is something... It's not there yet, but the CGRA variants have always been proposed and maybe there will be one in the future. So how do you... So there are at least three architectural variants. And how do you... How to have a software that will cope for each one? That's the something that's been... That this community has been tackling for a long time. There are some successes, like there are OpenACC, OpenMP. There are C++ variants that try to hide away some of these complexities and DSLs that's been mentioned and so forth. So there is constant challenge from that end and trials from those and it's some successes. But that's ultimately what we're striving for. But at the end of the day, when you have a legacy Fortran code, you really need to run. Then I don't think there is any other choice but to basically try to use architecture that's probably most fit for those types of codes. Because if you're not rewriting it, or at least not rewriting it in significance, because that's... You want to spend your time otherwise. In case you have to get the proper architecture of that. So other you try to invest in software to adapt to these architectural variants or you don't... So you just try to pick the best hardware. Now, I'm not going to say which one, but it depends on your code. But fortunately, I think each of the variations are coming up to speed. So Intel was in low, but now that there's Fujitsu chip, which will probably blow them out of the water, really. They will come up with competing chips that will try to compete with Fujitsu because otherwise they'll just lose their business. Because on CFD codes, the I644 effects is 48 times faster than Intel. And that's because of the inherent properties of the high bandwidth memory. So I'm sure Intel will come up with variants with high bandwidth memory at some point. Because otherwise, those use lose all the business. Yeah, I just want to add a few comments to that question. Actually, I was trying... When I was talking with the climate guys, they had the same issue. When GPU came out in, I think, 2008 or 2009, right? And it's only until recently that climate communities starting to look at GPUs. So they really... They don't want to change, right? But if you look at the problem from another perspective, it's actually a good... Maybe for a few decades, you face this kind of disruptive architecture change that motivates you to talk with computer scientists. This is a good way to promote interdisciplinary. Usually they don't talk to me. Now they have to talk to me because they don't know how to run the code on some way, type of light, or something like that. And the other thing is... I'm not sure in the domain of high energy physics, like in climate, I think it's good for a few decades if you really relook at the code and refactor it. Because some of the codes are really written a few decades away and nobody can really read it very smoothly. So it's good to actually reconstruct the code after a few decades. I think it can be good. We had a question from Sunita. So I'm Sunita from University of Delaware. I'm also a faculty. So it's very nice to see a faculty on the panel. So I'm going to follow up with your comment, Mikaela. We have heard so much about the need for applying computer science on real science code. And I recently had this fabulous conversation about RSE, Research Software Engineers. And I was relaying this to other faculty across departments, across universities where I got an opportunity. The challenge I see is if a student were to do engineering kind of work, which is equally important, it is perceived as, oh, it is not novel enough. And it doesn't qualify to give the student a PhD degree because there's no novelty because it's engineering work. And I got such feedback, which I totally disagreed to, because we need software developers, we need research software engineers to do the kind of science. And there are a lot of national lab and center on the podium as well as in the room. How is it that we can communicate back to universities that PhD is an RSE is equally important than a PhD in computer science, but the subcategory being RSE is important because that's what we need at the end of the day, be it high energy physics or be it climate or be it any interdisciplinary science. So how is it that we can take the message from you and be able to communicate back at the university level that that's very important and cannot be neglected? I'm in trouble. I actually think that there is even a worst scenario of what you are describing, that we are running out of PhD students who are talented because sometimes our institution don't understand these talents, but the national lab, research labs, industry understand and come and offer them fantastic salary. And we run out of excellent students who does work in software development sustainability and get a PhD and become faculties. And we may have also students that look at us and say, I don't want to become like you, I go and get money. So I think that is an effort that the entire community should do. I look at this, this is the first time in PASC and I look at this community that has been flourishing so quickly and I think that is a clear message that there is a need to create this community of PhD students who graduate with work that is defined as engineering or software engineering. And I think that conferences like PASC are a response and a solution to this kind of culture, to change it. So I am very pleased that PASC exists and I am looking forward to seeing more of PASC as a motivation for our work. I think the science of research software is incredibly important and that is one way to frame, I believe, Sanita, some of the points that you are making. And I see it as a huge growth opportunity for our community, very, very strongly needed. Can I make a comment? So, you know, super computer back in 19, let's say, 60s, like grade one, there was only a factor of like three to ten difference between a normal computer versus a super computer. Super computer was like only three to ten times faster. Now there is several orders of magnitude difference. So if you look at, like, again, those K for a computer, it has more than 150,000 nodes. It has more like eight million cores. So which is it? You know, just huge. In terms of cores, it is bigger than Google Cloud of the data center. So if you are in the application side or the system side, you know, if you can't find interesting problems, right, on such an infrastructure, be it, it can be a little smaller, but you know, it can't find interesting problems to solve on a system like that. I don't think you deserve a PhD. So, yeah. And I think we can find that. I think that is the problem is having the outside community acknowledging that that is an interesting problem. So I think it's really important to actually partner, like, for example, computer science to really partner with people who have problems on the application side and use these high machines to solve real problems. And there will be some real challenges on the computer side to resolve. So, and similarly for the other way around. Now, I think, you know, having fielding these infrastructures actually is truly important to advance science because otherwise, if you're working at skill, which is mundane, then you don't find these problems. Well, folks, we're at a hard stop here because of an event. I see we have more questions, but I want to have a let's give a hand to our panel. I think they did a great job.
 Hello, everyone. Welcome to our panel on challenges in productive and sustainable software for  Exascale and Beyond. My name is Lois Curfin McKinnis, and I am co-moderating this panel  with my partner, Rich Bookner. I'm a computational scientist at Argonne National Lab, and my  background is scalable numerical software and issues in interoperability among software  developed by different teams, as well as issues in software productivity and sustainability.  Today we will have a conversation with our international experts on various aspects of  software, and we'll begin by each one of them introducing themselves. So first, Satoshi  Matsuoka. So hi, I'm Satoshi Matsuoka. My current affiliation is Rican, Rican's Center for  Computational Science. I'm the director there, as well as I'm a professor at Tokenshove Technology,  and I have a very long career in this business, but I'm basically a systems guy. I've been building,  you know, built in software and hardware. I've been building systems like Zbami series of  supercomputers to our GPU, pioneering GPU systems, but now I'm responsible for the computer, and then  the next generation, Fugaku, the post-K Fugaku supercomputer, which will be one of the first  XFKL systems in the world, and we'll have more than 150,000 nodes, or about 8 million GPU cores.  Thanks, Satoshi. Next we have Mikaela Toffer. Thank you. I'm very excited to be here. My name is  Mikaela Toffer. I'm a professor in electrical engineering and computer science at the University  of Tennessee Knoxville. I come from the computational science simulations of molecular dynamics,  and I move through the process that many of us are moving through from compute intensive to  data intensive applications aspect of in-situ analytics and non-determinism. Thanks, Mikaela.  Peter Bauer. Thanks very much. My name is Peter Bauer. I'm with an international organization  that sits in the United Kingdom in Reading. It's called the European Center for Medium-Ranged  Weather Forecasts. I'm obviously representing weather and climate prediction as an application  here. We have a big supercomputer as well because we operate under very tight schedules. At ECMWF,  I'm the deputy director of the research department, which is our biggest department. We need to do  a lot of research to produce accurate forecasts. I'm leading a very large effort on scalability,  as we call it, which is basically all the computational science around our application,  the computational aspects, the data aspects, and the workflow aspects.  Thank you. How are you?  Right. I guess you should still remember me from the morning. How are you? I'm also from  Chihua University and the National Supercomputing Center in Wuxi. My job at the center is the  deputy director in charge of research and development. Basically, writing software for  some way type of light. My background, most of my education is in computer science, but  I'm a professor in the Department of Earth System Science. A lot of my efforts are actually  trying to use HPC to resolve Earth science-related problems.  Thank you. Rich.  Well, I'm very humbled to be here. I don't know if you folks know me, but I'm a reporter  for Inside HPC, and I've been coming here for a number of years. But if you're wondering  why there's two moderators, I'm here to get stories. As a storyteller, you're all going  to get to participate today at each of these seats, every other seat, or so there's a microphone.  So we have some prepared questions that Lois is going to ask, and at about the 30-minute  mark, we're going to ask you to participate, and we'll raise your hand. You hit a button  on there, it turns red, that means your mic is live, and we want you to tell us your stories,  ask the panel what's on your mind, and at the very end, we're going to share some logistics  about the evening dinner event. So with that, Lois, I think we should get started.  Thank you. So as we've already been discussing at this conference, software is a key cross-cutting  technology by which we do computational science. It's the way in which we collaborate and encapsulate  expertise across mathematics, computer science, and various domains in science and engineering.  Our challenges in software are getting more and more complex as we're dealing with disruptive  architectural changes, increasing demands of our software to tackle multi-scale, multi-physics  problems, to incorporate both simulations and data analytics, and likewise to have simulations  that are reproducible as well as sustainable. So today, our various panelists are going  to speak about some of their experiences in this space. So we'll begin by asking each  panelist, what do you consider to be the most pressing challenge in the space of software  productivity and sustainability both now and in the future, exascale and beyond? Satoshi?  Okay. So, you know, that topic has been actually one of the central topics during the career,  you know, on the systems person, both from software and hardware perspective. And the  challenge has always been how to balance the ecosystem. The important, you know, ecosystem  is very important. That's in software, of course, everywhere. And that's not just with  HPC, but very how to sustain the very broad software ecosystem versus trying to pursue  the bleeding edge. And, you know, try, and if you do too much later, what will end up  is you just die. So, you know, there's a very interesting webpage, you know, that supercomputing  vendor page if you search Google. And, you know, you get a list of whole, this whole  list of vendors that have passed away over the years. Because, and the reason they passed  away is predominantly because, you know, the software ecosystem, they can create the software  ecosystem to sustain the overall business. But if you do that too much, what happens  is you get very underperforming system because you'll be just buying into this very vanilla  thing. And, and thus, you know, you no longer are in the supercomputing realm. Because that's  the whole purpose of supercomputing to achieve performance. So how do you, so this reality  of how do you have this really very rich ecosystem versus how do you pursue the bleeding edge?  And so, you know, my, all my, a lot of my efforts in the past have been to how to realize  both at the same time. And I can give more exemplars, but I'll just pass it on for the  interest of time.  I think we need to sort of have a 360 degree vision of what is that our software does. We  started 10, 20 years ago, think about software as a simulation of a phenomenon. We were sort  of focused on one single aspect, which was the simulation itself. And now we look at  what our workflows are becoming. And it's such an heterogeneous path towards science  production and discovery. And so it's no longer just a single architecture, a single package,  but we integrate, we move from compute intensive to data intensive stages. And we have to plug  them together to work. And it looks easy. It looks quite trivial. But then when we do  that, and when we share that with other, that is a big challenge. And I think that if you  start looking at what the workflows are becoming and how we need to integrate them, we need  also to change our state of mind towards what is that we are doing with our software. So  I think that is a big challenge.  I probably have to give an answer slightly from an application point of view. I think  in the past we have been spoiled by general purpose chips, general purpose architectures,  general purpose compilers. Everything worked for and programming standards. Everything worked  for us. We could just afford to run our application with 5% sustained efficiency on computers  and still buy them and run them and fulfill our scientific needs. So while we have a very  good idea of where we want to go science wise in the next 5 or 10 years, we don't know where  the technological evolution is going to get us and whether it's actually supporting our  scientific ambitions. And there were two sessions today about weather and climate. And I'd say  at least 80% of the contents of the talks were actually about how to overcome this and  take things in our own hand. So on this domain specific language, for example, buzzword,  we only started to think about 5 years ago is now in everybody's talk and there's actually  living examples of how to do this. But that implies as well that there's actually a load  now, a development load on us to take care of ourselves rather than rely on external  factors just delivering this to us. You know, the number I can give is at the center of  the ECMWF, we have great staff, excellent staff, but we probably had like two, three,  four people max taking care of code optimization and adaptation. Today with the work we're  investing in developing our own, taking care of ourselves, it's at least 20 people, 25  people, you know, and that is reflected in many other places as well. So you have a shift  of responsibility because of this lack of natural evolution of standards and common technology.  Right. I didn't know we speak in the sequence of the seats. Otherwise...  Yeah, I would follow up on the application side actually. So the first sort of piece  of code that we tried to put onto some way type of light is the CESM, this Earth System  model with, you know, over one million lines of code. And so after that, you can imagine  it's a very frustrating process. We have roughly 20 students to go on the projects and after  three months, I think only one steer would like to continue because I, because I force  him, he's my student. So actually after this, I was thinking the toughest challenge is actually  we don't have students or say engineers with a multidisciplinary set of knowledge and skills.  Because when they look at this one million lines of code, they don't understand, I mean,  the computer can understand this. It's about cloud, not that cloud, it's the real cloud.  Or, you know, Typhoon or interesting phenomenon that you want to study. But for computer science  students, after three months, they will come to you and say that deep learning is very  sexy. I want that. So for me, I think the talents, the talents that can really understand  both science and engineering is probably one of the challenge that we're going to face.  Absolutely. Yeah. Do you have follow-ups at this point?  Well, real quick, I wanted to revisit this idea of ecosystems because Satoshi, the K computer  that you're going to supplant with the Fugaku machine, was kind of a one-off, right? It  was a spark machine with Linux. And it might have been the only one in the world.  There are lots of Linux in your previous company.  Well, at least for HPC, right? And now you're starting with ARM, which is not exactly a  big presence on the top 500, but you're going to have a machine of incredible scale, but  has its own ecosystem outside of HPC. Right.  What is the challenge there?  Well, the challenge, of course, the first challenge was convincing everyone, right,  that ecosystem was of great importance because the Japanese supercomputing vendors have had  a long history of not necessarily the computer vendors themselves because if you look at the  online history of Japanese computers, they've always created IBM compatibles, right? And  then they got sued. So there were a lot of stories there. So despite the fact that mainstream  computing that Japanese vendors have been creating basically copies of IBM and stuff  and also microprocessors, for HPC, for some reason, they had always committed to building  some proprietary stuff. And this continued for many years. So it took a lot of effort  to initially try to convince a partner like Fujitsu to say we go ARM to the heated discussion.  And now that was done. Then, of course, the second challenge was to how to come up with  enhancements of ARM, ISA, and the micro architecture in order to support HPC workloads. And there  were a lot of initial trials to try to adapt mobile processors in the ARM space to be used  in HPC, for example, Mont Blanc effort in Europe. And we had all the efforts and there  were several other efforts in the world. And, of course, the foregone conclusion was that  was not going to happen because inherently it was just for various reasons, it was just  too hard because these mobile devices for various reasons were not tailored to run at  scale. So then we had to go through all the exercises of how to add these features to  come up with a brand new micro architecture, but brand new instruction sets which were  turned into SVE. And then also have HP enhancements like, you know, putting HPM2 on a stack just  like GPUs. So that's the second challenge. The third challenge, of course, despite this  very broad ARM ecosystem and the other in this space, in the mobile space, there are  some tools which still needed to be ported over, over to HPC space. There are some unique  requirements like, you know, MPI or parallel file systems or some of the debugging tools  and also SV itself, you know, how to come up with the optimized SV library. So there are  a lot of challenges there. But overall, I think once these challenges are overcome,  we'll be at the very tip of this very humongous ARM ecosystem. And it's nowadays because the  security reasons become very important that the code would run ubiquitously from the edge  all the way to the cloud. So this very same code will run from the edge to the cloud,  which will be inherent advantage in the ARM ecosystem. So I think we have a pretty good  solution actually in our system, in our upcoming system.  Thanks. So already our panelists have talked about multidisciplinary collaboration, ecosystems,  teamwork. I wonder if you could comment about whether some of the aspects of the culture  and the reward structure in high-performance computing and computational science either  help or hinder some of those challenges that we're encountering. Why don't we start at  the other end? I think it's S-word.  S-word is unexpected. Yeah, actually, so I think we have some of the culture issues.  Like I think in each university, the president of the university would say, you have to collaborate  with different departments. But then after you write the paper, the paper only can go  through the credit of one single department. So after you become, let's say, after you  write a good paper and then the department has started to fight and then they become  angry with you. So that's the part. Currently, not working very well. We were proposing to  the president, you should just give double credit to all the departments related. But  currently it's not like that. And also there's one thing. So Satoshi was talking about actually  the basic software, like the compiler, the operating system, is actually a problem in  China now. If you look at the universities, there used to be good professors doing research  on that. But it's very difficult to write papers for compilers or operating systems.  So nowadays we don't have many professors doing that. So that's also another part of  the problem. How do you evaluate your research in terms of HPC-related software?  Yes, those are challenges. Peter?  Well again, application-oriented response. I think in our case, a lot of the reward comes  from what science can actually implement. That applies equally to the applied scientist,  the person developing the cloud scheme, as to the computational scientist. Because if  you happen to improve the forecast by a certain factor, you can see the result every day.  It saves lives actually. If you happen to be able to prepare a weather prediction system  for the next generation of supercomputers that looks entirely different than today, and you  can do that on time, you save money and lives again. So I think that is why we're actually  seeing, maybe opposite what you said in your first statement, is actually that I think  we can actually see at the interface between our science, the applied science, and the  computational science, a new generation of people coming from aerospace engineering or  other disciplines that have backgrounds in fluid dynamics, but at the same time, a very  good computational scientist coming into our discipline, because they understand that weather  and climate prediction has great societal impact, and they have a conscience that on  Google, they come to us, and they really want to improve the system. And apparently, working  in that environment is very rewarding. So I can see actually in our field that there's  obviously enough incentive and reward to draw people in and build up that interface that  didn't exist before.  I think we are in a sort of critical time in which we need probably to pause a moment  and think about how we evaluate the contribution of computational scientists when the research  is in between computer science and sciences. I think that what several of us maybe experienced  that we submit paper and we get back reviewed that tell us, oh, this is engineering, or  you are not really HPC, or you are not really computational chemistry. We need really to  pause and understand that putting together workflows, generating real science, it's not  trivial. And describing how to achieve this process is a contribution. So several of us  who review papers or assess the quality of our colleagues should be more careful in assessing  work as trivial because there is not a mathematical formula. And I think it's a transition that  we need to do because otherwise we will lose this opportunity of really on a long term  collaborate.  Yeah.  Okay. So when I became director at Recon CCS, so I try to restructure our directions a  bit because we were a little bit too focused on the simulation outcomes that are not broad  enough to be a more broader HPC, part of a broader HPC ecosystem. So my question was  what does Recon Center for Computational Science mean? So it's that, well, you know, we're  studying computing, obviously, and we're doing science about or related to computing because  our key element is computing. So I said, well, we should really strive in pursuing science  of computing, meaning that we, especially in high performance because that's what we're  doing. That's a subject matter of science. So we should really stress that. But also  we say science by computing because that's the more traditional computational science  where the outcomes of studies of computing will actually drive other sciences. And science  for computing, there are other scientific disciplines which would contribute to computing  like material science, when your device is, and so forth. And so we should collaborate  with those. So together, science of computing by computing for computing, which is kind  of stealing a little bit of a quote from a more, when US presidents were very honorable  and not computing first, something like that. So that's the direction. Now, as for the reward  structure, if you keep that in mind, I think the reward structure I think has become quite  enriched in the recent years because that really tells you that we're involved in a  very multidisciplinary area. And actually there's a very high demand for people of those  skill sets. And whether they become scientists or they start working at Google, they have  a very flexible career path. And as such, and very high demand, as such very high reward  would be at financial reward or some reward in some prestigious scientific institutions  or whatnot. I think there's a very, the reward themselves are quite enriched as compared to  the previous years. And this is, so the interest in computing has, I think, really had changed  the value systems of the young people at large because in Japan, there's a recent poll. We  asked the junior, like 13 year olds, what do you want to become when you become an adult?  So for a long time, people said, well, you know, we become a medical doctor. We want  to become a baseball player or soccer player for that matter. Or, you know, some things  like that. But now the top number one ranking professional is IT professional. So that,  by the way, number six is a YouTuber. So you have to take it with a little grain of salt.  But you know, actually that will accumulate talent. Basically, you know, when people aspire  to be, the best talents aspire to go into the field. Then you get very rich sets of  talents. So if we sustain a proper reward system such that no matter what they do, being  engulfed and being exposed to this multidisciplinary area to the extent that they have a very rich  career path, I think we will have a very promising future. We just have to make sure that all  the people go work for Google. We have to retain a little talent for ourselves.  I would like to agree and disagree with you, Satoshi.  Disagree. I agree with you that working across discipline is one of the most exciting jobs  that a person can have. I fully agree with you. I disagree with you that there is a community  that welcomed that. I think that today still we struggle to get ten years when we do internet  research. We still struggle to get jobs when we do interdiction. And probably we make jobs  in position like industry. But if you go in academia, it's still hard to make a career  if you work across disciplines.  Well, you know, so becoming a director at Recon is an interesting one. Well, we have  15 centers at Recon. And it's a very interesting experience career. It's much more interesting  to some extent than being a professor at this point, although I'm still a professor. The  reason is there are 15 other centers that have something related to biology. So you get  to basically converse with a lot of these directors, some of who are like Nobel Prize  candidates, very esteemed people. But they complain a lot about their center. And one  of the complaints they have is the, you know, one of the things I found out is that biology  has a completely different type of value system as opposed to physics or computer science  or whatnot. In fact, in biology, a lot of people are very narrow. I don't know if you  have biologists. I'm sorry if I'm not.  You're around somebody who's watching the rest.  They're very narrow skill sets. So it's very hard for them to change their career, especially  when you experiment. So compared to that, I think we have a very broad, you know, people's  skill sets are broad. So I don't know why they're having such problems finding good  jobs.  But it's not the problem in finding good jobs. It's sometimes a challenge in moving forward  in careers, especially in environments. You know, not everybody has the luck, the fortune  to work in a very dynamic environment. A lot of institutions still are small and have traditional  criteria for assessing the success of our students, the success of our faculty. And  there, even the success of paper. So we need to learn to be more open mind and revaluate  what is success in these careers. It's not an H index, which sometimes is not as high  as people that work in a single field. It's not a single paper, but it's this bringing  vision and how do you measure visions in a community that is interdisciplinary?  That's particularly challenging, I think, as we think about software and many contributions  we need to have to make effective software, as Peter talked about.  I mean, you work at the top university in China, so do you have similar problems?  Yeah, I think it's especially... So for me, I kind of resolved my issue because I demonstrated  that I can use HPC to solve problems that they cannot solve. But here, when I go to  the next level, it's like Michal said, I don't know whether I should compete with people  from computer science, they will say you are from Earth Systems Science Department. If  I compete with climate scientists, they will say you are a programmer, you are not a scientist.  So it's... But why do we need to compete? Isn't it more like that you have new disciplines  coming up at exactly the interfaces I mentioned before? You don't need to invade a computational  scientist or climate scientist area. There's new areas coming up at the interface that didn't  exist at all before. It's a new type of expertise and therefore there's new dynamics and new  incentives and new career paths. I definitely agree with that, as Michal also mentioned.  If I give up and I start up a company to do all these things that I can do, it would be  quite rewarding. But in universities, they make very clear categories. You have to fall  into one specific category. And there's funding. I mean, in the end, funding is a big issue.  If you can supply the funding for what you want to do, it doesn't matter what you want  to do. And there's big funding programs. So that's one of the issues.  But I want to go back to the software aspect and describe a small personal experience.  I report in my CV some of the software that with my group we developed. And I have a department  chair asking me, why do you report the software in your CV during an annual evaluation, which  is because it's a product that... So there is the culture of software is not as a product.  Few of us are able to associate a DOI to our software and distribute that. And that is  something that we should do more. You moved to a new university, right?  Yes. Precisely for that reason. Same for data sets.  Or data sets, exactly. Can we put DOI? Can we assess them?  But that exists now. So the infrastructure is there?  I think there is... Because I'm on these government committees and all that. And I think at the  very top level there is a growing recognition for example, need for open science and open  data sets. And of course we advocate for them. And people at the very senior level also bureaucrats  and so forth, they do understand the needs. So maybe it's people who are kind of stuck  in the middle, the old people who have very solved mindsets who will probably retire pretty  soon. And then we have a very brave new world where I think these multidisciplinary work  values and the new world and the new realm will be well evaluated. And I think that's...  And we can... Sort of being very forward looking institution as opposed to universities. National  lab some more than the epic. I'm seeing... I think... Seeing this tremendous change in  the culture.  All right, folks. Well, we're at the halfway point for this session and I'm a reporter,  so I have a million questions. We want to make this participatory. So if we have questions  from the audience, please raise your hand. There's going to be a microphone. We ask you  to use that so that the camera can hear you. So first... Who has a question for our panel  today? Don't be shy right here. Red is on.  Okay, now it's red. Now, actually the subject of the discussion was the scale and beyond.  Then we just went to the careers and these careers of numerical scientists is not very  estimated. It's not very nicely founded. And I'm coming from fusion community. It's exactly  the same problem we have. Exactly.  So physicists have careers, they have publications, but people who are doing codes or optimizing  codes. And now what I see already what's going on in Europe because you're presenting many  countries. We used to think in Europe that in Japan it's good. In China it's very nice.  In a way that there are people and there is funding to adapt existing, for example, codes  for applications to exascale. And architectures are developing together with codes and applications.  And now you didn't mention it at all. I wonder what do you think? For the moment we're just  looking at the tendencies that people are doing exascale based on GPUs or something. But nobody  is listening to us to use this. Because for example, if it scales badly, what we can do  and stuff like this, I would like to hear opinion of somebody. Can you comment on it?  Because you're talking about applications. But now we're doing architectures kind of  independently going to exascale. That's not what's completely untrue.  That's completely untrue. It's completely untrue maybe in Japan, but it's more or less  like this in Europe. And we worry about it. I'm in place scientific steering committee.  And we worry about it. And scientists will just keep talking about this. And so what  do you think about it? Is it because of funding? Is it because of organization? What kind of  efforts we could do to join development of high performance computing and applications?  I think I understood what you want to say. But please correct me if I didn't once I start.  I think partly what you're saying is true. And we had a very big discussion in Europe  over the past five years in the context of where large European infrastructures that  are eventually were in part managed by praise, for example, should go in the future. And  how do we get the applications along? And are these architectures actually at all designed  to fulfill what the individual applications need? And again, I think the level at which  that started was again in the old school thinking of we have a general purpose machine. It's  flop oriented as long as it's high enough in the top 500 list, it should be all right.  And then we make a separate efforts probably underfunded to get the applications on it.  And they just achieve the same performances as ever. I think that's how it started. I  think that's slightly changing now. I think there are increasingly there's an increasing  in awareness, for example, at the European Commission level, but also at national level,  national level, because for those of you who don't know, it's a co funded effort. So there's  a big European as United's investment and that is complemented by individual national  funding, which is a complicated thing. But this democracy, democracy. So in any case,  I think there is a step or movement towards getting increasingly increasingly domain specific  to recognize where the individual domains suffer to maybe really invest in co design  and not just try to optimize and adapt applications for pre designed HPC systems, but to help  them along at least at system level, if not at component level. So I think it's actually  coming along. And I hope that maybe not for the pre access scale systems, but for the  ones after or even the ones after we will actually will be in a different place where  we have not just four or five machines of the same type, but four or five machines that  serve specifically specific application groups that share the same problems. So you have different  flavors of technology that is available and maybe in part actually co designed to serve  applications better groups of applications better than today with the general purpose  machines. I think it's moving and I can add something related to the United States. I've  been asked to advise some university on what to do when they try to improve their HPC system.  And the attitude is to start with the big machine. But as the university go through surveys  that they reach out to their scientists, who are the ones that are listening in this case,  it come clearly out that is not a single machine that is better to invest in more domain specific  machine and embrace a group of scientists or domains. So there is some movement in that  direction. Of course, there is still the old fashioned tradition of the big top 500. And  I think there is a shifting also in that takes some time.  So you know, one, we are close game was developed. So basically the first thing that was dished  from the objective was to do well on the top 500. That was the first objective that was  thrown away. And the whole project started around 2009 even before K, the K computer.  So there was initial studies of the possible architectures in 2020 era, time frame, and  you know, possible alternatives. And then the next phase that happened in 2012 was to  basically start a set of what we call the feasibility study project by which we involved  all the Japanese application community like in its entirety. Everyone basically, well,  everyone who is using big machines. Not, you know, not 100,000 people, but at least people  who are using supercomputers. And to get them involved in the co-design process. And there  were four teams and I was responsible for one of the application teams from the system  perspective. And what we did was to basically group all the people, all the stakeholders  in a single room. And then in a short version, we forced them to write a 600 page report.  What would happen? What are the science, what are the science they project 10 years from  now? 5 years from now, 10 years from now. And what are the computational capacity requirements?  What are the algorithms they'll use and so forth. So if you look at the report and the  English summary, 60 page English summary, 600 page Japanese version. And you know, you  get the whole chart of the applications and the computational requirements and the scientific  outcome. And then based on those, the Fosuke Fugaku project was officially launched in  2014. And out of, but at the same time, it wasn't just a machine app development project.  The nine priority areas, priority sub-party areas were selected. And each of these areas  were asked to come up with one representative application code design targets. So nine codes,  and there were nine areas, so nine code design targets were chosen. And the whole design  process was basically to achieve up to 100 times performance boost given the capacity  requirements, 100 times performance boost when the average of about 40 over K for each  of the areas. And it's projected that, you know, we know, in fact, we know now because  we have the world machine, we will chip and everything, we will achieve this goal. So,  you know, it was a very orchestrated process, code design process by which we developed  Fugaku post-K. It was not to hit the top 500. It was not to, you know, those plot numbers.  It was not to take off the shelf components. All the components are new. But especially  designed for optimized for HPC and listening to all the application requirements.  Well, Lois, you manage a huge team of people preparing for these exascale machines. Are  you seeing this kind of frustration out there, or are you working to fix that? What would  you say?  Well, so I work in the exascale computing project, in particular, coordinate work in  math libraries, where the math library's teams all collaborate with scientific applications.  And I would say that we've been very pleased and happy with the multidisciplinary collaborations  that we're seeing. And we're making some good steps forward in code design. I will say that  we're still challenged, though I believe in terms of collaborative software, to develop  appropriate career paths for everyone who are participants in our broad teams. Research  software engineers are incredibly important for the work that we do. And I believe as  an international community, there's lots of opportunity for all of us to work together  to make sure that we can fully support our broad skills that we need in order to do collaborative  computational science.  Okay. Did that answer your question? Okay. Yeah. We have another right there. Got the  mic. Red is on.  So I do experimental high energy physics. And from our point of view in the community,  the next generation and the generation after that probably of high HPCs is an utter and  complete disaster. Intel has told us, don't worry, we'll help you port your code to A21.  NVIDIA has said the same for Perlmutter. I'm sure AMD will tell us the same for Frontier.  We have millions of lines of code. And we cannot afford to port to specific machines  every two years. Do you see any hope of convergence of technologies either on the hardware side  or on the software side that will spare us from this? It's just an impossible burden,  frankly.  I have an answer later. I think I mentioned it maybe earlier. Where I said that kind of  in our community, we're trying to take things on our own hand. I don't think there's technological  convergence. I'm not sure you want that, actually. I think you want actually heterogeneous  architectures because at least in our application, if we go through the big components that kind  of cost the most in our system, they have very different patterns in terms of how they  stress memory or what memory bandwidth they require or what data they produce and these  kind of things or what level of parallelism they actually allow or vectorization and these  things.  So I don't think you want actually one size fits all for that. So I think heterogeneous  architectures is probably good. But the different difficulties, of course, programming the different  types of hardware and that's where I said earlier, I think we at least try to take things  in our own hand and implement what's been mentioned many times before in many past conferences,  the so-called separation of concerns where we hide away or abstract away all these different  dimensions of adaptation and optimization away from the science code so that your actual  native science code becomes independent from the hardware and below you develop a big middle  layer that's the part we have to develop ourselves and just underneath at the bottom  end you have back ends that work for specific components on specific hardware and that's  exchangeable and ideally fully flexible. So whatever you're facing in five years or ten  years, been taken care of and you don't have to touch your top level code anymore. But  where of course not naive, we know this is very intrusive, it takes different thinking  in terms of computational science. You have to reeducate your scientists but they will  have less opportunities to break the code because they will never see the bottom layer  actually. So I think there's solutions to that but it's not going to be easy and it's  probably not something you can do incrementally. It's not like in the past where you do little  changes piece by piece every little year to your code. I think it's one big jump. It's  really a quantum leap in the real sense. There's nothing in between.  And I would like to sort of ask a question back which is what are our measures of success,  what are our metrics of success? If it is flops, yes we are constantly running and rewriting  and adapting. But if it is science, I think that it changes a lot the perspective of how  to reengineer software. It is not something that every time that there is a new processor  you have to go and run and rewrite everything. I think we need to change also our metrics  of success and it's in terms of science that we can achieve. If we look in that criteria  and that condition, the requirement to rewrite in the software is so urgent every time we  have a new platform.  Success for us means publishing a paper and the DOE has told us that we cannot use these  machines if we don't use the GPUs. So we cannot publish, we cannot produce science.  But the GPU is probably one of these major jump and then the question is do you really  need the last GPU? We go back also to the fact of what...  If we need a GPU, we need a computer.  Well, let's speak about accelerators in general.  Well, get the Fujitsu CPU and... Accelerate.  No, I mean, it's a CPU but it has...  Well, seriously. There's always... I mean, there are...  There are... I want to think... When you kind of structure the problem in various ways,  you know, most of the HPC codes are bandwidth bound, memory bandwidth bound. There are a  few that are exceptions like first principle, look at orbitals and some are looking... And  also look at dynamics and maybe like the DFT, maybe the other three instances that are playoffs  bound. But most of the other HPC codes are bandwidth bound.  So how do you drive the bandwidth? Well, architecturally, there are like three options. One is YCyMD,  which is like the now sort of rest in peace as the on-fi.  And then the proper version of this, which is the A64FX, which is done right, by the  way. The ARM people say they got the instructions  right too, but that's a different issue. Okay.  And then there's the SIMTI style dynamic threading plus vector lengths and that's the GPU.  So whether being at NVIDIA or NVIDIA or now Intel, GPUs basically have the same SIMR structure.  And then there is the more of a data flow like approach, which is something... It's  not there yet, but the CGRA variants have always been proposed and maybe there will  be one in the future. So how do you... So there are at least three architectural variants.  And how do you... How to have a software that will cope for each one? That's the something  that's been... That this community has been tackling for a long time.  There are some successes, like there are OpenACC, OpenMP. There are C++ variants that  try to hide away some of these complexities and DSLs that's been mentioned and so forth.  So there is constant challenge from that end and trials from those and it's some successes.  But that's ultimately what we're striving for. But at the end of the day, when you have  a legacy Fortran code, you really need to run. Then I don't think there is any other  choice but to basically try to use architecture that's probably most fit for those types of  codes. Because if you're not rewriting it, or at least not rewriting it in significance,  because that's... You want to spend your time otherwise.  In case you have to get the proper architecture of that. So other you try to invest in software  to adapt to these architectural variants or you don't... So you just try to pick the  best hardware. Now, I'm not going to say which one, but it depends on your code.  But fortunately, I think each of the variations are coming up to speed. So Intel was in low,  but now that there's Fujitsu chip, which will probably blow them out of the water, really.  They will come up with competing chips that will try to compete with Fujitsu because otherwise  they'll just lose their business. Because on CFD codes, the I644 effects is 48 times faster  than Intel. And that's because of the inherent properties of the high bandwidth memory. So  I'm sure Intel will come up with variants with high bandwidth memory at some point.  Because otherwise, those use lose all the business.  Yeah, I just want to add a few comments to that question. Actually, I was trying... When  I was talking with the climate guys, they had the same issue. When GPU came out in,  I think, 2008 or 2009, right? And it's only until recently that climate communities starting  to look at GPUs. So they really... They don't want to change, right? But if you look at the  problem from another perspective, it's actually a good... Maybe for a few decades, you face  this kind of disruptive architecture change that motivates you to talk with computer scientists.  This is a good way to promote interdisciplinary. Usually they don't talk to me. Now they have  to talk to me because they don't know how to run the code on some way, type of light,  or something like that. And the other thing is... I'm not sure in the domain of high energy  physics, like in climate, I think it's good for a few decades if you really relook at  the code and refactor it. Because some of the codes are really written a few decades  away and nobody can really read it very smoothly. So it's good to actually reconstruct the code  after a few decades. I think it can be good.  We had a question from Sunita.  So I'm Sunita from University of Delaware. I'm also a faculty. So it's very nice to see  a faculty on the panel. So I'm going to follow up with your comment, Mikaela. We have heard  so much about the need for applying computer science on real science code. And I recently  had this fabulous conversation about RSE, Research Software Engineers. And I was relaying  this to other faculty across departments, across universities where I got an opportunity.  The challenge I see is if a student were to do engineering kind of work, which is equally  important, it is perceived as, oh, it is not novel enough. And it doesn't qualify to give  the student a PhD degree because there's no novelty because it's engineering work.  And I got such feedback, which I totally disagreed to, because we need software developers,  we need research software engineers to do the kind of science. And there are a lot of national  lab and center on the podium as well as in the room. How is it that we can communicate  back to universities that PhD is an RSE is equally important than a PhD in computer science,  but the subcategory being RSE is important because that's what we need at the end of  the day, be it high energy physics or be it climate or be it any interdisciplinary science.  So how is it that we can take the message from you and be able to communicate back at  the university level that that's very important and cannot be neglected?  I'm in trouble. I actually think that there is even a worst scenario of what you are  describing, that we are running out of PhD students who are talented because sometimes  our institution don't understand these talents, but the national lab, research labs, industry  understand and come and offer them fantastic salary. And we run out of excellent students  who does work in software development sustainability and get a PhD and become faculties. And we  may have also students that look at us and say, I don't want to become like you, I go  and get money. So I think that is an effort that the entire community should do. I look  at this, this is the first time in PASC and I look at this community that has been flourishing  so quickly and I think that is a clear message that there is a need to create this community  of PhD students who graduate with work that is defined as engineering or software engineering.  And I think that conferences like PASC are a response and a solution to this kind of  culture, to change it. So I am very pleased that PASC exists and I am looking forward  to seeing more of PASC as a motivation for our work.  I think the science of research software is incredibly important and that is one way to  frame, I believe, Sanita, some of the points that you are making. And I see it as a huge  growth opportunity for our community, very, very strongly needed.  Can I make a comment? So, you know, super computer back in 19, let's say, 60s, like  grade one, there was only a factor of like three to ten difference between a normal computer  versus a super computer. Super computer was like only three to ten times faster. Now there  is several orders of magnitude difference. So if you look at, like, again, those K for  a computer, it has more than 150,000 nodes. It has more like eight million cores. So which  is it? You know, just huge. In terms of cores, it is bigger than Google Cloud of the data  center. So if you are in the application side or the system side, you know, if you can't  find interesting problems, right, on such an infrastructure, be it, it can be a little  smaller, but you know, it can't find interesting problems to solve on a system like that.  I don't think you deserve a PhD. So, yeah.  And I think we can find that. I think that is the problem is having the outside community  acknowledging that that is an interesting problem.  So I think it's really important to actually partner, like, for example, computer science  to really partner with people who have problems on the application side and use these high  machines to solve real problems. And there will be some real challenges on the computer  side to resolve. So, and similarly for the other way around.  Now, I think, you know, having fielding these infrastructures actually is truly important  to advance science because otherwise, if you're working at skill, which is mundane, then you  don't find these problems.  Well, folks, we're at a hard stop here because of an event. I see we have more questions,  but I want to have a let's give a hand to our panel. I think they did a great job.
_Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_0_107.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_108_1247.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_1248_1811.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_1812_2063.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_2064_2447.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_2448_2579.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_2580_3179.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_3180_3335.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_3336_3431.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_3432_4391.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_4392_6443.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_6444_6911.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_6912_7115.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_7116_8483.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_8484_9083.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_9084_12743.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_12744_12875.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_12876_15503.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_15504_16019.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_16020_16811.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_16812_16991.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_16992_17591.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_17592_21047.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_21048_23807.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_23808_24035.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_24036_24143.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_24144_24623.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_24624_25991.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_25992_27695.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_27696_28619.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_28620_31643.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_31644_33299.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_33300_33887.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_33888_34031.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_34032_35819.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_35820_36815.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_36816_36863.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_36864_37199.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_37200_37355.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_37356_37607.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_37608_38579.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_38580_39179.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_39180_41435.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_41436_44171.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_44172_44351.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_44352_46907.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_46908_47279.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_47280_47423.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_47424_49931.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_49932_50423.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_50424_50675.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_50676_54203.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_54204_54371.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_54372_55451.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_55452_57959.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_57960_60407.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_60408_60635.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_60636_62039.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_62040_62471.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_62472_63623.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_63624_63767.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_63768_64679.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_64680_68411.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_68412_69215.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_69216_75359.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_75360_75599.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_75600_76043.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_76044_77759.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_77760_79823.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_79824_81767.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_81768_82043.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_82044_82271.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_82272_84887.png _Panel__Exascale_and_Beyond_Challenges_in_Productive_and_Sus_84888_85183.png