Just-in-time compiling Java in 2020
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Doerr, Martin
Client compiler, server compiler, JVMCI, Graal ... What are we using today and how do they work together? In this talk I'll give and overview of the Just-in-time compilers included in OpenJDK and explain how to play with them. I'll also address Just-in-time compiler threads and resource related issues.
 So, hello everyone and welcome to my talk about the trust and time compilers in the OpenTrail. I'm Martin Dörr and I work for SAP. We are a small team working on the JVM at SAP in Waldorf and this time we have three time, three talks in a row. So we will also see two of my colleagues later on but let's get started with the agenda. So I will talk about the trust and time compilers which translates Java bytecode into machine code in the OpenTrail. I will also talk about how different compilers work together and I also like to address resource usage as well. So first of all, how many compilers do we have in the OpenTrail? We've already heard about two ones but how many do we have? We have one more. Three ones. So the one we haven't talked today yet is the client compiler which is also called C1. It compiles pretty quickly but with a lower optimization level and then we have the server compiler also called C2. We already had a talk about that one. It's kind of the opposite, it compiles slowly but therefore with a high optimization level. For example, it has a lot of loop optimizations and we've also heard about the escape analysis and there are still people working on improvements for that. Both compilers are available on a lot of platforms including PowerPC and S390 which are supported by our team and these two compilers are used by default. And we've also heard about Graal which is rather new in the OpenTrail.k. It is still experimental in the OpenTrail.k. That means it is not used by default. You need to switch it on if you want to use it and it is developed on GitHub. So updates get merged into the OpenTrail.k. Also to Graal is that it is written in Java. That is a big difference to the other two compilers. And that also does a lot of optimization. It has a more sophisticated escape analysis for example. Andrew has already shown a few things about Graal. So thanks for that. And it is optimized for dynamic languages. By the way, Graal compiler is also called GraalVM compiler. And I'd like to show a few things Andrew has already mentioned. I got one slide from Oracle. So thanks to Oracle for providing it. There are three different use cases of GraalVM. And the Graal compiler is always in the center of it. Together with the JVMCI, the Java Virtual Machine Compiler Interface. And the use case on the very left is the one which is available in the OpenTrail.k. So you have a Java application or Java methods which get compiled by Graal. And they run on the hotspot VM, the Java Virtual Machine of the OpenTrail.k. So that path on the very left is supported by the OpenTrail.k. And in addition to that, on the right hand side, you can see the native image technology Andrew also already mentioned. And everything gets pre-compiled. And there's something in between where only the Graal compiler is pre-compiled into a shared library. So that's the basic difference between this approach and this one. Here Graal compiler itself is pre-compiled in a shared library. So back to the different compilers. I'd like to compare performance a little bit. By the way, this is an old benchmark with an old JDK and an old garbage collector. But don't care about numbers. I think it's good to get a first impression about the performance of the different compilers. So at the bottom you can see interpreter for reference. So that means we are not using any just-in-time compiler. And you can get that by specifying the runtime option minus xint. That stands for interpreter mode. So you will only use the interpreter, no-jit compilers at all. And as you can see, the performance is pretty poor. Already much faster is the C1, the client compiler. You can select that, for example, by using this flag, tiered stop at level 3. That might sound a little bit complicated. And I have to note that level 3 means that C1 still performs profiling. So you won't get the best performance out of C1 by that. If you want to tune C1, you would select stop at level 1. And then you would get C1 without profiling. But in this case, I want the better profiling information. That's why I left it on. If you want to use the C2 compiler only, you can switch off tiered compilation. And then you get the blue line, which is already much faster. And the default configuration uses tiered compilation. And you get the fastest startup and the best peak performance. The best peak performance also because the profiling information is better. But I'll explain the tiered compilation stuff later on in more detail. So you should be able to understand that better at the end. But for those who hate this old stuff, I have also a slide with the latest JDK. So the same old benchmark with the latest JDK 15. And you can already see that the peak performance is better with C2 especially. And you can also see the green line, which is new. That is grahl. In order to use grahl, you need to use this switch, use JVMCI compiler, which is an experimental option. So you need to unlock it in addition. And grahl is the default JVMCI compiler. So you will get grahl by this flag. So peak performance of grahl is good even for this very traditional workload. Grahl performs this, of course, better for modern workloads. For example, if you run Scala, that's what Twitter does a lot. And I should also mention that the OpenShadeDK only contains the community edition of grahl. There's also an enterprise version available, which contains more optimizations. So you will get better performance with the enterprise edition. And you can also see that the startup takes longer. It takes a couple of seconds until here 4.5 seconds roughly to get peak performance. And that's due to the fact that grahl itself is written in Java. So the grahl compiler itself gets interpreted at the beginning. And then later on, hot methods get compiled by C1. And later on, they get compiled by which compiler? The grahl compiler itself. So grahl compiles itself. And that takes a few seconds. So this may be OK for large server applications, where you can afford spending a few seconds. But there's also a possibility to fix that if you need a quicker startup. And that's available with grahl VM. So the trahl VM has a feature called printflexfinal. And if you enable that, you will see all flag configurations, the VM sets for itself. And you can also find use trahlmci native library. And with grahl VM, that one is true by default. And that means the trahlm is using the pre-compiled shared library. So the grahl compiler is already pre-compiled. And you get a pretty good startup with that. So next, I've promised to explain tiered compilation a little bit. So tiered compilation is basically the answer to the question of how these different compilers work together. As already mentioned at the beginning, everything starts at the interpreter, which is tier zero. And then we have three different tiers for the C1 compiler. Tier one is C1 without any profiling. That is used only for trivial methods. And the C1 believes that it's not worth optimizing further. So we will stick on this trivial compilation. And then there's tier two. C1 uses reduced profiling. And it does that when it thinks there's too much work to do. So we just should make it quick. And the default tier for C1 is tier three. And you get the full profiling code compiled into the compiled method. And then finally, the tier four is for the highest optimization level. And it uses C2 compiler by default in OpenTraderK. And you can replace it by grahl if you enable it explicitly. You can also see the tiers when you enable print compilation. You can see which method gets compiled at which tier. And typically, most methods get started at tier three. Then you get also tier four method compiled by C2 in this case. But here's also a picture to explain that a little bit more in detail. Everything starts in the interpreter, as already mentioned, at the beginning. And the interpreter performs invocation counting. And once the invocation count of a method reaches a certain level, then a compile task gets generated in the C1 compile queue. A C1 compiler thread can pick it up and create a C1 compiled method, which is a tier three method in this example. And as already mentioned, tier three also does profiling, which includes invocation counting. So this compiled code still does invocation counting. And once a compiled method reaches or a method reaches this level, then a compile task gets generated in the C2 compile queue. And similar to C1, a C2 compiler thread can pick it up and create the fastest version of the method. And this is how it works for method invocations. But there may be long-running loops, which without any method invocations. And obviously, the invocation counting will not help in that case. That's why there's also back edge counting, which works similar. So it's almost the same slide. But here with back edge counters instead of the invocation counters with different limits. And what happens here? The compilers generate so-called OSR methods, which stands for on-stack replacement. And they are special methods. They have an entry point for the loop. And on-stack replacement is called this way, because an interpreted method gets removed from the stack frame. And it gets replaced by a compiled stack frame. That's why we call it on-stack replacement. So I've already talked about compiler threads. How many compiler threads are we using? Well, that depends on the machine we are running on. In the office, I have a 40 CPU Linux machine. And when using Printflex final, I can see that the VM selects CI compiler crown to 15. That is computed by a fancy formula. And one third of them are reserved for C1. And the remaining 10 in this case are reserved for C2 threads. And similar to compiler threads, the VM also decides on how many GC threads to use, which is 28 on my machine. And obviously, these numbers are pretty high for simple workloads. When you just do trivial things with your JVM, you don't need so many threads. We already heard this morning that threads are expensive. So we usually don't want that. And that's why we have implemented a new feature that was contributed by us. It's called Dynamic Number of Compiler Flats. We already shipped it with JDK 11. So it's not brand new, but it's the first time it is shown at a conference, I believe. And what we do by this new feature, we interpret these numbers as maximum numbers. So we start up to 15 compiler threads. And I'll get back to that later. But we start one thread of each type at startup and additional threads only on demand. There's a similar feature called Dynamic Number of GC Threads, which was already implemented by Oracle, which has switched it on by default with JDK 11. And with that, you get, of course, much lower resource usage. It's still possible to switch these features off to get the old behavior. So all compiler and GC threads get started at the VM startup. I have tuned all the memory settings to very low sizes. So the JVM should actually not use a lot of memory, but you can see virtual memory is pretty high here. And that's because of the threads. They reserve a lot of virtual memory, or they occupy virtual memory on Linux due to the G-LIB C. And if you don't switch off these new features, you can see we get a much lower virtual memory usage. It's from 6GB to 1.5 down. But it's not only about virtual memory. We, of course, also save other resources, but you can trace compiler threads also by this flag. It's a diagnostic flag, so you need to switch it on to enable to unlock these options. And as already mentioned, you can see that the JVM starts initially one compiler thread of each type, which is 1C2 and 1C1 thread. And they get kept alive for the whole lifetime of the JVM. And the other threads only get added on demand. That depends on the compile queue length and also on the available memory and code cache base, which is available, because we don't want to mess up things when the memory is already full. We don't want to start any further threads. And once these compiler threads don't have any work left to do, they will die after some time. And they die in the reverse order. They were generated, so we don't have any gaps in the compiler list. So that's the feature we are already using. And one remark on the memory usage of the compilers, C1 and C2 compilers, of course, use native memory. And in comparison to that, the Grail compiler uses Java heap. So that may be an issue, because your Java application uses the same heap, and you may need to configure a larger heap with the XMX flag. Otherwise, you may get out of memory issues. And it is also solved by using this shared library, because that uses a separate heap, which is part of the native image technology. So it doesn't use the regular Java heap, which you want to use for your application. So that's already it, what I wanted to tell. Maybe a few remarks. It is also possible to configure the compiler threads to use lower memory. For example, you can tune inlining. But of course, that may have performance implications. And it is also possible to set a node limit for the C1 compiler that will make it smaller, or will limit the memory it uses. But of course, that has always side effects. So I wouldn't recommend that in general. So I'm sure we have time for questions left. Excellent. Any questions? Yeah. We need a microphone. I see, still here. Stop. What? I was just wondering what the compiler thread count and heap size, or virtual memory size, or whatever sizes look like when you force tier one, when you only run with C1. I would assume it's fewer threads and less heap, but I didn't cover that. The virtual memory issue is due to the malloc arenas from Trilip C. And the first allocation already occupies a 128 megabyte block of heap. It's not really used. It's only virtual memory. So in most cases, it's not really a problem. But that is independent of which compiler it is, or which thread it is. It also happens with trowards, or with any other thread. There's also another way to fix that. You can configure the Trilip C to use less malloc arenas. There's a malloc arena max environment variable. And you can limit the memory by using that. That may have impacts on other performance things. Because if you have many native threads which perform a lot of concurrent mallocs, you may get issues with that. But for the TrWM itself, it works pretty well. We have tried that. We have experimented with using only one malloc arena. And the TrWM itself still works quite OK, because it has its own memory management. And we are not so much using many small concurrent mallocs. Good question, actually. Thanks. Further questions? So I have a question. For the server compiler and the client compiler, the code cache is managed by the sweeper. Is the same mechanism implemented for the Gral VM? The sweeper has a separate flat, so it's no longer part of the compiler flat. So I'm not aware of any relationship between Gral compiler and the sweeper. Maybe Andrew has a few thoughts about that. OK. No. So the short answer is? I'm not absolutely sure about that. So I wouldn't want to say unless I'm sure. But I do know that there's external, there's a change made to external code segments, not in the original code cache. And they're wrapped with a stub that points to them. So Gral is managing some of its own memory, I think. And I'm not sure how they get reclaimed. But Gral does know about deoptimization events. It may be also as a way that it can find out about the fact that something has been released. And there's a release protocol. I just don't know for sure. OK, thanks. I'm not really a Gral expert. I've worked a lot on C1 and C2, but not so much in Gral. But related to the code cache, there was a significant change back in the past. We only had the sweeper run by the compiler threads. And in the meantime, we have a dedicated sweeper thread. More questions? So I think we're done. Thanks, everyone, for your attention. Thank you.
 So, hello everyone and welcome to my talk about the trust and time compilers in the  OpenTrail.  I'm Martin Dörr and I work for SAP.  We are a small team working on the JVM at SAP in Waldorf and this time we have three  time, three talks in a row.  So we will also see two of my colleagues later on but let's get started with the agenda.  So I will talk about the trust and time compilers which translates Java bytecode into machine  code in the OpenTrail.  I will also talk about how different compilers work together and I also like to address resource  usage as well.  So first of all, how many compilers do we have in the OpenTrail?  We've already heard about two ones but how many do we have?  We have one more.  Three ones.  So the one we haven't talked today yet is the client compiler which is also called C1.  It compiles pretty quickly but with a lower optimization level and then we have the server  compiler also called C2.  We already had a talk about that one.  It's kind of the opposite, it compiles slowly but therefore with a high optimization level.  For example, it has a lot of loop optimizations and we've also heard about the escape analysis  and there are still people working on improvements for that.  Both compilers are available on a lot of platforms including PowerPC and S390 which are supported  by our team and these two compilers are used by default.  And we've also heard about Graal which is rather new in the OpenTrail.k.  It is still experimental in the OpenTrail.k.  That means it is not used by default.  You need to switch it on if you want to use it and it is developed on GitHub.  So updates get merged into the OpenTrail.k.  Also to Graal is that it is written in Java.  That is a big difference to the other two compilers.  And that also does a lot of optimization.  It has a more sophisticated escape analysis for example.  Andrew has already shown a few things about Graal.  So thanks for that.  And it is optimized for dynamic languages.  By the way, Graal compiler is also called GraalVM compiler.  And I'd like to show a few things Andrew has already mentioned.  I got one slide from Oracle.  So thanks to Oracle for providing it.  There are three different use cases of GraalVM.  And the Graal compiler is always in the center of it.  Together with the JVMCI, the Java Virtual Machine Compiler Interface.  And the use case on the very left is the one which is available in the OpenTrail.k.  So you have a Java application or Java methods which get compiled by Graal.  And they run on the hotspot VM, the Java Virtual Machine of the OpenTrail.k.  So that path on the very left is supported by the OpenTrail.k.  And in addition to that, on the right hand side, you can see the native image technology  Andrew also already mentioned.  And everything gets pre-compiled.  And there's something in between where only the Graal compiler is pre-compiled into a  shared library.  So that's the basic difference between this approach and this one.  Here Graal compiler itself is pre-compiled in a shared library.  So back to the different compilers.  I'd like to compare performance a little bit.  By the way, this is an old benchmark with an old JDK and an old garbage collector.  But don't care about numbers.  I think it's good to get a first impression about the performance of the different compilers.  So at the bottom you can see interpreter for reference.  So that means we are not using any just-in-time compiler.  And you can get that by specifying the runtime option minus xint.  That stands for interpreter mode.  So you will only use the interpreter, no-jit compilers at all.  And as you can see, the performance is pretty poor.  Already much faster is the C1, the client compiler.  You can select that, for example, by using this flag, tiered stop at level 3.  That might sound a little bit complicated.  And I have to note that level 3 means that C1 still performs profiling.  So you won't get the best performance out of C1 by that.  If you want to tune C1, you would select stop at level 1.  And then you would get C1 without profiling.  But in this case, I want the better profiling information.  That's why I left it on.  If you want to use the C2 compiler only, you can switch off tiered compilation.  And then you get the blue line, which is already much faster.  And the default configuration uses tiered compilation.  And you get the fastest startup and the best peak performance.  The best peak performance also because the profiling information is better.  But I'll explain the tiered compilation stuff later on in more detail.  So you should be able to understand that better at the end.  But for those who hate this old stuff, I have also a slide with the latest JDK.  So the same old benchmark with the latest JDK 15.  And you can already see that the peak performance is better with C2 especially.  And you can also see the green line, which is new.  That is grahl.  In order to use grahl, you need to use this switch, use JVMCI compiler, which is an experimental  option.  So you need to unlock it in addition.  And grahl is the default JVMCI compiler.  So you will get grahl by this flag.  So peak performance of grahl is good even for this very traditional workload.  Grahl performs this, of course, better for modern workloads.  For example, if you run Scala, that's what Twitter does a lot.  And I should also mention that the OpenShadeDK only contains the community edition of grahl.  There's also an enterprise version available, which contains more optimizations.  So you will get better performance with the enterprise edition.  And you can also see that the startup takes longer.  It takes a couple of seconds until here 4.5 seconds roughly to get peak performance.  And that's due to the fact that grahl itself is written in Java.  So the grahl compiler itself gets interpreted at the beginning.  And then later on, hot methods get compiled by C1.  And later on, they get compiled by which compiler?  The grahl compiler itself.  So grahl compiles itself.  And that takes a few seconds.  So this may be OK for large server applications, where you can afford spending a few seconds.  But there's also a possibility to fix that if you need a quicker startup.  And that's available with grahl VM.  So the trahl VM has a feature called printflexfinal.  And if you enable that, you will see all flag configurations, the VM sets for itself.  And you can also find use trahlmci native library.  And with grahl VM, that one is true by default.  And that means the trahlm is using the pre-compiled shared library.  So the grahl compiler is already pre-compiled.  And you get a pretty good startup with that.  So next, I've promised to explain tiered compilation a little bit.  So tiered compilation is basically the answer to the question of how these different compilers  work together.  As already mentioned at the beginning, everything starts at the interpreter, which is tier zero.  And then we have three different tiers for the C1 compiler.  Tier one is C1 without any profiling.  That is used only for trivial methods.  And the C1 believes that it's not worth optimizing further.  So we will stick on this trivial compilation.  And then there's tier two.  C1 uses reduced profiling.  And it does that when it thinks there's too much work to do.  So we just should make it quick.  And the default tier for C1 is tier three.  And you get the full profiling code compiled into the compiled method.  And then finally, the tier four is for the highest optimization level.  And it uses C2 compiler by default in OpenTraderK.  And you can replace it by grahl if you enable it explicitly.  You can also see the tiers when you enable print compilation.  You can see which method gets compiled at which tier.  And typically, most methods get started at tier three.  Then you get also tier four method compiled by C2 in this case.  But here's also a picture to explain that a little bit more in detail.  Everything starts in the interpreter, as already mentioned,  at the beginning.  And the interpreter performs invocation counting.  And once the invocation count of a method reaches a certain level,  then a compile task gets generated in the C1 compile queue.  A C1 compiler thread can pick it up and create a C1 compiled method,  which is a tier three method in this example.  And as already mentioned, tier three also does profiling,  which includes invocation counting.  So this compiled code still does invocation counting.  And once a compiled method reaches or a method reaches this level,  then a compile task gets generated in the C2 compile queue.  And similar to C1, a C2 compiler thread can pick it up  and create the fastest version of the method.  And this is how it works for method invocations.  But there may be long-running loops, which  without any method invocations.  And obviously, the invocation counting will not help in that case.  That's why there's also back edge counting, which works similar.  So it's almost the same slide.  But here with back edge counters instead of the invocation  counters with different limits.  And what happens here?  The compilers generate so-called OSR methods, which  stands for on-stack replacement.  And they are special methods.  They have an entry point for the loop.  And on-stack replacement is called this way,  because an interpreted method gets removed from the stack frame.  And it gets replaced by a compiled stack frame.  That's why we call it on-stack replacement.  So I've already talked about compiler threads.  How many compiler threads are we using?  Well, that depends on the machine we are running on.  In the office, I have a 40 CPU Linux machine.  And when using Printflex final, I  can see that the VM selects CI compiler crown to 15.  That is computed by a fancy formula.  And one third of them are reserved for C1.  And the remaining 10 in this case are reserved for C2 threads.  And similar to compiler threads, the VM also  decides on how many GC threads to use, which is 28 on my machine.  And obviously, these numbers are pretty high for simple workloads.  When you just do trivial things with your JVM,  you don't need so many threads.  We already heard this morning that threads are expensive.  So we usually don't want that.  And that's why we have implemented a new feature that  was contributed by us.  It's called Dynamic Number of Compiler Flats.  We already shipped it with JDK 11.  So it's not brand new, but it's the first time  it is shown at a conference, I believe.  And what we do by this new feature,  we interpret these numbers as maximum numbers.  So we start up to 15 compiler threads.  And I'll get back to that later.  But we start one thread of each type at startup  and additional threads only on demand.  There's a similar feature called Dynamic Number of GC Threads,  which was already implemented by Oracle,  which has switched it on by default with JDK 11.  And with that, you get, of course, much lower resource usage.  It's still possible to switch these features off  to get the old behavior.  So all compiler and GC threads get started at the VM startup.  I have tuned all the memory settings to very low sizes.  So the JVM should actually not use a lot of memory,  but you can see virtual memory is pretty high here.  And that's because of the threads.  They reserve a lot of virtual memory,  or they occupy virtual memory on Linux due to the G-LIB C.  And if you don't switch off these new features,  you can see we get a much lower virtual memory usage.  It's from 6GB to 1.5 down.  But it's not only about virtual memory.  We, of course, also save other resources,  but you can trace compiler threads also by this flag.  It's a diagnostic flag, so you need  to switch it on to enable to unlock these options.  And as already mentioned, you can see that the JVM starts  initially one compiler thread of each type, which is 1C2  and 1C1 thread.  And they get kept alive for the whole lifetime of the JVM.  And the other threads only get added on demand.  That depends on the compile queue length  and also on the available memory and code cache  base, which is available, because we don't want to mess up  things when the memory is already full.  We don't want to start any further threads.  And once these compiler threads don't have any work left to do,  they will die after some time.  And they die in the reverse order.  They were generated, so we don't have any gaps  in the compiler list.  So that's the feature we are already using.  And one remark on the memory usage of the compilers,  C1 and C2 compilers, of course, use native memory.  And in comparison to that, the Grail compiler uses Java heap.  So that may be an issue, because your Java application  uses the same heap, and you may need to configure a larger heap  with the XMX flag.  Otherwise, you may get out of memory issues.  And it is also solved by using this shared library,  because that uses a separate heap, which is part of the native  image technology.  So it doesn't use the regular Java heap, which you  want to use for your application.  So that's already it, what I wanted to tell.  Maybe a few remarks.  It is also possible to configure the compiler threads  to use lower memory.  For example, you can tune inlining.  But of course, that may have performance implications.  And it is also possible to set a node limit for the C1 compiler  that will make it smaller, or will limit the memory it uses.  But of course, that has always side effects.  So I wouldn't recommend that in general.  So I'm sure we have time for questions left.  Excellent.  Any questions?  Yeah.  We need a microphone.  I see, still here.  Stop.  What?  I was just wondering what the compiler thread count  and heap size, or virtual memory size, or whatever sizes  look like when you force tier one, when you only run with C1.  I would assume it's fewer threads and less heap,  but I didn't cover that.  The virtual memory issue is due to the malloc arenas  from Trilip C. And the first allocation already occupies  a 128 megabyte block of heap.  It's not really used.  It's only virtual memory.  So in most cases, it's not really a problem.  But that is independent of which compiler it is,  or which thread it is.  It also happens with trowards, or with any other thread.  There's also another way to fix that.  You can configure the Trilip C to use less malloc arenas.  There's a malloc arena max environment variable.  And you can limit the memory by using that.  That may have impacts on other performance things.  Because if you have many native threads which  perform a lot of concurrent mallocs,  you may get issues with that.  But for the TrWM itself, it works pretty well.  We have tried that.  We have experimented with using only one malloc arena.  And the TrWM itself still works quite OK,  because it has its own memory management.  And we are not so much using many small concurrent mallocs.  Good question, actually.  Thanks.  Further questions?  So I have a question.  For the server compiler and the client compiler,  the code cache is managed by the sweeper.  Is the same mechanism implemented for the Gral VM?  The sweeper has a separate flat, so it's no longer  part of the compiler flat.  So I'm not aware of any relationship  between Gral compiler and the sweeper.  Maybe Andrew has a few thoughts about that.  OK.  No.  So the short answer is?  I'm not absolutely sure about that.  So I wouldn't want to say unless I'm sure.  But I do know that there's external,  there's a change made to external code segments,  not in the original code cache.  And they're wrapped with a stub that points to them.  So Gral is managing some of its own memory, I think.  And I'm not sure how they get reclaimed.  But Gral does know about deoptimization events.  It may be also as a way that it can find out about the fact  that something has been released.  And there's a release protocol.  I just don't know for sure.  OK, thanks.  I'm not really a Gral expert.  I've worked a lot on C1 and C2, but not so much in Gral.  But related to the code cache, there  was a significant change back in the past.  We only had the sweeper run by the compiler threads.  And in the meantime, we have a dedicated sweeper thread.  More questions?  So I think we're done.  Thanks, everyone, for your attention.  Thank you.
jit2020_0_155.png jit2020_156_275.png jit2020_276_407.png jit2020_408_515.png jit2020_516_587.png jit2020_588_731.png jit2020_732_779.png jit2020_780_1559.png jit2020_1560_1859.png jit2020_1860_2495.png jit2020_2496_2819.png jit2020_2820_4259.png jit2020_4260_6155.png jit2020_6156_6227.png jit2020_6228_9491.png jit2020_9492_12587.png jit2020_12588_12755.png jit2020_12756_13655.png jit2020_13656_15659.png jit2020_15660_16235.png jit2020_16236_16319.png jit2020_16320_18299.png jit2020_18300_19703.png jit2020_19704_21203.png jit2020_21204_22811.png jit2020_22812_23699.png jit2020_23700_24299.png jit2020_24300_26207.png jit2020_26208_26507.png jit2020_26508_27155.png jit2020_27156_35495.png jit2020_35496_35610.png