{"push_id":967327634,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"168c1f0a1bae79c764efaf2d4d69558b856c4b9a","before":"4c1f4471cb5cfb0cb810be0025b4d5eb71a152d4","commits":[{"sha":"168c1f0a1bae79c764efaf2d4d69558b856c4b9a","author":{"email":"5e4896e4185c162a2e84b9528cd53bff19819528@apache.org","name":"Ufuk Celebi"},"message":"[FLINK-3120] [runtime] Manually configure Netty's ByteBufAllocator\n\ntl;dr Change default Netty configuration to be relative to number of slots,\ni.e. configure one memory arena (in PooledByteBufAllocator) per slot and use one\nevent loop thread per slot. Behaviour can still be manually overwritten. With\nthis change, we can expect 16 MB of direct memory allocated per task slot by\nNetty.\n\nProblem: We were using Netty's default PooledByteBufAllocator instance, which\nis subject to changing behaviour between Netty versions (happened between\nversions 4.0.27.Final and 4.0.28.Final resulting in increased memory\nconsumption) and whose default memory consumption depends on the number of\navailable cores in the system. This can be problematic for example in YARN\nsetups where users run one slot per task manager on machines with many cores,\nresulting in a relatively high number of allocated memory.\n\nSolution: We instantiate a PooledByteBufAllocator instance manually and wrap\nit as a NettyBufferPool. Our instance configures one arena per task slot as\ndefault. It's desirable to have the number of arenas match the number of event\nloop threads to minimize lock contention (Netty's default tried to ensure this\nas well), hence the number of threads is changed as well to match the number\nof slots as default. Both number of threads and arenas can still be manually\nconfigured.\n\nThis closes #1593.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-flink/commits/168c1f0a1bae79c764efaf2d4d69558b856c4b9a"}]}