<p>Neo is optimizing inference using compilation, which is different and often orthogonal to compression</p>
<ul>
<li><p><strong>compilation</strong> makes inference faster and lighter by specializing the prediction application, notably: (1) changing the environment in which the model runs, in particular replacing training frameworks by the least amount of necessary math libraries, (2) optimizing the model graph to be prediction-only and grouping together operators that can be, (3) specializing the runtime to use best the specific hardware and instructions available on a given target machine. Compilation is not supposed to change the model math, thereby doesn't change its footprint on disk</p>
</li>
<li><p><strong>compression</strong> makes inference faster by removing model weights or making them smaller (quantization). Weights can be removed by pruning (dropping weights that do not influence much results or distillation (training a small model to mimic a big model).</p>
</li>
</ul>
<p>At the time of this writing, SageMaker Neo is a managed compilation service. That being said, compilation and compression can be combined, and you can prune or distill your network before feeding it to Neo.</p>
<p>SageMaker Neo covers a large grid of hardware targets and model architectures, and consequently leverages numerous backends and optimizations. Neo internals are publicly documented in many places:</p>
<ul>
<li><p>According to <a href="https://aws.amazon.com/blogs/machine-learning/unlock-performance-gains-with-xgboost-amazon-sagemaker-neo-and-serverless-artillery/" rel="nofollow noreferrer">this blog</a>, Neo uses <a href="https://treelite.readthedocs.io/en/latest/" rel="nofollow noreferrer"><strong>Treelite</strong></a> for tree models optimization (<a href="https://mlsys.org/Conferences/doc/2018/196.pdf" rel="nofollow noreferrer"><em>Treelite: toolbox for decision tree deployment</em></a>, Cho et Li)</p>
</li>
<li><p>According to its <a href="https://aws.amazon.com/sagemaker/neo/" rel="nofollow noreferrer">landing page</a>, Neo uses <a href="https://tvm.apache.org/" rel="nofollow noreferrer"><strong>Apache TVM</strong></a> too. TVM is the leading open-source compiler, developed by <a href="https://tqchen.com/" rel="nofollow noreferrer">Tianqi Chen</a> and the <a href="https://github.com/dmlc" rel="nofollow noreferrer">DMLC</a> community (that also co-authored <a href="https://arxiv.org/abs/1603.02754" rel="nofollow noreferrer">XGBoost</a> and <a href="https://arxiv.org/abs/1512.01274" rel="nofollow noreferrer">MXNet</a>). TVM tricks are abundantly documented in <a href="https://arxiv.org/pdf/1802.04799.pdf" rel="nofollow noreferrer"><em>TVM: An Automated End-to-End Optimizing Compiler for Deep Learning</em></a> (Chen et al)</p>
</li>
<li><p>According to <a href="https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-neo-makes-it-easier-to-get-faster-inference-for-more-ml-models-with-nvidia-tensorrt/" rel="nofollow noreferrer">this blog</a>, Neo also sometimes leverages <a href="https://developer.nvidia.com/tensorrt" rel="nofollow noreferrer"><strong>NVIDIA TensorRT</strong></a>, the official inference optimization stack from NVIDIA</p>
</li>
<li><p>Neo also uses a number of Amazon-developed optimization:</p>
<ul>
<li><a href="https://arxiv.org/pdf/1907.02154.pdf" rel="nofollow noreferrer"><em>A Unified Optimization Approach for CNN Model Inference on Integrated GPUs</em></a> (Wang et al): <em>&quot;Our work is already deployed
in Amazon SageMaker Neo Service&quot;</em></li>
<li><a href="https://www.usenix.org/system/files/atc19-liu-yizhi.pdf" rel="nofollow noreferrer"><em>Optimizing CNN Model Inference on CPUs</em></a> (Liu et al)<em>&quot;NeoCPU is used in Amazon SageMaker Neo Service&quot;</em></li>
</ul>
</li>
</ul>
Which techniques are used by SageMaker Neo for model optimizations
<p>Does SageMaker Neo (SageMaker compilation job) use any techniques for model optimization? Are there any compression techniques used (distillation, quantization etc) to reduce the model size?</p>
<p>I found some description here (<a href="https://docs.aws.amazon.com/sagemaker/latest/dg/neo.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/sagemaker/latest/dg/neo.html</a>) regarding quantization but it's not clear how it could be used.</p>
<p>Thanks very much for any insight.</p>