这篇论文是ICLR 2017的一篇量化相关的论文。本文提出了一种渐进式量化的方法（ INQ ）：先分组量化，然后冻结已量化的部分并训练未量化的部分，重复以上步骤多次直到所有权重都被量化。这种渐进式量化的方法可以把一个预训练的全精度模型近乎无损地压缩成一个低精度的模型，在Alexnet，VGG，ResNet等模型上都表现良好，甚至可以压缩成三值模型，精确度比TWN等方法都要好。
亮点1：量化表征方式 variable-length encoding scheme
一般的量化方法（也就是谷歌的IAO算法）是这样的：
$scale=max(abs(W))$
$\alpha=(2^{bit-1}-1) \div scale$
$W=round(W \times \alpha) \div \alpha$
这种量化方式是最简单的对称式量化，其实就是把一段区域上所有的实数点对应到其最近的离散均匀分布的点上。INQ采用的量化方法有所不同，它是把权重映射到指定的正负2的n次幂或者0：
其中：
$n_{1} = floor(log{2}(\frac{4scale}{3}))$
$n_{2}=n_{1}+1-2^{bit-2}$
$n_{2} \leq n_{1}$
映射公式如下：
这样量化我觉得比IAO的方法要灵活，而且是一种非线性的量化。鉴于权重的分布一般是如下图所示的双峰分布：
这样的非线性量化可以很好地拟合双峰的位置（注意n1的选取）还有类似正态分布的特点（中间多、四周少，利用了幂函数的特点），使得量化后的权重与原权重相似度提高，权重的变化不那么大，有利于之后的重训练恢复精度的进行。缺点就是有点复杂，不够直观（不过这也不能叫缺点叭）。
亮点2：渐进式量化的策略 weight partition, group-wise quantization and re-training
首先，作者阐释了如何选择当前需量化的部分。论文中并没有明确说明选择的策略，只是笼统地说选择的策略是受了剪枝的启发。作者在这里引用了两篇论文：
Learning both weights and connections for efficient neural networks (Han et al., 2015)
Dynamic network surgery for efficient dnns (Guo et al., 2016)
然后，作者解释了为什么在重训练的时候要冻结已量化的部分，只训练未量化的部分。作者先指出得到最终的量化神经网络的本质是一个优化问题：
直接从头训练这个网络是不实际的，一来是浪费时间，我们可以利用全精度的模型，干嘛要从头训练？二来是很多论文都指出量化网络的学习能力没那么强，从头训练无法达到比较优秀的水平。考虑到哪些权重需要被量化是已经确定了的，我们可以通过多次迭代来逐渐量化。因此，以上优化问题等价于下面这个优化问题：
这个优化问题可以用SGD来求解，更新的方法是：
以上内容纯粹是论文凑字数，最重要的是看这个梯度更新。我们可以发现，如果T(i,j)=0，意思是这个权重被量化了，那么这个权重就不需要被更新。所以我们要得到最终的最优化模型，确实只需要梯度更新未量化的权重即可。
总体的算法如下所示：
总的来说就是用与剪枝类似的方法选择可以量化的权重，冻结量化了的权重并重训练未量化的权重，反复以上两步直到所有权重都量化完毕。其实还是很简单的一个idea，但效果很好😀。
实验结果 EXPERIMENTAL RESULTS
一个字：牛！
详细的训练参数论文中有给出（好像没给学习率），代码也开源了，感兴趣的小伙伴可以复现一下。
原作者的Caffe代码：https://github.com/AojunZhou/Incremental-Network-Quantization
非官方的Pytorch代码：https://github.com/Mxbonn/INQ-pytorch
我自己也写了个，太丑了就不放上来了……
需要注意的地方是部分权重冻结的代码要怎么写。Mxbonn是通过自定义optimizer来实现的，我是在梯度更新前保存需要冻结的权重，梯度更新后再把冻结的权重还原回去（有点ugly）。
附录给了一张5-bit AlexNet model量化后权重分布的表，大家可以参考下：
Your blog has become my go-to source for information on this subject. I have learned so much from your articles, and I always look forward to reading more.
Simply wanna input that you have a very nice internet site , I the style it really stands out. Myranda Guntar Arne
You made some nice points there. I did a search on the subject matter and found most guys will consent with your site. Tamqrah Napoleon Ruzich
Hi friends, pleasant article and pleasant urging commented here, I am actually enjoying by these. Margeaux Ron Herzberg
Unbelievably intriguing highlights that you have mentioned, say thanks a lot for submitting. Bette Gonzales Trinity
Some really nice and useful information on this site, as well I believe the design and style contains great features. Kristal Dionisio Duwe
Pretty! This has been a really wonderful post. Thank you for providing this information. Johna Marsh Bertrando
Very informative article post. Much thanks again. Cool. Jeanette Stearn Shawnee
I like it when individuals come together and share ideas. Arielle Greggory Osber
You made some decent points there. I did a search on the subject matter and found most people will go along with with your site. Vally Benjamin Rather
Simply wanna comment that you have a very decent website, I love the design it really stands out. Drusy Byran Lange
I like this article, useful stuff on here : D. Kimbra Allistir Soinski
Thank you for your blog article. Thanks Again. Really Great. Carlita Fields Boggers
Way cool! Some extremely valid points! I appreciate you writing this post and the rest of the website is very good. Kally Floyd Dorcy
Hey There. I found your blog the uuse of msn. That is a very smartly written article. Jourdan Sebastian Karb
I am perpetually thought about this, thanks for posting. Jorey Booth Rossuck
Some truly quality articles on this site, saved to bookmarks. Sybille Moe Roath
Remarkable! Its truly remarkable post, I have got much clear idea about from this paragraph. Carla Kennie Boehmer
You completed a few good points there. I did a search on the issue and found most people will have the same opinion with your blog. Auguste Sigfried Hedwig
I am so grateful for your blog. Thanks Again. Awesome. Darbie Taber Eirene
Hello. This article was extremely motivating, especially since I was looking for thoughts on this topic last Monday. Toni Murdoch Aili
You are not right. I am assured. I can prove it. Write to me in PM, we will talk. Pam Hal Tiga
My brother suggested I might like this web site. He was entirely right. Danyelle Nikolos Lars
I feel this is one of the such a lot vital info for me. Jackie Ripley Kissner
This article presents clear idea designed for the new visitors of blogging, that in fact how to do blogging. Lolita Royall Gemina
Thanks again for the post. Much thanks again. Really Cool. Gabbie Reilly Woolley
Im grateful for the blog post. Much thanks again. Keep writing. Marya Kerry Niles
Greetings! Very useful advice within this article! It is the little changes that produce the most significant changes. Zaria Shae Nobel
I feel genuinely thrilled to have seen your current webpage and appear forward in order to so many more enjoyable times reading here. Thanks a lot once again for all the particular details.