張量分解是用來壓縮深度學習網路的其中一種技術。Tucker分解(TD)與Canonical Polyadic分解(CPD)為兩種主要常見的分解方法；然而這兩種方法卻在模型壓縮上還未徹底地探究他們的性質。本論文除了研究這兩種在模型壓縮上的性質以外，基於實作上的觀察，我們也提出了融合這兩種技術之混和式張量分解(MTD)，在容許誤差範圍底下，達到更好的壓縮量。MTD考慮每一層convolution layer作適當的壓縮，併進一步的調整好維持效能。本論文以在CIFAR10上的模型VGG11以及VGG16來做壓縮，在準確率一個百分比的誤差底下，分別能達到32倍與37倍的壓縮比，這能達到比其他以張量分解的技術更好的壓縮量。

Tensor decomposition is one of the model reduction techniques for deep neural networks. Two commonly used methods are Tucker decomposition (TD) and Canonical Polyadic decomposition (CPD). However, their properties haven't been well understood for model compression. In this research, we studied the relation of model accuracy and compression ratio for TD and CPD applying to convolution neural networks (CNN). Based on the studied results, we developed a mixed tensor decomposition (MTD) algorithm to achieve better compression ratio while keeping similar accuracy as the original models. MTD selects the most suitable decomposition method for each layer, and further fine-tunes the models to recover the accuracy. We have conducted experiments using VGG11 and VGG16 with CIFAR10 dataset, and compared MTD with other tensor decomposition algorithms. The results show that MTD can achieve compression ratio 32 $\times$ and 37 $\times$ for VGG11 and VGG16 respectively with less than 1\% accuracy drops, which is much better than the state-of-the-art tensor decomposition algorithms for model compression.

[1]  Misha Denil et al. “Predicting parameters in deep learning”. Advances in Neural Information Processing Systems. 2013, pp. 2148–2156.
[2]  Yunchao Gong et al. “Compressing Deep Convolutional Networks using Vector Quan- tization”. CoRR abs/1412.6115 (2014). arXiv: 1412.6115. URL: http://arxiv. org/abs/1412.6115.
[3]  Jiaxiang Wu et al. “Quantized Convolutional Neural Networks for Mobile Devices”. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, pp. 4820–4828.
[4]  Song Han, Huizi Mao, and William J. Dally. “Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding”. 4th International Conference on Learning Representations, ICLR 2016, Conference Track Proceedings. 2016. URL: http://arxiv.org/abs/1510.00149.
[5]  Hyeji Kim, Muhammad Umar Karim Khan, and Chong-Min Kyung. “Efficient Neural Network Compression”. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019, pp. 12569–12577.
[6]  Jian Xue, Jinyu Li, and Yifan Gong. “Restructuring of Deep Neural Network Acoustic Models with Singular Value Decomposition”. Interspeech. 2013, pp. 2365–2369.
[7]  Tara N Sainath et al. “Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets”. 2013 IEEE international conference on acoustics, speech and signal processing. IEEE. 2013, pp. 6655–6659
[8]  Emily L Denton et al. “Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation”. Advances in Neural Information Processing Systems. 2014, pp. 1269–1277.
[9]  Vikas Sindhwani, Tara N. Sainath, and Sanjiv Kumar. “Structured Transforms for Small-Footprint Deep Learning”. Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada. Ed. by Corinna Cortes et al. 2015, pp. 3088–3096. URL: http://papers.nips.cc/paper/5869-structured- transforms-for-small-footprint-deep-learning.
[10]  Vadim Lebedev et al. “Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition”. 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings. 2015. URL: http://arxiv.org/ abs/1412.6553.
[11]  Yong-Deok Kim et al. “Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications”. 4th International Conference on Learning Representations, ICLR 2016, Conference Track Proceedings. 2016.
[12]  Alexander Novikov et al. “Tensorizing neural networks”. Advances in Neural Information Processing Systems. 2015, pp. 442–450.
[13]  Julia Gusak et al. “Automated Multi-Stage Compression of Neural Networks”. 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Work- shops 2019, Seoul, Korea (South), October 27-28, 2019. IEEE, 2019, pp. 2501–2508.
[14]  Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. “Tensor Decomposition for Compressing Recurrent Neural Network”. 2018 International Joint Conference on Neural Networks, IJCNN 2018, Rio de Janeiro, Brazil, July 8-13, 2018. IEEE, 2018, pp. 1–8. DOI: 10.1109/IJCNN.2018.8489213. URL: https://doi.org/10. 1109/IJCNN.2018.8489213.
[15]  Marcella Astrid and Seung-Ik Lee. “CP-decomposition with Tensor Power Method for Convolutional Neural Networks compression”. 2017 IEEE International Conference on Big Data and Smart Computing, BigComp 2017, Jeju Island, South Korea, February 13-16, 2017. IEEE, 2017, pp. 115–118. DOI: 10.1109/BIGCOMP.2017. 7881725. URL: https://doi.org/10.1109/BIGCOMP.2017.7881725.
[16]  Wei-Shiang Lin, Hao-Ning Wu, and Chih-Tsun Huang. “Accelerating Convolutional Neural Networks using Iterative Two-Pass Decomposition”. MA thesis. Department of Computer Science, 2018, p. 36.
[17]  Karen Simonyan and Andrew Zisserman. “Very Deep Convolutional Networks for Large-Scale Image Recognition”. 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015. URL: http://arxiv.org/abs/1409.1556.
[18]  Alex Krizhevsky, Geoffrey Hinton, et al. “Learning multiple layers of features from tiny images”. MA thesis. Department of Computer Science, 2009.
[19]  Tamara G. Kolda and Brett W. Bader. “Tensor Decompositions and Applications”. SIAM Review 51.3 (2009), pp. 455–500. DOI: 10.1137/07070111X. URL: https: //doi.org/10.1137/07070111X.
[20]  Andrew G. Howard et al. “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications”. CoRR abs/1704.04861 (2017). arXiv: 1704.04861. URL: http://arxiv.org/abs/1704.04861.
[21]  Shinichi Nakajima et al. “Global analytic solution of fully-observed variational Bayesian matrix factorization”. J. Mach. Learn. Res. 14.1 (2013), pp. 1–37. URL: http://dl. acm.org/citation.cfm?id=2502582.
[22]  Xiyu Yu et al. “On Compressing Deep Models by Low Rank and Sparse Decomposition”. 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. IEEE Computer Society, 2017, pp. 67–76. DOI: 10.1109/CVPR.2017.15. URL: https://doi.org/10.1109/CVPR.2017.15.
[23]  Dong Yu et al. “Exploiting Sparseness In Deep Neural Networks For Large Vocabulary Speech Recognition”. 2012 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE. 2012, pp. 4409–4412.
[24]  Lieven De Lathauwer. “Decompositions of a higher-order tensor in block terms—Part II: Definitions and uniqueness”. SIAM Journal on Matrix Analysis and Applications 30.3 (2008), pp. 1033–1066.
[25]  Haiping Lu, Konstantinos N Plataniotis, and Anastasios N Venetsanopoulos. “A survey of multilinear subspace learning for tensor data”. Pattern Recognition 44.7 (2011), pp. 1540–1551.
[26]  Andrzej Cichocki et al. “Tensor decompositions for signal processing applications: From two-way to multiway component analysis”. IEEE signal processing magazine 32.2 (2015), pp. 145–163.
[27]  Edwin Stoudenmire and David J Schwab. “Supervised learning with tensor networks”. Advances in Neural Information Processing Systems. 2016, pp. 4799–4807.
[28]  Stavros Efthymiou, Jack Hidary, and Stefan Leichenauer. “TensorNetwork for machine learning”. arXiv preprint arXiv:1906.06329 (2019).
[29]  Adam Paszke et al. “PyTorch: An Imperative Style, High-Performance Deep Learning Library”. Advances in Neural Information Processing Systems 32. Ed. by H. Wallach et al. Curran Associates, Inc., 2019, pp. 8024–8035. URL: http://papers. neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
[30]  Jean Kossaifi et al. “TensorLy: Tensor Learning in Python”. Journal of Machine Learning Research 20.26 (2019), pp. 1–6. URL: http://jmlr.org/papers/v20/18-277.html.
[31]  Jason Yosinski et al. “How transferable are features in deep neural networks?” Advances in neural information processing systems. 2014, pp. 3320–3328.
[32]  Christopher J. Hillar and Lek-Heng Lim. “Most Tensor Problems Are NP-Hard”. J. ACM 60.6 (2013), 45:1–45:39. DOI: 10.1145/2512329. URL: https://doi.org/ 10.1145/2512329.
[33]  Marcella Astrid, Seung-Ik Lee, and Beom-Su Seo. “Rank selection of CP-decomposed convolutional layers with variational bayesian matrix factorization”. 2018 20th International Conference on Advanced Communication Technology (ICACT). IEEE. 2018, pp. 347–350.
[34]  Taehyeon Kim, Jieun Lee, and Yoonsik Choe. “Bayesian Optimization-Based Global Optimal Rank Selection for Compression of Convolutional Neural Networks”. IEEE Access 8 (2020), pp. 17605–17618. DOI: 10.1109/ACCESS.2020.2968357. URL: https://doi.org/10.1109/ACCESS.2020.2968357.
[35]  Avinab Saha et al. “Fitness Based Layer Rank Selection Algorithm for Accelerating CNNs by Candecomp/Parafac (CP) Decompositions”. 2019 IEEE International Conference on Image Processing (ICIP). IEEE. 2019, pp. 3402–3406.
[36]  Kaiming He et al. “Deep Residual Learning for Image Recognition”. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016. IEEE Computer Society, 2016, pp. 770–778. DOI: 10.1109/CVPR.2016.90. URL: https: //doi.org/10.1109/CVPR.2016.90.
[37]  Christian Szegedy et al. “Going deeper with convolutions”. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015. IEEE Computer Society, 2015, pp. 1–9. DOI: 10.1109/CVPR.2015.7298594. URL: https://doi.org/10. 1109/CVPR.2015.7298594.