An adaptive two-dimensional video retargeting algorithm based on region-wised mosaic-guided approach is proposed in this paper. We first generate the saliency map of each frame with the weighted sum of PQSM and gradient value. Within camera motion, we generate the camera model and mosaic the video sequence, and then use the camera model to mosaic the saliency map of each frame to record the camera motion and the regions of moving objects. To alleviate the spatial deformation in each region, a semi-automatic segmentation step is presented to segment the region in panoramic mosaic map. Then, we directly resize the panoramic mosaic map to generate a set of global optimal scaling factor subject to the information loss, spatial coherence, and resource budget, and scaling factor ratio constraints. Moreover, for the application on the on-line system, we further proposed to release some constraints, and reformulated the problem in a simpler way that the optimization process can be implemented in the linear programming method, and accelerate the time consumption. Our experimental results show that the proposed method can achieve a better quality of temporal and spatial coherence in a systematic way, even video contains grave camera motion and object motion. Besides, our method accelerates the speed in optimization process for further on-line application.

科技日新月異，播放裝置的種類也隨之多元，藉由不同撥放裝置分享影音/視訊檔案逐漸受到喜愛。然而，受限於這些撥放裝置彼此螢幕解析度的差異，我們必須對影像/視訊畫面解析度進行調整才能在撥放裝置上撥放。最直觀的做法是對影像/視訊等比例的縮小，或者擷取符合畫面大小的解析度。然而，這類的作法容易造成嚴重的形變或訊號上的損失。因此，近年來已有不少方法被提出，希望在調整解析度的過程中能保留下人眼視覺感興趣的部分，且捨棄或縮小不被人眼所注意的區域，讓濃縮的結果相較於原本的影像/視訊能不被人眼發現。本文提出了利用全景接圖來引導視訊影像濃縮的方法。對於鏡頭的移動的處理，本文利用全景接圖之技術，紀錄視訊影像中的時間與空間相對位置，以確保在不同時間點上空間中相同位置的縮放比例相同。另外，為了讓不同區域上的縮放比例能相近，因此我們使用了半自動式的影像切割方法來對全景接圖劃分區域，以保持空間中物體的完整性。最後，利用最佳化方法，直接對全景接圖做解析度上的調整，這樣得到的濃縮比例不但能有效地保持空間的一致性，也能達到物體形狀的完整性。此外，對於二維的濃縮，我們認為視訊影像中的同一個物體其橫方向與縱方向濃縮的比例應受限於人眼感興趣的程度，即同一物體在不同方向的濃縮比例應是相關的。而為了往後及時運算的需求，我們提出將原本非線性的問題化簡為線性來加速最佳化的運算，然而從實驗數據上看來，即使如此會造成濃縮的視訊影像品質較差，但不至於有嚴重的缺陷；對於較為複雜的運鏡與物體移動的視訊影像，我們提出的方法也都還有不錯的濃縮結果。

[1] Shamir and O. Sorkine, “Visual media retargeting,” in ACM SIGGRAPH ASIA Courses (SIGGRAPH ASIA '09), 2009, pp. 1□13.
[2] S. Avidan and A. Shamir, “Seam carving for content-aware image resizing,” ACM Trans. Graphics, vol. 26, no. 3, pp. 16, 2007.
[3] M. Rubinstein, A. Shamir, and S. Avidan, “Improved seam carving for video retargeting,” ACM Trans. Graphics, vol. 27, no. 3, pp. 16, 2008.
[4] L. Wolf, M. Guttmann, and D. Cohen-Or, "Non-homogeneous content-driven video-retargeting," in Proc. IEEE Int. Conf. Comput. Vis., 2007, pp. 1□6, Rio de Janeiro, Brazil.
[5] P. Krähenbühl, M. Lang, A. Hornung, and M. Gross, “A system for retargeting of streaming video,” ACM Trans. Graphics, vol. 28, no. 5, 2009.
[6] J.-S. Kim, J.-H. Kim and C.-S. Kim, "Adaptive image and video retargeting technique based on Fourier analysis," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pp. 1730□1737, Sept. 2009, Kyoto, Japan.
[7] L. Shi, J. Wang, L. Y. Duan, and H. Lu, " Consumer video retargeting: context assisted spatial-temporal grid optimization,” in Proc. ACM Int. Conf. Multimedia, pp. 301−310, Oct. 2009, Beijing, China.
[8] S. Kopf, T. Haenselmann, D. Farin, and W. Effelsberg, " Automatic generation of summaries for the Web,” in Proc. IS&T/SPIE Conf. Storage and Retrieval for Media Databases, pp. 417–428, Jan. 2004, San Jose, USA.
[9] F. Liu and M. Gleicher, “Video retargeting: automating pan and scan,” in Proc. ACM Int. Conf. Multimedia, pp. 241□250. Oct. 2006, Santa Barbara, CA.
[10] Y.-S. Wang, H. Fu, O. Sorkine, T.-Y. Lee, and H.-P. Seidel, “Motion-aware temporal coherence for video resizing,” ACM Trans. Graphics, vol. 28, no. 5, 2009.
[11] W.-H. Cheng, C.-W. Wang, and J.-L. Wu, “Video adaptation for small display based on content recomposition,” IEEE Trans. Circuits Syst. Video Technol. vol. 17, no. 1, pp. 43□58, Jan. 2007
[12] V. Setlur, T. Lechner, M. Nienhaus and B. Gooch, “Retargeting images and video for preserving information saliency,” IEEE Computer Graphics and Applications, vol. 27, no. 5, pp. 80□88, Sept.□Oct. 2007.
[13] C.-K. Chiang, S.-F. Wang, Y.-L. Chen, and S.-H. Lai, “Fast JND-based video carving with GPU acceleration for real-time video retargeting,” IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 11, pp. 1588□1597, Nov. 2009.
[14] D. Han, X. Wu, and M. Sonka, “Optimal multiple surfaces searching for video/image resizing - a graph-theoretic approach,” in Proc. IEEE Int. Conf. Comput. Vis., pp. 1026□1033, 2009, Kyoto, Japan.
[15] S. Kopf, J. Kiess, H. Lemelson, and W. Effelsberg, “FSCAV-fast seam carving for size adaptation of videos,” in Proc. ACM Int. Conf. Multimedia, pp. 321□330, Oct. 2009, Beijing, China.
[16] W.-L. Chao, H.-H. Su, S.-Y. Chien, W. Hsu, and J.-J. Ding, “Coarse-to-fine temporal optimization for video retargeting based on seam carving,” in Proc. IEEE Int. Conf. Multimedia Expo, July 2011, Barcelona, Spain.
[17] Y.-F. Zhang, S.-M. Hu, and R. R. Martin, “Shrinkability maps for content-aware video resizing,” Computer Graphics Forum, vol. 27, no. 7, pp. 1797□1804, 2008.
[18] S.-F. Wang, and S.-H. Lai, “Compressibility-aware Media retargeting with structure preserving,” IEEE Trans. Image Process. vol. 20, no. 3, pp. 855□865, March 2011.
[19] Y. Luo, J. Yuan, P. Xue, and Q. Tian, “Salient region detection and its application to video retargeting,” in Proc. IEEE Int. Conf. Multimedia Expo, July 2011, Barcelona, Spain.
[20] T.-C. Yen, C.-M. Tsai, and C.-W. Lin, “Maintaining temporal coherence in video retargeting using mosaic-guided scaling,” IEEE Trans. Image Process. vol. 20, no. 8, pp. 2339–2351, Aug. 2011.
[21] Z. Lu, W. Lin, X. Yang, E. Ong, and S. Yao, “Modeling visual attention's modulatory aftereffects on visual sensitivity and quality evaluation,” IEEE Trans. Image Process., vol. 14, no. 11, pp. 1928□1942, Nov. 2005.
[22] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91□110, 2004.
[23] R. Szeliski, “Image alignment and stitching: a tutorial,” Foundations and Trends in Computer Graphics and Vision (FTCGV), vol. 2, no. 1, pp. 1□104, 2006.
[24] P. F. Felzenszwalb and D. P. Huttenlocher, "Efficient Graph-Based Image Segmentation," Int. J. Comput. Vis., Vol. 59, Num. 2, Sept. 2004.
[25] Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski, “A Bayesian approach to digital matting,” in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., Dec. 2001, Hawaii, U.S.A.
[26] Y. Guo, F. Liu, J. Shi, Z.-H. Zhou, and M. Gleicher, “Image retargeting using mesh parametrization,” IEEE Trans. Multimedia, vol. 11, no. 5, pp. 856□867, Aug. 2009.
[27] T. Ren, Y. Liu, and G. Wu, "Image retargeting based on global energy optimization,” in Proc. IEEE Int. Conf. Multimedia Expo, pp. 406□409. June 2009, New York, USA.