本篇論文探討適用於真實世界的場景分析所面臨的兩個困難處：對於類別會持續增加的真實世界，我們需要一個比完全監督式學習模型更加實用的方法；另外，對於物體於畫面中所佔面積很小或是鮮少出現的類別，要正確的將其標示出來仍非常具有挑戰性。在論文中，我們提出（一）善用已經存在之監督式學習模型可偵測出的類別資訊，以及（二）轉移真實世界資料庫中新類別的資訊，以達到適用於真實世界的場景分析之目的。我們透過提出的「內容適應」以及「類別知曉」馬可夫隨機場參數架構，同時結合並善用監督式學習的知識以及可轉移資訊，因此，我們提出的方法不需要事先訓練模型而且適用於真實世界中。在實驗中，我們透過SIFT Flow以及LMSun資料庫進行驗證，實驗結果顯示我們的方法在真實世界場景分析的假設下優於現有的方法。

This thesis addresses two major challenges in semantic segmentation for real-world data. First, with ever-increasing semantic labels, we need a more pragmatic approach other than existing fully-supervised methods. Second, semantic segmentation for very small or rarely-appeared objects are still very challenging for existing methods. In this thesis, we propose to (1) fully utilize the predicted label information from an existing supervised model and to (2) infer newly generated labels via label transfer from a real-world dataset. We propose a “content-adaptive” and “label-aware” MRF framework to jointly exploiting both the supervised and label-transferrable knowledge. The proposed method needs no off-line training and can easily adapt to real-world data. Experimental results on SIFT Flow and LMSun datasets demonstrate the effectiveness of the proposed method, and show promising performance over state-of-the-art methods under the real-world scenario.

[1] J. Long, E. Shelhamer and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. CVPR, 2015.
[2] A. Sharma, O. Tuzel, and D. W. Jacobs, “Deep hierarchical parsing for semantic segmentation,” in Proc. CVPR, 2015.
[3] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected CRFs,” in Proc. ICLR, 2015.
[4] H. Noh, S. Hong and B. Han, “Learning deconvolution network for semantic segmentation,” in Proc. ICCV, 2015.
[5] X. Qi, J. Shi, S. Liu, R. Liao and J. Jia, “Semantic segmentation with object clique potentials,” in Proc. ICCV, 2015.
[6] G. Lin, C. Shen, A. van den Hengel and I. Reid, “Efficient piecewise training of deep structured models for semantic segmentation,” in Proc. CVPR, 2016.
[7] J. Tighe and S. Lazebnik, “Finding things: Image parsing with regions and per-exemplar detectors,” in Proc. CVPR, 2013.
[8] J. Tighe, M. Niethammer and S. Lazebnik, “Scene parsing with object instances and occlusion ordering,” in Proc. CVPR, 2014.
[9] C. Farabet, C. Couprie, L. Najman, and Y. LeCun, “Learning hierarchical features for scene labeling,” IEEE Trans. PAMI, vol.35, no.8, Aug. 2013, pp.1915-1929.
[10] B. Shuai, G. Wang, Z. Zuo, B. Wang, and L. Zhao, “Integrating parametric and non-parametric models for scene labeling,” in Proc. CVPR, 2015.
[11] C. H. Ma, C. T. Hsu and B. Huet, “Nonparametric scene parsing with deep convolutional features and dense alignment,” in Proc. ICIP, 2015.
[12] C. Liu, J. Yuen and A. Torralba, “Nonparametric scene parsing via label transfer,” IEEE Trans. PAMI, vol.32, no.12, Dec. 2011, pp.2368-2382.
[13] J. Tighe and S. Lazebnik, “Superparsing: Scalable nonparametric image parsing with superpixels,” IJCV, vol.101, no.2, Jan. 2013, pp.329-349.
[14] F. Tung and J. J. Little, “CollageParsing: Nonparametric scene parsing by adaptive overlapping windows,” in Proc. ECCV, 2014.
[15] J. Yang, B. Price, S. Cohen and M. Yang, “Context driven scene parsing with attention to rare classes,” in Proc. CVPR, 2014.
[16] M. George, “Image parsing with a wide range of classes and scene-level context,” in Proc. CVPR, 2015.
[17] M. Rubinstein, C. Liu and W. T. Freeman, “Joint inference in weakly-annotated image datasets via dense correspondence,” IJCV, early access, 2016.
[18] Z. Lu, Z. Fu, T, Xiang, P. Han, Liwei, Wang, and X. Gao, “Learning from weak and noisy labels for semantic segmentation,” IEEE Trans. PAMI, early access, 2016.
[19] G. Papandreou, L. C. Chen, K. Murphy and A. L. Yuille, “Weakly- and semi-supervised learning of a DCNN for semantic image segmentation,” arXiv:1502.02734 [cs.CV], 2015.
[20] S. Hong, J. Oh, B. Han and H. Lee, “Learning transferrable knowledge for semantic segmentation with deep convolutional neural network,” arXiv:1512.07928 [cs.CV], 2015.
[21] P. H. Pinheiro and R. Collobert, “Recurrent convolutional neural networks for scene labeling,” in Proc. ICML, 2014.
[22] A. Krizhevsky, I. Sutskever and G. Hinton, “ImageNet classification with deep convolutional neural networks,” in Proc. NIPS, 2012.
[23] S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Toward real-time object detection with region proposal networks,” in Proc. NIPS, 2015.
[24] M. Everingham, L. V. Gool, C. K. I. Williams, J. Winn and A. Zisserman, “The PASCAL visual object classes (VOC) challenge,” IJCV, vol.88, no.2, June 2010, pp.303-338.
[25] Y. Boykov, O. Veksler and R. Zabin, “Fast approximate energy minimization via graph cuts,” IEEE Trans. PAMI, vol.23, no.11, Nov. 2001, pp.1222-1239.
[26] R. Mottaghi, X. Chen, X. Liu, N. G. Cho, S. W. Lee, S. Fidler, R. Urtasun and Al Yuille, “The role of context for object detection and semantic segmentation in the wild,” in Proc. CVPR, 2014.
[27] R. Girshick, J. Donahue, T. Darrell and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. CVPR, 2014.
[28] Model Zoo. https://github.com/shelhamer/fcn.berkeleyvision.org
[29] F. Tung and J. J. Little, “Scene parsing by nonparametric label transfer of content-adaptive windows,” CVIU, vol.143, Feb. 2016, pp.191-200.
[30] K. E. A. van de Sande, J. R. R. Uijlings, T. Gevers, A. W. M. Smeulders, “Segmentation as selective search for object recognition,” in Proc. ICCV, 2011.
[31] C. L. Zitnick, and P. Dollár, “EdgeBoxes: locating object proposals from edges,” in Proc. ECCV, 2014.
[32] C. Galleguillos, B. McFee and G. R. G. Lanckriet, “Iterative category discovery via multiple kernel metric learning,” IJCV, vol.108, no.1, May 2014, pp.115-132.
[33] S. Gould and Y. Zhang, “Patchmatchgraph: Building a graph of dense patch correspondences for label transfer,” in Proc. ECCV, 2012.
[34] S. Gould, J. Zhao, X. He and Y. Zhang, “Superpixel graph label transfer with learned distance metric,” in Proc. ECCV, 2014.