在本篇論文中，我們提出了一個修正3D影片的深度圖的方法。在計算深度圖的過程中，我們不只考慮空間的一致性，還使深度在不同時間保持一致。我們的方法可以被應用在動態的場景或是相機旋轉的情況下。在修正3D影片深度圖的相關研究中，大部分的研究都會用不同的方式修正動態物體跟靜態景物的深度。因為靜態物體的深度會在不同時間上保持一致，動態物體則不然。許多研究都會用二元圖將靜態的景物跟動態的物體分開作處理。而我們則是用機率的方式來描述一個像素屬於動態的可能性。相較於只有0與1的二元圖，我們的作法是比較有彈性的。在我們的方法中，我們先對每個時間點計算出對應的深度圖。由於計算過程中沒有考慮時間的一致性，這使得我們可能對一個靜態物體上在不同時間獲得不同的深度。為了結合時間的資訊，我們利用光流法來獲得相鄰影格的對應關係。若該像素屬於靜態景物，那我們將會使深度在時間以及空間上保持一致。但若像素屬於動態物體，那我們會重視深度在空間上的一致性，而不是時間。最後，我們利用迭代式最小平方法來求得最佳的深度影片。經由能量的最小化，我們可以讓修正後的深度圖更加準確並在時間軸上保持一致的深度。

In this thesis, we propose a novel method to refine the depth map from stereo video. The proposed method enforces the spatial and temporal consistency of the depth maps computed from stereo video. Most previous works on estimating depth map from videos applied different ways to recover the depth by first deciding if a pixel belongs to dynamic region or static region. Because the static region has the same depth over time, but the dynamic region does not. Most methods employ a binary map to separate two regions in an image. Our method employs a probabilistic framework to describe the probability for a dynamic pixel or a static pixel. Compared to the binary map, the probability map is more flexible. In the proposed approach, we estimate the initial disparity map for each frame. The initial disparity map is temporally inconsistent because it is estimated with a stereo matching method from a single pair of stereo images. We apply the optical flow to estimate the corresponding pixels between neighboring frames. If the pixel is static, we smooth its depth both spatially and temporally. If the pixel is dynamic, we just smooth its depth spatially. In the final step, we optimize the depth map by applying iterative reweighted least squares (IRLS). Our experiments show the refined depth maps by using the proposed algorithm are more accurate and temporally consistent through experiments on a number of different stereo videos.

[1] A. Bruhn, J.Weickert and C. Schn¨orr. Lucas/Kanade meets Horn/Schunk: combining local and global optical flow methods. In International Journal of Computer Vision (IJCV), 61(3):211–231, 2005.
[2] A. Klaus, M. Sormann and K. Karner. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. ICPR 2006.
[3] C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. High-quality video view interpolation using a layered representation. In SIGGRAPH, 2004, pp. 600-608. Available: http://research.microsoft.com/en-us/um/people/larryz/videoviewinterpolation.htm
[4] C. Liu. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral Thesis. Massachusetts Institute of Technology. May 2009.
[5] C. Liu, W. T. Freeman, E. H. Adelson and Y. Weiss. Human-assisted motion annotation. In CVPR, pp. 1-8, 2008
[6] C. Richardt, D. Orr, I. Davies, A. Criminisi, and N. A. Dodgson, Realtime Spatiotemporal Stereo Matching Using the Dual-Cross-Bilateral Grid. In ECCV, 2010. Available: http://www.cl.cam.ac.uk/research/rainbow/projects/dcbgrid/
[7] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR, 2011.
[8] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In IJCV, 47:7–42, 2002.
[9] F. Liu and V. Philomin. Disparity estimation in stereo sequences using scene flow. In BMVC, 2009.
[10] G. Zhang, J. Jia, T.-T. Wong, and H. Bao. Consistent depth maps recovery from a video sequence. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(6):974–988, 2009.
[11] I. Feldmann, M. Mueller, F. Zilly, R. Tanger, K. Mueller, A. Smolic, P. Kauff, and T. Wiegand. HHI Test Material for 3D Video. ISO/IEC JTC1/SC29/WG11 MPEG 2008/M15413, 2008.
[12] Larsen, E.S., Mordohai, P., Pollefeys, M., Fuchs, H.: Temporally consistent reconstruction from multiple video streams using enhanced belief propagation. In ICCV, pp. 1–8, 2007.
[13] K. Karsch, C. Liu and S.B. Kang. Depth Extraction from Video Using Non-parametric Sampling. In ECCV, 2012. Oral presentation. Available: http://www.kevinkarsch.com/depthtransfer/index.html
[14] M. Bleyer, C. Rother, and P. Kohli. Surface stereo with soft segmentation. In CVPR, 2010.
[15] M. Bleyer and M. Gelautz. A layered stereo algorithm using image segmentation and global visibility constraints. ICIP, 2004.
[16] M. Bleyer and M. Gelautz. Temporally consistent disparity maps from uncalibrated stereo videos. In ISPA, pages 383-387, sept. 2009.
[17] M. Bleyer, M. Gelautz, C. Rother, and C. Rhemann. A stereo approach that handles the matting problem via image warping. In CVPR, 2009.
[18] M. J. Hannah, Computer matching of areas in stereo images, Ph.D. thesis, Stanford University, 1974.
[19] M. Yang, X. Cao, and Q. Dai. Multiview video depth estimation with spatial-temporal consistency. In BMVC, 2010.
[20] Q. Yang, A Non-Local Cost Aggregation Method for Stereo Matching. In CVPR, 2012, 1402-1409.
[21] R. Khoshabeh , S. Chan and T. Nguyen. Spatio–temporal consistency in video disparity estimation. Proc. IEEE ICASSP, 2011
[22] S. H. Chan, P. E. Gill, and T. Q. Nguyen. An Augmented Lagrangian Method for Total Variation Image Restoration. In IEEE Transactions on Image Processing, 2010.
[23] T. Brox, A. Bruhn, N. Papenberg, and J.Weickert. High accuracy optical flow estimation based on a theory for warping. In ECCV, pages 25–36, 2004.
[24] W. Yang, G. Zhang, H. Bao, J. Kim and H. Y. Lee. Consistent Depth Maps Recovery from a Trinocular Video Sequence. In CVPR, 2012.
[25] Z. Wang and Z. Zheng. A region based stereo matching algorithm using cooperative optimization. In CVPR, pages 1-8, 2008.