Because of the Human-Computer Interaction (HCI) applications become more and more popular in our daily life, of which the most continent one is by hand gesture. Therefore, the recognition and reconstruction of hand gesture plays an importance role in the research area of related HCI topics. Using hand gesture to or HCI has the benefits containing intuitive, non-invasive, and user friendly. Here, we develop a muti-view-muti-camera-based methodology to capture the hand motion from different view angles.
In this thesis, we form a 3-D hand model with structural constraint and kinematical constraint from frontal view angle and side view angle. To estimate human hand state in high degree of freedom space, we use separable state based particle filter (SSBPF) to the track the finger motion. SSBPF separates the states of the variable into several different parts, and track each part with the particle filter, Furthermore we use different kind of features including silhouette, Chamfer distance, and depth map as the observation. By fusing these features in different view angles, the self-occlusion problem can be solved. Based on the hand model, different features, and SSBPF, we implement a human hand gesture reconstruction system, which can automatically reconstruct human hand model with a 3-D virtual hand with less than 19 degree average error.

由於人機介面的應用日益普及，相關的溝通方式包含了勢辨識、語音辨識、或是肢體語言辨識等等，都已經被廣泛研究並應用在日常生活之中，其中以利用手勢作為媒介的方式最為直覺。因此手勢辨識與重建一向是個相當重要的研究議題，使用手勢作為人機介面除了直覺之外，直觀、使用者親和力高與自然使用方式等優勢。然而以目前已經實現的系統或是設備當中，大多數都需要使用額外的設備附著於手部上或是使用特殊的標誌當作參數擷取的依據。由此可知，此方式對於特殊設備上的依賴與需求是非常高的，而相對的負擔這些特殊設備的成本也很昂貴，導致整體的系統發展受制於儀器設備上而無法有效率發展與應用。據此，本論文嘗試建立一套以多視角多攝影機針對手部擷取系統，降低設備需求以及增加使用者親和力。
在本論文當中，針對手部運動的情況，對手部三維模型進行限制，在使用這些限制之下，可增加辨識與估測手部模型的效率與準確性。而在以往相關的研究主題都是用單一攝影機，且限制手掌必須與攝影機垂直，此限制可避免在側面情況下的手指間相互遮蔽情況。而本論文使用多隻攝影機在側面的情況之下可有效的解決手指相互遮蔽情形，可正常估測與重建出相對應的三維手部模型。且在以往的研究當中，預估類似的高維度參數空間的系統是極為耗時的，為了有效率的實現在高維度空間中估測手部參數空間，我們採用Separable State Based Particle Filter (SSBPF)。使用此方法切割欲估測的參數空間成幾個子集合，接著仍以Particle Filter的原理估測在下個時間點之參數空間之參數。而使用多攝影機，系統亦採用多種不同的特徵空間，包含了景深、Chamfer Distance與輪廓作為追蹤的依據，結合以上幾點，本論文實現了以視覺為基礎之多視角多攝影機為主的手部動作參數擷取系統，同時本系統對於除了手腕全域運動(旋轉與位移)都正常估測之外，對與手指間的彎曲情形亦可估測出來，其結果對於精準度方面的平均誤差都在可容忍的範圍之內，同時當發生手指發生彼此間的遮蔽情形之下也可正確預估手部的彎曲角度。

[1] M. Bray, E. Koller-Meier, and L. Van Gool, "Smart particle filtering for 3D hand tracking," in IEEE Int. Conf. Automatic Face and Gesture Recognition., 2004, pp. 675-680.
[2] M. Bray, E. Koller-Meier, and L. Van Gool, "Smart particle filtering for high-dimensional tracking," Computer Vision and Image Understanding., vol. 106, issue 1, 2007, pp. 116-129.
[3] D. Huan and E. Charbon, "3D Hand Model Fitting for Virtual Keyboard System," in IEEE Workshop on Applications of Computer Vision., 2007, pp. 31-31.
[4] W. Ying, J. Y. Lin, and T. S. Huang, "Capturing natural hand articulation," in Proc. IEEE ICCV., 2001, vol.2,pp. 426-432.
[5] A. Erol, G. Bebis, M. Nicolescu, R. Boyle, and X. Twombly, "A Review on Vision-Based Full DOF Hand Motion Estimation," in Proc. IEEE CVPR., 2005, pp. 75-75.
[6] A. El-Sawah, C. Joslin, N. D. Georganas, and E. M. Petriu, "A Framework for 3D Hand Tracking and Gesture Recognition using Elements of Genetic Programming," in Proc. IEEE CRV., 2007, pp. 495-502.
[7] A. Erol, G. Bebis, M. Nicolescu, R. D. Boyle, and X. Twombly, "Vision-based hand pose estimation: A review," Computer Vision and Image Understanding., vol. 108, issue1-2, 2007, pp. 52-73.
[8] DataGlove, "5DT Fifth Dimension Technologies," http://www.5dt.com/products/pdataglove14.html.
[9] D. J. Sturman and D. Zeltzer, "A survey of glove-based input," IEEE Trans. on Computer Graphic and Application., vol. 14, issue 1, pp. 30-39, 1994.
[10] R. G. O'Hagan, A. Zelinsky, and S. Rougeaux, "Visual gesture interfaces for virtual environments," Interacting with Computers., vol. 14, issue 3, 2002, pp. 231-250.
[11] C.-C. Lien, "A scalable model-based hand posture analysis system," Machine Vision and Applications., vol. 16, No. 3, 2005, pp. 157-169.
[12] W. Ying and T. S. Huang, "View-independent recognition of hand postures," in Proc. IEEE CVPR, 2000, pp. 88-94 vol.2.
[13] V. Athitsos and S. Sclaroff, "Estimating 3D hand pose from a cluttered image," in Proc. IEEE CVPR., 2003, vol.2, pp. II-432-9.
[14] V. Athitsos and S. Sclaroff, "An appearance-based framework for 3D hand shape classification and camera viewpoint estimation," in Proc. IEEE FGR., 2002, pp. 40-45.
[15] A. Imai, N. Shimada, and Y. Shirai, "3-D hand posture recognition by training contour variation," Proc. IEEE FGR., 2004, pp. 895-900.
[16] C. Wen-Yan, C. Chu-Song, and H. Yi-Ping, "Appearance-guided particle filtering for articulated hand tracking," in Proc. IEEE CVPR., 2005, vol. 1, pp. 235-242.
[17] M. Bray, E. Koller-Meier, N. N. Schraudolph, and L. Van Gool, "Fast stochastic optimization for articulated structure tracking," Image and Vision Computing., vol. 25, issue 3, 2007, pp. 352-364.
[18] E. B. Sudderth, M. I. Mandel, W. T. Freeman, and A. S. Willsky, "Visual Hand Tracking Using Nonparametric Belief Propagation," Proc. IEEE CVPR., 2004, pp. 189-189.
[19] W. Ying, J. Lin, and T. S. Huang, "Analyzing and capturing articulated hand motion in image sequences," IEEE Trans. on Pattern Analysis and Machine Intelligence., vol. 27, no. 12, 2005, pp. 1910-1922.
[20] B. Stenger, P. R. S. Mendonca, and R. Cipolla, "Model-based 3D tracking of an articulated hand," in Proc. IEEE CVPR., 2001, vol.2, pp. II-310-II-315.
[21] "OpenGL." Available: http://www.opengl.org
[22] M. Isard and A. Blake, "CONDENSATION—Conditional Density Propagation for Visual Tracking," International Journal of Computer Vision., vol. 29, pp. 5-28, 1998.
[23] M. Isard and A. Blake, "ICONDENSATION: Unifying Low-Level and High-Level Tracking in a Stochastic Framework," in Proc. ECCV., 1998, p. 893.
[24] K. Nummiaro, E. Koller-Meier, and L. Van Gool, "An adaptive color-based particle filter," in Image and Vision Computing., vol. 21, issue 1, 2003, pp. 99-110.
[25] 翁精佑, "基於視覺之手部動作參數擷取系統," 清華大學碩士論文, 九十六年七月
[26] M. Kolsch and M. Turk, "Fast 2D Hand Tracking with Flocks of Features and Multi-Cue Integration," in Proc. IEEE CVPR., 2004, pp. 158-158.
[27] Z. Haiting, W. Xiaojuan, and H. Hui, "Research of a Real-time Hand Tracking Algorithm," in Proc. IEEE 0ICNN&B., 2005, vol. 2, pp. 1233-1235.
[28] H. Zhou, L. Xie, and X. Fang, "Visual Mouse: SIFT Detection and PCA Recognition," in Proc. IEEE CISW., 2007, pp. 263-266.
[29] C. Shan, T. Tan, and Y. Wei, "Real-time hand tracking using a mean shift embedded particle filter," Pattern Recognition., vol. 40, issue 7, 2007, pp. 1958-1970.
[30] Bumblebee2, "Point Grey Research " http://www.ptgrey.com/products/stereo.asp.
[31] G. Amayeh, G. Bebis, A. Erol, and M. Nicolescu, "A Component-Based Approach to Hand Verification," in Proc. IEEE CVPR., 2007, pp. 1-8.
[32] B. Stenger, A. Thayananthan, P. H. S. Torr, and R. Cipolla, "Estimating 3D hand pose using hierarchical multi-label classification," Image and Vision Computing., vol. 25, issue 12, 2007, pp. 1885-1894.