在知覺使用者介面(PUIs)中，利用指尖追蹤以辨識手勢是一種讓電腦察覺使用者的行為的常見方式。本論文旨在提出偵測指尖位置以及追蹤指尖軌跡的方法，讓使用者能夠藉著移動伸出的食指而在空中寫字。
本篇方法首先結合顏色及空間結構資訊作為雙鏡頭所拍攝出之左右影像的比對值，並利用整張影像同步做aggregation的概念取得視差圖。接著藉由此篇設計的立體匹配演算法而得的深度資訊，分割出手的區域與掌心的位置。而在追蹤使用者指尖的部分，針對指尖指向側邊以及指向鏡頭兩種情況，以兩種不同的偵測方法來處理。最後，我們也提出了偵測和追蹤時的角度範圍限制，以及指尖追蹤位置的改善方法，以提高追蹤的準確性。由實驗結果顯示，無論在室外或室內的環境，此方法皆能在11.29(frames/s) 的速度下，有效地追蹤出使用者在空中寫字時的指尖位置。

In the class of perceptive user interfaces (PUIs), gesture recognition with fingertips tracking is one of the natural ways of making the computer aware of what the user is doing. The aim of this thesis is to propose a method for detecting fingertip locations in stereo images and tracking fingertip trajectories across image frames. For demonstrating the quality performance, the applications of writing-in-the-air with stretched index finger is given.
The proposed method is working on both color images and the corresponding disparity maps using a designated stereo matching algorithm aware of the color and depth features of hands simultaneously. Within matching, the color and structure information are combined first to be the matching costs of left and right image taken from twin-lens cameras, and the concept of full-image aggregation is used to obtain the disparity maps. Then, the hand region is segmented and the palm center is located with the help of depth information. For tracking the position of user’s fingertip, two detection procedures are employed for two cases of gestures (pointing to the camera or pointing to the side). Finally, angular restriction and position refinement of fingertips are applied to enhance the accuracy. The experimental results demonstrate that our method performs well for both indoor and outdoor tracking at a speed of about 11.29 frames per second for the 640×480 images. Besides, its capabilities are further demonstrated in the application of writing-in-the-air.

[1] D. Scharstein and R. Szeliski, “A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,” International Journal of Computer Vision, vol. 47, no. 1-3, 2002, pp 7-42.
http://www.middlebury.edu/stereo/
[2] K. Yoon and S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 4, April 2006, pp.650-656.
[3] C. Richardt, D. Orr, I. Davies, A. Criminisi, and N. Dodgson, “Real-time spatiotemporal stereo matching using the dual-cross-bilateral grid,” European Conference on Computer Vision (ECCV), September 2010, vol. 6313, pp.510-523.
[4] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, “Fast cost-volume filtering for visual correspondence and beyond,” IEEE Transactions on Pattern Analysis and Machine Intelligence, February 2013, vol. 35, no 2, pp.504-511.
[5] X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang, “On Building an Accurate Stereo Matching System on Graphics Hardware,” IEEE International Conference on Computer Vision Workshops (ICCV Workshops), November 2011, pp. 464-474.
[6] C. Stentoumis, L. Grammatikopoulos, I. Kalisperakis, E. Petsa, and G. Karras, “A local Adaptive Approach for Dense Stereo Matching in Architectural Scene Reconstruction,” 5th International Workshop on 3D Virtual Reconstruction and Visualization of Complex Architectures (3D-ARCH), February 2013.
[7] Qingqing Yang, Dongxiao Li, Lianghao Wang, and Ming Zhang, “Full-image Guided Filtering for Fast Stereo Matching,” IEEE Signal Processing Letters, March 2013, vol. 20, no. 3, pp. 237-240.
[8] R. Khoshabeh, S. H. Chan, and T. Q. Nguyen, “Spatio-Temporal Consistency in Video Disparity Estimation,” International Conference on Acoustics, Speech and Signal Processing, May 2011, pp. 885-888.
[9] C. Richardt, D. Orr, I. Davies, A. Criminisi, and N. Dodgson, “Real-time spatiotemporal stereo matching using the dual-cross-bilateral grid,” European Conference on Computer Vision (ECCV), September 2010, vol. 6313, pp.510-523.
[10] Q. Yang, L. Wang, R. Yang, H. Stewénius, and D. Nistér, “Stereo matching with color-weighted correlation, hierarchical belief propagation and occlusion handling,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 3, March 2009, pp.492-504.
[11] Yong Seok Heo, Kyoung Mu Lee, and Sang Uk Lee, “Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space”, IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp. 445 – 452.
[12] Cuong Cao Pham and Jae Wook Jeon, “Domain Transformation-Based Efficient Cost Aggregation for Local Stereo Matching,” IEEE Transactions on Circuits and Systems for Video Technology, July 2013, vol. 23, no. 7, pp. 1119-1130.
[13] J. Kovac, P. Peer, F. Solina, “Human skin color clustering for face detection”, The IEEE Region 8 EUROCON 2003-International Conference on Computer as a Tool , September 2003, vol. 2, pp. 144-148.
[14] L. Jin, D. Yang, L. Zhen, and J. Huang, “A novel vision based finger-writing character recognition system,” 18th International Conference on Pattern Recognition (ICPR), August 2006, vol. 1, pp.1104-1107.
[15] D. Lee and S. Lee, “Vision-based finger action recognition by angle detection and contour analysis,” Electronics and Telecommunications Research Institute Journal, June. 2011, vol. 33, no. 3, pp. 415-422.
[16] Hui Liang, Junsong Yuan, and Daniel Thalmann, “3D Fingertip and Palm Tracking in Depth Image Sequences,” Proceedings of the 20th ACM international conference on Multimedia, October 2012, pp.785-788.
[17] Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, “Real-time Hand Tracking on Depth Images,” IEEE Visual Communications and Image Processing (VCIP), November 2011, pp. 1-4.
[18] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System,” Proceedings of the 4th International Conference on Internet Multimedia Computing and Service (ICIMCS), September 2012, pp.70-74.
[19] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW), July 2013, pp. 1 - 4.
[20] Zhou Ren, Junsong Yuan, and Zhengyou Zhang, “Robust Hand Gesture Recognition Based on Finger-Earth Mover’s Distance with a Commodity Depth Camera,” Proceedings of the 19th ACM International Conference on Multimedia, November 2011, pp. 1093-1096.
[21] Lingchen Chen, Feng Wang, Hui Deng, and Kaifan Ji, “A Survey on Hand Gesture Recognition,” International Conference on Computer Sciences and Applications (CSA), December 2013, pp. 313 – 316.