視覺訊息(手勢、臉部表情)的存取性是面對面溝通與透過電腦中介傳播來進行溝通中一個主要具有差異的性質。我們會普遍認為在視訊溝通與音訊溝通下，因為能夠接收對方的訊息減少(像是視訊時可能看不到對方的肢體動作)甚至看不到對方，有可能會負面地影響在溝通中的手勢使用，甚至影響到溝通中的一些性質，像是溝通的效率。不過目前仍然缺乏足夠的證據來證明溝通媒介的豐富性是否會影響到在溝通中的非語言行為，尤其是手勢使用。
如何透過電腦中介傳播工具的設計來促進溝通是另外一個重要的議題。我們認為在對話的過程中根據對話者的行為提供適當的訊息，像是在溝通過程中顯示手勢使用程度的視覺化資訊，可能有效地促使在溝通中使用較多的手勢。如果可以透過類似的方式來改變溝通中的行為，或許一方面可以提供較多的視覺訊息，另一方面可以防止隱私上的顧慮。這樣的溝通方式對於那些在溝通的時候不想顯現自己的身分與外表尤其是面對陌生人，或許是一個適合的選擇。
本篇論文，進行兩個研究實驗。第一個主要提出一個研究非語言行為的新方法，稱為”Kinect-taping”。透過體感裝置Kinect來記錄人們在溝通中的手部移動，並且以此為基礎，量化手勢的使用程度與彼此手勢的相似程度。我們使用這個方法來瞭解面對面溝通與透過電腦中介傳播來進行溝通中可視度對於溝通中手勢使用的影響。第二個以”Kinect-taping”為基礎，設計一個呈現溝通中使用手勢的程度的視覺化溝通介面，並且提供不同可視度的溝通媒介，來瞭解在這些溝通環境的設定下對於手勢使用的程度與隱私顧慮程度的影響。

One key difference between face-to-face (F2F) communication and computer-mediated communication (CMC) is the availability of visual cues. It is often assumed that the reduction of visibility in audio and video conferencing may negatively impact the use of gesture to communicate, and thus negatively influence the outcomes of communication. There remains limited understanding around how visibility and richness of communication media may influence the non-verbal aspect of communication, especially gesture use.
On the other hand, another important issue in the design of CMC tools is on how to enhance communication actively. We consider that it is possible to promote the use of gesture in communication by providing appropriate feedback such as visualization that displays one’s own and the conversational partner’s amount of gesture during the conversational process. If we can successfully increase the amount of gesture with graphical visualization, then it is possible for us to create a technology design that functions as well as F2F and video, while still prevents from some negative side effects such as disclosure of private information or communication anxiety. This may give users more options of CMC in everyday usage when they don’t want to reveal too much personal information, such as appearance, to their conversational partners.
In this thesis, I conducted two studies. First, I studied how visibility influences gesture use by “Kinect-taping” F2F communication and CMC in brainstorming groups. “Kinect-taping” (similar to the sense of videotaping) refers to an analytic technique of using motion sensors to record and analyze group members’ hand movements during communication. Based on the positional data of body parts obtained from the motion sensor (Microsoft’s Kinect in this work), the technique helps derive measures useful for the purpose of behavioral analytics, such as the amount of gesture use by individuals and the extent of gestural similarity in a conversational dyad. Second, I used the Kinect-taping technique as the basis to design a new visualization interface by showing the amount of one’s own and the partner’s gesture use during online communication. I evaluated the effects of the visualization on gesture use and perceived privacy threat for communication media with different levels of visibility. Implications to future research and design are discussed.

1. Aggarwal, J. K. & Cai, Q. (1999). Human motion analysis: A review. Computer Vision and Image Understanding, 73, 428-440.
2. Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645.
3. Bishop, T. Xbox launches Avatar Kinect: A review by my avatar. GeekWire (2011). http://www.geekwire.com/2011/finally-cartoon-chat-xbox-launches-avatar-kinect/
4. Biswas, K., & Basu, S. K. (2011). Gesture recognition using Microsoft Kinect. International Conference on Automation, Robotics and Applications (ICARA).
5. Bos N., Olson J., Gergle D., Olson G., & Wright Z. (2002). Effects of four computer-mediated communications channels on trust development. Proceedings of CHI 2002.
6. Boyle, M., Edwards, C. & Greenberg, S. The Effects of Filtered Video on Awareness and Privacy. Proceedings of CSCW 2000.
7. Chartrand, T.L., & Bargh, J.A. (1999). The chameleon effect: The perception behavior link and social interaction. Journal of Personality & Social Psychology, 76, 893-910.
8. Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127-149). Washington, DC: APA.
9. Daft, R. L., & Lengel, R. H. (1986). Organizational information requirement, media richness and structural design. Management Science, 32. 554-571.
10. DiMicco, J., M., Hollenbach, K., J., & Bender, W. (2006). Using visualization to review a group’s interaction dynamics. Proceedings of CHI EA 2006.
11. Driskell, J. E., & Radtke, P. H. (2003). The effect of gesture on speech production and comprehension. Human Factor, 45, 445-454.
12. Dunbar, N. E., Jensen, M. L., & Burgoon, J. K. (2011). A dyadic approach to the detection of deception. Proceedings of the Annual Hawaii International Conference on System Sciences (HICSS).
13. Filho, J., E., V., Inkpen, K., M., & Czerwinski, M. (2009). Image, appearance and vanity in the use of media spaces and video conference system. Proceedings of GROUP 2009.
14. Fussell, S. R., Setlock, L. D., Yang, J., Ou, J., Mauer, E. M., & Kramer, A. (2004). Gestures over video streams to support remote collaboration on physical tasks. Human-Computer Interaction, 19, 273-309.
15. Garrod, S., & Pickering, M. J. (2004). Why is conversation so easy? Trends in Cognitive Sci., 8, 8-11.
16. Gergle, D., & Clark, A. T. (2011). See what I’m saying? Using dyadic mobile eye tracking to study collaborative reference. Proceedings of CSCW 2011.
17. Giles, H. (1973). Accent mobility: A model and some data. Anthropological Linguistics, 15, 87-105.
18. Giles, H., Coupland, J. & Coupland, N. (1991). Contexts of Accommodation. NY: Cambridge University Press.
19. Goldin-Meadow, S. (1999). The role of gesture in communication and thinking. Trends in Cognitive Sciences, 3, 419-429.
20. Kenny, D. A., Mannetti, L., Pierro, A., & Livi, S. (2002). The statistical analysis of data from small groups. J. Personality and Social Psych., 83, 126-137.
21. Khoshelham, K, & Elberink, S. O. (2012). Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensor, 1437-1454.
22. Kim, T., Chang, A., Holland, L., & Pentland, A., S. (2008). Meeting mediator: enhancing group collaboration using sociometric feedback. Proceedings of CSCW 2008.
23. Krauss, R. M., Chen, Y., & Chawla, P. (1996). Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us? In M. Zanna (Ed.), Advances in Experimental Social Psychology (pp. 389-450). Academic Press.
24. Leshed, G., Perez, D., Hancock, J. T., Cosley, D., Birnholtz, J., Lee, S., McLeod, P., & Gay, G. (2009). Visualizing real-time language-based feedback on teamwork behavior in computer-mediated groups. Proceedings of CHI 2009.
25. Littell, R., Milliken, G. A., Stroup, W. W., & Wolfinger, R. D. (1996). SAS system for mixed models. SAS Inst.
26. Mauer, R. E., & Tindall, J. H. (1983). Effects of postural congruence on client’s perceptions of counselor empathy. J. Counseling Psych., 30, 158-163.
27. McNeill, D. (2005). Gesture and thought. Chicago: University of Chicago Press.
28. Niederhoffer, K., & Pennebaker, J.W. (2002). Linguistic style matching in social interaction. Journal of Language & Social Psychology, 21(4), 337-360.
29. Osborn, A. (1957). Applied Imagination. NY: Scribner.
30. Poppe, R., Van Der Zee, S., Heylen, D. K., & Taylor, P. J. (2013). AMAB: Automated measurement and analysis of body motion. Behavior Research Methods.
31. Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27, 169-190.
32. Reyes, M., Dominguez, G., & Escalera, S. (2011). Feature weighting in dynamic time warping for gesture recognition in depth data. Proceedings of ICCV Workshops 2011.
33. Roberto, H., Patil, S., White, D., Dawson, J., Whalen, P., & Kapadia, A. (2013). Attire: conveying information exposure through avatar apparel. Proceedings of CSCW 2013.
34. Scissors L. E., Gill A. J., Geraghty K., & Gergle D. (2009). In CMC we trust: the role of similarity. Proceedings of CHI 2009.
35. Setlock, L D., Fussell, S. R., & Neuwirth. C. (2004). Taking it out of context: collaborating within and across cultures in face-to-face settings and via instant messaging. Proc. of CSCW 2004.
36. Sproull, L., & Kiesler, S. (1986). Reducing social context cues: Electronic mail in organizational communications. Management Science, 32, 1492-1512.
37. Sturm, J., Herwijnen, O., H., Eyck, A., & Terken, J. (2007). Influencing social dynamics in meetings through a peripheral display. Proceedings of ICMI 2007.
38. Veinott, E., Olson, J., Olson, G., & Fu, X. (1999). Video helps remote work: Speakers who need to negotiate common ground benefit from seeing each other. In Proceedings of CHI 1999.
39. Wang, H-C. , Fussell, S. R., & Setlock, L. D. (2009). Cultural difference and adaptation of communication styles in computer-mediated group brainstorming. Proceedings of CHI 2009.
40. Wang, H-C., & Fussell S. R. (2010). Groups in groups: conversational similarity in online multicultural multiparty brainstorming. Proceedings of CSCW 2010.
41. Wang, H-.C., Fussell, S. R., & Cosley, D. (2011). From diversity to creativity: Stimulating group brainstorming with cultural differences and conversationally-retrieved pictures. Proceedings of CSCW 2011.
42. Whittaker, S., & O’Conaill, B. (1997). The role of vision in face-to-face and mediated communication. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 23-49).
43. Yabe, T., & Tanaka, K. (1999). Similarity Retrieval of Human Motion as Multi-stream Time Series Data. Proc. of International Symposium on Database Applications in Non-Traditional Environments (DANTE).
44. Zhao, Q-A., & Stasko, J. T. (1998). The Awareness-Privacy Tradeoff in Video Supported Informal Awareness: A Study of Image-Filtering Based Techniques.