在現今的網路上，利用軟體的弱點來入侵受害者的電腦系統有越來越多的趨勢。攻擊者最常使用的手法就是透過軟體的漏洞來植入一段可在受害者電腦上可執行的有害程式碼，此程式碼可再受害者的電腦上執行一些破壞工作，或是竊取使用者敏感的資料。這段有害程式碼會利用一些特定的Windows API 來完成此破壞工作。在這篇論文中，我們提出了一種利用監測Windows API 的異常使用狀況來偵測是否有未知的有害程式碼在執行。這個系統使用Windows API Hooking 以及有限狀態機來偵測有害程式碼。利用此二種工具來監視Windows API 的行為，並判斷是否為合法的使用行為。若是合法，則程式會正常執行下去；若是不合法的行為，就可以中斷其執行破壞的工作。至於行為的評估，則是分成二個階段：記錄階段跟監視階段。利用在此二階段中的有限狀態機來得到一個行為特徵值，然後監視階段所得的行為特徵值來跟記錄階段結果比對。若監視階段中得到的特徵值沒在記錄階段中發現，則有可疑的未知程式碼在執行。此系統的效能很不錯，且對系統的需求很低。而且，此系統也不需要原軟體的原始碼以及修改硬體。

Nowadays, there is a growing trend of attacks using vulnerabilities to compromise victim's computer system. Attackers often write shellcodes through these vulnerabilities to compromise the victim's system. These shellcodes are a kind of "harmful code", and use the specific Windows APIs to destroy the system or disclose sensitive data. However, the shellcodes should get the APIs’ addresses before they use it. In this thesis, we proposed an effective system using API hooking and finite state machine to detect shellcodes. Our system detects the shellcodes by monitoring the Windows APIs detect the shellcodes through the API hooks. We use a finite state machine (FSM) to evaluate the behavior of shellcodes. In Recoding Phase, the FSM stores the feature codes and compare in Monitor Phase. If the feature codes mismatch, our system detects shellcodes. Furthermore, our system is low cost and convenient since it does not need hardware support and source code of programs.

[1] Aleph One, "Smashing The Stack For Fun And Profit," Phrack 7(49). Available:
http://doc.bughunter.net/buffer-overflow/smash-stack.html, 1996.
[2] Jeffrey Richter, "Advanced Windows, 3rd ed," Microsoft Press, February 1997.
[3] C. Cowan, C. Pu, D. Maier, H. Hinton, J. Walpole, P. Bakke, S. Beattie, A. Grier,
P. Wagle, and Q. Zhang, “StackGuard: Automatic Adaptive Detection and
Prevention of Buffer-Overflow Attacks,” Proceedings of the 7th USENIX
Security Symposium, Jan. 1998
[4] G.Hunt and D.Brubacher(1999), "Detours: Binary Interception of Win32
Functions," Published in Proceedings of the 3rd USENIX Windows NT
Symposium. Seattle, WA, July 1999.
[5] Arash Baratloo, Timothy Tsai, and Navjot Singh, Bell Labs, , “Libsafe:
Protecting Critical Elements of Stacks,” Lucent Technologies, available:
http://www.bell-labs.com/org/11356/libsafe.html, December 25, 1999
[6] John Viega, J.T. Bloch, Tadayoshi Kohno, and Gary McGraw, “ITS4: A static
vulnerability scanner for C and C++ code,” In Proceedings of the 16th Annual
Computer Security Applications Conference, December 2000.
[7] “Microsoft Visual C++ Compiler Options: /gs (Control Stack Checking
Calls),”Online documentation, http://msdn.microsoft.com/library/default.asp?
url=/library/en-us/vccore/html/_core_.2f.gs.asp, 2001.
[8] Tzi-cker Chiueh and Fu-Hau Hsu, “RAD: A Compile-Time Solution to Buffer
Overflow Attacks,” 21st IEEE International Conference on Distributed
Computing Systems (ICDCS'01), p. 0409, April 2001
[9] Matt Pietrek, "An In-Depth Look into the Win32 Portable Executable File
Format," MSDN Library. Available:
http://msdn.microsoft.com/msdnmag/issues/02/02/PE/default.aspx, February
2002
[10] Mark E. Donaldson, "INSIDE THE BUFFER OVERFLOW ATTACK:
MECHANISM, METHOD, & PREVENTION," SANS Institute, 2002
[11] D.C. DuVarney, V.N. Venkatakrishnan, and S. Bhatkar, “SELF: A Transparent
Security Extension for ELF Binaries,” Proc. 2003 Workshop New Security
Paradigms, pp. 29-38, 2003.
[12] M. Prasad and T.-c. Chiueh, “A Binary Rewriting Defense against Stack Based
Overflow Attacks,” Proc. USENIX 2003 Ann. TechnicalConf., 2003.
[13] E. Haugh and M. Bishop, "Testing C Programs for Buffer Overflow
Vulnerabilities," Proceedings of the 2003 Symposium on Networked and
Distributed System Security (SNDSS 2003), Feb. 2003.
[14] Skape, "Understanding Windows Shellcode," nologin website.
Available:http://www.nologin.org/Downloads/Papers/win32-shellcode.pdf,
12/06/2003
[15] J. McGregor and D. Karig and Z. Shi and R. Lee, “A Processor Architecture
Defense against Buffer Overflow Attacks,” on Information Technology:
Research and Education (ITRE), pp. 243--250, Aug. 2003.
[16] C.Jesse, R.Rabek, I.Khazan, M.Scott, L.Robert and K.Cunningham, "Detection
of Injected, Dynamically Generated, and Obfuscated Malicious Code," In
Proceedings of the 2003 ACM workshop on Rapid Malcode October 2003.
[17] O. Ruwase and M. S. Lam, “A practical dynamic buffer overflow detector,” In
Network and Distributed System Security Symposium (NDSS), pages 159--169,
February 2004.
[18] Aishwarya Iyer, Liebrock, L.M., “Vulnerability scanning for buffer overflow,”
Information Technology: Coding and Computing, 2004. Proceedings ITCC
2004, April 2004.
[19] Mihai Christodorescu and Somesh Jha, “Testing Malware Detectors,” ISSTA’04,July 11–14, 2004
[20] Phrack Inc, "Bypassing 3rd Party Windows Buffer Overflow Protection," In
Phrack Magazine. Available:
http://www.phrack.org/phrack/62/p62-0x05_Bypassing_Win_BufferOverflow_P
rotection.txt, Jul-2004.
[21] C.Shannon and D.Moore. "The spread of the Witty worm," In Security & Privacy
Magazine, IEEE Vol 2, issue 4, pp. 46 - 50, July-Aug. 2004.
[22] Phrack Inc, "History and Advances in Windows Shellcode," In Phrack Magazine.
Available:http://www.phrack.org/phrack/62/p62-0x07Advances_in_Windows
_Shellcode.txt, Nov 2004.
[23] The MetaSploit Project, "ShellCode Archive," Metasploit Project offical website.
Available:http://metasploit.org/shellcode.html, Nov 2004.
[24] The ShellCode.org, "The ShellCode Writing," ShellCode.org website.
Available:http://shellcode.org/, Nov 2004.
[25] Elena Gabriela Barrantes, David H. Ackley, Stephanie Forrest and Darko
Stefanovic, “Randomized Instruction Set Emulation to Disrupt Binary Code
Injection Attacks,” ACM Transactions on Information and System Security
(TISSEC), Volume 8, Issue 1, February 2005.
[26] Microsoft Corp, "A detailed description of the Data Execution Prevention (DEP)
feature in Windows XP Service Pack 2 and Windows XP Tablet PC Edition 2005,"
Microsoft Corp’s support website.
Available:http://support.microsoft.com/kb/875352/en-us, Feb 2005.
[27] E.Levy. "Worm propagation and generic attacks," In Security & Privacy
Magazine, IEEE Vol 3, issue 2, pp.63 - 65, Mar-Apr 2005.
[28] French Security Incident Response Team, "SlimFTPd Multiple Commands
Remote Buffer Overflow Vulnerability," French Security Incident Response
Team’s website. Available: http://www.frsirt.com/english/advisories/2005/1168,
2005-07-22.
[29] Piotr Bania, “Windows Syscall Shellcode,” securityfocus, website:
http://www.securityfocus.com/infocus/1844, 2005-08-04.
[30] Steve Hanna, "Shellcoding for Linux and Windows Tutorial," vividmachines
website. Available: http://www.vividmachines.com/shellcode/shellcode.html,
Sept 2005.
[31] Manuel Costa, Jon Crowcroft, Miguel Castro, Antony Rowstron, Lidong Zhou,
Lintao Zhang and Paul Barham,”Vigilante: End-to-End Containment of Internet
Worms,” SOSP’05, October 23–26, 2005.
[32] Fanglu Guo, Yang Yu, Tzi-cker Chiueh, “Automated and Safe Vulnerability
Assessment,” Proceedings of the 21st Annual Computer Security Applications
Conference (ACSAC), Dec 2005.
[33] Hung-Min Sun, Yue-Hsun Lin, and Ming-Fung Wu, "API Monitoring System
for Defeating Worms and Exploits in MS-Windows System," In Conference of
the 11th Australasian Conference on Information Security and Privacy (ACISP),
LNCS 4058, pp. 159 – 170, 2006.
[34] The CERT, "CERT/cc statistics 1988-2005," CERT website. Available:
http://www.cert.org/stats/, January 16, 2006.
[35] Danny Nebenzahl, Mooly Sagiv, Avishai Wool, "Install-Time Vaccination of
Windows Executables to Defend against Stack Smashing Attacks," IEEE
Transactions on Dependable and Secure Computing, vol. 3, no. 1, pp. 78-90,
Jan-Mar, 2006.
[36] Microsoft Corp, "Overview of the Windows API," MSDN Library, Available:
http://msdn.microsoft.com/library/en-us/winprog/winprog/overview_of_the_wi
ndows_api.asp.
[37] DataRescue, IDA Pro, website: http://www.datarescue.com/idabase/
[38] WhitSoft Development and Matt Whitlock, Slimftpd, website:
http://www.whitsoftdev.com/slimftpd/.
[39] Trend Micro, PC-Cillin 2006, and website:
http://www.trendmicro.com/tw/products/desktop/pc-cillin/evaluate/features.htm.
[40] Symantec Corporation, Norton antivirus 2006, website:
http://www.symantec.com/home_homeoffice/products/virus_protection/nav200
6/index.html.
[41] Kaspersky Lab, Kaspersky Anti-Virus Personal, website:
http://www.kaspersky.com/personal.
[42] Boost Spirit library, website: http://spirit.sourceforge.net/
[43] Malware, from Wikipedia, website: http://en.wikipedia.org/wiki/Malware
[44] Snort IDS, website: http://www.snort.org
[45] StackShield, website: http://www.angelfire.com/sk/stackshield.
[46] PaX. https://pageexec.virtualave.net.
[47] 7-zip, http://www.7-zip.org/