In-depth packet inspection engines, which search the whole packet payload to identify packets of interest that contain certain patterns, are urgently required. The searching results from the inspection engines can be utilized in the network equipment for varied application-oriented management. The most important technology for fast packet inspection is an efficient multi-pattern matching algorithm, which performs exact string matching between packets and a large set of patterns. This study discusses state-of-the-art pattern matching algorithms and proposes three efficient multi-pattern matching algorithms for networks: a hierarchical multi-pattern matching algorithm (HMA), an enhanced hierarchical multi-pattern matching algorithm (EHMA), and an Aho-Corasick with Magic Structures (ACM) algorithm.
HMA and EHMA are built based on hierarchical and cluster-wise matching strategies. The hierarchical matching strategy of HMA and EHMA can efficiently reduce the number of external memory (L2) accesses and the amount of memory space. EHMA contributes modifications to HMA and includes the ideas of Sampling Windows and a Safety Shift Strategy. The Safety Shift Strategy can significantly speed up the scanning process of packet inspection. HMA and EHMA improve the average-case performance of multi-pattern matching, and are useful for the network equipment that locates at the general network environment.
Moreover, the proposed ACM presents a novel Magic Structure based on the Chinese Remainder Theorem. ACM needs only a small amount of memory space and does not increase computational time complexity. ACM has better worst-case performance than state-of-the-art algorithms, and is suitable for the network equipment that usually suffers heavy attacks or requires guaranteed performance.
In this study, the analyses and simulation results show that the proposed algorithms in this study outperform others. HMA and EHMA successfully reduce the average number of L2 memory accesses to about only 0.06–0.37 per code, and improve the performance to about 0.89–1161 times better than the state-of-the-art algorithms. The overall cost of ACM is about 1.1–459 times better than the existing implementations. In particular, HMA, EHMA, and ACM use only simple and easy instructions, and no special hardware is required. Therefore, the proposed multi-pattern matching algorithms are easy to be implemented in both hardware and software. Consequently, the proposed multi-pattern matching algorithms can be efficiently applied to packet inspection engines for network equipment.

[1] Snort, http://www.snort.org.
[2] Brian Caswell, Jay Beale, James C. Foster, and Jeremy Faircloth, "Snort 2.0 Intrusion Detection," Syngress, Feb, 2003.
[3] CERT/CC. "The Nimda worm has the potential to affect both user workstations (clients) running Windows 95, 98, ME, NT, or 2000 and servers running Windows NT and 2000." CERT Advisory CA-2001-26, Sep. 2001.
[4] Spyros Antonatos, Kostas G. Anagnostakis, and Evangelos P. Markatos, "Generating realistic workloads for network intrusion detection systems," ACM Workshop on Software and Performance, pp. 207–215, 2004.
[5] Martin Roesch, "Snort – Lightweight Intrusion Detection for Networks," Proceedings of the 13th Systems Administration Conference, pp. 229–238, 1999.
[6] Tomoaki Sato and Masa-aki Fukase, "Reconfigurable Hardware Implementation of Host-based IDS," the 9th Asia-Pacific Conference on Communication, Vol. 2, pp. 849–853, Penang, Malaysia, Sept. 2003.
[7] Mike Fisk and George Varghese, “Fast Content-Based Packet Handling for Intrusion Detection,” UCSD Technical Report CS2001-0670, May 2001.
[8] Alfred V. Aho and Margaret J. Corasick, "Efficient string matching: an aid to bibliographic search," Communications of the ACM, Vol. 18, Np. 6, pp. 330–340, June 1975.
[9] Sridhar Lakshmanamurthy, Kin-Yip Liu, Yim Pun, Larry Huston, and Uday Naik, "Network Processor Perfromance Analysis Methodology," Intel Technology Journal, Vol. 6, Aug. 2002.
[10] Ricardo A. Baeza-Yates, "Improved String Search," Software – Proctice and Experience, Vol. 19, No. 3, pp. 257–271, March 1989.
[11] Spyros Antonatos, Michalis Polychronakis, P. Akritidis, Kostas G. Anagnostakis, Evangelos P. Markatos, "Piranha: Fast and memory-efficient Pattern Matching for Intrusion Detection," Proceedings of the 20th IFIP International Information Security Conference (SEC2005), pp. 393–408, May 2005.
[12] Gordon Brebner and Delon Levi, "Networking on Chip with Platform FPGAs," Proceedings of 2003 IEEE International Conference on Field-Programmable Technology, pp. 13–20, Dec. 2003.
[13] Robert S. Boyer and Strother J. Moor, "A Fast String Searching Algorithm," Communications of the ACM, Vol. 20, No. 10, pp. 762–772, October 1977.
[14] Crispin Cowan, “Defcon Capture the Flag: Defending Vulnerable Code from Intense Attack,” Proceedings of DARPA Information Survivability Conference and Exposition, Washington DC, vol.2, pp. 71–72, April 2003.
[15] C. Jason Coit, Stuart Staniford, and Joseph McAlerney, "Towards Faster String Matching for Intrusion Detection or Exceeding the Speed of Snort," Proceedings of the 2nd DARPA Information Survivability Conference and Exposition, vol.1, pp. 367–371, 2001.
[16] Thomas H. Cormen, Dartmouth College, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein, "Introduction to Algorithms - 2nd Edition," MIT Press and McGraw-Hill, Sep. 2001.
[17] Sarang Dharmapurikar, Praveen Krishnamurthy, Todd Sproull, John lockwood, "Deep Packet Inspection using Parallel Bloom Filters," 11th Symposium on High Performance Interconnects, pp. 44–51, August 2003.
[18] Sarang Dharmapurikar and John Lockwood, "Fast and Scalable Pattern Matching for Network Intrusion Detection Systems," IEEE Journal on Selected Area in Communications, Vol. 24, No. 10, pp. 1781–1792, Oct. 2006.
[19] Ozgun Erdogan and Pei Cao, "Hash-AV: Fast Virus Signature Scanning by Cache-Resident Filters," Proceedings of IEEE Global Telecommunications Conference, Vol. 3, St. Louis, MO, Nov. 28, 2005.
[20] Mark Handley, Vern Paxson and Christian Kreibich, "Network Intrusion Detection: Evasion, Traffic Normalization, and End-to-End Protocol Semantics," Proceedings of the 9th USENIX Security Symposium, 2000.
[21] R. Nigel Horspool, "Practical Fast Searching in Strings," Sofetware Practice and Experience, Col. 10, No. 6, pp. 501–506, 1980.
[22] Intel Network Processors, http://www.intel.com/design/network/products/npfamily/index.htm
[23] Christopher Kruegel, Fredrik Valeur, Giovanni Vigna, and Richard Kemmerer, "Stateful Intrusion Detection for High-Speed Networks," Proceedings of IEEE Symposium on Security and Privacy, pp. 285, May 2002.
[24] Sun Kim and Yanggon Kim, "A Fast Multiple String-Pattern Matching Algorithm," 17th AoM/IAoM Interantional Conference on Computer Science, San Diego, CA, August, 1999.
[25] Vasilios Katos, "Network Intrusion Detection: Evaluating Clusterm Discriminant, and Logit Analysis," Information Sciences 177, pp.3060-3073, 2007.
[26] Hongbin Lu, Kai Zheng, Bin Liu, Xin Zhang, and Yunhao Liu, "A Memory-Efficient Parallel String Matching Architceture for High-Speed Intrusion Detection," IEEE Journal on Selected Area in Communications, Vol. 24, No. 10, pp. 1793–1804, Oct. 2006.
[27] Rong-Tai Liu, Nen-Fu Huang, Chih-Hao Chen and Chia-Nan Kao, "A Fast String Matching Algorithm for Network Processor-Based Intrusion Detection System," ACM Transactions in Embedded Computing Systems,Vol.3, Issue 3., pp. 614–633, Aug. 2004.
[28] Shaomeng Li, Jim Torresen, and Oddvar Soraasen, "Exploiting Reconfigurable Hardware for Network Security," Proceedings of the 11th Annual IEEE Symposium on Field-Programmable Custom Computing Machines, pp. 292, 2003.
[29] Evangelos P. Markatos, Spyros Antonatos, Michalis Polychronakis and Kostas Anagnostakis, "Exclusion-based Signature Matching for Intrusion Detection," Proceedings of IASTED International Conference on Communications and Computer Networks (CCN 2002), pp. 146–152, October 2002.
[30] Vern Paxson, "Bro: A System for Detecting Network Intruders in Real-Time," Computer Networks, Vol. 31, No. 23-24, pp. 2435–2463, 1999.
[31] Graham A. Stephen, "String Matching Algorithms," World Scientific (ISBN 981-02-1829-X), 1994.
[32] Taeshik Shon and Jongsub Moon, "A Hybrid Machine Learning Approach to Network Anomaly Detection," Information Sciences 177, pp. 3799–3821, 2007.
[33] Tzu-Fang Sheu, Nen-Fu Huang, Hung-Shen Wu, Ming-Chang Shih, and Yuang-Fang Huang, "On the Design of Network-Processor-Based Gigabit Multiple-Service Switch," Proceedings of IEEE ITRE 2005, Hsinchu, Taiwan, 2005.
[34] Nathan Tuck, Timothy Sherwood, Brad Calder, George Varghese, "Deterministic Memory –Efficient String Matching Algorithms for Intrusion Detection," Proceedings of the IEEE Infocom Conference, Vol. 4, pp. 2628–2639, Hong Kong, March 2004.
[35] Vitesse Network Processors, http://www.vitesse.com
[36] Sun Wu and Udi Manber, "A Fast Algorithm for Multi-Pattern Searching," Tech. Rep. TR94-17, Department of Computer Science, University of Arizona, May 1994.
[37] Zhenwei Yu, Jeffrey J. P. Tsai and Thomass Weigert, "An Automatically Tuning Intrusion Detection System," IEEE Transactions on Systems, Man and Cybernetics – Part B: Cybernetics, Vol. 37, No. 2, pp. 373–384, April 2007.
[38] Tzu-Fang Sheu, Nen-Fu Huang and Hsiao-Ping Lee, “ A Novel Hierarchical Matching Algorithm for Intrusion Detection Systems,” Proceedings of IEEE Global Telecommunications Conference (Globecom), St. Louis, Nov. 2005.
[39] Yuke Wang, “New Chinese Remainder Theorems,” Conference Record of the Thirty-Second Asilomar Conference on Signals, Systems & Computers. Vol. 1, pp. 165-171, Nov. 1998.
[40] Mikael Degermark, Andrej Brodnik, Svante Carlsson, and Stephen Pink, “Small forwarding tables for fast routing lookups,” In Proceedings of SIGCOMM, pages 3–14, 1997.
[41] W. Eatherton, Z. Dittia, and G. Varghese, “Tree bitmap: Hardware/software IP lookups with incremental updates,” ACM SIGCOMM Computer Communications Review, 34(2), 2004.
[42] Reetinder Sidhu and Viktor K. Prasanna, "Fast Regular Expression Matching using FPGAs,” IEEE Symposium on Field-Programmable Custom Computing Machines (FCCM01), April 2001.
[43] Yuke Wang, “New Chinese Remainder Theorems,” Thirty-Second Asilomar Conference on Signals, Systems & Computers. Vol. 1, pp. 165-171, Nov. 1998.
[44] Saman Amarasinghe, Walter Lee, Ben Greenwald, ”Strength Reduction of Integer Division and Modulo Operations,” M.I.T., 1999. http://www.cag.lcs.mit.edu/raw
[45] Torbjorn Granlund. Instruction Latencies and Throughput for AMD and Intel x86 processors. http://swox.com/doc/x86-timing.pdf. Sep. 2005.