簡易檢索 / 詳目顯示

研究生: 孫瑞廷
Juei-Ting Sun
論文名稱: 基於Windows平行檔案系統下的平行I/O
Parallel I/O for Windows Based Parallel File System
指導教授: 許雅三
Yarsun Hsu
口試委員:
學位類別: 碩士
Master
系所名稱: 電機資訊學院 - 電機工程學系
Department of Electrical Engineering
論文出版年: 2006
畢業學年度: 94
語文別: 英文
論文頁數: 70
中文關鍵詞: 平行檔案系統平行輸入輸出高效能輸入輸出視窗作業系統隨機視訊系統
外文關鍵詞: parallel file system, MPI-IO, VOD, .NET framework, C#, high performance I/O, Windows, DirectShow
相關次數: 點閱:3下載:0
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報
  • 近年來,因為處理器的迅速發展所以使其速度變得越來越快,然而,資料的存取系統卻無法跟上處理器的速度,這使得處理器會因為存取系統過於緩慢而常常處於閒置狀態,因此,平行檔案系統被廣泛的用來解決這個問題,除此之外,由於叢集電腦被使用於平行處理平台的普及率逐步上升,這造成使用於叢集電腦上的平行處理軟體需求量持續增加,但是,卻有可能缺乏平行I/O的系統,這讓平行檔案系統被要求為叢集電腦來提供高效能的平行I/O系統,雖然它非常適合用來解決因為貧乏的效能所導致的瓶頸,可是大多數現存的平行檔案系統都是基於像UNIX的作業系統,因為這個理由,我們利用微軟的.NET架構實做了一個基於視窗作業系統的平行檔案系統,這篇論文除了有描述這個系統的設計構思與實現方法還有做了些效能的測試,讀取跟寫入的效能都比只用單一磁碟還要好,結果顯示,寫入的效能最高可以達到110 MB/s,讀取的效能最好也有75 MB/s。

    在我們的平行檔案系統上,我們發展了一個簡單的隨選視訊系統,這個系統還有使用到微軟的DirectShow,另外,我們還實現了幾個主要的MPI-IO函數,這些在這篇論文中都有被討論到,利用平行檔案系統的優點,隨選視訊系統有足夠的能力可以供給客戶更好的服務品質,而且MPI-IO的效能也因受惠於此而有更傑出的性能。


    In recent year, the speed of CPU becomes more and more progressive. However, the development of the storage system is incompetent to catch up with that of CPU. For this reason, the parallel file system is widely used to overcome this problem. In addition, as the PC cluster has increased in popularity as a parallel computing platform, the demand for system software support is necessary for parallel computing to continue to grow on this famous platform. But it may be devoid of parallel I/O systems. The parallel file system is required for providing a high performance parallel I/O system for PC clusters. Although it is so applicable to solve the bottleneck due to poor I/O performance, most of the existing parallel file systems are based on UNIX-like operating systems. Therefore, we implement a Windows based parallel file system by using Microsoft .NET framework. Design and implementation of our system are described in this thesis and the performance evaluation is performed as well. Both write and read performance are better than those of only using a single disk. The results indicate that the write performance can achieve 110 MB/s at the utmost and the read performance can also reach a peak of 75 MB/s.

    We develop a simple VOD system by using Microsoft DirectShow and some primary methods of MPI-IO on top of our parallel file system. These two implementations are discussed in the thesis, too. Taking advantage of the parallel file system, the VOD system could have sufficient ability to supply higher QoS for clients and the performance of MPI-IO also benefits from it and attains outstanding results without suffering serious overhead.

    1 Introduction 1 1.1 Motivation and Objective . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Research Motivation . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.2 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Bene.ts of the Parallel File System . . . . . . . . . . . . . . . . . . . 3 1.3 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Background and Related Works 6 2.1 File and File System . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.1 File Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.2 Aspects of File System . . . . . . . . . . . . . . . . . . . . . . 7 2.1.3 Virtual File System . . . . . . . . . . . . . . . . . . . . . . . . 9 2.1.4 Parallel File System . . . . . . . . . . . . . . . . . . . . . . . 10 2.2 Related Researches . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2.1 Parallel File System . . . . . . . . . . . . . . . . . . . . . . . 11 2.2.2 MPI-IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3 Design and Implementation 16 3.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 Metadata Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.3 I/O Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4 Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4 VOD Prototype System 31 4.1 DirectShow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.2 Achievement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3 Functions of the Web Server . . . . . . . . . . . . . . . . . . . . . . . 35 4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5 Implementation of MPI-IO 41 5.1 MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.2 Accounts of Parallel I/O . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.3 MPI-IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 5.4 Noncontiguous Accesses and Collective I/O . . . . . . . . . . . . . . . 44 5.5 Processes of Implementation . . . . . . . . . . . . . . . . . . . . . . . 48 5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6 Performance Evaluation 53 6.1 Evaluation Environment . . . . . . . . . . . . . . . . . . . . . . . . . 53 6.2 Measurement of the Local Disk . . . . . . . . . . . . . . . . . . . . . 54 6.3 Measurement of the Parallel File System . . . . . . . . . . . . . . . . 57 6.3.1 Di¤erent I/O Nodes . . . . . . . . . . . . . . . . . . . . . . . 57 6.3.2 Di¤erent Striping Size . . . . . . . . . . . . . . . . . . . . . . 60 6.4 Measurement of MPI-IO . . . . . . . . . . . . . . . . . . . . . . . . . 63 7 Conclusion and Future Work 66 7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 References 68

    [1] S. Kleiman D., Walsh R. Sandberg, D. Goldberg, and B. Lyon, .Design and Im-
    plementation of the Sun Network Filesystem., in Summer USENIX Conference
    Proceedings, pages 119-130, June 1985.
    [2] Gene M. Amdahl, Validity of the Single Processor Approach to Achieve Large
    Scale Computing Capabilities, pages 79-81, Morgan Kaufmann Publishers Inc.,
    San Francisco, CA, USA, 2000.
    [3] N.R. Adiga, M. Blumrich, and T. Liebsch, .An Overview of the BlueGene/L
    Supercomputer., in Proceedings of the 2002 ACM/IEEE Conference on Super-
    computing, Baltimore, Maryland, 2002, pages 1-22.
    [4] Philip H. Carns, Walter B. Ligon III, Robert B. Ross, and Rajeev Thakur,
    .PVFS: A Parallel File System for Linux Clusters., in Fourth Annual Linux
    Showcase and Conference, Atlanta, GA, October 2000, pages 317-327.
    [5] The Parallel Virtual File System, version 2. http://www.pvfs.org/pvfs2.
    [6] Walter B. Ligon III and Robert B. Ross, .An Overview of the Parallel Virtual
    File System., in 1999 Extreme Linux Workgroup, June 1999.
    [7] José María Pérez, Jesús Carretero, and José Daniel García, .A Parallel File Sys-
    tem for Networks of Windows Workstations., in ACM International Conference
    on Supercomputing, 2004.
    [8] Christopher R. Hertel, Implementing CIFS: The Common Internet File System,
    Prentice-Hall, 2003.
    [9] Mark E. Russinovich and David A. Solomon, Microsoft Windows Internals,
    Fourth Edition: Microsoft Windows Server 2003, Windows XP, and Windows
    2000, Microsoft Press, 2004.
    [10] Rajeev Nagar, Windows NT File System Internals: A Developer.s Guide,
    O.Relly & Associates, September 1997.
    [11] Message Passing Interface (MPI) Forum. MPI-2: Extensions to Message Passing
    Interface, July 1997. http://www.mpi-forum.org/docs/docs.html.
    [12] R. D. Russel, .The Architecture of BPFS: A Basic Parallel File System., in Pro-
    ceedings of Second IASTED Int. Conference on Parallel and Distributed Com-
    puting and Networks, pages 214-220, Brisbanen, Australia, December 1998.
    [13] R. Thakur, W. Gropp, and E. Lusk, .On Implementing MPI-IO Portably and
    with High Performance., in Proceedings of the Sixth Workshop on Input/Output
    in Parallel and Distributed Systems, May 1999.
    [14] R. Thakur, W. Gropp, and E. Lusk, Users Guide for ROMIO: A High-
    Performance, Portable MPI-IO Implementation, Tech. Report ANL/MCS-TM-
    234, Mathematics and Computer Science Division, Argonne National Labora-
    tory, October 1997.
    [15] Samuel A. Fineberg, Parkson Wong, Bill Nitzberg, and Chris Kuszmaul, .PM-
    PIO - A Portable Implementation of MPI-IO., in Proceedings of the Sixth Sym-
    posium on the Frontiers of Massively Parallel Computation, pages 188-195, IEEE
    Computer Society Press, October 1996.
    [16] R. Thakur, W. Gropp, and E. Lusk, .An Abstract-Device Interface for Imple-
    menting Portable Parallel-I/O Interfaces., in Proceedings of the Sixth Sympo-
    sium on the Frontiers of Massively Parallel Computation, pages180-187, IEEE
    Computer Society Press, October 1996.
    [17] James V. Huber, Christopher L. Elford, Daniel A. Reed, Andrew A. Chien,
    and David S. Blumenthal, .PPFS: A High Performance Portable Parallel File
    System., in Proceedings of the Ninth ACM International Conference on Super-
    computing, pages 385-394, ACM Press, July 1995.
    [18] E. Schikuta, T. Fuerle, and H. Wanek, .ViPIOS: The Vienna Parallel In-
    put/Output System., in Proceedings of the Fourth International Euro-Par Con-
    ference on Parallel Processing, pages 953-958, September 1998.
    [19] David A. Patterson, Garth Gibson, and Randy H. Katz, .A Case for Redundant
    Arrays of Inexpensive Disks (RAID)., in Proceedings of the 1998 ACM SIGMOD
    International Conference on Mangement of Data, pages 109-116, June 1998.
    [20] S. Moyer and V. Sunderam, .PIOUS: A Scalable Parallel I/O System for
    Distributed Computing Environments., in Proceedings of the Scalable High-
    Performance Computing Conference, pages 71-78, 1994.
    [21] Intel Scalable System Division. Paragon System User.s Guide. Order Number
    312489-004, May 1995.
    [22] IBM Corp. IBM AIX Parallel I/O File System: Installation, Administration,
    and Use. Document Number SH34-6065-01, August 1995.
    [23] O. Krieger and M. Stumm, .HFS: A Performance-Oriented Flexible File System
    Based on Building-Block Compositions., in Proceedings of Fourth Workshop on
    Input/Output in Parallel and Distributed Systems, pages 95-108, ACM Press,
    May 1996.
    [24] Adam Sweeney, Doug Doucette, Wei Hu, Curtis Anderson, Mike Nishimoto, and
    Geo¤Peck, .Scalability in the XFS File System., in Proceedings of the USENIX
    1996 Annual Technical Conference, January 1996.

    無法下載圖示 全文公開日期 本全文未授權公開 (校內網路)
    全文公開日期 本全文未授權公開 (校外網路)

    QR CODE