Performance Comparison of iSCSI and NFS IP Storage protocols.
 TechnoMages home |
 PDF version. |
Introduction.
Currently two major approaches to file transfer in a storage network exist. Those approaches are so-called "block I/O" and "file I/O".
For the "block IO" approach, the SCSI protocol is the protocol of choice in the storage management arena. Different transport mechanisms, such as parallel SCSI, Fibre Channel (FC), or Internet SCSI (iSCSI) can be used to transfer SCSI protocol data.
SCSI is a mature and well established protocol used for fast access to storage devices. However, it has serious drawbacks, such as distance (several meters) and scalability (15 devices on a bus) limitations. While Fibre Channel (FC) has overcome some of SCSI drawbacks and became currently dominant protocol for SAN, it also has certain capability limitations. A high cost for FC SAN installation and maintenance, lack of security and access control mechanisms, lacking long distance capabilities (<500m regular, <10km with special and costly equipment), makes emerging protocols, such iSCSI and Infiniband (IBA), serious competitors to FC for a certain set of applications.
Among emerging storage protocols, iSCSI is a very promising mechanism for SCSI command transfer using IP network. Based on TCP/IP protocol, iSCSI uses TCP flow control, congestion control, segmentation mechanisms, and it is built upon the IP addressing and discovery mechanisms. Using existing gigabit Ethernet infrastructure, iSCSI can provide gigabit speeds comparable to FC and will have 10Gbit speeds available in the near future. iSCSI is more scalable with respect to the number of storage devices in the SAN then FC or IBA due to its addressing and discovery mechanisms. iSCSI is better suited then FC or IBA for deployment over WAN, since it leverages existing IP infra-structure, which has practically no distance limitations and has well-understood network security mechanisms.
For the "file IO" approach, NFS and CIFS protocols are practically the only protocols used today for network file system access. Like iSCSI, these protocols are based on TCP/IP protocol. The ability of NFS/CIFS to represent storage appliance as a local file system accessible by clients, allows file shared storage access. However, there is number of applications that do not require file sharing. Moreover, there are applications, whose performance is significantly degraded when accessing data using "file IO" approach, like databases and similar transaction-oriented applications. However, those applications do require IP connectivity between servers and clients, particularly in certain redundant failover configurations.
The purpose of this work is compare the performance of iSCSI and NFS using benchmarks that emulate different storage access patterns, such as streaming writes and reads (Bonnie), database access (IOgen) and small file access (PostMark).
Experimental Setup.
The experimental setup used for performance evaluation of iSCSI and NFS protocols is presented on Figures 1a and 1b.
Figure 1a. iSCSI setup.
Figure 1b. NFS setup.
As shown on Figures 1a and 1b, iSCSI and NFS experimental setups consist of the storage server and a client connected to each other by Gigabit Ethernet through Gigabit Ethernet NICs. For both setups the hardware of the client and the server machines is kept identical as described below.
The client machine is a PC that consists of SuperMicro 370DE6 motherboard, one 650MHz Pentium III processor, 1GB RAM, 64bit/33MHz PCI bus, copper 1000BaseT Ethernet NIC. For NFS experiments, the Linux 7.1 RedHat distribution with NFS 2 was set on the client machine. For CIFS experiments, the client was running Window 2000 operating system. For iSCSI setup the IBM iSCSI initiator, version 1.2.1, was used.
The storage server machine contains SuperMicro 370DLR motherboard, one 1GHz Pentium III processor, 512MB RAM, 64bit/33MHz PCI bus, copper 1000BaseT Ethernet NIC, and 8 ATA Maxtor 5400rpm 80GB hard disks. These hard disks are combined into RAID by 3Ware 7850 Escalade RAID controller, which presents them to the system as a single SCSI disk. The server machine runs the TechnoMages proprietary embedded Linux distribution with 2.4.17 Linux kernel.
As shown on Figure 1a, messages coming to the server through Gigabit Ethernet NIC are received by the TCP/IP network layer, which forwards iSCSI messages to iSCSI target. The latter passes iSCSI requests to the Storage Router, called Data Transport Processor, whose functionality is to transform iSCSI messages to SCSI commands and send them to SCSI initiator, which in turn sends SCSI commands to the RAID controller.
At a client side, as seen at Figure 1a, server´s RAID disk array is presented as locally attached SCSI disk by iSCSI initiator running on a client machine. The requests for File System received from user level are passed to iSCSI initiator, which sends them down to TCP/IP layer, and the latter transfer them to the network through Gigabit Ethernet NIC.
As shown on Figure 1b, messages that come to the server through Gigabit Ethernet NIC are received by the TCP/IP network layer, which forwards NFS related requests to NFS server. The latter passes file system requests to the file system layer. The file system communicates with SCSI initiator, which in turn sends SCSI commands to the RAID controller.
At a client side, as seen from Figure 1b, the user level file system requests are passed to NFS client, who sends those requests down to TCP/IP layer, from where they are transferred to the network through the Gigabit Ethernet NIC.
On Figures 2a and 2b we present the components of TCP/IP messages for both cases of iSCSI and NFS protocol messages encapsulated into TCP/IP.
Figure 2a. TCP/IP packet components for iSCSI protocol.
Figure 2b. TCP/IP packet components for NFS setup.
Figure 2a shows components of TCP/IP packet when it carries iSCSI protocol. The data are encapsulated into SCSI commands, to which iSCSI header is added, and iSCSI message is the part of TCP/IP message.
Figure 2b shows components of TCP/IP packet carrying NFS packets. The data are enclosed into NFS messages, which are further encapsulated into RPC (Remote Procedure Call) messages. The latter are the part of TCP/IP packets.
Benchmark Results
Bonnie and Bonnie++
Bonnie benchmark measures sequential I/O access to a file system as well as random seek rate. The sequential I/O is measured per character and per block. Bonnie also reports CPU utilization for all its measurements.
Bonnie initially creates a file of predefined size and then performs sequential write first per character and then per block. After that, Bonnie does sequential read also per character and per block. At last, the benchmark performs random seek operations.
Table 1. Bonnie results.
Table 2. Bonnie++ results.
As it is seen from Tables 1 and 2, while rate for sequential reads and writes performed by character are practically the same for both protocols, in case of sequential write per block iSCSI outperforms NFS by at least 10% and in case of sequential read by at least 20%. The intelligent rewrite rate of iSCSI is 50% higher then NFS. In addition, iSCSI random seek rate is 3 time higher then the rate of NFS.
PostMark
PostMark benchmark measures the performance of applications that extensively work with small files. Those applications are Internet related ones such as electronic mail, net news, and web-based commerce.
PostMark initially generates a large pool of files ranging in size and then performs a specified number of transactions, where each transaction consists of a pair of small transactions: file creation or deletion and file read or append.
PostMark provides a set of results, from which we report the transaction rate (in transactions per second), and read and write rates (in kilobytes per second). Our PostMark results were generated using ReiserFS file system.
Table 3. Postmark results.
As it is seen from Table 3, iSCSI rates for transaction, read, and write are from 200% to 300% higher then ones for NFS for the case of low to middle number of files and transactions. For very large number of files, the iSCSI outperforms NFS by at least 100%.
IOgen
IOgen benchmarks measures I/O rates for random reads and writes performed by multiple processes, thus emulating database type access.
When IOgen is configured, it is given a file for performing I/O. The ratio of read and write operations can be also specified, as well as a number of processes to perform I/O and a block size for I/O transactions.
At the beginning of each run, IOgen creates a specified number of child processes, each of which compute random offset on a given file and then do seek and I/O operation at that offset. At the end of a run, IOgen reports average I/O service and response time, number of I/O per second, and I/O rate.
For our experiment, we used a file of a size equal to 2GB.
Table 4. IOgen results.
As it is seen from Table 4, iSCSI throughput rates are higher then NFS rates by ~20% for pure writes and by ~400% for pure reads. As for service and response times, these iSCSI parameters are ~20% lower for pure writes and almost 400% lower for pure reads then ones for NFS.
Conclusions
Our results, generated by running some of industry standard benchmarks, show that iSCSI significantly outperforms NFS for situations when performing streaming, database like accesses and small file transactions.
So far, NFS and CIFS were the only protocols, which allowed IP access to File Systems, and were used for the most applications that require IP connectivity. However, not all of those applications need and some even degrade when using "file IO" access means. While iSCSI cannot provide shared file access, it would be quite suitable for applications that do not need sharing [TMI1], [TMI2].
While performance of both iSCSI and NFS is significantly affected by TCP/IP protocol, which uses quite a bit of CPU time on both server and client, thus lowering the throughput, there currently exist a number of efforts in the industry aiming to increase performance of TCP/IP using hardware solutions.
References
[IETF] Internet Engineering Task Force, http://www.ietf.org/html.charters/ips-charter.html
[SNIA] Storage Networking Industry Association, http://www.snia.org/
[TMI1] "Deploying iSCSI Storage Protocol for File Server Applications", TechnoMages, Inc., white paper
[TMI2] "Deploying iSCSI Storage Protocol for Failover Configurations", TechnoMages, Inc., white paper
 |
|
| Technomages Inc., 2003 | Home | Products | Support | Contacts |