Building a SAN for SMBs, part 4: Turning shared blocks into shared files

Hewlett-Packard's NAS 9000 provides a NAS-SAN fusion that combines the best of both worlds.

By Jack Fegreus

What do you call a server with dual Xeon processors, 4GB of RAM, dual load-balanced Gigabit Ethernet NICs, hardware-mirrored system drive array, hot-plug PCI-X backplane, remote hardware control over IP, not to mention fully redundant hot-swap components that exhaust any space that might have been available for storage components? Try a "NAS appliance," or "NAS-SAN gateway."

It should come as no surprise that we've gone from NAS-SAN confusion to NAS-SAN fusion. The concept is rather simple: Create a LAN gateway to a SAN. The gateway enjoys all of the scalability and high-performance benefits derived from block-level access on a SAN, as it manages the distribution of file-level data over a LAN.

One of the lead proponents behind the fusion of NAS and SAN is Hewlett-Packard. The company's NAS 9000 is a densely packed version of the ProLiant DL580 Generation 2 server, which is notably dense in everything except internal storage.

Naturally this leads to a perplexing question: How do you turn a robust compute server into a NAS appliance or NAS-SAN gateway?

The question is not as strange as it sounds, however. The ProLiant DL580 G2 server is designed as a four-way SMP server for environments that require maximum computing power. Sites looking for a server that can handle all of the tasks surrounding very compute-intensive applications require a lot more than just raw CPU power. More often than not, these sites are looking for guaranteed maximum uptime even more than a plethora of CPU cycles. As a result, the ProLiant DL580 is packed with high-availability features that make it an ideal choice for an enterprise-class storage server for which downtime can be a showstopper.

Nonetheless, a storage server is not exactly a storage appliance. The leap from compute server to storage appliance requires a very important software dimension: ease of use. To that end, Microsoft has provided a software bride for NAS hardware in an intriguing marriage of convenience.

At the heart of Microsoft's strategy is a very low-cost version of Windows Server 2003 dubbed Windows Storage Server 2003. The broader goals for storage appliances using this software include a quick-less than 15 minutes-setup time, a minimum number of maintenance chores, and access to enterprise-level features via the operating system and third-party software.

Figure 1: One of the ways that HP extends the basic Windows Storage Server 2003 Web interface is through the addition of both Basic and Advanced Lights-Out Management software functions. For example, administrators can power-off or power-on a server using the virtual power button.
Click here to enlarge image

Microsoft may be primarily targeting the fast-growth small to medium-sized business (SMB) market for filer appliances with Windows Storage Server 2003; however, it also foresees this software playing a role in enterprise scenarios. Services inherited from Windows Server 2003-such as the creation of point-in-time copies of data via the Volume Shadow Copy Service (VSS)-buttress this assumption.

Windows Storage Server 2003 does not include all of the event-logging policies of Windows Server 2003 that are designed to support the role of an application server. Policies such as the requirement to log the reason for a server shutdown are contrary to the role of an appliance. In their place is a new and somewhat troubling policy that forbids logging into the Windows Update service. The logic for such a policy makes sense in the context of a headless device. Nonetheless, in the context of an enterprise SAN gateway where both security and performance are paramount, such a policy raises troubling issues.

Windows Storage Server 2003 includes a Web-based user interface for remote system management, which is essential for a headless appliance. This interface is designed to be customized and extended by OEMs to provide value-added services for users. For example, HP extended this interface-which it dubs WebUI-to include hooks into HP's Lights-Out Remote Management software, management of HP SAN storage arrays, and configuration and management of NAS 9000 clusters.

To further enhance the role of Windows Storage Server 2003 as a NAS operating system, there is an advanced storage manager to filter files and provide robust quota management. The storage manager provides a mechanism to control which files can be stored by file type and attribute. Such file filtering restrictions can be applied to either directories or users. For enterprise-level installations, Microsoft Services for NFS has been bundled for Unix connectivity, and an enhanced version of Microsoft's distributed file system (DFS) is included for a widespread Windows domain.

Testing: One, two, three

To test the HP NAS 9000, we inserted a dual-port Emulex LightPulse 10000 host bus adapter (HBA) into its backplane. Not only does this NAS appliance come with no drives, but the system also ships without a SAN HBA. The choice of a Fibre Channel HBA is left to the user.

The Emulex LP10000 is a dual-channel HBA that combines ARM9 microprocessors with an advanced Fibre Channel protocol engine and 133MHz PCI-X support. This enterprise-class HBA features a large data buffer and supports cable runs of more than 50km.

Drivers for the LP10000 are part of the standard Windows Server 2003 distribution. As a result, the HP NAS 9000 immediately recognized the LP10000, and we were immediately connected to the SAN. Included in our SAN were two QLogic SANbox 5200 stackable switches along with an nStor 4520 Storage System (both of which were evaluated in previous Lab Reviews as part of our series on Building a SAN for SMBs).

Figure 2: Thanks to the Windows caching scheme, throughput using a 128MB file appeared to be faster on reads than is possible over a 100Mbps connection. We were able to eliminate the caching effects through the use of a 512MB file to verify that reads were running close to wire speed. Writes were slightly slower at 10.2MBps.
Click here to enlarge image

Equally powerful was our LAN connection. By having an operating system based on Windows Server 2003, HP was able to include its Windows NIC teaming utility, which works with up to four NICs. Using the two Gigabit Ethernet NICs installed on the server, we set up a team employing switch-assisted load balancing for both transmitted and received network packets.

Alternatively, the team can be set up for load balancing of just transmitted packets without switch interaction or in an active-passive configuration for fault tolerance. In our configuration, the utility created an HP Network Teaming Virtual Miniport that appeared to the operating system as an additional NIC, which had an aggregate throughput of 2Gbps.

Once this virtual NIC is created, it can be configured as if it were a real NIC. The actual NICs are never individually configured.

Figure 3: Using Samba to connect an SuSE Linux 9.0 client to the same Windows share provided very disappointing throughput performance. Both reads and writes turned to sludge, averaging just 4MBps.
Click here to enlarge image

We were now ready to configure our physical storage volumes and network shares. Since we were using an nStor 4520 disk array, we were not able to use one of the key extensions that HP added to the Web interface: SAN array management. HP's WebUI modules apply -only to HP storage arrays on the SAN. Nevertheless, we were able to install and host all of the SAN management software for the QLogic and nStor devices.

From the operating system perspective, the Volume Shadow Copy Service (VSS) is a highly touted feature for creating point-in-time snapshots-dubbed shadow copies-of volumes. VSS, which was introduced with Windows Server 2003, supports up to 64 shadow copies per volume and works at the block level.

Figure 4: Using NFS to connect the same SuSE Linux client, which was running NFS v2, provided much better performance on reads as throughput matched wire speed. Write performance was disappointing and required asynchronous writes between the client and HP NAS 9000 just to reach parity with a Samba connection to a CIFS share. Using the default synchronous writes, throughput was a glacial 0.6MBps.
Click here to enlarge image

From the point in time that a shadow copy ends, the next shadow copy begins by monitoring files. As a file's block structure changes, the original blocks are saved in a cache. This means the files in a shadow copy only contain the original (pre-modified) blocks. If a file is unchanged, then it has 0 blocks in the cache. As a result, the scheme saves only enough data to maintain a consistent temporal view of the data.

However, there is a drawback to this implementation, which creates snapshots on the NAS device. The previous versions of files and folders are only available over the LAN to clients running an updated version of Windows Explorer.

From an IT operations perspective, the HP Integrated Lights-Out Management extension to the WebUI is an intriguing and, likely, one of the most useful features for administrators. The HP NAS 9000's integrated Lights-Out (iLO) port is an ASIC-based Web interface that provides remote management for the server, independent of the state of either the host operating system or CPU.

Figure 5: Since Unix and Linux clients will more likely be servers or high-end workstations, we also tested a server client running SuSE 9.1 with NFS v3 over a Gigabit Ethernet connection. Throughput on reads jumped 6x, while writes showed a performance increase of 5x. Our benchmark pegged average reads at 68MBps and writes at 20MBps.
Click here to enlarge image

We had no problem accessing the iLO Web interface features running Mozilla on a Linux system. Central among the base features is a virtual power button that can be used to remotely operate the power button on a host, including turning a server on.

To run the Java-based Virtual Graphical Remote Console, we needed to run the iLO Web interface via IE on Windows. This Java-based applet turns a supported browser into a virtual desktop. It gives the user full control over the display, keyboard, and mouse of the host server and supports graphic modes that display remote host operations, including shutdown and start-up.

More importantly, a USB-based Virtual Media feature allows an IT administrator to use a standard diskette or CD on the client machine to install applications on the remote server or perform a recovery of a failed system. With the rapid growth in the use of hosting sites, the iLO port and Lights-Out Management software make this system, whether in the guise of the HP NAS 9000 or the ProLiant DL580 G2, a winner in terms of TCO.

Performance testing

To test baseline performance, we mapped a 36GB LUN to the NAS 9000 system from the nStor storage array. Since this array was exported from both ports on the nStor server to both ports on the Emulex HBA, Windows reports this single drive as four different disks-one for each logical connection. Following the first reboot of the server, three of the four drives will remain visible but will not be accessible from within the Windows disk management utility. Should the current connection path fail, Windows activates one of the other connections as the primary path.

All of this configuration can be done using the WebUI. There is a static page that displays basic drive information. Clicking on the "Advanced Disk Management" button will bring up the standard Windows Disk Management utility within the WebUI.

Once we configured the volume, we then exported the volume as both a Windows and NFS share under two different aliases. Exporting the drive as a Windows CIFS share was trivial. Exporting the drive as an NFS share was traumatic.

Microsoft acquired the basic code for its Microsoft Services for NFS from Intergraph. The team that originally developed the code at Intergraph then spun out as Shaffer Solutions, with rights to the same code foundation. The differences in the two products are significant.

Sharing a volume from a Windows server via NFS is a bit more difficult than sharing a volume via CIFS because of the fundamental differences in user accounts and security between Unix and Windows. The differences require the administrator to create a mapping between the users and groups in a Windows domain and the users and groups in a Network Information Service (NIS) domain. If an NIS server is not being used, then it is necessary to provide data files with the correct Unix user data.

There is, however, one little hitch: There is no documentation about the structure of these files. In fact, there is little documentation about anything concerning Microsoft Services for NFS.

To simplify matters, we used an NIS server to deliver the required NIS domain information. This shortened the task of configuring a working NFS share down to two days. Compounding the lack of documentation, the system also provides no error checking or feedback. It is therefore very easy to configure a mapping structure that will simply fail to authenticate any and all users. This behavior is further complicated by the behavior of some Linux distributions, including SuSE, to behave as if the service simply does not exist. Only by using the tools supplied by Shaffer Solutions with DiskAccess NFS client software for Windows were we able to confirm that the share did indeed exist.

After considerable experimentation time, we diagnosed the problem as the inability of Microsoft Services for NFS to handle anything other than a 1-to-1 mapping scheme from Windows users to Unix users. A Unix user can be mapped to multiple Windows users, but not vice versa.

Unlike the configuration scheme in DiskAccess (Shaffer's NFS server for Windows), the explicit mapping feature in Microsoft Services for NFS does not allow multiple Unix users to be mapped onto the same Windows user. In addition, Microsoft Services for NFS makes it optional to automatically create an implicit mapping when Unix and Windows usernames match. DiskAccess does this -automatically.

The problem we encountered relates to the apparent failure of Microsoft Services for NFS to check if both implicit and explicit mappings are enabled, which can easily create many-to-one mapping relationships.

Once we eliminated any explicit mappings to any Windows users that had an implicit map from an NIS user, our authorization problems were solved.

Having broken through the NFS mapping morass, we set up Windows XP and SuSE Linux 9.0 clients on HP Compaq Evo laptops to access both the CIFS and NFS shares, run our oblDisk benchmark, and determine baseline performance. Our goal was not to stress the HP NAS 9000, but rather to determine the best client access strategy for using it as a SAN gateway.

Each laptop was powered by a 1GHz P-III CPU and configured with 256MB of RAM. In our tests we ran the obl-Disk benchmark using 128MB and 512MB files. In the former case we expected to see the benchmark report throughput on reads that was greater than 11MBps as a result of caching and throughput on writes on the order of 8MBps to 10MBps, which would indicate that we were likely bottlenecked by the speed of our LAN connection.

We first ran our oblDisk benchmark locally on the HP NAS 9000 to calibrate the server headroom. Under optimal conditions with asynchronous I/O requests on small files, throughput was approximately 180MBps to 190MBps. With back-end access to the SAN at 2Gbps and front-end access to the LAN at 2Gbps via HP NIC teaming software, the only issues remaining were overhead and throughput related to our choice of file-sharing protocols for clients.

We then began our NAS testing using a Windows XP client connected to a CIFS share over a 100Mbps connection. To measure performance we used both the results reported by the benchmark, which include the effects of client-side caching, along with the actual traffic throughput measured on the QLogic switch between the nStor disk array and HP NAS 9000 server. Reads, which were being cached on the client, averaged 25MBps. Writes, which were not affected by any caching, averaged 10.2MBps. Meanwhile, the HP NAS 9000 was bundling reads coming from the laptop client and access data at about 18MBps.

Using Samba to connect a matching laptop running SuSE Linux 9.0 to the same CIFS share both reads and writes turned to sludge, with both averaging about 4MBps.

We then reconnected our SuSE Linux laptop to the HP NAS 9000 using the NFS share created using Microsoft Services for NFS. Read performance leaped up to wire speeds as our benchmark locked onto a very consistent throughput of about 11.2MBps with no indication of any caching on reads.

Write performance was a different matter that bordered on the bizarre. Using synchronous writes between the NAS 9000 and clients (which is the default), write throughput plummeted to a glacial 0.6MBps!

Letting writes rip by enabling asynchronous writes between the NAS 9000 and clients raised throughput back to 3.8MBps. Naturally, at that speed write performance was effectively synchronous as the HP NAS 9000 was hardly being taxed writing data to the nStor 4520 disk array at that rate.

Even given the fact that SuSE 9.0 is built on the Linux 2.4 kernel and implements NFS v2, the communications overhead implications for Microsoft Services for NFS are significant. Nonetheless, Linux and Unix systems are far from being found on the traditional knowledge worker's desktop.

If not on a server, these operating systems are likely to be found on high-end number-crunching or video-editing workstations with Gigabit Ethernet.

To test this scenario, we connected to the NFS share using a dual-Xeon server running SuSE 9.1, which is built on the Linux 2.6 kernel and implements NFS v3.

As expected, performance on reads jumped significantly as it had obviously been bottlenecked by the LAN connection speed. More importantly, along with the 6x improvement in read throughput, we measured a 5x improvement in write throughput. For many, if not most, workstation-class applications, an average I/O throughput of 68MBps on reads and -19MBps on writes is very good -performance.

With the ROI arguments for moving storage onto a SAN continuously getting stronger, the merits of SAN-NAS fusion are equally strengthened.

Jack Fegreus is technology director at Strategic Communications (www.strat comm.com). He can be reached at jfegreus@stratcomm.info.

InfoStor Labs senario

NAS-SAN gateways


HP NAS 9000

  • ProLiant DL580 G2 Server
    • Dual 2.8GHz Xeon CPUs
    • Dual Gigabit Ethernet NICs
    • 4GB RAM
    • Hot-plug PCI backbone
    • Redundant power supplies and fans
    • Integrated Lights-Out Management port
  • WebUI
    • Array Management Extensions
    • Basic and Advanced Lights-Out Management extensions
  • MS Windows Storage Server 2003
    • Windows Server 2003
    • Advanced Storage Manager
    • Microsoft Services for NFS
Emulex LightPulse 10000 HBA
  • Dual full-duplex Fibre Channel ports
  • 133/100MHz PCI compatibility
  • ARM9 processors


Two QLogic SANbox 5200 stackable switches

  • 16 2Gbps Fibre Channel ports (SFP)
  • Four 10Gbps ISL ports (copper)
  • Port-based incremental licensing
  • Non-disruptive code load and activation
QLogic SANsurfer Management Suite
  • Host-based for Linux, Solaris, Windows
  • Modules for switches and HBAs
  • Wizards for configuration, zoning, and security
  • Performance monitoring
  • Fabric health alerts
Two HP Compaq Evo laptop clients
  • 1.067GHz P-III CPU
  • 512MB PC133 SDRAM
  • Compaq Pro 100 NIC
Appro 1224Xi 1U Server
  • Dual 2.4GHz Intel Xeon CPUs
  • 1GB PC2100 DDR memory
  • 133MHz PCI-X expansion slot
  • Dual Gigabit Ethernet NICs
SuSE Linux 9.0 Professional
  • Linux 2.4.21 kernel
  • NFS v2
SuSE Linux 9.1 Professional
  • Linux Kernel 2.6.4
  • NFS v3
  • oblDisk v2.0


  • Complete configuration and management via WebUI
    • Virtual power on/off buttons for restart
    • Client floppy and CD remapped to virtual devices for the server for disaster recovery and software upgrades
  • Performance for Windows clients on 100Mbps Ethernet at wire speed for both reads and writes
  • Services for NFS lack wizards and help for complex configuration
  • Performance for NFS clients on 100Mbps Ethernet at wire speed only for reads
  • Asynchronous writes necessary or adequate throughput on NFS

This article was originally published on October 01, 2004