InfiniBand makes a comeback at SuperComputing show

By Ann Silverthorn

—SuperComputing 2005, an international conference on high-performance computing (HPC), networking, and storage, convened this week in Seattle under the theme "Gateway to Discovery." Attendees in research, education, government, and commerce who have asked for native InfiniBand in their storage products will discover that some vendors have been listening.

Between 1999 and 2001, InfiniBand was hailed as the answer to all bandwidth problems. A number of analysts predicted that InfiniBand would be the default way of connecting servers to other servers, storage devices, and networks. Some observers even predicted that InfiniBand might replace Fibre Channel SANs.

Clearly, Fibre Channel has prevailed. However, InfiniBand didn't die. It flourished in the HPC market, often connecting clusters of Linux servers. Now it's back as an interconnect between servers and storage, and it might be ready to challenge Fibre Channel again.

For example, LSI Logic's Engenio division introduced storage solutions with native InfiniBand this week at the SuperComputing conference The Engenio 6498 controller and 6498 storage system are sold through OEMs and can be used with either Fibre Channel or Serial ATA (SATA) disk drives. Engenio partner SGI today announced the InfiniteStorage TP9700 with native InfiniBand technology, which is based on Engenio's 6498.

"InfiniBand seems to be the interconnect of choice for inter-processor communications in Linux clusters," says Steve Gardner, director of product marketing at Engenio. "Those customers started asking why they needed a separate Fibre Channel network when they already have InfiniBand connectivity to each of their servers."

Gardner explains that building a completely separate network for storage means that users have to buy Fibre Channel infrastructure, complete with switches and host bus adapters (HBAs), or buy a converter that performs Fibre Channel to InfiniBand conversion. Both choices cause complication and impact network performance, he says.

"The basic premise of InfiniBand is that it's less expensive and fast and has low latency. A single InfiniBand pipe that we offer runs at 10Gbps and is an attractive interface for users that have large amounts of data to move around," says Gardner. "That's why we're starting with supercomputing sites. These companies have already invested in InfiniBand for their inter-processor communications. Now they want to put their storage on the same network to reduce costs and use higher-speed, single-pipe technology."

Brian Garrett, technical director for ESG Lab, was enthusiastic in his presentation, "InfiniBand Storage Technology" at the Storage Networking World conference last month. According to Garrett, InfiniBand costs nearly one-quarter the price of a traditional SAN and cuts backup time in half. "The compelling price/performance of InfiniBand should not be ignored," Garrett said in his presentation.

According to Craig Schultz, SGI's marketing manager for InfiniteStorage solutions, SGI has found that Fibre Channel components are 2x to 4x more expensive than InfiniBand. "InfiniBand pricing is closer to what we find in the networking arena for Ethernet," he says. "The price per port and actual costs are more affordable than Fibre Channel."

With today's announcement, SGI claims to be the only vendor to offer a complete, high-performance, native InfiniBand solution that includes both servers and storage integrated in an InfiniBand fabric. The TP9700 with native InfiniBand host connections can plug in directly to the InfiniBand fabric of server clusters and communicate directly with the storage without having to go through protocol conversion. The TP9700 has four InfiniBand ports, a 10Gbps interface, and up to 1.6GBps bandwidth. Partners Cisco, Mellanox, SilverStorm Technologies, and Voltaire provide the software, host channel adapters (HCAs), and other SAN components.

SGI TP9700 configurations with InfiniBand are priced from $103,550 for the minimum capacity, which includes four 73GB Fibre Channel drives.

As proof that storage is taking on a larger role at SuperComputing, a consortium called StorCloud (SGI is a member) was created in 2004 and continues this year. At the show, StorCloud will build HPC storage capability showcasing HPC storage technologies and applications. Among others, two of its goals are to provide 1PB or greater of randomly accessible storage and approach 1TB per second infrastructure bandwidth for SC2005 participants.

This article was originally published on November 15, 2005