More and more applications that dominate our storage world require streaming I/O. The main driver for streaming I/O is streaming video capture. From home security to police to (soon) drones, this has been the fastest growth area of the storage market for a number of years. And it’s expected to grow even faster each year for the foreseeable future.
Most file systems with small allocations are designed for multi-user IOPS, which is a totally different set of design requirements than streaming video. Of course what is done today with standard file systems like NTFS, EXT-4 and others is to add hardware and reduce the number of video streams being written per device given the file system efficiency. But that really does not solve the problem and has a high cost with more disk drives, communications channels, and of course added power requirements.
File system efficiency for streaming I/O is one of the things that is really never published and yet it is critically important for many reasons. My definitions of file system scalability is: what is the performance of the raw hardware (HBA/NICs, switches, storage controllers and storage) running I/O to the raw devices streaming, and not taking into account the impact of cache (unless the real applications can and will take advantage of the cache).
I wish vendors would publish this number to give people configuring systems an understanding of the performance they can expect. The scalability of the file system is a critical aspect to the requirements for streaming video capture and it amazes me that vendors – both storage (block and NAS) nor file system vendors – publish numbers on scalability. I wish someone had the money and time to test this, as it would be fascinating.
