Tape and solid state devices offering huge storage capacity are on the horizon, and that's likely to change storage hierarchies dramatically.
Linear Tape-Open (LTO ) cartridge capacity is exploding: LTO generation 7 (LTO-7) tapes, expected next year, will have more than double the capacity of the current generation.
HP, IBM and Quantum, the companies behind LTO, have confirmed that next gen cartridges will offer up to 15TB of compressed data storage, and published the specifications for third part manufacturers.
And it's not just LTO tape technology that is seeing an explosion in capacity: last year Sony announced tape technology that could result in tape cartridges with a capacity of 185TB, while in April IBM and Fujifilm demonstrated new technologies that cram 123 billion bits in a square inch of tape, equivalent to an LTO tape cartridge holding 220TB.
Nor are the technology breakthroughs confined to tape storage. At the 2015 Flash Memory Summit in August, Samsung announced a 16TB flash-based solid state drive (SSD), built using it 48-layer 3D TLC NAND technology. Although the drive isn't (yet) available to buy, and no price has been set for it, it does apparently mark the first time that a single solid state device has a larger capacity than a tape cartridge.
All this news should be enough to make traditional spinning disk storage makers nervous.
"Twenty five years ago we had a storage hierarchy of disk – optical – tape, but then optical got squeezed out," says Dave Russell, a Gartner research vice president. "Now for hard drives you can see flash encroaching at the high end and tape at the low end."
This ominous message is echoed by Jason Buffington, a senior analyst at Enterprise Strategy Group. "Hottest data should be on flash, and coldest data should be on tape. So you could argue that traditional disks are the things that are becoming antiquated in this modern world – what exactly is the purpose of spindles?" he asks.
Do these analysts really think that the days of hard drive based enterprise storage are numbered? Absolutely not – for the foreseeable future, at least.
For now something quite different could be about to happen: hard drives could start encroaching into tape territory. Consider this: Ultra-high capacity tape would likely have a huge cost per Gigabyte advantage over hard disk drives. But what if excess hard disk drive storage capacity is offered by cloud storage providers at a loss?
That would change the economic argument for using tape. And it's not hard to imagine a scenario where cloud storage providers make hard drive storage capacity available at below cost, but with some strings attached – if the alternative is leaving this storage capacity idle, generating no revenue at all.
In fact cloud storage already offer storage services at lower cost that their standard storage services, with strings attached – Google's Durable Reduced Availability (DRA) , and Amazon's Reduced Redundancy Storage (RRS) to name but two – so the idea is not so far-fetched.
There's little doubt that the way many people think about flash/disk/tape storage hierarchies is already starting to change: All flash arrays are becoming more common, as are hybrid systems, and tape is far from becoming the low-tech, slow, outdated storage medium that many thought it would.
But for now tape storage sales are healthier than they have ever been, and the LTO announcement is significant because when it comes to non-proprietary tape storage, LTO is now the only game in town, says David Russell.
"LTO has effectively won the war. SuperDLT, DAT - they've all gone away. If you are not going with a proprietary tape solution from Oracle or IBM, you're going to go with LTO," he says.
This is backed up figures from Santa Clara Consulting Group, which reckons that LTO tape cartridges account for about 95% of the market for tape.
One reason for LTO's success is that it has always a published roadmap for the technology. In addition to LTO-7, the most recent roadmap, published in 2014, shows LTO-8 having an expected compressed capacity of 32TB, LTO-9 with 62.5TB, and LTO-10 with a whopping 120TB.
New LTO generations are released approximately every two to three years.
What's important about this roadmap is that it is credible, believes Jason Buffington. "The power of the LTO roadmap is not just because it is forward facing. It's also because it proves that LTO can deliver what it has promised in the past," he says.
But he says that the key attraction of LTO is that the format has evolved slowly and consistently, maintaining the promise of read/write compatibility with the previous generation, and read compatibility going back two generations.
"In 25 years of data protection, the challenge of tape is that it has always been something that changes significantly every few years – and that's just what you don't want for long term retention," says Buffington. "But LTO evolves but has remained consistent – that's just what you do want, and that has power for many people."
LTO-10 is not likely to be available until about eight or so years from now, and its projected compressed capacity of 120TB pales into insignificance when compared to the sorts of capacities that may be enabled by the new Sony and IBM/Fujitsu technologies.
So it appears that the backwards compatibility that LTO offers comes at a price: the format is constrained in the capacity increases that can be delivered in successive generations.
David Russell thinks that this is definitely the case, but that this is not necessarily a bad think from a market adoption perspective – for now.
"LTO is shackled by backwards compatibility, yes, but the question is: does this harm the format, or quite the opposite: does it actually engender more customers by relieving them of worries about backward compatibility?"
He says the ability to read back two generations and write back one generation is a very strong selling point, because businesses then don't have to worry about carrying out manual data transfers.
It also means they have the luxury of staying behind on the upgrade path and sweating their existing assets a little longer than they might otherwise be able to. They can continue with LTO-6 until LTO-7 comes out – or wait for LTO-8, or even LTO-9.
But backwards compatibility may not always be such a strong selling point in the future, he warns. "For long term archives and workloads, if capacity becomes more important than compatibility then being shackled to previous generations could count against the format," Russell says.
If capacity does become all-important, is there a danger that ultra-high capacity tape cartridges offering 185TB or more could actually have too much capacity for their own good, despite offering unprecedentedly low cost per Gigabyte storage?
It's certainly possible to imagine a situation where a single cartridge holds so many objects that the tape needs to be accessed constantly – even if each individual object is likely to be accessed only very rarely. This could lead to contention issues, and perhaps even reliability problems if the cartridge has to be accessed far more often than is the case with a single LTO cartridge today.
Jason Buffington thinks too much capacity is likely to become a problem. He points out that enterprise data requirements are increasing, but not solely because they are storing more individual files or objects. It's also true that files and objects in general are getting larger.
For example, a Word document today is bigger than an old Word document, so even if the storage capacity of a tape is larger, it doesn't mean that the amount of documents you are storing is greater, he says.
What about the argument that it's ever a good idea to put too much data on a single "storage thing?"
David Russell believes that fears of storing "too many eggs in one basket" are probably unnecessary, because however much data is stored on a single device, you still need two or more geo-dispersed copies of data for redundancy.
"It is true to say that if you lose a higher capacity tape or something happens to it then more data is at risk, he says. "There is obviously greater concern over a single point of failure going from 6TB to 185TB or more in a single cartridge, but it is probably exaggerated," he concludes.