Is The Era of Cheap Disk Storage Over?

Demand for disk storage is exploding faster than manufacturers can supply it. That means hard drive prices must go up.

In the next few years enterprises will consume storage capacity faster than disk manufacturers can produce it, according to some forecasts. If something doesn't change that means the word is heading for a "storage gap": there won't be enough hard drives to accommodate all the data that enterprises want to store.

The reasons for the exploding demand for enterprise storage are well known and include the rise of Big Data implementations and the desire to "store everything" for later analysis. In the near future Internet of Things projects that generate vast quantities of data are also likely to require huge amounts of storage.  In 2015 a total of about 4 zettabytes (about 4 billion terabytes) of data will be generated, but in five years that volume is likely to have grown ten-fold to about 40 zettabytes, according to an IDC study.

So in the short to medium term at least the storage market is likely to be disrupted, according to Gene Ruth, a research director at Gartner. "Growth in capacity production is no longer accelerating, and is actually slowing down," he confirms. "It is not clear that it will be possible for capacity to match requirements, so in the next few years there are the conditions for a shortage in drives to put data on. For the next few years there could be a limitation."

In the past, disk manufacturers have introduced new technologies that allowed them to make disks with capacities that increased in line with the demand for storage while the cost per gigabyte actually fell. 

But the predicted growth in demand for storage is so high that the manufacturers can't keep up, according to Mark Whitby, a vice president at California-based disk vendor Seagate. "By 2020 companies  will be producing and storing ten times more data than today but we are bound by the laws of physics. Innovation has allowed us to keep the economics (of hard drives) very attractive, but we can’t keep up with this ten times growth over the next five years," he says.

Rather than making a similar number of hard disk drives  at similar prices with capacities that grow in line with the growth in demand, in the future companies like Seagate will have to make more drives, and these will have to be bigger, with more spinning platters in them, Whitby believes. And inevitably, the cost per gigabytes will rise.

That's unless and until manufacturers can invent new technologies to cram an order of magnitude more data onto drives and produce them at a price per gigabyte comparable to today's prices.

The problem is that neither of these two options – making more drives or making higher capacity drives with new technology – are things that can be done in the short term. "There is a two year lead time at least to bring a new manufacturing facility online," explains Whitby,  "and the reality is that it isn't just one factory that's required. Different parts are made in different factories."

 A new factory costs about $1.5 billion to get up and running,  and that's a significant investment, he adds. "We have to decide when to make an investment, because the industry's history is one of over-capacity and then under-capacity," he says. "But we also have to decide in which area to invest: more disks, or innovations that cram more data on."

In many ways the hard drive market is very similar to the oil market. At various times over the last fifty years or so there have been warnings that oil would run out or that increasing demand would outstrip supply – leading to serious shortages.

But in every case these scenarios have been avoided thanks to the power of market forces. The price mechanism has come to the rescue, altering the behaviour of oil consumers and producers in such a way as to stimulate supply and reduce demand. The price of oil has gone up until demand has subsided (due to consumers switching to cheaper alternative energy sources or by adopting new technologies such as more fuel efficient automobile engines). Or  supply has increased using previously uneconomic hydrocarbon extraction techniques, or completely new ones such as hydraulic fracturing, or fracking.

What we're likely to see in the storage market is something similar. The current period of low and falling hard drive storage costs may be about to end, replaced by a period of rising costs per gigabyte of data stored. The obvious questions then are how will this influence storage consumers' and producers' behaviours?

On the demand side, rising hard drive costs (per gigabyte stored) will make buying new hard drive storage capacity relatively less attractive compared to alternatives. So what are the alternatives?

One obvious alternative is flash storage, but this medium is far more expensive per gigabyte than hard drive storage.

However, enterprises do use flash storage when its higher cost can be justified by the performance benefits that it offers over hard drive based storage. This suggests that if the price differential decreases due to rising hard drive costs then at the margin it will make economic sense for some enterprises to buy flash storage for its performance benefits where before it didn't.

But flash storage usage is just a drop in the ocean compared to the amount of disk-based storage in use, so the effect of this substitution is likely to be negligible.

A far more likely scenario is that enterprises will look at the increased price differential between disk drives and low cost tape storage and decide that the drawbacks of tape – such as poorer performance – are worth swallowing in an increasing number of circumstances. "The available tape storage capacity is mind blowing, so we could see companies tiering off to tape, and accepting slower recovery speeds," says Mark Peters, a senior analyst at Enterprise Strategy Group.

He also suggests that existing capacity will have to be used more effectively. "We have become profligate with storage because it is cheap. We will have to find ways to be smarter."

For example, he suggests companies could make use of sliced storage systems that store encrypted chunks of their data on storage systems belonging to companies around the world that have excess, unused capacity.

Another way of using existing capacity more effectively might be to switch from data protection using RAID 1 - which simply mirrors or duplicates data to make additional copies, to a more storage efficient RAID level which provides similar protection. "Rising costs could push people to accept a 5% performance hit of using a different RAID level," he says. "The fact is that some people simply replicate their data as it is less hassles, and there is no performance hit."

Of course another alternative to disk based storage is not using any storage at all, and Peters reckons that most companies could probably delete more than 20% of the data they currently hold without any noticeable effect to the business and without breaching any regulatory requirements.

Gartner's Gene Ruth broadly agrees with this assessment. "It is curious but we are constantly telling clients about technologies to use storage more effectively like dedupe, thin provisioning and so on. But there is a big opportunity to use content management. People often fail to understand what they could do if they managed their data well. It gives you the confidence to press the delete key. At the moment there is not enough investment in technology that helps you to call out data and get rid of it."

He suggests that moving to cloud storage can be helpful in this regards, but not for the reason that you may imagine. Cloud storage may well be cheaper than storage on an enterprise storage array, but Ruth points out that by paying for what you use – rather than filling up a storage array that has been paid for in the past – can make a big difference.

"When companies get a monthly bill it can be a big motivator to cut down the storage they use," he explains.

When it comes to the supply side, there’s no doubt that new fabrication facilities will come online to ensure that higher volumes of hard drives can be made – eventually. But unless a new technology – the equivalent of fracking in the oil industry – makes disk storage plentiful and cheap to produce then simply making more hard drives won't drive the price down fast enough.

It's impossible to predict what such a future storage technology might be, although one promising possibility includes heat assisted magnetic recording (HAMR) drives. HAMR uses a small laser to heat the part of the disk that is about to be written to. The effect is to allow smaller bits to be written to the disk, increasing the potential areal density to about 5,000 Gb per square inch. That means that 3.5" disk drives with a capacity of 60TB may be possible, although in the near term they won't offer this type of capacity.

The good news then is that until new storage technology comes to the rescue, as it inevitably does, there are plenty of opportunities for enterprises to store less data and store the data they do need more efficiently.

But the bad news is that a period of cheap disk storage is likely to be coming to an end. "I don't think we will run out of storage," concludes Seagate's Mark Whitby, "but I think we are entering a period where storage prices will rise subtly. "