A Yottabyte of storage

From SusoSight

In my life I've seen us go from Kilobytes to Megabytes to Gigabytes to Terabytes and now just barely large arrays are now hitting Petabytes. Just the other day I was telling someone about how in the 90s we would joke at the concept of a terabyte. And now we are here. So when will we see a Yottabyte of storage. Here is my estimate. Note that in these numbers I'm mostly talking about consumer grade equipment.

  • 1960s - (Megabytes in mainframe use)
  • 1970s - Kilobytes (Megabytes in mainframe use)
    • Harddrives were never below a few megabytes in size, but things like floppy drives and volatile memory were.
    • The Illiac IV supercomputer from 1972 apparently had a special laser based metal film memory that could store up to 1 terabit (125 Gigabytes) of data. However this was write once memory.
    • 1972 - In the documentary "Computer Networks - The Heralds of Resource Sharing", Lawrence G. Roberts refers to weather files that were "10 to the 11th bit". This would be a 12.5 GB file, which seems insane for 1972, but lends credibility to the metal film memory mentioned previously.
  • 1980s - Megabytes (KB/MB era for memory)
    • (1981) 10MB Hard drive ($3398)
    • (1989) ~600MB Hard drives
  • 1990s - Gigabytes (MB era for memory)
    • (1990) - IBM 4GB hard drive ($20,000)
    • (1993) - 9GB hard drive (I think they were about $3000)
    • (1999) - 20GB hard drives
    • 1999 - Suso's backups took up maybe 10GB.
  • 2000s - Giga/Tera era (MB/GB era for memory)
    • (2004) - 5.625 petabytes Google server farm capacity [1]
    • (2004) - 128MB microSD card is available
    • 2006 - Suso's backups took up about 500GB
    • (2007) - 1TB drives
    • (2007) - Place I worked at had a 40TB SAN that cost over $250,000
    • (2009) - 2TB drives ($450)
  • 2010s - Tera era (GB era for memory and maybe TB by 2015)
    • 2010 - 2TB drives available (as low as $200)
    • 2010 - Suso's backups take up 1.6TB
    • 2010 - Highest memory capacity seen for a motherboard is about 192GB (Supermicro X8DTN+-F), but they don't make the 16GB DIMMs yet that it would take and 18x8GB (144GB) dimms would cost $10,800 alone.
    • 2012 - Someone on the BLUG list talking about a small scale SAN for a lab being 5-10TB.
    • 2014 - You can buy 128GB microSD cards for about $190 (SanDisk 128GB 128G microSDXC Ultra microSD micro SDHC SDXC Class 10 UHS-I C10 Memory Card)
    • 2014 (Dec) - Western Digital 6TB hard drives available for as cheap as $250 on Newegg
    • 2014 (Dec) - Sandisk 960GB SSD drives for $400 on Newegg, but various models go as high as $1200.
    • 2014 (Dec) - Sandisk 128GB class 10 microSD cards for $105
    • 2014 (Dec) - Nice home desktop on Dell's website comes with 32GB of RAM and 3TB hard drive for $2000. An Alienware laptop $(4800) has as much as 32GB of RAM and a 512GB SSD drive. Its possible to get a desktop motherboard with 64GB of RAM.
    • 2014 (Dec) - Supermicro 3U servers adversed as having up to 1.5TB of RAM, however only 32GB DIMMS could be found ($350 each) (24 slots on server).
    • 2015 (Oct) - 8TB Internal Hard drives for $600, 1TB "green" Internal hard drive for $50. 2TB Samsung SSD drive $709, 512GB SSD drive $130, 128 MicroSD card $50, 2x16GB RAM Dimms for $250, most laptops these days come with 8GB of RAM and a 1TB hard drive.
    • 2016 (Mar) - Samsung announces 15TB SSD SAS drive (PM1633A model number Samsung MZ-ILS15T0), leap frogging the capacity of current magnetic hard discs (still only 8TB available) for the first time. The initial cost was $10,000 but it was hard to find on sale anywhere. As of March 2019 you can buy it for $6500.
    • 2017 - It was around 2017 that we started to see SSD drives match the capacity of magnetic drives on the market at the time.
    • 2018 (Nov) - At Supercomputing '18, a vendor was showing a 16TB M.2 drive and a 1U server that could hold 36 of them for a total of 576GB in 1U.
    • 2019 (Feb) - I created an account on IU's Scholarly Data Archive which at the time had total storage capacity of around 60PB. I mounted their interface and 'df' reported that I had access to a 4PB drive, which is probably the front end cache. This was amazing enough that [my tweet https://twitter.com/climagic/status/1100141338325078022] about it received a strong response.
    • 2019 (Mar) - Newegg is selling the following:
      • SSD:
        • $79.99 -500GB SAMSUNG 860 EVO Series 2.5" 500GB SATA III V-NAND 3-bit MLC Internal Solid State Drive (SSD) MZ-76E500B/AM
        • $147.99 - 1TB SAMSUNG 860 EVO Series 2.5" 1TB SATA III V-NAND 3-bit MLC Internal Solid State Drive (SSD) MZ-76E1T0B/AM
        • $344.20 - 2TB SAMSUNG 860 EVO Series 2.5" 2TB SATA III 3D NAND Internal Solid State Drive (SSD) MZ-76E2T0B/AM
        • $734.70 - 4TB SAMSUNG 860 EVO Series 2.5" 4TB SATA III 3D NAND Internal Solid State Drive (SSD) MZ-76E4T0B/AM
        • $1262.54 - 7.68TB Micron 5200 5200 ECO 7.68 TB 2.5" Internal Solid State Drive - SATA - TAA Compliant - 540 MB/s
        • $2770.67 - 7.6TB - Seagate XS7680SE70103 7.6TB SAS 12Gb/s, 15mm, 1DWPD SSD (Enterprise drive)
        • $6114.99 - 15.36TB - Seagate Nytro 3330 XS15360SE70103 2.5" 15.36TB SAS 12Gb/s 3D eTLC Solid State Disk - Enterprise
        • $6500.00 - 15.36TB - SAMSUNG PM1633A SAS MZ-ILS15T0 (from serversupply.com)
        • Note: Many different brands and models of 4TB SSDs available, most between $700 and $2000.
      • Magnetic:
        • $40.43 - 250GB Seagate Pipeline HD.2 ST3250412CS 250GB 5900 RPM 8MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive -OEM
        • $21.19 - 500GB Seagate ST3500414CS 500GB 5900 RPM 16MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive
        • $23.89 - 500GB Western Digital AV-GP WD5000AVDS 500GB 7200 RPM 32MB Cache SATA 3.0Gb/s 3.5"
        • $26.00 - 500GB Western Digital RE3 WD5002ABYS 500GB 7200 RPM 16MB Cache SATA 3.0Gb/s 3.5" (Enterprise drive)
        • $29.99 - 500GB WD RE4 500 GB Enterprise Hard Drive: 3.5 Inch, 7200 RPM, SATA II, 64 MB Cache - WD5003ABYX (Enterprise drive)
        • $34.99 - 1TB Hitachi HUA721010KLA330 1TB 32MB 7200RPM Enterprise Grade 3.5" SATA2 Hard Drive (Enterprise)
        • $35.18 - 1TB Western Digital WD1001FAES - 1TB 3.5 SATA 7.2K 3GB/s Non Hot-Plug Hard Drive (caviar black)
        • $48.19 - 1TB WD RE4 WD1003FBYX 1TB 7200 RPM 64MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive (Enterprise)
        • $55.17 - 1TB Seagate Constellation ES.3 ST1000NM0033 1TB 7200 RPM 128MB Cache SATA 6.0Gb/s 3.5"
        • $39.99 - 2TB MaxDigitalData 2TB 32MB Cache 7200PM SATA 3.0Gb/s 3.5" Internal Surveillance Hard Drive
        • $43.99 - 2TB Hitachi Ultrastar A7K2000 2TB HUA722020ALA330 (0F10942) 2TB 32MB Cache 7200RPM SATA2
        • $49.19 - 2TB Seagate Constellation ES ST2000NM0001 2TB 7200 RPM 64MB Cache SAS 6Gb/s 3.5"
        • $53.50 - 2TB WD RE4 WD2003FYYS 2TB 7200 RPM 64MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive Bare (Enterprise)
        • $62.74 - 2TB WD Blue 2TB Desktop Hard Disk Drive - 5400 RPM SATA 6Gb/s 64MB Cache 3.5 Inch
        • $84.99 - 3TB Seagate BarraCuda ST3000DM008 3TB 7200 RPM 64MB Cache SATA 6.0Gb/s 3.5" Hard Drive Bare Drive
        • $179.99 - 3TB Seagate Enterprise Capacity 3.5 HDD 3TB 7200 RPM 512n SATA 6Gb/s 128MB Cache Internal Hard Drive ST3000NM0005 (Enterprise)
        • $98.99 - 4TB Seagate BarraCuda ST4000DM004 4TB 5400 RPM 256MB Cache SATA 6.0Gb/s 3.5" Hard Drives Bare Drive - OEM
        • $109.99 - 4TB TOSHIBA X300 4TB Desktop Hard Drive 7200 RPM 128MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive Retail
        • $139.99 - 4TB Seagate Enterprise Capacity 3.5 HDD 4TB 7200 RPM 512n SATA 6Gb/s 128MB Cache Internal Hard Drive (Enterprise)
        • $184.99 - 4TB WD Black 4TB Performance Desktop Hard Disk Drive - 7200 RPM SATA 6Gb/s 256MB Cache 3.5 Inch - WD4005FZBX
        • $189.99 - 8TB Seagate BarraCuda ST8000DM004 8TB 5400 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive Bare Drive - OEM
        • $229.99 - 8TB Seagate IronWolf 8TB NAS Hard Drive 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive ST8000VN0022
        • $263.65 - 8TB Seagate Enterprise Capacity 3.5 HDD 8TB 7200 RPM 512e SAS 12Gb/s 256MB Cache Internal Hard Drive
        • $299.55 - 8TB WD Red Pro WD8003FFBX 8TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive
        • $394.37 - 8TB HGST Ultrastar He8 HUH728080ALN600 (0F23662) 8TB 7200 RPM 128MB Cache SATA 6.0Gb/s 3.5"
        • $282.99 - 10TB WD Purple 10TB Surveillance Hard Disk Drive - 5400 RPM Class SATA 6Gb/s 256MB Cache 3.5 Inch - WD100PURZ
        • $303.99 - 10TB WD Red WD100EFAX 10TB 5400 RPM 256MB Cache SATA 6.0Gb/s 3.5" NAS Hard Drive Bare Drive
        • $334.99 - 10TB WD Purple WD101PURZ 10TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive
        • $339.99 - 10TB Seagate Enterprise Capacity 3.5 HDD 10TB (Helium) 7200 RPM SAS 12Gb/s 256MB Cache Standard Model 512e Internal Hard Drive ST10000NM0096
        • $374.26 - 10TB Seagate Enterprise Capacity 3.5 HDD 10TB (Helium) 7200 RPM SATA 6Gb/s 256MB Cache Standard 4Kn Internal
        • $395.25 - 10TB WD Red Pro WD101KFBX 10TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Hard Drive Bare Drive
        • $368.79 - 12TB Seagate HDD ST12000NM0027 12TB 3.5 7200RPM 256MB SAS 12GB s Enterprise Bare
        • $369.99 - 12TB Seagate Enterprise Capacity ST12000NM0007 12TB 7200 RPM SATA 6Gb/s 256MB Enterprise Hard Drive
        • $399.51 - 12TB WD Purple WD121PURZ 12TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive
        • $442.99 - 12TB Seagate IronWolf Pro ST12000NE0007 12TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive Bare
        • $489.99 - 12TB Western Digital Ultrastar 12TB DC HC520 7200 RPM SATA 6.0Gb/s 3.5" Data Center Internal Hard Drive - 0F30146
        • $499.99 - 14TB Seagate IronWolf 14TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive ST14000VN0008
        • $549.99 - 14TB Seagate BarraCuda Pro ST14000DM001 14TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive
        • $598.99 - 14TB WD Ultrastar DC HC500 HDD 14TB 7200 RPM SATA III 512e 512MB Cache 3.5-Inch Data Center Internal Hard Disk


The future

A lot depends on the industry's ability to invent something beyond high capacity magnetic storage. SSD drives are still in their infancy as far as capacity goes (although they are growing fast) and we still have yet to see a consumer grade holographic storage device. I think that some kind of chemical storage medium will be required to get us beyond the 2020s and maybe then we'll see another surge of invention that could take us to Yottabyte levels.

  • 2020s - Petabytes (TB for memory)
  • 2030s - Petabytes/Exabytes (Also the end of 32-bit timespace for unix)
  • 2040s - Exabytes
  • 2050s - Zettabytes
  • 2060s - Zettabytes/Yottabytes
  • 2070s - Yottabytes
  • 2090s - ????bytes (1024^9 so maybe a nonubyte, novembyte or enneabyte? They haven't been named yet but previous prefixes were based on the greek or latin name of the exponent of 1024^N, Novembytes, the official data size of your Thanksgiving dinner.)

No official prefixes to use after Yotta at this time.

So if I live to be 85 I MIGHT get to see yottabytes of storage in use. Of course if they don't fix the data transfer bottleneck we've been in, it will take until the rise of the next intelligent species on Earth to copy the data (1).

Uses for a Yottabyte

In the article data powers of 10, the author states as an example that all words ever spoken by humans would take up 5 exabytes. This is presumably all words spoken in text form (not audio data).

So following along those lines, I thought a good example for demostrating how big a yottabyte would be is to ask "How much storage space would be required for a video recording of the whole lifetime of every person's that ever lived. This surely would be the ultimate expression of data storage as if you recorded someone's life completely, you'd pretty much cover everything they cared about as well. And since you'd be recording every person ever, you'd have all the data. It would basically be more data than anyone could do anything with. Nobody would be able to watch even one whole recording of someone else's life without wasting their own life.

So first we need to know how many people have ever lived. A quick search on google reveals an estimate of 100 billion people, which sounds about right.

We'll say that the average lifetime is 40 years, which kinda accounts for the fact that over the past century or two lifetimes have increased in length dramatically. 40 years is 1,262,269,440 seconds.

Now for a format, let's go ahead and use a modern format of 1920x1080 HD. Quite a bit of data alone just for a 2 hour movie. The data rate of HD is close to 1MB per second, so we'll use that.

So we have this simple equation

1,262,269,440 seconds X 1MB (220) X 100,000,000,000 = 132,358,544,031,744,000,000,000,000 = 132 YB = 109 YiB

Of course this is just for a single copy of all the recordings without backups, so there you go. Humanity will probably find some need for 1000YiB drives anyways. Probably everyone would need to have a copy of everyone else's lifetime, even if they didn't have time to watch it all. Which is kinda what people who hoard massive amounts of media do right now.

  • (1) - 1 YB / 150MB a second / 86400 sec in day * 365 days in year = 243,726,007 years (and 150MB/sec is being generous)