Backblaze finds drive failures creeping up as disks age
(2022/08/03)
- Reference: 1659528914
- News link: https://www.theregister.co.uk/2022/08/03/backblaze_quarterly_report/
- Source link:
Cloud storage and backup provider Backblaze has published its latest quarterly report detailing the reliability of disk storage deployed in its datacenters.
[1]Backblaze had a total of 219,444 hard drives and SSDs in its bit barns at the end of calendar Q2, of which 4,020 are boot drives, so the company has focused on the 215,424 data drives it has under management for the report. A separate report on reliability of the SSDs will be published later.
For the three-month period, Backblaze says that the annualized failure rate (AFR) across all drives in its datacenters increased to 1.46 percent, compared with 1.22 percent in the previous quarter and 1.01 percent for a year ago, during Q2 of 2021.
[2]
However, the company said this appears to be related simply to the aging of its entire drive fleet and, it expected this AFR number to go down as older drives are retired over the coming year.
[3]
[4]
For example, Backblaze previously reported that the drive model exhibiting the lowest failure rate was a 6TB Seagate drive (ST6000DX000), also the oldest in the company’s inventory at the time. During the quarter, these drives did experience a couple of failures, but this was the first since Q3 of last year, and the drives now boast an average age of 86.7 months of service.
[5]
Click to enlarge
“At some point in the future we can expect these drives will be cycled out, but with their lifetime AFR at just 0.87 percent, they are not first in line,” said Backblaze’s principal cloud storage evangelist Andy Klein.
However, the next to oldest drives in Backblaze’s portfolio are 4TB Toshiba drives (model MD04ABA400V) which have been in service for an average 85.3 months and these recorded zero failures during Q2. The AFR for these drives is just 0.79 percent, but this comes with the caveat that the lifetime confidence interval gap for them is 1.3 percent, which means Backblaze is lacking enough data to be confident of the true AFR.
Other drives that Backblaze reveals as having zero failures during Q2 are an 8TB HGST model (HUH728080ALE604), as well as some 14TB and 16TB Toshiba drives (MG07ACA14TEY and MG08ACA16TA respectively). But, as with the 4TB Toshiba model, these drives have very wide confidence interval gaps because of a relatively limited number of data points.
[6]
Some older drives on Backblaze’s books that are definitely showing their age are 4TB Seagate models (ST4000DM000), which have increased their failure rate over the past four quarters and have now reached 3.42 percent. The company said it has initiated its drive cloning program for these drives as part of its data durability program, and the drives will be cycled out over the next few months.
For the lifetime failure rates of drives, Backblaze looked at a total of 215,011 hard drives, excluding those being used for testing purposes or models for which the company did not have at least 60 drives in operation.
[7]
Click to enlarge
Backblaze found that the lifetime AFR across all these drives is 1.39 percent, which is the same as during the previous quarter and down from the 1.45 percent found during the same quarter in 2021.
Three drives can be picked out from the table with the highest failure rates, and these are an 8TB HGST model (HUH728080ALE604) at 6.26 percent; a Seagate 14TB model (ST14000NM0138) at 4.86 percent; and Toshiba’s 16TB (MG08ACA16TA) with 3.57 percent.
However, Backblaze cautions that the number of “drive days” (the number of days all the drives of a specifc model were operational during the defined period) for these is on the low side, leading to a wide gap between the low and high confidence interval values and therefore lower confidence in those AFR figures.
[8]Backblaze report finds SSDs as reliable as HDDs
[9]Toshiba reveals 30TB disk drive to arrive by 2024
[10]Back up for a minute – Backblaze HD reliability stats show oldies can be goodies
[11]Backblaze, long a champion of home-grown hardware, succumbs to the lure of commodity servers
For this reason, the company has drawn up another table comprising drives which have a minimum drive days value of one million and are larger than 8TB in capacity. This cuts the number of models down to 13, and the model with the highest failure rate here is a 12TB Seagate model (ST12000NM0007) with an AFR of 2.03 percent.
[12]
Click to enlarge
Backblaze said it now classifies drive failures in its datacenters into two categories - reactive and proactive. The first is the traditional failure mode, where a drive is no longer functioning failed or not responding to commands. A proactive failure is where failure is judged to be imminent, and the drive is removed before it stops functioning. This is based on errors reported by the drive and confirmed by examining its onboard SMART statistics.
For those interested in analyzing the data for themselves, Backblaze makes the complete data set used to create this latest report available from its [13]Hard Drive Test Data page. ®
Get our [14]Tech Resources
[1] https://www.backblaze.com/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YuqbtRwovgk2@iAY8@C4DwAAAJA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YuqbtRwovgk2@iAY8@C4DwAAAJA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YuqbtRwovgk2@iAY8@C4DwAAAJA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://regmedia.co.uk/2022/08/03/bb_table_1.jpg
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YuqbtRwovgk2@iAY8@C4DwAAAJA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://regmedia.co.uk/2022/08/03/bb_table_2.jpg
[8] https://www.theregister.com/2022/03/04/backblaze_ssd_stats/
[9] https://www.theregister.com/2022/02/09/30tb_toshiba_disk_drive/
[10] https://www.theregister.com/2022/02/01/backblaze_hd_reliability_stats_show/
[11] https://www.theregister.com/2021/06/28/backblaze_dell_storage_servers/
[12] https://regmedia.co.uk/2022/08/03/bb_table_3.jpg
[13] https://www.backblaze.com/b2/hard-drive-test-data.html
[14] https://whitepapers.theregister.com/
[1]Backblaze had a total of 219,444 hard drives and SSDs in its bit barns at the end of calendar Q2, of which 4,020 are boot drives, so the company has focused on the 215,424 data drives it has under management for the report. A separate report on reliability of the SSDs will be published later.
For the three-month period, Backblaze says that the annualized failure rate (AFR) across all drives in its datacenters increased to 1.46 percent, compared with 1.22 percent in the previous quarter and 1.01 percent for a year ago, during Q2 of 2021.
[2]
However, the company said this appears to be related simply to the aging of its entire drive fleet and, it expected this AFR number to go down as older drives are retired over the coming year.
[3]
[4]
For example, Backblaze previously reported that the drive model exhibiting the lowest failure rate was a 6TB Seagate drive (ST6000DX000), also the oldest in the company’s inventory at the time. During the quarter, these drives did experience a couple of failures, but this was the first since Q3 of last year, and the drives now boast an average age of 86.7 months of service.
[5]
Click to enlarge
“At some point in the future we can expect these drives will be cycled out, but with their lifetime AFR at just 0.87 percent, they are not first in line,” said Backblaze’s principal cloud storage evangelist Andy Klein.
However, the next to oldest drives in Backblaze’s portfolio are 4TB Toshiba drives (model MD04ABA400V) which have been in service for an average 85.3 months and these recorded zero failures during Q2. The AFR for these drives is just 0.79 percent, but this comes with the caveat that the lifetime confidence interval gap for them is 1.3 percent, which means Backblaze is lacking enough data to be confident of the true AFR.
Other drives that Backblaze reveals as having zero failures during Q2 are an 8TB HGST model (HUH728080ALE604), as well as some 14TB and 16TB Toshiba drives (MG07ACA14TEY and MG08ACA16TA respectively). But, as with the 4TB Toshiba model, these drives have very wide confidence interval gaps because of a relatively limited number of data points.
[6]
Some older drives on Backblaze’s books that are definitely showing their age are 4TB Seagate models (ST4000DM000), which have increased their failure rate over the past four quarters and have now reached 3.42 percent. The company said it has initiated its drive cloning program for these drives as part of its data durability program, and the drives will be cycled out over the next few months.
For the lifetime failure rates of drives, Backblaze looked at a total of 215,011 hard drives, excluding those being used for testing purposes or models for which the company did not have at least 60 drives in operation.
[7]
Click to enlarge
Backblaze found that the lifetime AFR across all these drives is 1.39 percent, which is the same as during the previous quarter and down from the 1.45 percent found during the same quarter in 2021.
Three drives can be picked out from the table with the highest failure rates, and these are an 8TB HGST model (HUH728080ALE604) at 6.26 percent; a Seagate 14TB model (ST14000NM0138) at 4.86 percent; and Toshiba’s 16TB (MG08ACA16TA) with 3.57 percent.
However, Backblaze cautions that the number of “drive days” (the number of days all the drives of a specifc model were operational during the defined period) for these is on the low side, leading to a wide gap between the low and high confidence interval values and therefore lower confidence in those AFR figures.
[8]Backblaze report finds SSDs as reliable as HDDs
[9]Toshiba reveals 30TB disk drive to arrive by 2024
[10]Back up for a minute – Backblaze HD reliability stats show oldies can be goodies
[11]Backblaze, long a champion of home-grown hardware, succumbs to the lure of commodity servers
For this reason, the company has drawn up another table comprising drives which have a minimum drive days value of one million and are larger than 8TB in capacity. This cuts the number of models down to 13, and the model with the highest failure rate here is a 12TB Seagate model (ST12000NM0007) with an AFR of 2.03 percent.
[12]
Click to enlarge
Backblaze said it now classifies drive failures in its datacenters into two categories - reactive and proactive. The first is the traditional failure mode, where a drive is no longer functioning failed or not responding to commands. A proactive failure is where failure is judged to be imminent, and the drive is removed before it stops functioning. This is based on errors reported by the drive and confirmed by examining its onboard SMART statistics.
For those interested in analyzing the data for themselves, Backblaze makes the complete data set used to create this latest report available from its [13]Hard Drive Test Data page. ®
Get our [14]Tech Resources
[1] https://www.backblaze.com/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YuqbtRwovgk2@iAY8@C4DwAAAJA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YuqbtRwovgk2@iAY8@C4DwAAAJA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YuqbtRwovgk2@iAY8@C4DwAAAJA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://regmedia.co.uk/2022/08/03/bb_table_1.jpg
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YuqbtRwovgk2@iAY8@C4DwAAAJA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://regmedia.co.uk/2022/08/03/bb_table_2.jpg
[8] https://www.theregister.com/2022/03/04/backblaze_ssd_stats/
[9] https://www.theregister.com/2022/02/09/30tb_toshiba_disk_drive/
[10] https://www.theregister.com/2022/02/01/backblaze_hd_reliability_stats_show/
[11] https://www.theregister.com/2021/06/28/backblaze_dell_storage_servers/
[12] https://regmedia.co.uk/2022/08/03/bb_table_3.jpg
[13] https://www.backblaze.com/b2/hard-drive-test-data.html
[14] https://whitepapers.theregister.com/
Re: "their failure rate over the past four quarters [has] now reached 3.42 percent"
Ayemooth
Actually the 6TB Seagate was the one to buy 86.7 months ago. Who knows what the reliability of the currently-available equivalent is.
I'm shocked
Horst U Rodeinon
To learn that the failure rate for (electro-)mechanical devices increases with age. What's the next mind bogglingly mind boggling news we will be treated to? I can hardly wait!
Re: I'm shocked
b0llchit
What's the next mind bogglingly mind boggling news...
Good News! You are getting older and will eventually die.
"their failure rate over the past four quarters [has] now reached 3.42 percent"
Okay, I will acknowledge that 3.42% of over 200k drives hits the 7000 replacement mark, which is non-negligeable.
Except that it is just one model, not the entire park. Why replace the entire range of the model for just a 3.4% failure rate ?
That seems a bit wasteful to me.
That said, Backblaze is a treasure trove of drive reliability reports and can't stop saying thank you to them.
I now know that the 6TB Seagate drive is the thing to buy at the moment. I'll be thinking about it.