Uncle Sam has a datacenter waste problem
(2022/03/11)
- Reference: 1647024045
- News link: https://www.theregister.co.uk/2022/03/11/us_government_datacenter_use/
- Source link:
US government auditors want to save taxpayers' money by bolstering the capability and efficiency of Uncle Sam's far-flung stable of datacenters. Each federal agency's sites have a host of problems, unsurprisingly.
The US Government Accountability Office (GAO) kicked off its Datacenter Optimization Initiative (DCOI) in 2016 with the goal of reducing the massive financial and energy waste being generated by these warehouses of computers. We're not talking small change, either: in 2007, the Environmental Protection Agency estimated the annual electricity bill for federal government datacenters and servers was $450 million, and the Dept of Energy said each facility can use 100 to 200 times as much energy as a commercial building.
Despite all that energy and monetary expenditure, two years later the US Office of Management and Budget (OMB) found that government server utilization rates were as low as five percent.
[1]
"The high cost and low utilization led OMB to establish a coordinated, government-wide effort to improve the efficiency, performance, and environmental footprint of federal datacenter activities," the GAO said.
[2]
[3]
The GAO's [4]latest report
To make matters worse, the OMB reports that the number of federal government datacenters has actually increased since the DCOI was enacted, growing from 5,607 in 2016 to 5,916 by 2018.
[6]
All of this has led to closing datacenters completely: 96 were shuttered in fiscal year 2020, 51 had already happened when the report was written and 29 more were in the planning phase.
So, why did US government agencies not meet their savings goals in 2021? The reasons vary, and some indicate that the GAO's measurement methods may not be accounting for all potential paths to savings.
The Nuclear Regulatory Commission, for example, didn't meet its virtualization goals because it implemented a higher density of virtual servers on smaller, higher-performance hardware. This means it fell short in 2021, but in 2022 "targets have been adjusted to align its new infrastructure with the metric."
[7]
The Small Business Administration wasn't able to meet its virtualization goals due to technical gremlins – it couldn't fit virtualized servers on existing hardware; and budgetary holdups – it had no money to move servers to the cloud. Its plans for the future involve applying for a grant and hoping for modernization funds.
Heath and Human Services was unable to meet its energy monitoring goals, and that may not have even been their direct fault: "it did not meet advance energy metering due to a change in the number of valid datacenters counted to establish its energy metering target," the report said.
[8]US winds up national security team dedicated to Chinese espionage
[9]Face Off: IRS kills plan to verify taxpayers with facial recognition database
[10]US DoD staffer with top-secret clearance stole identities from work systems to apply for loans
[11]President Biden orders transformation of 'Federal Customer Experience'
Despite shortcomings in meeting particular metrics, the overall intent of the DCOI, to reduce datacenter costs, is being fulfilled to a degree. In FY 2020, the GAO reported that 14 agencies exceeded their cost-savings goals, four met their goals, and six met their goals on a technicality (they were aiming for zero-dollar savings and got there).
The data from 2021 so far isn't as cut and dry, though there's still hope to be found. Eight agencies exceeded their savings goals by August, and eight had and met the same zero-dollar savings goal as in 2020. Seven agencies said they hadn't met their goals as of August, but still planned to save a total of $267.92 million. ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.gao.gov/products/gao-22-105118
[5] https://www.gao.gov/assets/gao-22-105118.pdf
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/02/24/us_china_initiative_ended/
[9] https://www.theregister.com/2022/02/07/irs_facial_plan_scrapped/
[10] https://www.theregister.com/2022/01/27/dod_sharepoint_apple_white_house/
[11] https://www.theregister.com/2021/12/14/president_biden_orders_transformation_of/
[12] https://whitepapers.theregister.com/
The US Government Accountability Office (GAO) kicked off its Datacenter Optimization Initiative (DCOI) in 2016 with the goal of reducing the massive financial and energy waste being generated by these warehouses of computers. We're not talking small change, either: in 2007, the Environmental Protection Agency estimated the annual electricity bill for federal government datacenters and servers was $450 million, and the Dept of Energy said each facility can use 100 to 200 times as much energy as a commercial building.
Despite all that energy and monetary expenditure, two years later the US Office of Management and Budget (OMB) found that government server utilization rates were as low as five percent.
[1]
"The high cost and low utilization led OMB to establish a coordinated, government-wide effort to improve the efficiency, performance, and environmental footprint of federal datacenter activities," the GAO said.
[2]
[3]
The GAO's [4]latest report
[5]PDF
on the state of DCOI progress indicates "mixed progress" at best. Of the 24 agencies that fall under the DCOI's purview, only half met virtualization metrics, 16 met availability requirements, half were properly monitoring energy consumption, and 13 were properly reporting underutilized servers.To make matters worse, the OMB reports that the number of federal government datacenters has actually increased since the DCOI was enacted, growing from 5,607 in 2016 to 5,916 by 2018.
[6]
All of this has led to closing datacenters completely: 96 were shuttered in fiscal year 2020, 51 had already happened when the report was written and 29 more were in the planning phase.
So, why did US government agencies not meet their savings goals in 2021? The reasons vary, and some indicate that the GAO's measurement methods may not be accounting for all potential paths to savings.
The Nuclear Regulatory Commission, for example, didn't meet its virtualization goals because it implemented a higher density of virtual servers on smaller, higher-performance hardware. This means it fell short in 2021, but in 2022 "targets have been adjusted to align its new infrastructure with the metric."
[7]
The Small Business Administration wasn't able to meet its virtualization goals due to technical gremlins – it couldn't fit virtualized servers on existing hardware; and budgetary holdups – it had no money to move servers to the cloud. Its plans for the future involve applying for a grant and hoping for modernization funds.
Heath and Human Services was unable to meet its energy monitoring goals, and that may not have even been their direct fault: "it did not meet advance energy metering due to a change in the number of valid datacenters counted to establish its energy metering target," the report said.
[8]US winds up national security team dedicated to Chinese espionage
[9]Face Off: IRS kills plan to verify taxpayers with facial recognition database
[10]US DoD staffer with top-secret clearance stole identities from work systems to apply for loans
[11]President Biden orders transformation of 'Federal Customer Experience'
Despite shortcomings in meeting particular metrics, the overall intent of the DCOI, to reduce datacenter costs, is being fulfilled to a degree. In FY 2020, the GAO reported that 14 agencies exceeded their cost-savings goals, four met their goals, and six met their goals on a technicality (they were aiming for zero-dollar savings and got there).
The data from 2021 so far isn't as cut and dry, though there's still hope to be found. Eight agencies exceeded their savings goals by August, and eight had and met the same zero-dollar savings goal as in 2020. Seven agencies said they hadn't met their goals as of August, but still planned to save a total of $267.92 million. ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.gao.gov/products/gao-22-105118
[5] https://www.gao.gov/assets/gao-22-105118.pdf
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YivUmU-m33YaCXquZ3ZBJAAAAEI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/02/24/us_china_initiative_ended/
[9] https://www.theregister.com/2022/02/07/irs_facial_plan_scrapped/
[10] https://www.theregister.com/2022/01/27/dod_sharepoint_apple_white_house/
[11] https://www.theregister.com/2021/12/14/president_biden_orders_transformation_of/
[12] https://whitepapers.theregister.com/
Re: This is the result of an organizational problem
Yet Another Anonymous coward
And if you do report your datacenter is only running at 49% capacity it would obviously make sense to merge it with another organization that is using 51%. So you will get taken over unless you can get your utilization up above half.
Time for some bitcoin mining or just wrapping all tasks in docker containers till the fans are roaring
Re: This is the result of an organizational problem
ecofeco
It's Big Org culture everywhere.
At one job I had to trace down and find lost servers. How do you just lose a server, let alone several? Yet, they did.
quite a bit of growth
Nate Amsden
5000 data centers?
it was ~2000 data centers 12 years ago
https://www.datacenterknowledge.com/archives/2010/10/13/feds-discover-1000-more-data-centers
"The process defined a data center as any room larger than 500 square feet dedicated to data processing that meets the one of the four tier classifications defined by The Uptime Institute."
500 sq feet seems more like a server room than a data center(data center I'd say should start at 10k sq feet?), but I guess they lump them all together so the non tech people don't get too confused.
And there's me worrying...
Lost in Cyberspace
... about my home file server PC being wasteful, with an outlay of £140, and sucking up £5 of electricity a month.
This is the result of an organizational problem
I have no problem believing this is directly associated with the "culture" of US government.
A big part of it is going to be how department budgets "work" within the government. Basically, if you don't use it, you lose it. Any department coming in under budget, very likely will see their annual budget reduced by that amount the next year, because "obviously, you don't need it." This causes all kinds of hijinks around the end of the fiscal year. As a tech in the Marine Corps (yes, hence the screen name), back in the early 90's, I got to see it in stark contrast. We did circuit board repair in our shop. We repaired boards right down to the individual component on the board. As part of that, we had a yearly allotment of powdered gold to use for re-electroplating contacts on the boards. Come the end of the fiscal year, if we hadn't used it, we'd get a reduced allotment the next year. That would really cause issues if we had MORE boards to repair the next year. So, in typical USMC fashion, we were told to improvise, adapt, and overcome. Here's what would happen. EVERYTHING got plated until it was used up. Every circuit board we could get our hands on had their contacts plated. Standard contacts that didn't require gold got plated. Dog tags got plated. Pens got plated. Anything that would hold still long enough to get jammed into the electroplating machine, got plated.
So there's the first questionable practice.
Second comes from the fact that it is a Herculean effort to get rid of ANYTHING in government service. I have a friend that teaches IT courses in the US Job Corps system. It is an absolute nightmare to get rid of excess or outdated equipment. PCs that are 11 to 12 years old sit in stacks gathering dust because they're no longer of any use, but nobody wants to wade through the bureaucratic nightmare to get rid of any of it.
With those two behaviors in mind, is it really any stretch to imagine that government datacenters have racks crammed with equipment that is either surplus to requirement or beyond usefulness? New equipment keeps coming in whether it has a use or not, because otherwise the budget is lost and in the future they MIGHT need additional kit. Nothing gets thrown away so there are probably racks of legacy servers just sucking up power and not actually fulfilling any tasks because nobody wants to jump through the hoops required to retire it from service and dispose of it.
I am honestly surprised it's not actually worse than it is.