NASA overspent $15m on Oracle software because it was afraid an audit could cost more
(2023/01/13)
- Reference: 1673575361
- News link: https://www.theregister.co.uk/2023/01/13/nasa_software_oracle_overpayment/
- Source link:
NASA is rubbish at software asset management, has not implemented federal government guidance on how to address it, and as a result is spending too much on code it doesn't use – including $15 million on unused Oracle software alone, under a twelve-year-old license the space agency was afraid to examine.
So says the aerospace agency's Office of Inspector General, which on Thursday published a [1]report [PDF] that opens with the unflattering observation that NASA's software asset management (SAM) practices "currently expose the Agency to operational, financial, and cyber security risks with management of the software life cycle largely decentralized and ad hoc." The report rates NASA’s SAM capabilities as "Basic" – the lowest ranking on the four-tier scale the US government employs.
It gets worse. The report finds that NASA hasn't embraced best practice, or implemented the SAM systems that discover, inventory, and track license data as required by federal policy. The agency's Software Asset Management Office and Software Manager positions "are misaligned and do not report to the Chief Information Officer as required by federal policy."
NASA is examining 'how and why' Oracle licensing became so cumbersome and complex to manage
Nor does NASA have a consistent process for negotiating with software vendors, or handling license audits. The report suggests NASA is therefore exposed to higher costs and penalties for violations of software license agreements.
The report uses the example of NASA's Oracle deal to demonstrate the issues, detailing how the agency was "unwilling to risk a license audit by Oracle because of the lack of solid, centralized visibility into deployment and use of the software."
[2]
Officials in NASA's office of the CIO told the Office of Inspector General they "knew better than to try our luck with an audit."
[3]
[4]
"Simply put, merely the potential threat of being audited by the vendor encouraged overbuying when the accuracy of Agency Software Asset Management was suspect," the report states.
NASA therefore spent $15 million on Oracle software it didn't use. And it's probably been spending too much with Oracle since 2011 – the year when it signed up with Big Red to manage the end of the Space Shuttle program.
[5]
"The Oracle license overspend has been in effect for more than a decade," the report states. "The Agency has not sufficiently tracked the full cost of license expenditures for the life of the existing contract which includes multiple option years in a manner which would allow the full costs to be known."
The auditor estimates NASA "could have saved approximately $35 million over the past five years in fines and overpayments ($20 million in penalties plus $15 million in Oracle overspend)" and is therefore questioning the costs.
The report acknowledges that "funding and staffing shortfalls" have contributed to NASA's poor SAM capabilities.
[6]
But lock-in hasn't helped, either.
"NASA purchased large amounts of Oracle products to support Space Shuttle processing and other mission operations during that timeframe containing licensing terms that made transitioning to a competitor difficult due to proprietary technologies," the report explains.
[7]NASA boss says US may lose latest space race with China
[8]NASA retires Mars InSight mission after it enters ‘dead bus’ condition
[9]NASA awards $60m to Texas biz for 3D printing future Moon base
[10]Oracle's compliance cops now include Java in license audits
NASA's Oracle licenses are due for renewal in April 2023 and the report states that the agency's officials "are gathering requirements and examining 'how and why' Oracle licensing became so cumbersome and complex to manage."
"In parallel, the Agency is also reviewing the current and desired licensing environment to quantify the true cost of doing business with Oracle."
Another revelation in the document is that NASA paid $4.36 million in software license violation penalties during FY 2021 alone.
NASA was able to negotiate some fees down to zero but sent $3.85 million to SUSE and $415,000 to SAP. The auditor suspects other payments may have been made over the last five years – probably to the tune of $20 million.
On top of the Oracle mess, that's $35 million of bad software spend, which the auditor thinks could have been avoided had SAM been in place – at a likely cost of $3 million to implement and $2.5 million a year to operate.
The report also offers the following unpleasant observations:
Software downloaded with privileged access is not tracked for license compliance and life-cycle management, and NASA does not have a consistent, Agency-wide process for limiting privileged access or using "least privilege" permissions, which gives users only the software permissions necessary for their job. This deviation from best practices is a cyber security risk because software deployed within the Agency raises both cyber security and software license compliance risks.
The report calls for NASA to get SAM right immediately. Until it does, "the Agency risks procuring software in a costly and ineffective manner, as well as incurring tens of millions of dollars in penalties for license non-compliance."
NASA management largely accepted the report's findings and stated that a SAM pilot will commence in October 2023, but that agency-wide implementation will not be complete until 2027.
The auditor has previously found that NASA has [11]sub-par cyber security and [12]grossly underestimated the cost of a cloud storage migration .
Seriously NASA, it's not rocket science. ®
Get our [13]Tech Resources
[1] https://oig.nasa.gov/docs/IG-23-008.pdf
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/01/04/nasa_nelson_moon/
[8] https://www.theregister.com/2022/12/22/mars_insight_retired_lost_contact/
[9] https://www.theregister.com/2022/11/29/nasa_moon_base_3d_printing/
[10] https://www.theregister.com/2022/03/22/oracle_starts_to_include_java/
[11] https://www.theregister.com/2022/12/21/nasa_immature_cycbersecurity/
[12] https://www.theregister.com/2020/03/19/nasa_cloud_data_migration_mess/
[13] https://whitepapers.theregister.com/
So says the aerospace agency's Office of Inspector General, which on Thursday published a [1]report [PDF] that opens with the unflattering observation that NASA's software asset management (SAM) practices "currently expose the Agency to operational, financial, and cyber security risks with management of the software life cycle largely decentralized and ad hoc." The report rates NASA’s SAM capabilities as "Basic" – the lowest ranking on the four-tier scale the US government employs.
It gets worse. The report finds that NASA hasn't embraced best practice, or implemented the SAM systems that discover, inventory, and track license data as required by federal policy. The agency's Software Asset Management Office and Software Manager positions "are misaligned and do not report to the Chief Information Officer as required by federal policy."
NASA is examining 'how and why' Oracle licensing became so cumbersome and complex to manage
Nor does NASA have a consistent process for negotiating with software vendors, or handling license audits. The report suggests NASA is therefore exposed to higher costs and penalties for violations of software license agreements.
The report uses the example of NASA's Oracle deal to demonstrate the issues, detailing how the agency was "unwilling to risk a license audit by Oracle because of the lack of solid, centralized visibility into deployment and use of the software."
[2]
Officials in NASA's office of the CIO told the Office of Inspector General they "knew better than to try our luck with an audit."
[3]
[4]
"Simply put, merely the potential threat of being audited by the vendor encouraged overbuying when the accuracy of Agency Software Asset Management was suspect," the report states.
NASA therefore spent $15 million on Oracle software it didn't use. And it's probably been spending too much with Oracle since 2011 – the year when it signed up with Big Red to manage the end of the Space Shuttle program.
[5]
"The Oracle license overspend has been in effect for more than a decade," the report states. "The Agency has not sufficiently tracked the full cost of license expenditures for the life of the existing contract which includes multiple option years in a manner which would allow the full costs to be known."
The auditor estimates NASA "could have saved approximately $35 million over the past five years in fines and overpayments ($20 million in penalties plus $15 million in Oracle overspend)" and is therefore questioning the costs.
The report acknowledges that "funding and staffing shortfalls" have contributed to NASA's poor SAM capabilities.
[6]
But lock-in hasn't helped, either.
"NASA purchased large amounts of Oracle products to support Space Shuttle processing and other mission operations during that timeframe containing licensing terms that made transitioning to a competitor difficult due to proprietary technologies," the report explains.
[7]NASA boss says US may lose latest space race with China
[8]NASA retires Mars InSight mission after it enters ‘dead bus’ condition
[9]NASA awards $60m to Texas biz for 3D printing future Moon base
[10]Oracle's compliance cops now include Java in license audits
NASA's Oracle licenses are due for renewal in April 2023 and the report states that the agency's officials "are gathering requirements and examining 'how and why' Oracle licensing became so cumbersome and complex to manage."
"In parallel, the Agency is also reviewing the current and desired licensing environment to quantify the true cost of doing business with Oracle."
Another revelation in the document is that NASA paid $4.36 million in software license violation penalties during FY 2021 alone.
NASA was able to negotiate some fees down to zero but sent $3.85 million to SUSE and $415,000 to SAP. The auditor suspects other payments may have been made over the last five years – probably to the tune of $20 million.
On top of the Oracle mess, that's $35 million of bad software spend, which the auditor thinks could have been avoided had SAM been in place – at a likely cost of $3 million to implement and $2.5 million a year to operate.
The report also offers the following unpleasant observations:
Software downloaded with privileged access is not tracked for license compliance and life-cycle management, and NASA does not have a consistent, Agency-wide process for limiting privileged access or using "least privilege" permissions, which gives users only the software permissions necessary for their job. This deviation from best practices is a cyber security risk because software deployed within the Agency raises both cyber security and software license compliance risks.
The report calls for NASA to get SAM right immediately. Until it does, "the Agency risks procuring software in a costly and ineffective manner, as well as incurring tens of millions of dollars in penalties for license non-compliance."
NASA management largely accepted the report's findings and stated that a SAM pilot will commence in October 2023, but that agency-wide implementation will not be complete until 2027.
The auditor has previously found that NASA has [11]sub-par cyber security and [12]grossly underestimated the cost of a cloud storage migration .
Seriously NASA, it's not rocket science. ®
Get our [13]Tech Resources
[1] https://oig.nasa.gov/docs/IG-23-008.pdf
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y8E5zl0fJ0MzDrB35cGdjwAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/01/04/nasa_nelson_moon/
[8] https://www.theregister.com/2022/12/22/mars_insight_retired_lost_contact/
[9] https://www.theregister.com/2022/11/29/nasa_moon_base_3d_printing/
[10] https://www.theregister.com/2022/03/22/oracle_starts_to_include_java/
[11] https://www.theregister.com/2022/12/21/nasa_immature_cycbersecurity/
[12] https://www.theregister.com/2020/03/19/nasa_cloud_data_migration_mess/
[13] https://whitepapers.theregister.com/
Re: Once upon a time
Code For Broke
Can anyone help me understand for what reason NASA would have a big contract with both Oracle and SAP? I'm honestly curious.
Re: Once upon a time
Anonymous Coward
If was anything like my momentary experience, it's because changing the database is way beyond the skills of their typical contractors. One developer was intentionally creating SQL injection vulnerabilities because, the dev claimed, parameterized SQL queries hadn't been reviewed for approval. It was a mind-blowingly lazy excuse, being that I was the reviewer.
Potemkine!
What I understand from reading the IT press about Oracle is: never ever work with them or be screwed.
I guess I'm not the only one. Then I don't understand how it is possible Oracle still has customers?
Joe W
Yes. That's it. You have to be extremely weary of any, even small, change. The Oracle licenses are cat-puke-clear (and even from a distance have a similarly obnoxious smell). And they are this way on purpose (most feel). Having an audit will cost much more than just overspending on the licenses, they will "eventually" (i.e. they knew all along but waited) discover the "anomalies", and then incur back payments.
I really am curious: without an actual audit, how did they discover they were overspending? Were the people involved intimately familiar with the way Oracle licenses work? Do they know that you (well, more or less, there's some rules, some restrictions) pay for every single CPU that could potentially be used (so for the full server farm your VM could be running on)? 15M$ is not that much, in terms of Oracle licenses.
(and remind me again, how much does a single F35 cost really?)
Pascal Monett
IIRC, NASA doesn't have a single F35 on it's roster of flight vehicles.
bregister
As a nearly 40 year user of the database, its a good product, the rest of what that company does is not so great any more.
Once upon a time
I worked for the four-letter-acronym agency, and much as I cursed the extra work associated with the SAM requirements, I thought the group responsible for SAM (part of the NASA Shared Services Center) was extremely customer-focused, perhaps because the NSSC is staffed mostly by contractor personnel. It would be a shame if their excellent customer service (bringing on new software under license when requirements for such are demonstrated, negotiating reasonable license terms, &c.) were sacrificed to concentrating solely on the big-ticket items cited in this report. I know they helped the projects I worked on and many others to control costs and remain compliant with Agency requirements.