Who knew? Hadoop is over, says former Hortonworks guru Scott Gnau
(2020/10/21)
- Reference: 1603282507
- News link: https://www.theregister.co.uk/2020/10/21/gnau_on_hadoop_object_storage/
- Source link:
Interview Tech history comes in waves. One minute you’re riding a crest, the next you’re wiped out. Just ask Scott Gnau, former CTO of Hortonworks, the company once seen as the figurehead of the big data boom.
Now head of data platforms at InterSystems, Gnau has had time to reflect on the Hadoop legacy and where those with torrential data workloads should go next.
“If you define Hadoop as some of the initial projects are out there, clearly, it's over, right? And the marketplace has moved on,” he told The Register .
But there is a caveat. The “broader ecosystem of open source technologies for data management” has left a valuable legacy, he said.
Just over two years ago, Gnau was promoting [1]Hortonworks Data Platform 3.0 based on Apache Hadoop 3.1 as a "huge step forward, architecturally" . But the world has changed.
Gnau joined InterSystems, a proprietary data platform vendor with strengths in the health sector, in 2019, after the merger between Hortonworks and Cloudera. Pitched as a marriage of equals, the [2]deal valued the combined entity as $5.2bn . It now hovers around $3bn.
The demise of Hadoop, as a distributed file system, and as an investment darling, has been linked to the growing popularity of object storage, integrated with the cloud hyperscalers' data offerings in the form of AWS S3, Azure Blob Storage, and Google’s Object Storage, as they are seen as accessible and cheap.
“Clearly object store is having an impact on the entire marketplace,” Gnau said.
Indeed, Teradata, the data warehouse stalwart where Gnau spent nearly 20 years, hitched its wagon to the object storage train in 2019, saying it offered low cost, easy management, and unlimited scalability with its “native” support for the cloud repository.
Object storage
But as a person who has seen some ups and downs, he has a salutary warning for those hyping object storage as an elixir for all data ailments.
“There are advantages to object stores and there are disadvantages,” he said. “To the extent that people believe there's a single answer to all these solutions, they'll be wrong. Object storage is all the craze today, I'm sure there’ll be something that follows on after that.”
While object storage is very flexible and easy to use, it is not specialised to any particular data task, he said. “With generic solutions, you're then also riding the hardware innovation curves to deliver improvements over time. With the continued orders of magnitude increases in data available for managing transactions and analytics, I think that the supply of data will continue to outstrip the supply of money and hardware to process it. So, there will always be kind of this yin and yang between specialised [kit] and technology,” Gnau said.
At his new home, InterSystems, Gnau claims there is the ability to address both structured and unstructured data in the same system with its “unique” proprietary data platform.
Obviously, being company IP he is reluctant to spill the beans on the secret sauce, but Gnau outlined to us a schema-less system that is akin to a “massively intelligent key-value store” with some internal indexing that allows both high volume onboarding of data and efficient analytics. He said it also offers in-memory storage and - having a background of supporting medical systems requiring storage of doctors’ notes and medical images - it also supports unstructured data, Gnau said.
As it strives to broaden its appeal outside its core medical market, InterSystems has signed up Chadwicks Group, a builders’ merchant based in Ireland, to help integrate business systems and provide real-time reporting.
Newly rolled out is Adaptive Analytics, which integrates live data in InterSystems IRIS and popular third-party business intelligence tools such as Tableau, Microsoft Power BI, and Excel. The firm has also made a deal with visualisation specialist AtScale to offer business users the ability to create their own dashboards.
Having spent years at Teradata working with its massively parallel, traditional RDBMS and moving across to manage unstructured big data at Hortonworks, Gnau claims to have found the “best both of the worlds” at InterSystems.
Whether it will stay that way depends of course on how fast those worlds change and whether the proprietary system can keep up. ®
Get our [3]Tech Resources
[1] https://www.theregister.com/2018/06/18/hortonworks_enables_containerisation_in_latest_data_platform_release/
[2] https://www.theregister.com/2018/10/04/cloudera_hortonworks_merger/
[3] https://whitepapers.theregister.com/
Now head of data platforms at InterSystems, Gnau has had time to reflect on the Hadoop legacy and where those with torrential data workloads should go next.
“If you define Hadoop as some of the initial projects are out there, clearly, it's over, right? And the marketplace has moved on,” he told The Register .
But there is a caveat. The “broader ecosystem of open source technologies for data management” has left a valuable legacy, he said.
Just over two years ago, Gnau was promoting [1]Hortonworks Data Platform 3.0 based on Apache Hadoop 3.1 as a "huge step forward, architecturally" . But the world has changed.
Gnau joined InterSystems, a proprietary data platform vendor with strengths in the health sector, in 2019, after the merger between Hortonworks and Cloudera. Pitched as a marriage of equals, the [2]deal valued the combined entity as $5.2bn . It now hovers around $3bn.
The demise of Hadoop, as a distributed file system, and as an investment darling, has been linked to the growing popularity of object storage, integrated with the cloud hyperscalers' data offerings in the form of AWS S3, Azure Blob Storage, and Google’s Object Storage, as they are seen as accessible and cheap.
“Clearly object store is having an impact on the entire marketplace,” Gnau said.
Indeed, Teradata, the data warehouse stalwart where Gnau spent nearly 20 years, hitched its wagon to the object storage train in 2019, saying it offered low cost, easy management, and unlimited scalability with its “native” support for the cloud repository.
Object storage
But as a person who has seen some ups and downs, he has a salutary warning for those hyping object storage as an elixir for all data ailments.
“There are advantages to object stores and there are disadvantages,” he said. “To the extent that people believe there's a single answer to all these solutions, they'll be wrong. Object storage is all the craze today, I'm sure there’ll be something that follows on after that.”
While object storage is very flexible and easy to use, it is not specialised to any particular data task, he said. “With generic solutions, you're then also riding the hardware innovation curves to deliver improvements over time. With the continued orders of magnitude increases in data available for managing transactions and analytics, I think that the supply of data will continue to outstrip the supply of money and hardware to process it. So, there will always be kind of this yin and yang between specialised [kit] and technology,” Gnau said.
At his new home, InterSystems, Gnau claims there is the ability to address both structured and unstructured data in the same system with its “unique” proprietary data platform.
Obviously, being company IP he is reluctant to spill the beans on the secret sauce, but Gnau outlined to us a schema-less system that is akin to a “massively intelligent key-value store” with some internal indexing that allows both high volume onboarding of data and efficient analytics. He said it also offers in-memory storage and - having a background of supporting medical systems requiring storage of doctors’ notes and medical images - it also supports unstructured data, Gnau said.
As it strives to broaden its appeal outside its core medical market, InterSystems has signed up Chadwicks Group, a builders’ merchant based in Ireland, to help integrate business systems and provide real-time reporting.
Newly rolled out is Adaptive Analytics, which integrates live data in InterSystems IRIS and popular third-party business intelligence tools such as Tableau, Microsoft Power BI, and Excel. The firm has also made a deal with visualisation specialist AtScale to offer business users the ability to create their own dashboards.
Having spent years at Teradata working with its massively parallel, traditional RDBMS and moving across to manage unstructured big data at Hortonworks, Gnau claims to have found the “best both of the worlds” at InterSystems.
Whether it will stay that way depends of course on how fast those worlds change and whether the proprietary system can keep up. ®
Get our [3]Tech Resources
[1] https://www.theregister.com/2018/06/18/hortonworks_enables_containerisation_in_latest_data_platform_release/
[2] https://www.theregister.com/2018/10/04/cloudera_hortonworks_merger/
[3] https://whitepapers.theregister.com/
Apache Hadoop isnt dead yet... but dying...
Apache Hadoop has some serious flaws that were never corrected.
HDFS should have followed MapR many moons ago but didn't. When MapR came out with a better mouse trap, Cloudera expected and demanded that MapR open up its MapR-FS as a giveback to the community. (Fat chance of that happening.) Rather than update HDFS to compete, they let it go. They key here is a lack of innovation on the part of Cloudera and also Horton when it came out. Horton's go to market strategy was to be the 'keeper of Apache' claiming to have a closer fork to native Apache. (There were also other flaws in their business model.)
What also killed Apache Hadoop was YARN. Flawed from the beginning it was never really fixed.
As technology evolved, Hadoop didn't.
(BTW Gnau came to the party late....)
Today, HPE has the technical advantage by combining two 'best of breed' technologies (Blue Data , MapR) So do you still call it Hadoop?
Cloudera is dying... while they tout CDP and are raising prices to become profitable... take a look at the job postings from their major clients.
All indications is the move off CDP and onto the cloud.
And while everyone is ignoring the enterprise cloud, Cloudera isn't the best choice. HDFS is still the bottleneck.
Posted Anon for all the obvious reasons and even from what I said, there are a lot of people who can guess who I am. ;-)