If you want to connect a GPU direct to SSDs for AI training, this could be it
(2022/03/14)
- Reference: 1647289435
- News link: https://www.theregister.co.uk/2022/03/14/nvidia_gpu_data/
- Source link:
Nvidia, IBM, and university collaborators have a developed a system architecture they say will provide faster access to data storage for GPU-driven applications, including machine-learning training.
Dubbed Big accelerator Memory, aka BaM, this approach reduces the reliance of graphics processors and other hardware accelerators on general-purpose chips when it comes to accessing storage, which should improve capacity and performance.
"The goal of BaM is to extend GPU memory capacity and enhance the effective storage access bandwidth while providing high-level abstractions for the GPU threads to easily make on-demand, fine-grain access to massive data structures in the extended memory hierarchy," reads a [1]paper written by the team describing their design.
[2]
The BaM concept is a step by Nvidia to move conventional CPU-centric tasks on to GPU cores. Rather than relying on things like virtual address translation, page-fault-based on-demand loading of data, and other traditional CPU-centric mechanisms for handling large amounts of information, BaM instead provides software and a hardware architecture that allows Nvidia GPUs to fetch data direct from memory and storage without needing a CPU core to orchestrate it.
[3]
[4]
BaM has two main parts: a software-managed cache of GPU memory; and a software library for making requests from GPU threads to NVMe SSDs to request data. Essentially, the job of moving information between storage and GPU is handled by threads on the GPU cores, using RDMA, PCIe, and a custom Linux kernel driver so that SSDs can read and write GPU memory directly. This means algorithms running on the GPU to perform AI tasks, and other intensive workloads, can get at the data they need quickly, and in a way that's optimized for their data access patterns.
[5]
Diagrams from the paper comparing the traditional CPU-centric approach to accessing storage (a) to the GPU-led BaM approach (b) and how said GPU would be physically wired to the storage devices (c). Source: Qureshi et al. Click to enlarge
The researchers tested a prototype Linux-based BaM system using off-the-shelf GPUs and NVMe SSDs to prove it is a viable alternative to today's approach of having the host processor direct everything. Storage access can be parallelized, and I/O bandwidth is more efficiently used to boost application performance, we're told.
“A CPU-centric strategy causes excessive CPU-GPU synchronization overhead and/or I/O traffic amplification, diminishing the effective storage bandwidth for emerging applications with fine-grain data-dependent access patterns like graph and data analytics, recommender systems, and graph neural networks,” the researchers stated in their paper this month.
[6]Why Nvidia sees a future in software and services: Recurring revenue
[7]Nvidia, Apple noticeably absent from Intel-led chiplet interconnect collaboration
[8]Intel reveals GPU roadmap with hybrid integrated discrete graphics
[9]2021 in storage: We waited for a flash price revolution that never came. But about creativity? We can't complain
“With the software cache, BaM does not rely on virtual memory address translation and thus does not suffer from serialization events like TLB misses,” the authors, including Nvidia's chief scientist Bill Dally, who previously led Stanford's computer science department, noted.
"BaM provides a user-level library of highly concurrent NVMe submission/completion queues in GPU memory that enables GPU threads whose on-demand accesses miss from the software cache to make storage accesses in a high-throughput manner," they continued. "This user-level approach incurs little software overhead for each storage access and supports a high-degree of thread-level parallelism."
The team plans to open-source hardware and software optimization details for others to build such systems. ®
Get our [10]Tech Resources
[1] https://arxiv.org/abs/2203.04910
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yi-JGjRuzALmXxEKdpxeNQAAANQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yi-JGjRuzALmXxEKdpxeNQAAANQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yi-JGjRuzALmXxEKdpxeNQAAANQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://regmedia.co.uk/2022/03/11/diagram_bam_versus_cpu.jpg
[6] https://www.theregister.com/2022/03/10/nvidia_software_services/
[7] https://www.theregister.com/2022/03/03/nvidia_apple_intel_ucie/
[8] https://www.theregister.com/2022/02/17/intel_gpu_plans/
[9] https://www.theregister.com/2021/12/30/storage_2021_news_analysis/
[10] https://whitepapers.theregister.com/
Dubbed Big accelerator Memory, aka BaM, this approach reduces the reliance of graphics processors and other hardware accelerators on general-purpose chips when it comes to accessing storage, which should improve capacity and performance.
"The goal of BaM is to extend GPU memory capacity and enhance the effective storage access bandwidth while providing high-level abstractions for the GPU threads to easily make on-demand, fine-grain access to massive data structures in the extended memory hierarchy," reads a [1]paper written by the team describing their design.
[2]
The BaM concept is a step by Nvidia to move conventional CPU-centric tasks on to GPU cores. Rather than relying on things like virtual address translation, page-fault-based on-demand loading of data, and other traditional CPU-centric mechanisms for handling large amounts of information, BaM instead provides software and a hardware architecture that allows Nvidia GPUs to fetch data direct from memory and storage without needing a CPU core to orchestrate it.
[3]
[4]
BaM has two main parts: a software-managed cache of GPU memory; and a software library for making requests from GPU threads to NVMe SSDs to request data. Essentially, the job of moving information between storage and GPU is handled by threads on the GPU cores, using RDMA, PCIe, and a custom Linux kernel driver so that SSDs can read and write GPU memory directly. This means algorithms running on the GPU to perform AI tasks, and other intensive workloads, can get at the data they need quickly, and in a way that's optimized for their data access patterns.
[5]
Diagrams from the paper comparing the traditional CPU-centric approach to accessing storage (a) to the GPU-led BaM approach (b) and how said GPU would be physically wired to the storage devices (c). Source: Qureshi et al. Click to enlarge
The researchers tested a prototype Linux-based BaM system using off-the-shelf GPUs and NVMe SSDs to prove it is a viable alternative to today's approach of having the host processor direct everything. Storage access can be parallelized, and I/O bandwidth is more efficiently used to boost application performance, we're told.
“A CPU-centric strategy causes excessive CPU-GPU synchronization overhead and/or I/O traffic amplification, diminishing the effective storage bandwidth for emerging applications with fine-grain data-dependent access patterns like graph and data analytics, recommender systems, and graph neural networks,” the researchers stated in their paper this month.
[6]Why Nvidia sees a future in software and services: Recurring revenue
[7]Nvidia, Apple noticeably absent from Intel-led chiplet interconnect collaboration
[8]Intel reveals GPU roadmap with hybrid integrated discrete graphics
[9]2021 in storage: We waited for a flash price revolution that never came. But about creativity? We can't complain
“With the software cache, BaM does not rely on virtual memory address translation and thus does not suffer from serialization events like TLB misses,” the authors, including Nvidia's chief scientist Bill Dally, who previously led Stanford's computer science department, noted.
"BaM provides a user-level library of highly concurrent NVMe submission/completion queues in GPU memory that enables GPU threads whose on-demand accesses miss from the software cache to make storage accesses in a high-throughput manner," they continued. "This user-level approach incurs little software overhead for each storage access and supports a high-degree of thread-level parallelism."
The team plans to open-source hardware and software optimization details for others to build such systems. ®
Get our [10]Tech Resources
[1] https://arxiv.org/abs/2203.04910
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yi-JGjRuzALmXxEKdpxeNQAAANQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yi-JGjRuzALmXxEKdpxeNQAAANQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yi-JGjRuzALmXxEKdpxeNQAAANQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://regmedia.co.uk/2022/03/11/diagram_bam_versus_cpu.jpg
[6] https://www.theregister.com/2022/03/10/nvidia_software_services/
[7] https://www.theregister.com/2022/03/03/nvidia_apple_intel_ucie/
[8] https://www.theregister.com/2022/02/17/intel_gpu_plans/
[9] https://www.theregister.com/2021/12/30/storage_2021_news_analysis/
[10] https://whitepapers.theregister.com/
Bad Hombre
There were no AI training benchmark measurements done in the paper that I can find, only data analytics. You should change the title of the article to "... for analytics" to reflect that.
Steve Medway
Looks a hell of a lot like how the Radeon Pro SSG with it's onboard 2tb ssd works.
So it looks to me nvidia are almost six years behind AMD on this idea...
Whats old is new
So they have reinvented a form of DMA (Direct Memory Access), just with a different endpoint attached to it ?
Wasn't this problem solved in the 1970's ?