MM Change Slated For Linux 7.4 Yields +22904539.81% In One Metric, More Modest Wins In Others
([Linux Kernel] 53 Minutes Ago
Sheaves)
- Reference: 0001660265
- News link: https://www.phoronix.com/news/Linux-7.4-Refill-Sheaves-Barn
- Source link:
Queued this week into the slab/for-next Git branch ahead of next month's Linux 7.4 merge window is a memory management change for refilling pre-filled sheaves from the barn. In some cases this yields wild improvements to specific synthetic benchmarks.
[1]Sheaves are the per-CPU, array-based caching layer for the Linux kernel's memory allocator. Sheaves have seen a lot of work since their introduction to the kernel last year and even more optimization work is on the way for Linux 7.4.
Hao Li explained of the memory management code change for refilling prefilled sheaves from the barn:
"Currently, when the prefill API refills a non-full sheaf, it takes the objects from partial slabs and never from the full sheaves in the barn, so once the barn's full list becomes saturated, it stays saturated.
For objects freed via kfree_rcu(), every RCU sheaf then has to be flushed to slabs because the barn's full list has no room.
To fix this, let the sheaf refill from the barn first, and introduce a partial sheaf in the barn, which holds the leftover objects.
Only the prefill path needs the partial sheaf. The generic allocation path (__pcs_replace_empty_main()) exchanges an empty sheaf for a full one from the barn, so nothing is left over. The prefill path refills a sheaf that is not necessarily empty, so taking objects from the barn usually leaves leftovers, and the partial sheaf is where they are kept. refill_sheaf() only takes objects from partial slabs and never involves the partial sheaf.
The sheaf is refilled by copying objects from the partial sheaf, and then from a full sheaf taken from the barn if it is still not full. A sheaf that objects were copied from stays in the barn as the partial sheaf if it still holds objects, or goes on the empty list if it is empty. The sheaf being refilled is never replaced.
The gain comes from two sides: every full sheaf taken out makes room on the barn's full list for a future RCU sheaf, and refilling from the barn is cheaper than refilling from partial slabs under list_lock."
The benchmarks with [2]the patch show an overall 24% improvement to a will-it-scale mmap1 test. The impact in the barn_get performance is +22904539.81%, barn_put is +21422544.07%, and the other metrics tend to be double-digit percentage improvements over the baseline.
Quite an impact out of 81 lines of new code. [3]The patch is destined for the Linux 7.4 merge window happening in late October.
[1] https://www.phoronix.com/search/sheaves
[2] https://git.kernel.org/pub/scm/linux/kernel/git/mm/slab.git/commit/?h=slab/for-next&id=1b87ea06a34b213e45e3e0c4effd5043f15046ed
[3] https://git.kernel.org/pub/scm/linux/kernel/git/mm/slab.git/commit/?h=slab/for-next&id=1b87ea06a34b213e45e3e0c4effd5043f15046ed
[1]Sheaves are the per-CPU, array-based caching layer for the Linux kernel's memory allocator. Sheaves have seen a lot of work since their introduction to the kernel last year and even more optimization work is on the way for Linux 7.4.
Hao Li explained of the memory management code change for refilling prefilled sheaves from the barn:
"Currently, when the prefill API refills a non-full sheaf, it takes the objects from partial slabs and never from the full sheaves in the barn, so once the barn's full list becomes saturated, it stays saturated.
For objects freed via kfree_rcu(), every RCU sheaf then has to be flushed to slabs because the barn's full list has no room.
To fix this, let the sheaf refill from the barn first, and introduce a partial sheaf in the barn, which holds the leftover objects.
Only the prefill path needs the partial sheaf. The generic allocation path (__pcs_replace_empty_main()) exchanges an empty sheaf for a full one from the barn, so nothing is left over. The prefill path refills a sheaf that is not necessarily empty, so taking objects from the barn usually leaves leftovers, and the partial sheaf is where they are kept. refill_sheaf() only takes objects from partial slabs and never involves the partial sheaf.
The sheaf is refilled by copying objects from the partial sheaf, and then from a full sheaf taken from the barn if it is still not full. A sheaf that objects were copied from stays in the barn as the partial sheaf if it still holds objects, or goes on the empty list if it is empty. The sheaf being refilled is never replaced.
The gain comes from two sides: every full sheaf taken out makes room on the barn's full list for a future RCU sheaf, and refilling from the barn is cheaper than refilling from partial slabs under list_lock."
The benchmarks with [2]the patch show an overall 24% improvement to a will-it-scale mmap1 test. The impact in the barn_get performance is +22904539.81%, barn_put is +21422544.07%, and the other metrics tend to be double-digit percentage improvements over the baseline.
Quite an impact out of 81 lines of new code. [3]The patch is destined for the Linux 7.4 merge window happening in late October.
[1] https://www.phoronix.com/search/sheaves
[2] https://git.kernel.org/pub/scm/linux/kernel/git/mm/slab.git/commit/?h=slab/for-next&id=1b87ea06a34b213e45e3e0c4effd5043f15046ed
[3] https://git.kernel.org/pub/scm/linux/kernel/git/mm/slab.git/commit/?h=slab/for-next&id=1b87ea06a34b213e45e3e0c4effd5043f15046ed