In the Linux kernel, the following vulnerability has been resolved:
ceph: do not repeat ceph_trim_dentries() if no progress possible
ceph_cap_reclaim_work() re-queues itself for as long as
ceph_trim_dentries() returns -EAGAIN, which happens whenever a lease
walk exhausts its `nr_to_scan` budget. This creates a busy loop that
consumes CPU without making any progress when there is nothing to
reclaim: with no cap pressure (`count==0`) and every scanned lease
still valid, each pass runs the full scan budget down to zero and
returns `-EAGAIN`, only to be queued again immediately.
The dir-lease walk made this worse. When `expire_dir_lease` is
`false` (i.e. we have no intention of reclaiming dir leases),
__dir_lease_check() returned `TOUCH` for every valid lease. `TOUCH`
moves the dentry to the tail of the list and resets `di->time` via
__dentry_dir_lease_touch(), so a walk over N valid leases pointlessly
rewrote the list, refreshed the timestamps (preventing them from ever
aging out) and always drained `nr_to_scan`, guaranteeing the `-EAGAIN`
requeue.
Fix this in three steps:
– Return `KEEP` instead of `TOUCH` when `expire_dir_lease` is
`false`. If we are not going to reclaim the lease, leave it in
place instead of churning the list and resetting its timestamp; the
walk then terminates naturally (or via `STOP` at the first fresh
lease).
– Only return `-EAGAIN` from the first (dentry-lease) walk when something
was actually freed. A full batch that frees nothing means retrying
the same list immediately is futile; fall through to the dir-lease
walk instead.
– After both walks, bail out with success (0) when nothing was freed
and there is no cap pressure (`count==0`). There is no reason to
keep retrying when we are not over the cap limit and made no
progress.
Under real cap pressure (`count>0`) the reclaim path is unchanged and
still retries via `-EAGAIN`.
Without this patch, I saw 500 ceph_trim_dentries() calls per second on
our web servers. This is very visible in `/proc/lock_stat` (5 minute
capture):
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 126180 128218 0.04 8063.44 15986965.20 124.69 1573354 5296812 0.04 8291.28 74164526.48 14.00
———————–
&mdsc->dentry_list_lock 111736 [] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 2631 [] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 3878 [] __ceph_dentry_lease_touch+0x5c/0xa8
&mdsc->dentry_list_lock 9973 [] __dentry_lease_unlist+0x50/0xa0
———————–
&mdsc->dentry_list_lock 123621 [] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 1822 [] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 2720 [] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 55 [] __ceph_dentry_lease_touch+0x5c/0xa8
With this patch:
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 1203 1215 0.16 408.88 33082.88 27.23 4320501 7357389 0.04 500.64 1961578.00 0.27
———————–
&mdsc->dentry_list_lock 1029 [] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 1
—truncated—
