blk-cgroup: protect blkgs with blkcg_mutex#1033
Open
blktests-ci[bot] wants to merge 17 commits into
Open
Conversation
added 17 commits
July 4, 2026 20:48
bio_set_dev() is about to become explicitly sleepable because it can associate the bio with a blkg for the destination queue. NVMe failover can run from request completion context, and nvme_failover_req() also holds head->requeue_lock with interrupts disabled while it steals bios from the failed request. Calling bio_set_dev() there is not safe once the helper is allowed to sleep. The requeue lock only protects head->requeue_list. Keep the list manipulation under that lock, but defer retargeting to nvme_requeue_work(), which already drains the list from process context before resubmitting each bio. The bios remain private to the requeue list until the worker pops them, so moving the device switch there preserves the existing retry flow while avoiding a sleepable helper in completion context. Signed-off-by: Yu Kuai <yukuai@fygo.io>
bio_set_dev() is about to become explicitly sleepable because it can associate the bio with a blkg for the destination queue. pool_map() calls bio_set_dev() while holding pool->lock with interrupts disabled, which would be invalid once bio_set_dev() may sleep. The lock is not needed in this map path. The pool target is a singleton mapping and pool_map() only reads pt->data_dev, which is a target-private device reference acquired during construction and released during target destruction. It does not inspect or modify pool state protected by pool->lock. Remove the lock so the remap stays in the normal sleepable DM map context while the data device pointer remains stable for the table lifetime. Signed-off-by: Yu Kuai <yukuai@fygo.io>
bio_set_dev() is about to become explicitly sleepable. It currently updates the bio's target device and then associates the bio with the destination queue's blkcg state. After blkcg lookup/creation is moved under the queue's blkcg_mutex, that association may take blkcg_mutex and allocate a new blkg. Callers therefore must not invoke bio_set_dev() from atomic or otherwise non-sleepable sections. snapshot_map() has several remap decisions inside dm_exception_table_lock(), which nests the completed and pending exception hash-table spinlocks. Those locks protect the lookup result, pending-exception insertion, pe->started, and the pending bio lists until the bio has either been returned to DM core or queued on the pending exception. Dropping the locks just to call bio_set_dev() would require revalidating the exception state and preserving the pending-list ordering rules; calling a sleepable bio_set_dev() while holding the spinlocks is not allowed either. Split out snapshot_bio_set_dev() for these locked remap decisions. It only performs the non-sleeping part of bio_set_dev(): clear BIO_REMAPPED, clear BIO_BPS_THROTTLED when the bdev changes, and update bi_bdev. It deliberately does not associate the bio with a blkg while snapshot locks are held. This does not lose blkcg attribution for the normal DM_MAPIO_REMAPPED case. After the target returns, DM core submits the mapped bio through dm_submit_bio_remap(), and that helper clones the blkg association from the original bio in the normal submission context. Some snapshot bios are not submitted by DM core immediately. Writes waiting for a pending exception and bios queued during snapshot merge are kept on snapshot-owned lists and submitted later after copy or merge completion. Once bio_set_dev() is no longer used in the locked path, these delayed bios also need their blkcg association restored at submission time. Submit those bios through dm_submit_bio_remap() instead of submit_bio_noacct() so the association is cloned from the original bio after the snapshot locks have been released. Signed-off-by: Yu Kuai <yukuai@fygo.io>
Throttle currently uses queue_lock for both blkcg topology and its own runtime state. This blocks moving blkg topology protection to blkcg_mutex cleanly. Add a throttle-private spinlock and use it for throttle service queues, pending timers, runtime counters and config updates. Keep queue_lock only where the current intermediate code still walks blkcg topology. Signed-off-by: Yu Kuai <yukuai@fygo.io>
Add bio_alloc_atomic() for callers that need a GFP_ATOMIC bio from the default bio set but cannot safely pass a bdev during allocation. The helper returns an unattached bio, leaving callers to set bi_bdev and attach blkcg state explicitly before submission. Use the helper for virtio-pmem flush child bios and OCFS2 heartbeat I/O. Both allocate bios from atomic paths and must avoid creating missing blkgs once blkg creation is protected by q->blkcg_mutex. virtio-pmem clones the parent bio's blkg association; OCFS2 binds heartbeat I/O to the root blkg. Signed-off-by: Yu Kuai <yukuai@fygo.io>
Allow bio association helpers to be called from non-blocking paths by returning whether the association succeeded and by taking a nowait argument. The normal callers pass nowait=false and keep the existing behavior of creating missing blkgs. For nowait=true, the helper only succeeds when the needed blkg already exists. This lets callers set or clone a bio's bdev without entering the sleepable missing-blkg creation path. Signed-off-by: Yu Kuai <yukuai@fygo.io>
bio_alloc_clone(), bio_init_clone(), and bio_alloc_bioset() can be called with non-blocking GFP masks. Passing a bdev into bio initialization may need to associate blkcg state and, after missing blkg creation is serialized by q->blkcg_mutex, that association can sleep. Keep the generic block layer simple by letting bio_alloc_bioset() handle this case directly. Non-blocking allocations initialize the bio without a bdev, set the bdev fields, and associate the blkg with nowait=true. If the needed blkg is missing and would have to be created, allocation fails normally so the caller can retry from a blocking context. Blocking callers keep the existing allocation-time association behavior. Signed-off-by: Yu Kuai <yukuai@fygo.io>
cached_dev_cache_miss() allocates cache_bio with GFP_NOWAIT. Passing a bdev to bio_alloc_bioset() can attach blkcg state and sleep to create a missing blkg after blkg lookup is protected by q->blkcg_mutex. Use the nowait bio allocation/association path. If the cache bio needs a missing blkg to be created, fail the association and fall back to the existing miss submission path. journal_write_unlocked() also resets journal bios while holding the journal spinlock. Reset those bios without a bdev, set bi_bdev while still under the lock, and associate blkcg after dropping the lock. Signed-off-by: Yu Kuai <yukuai@fygo.io>
dm-bufio allocates a bio with bio_kmalloc(GFP_NOWAIT) and then initializes it with the target bdev. That initialization can attach blkcg state and sleep to create a missing blkg once blkg lookup is protected by q->blkcg_mutex. Initialize the bio without a bdev, set the bdev fields, and associate blkcg with nowait=true. Fall back to dm_io if a missing blkg would need to be created. Signed-off-by: Yu Kuai <yukuai@fygo.io>
dm-pcache may preallocate backing requests with GFP_NOWAIT and initialize the embedded bio with bio_init_clone(). Non-blocking clone initialization can now fail if cloning the blkg association would need to create a blkg. Check the return value and free the preallocated request on failure so the existing caller can retry through its GFP_NOIO preallocation path. Signed-off-by: Yu Kuai <yukuai@fygo.io>
blk_alloc_discard_bio() and blk_rq_map_bio_alloc() can be used with non-blocking GFP masks. Their bio allocation now handles bdev association in nowait mode, so the helpers can pass the target bdev directly and avoid local open-coded association paths. The discard helper can also be reached from io_uring with GFP_NOWAIT. Keep its long-loop cond_resched() only for blocking callers. Signed-off-by: Yu Kuai <yukuai@fygo.io>
DM allocates normal NOWAIT target clones with GFP_NOWAIT. Targets that set needs_bio_set_dev can therefore make alloc_tio() associate blkcg state from a non-blocking allocation path, which may sleep while creating a missing blkg after blkg lookup is protected by q->blkcg_mutex. Set the default bdev without blkcg association first, then associate blkcg with nowait=true for non-blocking allocations. If a blkg would need creating, fail the NOWAIT allocation with BLK_STS_AGAIN. Targets that advertise DM_TARGET_NOWAIT may also remap bios in their map functions. Those remaps update only the bdev for NOWAIT bios, then DM submission clones the original bio's blkg association with nowait=true before lower submission. If that would need to sleep, complete the clone with BLK_STS_AGAIN. Signed-off-by: Yu Kuai <yukuai@fygo.io>
bfq_bio_bfqg() is called while bfqd->lock is held from the merge and request insertion paths. It walks bio->bi_blkg and its parent chain to find the closest online BFQ group, and also updates bio->bi_blkg when the original association points at an offline or otherwise unusable blkg. After missing blkg creation is protected by q->blkcg_mutex, bio_associate_blkg_from_css() can sleep on lookup misses. BFQ must not call it while holding bfqd->lock. The blkg BFQ wants is already known from the existing bio->bi_blkg ancestry walk, so update bio->bi_blkg by swapping references to that existing blkg directly instead of looking it up again by css. Signed-off-by: Yu Kuai <yukuai@fygo.io>
queue_lock is still needed by block core users, but blkcg no longer needs it for blkg topology now that throttle runtime state has a private lock. Move queue-local blkg synchronization to q->blkcg_mutex. Hold it while looking up, creating and destroying blkgs, while preparing and undoing configuration, and while activating or deactivating policies. Update the BFQ, iocost, iolatency and throttle paths which walk q->blkg_list or access per-blkg policy state to use the same lock. blkcg->lock still protects blkcg-local radix tree and list updates. Some lookups under blkcg_mutex can race with blkcg updates done for other queues, so keep those lookups in RCU read-side critical sections. In particular, protect the parent lookup in blkg_create() and the parent walk in blkg_lookup_create(). Nowait bio association remains non-blocking after the lock conversion: if RCU lookup misses, preemptible task-context callers can try q->blkcg_mutex and create the missing blkg without sleeping. Atomic callers, contended mutexes, or allocation failures keep the fail-fast behavior. Signed-off-by: Yu Kuai <yukuai@fygo.io>
blkg creation is now serialized by q->blkcg_mutex and no longer runs under q->queue_lock. The radix tree is initialized with GFP_NOWAIT, so radix_tree_insert() cannot sleep while blkcg->lock is held and the old preload dance is no longer needed. Remove the preload calls and the associated unwind path. Signed-off-by: Yu Kuai <yukuai@fygo.io>
After radix tree preloading is gone, callers no longer need to allocate a blkg before entering blkg_create(). Move allocation into blkg_create() and pass the desired GFP mask instead. Use GFP_NOIO for runtime and config blkg creation so slow paths can sleep without recursing into IO reclaim, keep GFP_KERNEL for root blkg setup, and use GFP_ATOMIC when nowait bio association creates a missing blkg after a successful q->blkcg_mutex trylock. Signed-off-by: Yu Kuai <yukuai@fygo.io>
blkg_conf_prep() open-codes the same parent walk and blkg creation that blkg_lookup_create() already performs. Make blkg_lookup_create() report whether the target blkg was created or found while still returning the closest existing blkg on failure, then have blkg_conf_prep() use the helper and treat errors as config failures. This keeps the bio association path's closest-blkg fallback and removes the duplicate config path loop. Signed-off-by: Yu Kuai <yukuai@fygo.io>
Author
|
Upstream branch: 87320be |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: blk-cgroup: protect blkgs with blkcg_mutex
version: 1
url: https://patchwork.kernel.org/project/linux-block/list/?series=1121611