summaryrefslogtreecommitdiffstats
path: root/library/std/src/sync/once.rs
diff options
context:
space:
mode:
Diffstat (limited to 'library/std/src/sync/once.rs')
-rw-r--r--library/std/src/sync/once.rs312
1 files changed, 23 insertions, 289 deletions
diff --git a/library/std/src/sync/once.rs b/library/std/src/sync/once.rs
index a7feea588..0f25417d6 100644
--- a/library/std/src/sync/once.rs
+++ b/library/std/src/sync/once.rs
@@ -3,99 +3,12 @@
//! This primitive is meant to be used to run one-time initialization. An
//! example use case would be for initializing an FFI library.
-// A "once" is a relatively simple primitive, and it's also typically provided
-// by the OS as well (see `pthread_once` or `InitOnceExecuteOnce`). The OS
-// primitives, however, tend to have surprising restrictions, such as the Unix
-// one doesn't allow an argument to be passed to the function.
-//
-// As a result, we end up implementing it ourselves in the standard library.
-// This also gives us the opportunity to optimize the implementation a bit which
-// should help the fast path on call sites. Consequently, let's explain how this
-// primitive works now!
-//
-// So to recap, the guarantees of a Once are that it will call the
-// initialization closure at most once, and it will never return until the one
-// that's running has finished running. This means that we need some form of
-// blocking here while the custom callback is running at the very least.
-// Additionally, we add on the restriction of **poisoning**. Whenever an
-// initialization closure panics, the Once enters a "poisoned" state which means
-// that all future calls will immediately panic as well.
-//
-// So to implement this, one might first reach for a `Mutex`, but those cannot
-// be put into a `static`. It also gets a lot harder with poisoning to figure
-// out when the mutex needs to be deallocated because it's not after the closure
-// finishes, but after the first successful closure finishes.
-//
-// All in all, this is instead implemented with atomics and lock-free
-// operations! Whee! Each `Once` has one word of atomic state, and this state is
-// CAS'd on to determine what to do. There are four possible state of a `Once`:
-//
-// * Incomplete - no initialization has run yet, and no thread is currently
-// using the Once.
-// * Poisoned - some thread has previously attempted to initialize the Once, but
-// it panicked, so the Once is now poisoned. There are no other
-// threads currently accessing this Once.
-// * Running - some thread is currently attempting to run initialization. It may
-// succeed, so all future threads need to wait for it to finish.
-// Note that this state is accompanied with a payload, described
-// below.
-// * Complete - initialization has completed and all future calls should finish
-// immediately.
-//
-// With 4 states we need 2 bits to encode this, and we use the remaining bits
-// in the word we have allocated as a queue of threads waiting for the thread
-// responsible for entering the RUNNING state. This queue is just a linked list
-// of Waiter nodes which is monotonically increasing in size. Each node is
-// allocated on the stack, and whenever the running closure finishes it will
-// consume the entire queue and notify all waiters they should try again.
-//
-// You'll find a few more details in the implementation, but that's the gist of
-// it!
-//
-// Atomic orderings:
-// When running `Once` we deal with multiple atomics:
-// `Once.state_and_queue` and an unknown number of `Waiter.signaled`.
-// * `state_and_queue` is used (1) as a state flag, (2) for synchronizing the
-// result of the `Once`, and (3) for synchronizing `Waiter` nodes.
-// - At the end of the `call_inner` function we have to make sure the result
-// of the `Once` is acquired. So every load which can be the only one to
-// load COMPLETED must have at least Acquire ordering, which means all
-// three of them.
-// - `WaiterQueue::Drop` is the only place that may store COMPLETED, and
-// must do so with Release ordering to make the result available.
-// - `wait` inserts `Waiter` nodes as a pointer in `state_and_queue`, and
-// needs to make the nodes available with Release ordering. The load in
-// its `compare_exchange` can be Relaxed because it only has to compare
-// the atomic, not to read other data.
-// - `WaiterQueue::Drop` must see the `Waiter` nodes, so it must load
-// `state_and_queue` with Acquire ordering.
-// - There is just one store where `state_and_queue` is used only as a
-// state flag, without having to synchronize data: switching the state
-// from INCOMPLETE to RUNNING in `call_inner`. This store can be Relaxed,
-// but the read has to be Acquire because of the requirements mentioned
-// above.
-// * `Waiter.signaled` is both used as a flag, and to protect a field with
-// interior mutability in `Waiter`. `Waiter.thread` is changed in
-// `WaiterQueue::Drop` which then sets `signaled` with Release ordering.
-// After `wait` loads `signaled` with Acquire and sees it is true, it needs to
-// see the changes to drop the `Waiter` struct correctly.
-// * There is one place where the two atomics `Once.state_and_queue` and
-// `Waiter.signaled` come together, and might be reordered by the compiler or
-// processor. Because both use Acquire ordering such a reordering is not
-// allowed, so no need for SeqCst.
-
#[cfg(all(test, not(target_os = "emscripten")))]
mod tests;
-use crate::cell::Cell;
use crate::fmt;
-use crate::marker;
use crate::panic::{RefUnwindSafe, UnwindSafe};
-use crate::ptr;
-use crate::sync::atomic::{AtomicBool, AtomicPtr, Ordering};
-use crate::thread::{self, Thread};
-
-type Masked = ();
+use crate::sys_common::once as sys;
/// A synchronization primitive which can be used to run a one-time global
/// initialization. Useful for one-time initialization for FFI or related
@@ -114,19 +27,9 @@ type Masked = ();
/// ```
#[stable(feature = "rust1", since = "1.0.0")]
pub struct Once {
- // `state_and_queue` is actually a pointer to a `Waiter` with extra state
- // bits, so we add the `PhantomData` appropriately.
- state_and_queue: AtomicPtr<Masked>,
- _marker: marker::PhantomData<*const Waiter>,
+ inner: sys::Once,
}
-// The `PhantomData` of a raw pointer removes these two auto traits, but we
-// enforce both below in the implementation so this should be safe to add.
-#[stable(feature = "rust1", since = "1.0.0")]
-unsafe impl Sync for Once {}
-#[stable(feature = "rust1", since = "1.0.0")]
-unsafe impl Send for Once {}
-
#[stable(feature = "sync_once_unwind_safe", since = "1.59.0")]
impl UnwindSafe for Once {}
@@ -136,10 +39,8 @@ impl RefUnwindSafe for Once {}
/// State yielded to [`Once::call_once_force()`]’s closure parameter. The state
/// can be used to query the poison status of the [`Once`].
#[stable(feature = "once_poison", since = "1.51.0")]
-#[derive(Debug)]
pub struct OnceState {
- poisoned: bool,
- set_state_on_drop_to: Cell<*mut Masked>,
+ pub(crate) inner: sys::OnceState,
}
/// Initialization value for static [`Once`] values.
@@ -159,38 +60,6 @@ pub struct OnceState {
)]
pub const ONCE_INIT: Once = Once::new();
-// Four states that a Once can be in, encoded into the lower bits of
-// `state_and_queue` in the Once structure.
-const INCOMPLETE: usize = 0x0;
-const POISONED: usize = 0x1;
-const RUNNING: usize = 0x2;
-const COMPLETE: usize = 0x3;
-
-// Mask to learn about the state. All other bits are the queue of waiters if
-// this is in the RUNNING state.
-const STATE_MASK: usize = 0x3;
-
-// Representation of a node in the linked list of waiters, used while in the
-// RUNNING state.
-// Note: `Waiter` can't hold a mutable pointer to the next thread, because then
-// `wait` would both hand out a mutable reference to its `Waiter` node, and keep
-// a shared reference to check `signaled`. Instead we hold shared references and
-// use interior mutability.
-#[repr(align(4))] // Ensure the two lower bits are free to use as state bits.
-struct Waiter {
- thread: Cell<Option<Thread>>,
- signaled: AtomicBool,
- next: *const Waiter,
-}
-
-// Head of a linked list of waiters.
-// Every node is a struct on the stack of a waiting thread.
-// Will wake up the waiters when it gets dropped, i.e. also on panic.
-struct WaiterQueue<'a> {
- state_and_queue: &'a AtomicPtr<Masked>,
- set_state_on_drop_to: *mut Masked,
-}
-
impl Once {
/// Creates a new `Once` value.
#[inline]
@@ -198,10 +67,7 @@ impl Once {
#[rustc_const_stable(feature = "const_once_new", since = "1.32.0")]
#[must_use]
pub const fn new() -> Once {
- Once {
- state_and_queue: AtomicPtr::new(ptr::invalid_mut(INCOMPLETE)),
- _marker: marker::PhantomData,
- }
+ Once { inner: sys::Once::new() }
}
/// Performs an initialization routine once and only once. The given closure
@@ -261,6 +127,7 @@ impl Once {
/// This is similar to [poisoning with mutexes][poison].
///
/// [poison]: struct.Mutex.html#poisoning
+ #[inline]
#[stable(feature = "rust1", since = "1.0.0")]
#[track_caller]
pub fn call_once<F>(&self, f: F)
@@ -268,12 +135,12 @@ impl Once {
F: FnOnce(),
{
// Fast path check
- if self.is_completed() {
+ if self.inner.is_completed() {
return;
}
let mut f = Some(f);
- self.call_inner(false, &mut |_| f.take().unwrap()());
+ self.inner.call(false, &mut |_| f.take().unwrap()());
}
/// Performs the same function as [`call_once()`] except ignores poisoning.
@@ -320,18 +187,19 @@ impl Once {
/// // once any success happens, we stop propagating the poison
/// INIT.call_once(|| {});
/// ```
+ #[inline]
#[stable(feature = "once_poison", since = "1.51.0")]
pub fn call_once_force<F>(&self, f: F)
where
F: FnOnce(&OnceState),
{
// Fast path check
- if self.is_completed() {
+ if self.inner.is_completed() {
return;
}
let mut f = Some(f);
- self.call_inner(true, &mut |p| f.take().unwrap()(p));
+ self.inner.call(true, &mut |p| f.take().unwrap()(p));
}
/// Returns `true` if some [`call_once()`] call has completed
@@ -378,119 +246,7 @@ impl Once {
#[stable(feature = "once_is_completed", since = "1.43.0")]
#[inline]
pub fn is_completed(&self) -> bool {
- // An `Acquire` load is enough because that makes all the initialization
- // operations visible to us, and, this being a fast path, weaker
- // ordering helps with performance. This `Acquire` synchronizes with
- // `Release` operations on the slow path.
- self.state_and_queue.load(Ordering::Acquire).addr() == COMPLETE
- }
-
- // This is a non-generic function to reduce the monomorphization cost of
- // using `call_once` (this isn't exactly a trivial or small implementation).
- //
- // Additionally, this is tagged with `#[cold]` as it should indeed be cold
- // and it helps let LLVM know that calls to this function should be off the
- // fast path. Essentially, this should help generate more straight line code
- // in LLVM.
- //
- // Finally, this takes an `FnMut` instead of a `FnOnce` because there's
- // currently no way to take an `FnOnce` and call it via virtual dispatch
- // without some allocation overhead.
- #[cold]
- #[track_caller]
- fn call_inner(&self, ignore_poisoning: bool, init: &mut dyn FnMut(&OnceState)) {
- let mut state_and_queue = self.state_and_queue.load(Ordering::Acquire);
- loop {
- match state_and_queue.addr() {
- COMPLETE => break,
- POISONED if !ignore_poisoning => {
- // Panic to propagate the poison.
- panic!("Once instance has previously been poisoned");
- }
- POISONED | INCOMPLETE => {
- // Try to register this thread as the one RUNNING.
- let exchange_result = self.state_and_queue.compare_exchange(
- state_and_queue,
- ptr::invalid_mut(RUNNING),
- Ordering::Acquire,
- Ordering::Acquire,
- );
- if let Err(old) = exchange_result {
- state_and_queue = old;
- continue;
- }
- // `waiter_queue` will manage other waiting threads, and
- // wake them up on drop.
- let mut waiter_queue = WaiterQueue {
- state_and_queue: &self.state_and_queue,
- set_state_on_drop_to: ptr::invalid_mut(POISONED),
- };
- // Run the initialization function, letting it know if we're
- // poisoned or not.
- let init_state = OnceState {
- poisoned: state_and_queue.addr() == POISONED,
- set_state_on_drop_to: Cell::new(ptr::invalid_mut(COMPLETE)),
- };
- init(&init_state);
- waiter_queue.set_state_on_drop_to = init_state.set_state_on_drop_to.get();
- break;
- }
- _ => {
- // All other values must be RUNNING with possibly a
- // pointer to the waiter queue in the more significant bits.
- assert!(state_and_queue.addr() & STATE_MASK == RUNNING);
- wait(&self.state_and_queue, state_and_queue);
- state_and_queue = self.state_and_queue.load(Ordering::Acquire);
- }
- }
- }
- }
-}
-
-fn wait(state_and_queue: &AtomicPtr<Masked>, mut current_state: *mut Masked) {
- // Note: the following code was carefully written to avoid creating a
- // mutable reference to `node` that gets aliased.
- loop {
- // Don't queue this thread if the status is no longer running,
- // otherwise we will not be woken up.
- if current_state.addr() & STATE_MASK != RUNNING {
- return;
- }
-
- // Create the node for our current thread.
- let node = Waiter {
- thread: Cell::new(Some(thread::current())),
- signaled: AtomicBool::new(false),
- next: current_state.with_addr(current_state.addr() & !STATE_MASK) as *const Waiter,
- };
- let me = &node as *const Waiter as *const Masked as *mut Masked;
-
- // Try to slide in the node at the head of the linked list, making sure
- // that another thread didn't just replace the head of the linked list.
- let exchange_result = state_and_queue.compare_exchange(
- current_state,
- me.with_addr(me.addr() | RUNNING),
- Ordering::Release,
- Ordering::Relaxed,
- );
- if let Err(old) = exchange_result {
- current_state = old;
- continue;
- }
-
- // We have enqueued ourselves, now lets wait.
- // It is important not to return before being signaled, otherwise we
- // would drop our `Waiter` node and leave a hole in the linked list
- // (and a dangling reference). Guard against spurious wakeups by
- // reparking ourselves until we are signaled.
- while !node.signaled.load(Ordering::Acquire) {
- // If the managing thread happens to signal and unpark us before we
- // can park ourselves, the result could be this thread never gets
- // unparked. Luckily `park` comes with the guarantee that if it got
- // an `unpark` just before on an unparked thread it does not park.
- thread::park();
- }
- break;
+ self.inner.is_completed()
}
}
@@ -501,37 +257,6 @@ impl fmt::Debug for Once {
}
}
-impl Drop for WaiterQueue<'_> {
- fn drop(&mut self) {
- // Swap out our state with however we finished.
- let state_and_queue =
- self.state_and_queue.swap(self.set_state_on_drop_to, Ordering::AcqRel);
-
- // We should only ever see an old state which was RUNNING.
- assert_eq!(state_and_queue.addr() & STATE_MASK, RUNNING);
-
- // Walk the entire linked list of waiters and wake them up (in lifo
- // order, last to register is first to wake up).
- unsafe {
- // Right after setting `node.signaled = true` the other thread may
- // free `node` if there happens to be has a spurious wakeup.
- // So we have to take out the `thread` field and copy the pointer to
- // `next` first.
- let mut queue =
- state_and_queue.with_addr(state_and_queue.addr() & !STATE_MASK) as *const Waiter;
- while !queue.is_null() {
- let next = (*queue).next;
- let thread = (*queue).thread.take().unwrap();
- (*queue).signaled.store(true, Ordering::Release);
- // ^- FIXME (maybe): This is another case of issue #55005
- // `store()` has a potentially dangling ref to `signaled`.
- queue = next;
- thread.unpark();
- }
- }
- }
-}
-
impl OnceState {
/// Returns `true` if the associated [`Once`] was poisoned prior to the
/// invocation of the closure passed to [`Once::call_once_force()`].
@@ -568,13 +293,22 @@ impl OnceState {
/// assert!(!state.is_poisoned());
/// });
#[stable(feature = "once_poison", since = "1.51.0")]
+ #[inline]
pub fn is_poisoned(&self) -> bool {
- self.poisoned
+ self.inner.is_poisoned()
}
/// Poison the associated [`Once`] without explicitly panicking.
- // NOTE: This is currently only exposed for the `lazy` module
+ // NOTE: This is currently only exposed for `OnceLock`.
+ #[inline]
pub(crate) fn poison(&self) {
- self.set_state_on_drop_to.set(ptr::invalid_mut(POISONED));
+ self.inner.poison();
+ }
+}
+
+#[stable(feature = "std_debug", since = "1.16.0")]
+impl fmt::Debug for OnceState {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("OnceState").field("poisoned", &self.is_poisoned()).finish()
}
}
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-19 19:41:24 +0000