/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- * vim: set ts=8 sts=2 et sw=2 tw=80: * This Source Code Form is subject to the terms of the Mozilla Public * License, v. 2.0. If a copy of the MPL was not distributed with this * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ /* * [SMDOC] Garbage Collector * * This code implements an incremental mark-and-sweep garbage collector, with * most sweeping carried out in the background on a parallel thread. * * Full vs. zone GC * ---------------- * * The collector can collect all zones at once, or a subset. These types of * collection are referred to as a full GC and a zone GC respectively. * * It is possible for an incremental collection that started out as a full GC to * become a zone GC if new zones are created during the course of the * collection. * * Incremental collection * ---------------------- * * For a collection to be carried out incrementally the following conditions * must be met: * - the collection must be run by calling js::GCSlice() rather than js::GC() * - the GC parameter JSGC_INCREMENTAL_GC_ENABLED must be true. * * The last condition is an engine-internal mechanism to ensure that incremental * collection is not carried out without the correct barriers being implemented. * For more information see 'Incremental marking' below. * * If the collection is not incremental, all foreground activity happens inside * a single call to GC() or GCSlice(). However the collection is not complete * until the background sweeping activity has finished. * * An incremental collection proceeds as a series of slices, interleaved with * mutator activity, i.e. running JavaScript code. Slices are limited by a time * budget. The slice finishes as soon as possible after the requested time has * passed. * * Collector states * ---------------- * * The collector proceeds through the following states, the current state being * held in JSRuntime::gcIncrementalState: * * - Prepare - unmarks GC things, discards JIT code and other setup * - MarkRoots - marks the stack and other roots * - Mark - incrementally marks reachable things * - Sweep - sweeps zones in groups and continues marking unswept zones * - Finalize - performs background finalization, concurrent with mutator * - Compact - incrementally compacts by zone * - Decommit - performs background decommit and chunk removal * * Roots are marked in the first MarkRoots slice; this is the start of the GC * proper. The following states can take place over one or more slices. * * In other words an incremental collection proceeds like this: * * Slice 1: Prepare: Starts background task to unmark GC things * * ... JS code runs, background unmarking finishes ... * * Slice 2: MarkRoots: Roots are pushed onto the mark stack. * Mark: The mark stack is processed by popping an element, * marking it, and pushing its children. * * ... JS code runs ... * * Slice 3: Mark: More mark stack processing. * * ... JS code runs ... * * Slice n-1: Mark: More mark stack processing. * * ... JS code runs ... * * Slice n: Mark: Mark stack is completely drained. * Sweep: Select first group of zones to sweep and sweep them. * * ... JS code runs ... * * Slice n+1: Sweep: Mark objects in unswept zones that were newly * identified as alive (see below). Then sweep more zone * sweep groups. * * ... JS code runs ... * * Slice n+2: Sweep: Mark objects in unswept zones that were newly * identified as alive. Then sweep more zones. * * ... JS code runs ... * * Slice m: Sweep: Sweeping is finished, and background sweeping * started on the helper thread. * * ... JS code runs, remaining sweeping done on background thread ... * * When background sweeping finishes the GC is complete. * * Incremental marking * ------------------- * * Incremental collection requires close collaboration with the mutator (i.e., * JS code) to guarantee correctness. * * - During an incremental GC, if a memory location (except a root) is written * to, then the value it previously held must be marked. Write barriers * ensure this. * * - Any object that is allocated during incremental GC must start out marked. * * - Roots are marked in the first slice and hence don't need write barriers. * Roots are things like the C stack and the VM stack. * * The problem that write barriers solve is that between slices the mutator can * change the object graph. We must ensure that it cannot do this in such a way * that makes us fail to mark a reachable object (marking an unreachable object * is tolerable). * * We use a snapshot-at-the-beginning algorithm to do this. This means that we * promise to mark at least everything that is reachable at the beginning of * collection. To implement it we mark the old contents of every non-root memory * location written to by the mutator while the collection is in progress, using * write barriers. This is described in gc/Barrier.h. * * Incremental sweeping * -------------------- * * Sweeping is difficult to do incrementally because object finalizers must be * run at the start of sweeping, before any mutator code runs. The reason is * that some objects use their finalizers to remove themselves from caches. If * mutator code was allowed to run after the start of sweeping, it could observe * the state of the cache and create a new reference to an object that was just * about to be destroyed. * * Sweeping all finalizable objects in one go would introduce long pauses, so * instead sweeping broken up into groups of zones. Zones which are not yet * being swept are still marked, so the issue above does not apply. * * The order of sweeping is restricted by cross compartment pointers - for * example say that object |a| from zone A points to object |b| in zone B and * neither object was marked when we transitioned to the Sweep phase. Imagine we * sweep B first and then return to the mutator. It's possible that the mutator * could cause |a| to become alive through a read barrier (perhaps it was a * shape that was accessed via a shape table). Then we would need to mark |b|, * which |a| points to, but |b| has already been swept. * * So if there is such a pointer then marking of zone B must not finish before * marking of zone A. Pointers which form a cycle between zones therefore * restrict those zones to being swept at the same time, and these are found * using Tarjan's algorithm for finding the strongly connected components of a * graph. * * GC things without finalizers, and things with finalizers that are able to run * in the background, are swept on the background thread. This accounts for most * of the sweeping work. * * Reset * ----- * * During incremental collection it is possible, although unlikely, for * conditions to change such that incremental collection is no longer safe. In * this case, the collection is 'reset' by resetIncrementalGC(). If we are in * the mark state, this just stops marking, but if we have started sweeping * already, we continue non-incrementally until we have swept the current sweep * group. Following a reset, a new collection is started. * * Compacting GC * ------------- * * Compacting GC happens at the end of a major GC as part of the last slice. * There are three parts: * * - Arenas are selected for compaction. * - The contents of those arenas are moved to new arenas. * - All references to moved things are updated. * * Collecting Atoms * ---------------- * * Atoms are collected differently from other GC things. They are contained in * a special zone and things in other zones may have pointers to them that are * not recorded in the cross compartment pointer map. Each zone holds a bitmap * with the atoms it might be keeping alive, and atoms are only collected if * they are not included in any zone's atom bitmap. See AtomMarking.cpp for how * this bitmap is managed. */ #include "gc/GC-inl.h" #include "mozilla/DebugOnly.h" #include "mozilla/MacroForEach.h" #include "mozilla/MemoryReporting.h" #include "mozilla/Range.h" #include "mozilla/ScopeExit.h" #include "mozilla/TextUtils.h" #include "mozilla/TimeStamp.h" #include "mozilla/Unused.h" #include #include #include #include #include #ifndef XP_WIN # include # include #endif #include "jsapi.h" #include "jsfriendapi.h" #include "jstypes.h" #include "builtin/FinalizationRegistryObject.h" #include "builtin/WeakRefObject.h" #include "debugger/DebugAPI.h" #include "gc/ClearEdgesTracer.h" #include "gc/FindSCCs.h" #include "gc/FreeOp.h" #include "gc/GCInternals.h" #include "gc/GCLock.h" #include "gc/GCProbes.h" #include "gc/Memory.h" #include "gc/ParallelWork.h" #include "gc/Policy.h" #include "gc/WeakMap.h" #include "jit/BaselineJIT.h" #include "jit/JitCode.h" #include "jit/JitcodeMap.h" #include "jit/JitRealm.h" #include "jit/JitRuntime.h" #include "jit/JitZone.h" #include "jit/MacroAssembler.h" // js::jit::CodeAlignment #include "js/Object.h" // JS::GetClass #include "js/SliceBudget.h" #include "proxy/DeadObjectProxy.h" #include "util/DifferentialTesting.h" #include "util/Poison.h" #include "util/Windows.h" #include "vm/BigIntType.h" #include "vm/GeckoProfiler.h" #include "vm/HelperThreadState.h" #include "vm/JSAtom.h" #include "vm/JSContext.h" #include "vm/JSObject.h" #include "vm/JSScript.h" #include "vm/Printer.h" #include "vm/ProxyObject.h" #include "vm/Realm.h" #include "vm/Shape.h" #include "vm/StringType.h" #include "vm/SymbolType.h" #include "vm/Time.h" #include "vm/TraceLogging.h" #include "vm/WrapperObject.h" #include "gc/Heap-inl.h" #include "gc/Marking-inl.h" #include "gc/Nursery-inl.h" #include "gc/PrivateIterators-inl.h" #include "gc/Zone-inl.h" #include "vm/GeckoProfiler-inl.h" #include "vm/JSObject-inl.h" #include "vm/JSScript-inl.h" #include "vm/Stack-inl.h" #include "vm/StringType-inl.h" using namespace js; using namespace js::gc; using mozilla::Maybe; using mozilla::Nothing; using mozilla::Some; using mozilla::TimeDuration; using mozilla::TimeStamp; using JS::AutoGCRooter; /* Increase the IGC marking slice time if we are in highFrequencyGC mode. */ static constexpr int IGC_MARK_SLICE_MULTIPLIER = 2; const AllocKind gc::slotsToThingKind[] = { // clang-format off /* 0 */ AllocKind::OBJECT0, AllocKind::OBJECT2, AllocKind::OBJECT2, AllocKind::OBJECT4, /* 4 */ AllocKind::OBJECT4, AllocKind::OBJECT8, AllocKind::OBJECT8, AllocKind::OBJECT8, /* 8 */ AllocKind::OBJECT8, AllocKind::OBJECT12, AllocKind::OBJECT12, AllocKind::OBJECT12, /* 12 */ AllocKind::OBJECT12, AllocKind::OBJECT16, AllocKind::OBJECT16, AllocKind::OBJECT16, /* 16 */ AllocKind::OBJECT16 // clang-format on }; // Check that reserved bits of a Cell are compatible with our typical allocators // since most derived classes will store a pointer in the first word. static const size_t MinFirstWordAlignment = 1u << CellFlagBitsReservedForGC; static_assert(js::detail::LIFO_ALLOC_ALIGN >= MinFirstWordAlignment, "CellFlagBitsReservedForGC should support LifoAlloc"); static_assert(CellAlignBytes >= MinFirstWordAlignment, "CellFlagBitsReservedForGC should support gc::Cell"); static_assert(js::jit::CodeAlignment >= MinFirstWordAlignment, "CellFlagBitsReservedForGC should support JIT code"); static_assert(js::gc::JSClassAlignBytes >= MinFirstWordAlignment, "CellFlagBitsReservedForGC should support JSClass pointers"); static_assert(js::ScopeDataAlignBytes >= MinFirstWordAlignment, "CellFlagBitsReservedForGC should support scope data pointers"); static_assert(std::size(slotsToThingKind) == SLOTS_TO_THING_KIND_LIMIT, "We have defined a slot count for each kind."); #define CHECK_THING_SIZE(allocKind, traceKind, type, sizedType, bgFinal, \ nursery, compact) \ static_assert(sizeof(sizedType) >= SortedArenaList::MinThingSize, \ #sizedType " is smaller than SortedArenaList::MinThingSize!"); \ static_assert(sizeof(sizedType) >= sizeof(FreeSpan), \ #sizedType " is smaller than FreeSpan"); \ static_assert(sizeof(sizedType) % CellAlignBytes == 0, \ "Size of " #sizedType " is not a multiple of CellAlignBytes"); \ static_assert(sizeof(sizedType) >= MinCellSize, \ "Size of " #sizedType " is smaller than the minimum size"); FOR_EACH_ALLOCKIND(CHECK_THING_SIZE); #undef CHECK_THING_SIZE template struct ArenaLayout { static constexpr size_t thingSize() { return sizeof(T); } static constexpr size_t thingsPerArena() { return (ArenaSize - ArenaHeaderSize) / thingSize(); } static constexpr size_t firstThingOffset() { return ArenaSize - thingSize() * thingsPerArena(); } }; const uint8_t Arena::ThingSizes[] = { #define EXPAND_THING_SIZE(_1, _2, _3, sizedType, _4, _5, _6) \ ArenaLayout::thingSize(), FOR_EACH_ALLOCKIND(EXPAND_THING_SIZE) #undef EXPAND_THING_SIZE }; const uint8_t Arena::FirstThingOffsets[] = { #define EXPAND_FIRST_THING_OFFSET(_1, _2, _3, sizedType, _4, _5, _6) \ ArenaLayout::firstThingOffset(), FOR_EACH_ALLOCKIND(EXPAND_FIRST_THING_OFFSET) #undef EXPAND_FIRST_THING_OFFSET }; const uint8_t Arena::ThingsPerArena[] = { #define EXPAND_THINGS_PER_ARENA(_1, _2, _3, sizedType, _4, _5, _6) \ ArenaLayout::thingsPerArena(), FOR_EACH_ALLOCKIND(EXPAND_THINGS_PER_ARENA) #undef EXPAND_THINGS_PER_ARENA }; FreeSpan FreeLists::emptySentinel; struct js::gc::FinalizePhase { gcstats::PhaseKind statsPhase; AllocKinds kinds; }; /* * Finalization order for objects swept incrementally on the main thread. */ static constexpr FinalizePhase ForegroundObjectFinalizePhase = { gcstats::PhaseKind::SWEEP_OBJECT, {AllocKind::OBJECT0, AllocKind::OBJECT2, AllocKind::OBJECT4, AllocKind::OBJECT8, AllocKind::OBJECT12, AllocKind::OBJECT16}}; /* * Finalization order for GC things swept incrementally on the main thread. */ static constexpr FinalizePhase ForegroundNonObjectFinalizePhase = { gcstats::PhaseKind::SWEEP_SCRIPT, {AllocKind::SCRIPT, AllocKind::JITCODE}}; /* * Finalization order for GC things swept on the background thread. */ static constexpr FinalizePhase BackgroundFinalizePhases[] = { {gcstats::PhaseKind::SWEEP_OBJECT, {AllocKind::FUNCTION, AllocKind::FUNCTION_EXTENDED, AllocKind::OBJECT0_BACKGROUND, AllocKind::OBJECT2_BACKGROUND, AllocKind::ARRAYBUFFER4, AllocKind::OBJECT4_BACKGROUND, AllocKind::ARRAYBUFFER8, AllocKind::OBJECT8_BACKGROUND, AllocKind::ARRAYBUFFER12, AllocKind::OBJECT12_BACKGROUND, AllocKind::ARRAYBUFFER16, AllocKind::OBJECT16_BACKGROUND}}, {gcstats::PhaseKind::SWEEP_SCOPE, { AllocKind::SCOPE, }}, {gcstats::PhaseKind::SWEEP_REGEXP_SHARED, { AllocKind::REGEXP_SHARED, }}, {gcstats::PhaseKind::SWEEP_STRING, {AllocKind::FAT_INLINE_STRING, AllocKind::STRING, AllocKind::EXTERNAL_STRING, AllocKind::FAT_INLINE_ATOM, AllocKind::ATOM, AllocKind::SYMBOL, AllocKind::BIGINT}}, {gcstats::PhaseKind::SWEEP_SHAPE, {AllocKind::SHAPE, AllocKind::ACCESSOR_SHAPE, AllocKind::BASE_SHAPE, AllocKind::OBJECT_GROUP}}}; void Arena::unmarkAll() { MarkBitmapWord* arenaBits = chunk()->markBits.arenaBits(this); for (size_t i = 0; i < ArenaBitmapWords; i++) { arenaBits[i] = 0; } } void Arena::unmarkPreMarkedFreeCells() { for (ArenaFreeCellIter cell(this); !cell.done(); cell.next()) { MOZ_ASSERT(cell->isMarkedBlack()); cell->unmark(); } } #ifdef DEBUG void Arena::checkNoMarkedFreeCells() { for (ArenaFreeCellIter cell(this); !cell.done(); cell.next()) { MOZ_ASSERT(!cell->isMarkedAny()); } } void Arena::checkAllCellsMarkedBlack() { for (ArenaCellIter cell(this); !cell.done(); cell.next()) { MOZ_ASSERT(cell->isMarkedBlack()); } } #endif #if defined(DEBUG) || defined(JS_GC_ZEAL) void Arena::checkNoMarkedCells() { for (ArenaCellIter cell(this); !cell.done(); cell.next()) { MOZ_ASSERT(!cell->isMarkedAny()); } } #endif /* static */ void Arena::staticAsserts() { static_assert(size_t(AllocKind::LIMIT) <= 255, "All AllocKinds and AllocKind::LIMIT must fit in a uint8_t."); static_assert(std::size(ThingSizes) == AllocKindCount, "We haven't defined all thing sizes."); static_assert(std::size(FirstThingOffsets) == AllocKindCount, "We haven't defined all offsets."); static_assert(std::size(ThingsPerArena) == AllocKindCount, "We haven't defined all counts."); } /* static */ inline void Arena::checkLookupTables() { #ifdef DEBUG for (size_t i = 0; i < AllocKindCount; i++) { MOZ_ASSERT( FirstThingOffsets[i] + ThingsPerArena[i] * ThingSizes[i] == ArenaSize, "Inconsistent arena lookup table data"); } #endif } template inline size_t Arena::finalize(JSFreeOp* fop, AllocKind thingKind, size_t thingSize) { /* Enforce requirements on size of T. */ MOZ_ASSERT(thingSize % CellAlignBytes == 0); MOZ_ASSERT(thingSize >= MinCellSize); MOZ_ASSERT(thingSize <= 255); MOZ_ASSERT(allocated()); MOZ_ASSERT(thingKind == getAllocKind()); MOZ_ASSERT(thingSize == getThingSize()); MOZ_ASSERT(!onDelayedMarkingList_); uint_fast16_t firstThing = firstThingOffset(thingKind); uint_fast16_t firstThingOrSuccessorOfLastMarkedThing = firstThing; uint_fast16_t lastThing = ArenaSize - thingSize; FreeSpan newListHead; FreeSpan* newListTail = &newListHead; size_t nmarked = 0, nfinalized = 0; for (ArenaCellIterUnderFinalize cell(this); !cell.done(); cell.next()) { T* t = cell.as(); if (t->asTenured().isMarkedAny()) { uint_fast16_t thing = uintptr_t(t) & ArenaMask; if (thing != firstThingOrSuccessorOfLastMarkedThing) { // We just finished passing over one or more free things, // so record a new FreeSpan. newListTail->initBounds(firstThingOrSuccessorOfLastMarkedThing, thing - thingSize, this); newListTail = newListTail->nextSpanUnchecked(this); } firstThingOrSuccessorOfLastMarkedThing = thing + thingSize; nmarked++; } else { t->finalize(fop); AlwaysPoison(t, JS_SWEPT_TENURED_PATTERN, thingSize, MemCheckKind::MakeUndefined); gcprobes::TenuredFinalize(t); nfinalized++; } } if (thingKind == AllocKind::STRING || thingKind == AllocKind::FAT_INLINE_STRING) { zone->markedStrings += nmarked; zone->finalizedStrings += nfinalized; } if (nmarked == 0) { // Do nothing. The caller will update the arena appropriately. MOZ_ASSERT(newListTail == &newListHead); DebugOnlyPoison(data, JS_SWEPT_TENURED_PATTERN, sizeof(data), MemCheckKind::MakeUndefined); return nmarked; } MOZ_ASSERT(firstThingOrSuccessorOfLastMarkedThing != firstThing); uint_fast16_t lastMarkedThing = firstThingOrSuccessorOfLastMarkedThing - thingSize; if (lastThing == lastMarkedThing) { // If the last thing was marked, we will have already set the bounds of // the final span, and we just need to terminate the list. newListTail->initAsEmpty(); } else { // Otherwise, end the list with a span that covers the final stretch of free // things. newListTail->initFinal(firstThingOrSuccessorOfLastMarkedThing, lastThing, this); } firstFreeSpan = newListHead; #ifdef DEBUG size_t nfree = numFreeThings(thingSize); MOZ_ASSERT(nfree + nmarked == thingsPerArena(thingKind)); #endif return nmarked; } // Finalize arenas from src list, releasing empty arenas if keepArenas wasn't // specified and inserting the others into the appropriate destination size // bins. template static inline bool FinalizeTypedArenas(JSFreeOp* fop, Arena** src, SortedArenaList& dest, AllocKind thingKind, SliceBudget& budget) { AutoSetThreadIsFinalizing setThreadUse; size_t thingSize = Arena::thingSize(thingKind); size_t thingsPerArena = Arena::thingsPerArena(thingKind); while (Arena* arena = *src) { Arena* next = arena->next; MOZ_ASSERT_IF(next, next->zone == arena->zone); *src = next; size_t nmarked = arena->finalize(fop, thingKind, thingSize); size_t nfree = thingsPerArena - nmarked; if (nmarked) { dest.insertAt(arena, nfree); } else { arena->chunk()->recycleArena(arena, dest, thingsPerArena); } budget.step(thingsPerArena); if (budget.isOverBudget()) { return false; } } return true; } /* * Finalize the list of areans. */ static bool FinalizeArenas(JSFreeOp* fop, Arena** src, SortedArenaList& dest, AllocKind thingKind, SliceBudget& budget) { switch (thingKind) { #define EXPAND_CASE(allocKind, traceKind, type, sizedType, bgFinal, nursery, \ compact) \ case AllocKind::allocKind: \ return FinalizeTypedArenas(fop, src, dest, thingKind, budget); FOR_EACH_ALLOCKIND(EXPAND_CASE) #undef EXPAND_CASE default: MOZ_CRASH("Invalid alloc kind"); } } TenuredChunk* ChunkPool::pop() { MOZ_ASSERT(bool(head_) == bool(count_)); if (!count_) { return nullptr; } return remove(head_); } void ChunkPool::push(TenuredChunk* chunk) { MOZ_ASSERT(!chunk->info.next); MOZ_ASSERT(!chunk->info.prev); chunk->info.next = head_; if (head_) { head_->info.prev = chunk; } head_ = chunk; ++count_; } TenuredChunk* ChunkPool::remove(TenuredChunk* chunk) { MOZ_ASSERT(count_ > 0); MOZ_ASSERT(contains(chunk)); if (head_ == chunk) { head_ = chunk->info.next; } if (chunk->info.prev) { chunk->info.prev->info.next = chunk->info.next; } if (chunk->info.next) { chunk->info.next->info.prev = chunk->info.prev; } chunk->info.next = chunk->info.prev = nullptr; --count_; return chunk; } // We could keep the chunk pool sorted, but that's likely to be more expensive. // This sort is nlogn, but keeping it sorted is likely to be m*n, with m being // the number of operations (likely higher than n). void ChunkPool::sort() { // Only sort if the list isn't already sorted. if (!isSorted()) { head_ = mergeSort(head(), count()); // Fixup prev pointers. TenuredChunk* prev = nullptr; for (TenuredChunk* cur = head_; cur; cur = cur->info.next) { cur->info.prev = prev; prev = cur; } } MOZ_ASSERT(verify()); MOZ_ASSERT(isSorted()); } TenuredChunk* ChunkPool::mergeSort(TenuredChunk* list, size_t count) { MOZ_ASSERT(bool(list) == bool(count)); if (count < 2) { return list; } size_t half = count / 2; // Split; TenuredChunk* front = list; TenuredChunk* back; { TenuredChunk* cur = list; for (size_t i = 0; i < half - 1; i++) { MOZ_ASSERT(cur); cur = cur->info.next; } back = cur->info.next; cur->info.next = nullptr; } front = mergeSort(front, half); back = mergeSort(back, count - half); // Merge list = nullptr; TenuredChunk** cur = &list; while (front || back) { if (!front) { *cur = back; break; } if (!back) { *cur = front; break; } // Note that the sort is stable due to the <= here. Nothing depends on // this but it could. if (front->info.numArenasFree <= back->info.numArenasFree) { *cur = front; front = front->info.next; cur = &(*cur)->info.next; } else { *cur = back; back = back->info.next; cur = &(*cur)->info.next; } } return list; } bool ChunkPool::isSorted() const { uint32_t last = 1; for (TenuredChunk* cursor = head_; cursor; cursor = cursor->info.next) { if (cursor->info.numArenasFree < last) { return false; } last = cursor->info.numArenasFree; } return true; } #ifdef DEBUG bool ChunkPool::contains(TenuredChunk* chunk) const { verify(); for (TenuredChunk* cursor = head_; cursor; cursor = cursor->info.next) { if (cursor == chunk) { return true; } } return false; } bool ChunkPool::verify() const { MOZ_ASSERT(bool(head_) == bool(count_)); uint32_t count = 0; for (TenuredChunk* cursor = head_; cursor; cursor = cursor->info.next, ++count) { MOZ_ASSERT_IF(cursor->info.prev, cursor->info.prev->info.next == cursor); MOZ_ASSERT_IF(cursor->info.next, cursor->info.next->info.prev == cursor); } MOZ_ASSERT(count_ == count); return true; } #endif void ChunkPool::Iter::next() { MOZ_ASSERT(!done()); current_ = current_->info.next; } inline bool GCRuntime::tooManyEmptyChunks(const AutoLockGC& lock) { return emptyChunks(lock).count() > tunables.minEmptyChunkCount(lock); } ChunkPool GCRuntime::expireEmptyChunkPool(const AutoLockGC& lock) { MOZ_ASSERT(emptyChunks(lock).verify()); MOZ_ASSERT(tunables.minEmptyChunkCount(lock) <= tunables.maxEmptyChunkCount()); ChunkPool expired; while (tooManyEmptyChunks(lock)) { TenuredChunk* chunk = emptyChunks(lock).pop(); prepareToFreeChunk(chunk->info); expired.push(chunk); } MOZ_ASSERT(expired.verify()); MOZ_ASSERT(emptyChunks(lock).verify()); MOZ_ASSERT(emptyChunks(lock).count() <= tunables.maxEmptyChunkCount()); MOZ_ASSERT(emptyChunks(lock).count() <= tunables.minEmptyChunkCount(lock)); return expired; } static void FreeChunkPool(ChunkPool& pool) { for (ChunkPool::Iter iter(pool); !iter.done();) { TenuredChunk* chunk = iter.get(); iter.next(); pool.remove(chunk); MOZ_ASSERT(!chunk->info.numArenasFreeCommitted); UnmapPages(static_cast(chunk), ChunkSize); } MOZ_ASSERT(pool.count() == 0); } void GCRuntime::freeEmptyChunks(const AutoLockGC& lock) { FreeChunkPool(emptyChunks(lock)); } inline void GCRuntime::prepareToFreeChunk(TenuredChunkInfo& info) { MOZ_ASSERT(numArenasFreeCommitted >= info.numArenasFreeCommitted); numArenasFreeCommitted -= info.numArenasFreeCommitted; stats().count(gcstats::COUNT_DESTROY_CHUNK); #ifdef DEBUG /* * Let FreeChunkPool detect a missing prepareToFreeChunk call before it * frees chunk. */ info.numArenasFreeCommitted = 0; #endif } inline void GCRuntime::updateOnArenaFree() { ++numArenasFreeCommitted; } void TenuredChunk::addArenaToFreeList(GCRuntime* gc, Arena* arena) { MOZ_ASSERT(!arena->allocated()); arena->next = info.freeArenasHead; info.freeArenasHead = arena; ++info.numArenasFreeCommitted; ++info.numArenasFree; gc->updateOnArenaFree(); } void TenuredChunk::addArenaToDecommittedList(const Arena* arena) { ++info.numArenasFree; decommittedArenas[TenuredChunk::arenaIndex(arena->address())] = true; } void TenuredChunk::recycleArena(Arena* arena, SortedArenaList& dest, size_t thingsPerArena) { arena->setAsFullyUnused(); dest.insertAt(arena, thingsPerArena); } void TenuredChunk::releaseArena(GCRuntime* gc, Arena* arena, const AutoLockGC& lock) { addArenaToFreeList(gc, arena); updateChunkListAfterFree(gc, lock); } bool TenuredChunk::decommitOneFreeArena(GCRuntime* gc, AutoLockGC& lock) { MOZ_ASSERT(info.numArenasFreeCommitted > 0); Arena* arena = fetchNextFreeArena(gc); updateChunkListAfterAlloc(gc, lock); bool ok; { AutoUnlockGC unlock(lock); ok = MarkPagesUnusedSoft(arena, ArenaSize); } if (ok) { addArenaToDecommittedList(arena); } else { addArenaToFreeList(gc, arena); } updateChunkListAfterFree(gc, lock); return ok; } void TenuredChunk::decommitFreeArenasWithoutUnlocking(const AutoLockGC& lock) { for (size_t i = 0; i < ArenasPerChunk; ++i) { if (decommittedArenas[i] || arenas[i].allocated()) { continue; } if (MarkPagesUnusedSoft(&arenas[i], ArenaSize)) { info.numArenasFreeCommitted--; decommittedArenas[i] = true; } } } void TenuredChunk::updateChunkListAfterAlloc(GCRuntime* gc, const AutoLockGC& lock) { if (MOZ_UNLIKELY(!hasAvailableArenas())) { gc->availableChunks(lock).remove(this); gc->fullChunks(lock).push(this); } } void TenuredChunk::updateChunkListAfterFree(GCRuntime* gc, const AutoLockGC& lock) { if (info.numArenasFree == 1) { gc->fullChunks(lock).remove(this); gc->availableChunks(lock).push(this); } else if (!unused()) { MOZ_ASSERT(gc->availableChunks(lock).contains(this)); } else { MOZ_ASSERT(unused()); gc->availableChunks(lock).remove(this); decommitAllArenas(); MOZ_ASSERT(info.numArenasFreeCommitted == 0); gc->recycleChunk(this, lock); } } void GCRuntime::releaseArena(Arena* arena, const AutoLockGC& lock) { MOZ_ASSERT(arena->allocated()); MOZ_ASSERT(!arena->onDelayedMarkingList()); arena->zone->gcHeapSize.removeGCArena(); arena->release(lock); arena->chunk()->releaseArena(this, arena, lock); } GCRuntime::GCRuntime(JSRuntime* rt) : rt(rt), systemZone(nullptr), atomsZone(nullptr), heapState_(JS::HeapState::Idle), stats_(this), marker(rt), heapSize(nullptr), helperThreadRatio(TuningDefaults::HelperThreadRatio), maxHelperThreads(TuningDefaults::MaxHelperThreads), helperThreadCount(1), rootsHash(256), nextCellUniqueId_(LargestTaggedNullCellPointer + 1), // Ensure disjoint from null tagged pointers. numArenasFreeCommitted(0), verifyPreData(nullptr), lastGCStartTime_(ReallyNow()), lastGCEndTime_(ReallyNow()), incrementalGCEnabled(TuningDefaults::IncrementalGCEnabled), perZoneGCEnabled(TuningDefaults::PerZoneGCEnabled), numActiveZoneIters(0), cleanUpEverything(false), grayBufferState(GCRuntime::GrayBufferState::Unused), grayBitsValid(false), majorGCTriggerReason(JS::GCReason::NO_REASON), fullGCForAtomsRequested_(false), minorGCNumber(0), majorGCNumber(0), number(0), sliceNumber(0), isFull(false), incrementalState(gc::State::NotActive), initialState(gc::State::NotActive), useZeal(false), lastMarkSlice(false), safeToYield(true), markOnBackgroundThreadDuringSweeping(false), sweepOnBackgroundThread(false), requestSliceAfterBackgroundTask(false), lifoBlocksToFree((size_t)JSContext::TEMP_LIFO_ALLOC_PRIMARY_CHUNK_SIZE), lifoBlocksToFreeAfterMinorGC( (size_t)JSContext::TEMP_LIFO_ALLOC_PRIMARY_CHUNK_SIZE), sweepGroupIndex(0), sweepGroups(nullptr), currentSweepGroup(nullptr), sweepZone(nullptr), hasMarkedGrayRoots(false), abortSweepAfterCurrentGroup(false), sweepMarkResult(IncrementalProgress::NotFinished), startedCompacting(false), relocatedArenasToRelease(nullptr), zonesCompacted(0), #ifdef JS_GC_ZEAL markingValidator(nullptr), #endif defaultTimeBudgetMS_(TuningDefaults::DefaultTimeBudgetMS), incrementalAllowed(true), compactingEnabled(TuningDefaults::CompactingEnabled), rootsRemoved(false), #ifdef JS_GC_ZEAL zealModeBits(0), zealFrequency(0), nextScheduled(0), deterministicOnly(false), zealSliceBudget(0), selectedForMarking(rt), #endif fullCompartmentChecks(false), gcCallbackDepth(0), alwaysPreserveCode(false), lowMemoryState(false), lock(mutexid::GCLock), allocTask(this, emptyChunks_.ref()), unmarkTask(this), markTask(this), sweepTask(this), freeTask(this), decommitTask(this), nursery_(this), storeBuffer_(rt, nursery()) { marker.setIncrementalGCEnabled(incrementalGCEnabled); } #ifdef JS_GC_ZEAL void GCRuntime::getZealBits(uint32_t* zealBits, uint32_t* frequency, uint32_t* scheduled) { *zealBits = zealModeBits; *frequency = zealFrequency; *scheduled = nextScheduled; } const char gc::ZealModeHelpText[] = " Specifies how zealous the garbage collector should be. Some of these " "modes can\n" " be set simultaneously, by passing multiple level options, e.g. \"2;4\" " "will activate\n" " both modes 2 and 4. Modes can be specified by name or number.\n" " \n" " Values:\n" " 0: (None) Normal amount of collection (resets all modes)\n" " 1: (RootsChange) Collect when roots are added or removed\n" " 2: (Alloc) Collect when every N allocations (default: 100)\n" " 4: (VerifierPre) Verify pre write barriers between instructions\n" " 6: (YieldBeforeRootMarking) Incremental GC in two slices that yields " "before root marking\n" " 7: (GenerationalGC) Collect the nursery every N nursery allocations\n" " 8: (YieldBeforeMarking) Incremental GC in two slices that yields " "between\n" " the root marking and marking phases\n" " 9: (YieldBeforeSweeping) Incremental GC in two slices that yields " "between\n" " the marking and sweeping phases\n" " 10: (IncrementalMultipleSlices) Incremental GC in many slices\n" " 11: (IncrementalMarkingValidator) Verify incremental marking\n" " 12: (ElementsBarrier) Use the individual element post-write barrier\n" " regardless of elements size\n" " 13: (CheckHashTablesOnMinorGC) Check internal hashtables on minor GC\n" " 14: (Compact) Perform a shrinking collection every N allocations\n" " 15: (CheckHeapAfterGC) Walk the heap to check its integrity after " "every GC\n" " 16: (CheckNursery) Check nursery integrity on minor GC\n" " 17: (YieldBeforeSweepingAtoms) Incremental GC in two slices that " "yields\n" " before sweeping the atoms table\n" " 18: (CheckGrayMarking) Check gray marking invariants after every GC\n" " 19: (YieldBeforeSweepingCaches) Incremental GC in two slices that " "yields\n" " before sweeping weak caches\n" " 21: (YieldBeforeSweepingObjects) Incremental GC in two slices that " "yields\n" " before sweeping foreground finalized objects\n" " 22: (YieldBeforeSweepingNonObjects) Incremental GC in two slices that " "yields\n" " before sweeping non-object GC things\n" " 23: (YieldBeforeSweepingShapeTrees) Incremental GC in two slices that " "yields\n" " before sweeping shape trees\n" " 24: (CheckWeakMapMarking) Check weak map marking invariants after " "every GC\n" " 25: (YieldWhileGrayMarking) Incremental GC in two slices that yields\n" " during gray marking\n"; // The set of zeal modes that control incremental slices. These modes are // mutually exclusive. static const mozilla::EnumSet IncrementalSliceZealModes = { ZealMode::YieldBeforeRootMarking, ZealMode::YieldBeforeMarking, ZealMode::YieldBeforeSweeping, ZealMode::IncrementalMultipleSlices, ZealMode::YieldBeforeSweepingAtoms, ZealMode::YieldBeforeSweepingCaches, ZealMode::YieldBeforeSweepingObjects, ZealMode::YieldBeforeSweepingNonObjects, ZealMode::YieldBeforeSweepingShapeTrees}; void GCRuntime::setZeal(uint8_t zeal, uint32_t frequency) { MOZ_ASSERT(zeal <= unsigned(ZealMode::Limit)); if (verifyPreData) { VerifyBarriers(rt, PreBarrierVerifier); } if (zeal == 0) { if (hasZealMode(ZealMode::GenerationalGC)) { evictNursery(JS::GCReason::DEBUG_GC); nursery().leaveZealMode(); } if (isIncrementalGCInProgress()) { finishGC(JS::GCReason::DEBUG_GC); } } ZealMode zealMode = ZealMode(zeal); if (zealMode == ZealMode::GenerationalGC) { evictNursery(JS::GCReason::DEBUG_GC); nursery().enterZealMode(); } // Some modes are mutually exclusive. If we're setting one of those, we // first reset all of them. if (IncrementalSliceZealModes.contains(zealMode)) { for (auto mode : IncrementalSliceZealModes) { clearZealMode(mode); } } bool schedule = zealMode >= ZealMode::Alloc; if (zeal != 0) { zealModeBits |= 1 << unsigned(zeal); } else { zealModeBits = 0; } zealFrequency = frequency; nextScheduled = schedule ? frequency : 0; } void GCRuntime::unsetZeal(uint8_t zeal) { MOZ_ASSERT(zeal <= unsigned(ZealMode::Limit)); ZealMode zealMode = ZealMode(zeal); if (!hasZealMode(zealMode)) { return; } if (verifyPreData) { VerifyBarriers(rt, PreBarrierVerifier); } if (zealMode == ZealMode::GenerationalGC) { evictNursery(JS::GCReason::DEBUG_GC); nursery().leaveZealMode(); } clearZealMode(zealMode); if (zealModeBits == 0) { if (isIncrementalGCInProgress()) { finishGC(JS::GCReason::DEBUG_GC); } zealFrequency = 0; nextScheduled = 0; } } void GCRuntime::setNextScheduled(uint32_t count) { nextScheduled = count; } using CharRange = mozilla::Range; using CharRangeVector = Vector; static bool ParseZealModeName(CharRange text, uint32_t* modeOut) { struct ModeInfo { const char* name; size_t length; uint32_t value; }; static const ModeInfo zealModes[] = {{"None", 0}, # define ZEAL_MODE(name, value) {# name, strlen(# name), value}, JS_FOR_EACH_ZEAL_MODE(ZEAL_MODE) # undef ZEAL_MODE }; for (auto mode : zealModes) { if (text.length() == mode.length && memcmp(text.begin().get(), mode.name, mode.length) == 0) { *modeOut = mode.value; return true; } } return false; } static bool ParseZealModeNumericParam(CharRange text, uint32_t* paramOut) { if (text.length() == 0) { return false; } for (auto c : text) { if (!mozilla::IsAsciiDigit(c)) { return false; } } *paramOut = atoi(text.begin().get()); return true; } static bool SplitStringBy(CharRange text, char delimiter, CharRangeVector* result) { auto start = text.begin(); for (auto ptr = start; ptr != text.end(); ptr++) { if (*ptr == delimiter) { if (!result->emplaceBack(start, ptr)) { return false; } start = ptr + 1; } } return result->emplaceBack(start, text.end()); } static bool PrintZealHelpAndFail() { fprintf(stderr, "Format: JS_GC_ZEAL=level(;level)*[,N]\n"); fputs(ZealModeHelpText, stderr); return false; } bool GCRuntime::parseAndSetZeal(const char* str) { // Set the zeal mode from a string consisting of one or more mode specifiers // separated by ';', optionally followed by a ',' and the trigger frequency. // The mode specifiers can by a mode name or its number. auto text = CharRange(str, strlen(str)); CharRangeVector parts; if (!SplitStringBy(text, ',', &parts)) { return false; } if (parts.length() == 0 || parts.length() > 2) { return PrintZealHelpAndFail(); } uint32_t frequency = JS_DEFAULT_ZEAL_FREQ; if (parts.length() == 2 && !ParseZealModeNumericParam(parts[1], &frequency)) { return PrintZealHelpAndFail(); } CharRangeVector modes; if (!SplitStringBy(parts[0], ';', &modes)) { return false; } for (const auto& descr : modes) { uint32_t mode; if (!ParseZealModeName(descr, &mode) && !(ParseZealModeNumericParam(descr, &mode) && mode <= unsigned(ZealMode::Limit))) { return PrintZealHelpAndFail(); } setZeal(mode, frequency); } return true; } const char* js::gc::AllocKindName(AllocKind kind) { static const char* const names[] = { # define EXPAND_THING_NAME(allocKind, _1, _2, _3, _4, _5, _6) # allocKind, FOR_EACH_ALLOCKIND(EXPAND_THING_NAME) # undef EXPAND_THING_NAME }; static_assert(std::size(names) == AllocKindCount, "names array should have an entry for every AllocKind"); size_t i = size_t(kind); MOZ_ASSERT(i < std::size(names)); return names[i]; } void js::gc::DumpArenaInfo() { fprintf(stderr, "Arena header size: %zu\n\n", ArenaHeaderSize); fprintf(stderr, "GC thing kinds:\n"); fprintf(stderr, "%25s %8s %8s %8s\n", "AllocKind:", "Size:", "Count:", "Padding:"); for (auto kind : AllAllocKinds()) { fprintf(stderr, "%25s %8zu %8zu %8zu\n", AllocKindName(kind), Arena::thingSize(kind), Arena::thingsPerArena(kind), Arena::firstThingOffset(kind) - ArenaHeaderSize); } } #endif // JS_GC_ZEAL bool GCRuntime::init(uint32_t maxbytes) { MOZ_ASSERT(SystemPageSize()); Arena::checkLookupTables(); { AutoLockGCBgAlloc lock(this); MOZ_ALWAYS_TRUE(tunables.setParameter(JSGC_MAX_BYTES, maxbytes, lock)); const char* size = getenv("JSGC_MARK_STACK_LIMIT"); if (size) { setMarkStackLimit(atoi(size), lock); } if (!nursery().init(lock)) { return false; } const char* pretenureThresholdStr = getenv("JSGC_PRETENURE_THRESHOLD"); if (pretenureThresholdStr && pretenureThresholdStr[0]) { char* last; long pretenureThreshold = strtol(pretenureThresholdStr, &last, 10); if (last[0] || !tunables.setParameter(JSGC_PRETENURE_THRESHOLD, pretenureThreshold, lock)) { fprintf(stderr, "Invalid value for JSGC_PRETENURE_THRESHOLD: %s\n", pretenureThresholdStr); } } } #ifdef JS_GC_ZEAL const char* zealSpec = getenv("JS_GC_ZEAL"); if (zealSpec && zealSpec[0] && !parseAndSetZeal(zealSpec)) { return false; } #endif if (!marker.init() || !initSweepActions()) { return false; } gcprobes::Init(this); updateHelperThreadCount(); return true; } void GCRuntime::freezeSelfHostingZone() { MOZ_ASSERT(!selfHostingZoneFrozen); MOZ_ASSERT(!isIncrementalGCInProgress()); for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { MOZ_ASSERT(!zone->isGCScheduled()); if (zone->isSelfHostingZone()) { zone->scheduleGC(); } } gc(GC_SHRINK, JS::GCReason::INIT_SELF_HOSTING); selfHostingZoneFrozen = true; } void GCRuntime::finish() { MOZ_ASSERT(inPageLoadCount == 0); // Wait for nursery background free to end and disable it to release memory. if (nursery().isEnabled()) { nursery().disable(); } // Wait until the background finalization and allocation stops and the // helper thread shuts down before we forcefully release any remaining GC // memory. sweepTask.join(); freeTask.join(); allocTask.cancelAndWait(); decommitTask.cancelAndWait(); #ifdef JS_GC_ZEAL // Free memory associated with GC verification. finishVerifier(); #endif // Delete all remaining zones. if (rt->gcInitialized) { for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { AutoSetThreadIsSweeping threadIsSweeping(zone); for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next()) { for (RealmsInCompartmentIter realm(comp); !realm.done(); realm.next()) { js_delete(realm.get()); } comp->realms().clear(); js_delete(comp.get()); } zone->compartments().clear(); js_delete(zone.get()); } } zones().clear(); FreeChunkPool(fullChunks_.ref()); FreeChunkPool(availableChunks_.ref()); FreeChunkPool(emptyChunks_.ref()); gcprobes::Finish(this); nursery().printTotalProfileTimes(); stats().printTotalProfileTimes(); } bool GCRuntime::setParameter(JSGCParamKey key, uint32_t value) { MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); waitBackgroundSweepEnd(); AutoLockGC lock(this); return setParameter(key, value, lock); } bool GCRuntime::setParameter(JSGCParamKey key, uint32_t value, AutoLockGC& lock) { switch (key) { case JSGC_SLICE_TIME_BUDGET_MS: defaultTimeBudgetMS_ = value ? value : SliceBudget::UnlimitedTimeBudget; break; case JSGC_MARK_STACK_LIMIT: if (value == 0) { return false; } setMarkStackLimit(value, lock); break; case JSGC_INCREMENTAL_GC_ENABLED: setIncrementalGCEnabled(value != 0); break; case JSGC_PER_ZONE_GC_ENABLED: perZoneGCEnabled = value != 0; break; case JSGC_COMPACTING_ENABLED: compactingEnabled = value != 0; break; case JSGC_INCREMENTAL_WEAKMAP_ENABLED: marker.incrementalWeakMapMarkingEnabled = value != 0; break; case JSGC_HELPER_THREAD_RATIO: if (rt->parentRuntime) { // Don't allow this to be set for worker runtimes. return false; } if (value == 0) { return false; } helperThreadRatio = double(value) / 100.0; updateHelperThreadCount(); break; case JSGC_MAX_HELPER_THREADS: if (rt->parentRuntime) { // Don't allow this to be set for worker runtimes. return false; } if (value == 0) { return false; } maxHelperThreads = value; updateHelperThreadCount(); break; default: if (!tunables.setParameter(key, value, lock)) { return false; } updateAllGCStartThresholds(lock); } return true; } void GCRuntime::resetParameter(JSGCParamKey key) { MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); waitBackgroundSweepEnd(); AutoLockGC lock(this); resetParameter(key, lock); } void GCRuntime::resetParameter(JSGCParamKey key, AutoLockGC& lock) { switch (key) { case JSGC_SLICE_TIME_BUDGET_MS: defaultTimeBudgetMS_ = TuningDefaults::DefaultTimeBudgetMS; break; case JSGC_MARK_STACK_LIMIT: setMarkStackLimit(MarkStack::DefaultCapacity, lock); break; case JSGC_INCREMENTAL_GC_ENABLED: setIncrementalGCEnabled(TuningDefaults::IncrementalGCEnabled); break; case JSGC_PER_ZONE_GC_ENABLED: perZoneGCEnabled = TuningDefaults::PerZoneGCEnabled; break; case JSGC_COMPACTING_ENABLED: compactingEnabled = TuningDefaults::CompactingEnabled; break; case JSGC_INCREMENTAL_WEAKMAP_ENABLED: marker.incrementalWeakMapMarkingEnabled = TuningDefaults::IncrementalWeakMapMarkingEnabled; break; case JSGC_HELPER_THREAD_RATIO: if (rt->parentRuntime) { return; } helperThreadRatio = TuningDefaults::HelperThreadRatio; updateHelperThreadCount(); break; case JSGC_MAX_HELPER_THREADS: if (rt->parentRuntime) { return; } maxHelperThreads = TuningDefaults::MaxHelperThreads; updateHelperThreadCount(); break; default: tunables.resetParameter(key, lock); updateAllGCStartThresholds(lock); } } uint32_t GCRuntime::getParameter(JSGCParamKey key) { MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); AutoLockGC lock(this); return getParameter(key, lock); } uint32_t GCRuntime::getParameter(JSGCParamKey key, const AutoLockGC& lock) { switch (key) { case JSGC_MAX_BYTES: return uint32_t(tunables.gcMaxBytes()); case JSGC_MIN_NURSERY_BYTES: MOZ_ASSERT(tunables.gcMinNurseryBytes() < UINT32_MAX); return uint32_t(tunables.gcMinNurseryBytes()); case JSGC_MAX_NURSERY_BYTES: MOZ_ASSERT(tunables.gcMaxNurseryBytes() < UINT32_MAX); return uint32_t(tunables.gcMaxNurseryBytes()); case JSGC_BYTES: return uint32_t(heapSize.bytes()); case JSGC_NURSERY_BYTES: return nursery().capacity(); case JSGC_NUMBER: return uint32_t(number); case JSGC_MAJOR_GC_NUMBER: return uint32_t(majorGCNumber); case JSGC_MINOR_GC_NUMBER: return uint32_t(minorGCNumber); case JSGC_INCREMENTAL_GC_ENABLED: return incrementalGCEnabled; case JSGC_PER_ZONE_GC_ENABLED: return perZoneGCEnabled; case JSGC_UNUSED_CHUNKS: return uint32_t(emptyChunks(lock).count()); case JSGC_TOTAL_CHUNKS: return uint32_t(fullChunks(lock).count() + availableChunks(lock).count() + emptyChunks(lock).count()); case JSGC_SLICE_TIME_BUDGET_MS: if (defaultTimeBudgetMS_.ref() == SliceBudget::UnlimitedTimeBudget) { return 0; } else { MOZ_RELEASE_ASSERT(defaultTimeBudgetMS_ >= 0); MOZ_RELEASE_ASSERT(defaultTimeBudgetMS_ <= UINT32_MAX); return uint32_t(defaultTimeBudgetMS_); } case JSGC_MARK_STACK_LIMIT: return marker.maxCapacity(); case JSGC_HIGH_FREQUENCY_TIME_LIMIT: return tunables.highFrequencyThreshold().ToMilliseconds(); case JSGC_SMALL_HEAP_SIZE_MAX: return tunables.smallHeapSizeMaxBytes() / 1024 / 1024; case JSGC_LARGE_HEAP_SIZE_MIN: return tunables.largeHeapSizeMinBytes() / 1024 / 1024; case JSGC_HIGH_FREQUENCY_SMALL_HEAP_GROWTH: return uint32_t(tunables.highFrequencySmallHeapGrowth() * 100); case JSGC_HIGH_FREQUENCY_LARGE_HEAP_GROWTH: return uint32_t(tunables.highFrequencyLargeHeapGrowth() * 100); case JSGC_LOW_FREQUENCY_HEAP_GROWTH: return uint32_t(tunables.lowFrequencyHeapGrowth() * 100); case JSGC_ALLOCATION_THRESHOLD: return tunables.gcZoneAllocThresholdBase() / 1024 / 1024; case JSGC_SMALL_HEAP_INCREMENTAL_LIMIT: return uint32_t(tunables.smallHeapIncrementalLimit() * 100); case JSGC_LARGE_HEAP_INCREMENTAL_LIMIT: return uint32_t(tunables.largeHeapIncrementalLimit() * 100); case JSGC_MIN_EMPTY_CHUNK_COUNT: return tunables.minEmptyChunkCount(lock); case JSGC_MAX_EMPTY_CHUNK_COUNT: return tunables.maxEmptyChunkCount(); case JSGC_COMPACTING_ENABLED: return compactingEnabled; case JSGC_INCREMENTAL_WEAKMAP_ENABLED: return marker.incrementalWeakMapMarkingEnabled; case JSGC_NURSERY_FREE_THRESHOLD_FOR_IDLE_COLLECTION: return tunables.nurseryFreeThresholdForIdleCollection(); case JSGC_NURSERY_FREE_THRESHOLD_FOR_IDLE_COLLECTION_PERCENT: return uint32_t(tunables.nurseryFreeThresholdForIdleCollectionFraction() * 100.0f); case JSGC_PRETENURE_THRESHOLD: return uint32_t(tunables.pretenureThreshold() * 100); case JSGC_PRETENURE_GROUP_THRESHOLD: return tunables.pretenureGroupThreshold(); case JSGC_PRETENURE_STRING_THRESHOLD: return uint32_t(tunables.pretenureStringThreshold() * 100); case JSGC_STOP_PRETENURE_STRING_THRESHOLD: return uint32_t(tunables.stopPretenureStringThreshold() * 100); case JSGC_MIN_LAST_DITCH_GC_PERIOD: return tunables.minLastDitchGCPeriod().ToSeconds(); case JSGC_ZONE_ALLOC_DELAY_KB: return tunables.zoneAllocDelayBytes() / 1024; case JSGC_MALLOC_THRESHOLD_BASE: return tunables.mallocThresholdBase() / 1024 / 1024; case JSGC_MALLOC_GROWTH_FACTOR: return uint32_t(tunables.mallocGrowthFactor() * 100); case JSGC_CHUNK_BYTES: return ChunkSize; case JSGC_HELPER_THREAD_RATIO: MOZ_ASSERT(helperThreadRatio > 0.0); return uint32_t(helperThreadRatio * 100.0); case JSGC_MAX_HELPER_THREADS: MOZ_ASSERT(maxHelperThreads <= UINT32_MAX); return maxHelperThreads; case JSGC_HELPER_THREAD_COUNT: return helperThreadCount; default: MOZ_CRASH("Unknown parameter key"); } } void GCRuntime::setMarkStackLimit(size_t limit, AutoLockGC& lock) { MOZ_ASSERT(!JS::RuntimeHeapIsBusy()); AutoUnlockGC unlock(lock); AutoStopVerifyingBarriers pauseVerification(rt, false); marker.setMaxCapacity(limit); } void GCRuntime::setIncrementalGCEnabled(bool enabled) { incrementalGCEnabled = enabled; marker.setIncrementalGCEnabled(enabled); } void GCRuntime::updateHelperThreadCount() { if (!CanUseExtraThreads()) { // startTask will run the work on the main thread if the count is 1. MOZ_ASSERT(helperThreadCount == 1); return; } // The count of helper threads used for GC tasks is process wide. Don't set it // for worker JS runtimes. if (rt->parentRuntime) { helperThreadCount = rt->parentRuntime->gc.helperThreadCount; return; } double cpuCount = HelperThreadState().cpuCount; size_t target = size_t(cpuCount * helperThreadRatio.ref()); helperThreadCount = mozilla::Clamp(target, size_t(1), maxHelperThreads.ref()); HelperThreadState().ensureThreadCount(helperThreadCount); AutoLockHelperThreadState lock; HelperThreadState().setGCParallelThreadCount(helperThreadCount, lock); } bool GCRuntime::addBlackRootsTracer(JSTraceDataOp traceOp, void* data) { AssertHeapIsIdle(); return !!blackRootTracers.ref().append( Callback(traceOp, data)); } void GCRuntime::removeBlackRootsTracer(JSTraceDataOp traceOp, void* data) { // Can be called from finalizers for (size_t i = 0; i < blackRootTracers.ref().length(); i++) { Callback* e = &blackRootTracers.ref()[i]; if (e->op == traceOp && e->data == data) { blackRootTracers.ref().erase(e); break; } } } void GCRuntime::setGrayRootsTracer(JSTraceDataOp traceOp, void* data) { AssertHeapIsIdle(); grayRootTracer.ref() = {traceOp, data}; } void GCRuntime::clearBlackAndGrayRootTracers() { MOZ_ASSERT(rt->isBeingDestroyed()); blackRootTracers.ref().clear(); setGrayRootsTracer(nullptr, nullptr); } void GCRuntime::setGCCallback(JSGCCallback callback, void* data) { gcCallback.ref() = {callback, data}; } void GCRuntime::callGCCallback(JSGCStatus status, JS::GCReason reason) const { const auto& callback = gcCallback.ref(); MOZ_ASSERT(callback.op); callback.op(rt->mainContextFromOwnThread(), status, reason, callback.data); } void GCRuntime::setObjectsTenuredCallback(JSObjectsTenuredCallback callback, void* data) { tenuredCallback.ref() = {callback, data}; } void GCRuntime::callObjectsTenuredCallback() { JS::AutoSuppressGCAnalysis nogc; const auto& callback = tenuredCallback.ref(); if (callback.op) { callback.op(rt->mainContextFromOwnThread(), callback.data); } } bool GCRuntime::addFinalizeCallback(JSFinalizeCallback callback, void* data) { return finalizeCallbacks.ref().append( Callback(callback, data)); } template static void EraseCallback(CallbackVector& vector, F callback) { for (Callback* p = vector.begin(); p != vector.end(); p++) { if (p->op == callback) { vector.erase(p); return; } } } void GCRuntime::removeFinalizeCallback(JSFinalizeCallback callback) { EraseCallback(finalizeCallbacks.ref(), callback); } void GCRuntime::callFinalizeCallbacks(JSFreeOp* fop, JSFinalizeStatus status) const { for (auto& p : finalizeCallbacks.ref()) { p.op(fop, status, p.data); } } void GCRuntime::setHostCleanupFinalizationRegistryCallback( JSHostCleanupFinalizationRegistryCallback callback, void* data) { hostCleanupFinalizationRegistryCallback.ref() = {callback, data}; } void GCRuntime::callHostCleanupFinalizationRegistryCallback( JSFunction* doCleanup, GlobalObject* incumbentGlobal) { JS::AutoSuppressGCAnalysis nogc; const auto& callback = hostCleanupFinalizationRegistryCallback.ref(); if (callback.op) { callback.op(doCleanup, incumbentGlobal, callback.data); } } bool GCRuntime::addWeakPointerZonesCallback(JSWeakPointerZonesCallback callback, void* data) { return updateWeakPointerZonesCallbacks.ref().append( Callback(callback, data)); } void GCRuntime::removeWeakPointerZonesCallback( JSWeakPointerZonesCallback callback) { EraseCallback(updateWeakPointerZonesCallbacks.ref(), callback); } void GCRuntime::callWeakPointerZonesCallbacks() const { JSContext* cx = rt->mainContextFromOwnThread(); for (auto const& p : updateWeakPointerZonesCallbacks.ref()) { p.op(cx, p.data); } } bool GCRuntime::addWeakPointerCompartmentCallback( JSWeakPointerCompartmentCallback callback, void* data) { return updateWeakPointerCompartmentCallbacks.ref().append( Callback(callback, data)); } void GCRuntime::removeWeakPointerCompartmentCallback( JSWeakPointerCompartmentCallback callback) { EraseCallback(updateWeakPointerCompartmentCallbacks.ref(), callback); } void GCRuntime::callWeakPointerCompartmentCallbacks( JS::Compartment* comp) const { JSContext* cx = rt->mainContextFromOwnThread(); for (auto const& p : updateWeakPointerCompartmentCallbacks.ref()) { p.op(cx, comp, p.data); } } JS::GCSliceCallback GCRuntime::setSliceCallback(JS::GCSliceCallback callback) { return stats().setSliceCallback(callback); } JS::GCNurseryCollectionCallback GCRuntime::setNurseryCollectionCallback( JS::GCNurseryCollectionCallback callback) { return stats().setNurseryCollectionCallback(callback); } JS::DoCycleCollectionCallback GCRuntime::setDoCycleCollectionCallback( JS::DoCycleCollectionCallback callback) { const auto prior = gcDoCycleCollectionCallback.ref(); gcDoCycleCollectionCallback.ref() = {callback, nullptr}; return prior.op; } void GCRuntime::callDoCycleCollectionCallback(JSContext* cx) { const auto& callback = gcDoCycleCollectionCallback.ref(); if (callback.op) { callback.op(cx); } } bool GCRuntime::addRoot(Value* vp, const char* name) { /* * Sometimes Firefox will hold weak references to objects and then convert * them to strong references by calling AddRoot (e.g., via PreserveWrapper, * or ModifyBusyCount in workers). We need a read barrier to cover these * cases. */ MOZ_ASSERT(vp); Value value = *vp; if (value.isGCThing()) { ValuePreWriteBarrier(value); } return rootsHash.ref().put(vp, name); } void GCRuntime::removeRoot(Value* vp) { rootsHash.ref().remove(vp); notifyRootsRemoved(); } extern JS_FRIEND_API bool js::AddRawValueRoot(JSContext* cx, Value* vp, const char* name) { MOZ_ASSERT(vp); MOZ_ASSERT(name); bool ok = cx->runtime()->gc.addRoot(vp, name); if (!ok) { JS_ReportOutOfMemory(cx); } return ok; } extern JS_FRIEND_API void js::RemoveRawValueRoot(JSContext* cx, Value* vp) { cx->runtime()->gc.removeRoot(vp); } /* Compacting GC */ bool js::gc::IsCurrentlyAnimating(const TimeStamp& lastAnimationTime, const TimeStamp& currentTime) { // Assume that we're currently animating if js::NotifyAnimationActivity has // been called in the last second. static const auto oneSecond = TimeDuration::FromSeconds(1); return !lastAnimationTime.IsNull() && currentTime < (lastAnimationTime + oneSecond); } bool GCRuntime::shouldCompact() { // Compact on shrinking GC if enabled. Skip compacting in incremental GCs // if we are currently animating, unless the user is inactive or we're // responding to memory pressure. if (invocationKind != GC_SHRINK || !isCompactingGCEnabled()) { return false; } if (initialReason == JS::GCReason::USER_INACTIVE || initialReason == JS::GCReason::MEM_PRESSURE) { return true; } return !isIncremental || !IsCurrentlyAnimating(rt->lastAnimationTime, TimeStamp::Now()); } bool GCRuntime::isCompactingGCEnabled() const { return compactingEnabled && rt->mainContextFromOwnThread()->compactingDisabledCount == 0; } AutoDisableCompactingGC::AutoDisableCompactingGC(JSContext* cx) : cx(cx) { ++cx->compactingDisabledCount; if (cx->runtime()->gc.isIncrementalGCInProgress() && cx->runtime()->gc.isCompactingGc()) { FinishGC(cx); } } AutoDisableCompactingGC::~AutoDisableCompactingGC() { MOZ_ASSERT(cx->compactingDisabledCount > 0); --cx->compactingDisabledCount; } bool GCRuntime::canRelocateZone(Zone* zone) const { if (zone->isAtomsZone()) { return false; } if (zone->isSelfHostingZone() && selfHostingZoneFrozen) { return false; } return true; } #ifdef DEBUG void js::gc::ArenaList::dump() { fprintf(stderr, "ArenaList %p:", this); if (cursorp_ == &head_) { fprintf(stderr, " *"); } for (Arena* arena = head(); arena; arena = arena->next) { fprintf(stderr, " %p", arena); if (cursorp_ == &arena->next) { fprintf(stderr, " *"); } } fprintf(stderr, "\n"); } #endif Arena* ArenaList::removeRemainingArenas(Arena** arenap) { // This is only ever called to remove arenas that are after the cursor, so // we don't need to update it. #ifdef DEBUG for (Arena* arena = *arenap; arena; arena = arena->next) { MOZ_ASSERT(cursorp_ != &arena->next); } #endif Arena* remainingArenas = *arenap; *arenap = nullptr; check(); return remainingArenas; } static bool ShouldRelocateAllArenas(JS::GCReason reason) { return reason == JS::GCReason::DEBUG_GC; } /* * Choose which arenas to relocate all cells from. Return an arena cursor that * can be passed to removeRemainingArenas(). */ Arena** ArenaList::pickArenasToRelocate(size_t& arenaTotalOut, size_t& relocTotalOut) { // Relocate the greatest number of arenas such that the number of used cells // in relocated arenas is less than or equal to the number of free cells in // unrelocated arenas. In other words we only relocate cells we can move // into existing arenas, and we choose the least full areans to relocate. // // This is made easier by the fact that the arena list has been sorted in // descending order of number of used cells, so we will always relocate a // tail of the arena list. All we need to do is find the point at which to // start relocating. check(); if (isCursorAtEnd()) { return nullptr; } Arena** arenap = cursorp_; // Next arena to consider for relocation. size_t previousFreeCells = 0; // Count of free cells before arenap. size_t followingUsedCells = 0; // Count of used cells after arenap. size_t fullArenaCount = 0; // Number of full arenas (not relocated). size_t nonFullArenaCount = 0; // Number of non-full arenas (considered for relocation). size_t arenaIndex = 0; // Index of the next arena to consider. for (Arena* arena = head_; arena != *cursorp_; arena = arena->next) { fullArenaCount++; } for (Arena* arena = *cursorp_; arena; arena = arena->next) { followingUsedCells += arena->countUsedCells(); nonFullArenaCount++; } mozilla::DebugOnly lastFreeCells(0); size_t cellsPerArena = Arena::thingsPerArena((*arenap)->getAllocKind()); while (*arenap) { Arena* arena = *arenap; if (followingUsedCells <= previousFreeCells) { break; } size_t freeCells = arena->countFreeCells(); size_t usedCells = cellsPerArena - freeCells; followingUsedCells -= usedCells; #ifdef DEBUG MOZ_ASSERT(freeCells >= lastFreeCells); lastFreeCells = freeCells; #endif previousFreeCells += freeCells; arenap = &arena->next; arenaIndex++; } size_t relocCount = nonFullArenaCount - arenaIndex; MOZ_ASSERT(relocCount < nonFullArenaCount); MOZ_ASSERT((relocCount == 0) == (!*arenap)); arenaTotalOut += fullArenaCount + nonFullArenaCount; relocTotalOut += relocCount; return arenap; } #ifdef DEBUG inline bool PtrIsInRange(const void* ptr, const void* start, size_t length) { return uintptr_t(ptr) - uintptr_t(start) < length; } #endif static void RelocateCell(Zone* zone, TenuredCell* src, AllocKind thingKind, size_t thingSize) { JS::AutoSuppressGCAnalysis nogc(TlsContext.get()); // Allocate a new cell. MOZ_ASSERT(zone == src->zone()); TenuredCell* dst = AllocateCellInGC(zone, thingKind); // Copy source cell contents to destination. memcpy(dst, src, thingSize); // Move any uid attached to the object. src->zone()->transferUniqueId(dst, src); if (IsObjectAllocKind(thingKind)) { auto* srcObj = static_cast(static_cast(src)); auto* dstObj = static_cast(static_cast(dst)); if (srcObj->isNative()) { NativeObject* srcNative = &srcObj->as(); NativeObject* dstNative = &dstObj->as(); // Fixup the pointer to inline object elements if necessary. if (srcNative->hasFixedElements()) { uint32_t numShifted = srcNative->getElementsHeader()->numShiftedElements(); dstNative->setFixedElements(numShifted); } } else if (srcObj->is()) { if (srcObj->as().usingInlineValueArray()) { dstObj->as().setInlineValueArray(); } } // Call object moved hook if present. if (JSObjectMovedOp op = srcObj->getClass()->extObjectMovedOp()) { op(dstObj, srcObj); } MOZ_ASSERT_IF( dstObj->isNative(), !PtrIsInRange( (const Value*)dstObj->as().getDenseElements(), src, thingSize)); } // Copy the mark bits. dst->copyMarkBitsFrom(src); // Poison the source cell contents except for the forwarding flag and pointer // which will be stored in the first word. We can't do this for native object // with fixed elements because this would overwrite the element flags and // these are needed when updating COW elements referred to by other objects. #ifdef DEBUG JSObject* srcObj = IsObjectAllocKind(thingKind) ? static_cast(static_cast(src)) : nullptr; if (!srcObj || !srcObj->isNative() || !srcObj->as().hasFixedElements()) { AlwaysPoison(reinterpret_cast(src) + sizeof(uintptr_t), JS_MOVED_TENURED_PATTERN, thingSize - sizeof(uintptr_t), MemCheckKind::MakeNoAccess); } #endif // Mark source cell as forwarded and leave a pointer to the destination. RelocationOverlay::forwardCell(src, dst); } static void RelocateArena(Arena* arena, SliceBudget& sliceBudget) { MOZ_ASSERT(arena->allocated()); MOZ_ASSERT(!arena->onDelayedMarkingList()); MOZ_ASSERT(arena->bufferedCells()->isEmpty()); Zone* zone = arena->zone; AllocKind thingKind = arena->getAllocKind(); size_t thingSize = arena->getThingSize(); for (ArenaCellIterUnderGC cell(arena); !cell.done(); cell.next()) { RelocateCell(zone, cell, thingKind, thingSize); sliceBudget.step(); } #ifdef DEBUG for (ArenaCellIterUnderGC cell(arena); !cell.done(); cell.next()) { TenuredCell* src = cell; MOZ_ASSERT(src->isForwarded()); TenuredCell* dest = Forwarded(src); MOZ_ASSERT(src->isMarkedBlack() == dest->isMarkedBlack()); MOZ_ASSERT(src->isMarkedGray() == dest->isMarkedGray()); } #endif } #ifdef DEBUG static inline bool CanProtectArenas() { // On some systems the page size is larger than the size of an arena so we // can't change the mapping permissions per arena. return SystemPageSize() <= ArenaSize; } #endif static inline bool ShouldProtectRelocatedArenas(JS::GCReason reason) { // For zeal mode collections we don't release the relocated arenas // immediately. Instead we protect them and keep them around until the next // collection so we can catch any stray accesses to them. #ifdef DEBUG return reason == JS::GCReason::DEBUG_GC && CanProtectArenas(); #else return false; #endif } /* * Relocate all arenas identified by pickArenasToRelocate: for each arena, * relocate each cell within it, then add it to a list of relocated arenas. */ Arena* ArenaList::relocateArenas(Arena* toRelocate, Arena* relocated, SliceBudget& sliceBudget, gcstats::Statistics& stats) { check(); while (Arena* arena = toRelocate) { toRelocate = arena->next; RelocateArena(arena, sliceBudget); // Prepend to list of relocated arenas arena->next = relocated; relocated = arena; stats.count(gcstats::COUNT_ARENA_RELOCATED); } check(); return relocated; } // Skip compacting zones unless we can free a certain proportion of their GC // heap memory. static const float MIN_ZONE_RECLAIM_PERCENT = 2.0; static bool ShouldRelocateZone(size_t arenaCount, size_t relocCount, JS::GCReason reason) { if (relocCount == 0) { return false; } if (IsOOMReason(reason)) { return true; } return (relocCount * 100.0f) / arenaCount >= MIN_ZONE_RECLAIM_PERCENT; } static AllocKinds CompactingAllocKinds() { AllocKinds result; for (AllocKind kind : AllAllocKinds()) { if (IsCompactingKind(kind)) { result += kind; } } return result; } bool ArenaLists::relocateArenas(Arena*& relocatedListOut, JS::GCReason reason, SliceBudget& sliceBudget, gcstats::Statistics& stats) { // This is only called from the main thread while we are doing a GC, so // there is no need to lock. MOZ_ASSERT(CurrentThreadCanAccessRuntime(runtime())); MOZ_ASSERT(runtime()->gc.isHeapCompacting()); MOZ_ASSERT(!runtime()->gc.isBackgroundSweeping()); // Relocate all compatible kinds AllocKinds allocKindsToRelocate = CompactingAllocKinds(); // Clear all the free lists. clearFreeLists(); if (ShouldRelocateAllArenas(reason)) { zone_->prepareForCompacting(); for (auto kind : allocKindsToRelocate) { ArenaList& al = arenaList(kind); Arena* allArenas = al.head(); al.clear(); relocatedListOut = al.relocateArenas(allArenas, relocatedListOut, sliceBudget, stats); } } else { size_t arenaCount = 0; size_t relocCount = 0; AllAllocKindArray toRelocate; for (auto kind : allocKindsToRelocate) { toRelocate[kind] = arenaList(kind).pickArenasToRelocate(arenaCount, relocCount); } if (!ShouldRelocateZone(arenaCount, relocCount, reason)) { return false; } zone_->prepareForCompacting(); for (auto kind : allocKindsToRelocate) { if (toRelocate[kind]) { ArenaList& al = arenaList(kind); Arena* arenas = al.removeRemainingArenas(toRelocate[kind]); relocatedListOut = al.relocateArenas(arenas, relocatedListOut, sliceBudget, stats); } } } return true; } bool GCRuntime::relocateArenas(Zone* zone, JS::GCReason reason, Arena*& relocatedListOut, SliceBudget& sliceBudget) { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::COMPACT_MOVE); MOZ_ASSERT(!zone->isPreservingCode()); MOZ_ASSERT(canRelocateZone(zone)); js::CancelOffThreadIonCompile(rt, JS::Zone::Compact); if (!zone->arenas.relocateArenas(relocatedListOut, reason, sliceBudget, stats())) { return false; } #ifdef DEBUG // Check that we did as much compaction as we should have. There // should always be less than one arena's worth of free cells. for (auto kind : CompactingAllocKinds()) { ArenaList& al = zone->arenas.arenaList(kind); size_t freeCells = 0; for (Arena* arena = al.arenaAfterCursor(); arena; arena = arena->next) { freeCells += arena->countFreeCells(); } MOZ_ASSERT(freeCells < Arena::thingsPerArena(kind)); } #endif return true; } template inline T* MovingTracer::onEdge(T* thing) { if (thing->runtimeFromAnyThread() == runtime() && IsForwarded(thing)) { thing = Forwarded(thing); } return thing; } JSObject* MovingTracer::onObjectEdge(JSObject* obj) { return onEdge(obj); } Shape* MovingTracer::onShapeEdge(Shape* shape) { return onEdge(shape); } JSString* MovingTracer::onStringEdge(JSString* string) { return onEdge(string); } js::BaseScript* MovingTracer::onScriptEdge(js::BaseScript* script) { return onEdge(script); } BaseShape* MovingTracer::onBaseShapeEdge(BaseShape* base) { return onEdge(base); } Scope* MovingTracer::onScopeEdge(Scope* scope) { return onEdge(scope); } RegExpShared* MovingTracer::onRegExpSharedEdge(RegExpShared* shared) { return onEdge(shared); } BigInt* MovingTracer::onBigIntEdge(BigInt* bi) { return onEdge(bi); } ObjectGroup* MovingTracer::onObjectGroupEdge(ObjectGroup* group) { return onEdge(group); } JS::Symbol* MovingTracer::onSymbolEdge(JS::Symbol* sym) { MOZ_ASSERT(!sym->isForwarded()); return sym; } jit::JitCode* MovingTracer::onJitCodeEdge(jit::JitCode* jit) { MOZ_ASSERT(!jit->isForwarded()); return jit; } void Zone::prepareForCompacting() { JSFreeOp* fop = runtimeFromMainThread()->defaultFreeOp(); discardJitCode(fop); } void GCRuntime::sweepZoneAfterCompacting(MovingTracer* trc, Zone* zone) { MOZ_ASSERT(zone->isCollecting()); sweepFinalizationRegistries(zone); zone->weakRefMap().sweep(&storeBuffer()); { zone->sweepWeakMaps(); for (auto* cache : zone->weakCaches()) { cache->sweep(nullptr); } } if (jit::JitZone* jitZone = zone->jitZone()) { jitZone->traceWeak(trc); } for (RealmsInZoneIter r(zone); !r.done(); r.next()) { r->traceWeakRegExps(trc); r->traceWeakSavedStacks(trc); r->tracekWeakVarNames(trc); r->traceWeakObjects(trc); r->traceWeakSelfHostingScriptSource(trc); r->sweepDebugEnvironments(); r->traceWeakEdgesInJitRealm(trc); r->traceWeakObjectRealm(trc); r->traceWeakTemplateObjects(trc); } } template static inline void UpdateCellPointers(MovingTracer* trc, T* cell) { // We only update unmoved GC things or the new copy of moved GC things, never // the old copy. If this happened it could clear the forwarded flag which // could lead to pointers to the old copy not being updated. MOZ_ASSERT(!cell->isForwarded()); cell->fixupAfterMovingGC(); cell->traceChildren(trc); } template static void UpdateArenaPointersTyped(MovingTracer* trc, Arena* arena) { for (ArenaCellIterUnderGC cell(arena); !cell.done(); cell.next()) { UpdateCellPointers(trc, cell.as()); } } static bool CanUpdateKindInBackground(AllocKind kind) { // We try to update as many GC things in parallel as we can, but there are // kinds for which this might not be safe: // - we assume JSObjects that are foreground finalized are not safe to // update in parallel // - updating a shape touches child shapes in fixupShapeTreeAfterMovingGC() return js::gc::IsBackgroundFinalized(kind) && !IsShapeAllocKind(kind) && kind != AllocKind::BASE_SHAPE; } /* * Update the internal pointers for all cells in an arena. */ static void UpdateArenaPointers(MovingTracer* trc, Arena* arena) { AllocKind kind = arena->getAllocKind(); MOZ_ASSERT_IF(!CanUpdateKindInBackground(kind), CurrentThreadCanAccessRuntime(trc->runtime())); switch (kind) { #define EXPAND_CASE(allocKind, traceKind, type, sizedType, bgFinal, nursery, \ compact) \ case AllocKind::allocKind: \ UpdateArenaPointersTyped(trc, arena); \ return; FOR_EACH_ALLOCKIND(EXPAND_CASE) #undef EXPAND_CASE default: MOZ_CRASH("Invalid alloc kind for UpdateArenaPointers"); } } struct ArenaListSegment { Arena* begin; Arena* end; }; /* * Update the internal pointers for all arenas in a segment of an arena list. * * Returns the number of steps to count against the slice budget. */ static size_t UpdateArenaListSegmentPointers(GCRuntime* gc, const ArenaListSegment& arenas) { MOZ_ASSERT(arenas.begin); MovingTracer trc(gc->rt); size_t count = 0; for (Arena* arena = arenas.begin; arena != arenas.end; arena = arena->next) { UpdateArenaPointers(&trc, arena); count++; } return count * 256; } class ArenasToUpdate { // Maximum number of arenas to update in one block. #ifdef DEBUG static const unsigned MaxArenasToProcess = 16; #else static const unsigned MaxArenasToProcess = 256; #endif public: explicit ArenasToUpdate(Zone* zone); ArenasToUpdate(Zone* zone, const AllocKinds& kinds); bool done() const { return !segmentBegin; } ArenaListSegment get() const { MOZ_ASSERT(!done()); return {segmentBegin, segmentEnd}; } void next(); private: Maybe kinds; // Selects which thing kinds to update. Zone* zone; // Zone to process. AllocKind kind = AllocKind::FIRST; // Current alloc kind to process. Arena* segmentBegin = nullptr; Arena* segmentEnd = nullptr; static AllocKind nextAllocKind(AllocKind i) { return AllocKind(uint8_t(i) + 1); } void settle(); void findSegmentEnd(); }; ArenasToUpdate::ArenasToUpdate(Zone* zone) : zone(zone) { settle(); } ArenasToUpdate::ArenasToUpdate(Zone* zone, const AllocKinds& kinds) : kinds(Some(kinds)), zone(zone) { settle(); } void ArenasToUpdate::settle() { // Called when we have set |kind| to a new kind. Sets |arena| to the next // arena or null if there are no more arenas to update. MOZ_ASSERT(!segmentBegin); for (; kind < AllocKind::LIMIT; kind = nextAllocKind(kind)) { if (kinds && !kinds.ref().contains(kind)) { continue; } Arena* arena = zone->arenas.getFirstArena(kind); if (arena) { segmentBegin = arena; findSegmentEnd(); break; } } } void ArenasToUpdate::findSegmentEnd() { // Take up to MaxArenasToProcess arenas from the list starting at // |segmentBegin| and set |segmentEnd|. Arena* arena = segmentBegin; for (size_t i = 0; arena && i < MaxArenasToProcess; i++) { arena = arena->next; } segmentEnd = arena; } void ArenasToUpdate::next() { MOZ_ASSERT(!done()); segmentBegin = segmentEnd; if (segmentBegin) { findSegmentEnd(); return; } kind = nextAllocKind(kind); settle(); } static AllocKinds ForegroundUpdateKinds(AllocKinds kinds) { AllocKinds result; for (AllocKind kind : kinds) { if (!CanUpdateKindInBackground(kind)) { result += kind; } } return result; } void GCRuntime::updateTypeDescrObjects(MovingTracer* trc, Zone* zone) { // We need to update each type descriptor object and any objects stored in // its reserved slots, since some of these contain array objects that also // need to be updated. Do not update any non-reserved slots, since they might // point back to unprocessed descriptor objects. zone->typeDescrObjects().sweep(nullptr); for (auto r = zone->typeDescrObjects().all(); !r.empty(); r.popFront()) { MOZ_ASSERT(MaybeForwardedObjectClass(r.front())->isNative()); NativeObject* obj = static_cast(r.front()); UpdateCellPointers(trc, obj); MOZ_ASSERT(JSCLASS_RESERVED_SLOTS(MaybeForwardedObjectClass(obj)) == TypeDescr::SlotCount); for (size_t i = 0; i < TypeDescr::SlotCount; i++) { Value value = obj->getSlot(i); if (value.isObject()) { UpdateCellPointers(trc, &value.toObject()); } } } } void GCRuntime::updateCellPointers(Zone* zone, AllocKinds kinds) { AllocKinds fgKinds = ForegroundUpdateKinds(kinds); AllocKinds bgKinds = kinds - fgKinds; ArenasToUpdate fgArenas(zone, fgKinds); ArenasToUpdate bgArenas(zone, bgKinds); AutoLockHelperThreadState lock; AutoRunParallelWork bgTasks(this, UpdateArenaListSegmentPointers, gcstats::PhaseKind::COMPACT_UPDATE_CELLS, bgArenas, SliceBudget::unlimited(), lock); AutoUnlockHelperThreadState unlock(lock); for (; !fgArenas.done(); fgArenas.next()) { UpdateArenaListSegmentPointers(this, fgArenas.get()); } } // After cells have been relocated any pointers to a cell's old locations must // be updated to point to the new location. This happens by iterating through // all cells in heap and tracing their children (non-recursively) to update // them. // // This is complicated by the fact that updating a GC thing sometimes depends on // making use of other GC things. After a moving GC these things may not be in // a valid state since they may contain pointers which have not been updated // yet. // // The main dependencies are: // // - Updating a JSObject makes use of its shape // - Updating a typed object makes use of its type descriptor object // // This means we require at least three phases for update: // // 1) shapes // 2) typed object type descriptor objects // 3) all other objects // // Also, there can be data races calling IsForwarded() on the new location of a // cell whose first word is being updated in parallel on another thread. This // easiest way to avoid this is to not store a GC pointer in the first word of a // cell. Otherwise this can be avoided by updating different kinds of cell in // different phases. // // Since we want to minimize the number of phases, arrange kinds into three // arbitrary phases. static constexpr AllocKinds UpdatePhaseOne{ AllocKind::SCRIPT, AllocKind::BASE_SHAPE, AllocKind::SHAPE, AllocKind::ACCESSOR_SHAPE, AllocKind::OBJECT_GROUP, AllocKind::STRING, AllocKind::JITCODE, AllocKind::REGEXP_SHARED, AllocKind::SCOPE}; // UpdatePhaseTwo is typed object descriptor objects. static constexpr AllocKinds UpdatePhaseThree{AllocKind::FUNCTION, AllocKind::FUNCTION_EXTENDED, AllocKind::OBJECT0, AllocKind::OBJECT0_BACKGROUND, AllocKind::OBJECT2, AllocKind::OBJECT2_BACKGROUND, AllocKind::ARRAYBUFFER4, AllocKind::OBJECT4, AllocKind::OBJECT4_BACKGROUND, AllocKind::ARRAYBUFFER8, AllocKind::OBJECT8, AllocKind::OBJECT8_BACKGROUND, AllocKind::ARRAYBUFFER12, AllocKind::OBJECT12, AllocKind::OBJECT12_BACKGROUND, AllocKind::ARRAYBUFFER16, AllocKind::OBJECT16, AllocKind::OBJECT16_BACKGROUND}; void GCRuntime::updateAllCellPointers(MovingTracer* trc, Zone* zone) { updateCellPointers(zone, UpdatePhaseOne); // UpdatePhaseTwo: Update TypeDescrs before all other objects as typed // objects access these objects when we trace them. updateTypeDescrObjects(trc, zone); updateCellPointers(zone, UpdatePhaseThree); } /* * Update pointers to relocated cells in a single zone by doing a traversal of * that zone's arenas and calling per-zone sweep hooks. * * The latter is necessary to update weak references which are not marked as * part of the traversal. */ void GCRuntime::updateZonePointersToRelocatedCells(Zone* zone) { MOZ_ASSERT(!rt->isBeingDestroyed()); MOZ_ASSERT(zone->isGCCompacting()); AutoTouchingGrayThings tgt; gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::COMPACT_UPDATE); MovingTracer trc(rt); zone->fixupAfterMovingGC(); zone->fixupScriptMapsAfterMovingGC(&trc); // Fixup compartment global pointers as these get accessed during marking. for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next()) { comp->fixupAfterMovingGC(&trc); } zone->externalStringCache().purge(); zone->functionToStringCache().purge(); rt->caches().stringToAtomCache.purge(); // Iterate through all cells that can contain relocatable pointers to update // them. Since updating each cell is independent we try to parallelize this // as much as possible. updateAllCellPointers(&trc, zone); // Mark roots to update them. { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::MARK_ROOTS); WeakMapBase::traceZone(zone, &trc); } // Sweep everything to fix up weak pointers. sweepZoneAfterCompacting(&trc, zone); // Call callbacks to get the rest of the system to fixup other untraced // pointers. for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next()) { callWeakPointerCompartmentCallbacks(comp); } } /* * Update runtime-wide pointers to relocated cells. */ void GCRuntime::updateRuntimePointersToRelocatedCells(AutoGCSession& session) { MOZ_ASSERT(!rt->isBeingDestroyed()); gcstats::AutoPhase ap1(stats(), gcstats::PhaseKind::COMPACT_UPDATE); MovingTracer trc(rt); Zone::fixupAllCrossCompartmentWrappersAfterMovingGC(&trc); rt->geckoProfiler().fixupStringsMapAfterMovingGC(); // Mark roots to update them. traceRuntimeForMajorGC(&trc, session); { gcstats::AutoPhase ap2(stats(), gcstats::PhaseKind::MARK_ROOTS); DebugAPI::traceAllForMovingGC(&trc); DebugAPI::traceCrossCompartmentEdges(&trc); // Mark all gray roots. We call the trace callback to get the current set. traceEmbeddingGrayRoots(&trc); Compartment::traceIncomingCrossCompartmentEdgesForZoneGC( &trc, Compartment::GrayEdges); } // Sweep everything to fix up weak pointers. DebugAPI::sweepAll(rt->defaultFreeOp()); jit::JitRuntime::TraceWeakJitcodeGlobalTable(rt, &trc); for (JS::detail::WeakCacheBase* cache : rt->weakCaches()) { cache->sweep(nullptr); } // Type inference may put more blocks here to free. { AutoLockHelperThreadState lock; lifoBlocksToFree.ref().freeAll(); } // Call callbacks to get the rest of the system to fixup other untraced // pointers. callWeakPointerZonesCallbacks(); } void GCRuntime::clearRelocatedArenas(Arena* arenaList, JS::GCReason reason) { AutoLockGC lock(this); clearRelocatedArenasWithoutUnlocking(arenaList, reason, lock); } void GCRuntime::clearRelocatedArenasWithoutUnlocking(Arena* arenaList, JS::GCReason reason, const AutoLockGC& lock) { // Clear the relocated arenas, now containing only forwarding pointers while (arenaList) { Arena* arena = arenaList; arenaList = arenaList->next; // Clear the mark bits arena->unmarkAll(); // Mark arena as empty arena->setAsFullyUnused(); #ifdef DEBUG // The cell contents have been partially marked no access in RelocateCell, // so we need to mark the region as undefined again so we can poison it. SetMemCheckKind(reinterpret_cast(arena->thingsStart()), arena->getThingsSpan(), MemCheckKind::MakeUndefined); #endif AlwaysPoison(reinterpret_cast(arena->thingsStart()), JS_MOVED_TENURED_PATTERN, arena->getThingsSpan(), MemCheckKind::MakeNoAccess); // Don't count arenas as being freed by the GC if we purposely moved // everything to new arenas, as that will already have allocated a similar // number of arenas. This only happens for collections triggered by GC zeal. bool allArenasRelocated = ShouldRelocateAllArenas(reason); arena->zone->gcHeapSize.removeBytes(ArenaSize, !allArenasRelocated); // Release the arena but don't return it to the chunk yet. arena->release(lock); } } void GCRuntime::protectAndHoldArenas(Arena* arenaList) { for (Arena* arena = arenaList; arena;) { MOZ_ASSERT(!arena->allocated()); Arena* next = arena->next; if (!next) { // Prepend to hold list before we protect the memory. arena->next = relocatedArenasToRelease; relocatedArenasToRelease = arenaList; } ProtectPages(arena, ArenaSize); arena = next; } } void GCRuntime::unprotectHeldRelocatedArenas() { for (Arena* arena = relocatedArenasToRelease; arena; arena = arena->next) { UnprotectPages(arena, ArenaSize); MOZ_ASSERT(!arena->allocated()); } } void GCRuntime::releaseRelocatedArenas(Arena* arenaList) { AutoLockGC lock(this); releaseRelocatedArenasWithoutUnlocking(arenaList, lock); } void GCRuntime::releaseRelocatedArenasWithoutUnlocking(Arena* arenaList, const AutoLockGC& lock) { // Release relocated arenas previously cleared with clearRelocatedArenas(). while (arenaList) { Arena* arena = arenaList; arenaList = arenaList->next; // We already updated the memory accounting so just call // Chunk::releaseArena. arena->chunk()->releaseArena(this, arena, lock); } } // In debug mode we don't always release relocated arenas straight away. // Sometimes protect them instead and hold onto them until the next GC sweep // phase to catch any pointers to them that didn't get forwarded. void GCRuntime::releaseHeldRelocatedArenas() { #ifdef DEBUG unprotectHeldRelocatedArenas(); Arena* arenas = relocatedArenasToRelease; relocatedArenasToRelease = nullptr; releaseRelocatedArenas(arenas); #endif } void GCRuntime::releaseHeldRelocatedArenasWithoutUnlocking( const AutoLockGC& lock) { #ifdef DEBUG unprotectHeldRelocatedArenas(); releaseRelocatedArenasWithoutUnlocking(relocatedArenasToRelease, lock); relocatedArenasToRelease = nullptr; #endif } FreeLists::FreeLists() { for (auto i : AllAllocKinds()) { freeLists_[i] = &emptySentinel; } } ArenaLists::ArenaLists(Zone* zone) : zone_(zone), freeLists_(zone), arenaLists_(zone), newArenasInMarkPhase_(zone), arenasToSweep_(), incrementalSweptArenaKind(zone, AllocKind::LIMIT), incrementalSweptArenas(zone), gcShapeArenasToUpdate(zone, nullptr), gcAccessorShapeArenasToUpdate(zone, nullptr), savedEmptyArenas(zone, nullptr) { for (auto i : AllAllocKinds()) { concurrentUse(i) = ConcurrentUse::None; arenasToSweep(i) = nullptr; } } void ReleaseArenas(JSRuntime* rt, Arena* arena, const AutoLockGC& lock) { Arena* next; for (; arena; arena = next) { next = arena->next; rt->gc.releaseArena(arena, lock); } } void ReleaseArenaList(JSRuntime* rt, ArenaList& arenaList, const AutoLockGC& lock) { ReleaseArenas(rt, arenaList.head(), lock); arenaList.clear(); } ArenaLists::~ArenaLists() { AutoLockGC lock(runtime()); for (auto i : AllAllocKinds()) { /* * We can only call this during the shutdown after the last GC when * the background finalization is disabled. */ MOZ_ASSERT(concurrentUse(i) == ConcurrentUse::None); ReleaseArenaList(runtime(), arenaList(i), lock); } ReleaseArenaList(runtime(), incrementalSweptArenas.ref(), lock); ReleaseArenas(runtime(), savedEmptyArenas, lock); } void ArenaLists::queueForForegroundSweep(JSFreeOp* fop, const FinalizePhase& phase) { gcstats::AutoPhase ap(fop->runtime()->gc.stats(), phase.statsPhase); for (auto kind : phase.kinds) { queueForForegroundSweep(kind); } } void ArenaLists::queueForForegroundSweep(AllocKind thingKind) { MOZ_ASSERT(!IsBackgroundFinalized(thingKind)); MOZ_ASSERT(concurrentUse(thingKind) == ConcurrentUse::None); MOZ_ASSERT(!arenasToSweep(thingKind)); arenasToSweep(thingKind) = arenaList(thingKind).head(); arenaList(thingKind).clear(); } void ArenaLists::queueForBackgroundSweep(JSFreeOp* fop, const FinalizePhase& phase) { gcstats::AutoPhase ap(fop->runtime()->gc.stats(), phase.statsPhase); for (auto kind : phase.kinds) { queueForBackgroundSweep(kind); } } inline void ArenaLists::queueForBackgroundSweep(AllocKind thingKind) { MOZ_ASSERT(IsBackgroundFinalized(thingKind)); MOZ_ASSERT(concurrentUse(thingKind) == ConcurrentUse::None); ArenaList* al = &arenaList(thingKind); arenasToSweep(thingKind) = al->head(); arenaList(thingKind).clear(); if (arenasToSweep(thingKind)) { concurrentUse(thingKind) = ConcurrentUse::BackgroundFinalize; } else { arenaList(thingKind) = std::move(newArenasInMarkPhase(thingKind)); } } /*static*/ void ArenaLists::backgroundFinalize(JSFreeOp* fop, Arena* listHead, Arena** empty) { MOZ_ASSERT(listHead); MOZ_ASSERT(empty); AllocKind thingKind = listHead->getAllocKind(); Zone* zone = listHead->zone; size_t thingsPerArena = Arena::thingsPerArena(thingKind); SortedArenaList finalizedSorted(thingsPerArena); auto unlimited = SliceBudget::unlimited(); FinalizeArenas(fop, &listHead, finalizedSorted, thingKind, unlimited); MOZ_ASSERT(!listHead); finalizedSorted.extractEmpty(empty); // When arenas are queued for background finalization, all arenas are moved to // arenasToSweep, leaving the arena list empty. However, new arenas may be // allocated before background finalization finishes; now that finalization is // complete, we want to merge these lists back together. ArenaLists* lists = &zone->arenas; ArenaList& al = lists->arenaList(thingKind); // Flatten |finalizedSorted| into a regular ArenaList. ArenaList finalized = finalizedSorted.toArenaList(); // We must take the GC lock to be able to safely modify the ArenaList; // however, this does not by itself make the changes visible to all threads, // as not all threads take the GC lock to read the ArenaLists. // That safety is provided by the ReleaseAcquire memory ordering of the // background finalize state, which we explicitly set as the final step. { AutoLockGC lock(lists->runtimeFromAnyThread()); MOZ_ASSERT(lists->concurrentUse(thingKind) == ConcurrentUse::BackgroundFinalize); // Join |al| and |finalized| into a single list. ArenaList allocatedDuringSweep = std::move(al); al = std::move(finalized); al.insertListWithCursorAtEnd(lists->newArenasInMarkPhase(thingKind)); al.insertListWithCursorAtEnd(allocatedDuringSweep); lists->newArenasInMarkPhase(thingKind).clear(); lists->arenasToSweep(thingKind) = nullptr; } lists->concurrentUse(thingKind) = ConcurrentUse::None; } Arena* ArenaLists::takeSweptEmptyArenas() { Arena* arenas = savedEmptyArenas; savedEmptyArenas = nullptr; return arenas; } void ArenaLists::queueForegroundThingsForSweep() { gcShapeArenasToUpdate = arenasToSweep(AllocKind::SHAPE); gcAccessorShapeArenasToUpdate = arenasToSweep(AllocKind::ACCESSOR_SHAPE); } void ArenaLists::checkGCStateNotInUse() { // Called before and after collection to check the state is as expected. #ifdef DEBUG checkSweepStateNotInUse(); for (auto i : AllAllocKinds()) { MOZ_ASSERT(newArenasInMarkPhase(i).isEmpty()); } #endif } void ArenaLists::checkSweepStateNotInUse() { #ifdef DEBUG checkNoArenasToUpdate(); MOZ_ASSERT(incrementalSweptArenaKind == AllocKind::LIMIT); MOZ_ASSERT(incrementalSweptArenas.ref().isEmpty()); MOZ_ASSERT(!savedEmptyArenas); for (auto i : AllAllocKinds()) { MOZ_ASSERT(concurrentUse(i) == ConcurrentUse::None); MOZ_ASSERT(!arenasToSweep(i)); } #endif } void ArenaLists::checkNoArenasToUpdate() { MOZ_ASSERT(!gcShapeArenasToUpdate); MOZ_ASSERT(!gcAccessorShapeArenasToUpdate); } void ArenaLists::checkNoArenasToUpdateForKind(AllocKind kind) { #ifdef DEBUG switch (kind) { case AllocKind::SHAPE: MOZ_ASSERT(!gcShapeArenasToUpdate); break; case AllocKind::ACCESSOR_SHAPE: MOZ_ASSERT(!gcShapeArenasToUpdate); break; default: break; } #endif } TimeStamp SliceBudget::unlimitedDeadline; void SliceBudget::Init() { MOZ_ASSERT(!unlimitedDeadline); uint64_t oneYearsInSeconds = 365 * 24 * 60 * 60; unlimitedDeadline = ReallyNow() + TimeDuration::FromSeconds(100 * oneYearsInSeconds); } SliceBudget::SliceBudget() : timeBudget(UnlimitedTimeBudget), workBudget(UnlimitedWorkBudget) { makeUnlimited(); } SliceBudget::SliceBudget(TimeBudget time) : timeBudget(time), workBudget(UnlimitedWorkBudget) { if (time.budget < 0) { makeUnlimited(); } else { // Note: TimeBudget(0) is equivalent to WorkBudget(CounterReset). deadline = ReallyNow() + TimeDuration::FromMilliseconds(time.budget); counter = CounterReset; } } SliceBudget::SliceBudget(WorkBudget work) : timeBudget(UnlimitedTimeBudget), workBudget(work) { if (work.budget < 0) { makeUnlimited(); } else { deadline = TimeStamp(); counter = work.budget; } } int SliceBudget::describe(char* buffer, size_t maxlen) const { if (isUnlimited()) { return snprintf(buffer, maxlen, "unlimited"); } else if (isWorkBudget()) { return snprintf(buffer, maxlen, "work(%" PRId64 ")", workBudget.budget); } else { return snprintf(buffer, maxlen, "%" PRId64 "ms", timeBudget.budget); } } bool SliceBudget::checkOverBudget() { if (deadline.IsNull()) { return true; } bool over = ReallyNow() >= deadline; if (!over) { counter = CounterReset; } return over; } void GCRuntime::requestMajorGC(JS::GCReason reason) { MOZ_ASSERT_IF(reason != JS::GCReason::BG_TASK_FINISHED, !CurrentThreadIsPerformingGC()); if (majorGCRequested()) { return; } majorGCTriggerReason = reason; rt->mainContextFromAnyThread()->requestInterrupt(InterruptReason::GC); } void Nursery::requestMinorGC(JS::GCReason reason) const { MOZ_ASSERT(CurrentThreadCanAccessRuntime(runtime())); if (minorGCRequested()) { return; } minorGCTriggerReason_ = reason; runtime()->mainContextFromOwnThread()->requestInterrupt(InterruptReason::GC); } bool GCRuntime::triggerGC(JS::GCReason reason) { /* * Don't trigger GCs if this is being called off the main thread from * onTooMuchMalloc(). */ if (!CurrentThreadCanAccessRuntime(rt)) { return false; } /* GC is already running. */ if (JS::RuntimeHeapIsCollecting()) { return false; } JS::PrepareForFullGC(rt->mainContextFromOwnThread()); requestMajorGC(reason); return true; } void GCRuntime::maybeTriggerGCAfterAlloc(Zone* zone) { if (!CurrentThreadCanAccessRuntime(rt)) { // Zones in use by a helper thread can't be collected. MOZ_ASSERT(zone->usedByHelperThread() || zone->isAtomsZone()); return; } MOZ_ASSERT(!JS::RuntimeHeapIsCollecting()); TriggerResult trigger = checkHeapThreshold(zone, zone->gcHeapSize, zone->gcHeapThreshold); if (trigger.shouldTrigger) { // Start or continue an in progress incremental GC. We do this to try to // avoid performing non-incremental GCs on zones which allocate a lot of // data, even when incremental slices can't be triggered via scheduling in // the event loop. triggerZoneGC(zone, JS::GCReason::ALLOC_TRIGGER, trigger.usedBytes, trigger.thresholdBytes); } } void js::gc::MaybeMallocTriggerZoneGC(JSRuntime* rt, ZoneAllocator* zoneAlloc, const HeapSize& heap, const HeapThreshold& threshold, JS::GCReason reason) { rt->gc.maybeTriggerGCAfterMalloc(Zone::from(zoneAlloc), heap, threshold, reason); } void GCRuntime::maybeTriggerGCAfterMalloc(Zone* zone) { if (maybeTriggerGCAfterMalloc(zone, zone->mallocHeapSize, zone->mallocHeapThreshold, JS::GCReason::TOO_MUCH_MALLOC)) { return; } maybeTriggerGCAfterMalloc(zone, zone->jitHeapSize, zone->jitHeapThreshold, JS::GCReason::TOO_MUCH_JIT_CODE); } bool GCRuntime::maybeTriggerGCAfterMalloc(Zone* zone, const HeapSize& heap, const HeapThreshold& threshold, JS::GCReason reason) { if (!CurrentThreadCanAccessRuntime(rt)) { // Zones in use by a helper thread can't be collected. Also ignore malloc // during sweeping, for example when we resize hash tables. MOZ_ASSERT(zone->usedByHelperThread() || zone->isAtomsZone() || JS::RuntimeHeapIsBusy()); return false; } if (rt->heapState() != JS::HeapState::Idle) { return false; } TriggerResult trigger = checkHeapThreshold(zone, heap, threshold); if (!trigger.shouldTrigger) { return false; } // Trigger a zone GC. budgetIncrementalGC() will work out whether to do an // incremental or non-incremental collection. triggerZoneGC(zone, reason, trigger.usedBytes, trigger.thresholdBytes); return true; } TriggerResult GCRuntime::checkHeapThreshold( Zone* zone, const HeapSize& heapSize, const HeapThreshold& heapThreshold) { MOZ_ASSERT_IF(heapThreshold.hasSliceThreshold(), zone->wasGCStarted()); size_t usedBytes = heapSize.bytes(); size_t thresholdBytes = zone->gcState() > Zone::Prepare ? heapThreshold.sliceBytes() : heapThreshold.startBytes(); size_t niThreshold = heapThreshold.incrementalLimitBytes(); MOZ_ASSERT(niThreshold >= thresholdBytes); if (usedBytes < thresholdBytes) { return TriggerResult{false, 0, 0}; } // Don't trigger incremental slices during background sweeping or decommit, as // these will have no effect. A slice will be triggered automatically when // these tasks finish. if (usedBytes < niThreshold && zone->wasGCStarted() && (state() == State::Finalize || state() == State::Decommit)) { return TriggerResult{false, 0, 0}; } // Start or continue an in progress incremental GC. return TriggerResult{true, usedBytes, thresholdBytes}; } bool GCRuntime::triggerZoneGC(Zone* zone, JS::GCReason reason, size_t used, size_t threshold) { MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); /* GC is already running. */ if (JS::RuntimeHeapIsBusy()) { return false; } #ifdef JS_GC_ZEAL if (hasZealMode(ZealMode::Alloc)) { MOZ_RELEASE_ASSERT(triggerGC(reason)); return true; } #endif if (zone->isAtomsZone()) { /* We can't do a zone GC of just the atoms zone. */ if (rt->hasHelperThreadZones()) { /* We can't collect atoms while off-thread parsing is allocating. */ fullGCForAtomsRequested_ = true; return false; } stats().recordTrigger(used, threshold); MOZ_RELEASE_ASSERT(triggerGC(reason)); return true; } stats().recordTrigger(used, threshold); zone->scheduleGC(); requestMajorGC(reason); return true; } void GCRuntime::maybeGC() { MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); #ifdef JS_GC_ZEAL if (hasZealMode(ZealMode::Alloc) || hasZealMode(ZealMode::RootsChange)) { JS::PrepareForFullGC(rt->mainContextFromOwnThread()); gc(GC_NORMAL, JS::GCReason::DEBUG_GC); return; } #endif if (gcIfRequested()) { return; } if (isIncrementalGCInProgress()) { return; } bool scheduledZones = false; for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { if (checkEagerAllocTrigger(zone->gcHeapSize, zone->gcHeapThreshold) || checkEagerAllocTrigger(zone->mallocHeapSize, zone->mallocHeapThreshold)) { zone->scheduleGC(); scheduledZones = true; } } if (scheduledZones) { startGC(GC_NORMAL, JS::GCReason::EAGER_ALLOC_TRIGGER); } } bool GCRuntime::checkEagerAllocTrigger(const HeapSize& size, const HeapThreshold& threshold) { double thresholdBytes = threshold.eagerAllocTrigger(schedulingState.inHighFrequencyGCMode()); double usedBytes = size.bytes(); if (usedBytes <= 1024 * 1024 || usedBytes < thresholdBytes) { return false; } stats().recordTrigger(usedBytes, thresholdBytes); return true; } void GCRuntime::triggerFullGCForAtoms(JSContext* cx) { MOZ_ASSERT(fullGCForAtomsRequested_); MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); MOZ_ASSERT(!JS::RuntimeHeapIsCollecting()); MOZ_ASSERT(cx->canCollectAtoms()); fullGCForAtomsRequested_ = false; MOZ_RELEASE_ASSERT(triggerGC(JS::GCReason::DELAYED_ATOMS_GC)); } void GCRuntime::startDecommit() { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::DECOMMIT); #ifdef DEBUG MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); MOZ_ASSERT(decommitTask.isIdle()); { AutoLockGC lock(this); MOZ_ASSERT(fullChunks(lock).verify()); MOZ_ASSERT(availableChunks(lock).verify()); MOZ_ASSERT(emptyChunks(lock).verify()); // Verify that all entries in the empty chunks pool are already decommitted. for (ChunkPool::Iter chunk(emptyChunks(lock)); !chunk.done(); chunk.next()) { MOZ_ASSERT(chunk->info.numArenasFreeCommitted == 0); } } #endif // If we are allocating heavily enough to trigger "high frequency" GC, then // skip decommit so that we do not compete with the mutator. However if we're // doing a shrinking GC we always decommit to release as much memory as // possible. if (schedulingState.inHighFrequencyGCMode() && !cleanUpEverything) { return; } { AutoLockGC lock(this); if (availableChunks(lock).empty() && !tooManyEmptyChunks(lock)) { return; // Nothing to do. } } #ifdef DEBUG { AutoLockHelperThreadState lock; MOZ_ASSERT(!requestSliceAfterBackgroundTask); } #endif if (sweepOnBackgroundThread) { decommitTask.start(); return; } decommitTask.runFromMainThread(); } void js::gc::BackgroundDecommitTask::run(AutoLockHelperThreadState& lock) { { AutoUnlockHelperThreadState unlock(lock); ChunkPool emptyChunksToFree; { AutoLockGC gcLock(gc); // To help minimize the total number of chunks needed over time, sort the // available chunks list so that we allocate into more-used chunks first. gc->availableChunks(gcLock).sort(); gc->decommitFreeArenas(cancel_, gcLock); emptyChunksToFree = gc->expireEmptyChunkPool(gcLock); } FreeChunkPool(emptyChunksToFree); } gc->maybeRequestGCAfterBackgroundTask(lock); } // Called from a background thread to decommit free arenas. Releases the GC // lock. void GCRuntime::decommitFreeArenas(const bool& cancel, AutoLockGC& lock) { // Since we release the GC lock while doing the decommit syscall below, // it is dangerous to iterate the available list directly, as the active // thread could modify it concurrently. Instead, we build and pass an // explicit Vector containing the Chunks we want to visit. Vector chunksToDecommit; for (ChunkPool::Iter chunk(availableChunks(lock)); !chunk.done(); chunk.next()) { if (chunk->info.numArenasFreeCommitted != 0 && !chunksToDecommit.append(chunk)) { onOutOfMallocMemory(lock); return; } } for (TenuredChunk* chunk : chunksToDecommit) { // The arena list is not doubly-linked, so we have to work in the free // list order and not in the natural order. while (chunk->info.numArenasFreeCommitted && !cancel) { if (!chunk->decommitOneFreeArena(this, lock)) { // If we are low enough on memory that we can't update the page // tables, break out of the loop. break; } } } } // Do all possible decommit immediately from the current thread without // releasing the GC lock or allocating any memory. void GCRuntime::decommitFreeArenasWithoutUnlocking(const AutoLockGC& lock) { for (ChunkPool::Iter chunk(availableChunks(lock)); !chunk.done(); chunk.next()) { chunk->decommitFreeArenasWithoutUnlocking(lock); } MOZ_ASSERT(availableChunks(lock).verify()); } void GCRuntime::maybeRequestGCAfterBackgroundTask( const AutoLockHelperThreadState& lock) { if (requestSliceAfterBackgroundTask) { // Trigger a slice so the main thread can continue the collection // immediately. requestSliceAfterBackgroundTask = false; requestMajorGC(JS::GCReason::BG_TASK_FINISHED); } } void GCRuntime::cancelRequestedGCAfterBackgroundTask() { MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); #ifdef DEBUG { AutoLockHelperThreadState lock; MOZ_ASSERT(!requestSliceAfterBackgroundTask); } #endif majorGCTriggerReason.compareExchange(JS::GCReason::BG_TASK_FINISHED, JS::GCReason::NO_REASON); } void GCRuntime::sweepBackgroundThings(ZoneList& zones) { if (zones.isEmpty()) { return; } JSFreeOp fop(nullptr); // Sweep zones in order. The atoms zone must be finalized last as other // zones may have direct pointers into it. while (!zones.isEmpty()) { Zone* zone = zones.removeFront(); MOZ_ASSERT(zone->isGCFinished()); Arena* emptyArenas = zone->arenas.takeSweptEmptyArenas(); AutoSetThreadIsSweeping threadIsSweeping(zone); // We must finalize thing kinds in the order specified by // BackgroundFinalizePhases. for (auto phase : BackgroundFinalizePhases) { for (auto kind : phase.kinds) { Arena* arenas = zone->arenas.arenasToSweep(kind); MOZ_RELEASE_ASSERT(uintptr_t(arenas) != uintptr_t(-1)); if (arenas) { ArenaLists::backgroundFinalize(&fop, arenas, &emptyArenas); } } } // Release any arenas that are now empty. // // Empty arenas are only released after everything has been finalized so // that it's still possible to get a thing's zone after the thing has been // finalized. The HeapPtr destructor depends on this, and this allows // HeapPtrs between things of different alloc kind regardless of // finalization order. // // Periodically drop and reaquire the GC lock every so often to avoid // blocking the main thread from allocating chunks. static const size_t LockReleasePeriod = 32; while (emptyArenas) { AutoLockGC lock(this); for (size_t i = 0; i < LockReleasePeriod && emptyArenas; i++) { Arena* arena = emptyArenas; emptyArenas = emptyArenas->next; releaseArena(arena, lock); } } } } void GCRuntime::assertBackgroundSweepingFinished() { #ifdef DEBUG { AutoLockHelperThreadState lock; MOZ_ASSERT(backgroundSweepZones.ref().isEmpty()); } for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { for (auto i : AllAllocKinds()) { MOZ_ASSERT(!zone->arenas.arenasToSweep(i)); MOZ_ASSERT(zone->arenas.doneBackgroundFinalize(i)); } } #endif } void GCRuntime::queueZonesAndStartBackgroundSweep(ZoneList& zones) { { AutoLockHelperThreadState lock; MOZ_ASSERT(!requestSliceAfterBackgroundTask); backgroundSweepZones.ref().transferFrom(zones); if (sweepOnBackgroundThread) { sweepTask.startOrRunIfIdle(lock); } } if (!sweepOnBackgroundThread) { sweepTask.join(); sweepTask.runFromMainThread(); } } void BackgroundSweepTask::run(AutoLockHelperThreadState& lock) { AutoTraceLog logSweeping(TraceLoggerForCurrentThread(), TraceLogger_GCSweeping); gc->sweepFromBackgroundThread(lock); } void GCRuntime::sweepFromBackgroundThread(AutoLockHelperThreadState& lock) { do { ZoneList zones; zones.transferFrom(backgroundSweepZones.ref()); AutoUnlockHelperThreadState unlock(lock); sweepBackgroundThings(zones); // The main thread may call queueZonesAndStartBackgroundSweep() while this // is running so we must check there is no more work after releasing the // lock. } while (!backgroundSweepZones.ref().isEmpty()); maybeRequestGCAfterBackgroundTask(lock); } void GCRuntime::waitBackgroundSweepEnd() { sweepTask.join(); if (state() != State::Sweep) { assertBackgroundSweepingFinished(); } } void GCRuntime::queueUnusedLifoBlocksForFree(LifoAlloc* lifo) { MOZ_ASSERT(JS::RuntimeHeapIsBusy()); AutoLockHelperThreadState lock; lifoBlocksToFree.ref().transferUnusedFrom(lifo); } void GCRuntime::queueAllLifoBlocksForFree(LifoAlloc* lifo) { MOZ_ASSERT(JS::RuntimeHeapIsBusy()); AutoLockHelperThreadState lock; lifoBlocksToFree.ref().transferFrom(lifo); } void GCRuntime::queueAllLifoBlocksForFreeAfterMinorGC(LifoAlloc* lifo) { lifoBlocksToFreeAfterMinorGC.ref().transferFrom(lifo); } void GCRuntime::queueBuffersForFreeAfterMinorGC(Nursery::BufferSet& buffers) { AutoLockHelperThreadState lock; if (!buffersToFreeAfterMinorGC.ref().empty()) { // In the rare case that this hasn't processed the buffers from a previous // minor GC we have to wait here. MOZ_ASSERT(!freeTask.isIdle(lock)); freeTask.joinWithLockHeld(lock); } MOZ_ASSERT(buffersToFreeAfterMinorGC.ref().empty()); std::swap(buffersToFreeAfterMinorGC.ref(), buffers); } void GCRuntime::startBackgroundFree() { AutoLockHelperThreadState lock; freeTask.startOrRunIfIdle(lock); } void BackgroundFreeTask::run(AutoLockHelperThreadState& lock) { AutoTraceLog logFreeing(TraceLoggerForCurrentThread(), TraceLogger_GCFree); gc->freeFromBackgroundThread(lock); } void GCRuntime::freeFromBackgroundThread(AutoLockHelperThreadState& lock) { do { LifoAlloc lifoBlocks(JSContext::TEMP_LIFO_ALLOC_PRIMARY_CHUNK_SIZE); lifoBlocks.transferFrom(&lifoBlocksToFree.ref()); Nursery::BufferSet buffers; std::swap(buffers, buffersToFreeAfterMinorGC.ref()); AutoUnlockHelperThreadState unlock(lock); lifoBlocks.freeAll(); JSFreeOp* fop = TlsContext.get()->defaultFreeOp(); for (Nursery::BufferSet::Range r = buffers.all(); !r.empty(); r.popFront()) { // Malloc memory associated with nursery objects is not tracked as these // are assumed to be short lived. fop->freeUntracked(r.front()); } } while (!lifoBlocksToFree.ref().isEmpty() || !buffersToFreeAfterMinorGC.ref().empty()); } void GCRuntime::waitBackgroundFreeEnd() { freeTask.join(); } /* static */ bool UniqueIdGCPolicy::needsSweep(Cell** cellp, uint64_t*) { Cell* cell = *cellp; return MapGCThingTyped(cell, cell->getTraceKind(), [](auto t) { mozilla::DebugOnly prior = t; bool result = IsAboutToBeFinalizedUnbarriered(&t); // Sweep should not have to deal with moved pointers, since moving GC // handles updating the UID table manually. MOZ_ASSERT(t == prior); return result; }); } void JS::Zone::sweepUniqueIds() { uniqueIds().sweep(); } void Realm::destroy(JSFreeOp* fop) { JSRuntime* rt = fop->runtime(); if (auto callback = rt->destroyRealmCallback) { callback(fop, this); } if (principals()) { JS_DropPrincipals(rt->mainContextFromOwnThread(), principals()); } // Bug 1560019: Malloc memory associated with a zone but not with a specific // GC thing is not currently tracked. fop->deleteUntracked(this); } void Compartment::destroy(JSFreeOp* fop) { JSRuntime* rt = fop->runtime(); if (auto callback = rt->destroyCompartmentCallback) { callback(fop, this); } // Bug 1560019: Malloc memory associated with a zone but not with a specific // GC thing is not currently tracked. fop->deleteUntracked(this); rt->gc.stats().sweptCompartment(); } void Zone::destroy(JSFreeOp* fop) { MOZ_ASSERT(compartments().empty()); JSRuntime* rt = fop->runtime(); if (auto callback = rt->destroyZoneCallback) { callback(fop, this); } // Bug 1560019: Malloc memory associated with a zone but not with a specific // GC thing is not currently tracked. fop->deleteUntracked(this); fop->runtime()->gc.stats().sweptZone(); } /* * It's simpler if we preserve the invariant that every zone (except the atoms * zone) has at least one compartment, and every compartment has at least one * realm. If we know we're deleting the entire zone, then sweepCompartments is * allowed to delete all compartments. In this case, |keepAtleastOne| is false. * If any cells remain alive in the zone, set |keepAtleastOne| true to prohibit * sweepCompartments from deleting every compartment. Instead, it preserves an * arbitrary compartment in the zone. */ void Zone::sweepCompartments(JSFreeOp* fop, bool keepAtleastOne, bool destroyingRuntime) { MOZ_ASSERT(!compartments().empty()); MOZ_ASSERT_IF(destroyingRuntime, !keepAtleastOne); Compartment** read = compartments().begin(); Compartment** end = compartments().end(); Compartment** write = read; while (read < end) { Compartment* comp = *read++; /* * Don't delete the last compartment and realm if keepAtleastOne is * still true, meaning all the other compartments were deleted. */ bool keepAtleastOneRealm = read == end && keepAtleastOne; comp->sweepRealms(fop, keepAtleastOneRealm, destroyingRuntime); if (!comp->realms().empty()) { *write++ = comp; keepAtleastOne = false; } else { comp->destroy(fop); } } compartments().shrinkTo(write - compartments().begin()); MOZ_ASSERT_IF(keepAtleastOne, !compartments().empty()); MOZ_ASSERT_IF(destroyingRuntime, compartments().empty()); } void Compartment::sweepRealms(JSFreeOp* fop, bool keepAtleastOne, bool destroyingRuntime) { MOZ_ASSERT(!realms().empty()); MOZ_ASSERT_IF(destroyingRuntime, !keepAtleastOne); Realm** read = realms().begin(); Realm** end = realms().end(); Realm** write = read; while (read < end) { Realm* realm = *read++; /* * Don't delete the last realm if keepAtleastOne is still true, meaning * all the other realms were deleted. */ bool dontDelete = read == end && keepAtleastOne; if ((realm->marked() || dontDelete) && !destroyingRuntime) { *write++ = realm; keepAtleastOne = false; } else { realm->destroy(fop); } } realms().shrinkTo(write - realms().begin()); MOZ_ASSERT_IF(keepAtleastOne, !realms().empty()); MOZ_ASSERT_IF(destroyingRuntime, realms().empty()); } void GCRuntime::deleteEmptyZone(Zone* zone) { MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); MOZ_ASSERT(zone->compartments().empty()); for (auto& i : zones()) { if (i == zone) { zones().erase(&i); zone->destroy(rt->defaultFreeOp()); return; } } MOZ_CRASH("Zone not found"); } void GCRuntime::sweepZones(JSFreeOp* fop, bool destroyingRuntime) { MOZ_ASSERT_IF(destroyingRuntime, numActiveZoneIters == 0); if (numActiveZoneIters) { return; } assertBackgroundSweepingFinished(); Zone** read = zones().begin(); Zone** end = zones().end(); Zone** write = read; while (read < end) { Zone* zone = *read++; if (zone->wasGCStarted()) { MOZ_ASSERT(!zone->isQueuedForBackgroundSweep()); const bool zoneIsDead = zone->arenas.arenaListsAreEmpty() && !zone->hasMarkedRealms(); MOZ_ASSERT_IF(destroyingRuntime, zoneIsDead); if (zoneIsDead) { AutoSetThreadIsSweeping threadIsSweeping(zone); zone->arenas.checkEmptyFreeLists(); zone->sweepCompartments(fop, false, destroyingRuntime); MOZ_ASSERT(zone->compartments().empty()); MOZ_ASSERT(zone->typeDescrObjects().empty()); zone->destroy(fop); continue; } zone->sweepCompartments(fop, true, destroyingRuntime); } *write++ = zone; } zones().shrinkTo(write - zones().begin()); } void ArenaLists::checkEmptyArenaList(AllocKind kind) { MOZ_ASSERT(arenaList(kind).isEmpty()); } class MOZ_RAII AutoRunParallelTask : public GCParallelTask { // This class takes a pointer to a member function of GCRuntime. using TaskFunc = JS_MEMBER_FN_PTR_TYPE(GCRuntime, void); TaskFunc func_; gcstats::PhaseKind phase_; AutoLockHelperThreadState& lock_; public: AutoRunParallelTask(GCRuntime* gc, TaskFunc func, gcstats::PhaseKind phase, AutoLockHelperThreadState& lock) : GCParallelTask(gc), func_(func), phase_(phase), lock_(lock) { gc->startTask(*this, phase_, lock_); } ~AutoRunParallelTask() { gc->joinTask(*this, phase_, lock_); } void run(AutoLockHelperThreadState& lock) override { AutoUnlockHelperThreadState unlock(lock); // The hazard analysis can't tell what the call to func_ will do but it's // not allowed to GC. JS::AutoSuppressGCAnalysis nogc; // Call pointer to member function on |gc|. JS_CALL_MEMBER_FN_PTR(gc, func_); } }; void GCRuntime::purgeRuntimeForMinorGC() { // If external strings become nursery allocable, remember to call // zone->externalStringCache().purge() (and delete this assert.) MOZ_ASSERT(!IsNurseryAllocable(AllocKind::EXTERNAL_STRING)); for (ZonesIter zone(this, SkipAtoms); !zone.done(); zone.next()) { zone->functionToStringCache().purge(); } rt->caches().purgeForMinorGC(rt); } void GCRuntime::purgeRuntime() { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::PURGE); for (GCRealmsIter realm(rt); !realm.done(); realm.next()) { realm->purge(); } for (GCZonesIter zone(this); !zone.done(); zone.next()) { zone->purgeAtomCache(); zone->externalStringCache().purge(); zone->functionToStringCache().purge(); } JSContext* cx = rt->mainContextFromOwnThread(); queueUnusedLifoBlocksForFree(&cx->tempLifoAlloc()); cx->interpreterStack().purge(rt); cx->frontendCollectionPool().purge(); rt->caches().purge(); if (auto cache = rt->maybeThisRuntimeSharedImmutableStrings()) { cache->purge(); } MOZ_ASSERT(unmarkGrayStack.empty()); unmarkGrayStack.clearAndFree(); // If we're the main runtime, tell helper threads to free their unused // memory when they are next idle. if (!rt->parentRuntime) { HelperThreadState().triggerFreeUnusedMemory(); } } bool GCRuntime::shouldPreserveJITCode(Realm* realm, const TimeStamp& currentTime, JS::GCReason reason, bool canAllocateMoreCode) { if (cleanUpEverything) { return false; } if (!canAllocateMoreCode) { return false; } if (alwaysPreserveCode) { return true; } if (realm->preserveJitCode()) { return true; } if (IsCurrentlyAnimating(realm->lastAnimationTime, currentTime)) { return true; } if (reason == JS::GCReason::DEBUG_GC) { return true; } return false; } #ifdef DEBUG class CompartmentCheckTracer final : public JS::CallbackTracer { void onChild(const JS::GCCellPtr& thing) override; public: explicit CompartmentCheckTracer(JSRuntime* rt) : JS::CallbackTracer(rt, JS::TracerKind::Callback, JS::WeakEdgeTraceAction::Skip), src(nullptr), zone(nullptr), compartment(nullptr) {} Cell* src; JS::TraceKind srcKind; Zone* zone; Compartment* compartment; }; static bool InCrossCompartmentMap(JSRuntime* rt, JSObject* src, JS::GCCellPtr dst) { // Cross compartment edges are either in the cross compartment map or in a // debugger weakmap. Compartment* srccomp = src->compartment(); if (dst.is()) { if (ObjectWrapperMap::Ptr p = srccomp->lookupWrapper(&dst.as())) { if (*p->value().unsafeGet() == src) { return true; } } } if (DebugAPI::edgeIsInDebuggerWeakmap(rt, src, dst)) { return true; } return false; } void CompartmentCheckTracer::onChild(const JS::GCCellPtr& thing) { Compartment* comp = MapGCThingTyped(thing, [](auto t) { return t->maybeCompartment(); }); if (comp && compartment) { MOZ_ASSERT( comp == compartment || (srcKind == JS::TraceKind::Object && InCrossCompartmentMap(runtime(), static_cast(src), thing))); } else { TenuredCell* tenured = &thing.asCell()->asTenured(); Zone* thingZone = tenured->zoneFromAnyThread(); MOZ_ASSERT(thingZone == zone || thingZone->isAtomsZone()); } } void GCRuntime::checkForCompartmentMismatches() { JSContext* cx = rt->mainContextFromOwnThread(); if (cx->disableStrictProxyCheckingCount) { return; } CompartmentCheckTracer trc(rt); AutoAssertEmptyNursery empty(cx); for (ZonesIter zone(this, SkipAtoms); !zone.done(); zone.next()) { trc.zone = zone; for (auto thingKind : AllAllocKinds()) { for (auto i = zone->cellIterUnsafe(thingKind, empty); !i.done(); i.next()) { trc.src = i.getCell(); trc.srcKind = MapAllocToTraceKind(thingKind); trc.compartment = MapGCThingTyped( trc.src, trc.srcKind, [](auto t) { return t->maybeCompartment(); }); JS::TraceChildren(&trc, JS::GCCellPtr(trc.src, trc.srcKind)); } } } } #endif static void RelazifyFunctions(Zone* zone, AllocKind kind) { MOZ_ASSERT(kind == AllocKind::FUNCTION || kind == AllocKind::FUNCTION_EXTENDED); JSRuntime* rt = zone->runtimeFromMainThread(); AutoAssertEmptyNursery empty(rt->mainContextFromOwnThread()); for (auto i = zone->cellIterUnsafe(kind, empty); !i.done(); i.next()) { JSFunction* fun = &i->as(); // When iterating over the GC-heap, we may encounter function objects that // are incomplete (missing a BaseScript when we expect one). We must check // for this case before we can call JSFunction::hasBytecode(). if (fun->isIncomplete()) { continue; } if (fun->hasBytecode()) { fun->maybeRelazify(rt); } } } static bool ShouldCollectZone(Zone* zone, JS::GCReason reason) { // If we are repeating a GC because we noticed dead compartments haven't // been collected, then only collect zones containing those compartments. if (reason == JS::GCReason::COMPARTMENT_REVIVED) { for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next()) { if (comp->gcState.scheduledForDestruction) { return true; } } return false; } // Otherwise we only collect scheduled zones. if (!zone->isGCScheduled()) { return false; } // If canCollectAtoms() is false then parsing is currently happening on // another thread, in which case we don't have information about which atoms // are roots, so we must skip collecting atoms. // // Note that only affects the first slice of an incremental GC since root // marking is completed before we return to the mutator. // // Off-thread parsing is inhibited after the start of GC which prevents // races between creating atoms during parsing and sweeping atoms on the // main thread. // // Otherwise, we always schedule a GC in the atoms zone so that atoms which // the other collected zones are using are marked, and we can update the // set of atoms in use by the other collected zones at the end of the GC. if (zone->isAtomsZone()) { return TlsContext.get()->canCollectAtoms(); } return zone->canCollect(); } bool GCRuntime::prepareZonesForCollection(JS::GCReason reason, bool* isFullOut) { #ifdef DEBUG /* Assert that zone state is as we expect */ for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { MOZ_ASSERT(!zone->isCollecting()); MOZ_ASSERT_IF(!zone->isAtomsZone(), !zone->compartments().empty()); for (auto i : AllAllocKinds()) { MOZ_ASSERT(!zone->arenas.arenasToSweep(i)); } } #endif *isFullOut = true; bool any = false; for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { /* Set up which zones will be collected. */ bool shouldCollect = ShouldCollectZone(zone, reason); if (shouldCollect) { MOZ_ASSERT(zone->canCollect()); any = true; zone->changeGCState(Zone::NoGC, Zone::Prepare); } else if (zone->canCollect()) { *isFullOut = false; } zone->setWasCollected(shouldCollect); } /* * Check that we do collect the atoms zone if we triggered a GC for that * purpose. */ MOZ_ASSERT_IF(reason == JS::GCReason::DELAYED_ATOMS_GC, atomsZone->isGCPreparing()); /* Check that at least one zone is scheduled for collection. */ return any; } void GCRuntime::discardJITCodeForGC() { js::CancelOffThreadIonCompile(rt, JS::Zone::Prepare); for (GCZonesIter zone(this); !zone.done(); zone.next()) { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::MARK_DISCARD_CODE); zone->discardJitCode(rt->defaultFreeOp(), Zone::DiscardBaselineCode, Zone::DiscardJitScripts); } } void GCRuntime::relazifyFunctionsForShrinkingGC() { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::RELAZIFY_FUNCTIONS); for (GCZonesIter zone(this); !zone.done(); zone.next()) { if (zone->isSelfHostingZone()) { continue; } RelazifyFunctions(zone, AllocKind::FUNCTION); RelazifyFunctions(zone, AllocKind::FUNCTION_EXTENDED); } } void GCRuntime::purgeShapeCachesForShrinkingGC() { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::PURGE_SHAPE_CACHES); for (GCZonesIter zone(this); !zone.done(); zone.next()) { if (!canRelocateZone(zone) || zone->keepShapeCaches()) { continue; } for (auto baseShape = zone->cellIterUnsafe(); !baseShape.done(); baseShape.next()) { baseShape->maybePurgeCache(rt->defaultFreeOp()); } } } // The debugger keeps track of the URLs for the sources of each realm's scripts. // These URLs are purged on shrinking GCs. void GCRuntime::purgeSourceURLsForShrinkingGC() { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::PURGE_SOURCE_URLS); for (GCZonesIter zone(this); !zone.done(); zone.next()) { // URLs are not tracked for realms in the system zone. if (!canRelocateZone(zone) || zone->isSystemZone()) { continue; } for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next()) { for (RealmsInCompartmentIter realm(comp); !realm.done(); realm.next()) { GlobalObject* global = realm.get()->unsafeUnbarrieredMaybeGlobal(); if (global) { global->clearSourceURLSHolder(); } } } } } void GCRuntime::unmarkWeakMaps() { for (GCZonesIter zone(this); !zone.done(); zone.next()) { /* Unmark all weak maps in the zones being collected. */ WeakMapBase::unmarkZone(zone); } } bool GCRuntime::beginPreparePhase(JS::GCReason reason, AutoGCSession& session) { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::PREPARE); if (!prepareZonesForCollection(reason, &isFull.ref())) { return false; } /* Check it's safe to access the atoms zone if we are collecting it. */ if (atomsZone->isCollecting()) { session.maybeCheckAtomsAccess.emplace(rt); } /* * Start a parallel task to clear all mark state for the zones we are * collecting. This is linear in the size of the heap we are collecting and so * can be slow. This happens concurrently with the mutator and GC proper does * not start until this is complete. */ setParallelUnmarkEnabled(true); unmarkTask.initZones(); unmarkTask.start(); /* * Process any queued source compressions during the start of a major * GC. */ if (!IsShutdownReason(reason) && reason != JS::GCReason::ROOTS_REMOVED && reason != JS::GCReason::XPCONNECT_SHUTDOWN) { StartHandlingCompressionsOnGC(rt); } return true; } void BackgroundUnmarkTask::initZones() { MOZ_ASSERT(isIdle()); MOZ_ASSERT(zones.empty()); MOZ_ASSERT(!isCancelled()); // We can't safely iterate the zones vector from another thread so we copy the // zones to be collected into another vector. AutoEnterOOMUnsafeRegion oomUnsafe; for (GCZonesIter zone(gc); !zone.done(); zone.next()) { if (!zones.append(zone.get())) { oomUnsafe.crash("BackgroundUnmarkTask::initZones"); } } } void BackgroundUnmarkTask::run(AutoLockHelperThreadState& helperTheadLock) { AutoUnlockHelperThreadState unlock(helperTheadLock); AutoTraceLog log(TraceLoggerForCurrentThread(), TraceLogger_GCUnmarking); // We need to hold the GC lock while traversing the arena lists. AutoLockGC gcLock(gc); unmarkZones(gcLock); zones.clear(); } void BackgroundUnmarkTask::unmarkZones(AutoLockGC& lock) { for (Zone* zone : zones) { for (auto kind : AllAllocKinds()) { for (ArenaIter arena(zone, kind); !arena.done(); arena.next()) { AutoUnlockGC unlock(lock); arena->unmarkAll(); if (isCancelled()) { return; } } } } } void GCRuntime::endPreparePhase(JS::GCReason reason) { MOZ_ASSERT(unmarkTask.isIdle()); setParallelUnmarkEnabled(false); for (GCZonesIter zone(this); !zone.done(); zone.next()) { /* * In an incremental GC, clear the area free lists to ensure that subsequent * allocations refill them and end up marking new cells back. See * arenaAllocatedDuringGC(). */ zone->arenas.clearFreeLists(); zone->arenas.checkGCStateNotInUse(); zone->markedStrings = 0; zone->finalizedStrings = 0; zone->setPreservingCode(false); #ifdef JS_GC_ZEAL if (hasZealMode(ZealMode::YieldBeforeRootMarking)) { for (auto kind : AllAllocKinds()) { for (ArenaIter arena(zone, kind); !arena.done(); arena.next()) { arena->checkNoMarkedCells(); } } } #endif } // Discard JIT code more aggressively if the process is approaching its // executable code limit. bool canAllocateMoreCode = jit::CanLikelyAllocateMoreExecutableMemory(); auto currentTime = ReallyNow(); for (CompartmentsIter c(rt); !c.done(); c.next()) { c->gcState.scheduledForDestruction = false; c->gcState.maybeAlive = false; c->gcState.hasEnteredRealm = false; for (RealmsInCompartmentIter r(c); !r.done(); r.next()) { if (r->shouldTraceGlobal() || !r->zone()->isGCScheduled()) { c->gcState.maybeAlive = true; } if (shouldPreserveJITCode(r, currentTime, reason, canAllocateMoreCode)) { r->zone()->setPreservingCode(true); } if (r->hasBeenEnteredIgnoringJit()) { c->gcState.hasEnteredRealm = true; } } } if (!cleanUpEverything && canAllocateMoreCode) { jit::JitActivationIterator activation(rt->mainContextFromOwnThread()); if (!activation.done()) { activation->compartment()->zone()->setPreservingCode(true); } } /* * Perform remaining preparation work that must take place in the first true * GC slice. */ { gcstats::AutoPhase ap1(stats(), gcstats::PhaseKind::PREPARE); AutoLockHelperThreadState helperLock; /* Clear mark state for WeakMaps in parallel with other work. */ AutoRunParallelTask unmarkWeakMaps(this, &GCRuntime::unmarkWeakMaps, gcstats::PhaseKind::UNMARK_WEAKMAPS, helperLock); /* * Buffer gray roots for incremental collections. This is linear in the * number of roots which can be in the tens of thousands. Do this in * parallel with the rest of this block. */ Maybe bufferGrayRootsTask; if (isIncremental) { bufferGrayRootsTask.emplace(this, &GCRuntime::bufferGrayRoots, gcstats::PhaseKind::BUFFER_GRAY_ROOTS, helperLock); } AutoUnlockHelperThreadState unlock(helperLock); // Discard JIT code. For incremental collections, the sweep phase will // also discard JIT code. discardJITCodeForGC(); startBackgroundFreeAfterMinorGC(); /* * Relazify functions after discarding JIT code (we can't relazify * functions with JIT code) and before the actual mark phase, so that * the current GC can collect the JSScripts we're unlinking here. We do * this only when we're performing a shrinking GC, as too much * relazification can cause performance issues when we have to reparse * the same functions over and over. */ if (invocationKind == GC_SHRINK) { relazifyFunctionsForShrinkingGC(); purgeShapeCachesForShrinkingGC(); purgeSourceURLsForShrinkingGC(); } /* * We must purge the runtime at the beginning of an incremental GC. The * danger if we purge later is that the snapshot invariant of * incremental GC will be broken, as follows. If some object is * reachable only through some cache (say the dtoaCache) then it will * not be part of the snapshot. If we purge after root marking, then * the mutator could obtain a pointer to the object and start using * it. This object might never be marked, so a GC hazard would exist. */ purgeRuntime(); if (IsShutdownReason(reason)) { /* Clear any engine roots that may hold external data live. */ for (GCZonesIter zone(this); !zone.done(); zone.next()) { zone->clearRootsForShutdownGC(); } } } #ifdef DEBUG if (fullCompartmentChecks) { checkForCompartmentMismatches(); } #endif } void GCRuntime::beginMarkPhase(AutoGCSession& session) { /* * Mark phase. */ gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::MARK); // This is the slice we actually start collecting. The number can be used to // check whether a major GC has started so we must not increment it until we // get here. incMajorGcNumber(); marker.start(); GCMarker* gcmarker = ▮ gcmarker->clearMarkCount(); for (GCZonesIter zone(this); !zone.done(); zone.next()) { // Incremental marking barriers are enabled at this point. zone->changeGCState(Zone::Prepare, Zone::MarkBlackOnly); } if (rt->isBeingDestroyed()) { checkNoRuntimeRoots(session); } else { traceRuntimeForMajorGC(gcmarker, session); } if (isIncremental) { findDeadCompartments(); } updateMemoryCountersOnGCStart(); stats().measureInitialHeapSize(); } void GCRuntime::findDeadCompartments() { gcstats::AutoPhase ap1(stats(), gcstats::PhaseKind::MARK_ROOTS); gcstats::AutoPhase ap2(stats(), gcstats::PhaseKind::MARK_COMPARTMENTS); /* * This code ensures that if a compartment is "dead", then it will be * collected in this GC. A compartment is considered dead if its maybeAlive * flag is false. The maybeAlive flag is set if: * * (1) the compartment has been entered (set in beginMarkPhase() above) * (2) the compartment's zone is not being collected (set in * beginMarkPhase() above) * (3) an object in the compartment was marked during root marking, either * as a black root or a gray root (set in RootMarking.cpp), or * (4) the compartment has incoming cross-compartment edges from another * compartment that has maybeAlive set (set by this method). * * If the maybeAlive is false, then we set the scheduledForDestruction flag. * At the end of the GC, we look for compartments where * scheduledForDestruction is true. These are compartments that were somehow * "revived" during the incremental GC. If any are found, we do a special, * non-incremental GC of those compartments to try to collect them. * * Compartments can be revived for a variety of reasons. On reason is bug * 811587, where a reflector that was dead can be revived by DOM code that * still refers to the underlying DOM node. * * Read barriers and allocations can also cause revival. This might happen * during a function like JS_TransplantObject, which iterates over all * compartments, live or dead, and operates on their objects. See bug 803376 * for details on this problem. To avoid the problem, we try to avoid * allocation and read barriers during JS_TransplantObject and the like. */ // Propagate the maybeAlive flag via cross-compartment edges. Vector workList; for (CompartmentsIter comp(rt); !comp.done(); comp.next()) { if (comp->gcState.maybeAlive) { if (!workList.append(comp)) { return; } } } while (!workList.empty()) { Compartment* comp = workList.popCopy(); for (Compartment::WrappedObjectCompartmentEnum e(comp); !e.empty(); e.popFront()) { Compartment* dest = e.front(); if (!dest->gcState.maybeAlive) { dest->gcState.maybeAlive = true; if (!workList.append(dest)) { return; } } } } // Set scheduledForDestruction based on maybeAlive. for (GCCompartmentsIter comp(rt); !comp.done(); comp.next()) { MOZ_ASSERT(!comp->gcState.scheduledForDestruction); if (!comp->gcState.maybeAlive) { comp->gcState.scheduledForDestruction = true; } } } void GCRuntime::updateMemoryCountersOnGCStart() { heapSize.updateOnGCStart(); // Update memory counters for the zones we are collecting. for (GCZonesIter zone(this); !zone.done(); zone.next()) { zone->updateMemoryCountersOnGCStart(); } } template IncrementalProgress GCRuntime::markWeakReferences( SliceBudget& incrementalBudget) { gcstats::AutoPhase ap1(stats(), gcstats::PhaseKind::SWEEP_MARK_WEAK); auto unlimited = SliceBudget::unlimited(); SliceBudget& budget = marker.incrementalWeakMapMarkingEnabled ? incrementalBudget : unlimited; // We may have already entered weak marking mode. if (!marker.isWeakMarking() && marker.enterWeakMarkingMode()) { // Do not rely on the information about not-yet-marked weak keys that have // been collected by barriers. Clear out the gcWeakKeys entries and rebuild // the full table. Note that this a cross-zone operation; delegate zone // entries will be populated by map zone traversals, so everything needs to // be cleared first, then populated. if (!marker.incrementalWeakMapMarkingEnabled) { for (ZoneIterT zone(this); !zone.done(); zone.next()) { AutoEnterOOMUnsafeRegion oomUnsafe; if (!zone->gcWeakKeys().clear()) { oomUnsafe.crash("clearing weak keys when entering weak marking mode"); } } } for (ZoneIterT zone(this); !zone.done(); zone.next()) { if (zone->enterWeakMarkingMode(&marker, budget) == NotFinished) { MOZ_ASSERT(marker.incrementalWeakMapMarkingEnabled); marker.leaveWeakMarkingMode(); return NotFinished; } } } #ifdef DEBUG for (ZoneIterT zone(this); !zone.done(); zone.next()) { zone->checkWeakMarkingMode(); } #endif // This is not strictly necessary; if we yield here, we could run the mutator // in weak marking mode and unmark gray would end up doing the key lookups. // But it seems better to not slow down barriers. Re-entering weak marking // mode will be fast since already-processed markables have been removed. auto leaveOnExit = mozilla::MakeScopeExit([&] { marker.leaveWeakMarkingMode(); }); bool markedAny = true; while (markedAny) { if (!marker.markUntilBudgetExhausted(budget)) { MOZ_ASSERT(marker.incrementalWeakMapMarkingEnabled); return NotFinished; } markedAny = false; if (!marker.isWeakMarking()) { for (ZoneIterT zone(this); !zone.done(); zone.next()) { markedAny |= WeakMapBase::markZoneIteratively(zone, &marker); } } markedAny |= jit::JitRuntime::MarkJitcodeGlobalTableIteratively(&marker); } MOZ_ASSERT(marker.isDrained()); return Finished; } IncrementalProgress GCRuntime::markWeakReferencesInCurrentGroup( SliceBudget& budget) { return markWeakReferences(budget); } template void GCRuntime::markGrayRoots(gcstats::PhaseKind phase) { MOZ_ASSERT(marker.markColor() == MarkColor::Gray); gcstats::AutoPhase ap(stats(), phase); if (hasValidGrayRootsBuffer()) { for (ZoneIterT zone(this); !zone.done(); zone.next()) { markBufferedGrayRoots(zone); } } else { MOZ_ASSERT(!isIncremental); traceEmbeddingGrayRoots(&marker); Compartment::traceIncomingCrossCompartmentEdgesForZoneGC( &marker, Compartment::GrayEdges); } } IncrementalProgress GCRuntime::markAllWeakReferences() { SliceBudget budget = SliceBudget::unlimited(); return markWeakReferences(budget); } void GCRuntime::markAllGrayReferences(gcstats::PhaseKind phase) { markGrayRoots(phase); drainMarkStack(); } void GCRuntime::dropStringWrappers() { /* * String "wrappers" are dropped on GC because their presence would require * us to sweep the wrappers in all compartments every time we sweep a * compartment group. */ for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { zone->dropStringWrappersOnGC(); } } /* * Group zones that must be swept at the same time. * * From the point of view of the mutator, groups of zones transition atomically * from marking to sweeping. If compartment A has an edge to an unmarked object * in compartment B, then we must not start sweeping A in a later slice than we * start sweeping B. That's because a write barrier in A could lead to the * unmarked object in B becoming marked. However, if we had already swept that * object, we would be in trouble. * * If we consider these dependencies as a graph, then all the compartments in * any strongly-connected component of this graph must start sweeping in the * same slice. * * Tarjan's algorithm is used to calculate the components. */ bool Compartment::findSweepGroupEdges() { Zone* source = zone(); for (WrappedObjectCompartmentEnum e(this); !e.empty(); e.popFront()) { Compartment* targetComp = e.front(); Zone* target = targetComp->zone(); if (!target->isGCMarking() || source->hasSweepGroupEdgeTo(target)) { continue; } for (ObjectWrapperEnum e(this, targetComp); !e.empty(); e.popFront()) { JSObject* key = e.front().mutableKey(); MOZ_ASSERT(key->zone() == target); // Add an edge to the wrapped object's zone to ensure that the wrapper // zone is not still being marked when we start sweeping the wrapped zone. // As an optimization, if the wrapped object is already marked black there // is no danger of later marking and we can skip this. if (key->isMarkedBlack()) { continue; } if (!source->addSweepGroupEdgeTo(target)) { return false; } // We don't need to consider any more wrappers for this target // compartment since we already added an edge. break; } } return true; } bool Zone::findSweepGroupEdges(Zone* atomsZone) { // Any zone may have a pointer to an atom in the atoms zone, and these aren't // in the cross compartment map. if (atomsZone->wasGCStarted() && !addSweepGroupEdgeTo(atomsZone)) { return false; } for (CompartmentsInZoneIter comp(this); !comp.done(); comp.next()) { if (!comp->findSweepGroupEdges()) { return false; } } return WeakMapBase::findSweepGroupEdgesForZone(this); } static bool AddEdgesForMarkQueue(GCMarker& marker) { #ifdef DEBUG // For testing only. // // Add edges between all objects mentioned in the test mark queue, since // otherwise they will get marked in a different order than their sweep // groups. Note that this is only done at the beginning of an incremental // collection, so it is possible for objects to be added later that do not // follow the sweep group ordering. These objects will wait until their sweep // group comes up, or will be skipped if their sweep group is already past. JS::Zone* prevZone = nullptr; for (size_t i = 0; i < marker.markQueue.length(); i++) { Value val = marker.markQueue[i].get().unbarrieredGet(); if (!val.isObject()) { continue; } JSObject* obj = &val.toObject(); JS::Zone* zone = obj->zone(); if (!zone->isGCMarking()) { continue; } if (prevZone && prevZone != zone) { if (!prevZone->addSweepGroupEdgeTo(zone)) { return false; } } prevZone = zone; } #endif return true; } bool GCRuntime::findSweepGroupEdges() { for (GCZonesIter zone(this); !zone.done(); zone.next()) { if (!zone->findSweepGroupEdges(atomsZone)) { return false; } } if (!AddEdgesForMarkQueue(marker)) { return false; } return DebugAPI::findSweepGroupEdges(rt); } void GCRuntime::groupZonesForSweeping(JS::GCReason reason) { #ifdef DEBUG for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { MOZ_ASSERT(zone->gcSweepGroupEdges().empty()); } #endif JSContext* cx = rt->mainContextFromOwnThread(); ZoneComponentFinder finder(cx->nativeStackLimit[JS::StackForSystemCode]); if (!isIncremental || !findSweepGroupEdges()) { finder.useOneComponent(); } // Use one component for two-slice zeal modes. if (useZeal && hasIncrementalTwoSliceZealMode()) { finder.useOneComponent(); } for (GCZonesIter zone(this); !zone.done(); zone.next()) { MOZ_ASSERT(zone->isGCMarking()); finder.addNode(zone); } sweepGroups = finder.getResultsList(); currentSweepGroup = sweepGroups; sweepGroupIndex = 1; for (GCZonesIter zone(this); !zone.done(); zone.next()) { zone->clearSweepGroupEdges(); } #ifdef DEBUG unsigned idx = sweepGroupIndex; for (Zone* head = currentSweepGroup; head; head = head->nextGroup()) { for (Zone* zone = head; zone; zone = zone->nextNodeInGroup()) { MOZ_ASSERT(zone->isGCMarking()); zone->gcSweepGroupIndex = idx; } idx++; } MOZ_ASSERT_IF(!isIncremental, !currentSweepGroup->nextGroup()); for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { MOZ_ASSERT(zone->gcSweepGroupEdges().empty()); } #endif } static void ResetGrayList(Compartment* comp); void GCRuntime::getNextSweepGroup() { currentSweepGroup = currentSweepGroup->nextGroup(); ++sweepGroupIndex; if (!currentSweepGroup) { abortSweepAfterCurrentGroup = false; return; } MOZ_ASSERT_IF(abortSweepAfterCurrentGroup, !isIncremental); if (!isIncremental) { ZoneComponentFinder::mergeGroups(currentSweepGroup); } for (Zone* zone = currentSweepGroup; zone; zone = zone->nextNodeInGroup()) { MOZ_ASSERT(zone->isGCMarkingBlackOnly()); MOZ_ASSERT(!zone->isQueuedForBackgroundSweep()); } if (abortSweepAfterCurrentGroup) { joinTask(markTask, gcstats::PhaseKind::SWEEP_MARK); // Abort collection of subsequent sweep groups. for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { MOZ_ASSERT(!zone->gcNextGraphComponent); zone->changeGCState(Zone::MarkBlackOnly, Zone::NoGC); zone->arenas.unmarkPreMarkedFreeCells(); zone->arenas.mergeNewArenasInMarkPhase(); zone->gcGrayRoots().Clear(); zone->clearGCSliceThresholds(); } for (SweepGroupCompartmentsIter comp(rt); !comp.done(); comp.next()) { ResetGrayList(comp); } abortSweepAfterCurrentGroup = false; currentSweepGroup = nullptr; } hasMarkedGrayRoots = false; } /* * Gray marking: * * At the end of collection, anything reachable from a gray root that has not * otherwise been marked black must be marked gray. * * This means that when marking things gray we must not allow marking to leave * the current compartment group, as that could result in things being marked * gray when they might subsequently be marked black. To achieve this, when we * find a cross compartment pointer we don't mark the referent but add it to a * singly-linked list of incoming gray pointers that is stored with each * compartment. * * The list head is stored in Compartment::gcIncomingGrayPointers and contains * cross compartment wrapper objects. The next pointer is stored in the second * extra slot of the cross compartment wrapper. * * The list is created during gray marking when one of the * MarkCrossCompartmentXXX functions is called for a pointer that leaves the * current compartent group. This calls DelayCrossCompartmentGrayMarking to * push the referring object onto the list. * * The list is traversed and then unlinked in * GCRuntime::markIncomingCrossCompartmentPointers. */ static bool IsGrayListObject(JSObject* obj) { MOZ_ASSERT(obj); return obj->is() && !IsDeadProxyObject(obj); } /* static */ unsigned ProxyObject::grayLinkReservedSlot(JSObject* obj) { MOZ_ASSERT(IsGrayListObject(obj)); return CrossCompartmentWrapperObject::GrayLinkReservedSlot; } #ifdef DEBUG static void AssertNotOnGrayList(JSObject* obj) { MOZ_ASSERT_IF( IsGrayListObject(obj), GetProxyReservedSlot(obj, ProxyObject::grayLinkReservedSlot(obj)) .isUndefined()); } #endif static void AssertNoWrappersInGrayList(JSRuntime* rt) { #ifdef DEBUG for (CompartmentsIter c(rt); !c.done(); c.next()) { MOZ_ASSERT(!c->gcIncomingGrayPointers); for (Compartment::ObjectWrapperEnum e(c); !e.empty(); e.popFront()) { AssertNotOnGrayList(e.front().value().unbarrieredGet()); } } #endif } static JSObject* CrossCompartmentPointerReferent(JSObject* obj) { MOZ_ASSERT(IsGrayListObject(obj)); return &obj->as().private_().toObject(); } static JSObject* NextIncomingCrossCompartmentPointer(JSObject* prev, bool unlink) { unsigned slot = ProxyObject::grayLinkReservedSlot(prev); JSObject* next = GetProxyReservedSlot(prev, slot).toObjectOrNull(); MOZ_ASSERT_IF(next, IsGrayListObject(next)); if (unlink) { SetProxyReservedSlot(prev, slot, UndefinedValue()); } return next; } void js::gc::DelayCrossCompartmentGrayMarking(JSObject* src) { MOZ_ASSERT(IsGrayListObject(src)); MOZ_ASSERT(src->isMarkedGray()); AutoTouchingGrayThings tgt; /* Called from MarkCrossCompartmentXXX functions. */ unsigned slot = ProxyObject::grayLinkReservedSlot(src); JSObject* dest = CrossCompartmentPointerReferent(src); Compartment* comp = dest->compartment(); if (GetProxyReservedSlot(src, slot).isUndefined()) { SetProxyReservedSlot(src, slot, ObjectOrNullValue(comp->gcIncomingGrayPointers)); comp->gcIncomingGrayPointers = src; } else { MOZ_ASSERT(GetProxyReservedSlot(src, slot).isObjectOrNull()); } #ifdef DEBUG /* * Assert that the object is in our list, also walking the list to check its * integrity. */ JSObject* obj = comp->gcIncomingGrayPointers; bool found = false; while (obj) { if (obj == src) { found = true; } obj = NextIncomingCrossCompartmentPointer(obj, false); } MOZ_ASSERT(found); #endif } void GCRuntime::markIncomingCrossCompartmentPointers(MarkColor color) { gcstats::AutoPhase ap(stats(), color == MarkColor::Black ? gcstats::PhaseKind::SWEEP_MARK_INCOMING_BLACK : gcstats::PhaseKind::SWEEP_MARK_INCOMING_GRAY); bool unlinkList = color == MarkColor::Gray; for (SweepGroupCompartmentsIter c(rt); !c.done(); c.next()) { MOZ_ASSERT(c->zone()->isGCMarking()); MOZ_ASSERT_IF(color == MarkColor::Gray, c->zone()->isGCMarkingBlackAndGray()); MOZ_ASSERT_IF(c->gcIncomingGrayPointers, IsGrayListObject(c->gcIncomingGrayPointers)); for (JSObject* src = c->gcIncomingGrayPointers; src; src = NextIncomingCrossCompartmentPointer(src, unlinkList)) { JSObject* dst = CrossCompartmentPointerReferent(src); MOZ_ASSERT(dst->compartment() == c); if (color == MarkColor::Gray) { if (src->asTenured().isMarkedGray()) { TraceManuallyBarrieredEdge(&marker, &dst, "cross-compartment gray pointer"); } } else { if (src->asTenured().isMarkedBlack()) { TraceManuallyBarrieredEdge(&marker, &dst, "cross-compartment black pointer"); } } } if (unlinkList) { c->gcIncomingGrayPointers = nullptr; } } } static bool RemoveFromGrayList(JSObject* wrapper) { AutoTouchingGrayThings tgt; if (!IsGrayListObject(wrapper)) { return false; } unsigned slot = ProxyObject::grayLinkReservedSlot(wrapper); if (GetProxyReservedSlot(wrapper, slot).isUndefined()) { return false; /* Not on our list. */ } JSObject* tail = GetProxyReservedSlot(wrapper, slot).toObjectOrNull(); SetProxyReservedSlot(wrapper, slot, UndefinedValue()); Compartment* comp = CrossCompartmentPointerReferent(wrapper)->compartment(); JSObject* obj = comp->gcIncomingGrayPointers; if (obj == wrapper) { comp->gcIncomingGrayPointers = tail; return true; } while (obj) { unsigned slot = ProxyObject::grayLinkReservedSlot(obj); JSObject* next = GetProxyReservedSlot(obj, slot).toObjectOrNull(); if (next == wrapper) { js::detail::SetProxyReservedSlotUnchecked(obj, slot, ObjectOrNullValue(tail)); return true; } obj = next; } MOZ_CRASH("object not found in gray link list"); } static void ResetGrayList(Compartment* comp) { JSObject* src = comp->gcIncomingGrayPointers; while (src) { src = NextIncomingCrossCompartmentPointer(src, true); } comp->gcIncomingGrayPointers = nullptr; } #ifdef DEBUG static bool HasIncomingCrossCompartmentPointers(JSRuntime* rt) { for (SweepGroupCompartmentsIter c(rt); !c.done(); c.next()) { if (c->gcIncomingGrayPointers) { return true; } } return false; } #endif void js::NotifyGCNukeWrapper(JSObject* wrapper) { MOZ_ASSERT(IsCrossCompartmentWrapper(wrapper)); /* * References to target of wrapper are being removed, we no longer have to * remember to mark it. */ RemoveFromGrayList(wrapper); /* * Clean up WeakRef maps which might include this wrapper. */ JSObject* target = UncheckedUnwrapWithoutExpose(wrapper); if (target->is()) { WeakRefObject* weakRef = &target->as(); GCRuntime* gc = &weakRef->runtimeFromMainThread()->gc; if (weakRef->target() && gc->unregisterWeakRefWrapper(wrapper)) { weakRef->setTarget(nullptr); } } /* * Clean up FinalizationRecord record objects which might be the target of * this wrapper. */ if (target->is()) { auto* record = &target->as(); FinalizationRegistryObject::unregisterRecord(record); } } enum { JS_GC_SWAP_OBJECT_A_REMOVED = 1 << 0, JS_GC_SWAP_OBJECT_B_REMOVED = 1 << 1 }; unsigned js::NotifyGCPreSwap(JSObject* a, JSObject* b) { /* * Two objects in the same compartment are about to have had their contents * swapped. If either of them are in our gray pointer list, then we remove * them from the lists, returning a bitset indicating what happened. */ return (RemoveFromGrayList(a) ? JS_GC_SWAP_OBJECT_A_REMOVED : 0) | (RemoveFromGrayList(b) ? JS_GC_SWAP_OBJECT_B_REMOVED : 0); } void js::NotifyGCPostSwap(JSObject* a, JSObject* b, unsigned removedFlags) { /* * Two objects in the same compartment have had their contents swapped. If * either of them were in our gray pointer list, we re-add them again. */ if (removedFlags & JS_GC_SWAP_OBJECT_A_REMOVED) { DelayCrossCompartmentGrayMarking(b); } if (removedFlags & JS_GC_SWAP_OBJECT_B_REMOVED) { DelayCrossCompartmentGrayMarking(a); } } static inline void MaybeCheckWeakMapMarking(GCRuntime* gc) { #if defined(JS_GC_ZEAL) || defined(DEBUG) bool shouldCheck; # if defined(DEBUG) shouldCheck = true; # else shouldCheck = gc->hasZealMode(ZealMode::CheckWeakMapMarking); # endif if (shouldCheck) { for (SweepGroupZonesIter zone(gc); !zone.done(); zone.next()) { MOZ_RELEASE_ASSERT(WeakMapBase::checkMarkingForZone(zone)); } } #endif } IncrementalProgress GCRuntime::markGrayReferencesInCurrentGroup( JSFreeOp* fop, SliceBudget& budget) { MOZ_ASSERT(!markOnBackgroundThreadDuringSweeping); MOZ_ASSERT(marker.isDrained()); MOZ_ASSERT(marker.markColor() == MarkColor::Black); if (hasMarkedGrayRoots) { return Finished; } MOZ_ASSERT(cellsToAssertNotGray.ref().empty()); gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_MARK); // Mark any incoming gray pointers from previously swept compartments that // have subsequently been marked black. This can occur when gray cells // become black by the action of UnmarkGray. markIncomingCrossCompartmentPointers(MarkColor::Black); drainMarkStack(); // Change state of current group to MarkGray to restrict marking to this // group. Note that there may be pointers to the atoms zone, and // these will be marked through, as they are not marked with // TraceCrossCompartmentEdge. for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { zone->changeGCState(Zone::MarkBlackOnly, Zone::MarkBlackAndGray); } AutoSetMarkColor setColorGray(marker, MarkColor::Gray); marker.setMainStackColor(MarkColor::Gray); // Mark incoming gray pointers from previously swept compartments. markIncomingCrossCompartmentPointers(MarkColor::Gray); markGrayRoots(gcstats::PhaseKind::SWEEP_MARK_GRAY); hasMarkedGrayRoots = true; #ifdef JS_GC_ZEAL if (shouldYieldForZeal(ZealMode::YieldWhileGrayMarking)) { return NotFinished; } #endif if (markUntilBudgetExhausted(budget) == NotFinished) { return NotFinished; } marker.setMainStackColor(MarkColor::Black); return Finished; } IncrementalProgress GCRuntime::endMarkingSweepGroup(JSFreeOp* fop, SliceBudget& budget) { MOZ_ASSERT(!markOnBackgroundThreadDuringSweeping); MOZ_ASSERT(marker.isDrained()); MOZ_ASSERT(marker.markColor() == MarkColor::Black); MOZ_ASSERT(!HasIncomingCrossCompartmentPointers(rt)); gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_MARK); if (markWeakReferencesInCurrentGroup(budget) == NotFinished) { return NotFinished; } AutoSetMarkColor setColorGray(marker, MarkColor::Gray); marker.setMainStackColor(MarkColor::Gray); // Mark transitively inside the current compartment group. if (markWeakReferencesInCurrentGroup(budget) == NotFinished) { return NotFinished; } MOZ_ASSERT(marker.isDrained()); // We must not yield after this point before we start sweeping the group. safeToYield = false; MaybeCheckWeakMapMarking(this); return Finished; } // Causes the given WeakCache to be swept when run. class ImmediateSweepWeakCacheTask : public GCParallelTask { Zone* zone; JS::detail::WeakCacheBase& cache; ImmediateSweepWeakCacheTask(const ImmediateSweepWeakCacheTask&) = delete; public: ImmediateSweepWeakCacheTask(GCRuntime* gc, Zone* zone, JS::detail::WeakCacheBase& wc) : GCParallelTask(gc), zone(zone), cache(wc) {} ImmediateSweepWeakCacheTask(ImmediateSweepWeakCacheTask&& other) : GCParallelTask(std::move(other)), zone(other.zone), cache(other.cache) {} void run(AutoLockHelperThreadState& lock) override { AutoUnlockHelperThreadState unlock(lock); AutoSetThreadIsSweeping threadIsSweeping(zone); cache.sweep(&gc->storeBuffer()); } }; void GCRuntime::updateAtomsBitmap() { DenseBitmap marked; if (atomMarking.computeBitmapFromChunkMarkBits(rt, marked)) { for (GCZonesIter zone(this); !zone.done(); zone.next()) { atomMarking.refineZoneBitmapForCollectedZone(zone, marked); } } else { // Ignore OOM in computeBitmapFromChunkMarkBits. The // refineZoneBitmapForCollectedZone call can only remove atoms from the // zone bitmap, so it is conservative to just not call it. } atomMarking.markAtomsUsedByUncollectedZones(rt); // For convenience sweep these tables non-incrementally as part of bitmap // sweeping; they are likely to be much smaller than the main atoms table. rt->symbolRegistry().sweep(); SweepingTracer trc(rt); for (RealmsIter realm(this); !realm.done(); realm.next()) { realm->tracekWeakVarNames(&trc); } } void GCRuntime::sweepCCWrappers() { AutoSetThreadIsSweeping threadIsSweeping; // This can touch all zones. for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { zone->sweepAllCrossCompartmentWrappers(); } } void GCRuntime::sweepMisc() { SweepingTracer trc(rt); for (SweepGroupRealmsIter r(this); !r.done(); r.next()) { AutoSetThreadIsSweeping threadIsSweeping(r->zone()); r->traceWeakObjects(&trc); r->traceWeakTemplateObjects(&trc); r->traceWeakSavedStacks(&trc); r->traceWeakSelfHostingScriptSource(&trc); r->traceWeakObjectRealm(&trc); r->traceWeakRegExps(&trc); } } void GCRuntime::sweepCompressionTasks() { JSRuntime* runtime = rt; // Attach finished compression tasks. AutoLockHelperThreadState lock; AttachFinishedCompressions(runtime, lock); SweepPendingCompressions(lock); } void GCRuntime::sweepWeakMaps() { AutoSetThreadIsSweeping threadIsSweeping; // This may touch any zone. for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { /* No need to look up any more weakmap keys from this sweep group. */ AutoEnterOOMUnsafeRegion oomUnsafe; if (!zone->gcWeakKeys().clear()) { oomUnsafe.crash("clearing weak keys in beginSweepingSweepGroup()"); } // Lock the storebuffer since this may access it when rehashing or resizing // the tables. AutoLockStoreBuffer lock(&storeBuffer()); zone->sweepWeakMaps(); } } void GCRuntime::sweepUniqueIds() { for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { AutoSetThreadIsSweeping threadIsSweeping(zone); zone->sweepUniqueIds(); } } void GCRuntime::sweepWeakRefs() { for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { AutoSetThreadIsSweeping threadIsSweeping(zone); zone->weakRefMap().sweep(&storeBuffer()); } } void GCRuntime::sweepFinalizationRegistriesOnMainThread() { // This calls back into the browser which expects to be called from the main // thread. gcstats::AutoPhase ap1(stats(), gcstats::PhaseKind::SWEEP_COMPARTMENTS); gcstats::AutoPhase ap2(stats(), gcstats::PhaseKind::SWEEP_FINALIZATION_REGISTRIES); AutoLockStoreBuffer lock(&storeBuffer()); for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { sweepFinalizationRegistries(zone); } } void GCRuntime::startTask(GCParallelTask& task, gcstats::PhaseKind phase, AutoLockHelperThreadState& lock) { if (!CanUseExtraThreads()) { AutoUnlockHelperThreadState unlock(lock); task.runFromMainThread(); stats().recordParallelPhase(phase, task.duration()); return; } task.startWithLockHeld(lock); } void GCRuntime::joinTask(GCParallelTask& task, gcstats::PhaseKind phase, AutoLockHelperThreadState& lock) { // This is similar to GCParallelTask::joinWithLockHeld but handles recording // execution and wait time. if (task.isIdle(lock)) { return; } if (task.isDispatched(lock)) { // If the task was dispatched but has not yet started then cancel the task // and run it from the main thread. This stops us from blocking here when // the helper threads are busy with other tasks. task.cancelDispatchedTask(lock); AutoUnlockHelperThreadState unlock(lock); task.runFromMainThread(); } else { // Otherwise wait for the task to complete. gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::JOIN_PARALLEL_TASKS); task.joinRunningOrFinishedTask(lock); } stats().recordParallelPhase(phase, task.duration()); } void GCRuntime::joinTask(GCParallelTask& task, gcstats::PhaseKind phase) { AutoLockHelperThreadState lock; joinTask(task, phase, lock); } void GCRuntime::sweepDebuggerOnMainThread(JSFreeOp* fop) { AutoLockStoreBuffer lock(&storeBuffer()); // Detach unreachable debuggers and global objects from each other. // This can modify weakmaps and so must happen before weakmap sweeping. DebugAPI::sweepAll(fop); gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_COMPARTMENTS); // Sweep debug environment information. This performs lookups in the Zone's // unique IDs table and so must not happen in parallel with sweeping that // table. { gcstats::AutoPhase ap2(stats(), gcstats::PhaseKind::SWEEP_MISC); for (SweepGroupRealmsIter r(rt); !r.done(); r.next()) { r->sweepDebugEnvironments(); } } } void GCRuntime::sweepJitDataOnMainThread(JSFreeOp* fop) { SweepingTracer trc(rt); { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_JIT_DATA); if (initialState != State::NotActive) { // Cancel any active or pending off thread compilations. We also did // this before marking (in DiscardJITCodeForGC) so this is a no-op // for non-incremental GCs. js::CancelOffThreadIonCompile(rt, JS::Zone::Sweep); } // Bug 1071218: the following method has not yet been refactored to // work on a single zone-group at once. // Sweep entries containing about-to-be-finalized JitCode and // update relocated TypeSet::Types inside the JitcodeGlobalTable. jit::JitRuntime::TraceWeakJitcodeGlobalTable(rt, &trc); } if (initialState != State::NotActive) { gcstats::AutoPhase apdc(stats(), gcstats::PhaseKind::SWEEP_DISCARD_CODE); for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { zone->discardJitCode(fop); } } // JitZone/JitRealm must be swept *after* discarding JIT code, because // Zone::discardJitCode might access CacheIRStubInfos deleted here. { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_JIT_DATA); for (SweepGroupRealmsIter r(rt); !r.done(); r.next()) { r->traceWeakEdgesInJitRealm(&trc); } for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { if (jit::JitZone* jitZone = zone->jitZone()) { jitZone->traceWeak(&trc); } } } } using WeakCacheTaskVector = mozilla::Vector; // Call a functor for all weak caches that need to be swept in the current // sweep group. template static inline bool IterateWeakCaches(JSRuntime* rt, Functor f) { for (SweepGroupZonesIter zone(rt); !zone.done(); zone.next()) { for (JS::detail::WeakCacheBase* cache : zone->weakCaches()) { if (!f(cache, zone.get())) { return false; } } } for (JS::detail::WeakCacheBase* cache : rt->weakCaches()) { if (!f(cache, nullptr)) { return false; } } return true; } static bool PrepareWeakCacheTasks(JSRuntime* rt, WeakCacheTaskVector* immediateTasks) { // Start incremental sweeping for caches that support it or add to a vector // of sweep tasks to run on a helper thread. MOZ_ASSERT(immediateTasks->empty()); bool ok = IterateWeakCaches(rt, [&](JS::detail::WeakCacheBase* cache, Zone* zone) { if (!cache->needsSweep()) { return true; } // Caches that support incremental sweeping will be swept later. if (zone && cache->setNeedsIncrementalBarrier(true)) { return true; } return immediateTasks->emplaceBack(&rt->gc, zone, *cache); }); if (!ok) { immediateTasks->clearAndFree(); } return ok; } static void SweepAllWeakCachesOnMainThread(JSRuntime* rt) { // If we ran out of memory, do all the work on the main thread. gcstats::AutoPhase ap(rt->gc.stats(), gcstats::PhaseKind::SWEEP_WEAK_CACHES); IterateWeakCaches(rt, [&](JS::detail::WeakCacheBase* cache, Zone* zone) { if (cache->needsIncrementalBarrier()) { cache->setNeedsIncrementalBarrier(false); } cache->sweep(&rt->gc.storeBuffer()); return true; }); } IncrementalProgress GCRuntime::beginSweepingSweepGroup(JSFreeOp* fop, SliceBudget& budget) { /* * Begin sweeping the group of zones in currentSweepGroup, performing * actions that must be done before yielding to caller. */ using namespace gcstats; AutoSCC scc(stats(), sweepGroupIndex); bool sweepingAtoms = false; for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { /* Set the GC state to sweeping. */ zone->changeGCState(Zone::MarkBlackAndGray, Zone::Sweep); /* Purge the ArenaLists before sweeping. */ zone->arenas.checkSweepStateNotInUse(); zone->arenas.unmarkPreMarkedFreeCells(); zone->arenas.clearFreeLists(); if (zone->isAtomsZone()) { sweepingAtoms = true; } } #ifdef JS_GC_ZEAL validateIncrementalMarking(); #endif #ifdef DEBUG for (auto cell : cellsToAssertNotGray.ref()) { JS::AssertCellIsNotGray(cell); } cellsToAssertNotGray.ref().clearAndFree(); #endif { AutoLockStoreBuffer lock(&storeBuffer()); AutoPhase ap(stats(), PhaseKind::FINALIZE_START); callFinalizeCallbacks(fop, JSFINALIZE_GROUP_PREPARE); { AutoPhase ap2(stats(), PhaseKind::WEAK_ZONES_CALLBACK); callWeakPointerZonesCallbacks(); } { AutoPhase ap2(stats(), PhaseKind::WEAK_COMPARTMENT_CALLBACK); for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next()) { callWeakPointerCompartmentCallbacks(comp); } } } callFinalizeCallbacks(fop, JSFINALIZE_GROUP_START); } // Updating the atom marking bitmaps. This marks atoms referenced by // uncollected zones so cannot be done in parallel with the other sweeping // work below. if (sweepingAtoms) { AutoPhase ap(stats(), PhaseKind::UPDATE_ATOMS_BITMAP); updateAtomsBitmap(); } AutoSetThreadIsSweeping threadIsSweeping; sweepDebuggerOnMainThread(fop); { AutoLockHelperThreadState lock; AutoPhase ap(stats(), PhaseKind::SWEEP_COMPARTMENTS); AutoRunParallelTask sweepCCWrappers(this, &GCRuntime::sweepCCWrappers, PhaseKind::SWEEP_CC_WRAPPER, lock); AutoRunParallelTask sweepMisc(this, &GCRuntime::sweepMisc, PhaseKind::SWEEP_MISC, lock); AutoRunParallelTask sweepCompTasks(this, &GCRuntime::sweepCompressionTasks, PhaseKind::SWEEP_COMPRESSION, lock); AutoRunParallelTask sweepWeakMaps(this, &GCRuntime::sweepWeakMaps, PhaseKind::SWEEP_WEAKMAPS, lock); AutoRunParallelTask sweepUniqueIds(this, &GCRuntime::sweepUniqueIds, PhaseKind::SWEEP_UNIQUEIDS, lock); AutoRunParallelTask sweepWeakRefs(this, &GCRuntime::sweepWeakRefs, PhaseKind::SWEEP_WEAKREFS, lock); WeakCacheTaskVector sweepCacheTasks; bool canSweepWeakCachesOffThread = PrepareWeakCacheTasks(rt, &sweepCacheTasks); if (canSweepWeakCachesOffThread) { weakCachesToSweep.ref().emplace(currentSweepGroup); for (auto& task : sweepCacheTasks) { startTask(task, PhaseKind::SWEEP_WEAK_CACHES, lock); } } { AutoUnlockHelperThreadState unlock(lock); sweepJitDataOnMainThread(fop); if (!canSweepWeakCachesOffThread) { MOZ_ASSERT(sweepCacheTasks.empty()); SweepAllWeakCachesOnMainThread(rt); } } for (auto& task : sweepCacheTasks) { joinTask(task, PhaseKind::SWEEP_WEAK_CACHES, lock); } } if (sweepingAtoms) { startSweepingAtomsTable(); } // FinalizationRegistry sweeping touches weak maps and so must not run in // parallel with that. This triggers a read barrier and can add marking work // for zones that are still marking. sweepFinalizationRegistriesOnMainThread(); // Queue all GC things in all zones for sweeping, either on the foreground // or on the background thread. for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { zone->arenas.queueForForegroundSweep(fop, ForegroundObjectFinalizePhase); zone->arenas.queueForForegroundSweep(fop, ForegroundNonObjectFinalizePhase); for (const auto& phase : BackgroundFinalizePhases) { zone->arenas.queueForBackgroundSweep(fop, phase); } zone->arenas.queueForegroundThingsForSweep(); } MOZ_ASSERT(!sweepZone); safeToYield = true; markOnBackgroundThreadDuringSweeping = CanUseExtraThreads(); return Finished; } #ifdef JS_GC_ZEAL bool GCRuntime::shouldYieldForZeal(ZealMode mode) { bool yield = useZeal && hasZealMode(mode); // Only yield on the first sweep slice for this mode. bool firstSweepSlice = initialState != State::Sweep; if (mode == ZealMode::IncrementalMultipleSlices && !firstSweepSlice) { yield = false; } return yield; } #endif IncrementalProgress GCRuntime::endSweepingSweepGroup(JSFreeOp* fop, SliceBudget& budget) { // This is to prevent a race between markTask checking the zone state and // us changing it below. if (joinBackgroundMarkTask() == NotFinished) { return NotFinished; } MOZ_ASSERT(marker.isDrained()); // Disable background marking during sweeping until we start sweeping the next // zone group. markOnBackgroundThreadDuringSweeping = false; { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::FINALIZE_END); AutoLockStoreBuffer lock(&storeBuffer()); JSFreeOp fop(rt); callFinalizeCallbacks(&fop, JSFINALIZE_GROUP_END); } /* Free LIFO blocks on a background thread if possible. */ startBackgroundFree(); /* Update the GC state for zones we have swept. */ for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { if (jit::JitZone* jitZone = zone->jitZone()) { // Clear out any small pools that we're hanging on to. jitZone->execAlloc().purge(); } AutoLockGC lock(this); zone->changeGCState(Zone::Sweep, Zone::Finished); zone->arenas.unmarkPreMarkedFreeCells(); zone->arenas.checkNoArenasToUpdate(); } /* * Start background thread to sweep zones if required, sweeping the atoms * zone last if present. */ bool sweepAtomsZone = false; ZoneList zones; for (SweepGroupZonesIter zone(this); !zone.done(); zone.next()) { if (zone->isAtomsZone()) { sweepAtomsZone = true; } else { zones.append(zone); } } if (sweepAtomsZone) { zones.append(atomsZone); } queueZonesAndStartBackgroundSweep(zones); return Finished; } IncrementalProgress GCRuntime::markDuringSweeping(JSFreeOp* fop, SliceBudget& budget) { MOZ_ASSERT(markTask.isIdle()); if (marker.isDrained()) { return Finished; } if (markOnBackgroundThreadDuringSweeping) { AutoLockHelperThreadState lock; MOZ_ASSERT(markTask.isIdle(lock)); markTask.setBudget(budget); markTask.startOrRunIfIdle(lock); return Finished; // This means don't yield to the mutator here. } gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_MARK); return markUntilBudgetExhausted(budget); } void GCRuntime::beginSweepPhase(JS::GCReason reason, AutoGCSession& session) { /* * Sweep phase. * * Finalize as we sweep, outside of lock but with RuntimeHeapIsBusy() * true so that any attempt to allocate a GC-thing from a finalizer will * fail, rather than nest badly and leave the unmarked newborn to be swept. */ MOZ_ASSERT(!abortSweepAfterCurrentGroup); MOZ_ASSERT(!markOnBackgroundThreadDuringSweeping); releaseHeldRelocatedArenas(); #ifdef JS_GC_ZEAL computeNonIncrementalMarkingForValidation(session); #endif gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP); hasMarkedGrayRoots = false; AssertNoWrappersInGrayList(rt); dropStringWrappers(); groupZonesForSweeping(reason); sweepActions->assertFinished(); } bool ArenaLists::foregroundFinalize(JSFreeOp* fop, AllocKind thingKind, SliceBudget& sliceBudget, SortedArenaList& sweepList) { checkNoArenasToUpdateForKind(thingKind); // Arenas are released for use for new allocations as soon as the finalizers // for that allocation kind have run. This means that a cell's finalizer can // safely use IsAboutToBeFinalized to check other cells of the same alloc // kind, but not of different alloc kinds: the other arena may have already // had new objects allocated in it, and since we allocate black, // IsAboutToBeFinalized will return false even though the referent we intended // to check is long gone. if (!FinalizeArenas(fop, &arenasToSweep(thingKind), sweepList, thingKind, sliceBudget)) { // Copy the current contents of sweepList so that ArenaIter can find them. incrementalSweptArenaKind = thingKind; incrementalSweptArenas.ref().clear(); incrementalSweptArenas = sweepList.toArenaList(); return false; } // Clear the list of swept arenas now these are moving back to the main arena // lists. incrementalSweptArenaKind = AllocKind::LIMIT; incrementalSweptArenas.ref().clear(); sweepList.extractEmpty(&savedEmptyArenas.ref()); ArenaList& al = arenaList(thingKind); ArenaList allocatedDuringSweep = std::move(al); al = sweepList.toArenaList(); al.insertListWithCursorAtEnd(newArenasInMarkPhase(thingKind)); al.insertListWithCursorAtEnd(allocatedDuringSweep); newArenasInMarkPhase(thingKind).clear(); return true; } void js::gc::BackgroundMarkTask::run(AutoLockHelperThreadState& lock) { AutoUnlockHelperThreadState unlock(lock); // Time reporting is handled separately for parallel tasks. gc->sweepMarkResult = gc->markUntilBudgetExhausted(this->budget, GCMarker::DontReportMarkTime); } IncrementalProgress GCRuntime::joinBackgroundMarkTask() { AutoLockHelperThreadState lock; if (markTask.isIdle(lock)) { return Finished; } joinTask(markTask, gcstats::PhaseKind::SWEEP_MARK, lock); IncrementalProgress result = sweepMarkResult; sweepMarkResult = Finished; return result; } IncrementalProgress GCRuntime::markUntilBudgetExhausted( SliceBudget& sliceBudget, GCMarker::ShouldReportMarkTime reportTime) { // Run a marking slice and return whether the stack is now empty. AutoMajorGCProfilerEntry s(this); #ifdef DEBUG AutoSetThreadIsMarking threadIsMarking; #endif // DEBUG if (marker.processMarkQueue() == GCMarker::QueueYielded) { return NotFinished; } return marker.markUntilBudgetExhausted(sliceBudget, reportTime) ? Finished : NotFinished; } void GCRuntime::drainMarkStack() { auto unlimited = SliceBudget::unlimited(); MOZ_RELEASE_ASSERT(marker.markUntilBudgetExhausted(unlimited)); } static void SweepThing(JSFreeOp* fop, Shape* shape) { if (!shape->isMarkedAny()) { shape->sweep(fop); } } template static bool SweepArenaList(JSFreeOp* fop, Arena** arenasToSweep, SliceBudget& sliceBudget) { while (Arena* arena = *arenasToSweep) { MOZ_ASSERT(arena->zone->isGCSweeping()); for (ArenaCellIterUnderGC cell(arena); !cell.done(); cell.next()) { SweepThing(fop, cell.as()); } Arena* next = arena->next; MOZ_ASSERT_IF(next, next->zone == arena->zone); *arenasToSweep = next; AllocKind kind = MapTypeToFinalizeKind::kind; sliceBudget.step(Arena::thingsPerArena(kind)); if (sliceBudget.isOverBudget()) { return false; } } return true; } void GCRuntime::startSweepingAtomsTable() { auto& maybeAtoms = maybeAtomsToSweep.ref(); MOZ_ASSERT(maybeAtoms.isNothing()); AtomsTable* atomsTable = rt->atomsForSweeping(); if (!atomsTable) { return; } // Create secondary tables to hold new atoms added while we're sweeping the // main tables incrementally. if (!atomsTable->startIncrementalSweep()) { SweepingTracer trc(rt); atomsTable->traceWeak(&trc); return; } // Initialize remaining atoms to sweep. maybeAtoms.emplace(*atomsTable); } IncrementalProgress GCRuntime::sweepAtomsTable(JSFreeOp* fop, SliceBudget& budget) { if (!atomsZone->isGCSweeping()) { return Finished; } gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_ATOMS_TABLE); auto& maybeAtoms = maybeAtomsToSweep.ref(); if (!maybeAtoms) { return Finished; } if (!rt->atomsForSweeping()->sweepIncrementally(maybeAtoms.ref(), budget)) { return NotFinished; } maybeAtoms.reset(); return Finished; } static size_t IncrementalSweepWeakCache(GCRuntime* gc, const WeakCacheToSweep& item) { AutoSetThreadIsSweeping threadIsSweeping(item.zone); JS::detail::WeakCacheBase* cache = item.cache; MOZ_ASSERT(cache->needsIncrementalBarrier()); size_t steps = cache->sweep(&gc->storeBuffer()); cache->setNeedsIncrementalBarrier(false); return steps; } WeakCacheSweepIterator::WeakCacheSweepIterator(JS::Zone* sweepGroup) : sweepZone(sweepGroup), sweepCache(sweepZone->weakCaches().getFirst()) { settle(); } bool WeakCacheSweepIterator::done() const { return !sweepZone; } WeakCacheToSweep WeakCacheSweepIterator::get() const { MOZ_ASSERT(!done()); return {sweepCache, sweepZone}; } void WeakCacheSweepIterator::next() { MOZ_ASSERT(!done()); sweepCache = sweepCache->getNext(); settle(); } void WeakCacheSweepIterator::settle() { while (sweepZone) { while (sweepCache && !sweepCache->needsIncrementalBarrier()) { sweepCache = sweepCache->getNext(); } if (sweepCache) { break; } sweepZone = sweepZone->nextNodeInGroup(); if (sweepZone) { sweepCache = sweepZone->weakCaches().getFirst(); } } MOZ_ASSERT((!sweepZone && !sweepCache) || (sweepCache && sweepCache->needsIncrementalBarrier())); } IncrementalProgress GCRuntime::sweepWeakCaches(JSFreeOp* fop, SliceBudget& budget) { if (weakCachesToSweep.ref().isNothing()) { return Finished; } gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_COMPARTMENTS); WeakCacheSweepIterator& work = weakCachesToSweep.ref().ref(); AutoLockHelperThreadState lock; { AutoRunParallelWork runWork(this, IncrementalSweepWeakCache, gcstats::PhaseKind::SWEEP_WEAK_CACHES, work, budget, lock); AutoUnlockHelperThreadState unlock(lock); } if (work.done()) { weakCachesToSweep.ref().reset(); return Finished; } return NotFinished; } IncrementalProgress GCRuntime::finalizeAllocKind(JSFreeOp* fop, SliceBudget& budget) { MOZ_ASSERT(sweepZone->isGCSweeping()); // Set the number of things per arena for this AllocKind. size_t thingsPerArena = Arena::thingsPerArena(sweepAllocKind); auto& sweepList = incrementalSweepList.ref(); sweepList.setThingsPerArena(thingsPerArena); AutoSetThreadIsSweeping threadIsSweeping(sweepZone); if (!sweepZone->arenas.foregroundFinalize(fop, sweepAllocKind, budget, sweepList)) { return NotFinished; } // Reset the slots of the sweep list that we used. sweepList.reset(thingsPerArena); return Finished; } IncrementalProgress GCRuntime::sweepShapeTree(JSFreeOp* fop, SliceBudget& budget) { // Remove dead shapes from the shape tree, but don't finalize them yet. gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP_SHAPE); ArenaLists& al = sweepZone->arenas; if (!SweepArenaList(fop, &al.gcShapeArenasToUpdate.ref(), budget)) { return NotFinished; } if (!SweepArenaList( fop, &al.gcAccessorShapeArenasToUpdate.ref(), budget)) { return NotFinished; } return Finished; } // An iterator for a standard container that provides an STL-like begin()/end() // interface. This iterator provides a done()/get()/next() style interface. template class ContainerIter { using Iter = decltype(std::declval().begin()); using Elem = decltype(*std::declval()); Iter iter; const Iter end; public: explicit ContainerIter(const Container& container) : iter(container.begin()), end(container.end()) {} bool done() const { return iter == end; } Elem get() const { return *iter; } void next() { MOZ_ASSERT(!done()); ++iter; } }; // IncrementalIter is a template class that makes a normal iterator into one // that can be used to perform incremental work by using external state that // persists between instantiations. The state is only initialised on the first // use and subsequent uses carry on from the previous state. template struct IncrementalIter { using State = Maybe; using Elem = decltype(std::declval().get()); private: State& maybeIter; public: template explicit IncrementalIter(State& maybeIter, Args&&... args) : maybeIter(maybeIter) { if (maybeIter.isNothing()) { maybeIter.emplace(std::forward(args)...); } } ~IncrementalIter() { if (done()) { maybeIter.reset(); } } bool done() const { return maybeIter.ref().done(); } Elem get() const { return maybeIter.ref().get(); } void next() { maybeIter.ref().next(); } }; // Iterate through the sweep groups created by // GCRuntime::groupZonesForSweeping(). class js::gc::SweepGroupsIter { GCRuntime* gc; public: explicit SweepGroupsIter(JSRuntime* rt) : gc(&rt->gc) { MOZ_ASSERT(gc->currentSweepGroup); } bool done() const { return !gc->currentSweepGroup; } Zone* get() const { return gc->currentSweepGroup; } void next() { MOZ_ASSERT(!done()); gc->getNextSweepGroup(); } }; namespace sweepaction { // Implementation of the SweepAction interface that calls a method on GCRuntime. class SweepActionCall final : public SweepAction { using Method = IncrementalProgress (GCRuntime::*)(JSFreeOp* fop, SliceBudget& budget); Method method; public: explicit SweepActionCall(Method m) : method(m) {} IncrementalProgress run(Args& args) override { return (args.gc->*method)(args.fop, args.budget); } void assertFinished() const override {} }; // Implementation of the SweepAction interface that yields in a specified zeal // mode. class SweepActionMaybeYield final : public SweepAction { #ifdef JS_GC_ZEAL ZealMode mode; bool isYielding; #endif public: explicit SweepActionMaybeYield(ZealMode mode) #ifdef JS_GC_ZEAL : mode(mode), isYielding(false) #endif { } IncrementalProgress run(Args& args) override { #ifdef JS_GC_ZEAL if (!isYielding && args.gc->shouldYieldForZeal(mode)) { isYielding = true; return NotFinished; } isYielding = false; #endif return Finished; } void assertFinished() const override { MOZ_ASSERT(!isYielding); } // These actions should be skipped if GC zeal is not configured. #ifndef JS_GC_ZEAL bool shouldSkip() override { return true; } #endif }; // Implementation of the SweepAction interface that calls a list of actions in // sequence. class SweepActionSequence final : public SweepAction { using ActionVector = Vector, 0, SystemAllocPolicy>; using Iter = IncrementalIter>; ActionVector actions; typename Iter::State iterState; public: bool init(UniquePtr* acts, size_t count) { for (size_t i = 0; i < count; i++) { auto& action = acts[i]; if (!action) { return false; } if (action->shouldSkip()) { continue; } if (!actions.emplaceBack(std::move(action))) { return false; } } return true; } IncrementalProgress run(Args& args) override { for (Iter iter(iterState, actions); !iter.done(); iter.next()) { if (iter.get()->run(args) == NotFinished) { return NotFinished; } } return Finished; } void assertFinished() const override { MOZ_ASSERT(iterState.isNothing()); for (const auto& action : actions) { action->assertFinished(); } } }; template class SweepActionForEach final : public SweepAction { using Elem = decltype(std::declval().get()); using IncrIter = IncrementalIter; Init iterInit; Elem* elemOut; UniquePtr action; typename IncrIter::State iterState; public: SweepActionForEach(const Init& init, Elem* maybeElemOut, UniquePtr action) : iterInit(init), elemOut(maybeElemOut), action(std::move(action)) {} IncrementalProgress run(Args& args) override { MOZ_ASSERT_IF(elemOut, *elemOut == Elem()); auto clearElem = mozilla::MakeScopeExit([&] { setElem(Elem()); }); for (IncrIter iter(iterState, iterInit); !iter.done(); iter.next()) { setElem(iter.get()); if (action->run(args) == NotFinished) { return NotFinished; } } return Finished; } void assertFinished() const override { MOZ_ASSERT(iterState.isNothing()); MOZ_ASSERT_IF(elemOut, *elemOut == Elem()); action->assertFinished(); } private: void setElem(const Elem& value) { if (elemOut) { *elemOut = value; } } }; static UniquePtr Call(IncrementalProgress (GCRuntime::*method)( JSFreeOp* fop, SliceBudget& budget)) { return MakeUnique(method); } static UniquePtr MaybeYield(ZealMode zealMode) { return MakeUnique(zealMode); } template static UniquePtr Sequence(UniquePtr first, Rest... rest) { UniquePtr actions[] = {std::move(first), std::move(rest)...}; auto seq = MakeUnique(); if (!seq || !seq->init(actions, std::size(actions))) { return nullptr; } return UniquePtr(std::move(seq)); } static UniquePtr RepeatForSweepGroup( JSRuntime* rt, UniquePtr action) { if (!action) { return nullptr; } using Action = SweepActionForEach; return js::MakeUnique(rt, nullptr, std::move(action)); } static UniquePtr ForEachZoneInSweepGroup( JSRuntime* rt, Zone** zoneOut, UniquePtr action) { if (!action) { return nullptr; } using Action = SweepActionForEach; return js::MakeUnique(rt, zoneOut, std::move(action)); } static UniquePtr ForEachAllocKind(AllocKinds kinds, AllocKind* kindOut, UniquePtr action) { if (!action) { return nullptr; } using Action = SweepActionForEach, AllocKinds>; return js::MakeUnique(kinds, kindOut, std::move(action)); } } // namespace sweepaction bool GCRuntime::initSweepActions() { using namespace sweepaction; using sweepaction::Call; sweepActions.ref() = RepeatForSweepGroup( rt, Sequence( Call(&GCRuntime::markGrayReferencesInCurrentGroup), Call(&GCRuntime::endMarkingSweepGroup), Call(&GCRuntime::beginSweepingSweepGroup), MaybeYield(ZealMode::IncrementalMultipleSlices), MaybeYield(ZealMode::YieldBeforeSweepingAtoms), Call(&GCRuntime::sweepAtomsTable), MaybeYield(ZealMode::YieldBeforeSweepingCaches), Call(&GCRuntime::sweepWeakCaches), ForEachZoneInSweepGroup( rt, &sweepZone.ref(), Sequence(MaybeYield(ZealMode::YieldBeforeSweepingObjects), ForEachAllocKind(ForegroundObjectFinalizePhase.kinds, &sweepAllocKind.ref(), Call(&GCRuntime::finalizeAllocKind)), MaybeYield(ZealMode::YieldBeforeSweepingNonObjects), ForEachAllocKind(ForegroundNonObjectFinalizePhase.kinds, &sweepAllocKind.ref(), Call(&GCRuntime::finalizeAllocKind)), MaybeYield(ZealMode::YieldBeforeSweepingShapeTrees), Call(&GCRuntime::sweepShapeTree))), Call(&GCRuntime::endSweepingSweepGroup))); return sweepActions != nullptr; } IncrementalProgress GCRuntime::performSweepActions(SliceBudget& budget) { AutoMajorGCProfilerEntry s(this); gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP); JSFreeOp fop(rt); // Don't trigger pre-barriers when finalizing. AutoDisableBarriers disableBarriers(this); // Drain the mark stack, possibly in a parallel task if we're in a part of // sweeping that allows it. // // In the first sweep slice where we must not yield to the mutator until we've // starting sweeping a sweep group but in that case the stack must be empty // already. MOZ_ASSERT(initialState <= State::Sweep); MOZ_ASSERT_IF(initialState != State::Sweep, marker.isDrained()); if (initialState == State::Sweep && markDuringSweeping(&fop, budget) == NotFinished) { return NotFinished; } // Then continue running sweep actions. SweepAction::Args args{this, &fop, budget}; IncrementalProgress sweepProgress = sweepActions->run(args); IncrementalProgress markProgress = joinBackgroundMarkTask(); if (sweepProgress == Finished && markProgress == Finished) { return Finished; } MOZ_ASSERT(isIncremental); return NotFinished; } bool GCRuntime::allCCVisibleZonesWereCollected() { // Calculate whether the gray marking state is now valid. // // The gray bits change from invalid to valid if we finished a full GC from // the point of view of the cycle collector. We ignore the following: // // - Helper thread zones, as these are not reachable from the main heap. // - The atoms zone, since strings and symbols are never marked gray. // - Empty zones. // // These exceptions ensure that when the CC requests a full GC the gray mark // state ends up valid even it we don't collect all of the zones. for (ZonesIter zone(this, SkipAtoms); !zone.done(); zone.next()) { if (!zone->isCollecting() && !zone->usedByHelperThread() && !zone->arenas.arenaListsAreEmpty()) { return false; } } return true; } void GCRuntime::endSweepPhase(bool destroyingRuntime) { MOZ_ASSERT(!markOnBackgroundThreadDuringSweeping); sweepActions->assertFinished(); gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::SWEEP); JSFreeOp fop(rt); MOZ_ASSERT_IF(destroyingRuntime, !sweepOnBackgroundThread); { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::DESTROY); /* * Sweep script filenames after sweeping functions in the generic loop * above. In this way when a scripted function's finalizer destroys the * script and calls rt->destroyScriptHook, the hook can still access the * script's filename. See bug 323267. */ SweepScriptData(rt); } { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::FINALIZE_END); AutoLockStoreBuffer lock(&storeBuffer()); callFinalizeCallbacks(&fop, JSFINALIZE_COLLECTION_END); if (allCCVisibleZonesWereCollected()) { grayBitsValid = true; } } #ifdef JS_GC_ZEAL finishMarkingValidation(); #endif #ifdef DEBUG for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { for (auto i : AllAllocKinds()) { MOZ_ASSERT_IF(!IsBackgroundFinalized(i) || !sweepOnBackgroundThread, !zone->arenas.arenasToSweep(i)); } } #endif AssertNoWrappersInGrayList(rt); } void GCRuntime::beginCompactPhase() { MOZ_ASSERT(!isBackgroundSweeping()); assertBackgroundSweepingFinished(); gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::COMPACT); MOZ_ASSERT(zonesToMaybeCompact.ref().isEmpty()); for (GCZonesIter zone(this); !zone.done(); zone.next()) { if (canRelocateZone(zone)) { zonesToMaybeCompact.ref().append(zone); } } MOZ_ASSERT(!relocatedArenasToRelease); startedCompacting = true; zonesCompacted = 0; } IncrementalProgress GCRuntime::compactPhase(JS::GCReason reason, SliceBudget& sliceBudget, AutoGCSession& session) { assertBackgroundSweepingFinished(); MOZ_ASSERT(startedCompacting); AutoMajorGCProfilerEntry s(this); gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::COMPACT); // TODO: JSScripts can move. If the sampler interrupts the GC in the // middle of relocating an arena, invalid JSScript pointers may be // accessed. Suppress all sampling until a finer-grained solution can be // found. See bug 1295775. AutoSuppressProfilerSampling suppressSampling(rt->mainContextFromOwnThread()); ZoneList relocatedZones; Arena* relocatedArenas = nullptr; while (!zonesToMaybeCompact.ref().isEmpty()) { Zone* zone = zonesToMaybeCompact.ref().front(); zonesToMaybeCompact.ref().removeFront(); MOZ_ASSERT(nursery().isEmpty()); zone->changeGCState(Zone::Finished, Zone::Compact); if (relocateArenas(zone, reason, relocatedArenas, sliceBudget)) { updateZonePointersToRelocatedCells(zone); relocatedZones.append(zone); zonesCompacted++; } else { zone->changeGCState(Zone::Compact, Zone::Finished); } if (sliceBudget.isOverBudget()) { break; } } if (!relocatedZones.isEmpty()) { updateRuntimePointersToRelocatedCells(session); do { Zone* zone = relocatedZones.front(); relocatedZones.removeFront(); zone->changeGCState(Zone::Compact, Zone::Finished); } while (!relocatedZones.isEmpty()); } clearRelocatedArenas(relocatedArenas, reason); if (ShouldProtectRelocatedArenas(reason)) { protectAndHoldArenas(relocatedArenas); } else { releaseRelocatedArenas(relocatedArenas); } // Clear caches that can contain cell pointers. rt->caches().purgeForCompaction(); #ifdef DEBUG checkHashTablesAfterMovingGC(); #endif return zonesToMaybeCompact.ref().isEmpty() ? Finished : NotFinished; } void GCRuntime::endCompactPhase() { startedCompacting = false; } void GCRuntime::finishCollection() { assertBackgroundSweepingFinished(); MOZ_ASSERT(marker.isDrained()); marker.stop(); clearBufferedGrayRoots(); maybeStopStringPretenuring(); { AutoLockGC lock(this); updateGCThresholdsAfterCollection(lock); } for (GCZonesIter zone(this); !zone.done(); zone.next()) { zone->changeGCState(Zone::Finished, Zone::NoGC); zone->notifyObservingDebuggers(); } #ifdef JS_GC_ZEAL clearSelectedForMarking(); #endif auto currentTime = ReallyNow(); schedulingState.updateHighFrequencyMode(lastGCEndTime_, currentTime, tunables); lastGCEndTime_ = currentTime; checkGCStateNotInUse(); } void GCRuntime::checkGCStateNotInUse() { #ifdef DEBUG MOZ_ASSERT(!marker.isActive()); for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { if (zone->wasCollected()) { zone->arenas.checkGCStateNotInUse(); } MOZ_ASSERT(!zone->wasGCStarted()); MOZ_ASSERT(!zone->needsIncrementalBarrier()); MOZ_ASSERT(!zone->isOnList()); } MOZ_ASSERT(zonesToMaybeCompact.ref().isEmpty()); MOZ_ASSERT(cellsToAssertNotGray.ref().empty()); AutoLockHelperThreadState lock; MOZ_ASSERT(!requestSliceAfterBackgroundTask); #endif } void GCRuntime::maybeStopStringPretenuring() { for (GCZonesIter zone(this); !zone.done(); zone.next()) { if (zone->allocNurseryStrings) { continue; } // Count the number of strings before the major GC. size_t numStrings = zone->markedStrings + zone->finalizedStrings; double rate = double(zone->finalizedStrings) / double(numStrings); if (rate > tunables.stopPretenureStringThreshold()) { CancelOffThreadIonCompile(zone); bool preserving = zone->isPreservingCode(); zone->setPreservingCode(false); zone->discardJitCode(rt->defaultFreeOp()); zone->setPreservingCode(preserving); for (RealmsInZoneIter r(zone); !r.done(); r.next()) { if (jit::JitRealm* jitRealm = r->jitRealm()) { jitRealm->discardStubs(); jitRealm->setStringsCanBeInNursery(true); } } zone->markedStrings = 0; zone->finalizedStrings = 0; zone->allocNurseryStrings = true; } } } void GCRuntime::updateGCThresholdsAfterCollection(const AutoLockGC& lock) { for (GCZonesIter zone(this); !zone.done(); zone.next()) { zone->clearGCSliceThresholds(); zone->updateGCStartThresholds(*this, invocationKind, lock); } } void GCRuntime::updateAllGCStartThresholds(const AutoLockGC& lock) { for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { zone->updateGCStartThresholds(*this, GC_NORMAL, lock); } } static const char* GCHeapStateToLabel(JS::HeapState heapState) { switch (heapState) { case JS::HeapState::MinorCollecting: return "js::Nursery::collect"; case JS::HeapState::MajorCollecting: return "js::GCRuntime::collect"; default: MOZ_CRASH("Unexpected heap state when pushing GC profiling stack frame"); } MOZ_ASSERT_UNREACHABLE("Should have exhausted every JS::HeapState variant!"); return nullptr; } static JS::ProfilingCategoryPair GCHeapStateToProfilingCategory( JS::HeapState heapState) { return heapState == JS::HeapState::MinorCollecting ? JS::ProfilingCategoryPair::GCCC_MinorGC : JS::ProfilingCategoryPair::GCCC_MajorGC; } /* Start a new heap session. */ AutoHeapSession::AutoHeapSession(GCRuntime* gc, JS::HeapState heapState) : gc(gc), prevState(gc->heapState_) { MOZ_ASSERT(CurrentThreadCanAccessRuntime(gc->rt)); MOZ_ASSERT(prevState == JS::HeapState::Idle || (prevState == JS::HeapState::MajorCollecting && heapState == JS::HeapState::MinorCollecting)); MOZ_ASSERT(heapState != JS::HeapState::Idle); gc->heapState_ = heapState; if (heapState == JS::HeapState::MinorCollecting || heapState == JS::HeapState::MajorCollecting) { profilingStackFrame.emplace(gc->rt->mainContextFromOwnThread(), GCHeapStateToLabel(heapState), GCHeapStateToProfilingCategory(heapState)); } } AutoHeapSession::~AutoHeapSession() { MOZ_ASSERT(JS::RuntimeHeapIsBusy()); gc->heapState_ = prevState; } static const char* MajorGCStateToLabel(State state) { switch (state) { case State::Mark: return "js::GCRuntime::markUntilBudgetExhausted"; case State::Sweep: return "js::GCRuntime::performSweepActions"; case State::Compact: return "js::GCRuntime::compactPhase"; default: MOZ_CRASH("Unexpected heap state when pushing GC profiling stack frame"); } MOZ_ASSERT_UNREACHABLE("Should have exhausted every State variant!"); return nullptr; } static JS::ProfilingCategoryPair MajorGCStateToProfilingCategory(State state) { switch (state) { case State::Mark: return JS::ProfilingCategoryPair::GCCC_MajorGC_Mark; case State::Sweep: return JS::ProfilingCategoryPair::GCCC_MajorGC_Sweep; case State::Compact: return JS::ProfilingCategoryPair::GCCC_MajorGC_Compact; default: MOZ_CRASH("Unexpected heap state when pushing GC profiling stack frame"); } } AutoMajorGCProfilerEntry::AutoMajorGCProfilerEntry(GCRuntime* gc) : AutoGeckoProfilerEntry(gc->rt->mainContextFromAnyThread(), MajorGCStateToLabel(gc->state()), MajorGCStateToProfilingCategory(gc->state())) { MOZ_ASSERT(gc->heapState() == JS::HeapState::MajorCollecting); } JS_PUBLIC_API JS::HeapState JS::RuntimeHeapState() { return TlsContext.get()->runtime()->gc.heapState(); } GCRuntime::IncrementalResult GCRuntime::resetIncrementalGC( GCAbortReason reason) { // Drop as much work as possible from an ongoing incremental GC so // we can start a new GC after it has finished. if (incrementalState == State::NotActive) { return IncrementalResult::Ok; } AutoGCSession session(this, JS::HeapState::MajorCollecting); switch (incrementalState) { case State::NotActive: case State::MarkRoots: case State::Finish: MOZ_CRASH("Unexpected GC state in resetIncrementalGC"); break; case State::Prepare: unmarkTask.cancelAndWait(); setParallelUnmarkEnabled(false); for (GCZonesIter zone(this); !zone.done(); zone.next()) { zone->changeGCState(Zone::Prepare, Zone::NoGC); } incrementalState = State::NotActive; checkGCStateNotInUse(); break; case State::Mark: { // Cancel any ongoing marking. marker.reset(); clearBufferedGrayRoots(); for (GCCompartmentsIter c(rt); !c.done(); c.next()) { ResetGrayList(c); } for (GCZonesIter zone(this); !zone.done(); zone.next()) { zone->changeGCState(Zone::MarkBlackOnly, Zone::NoGC); zone->clearGCSliceThresholds(); zone->arenas.unmarkPreMarkedFreeCells(); zone->arenas.mergeNewArenasInMarkPhase(); } { AutoLockHelperThreadState lock; lifoBlocksToFree.ref().freeAll(); } lastMarkSlice = false; incrementalState = State::Finish; MOZ_ASSERT(!marker.shouldCheckCompartments()); break; } case State::Sweep: { // Finish sweeping the current sweep group, then abort. for (CompartmentsIter c(rt); !c.done(); c.next()) { c->gcState.scheduledForDestruction = false; } abortSweepAfterCurrentGroup = true; isCompacting = false; break; } case State::Finalize: { isCompacting = false; break; } case State::Compact: { // Skip any remaining zones that would have been compacted. MOZ_ASSERT(isCompacting); startedCompacting = true; zonesToMaybeCompact.ref().clear(); break; } case State::Decommit: { break; } } stats().reset(reason); return IncrementalResult::ResetIncremental; } AutoDisableBarriers::AutoDisableBarriers(GCRuntime* gc) : gc(gc) { /* * Clear needsIncrementalBarrier early so we don't do any write barriers * during sweeping. */ for (GCZonesIter zone(gc); !zone.done(); zone.next()) { if (zone->isGCMarking()) { MOZ_ASSERT(zone->needsIncrementalBarrier()); zone->setNeedsIncrementalBarrier(false); } MOZ_ASSERT(!zone->needsIncrementalBarrier()); } } AutoDisableBarriers::~AutoDisableBarriers() { for (GCZonesIter zone(gc); !zone.done(); zone.next()) { MOZ_ASSERT(!zone->needsIncrementalBarrier()); if (zone->isGCMarking()) { zone->setNeedsIncrementalBarrier(true); } } } static bool ShouldCleanUpEverything(JS::GCReason reason, JSGCInvocationKind gckind) { // During shutdown, we must clean everything up, for the sake of leak // detection. When a runtime has no contexts, or we're doing a GC before a // shutdown CC, those are strong indications that we're shutting down. return IsShutdownReason(reason) || gckind == GC_SHRINK; } static bool ShouldSweepOnBackgroundThread(JS::GCReason reason) { return reason != JS::GCReason::DESTROY_RUNTIME && CanUseExtraThreads(); } static bool NeedToCollectNursery(GCRuntime* gc) { return !gc->nursery().isEmpty() || !gc->storeBuffer().isEmpty(); } #ifdef DEBUG static const char* DescribeBudget(const SliceBudget& budget) { MOZ_ASSERT(TlsContext.get()->isMainThreadContext()); constexpr size_t length = 32; static char buffer[length]; budget.describe(buffer, length); return buffer; } #endif void GCRuntime::incrementalSlice(SliceBudget& budget, const MaybeInvocationKind& gckind, JS::GCReason reason) { MOZ_ASSERT_IF(isIncrementalGCInProgress(), isIncremental); AutoSetThreadIsPerformingGC performingGC; AutoGCSession session(this, JS::HeapState::MajorCollecting); // We don't allow off-thread parsing to start while we're doing an // incremental GC of the atoms zone. if (rt->activeGCInAtomsZone()) { session.maybeCheckAtomsAccess.emplace(rt); } bool destroyingRuntime = (reason == JS::GCReason::DESTROY_RUNTIME); initialState = incrementalState; isIncremental = !budget.isUnlimited(); #ifdef JS_GC_ZEAL // Do the incremental collection type specified by zeal mode if the collection // was triggered by runDebugGC() and incremental GC has not been cancelled by // resetIncrementalGC(). useZeal = isIncremental && reason == JS::GCReason::DEBUG_GC; #endif #ifdef DEBUG stats().log("Incremental: %d, lastMarkSlice: %d, useZeal: %d, budget: %s", bool(isIncremental), bool(lastMarkSlice), bool(useZeal), DescribeBudget(budget)); #endif if (useZeal && hasIncrementalTwoSliceZealMode()) { // Yields between slices occurs at predetermined points in these modes; the // budget is not used. |isIncremental| is still true. stats().log("Using unlimited budget for two-slice zeal mode"); budget.makeUnlimited(); } switch (incrementalState) { case State::NotActive: invocationKind = gckind.valueOr(GC_NORMAL); initialReason = reason; cleanUpEverything = ShouldCleanUpEverything(reason, invocationKind); sweepOnBackgroundThread = ShouldSweepOnBackgroundThread(reason); isCompacting = shouldCompact(); MOZ_ASSERT(!lastMarkSlice); rootsRemoved = false; lastGCStartTime_ = ReallyNow(); #ifdef DEBUG for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { zone->gcSweepGroupIndex = 0; } #endif incrementalState = State::Prepare; if (!beginPreparePhase(reason, session)) { incrementalState = State::NotActive; break; } if (useZeal && hasZealMode(ZealMode::YieldBeforeRootMarking)) { break; } [[fallthrough]]; case State::Prepare: if (waitForBackgroundTask(unmarkTask, budget, DontTriggerSliceWhenFinished) == NotFinished) { break; } incrementalState = State::MarkRoots; [[fallthrough]]; case State::MarkRoots: if (NeedToCollectNursery(this)) { collectNurseryFromMajorGC(gckind, reason); } endPreparePhase(reason); beginMarkPhase(session); // If we needed delayed marking for gray roots, then collect until done. if (isIncremental && !hasValidGrayRootsBuffer()) { budget.makeUnlimited(); isIncremental = false; stats().nonincremental(GCAbortReason::GrayRootBufferingFailed); } incrementalState = State::Mark; if (useZeal && hasZealMode(ZealMode::YieldBeforeMarking)) { break; } [[fallthrough]]; case State::Mark: if (mightSweepInThisSlice(budget.isUnlimited())) { // Trace wrapper rooters before marking if we might start sweeping in // this slice. rt->mainContextFromOwnThread()->traceWrapperGCRooters(&marker); } { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::MARK); if (markUntilBudgetExhausted(budget) == NotFinished) { break; } } MOZ_ASSERT(marker.isDrained()); /* * There are a number of reasons why we break out of collection here, * either ending the slice or to run a new interation of the loop in * GCRuntime::collect() */ /* * In incremental GCs where we have already performed more than one * slice we yield after marking with the aim of starting the sweep in * the next slice, since the first slice of sweeping can be expensive. * * This is modified by the various zeal modes. We don't yield in * YieldBeforeMarking mode and we always yield in YieldBeforeSweeping * mode. * * We will need to mark anything new on the stack when we resume, so * we stay in Mark state. */ if (isIncremental && !lastMarkSlice) { if ((initialState == State::Mark && !(useZeal && hasZealMode(ZealMode::YieldBeforeMarking))) || (useZeal && hasZealMode(ZealMode::YieldBeforeSweeping))) { lastMarkSlice = true; stats().log("Yielding before starting sweeping"); break; } } incrementalState = State::Sweep; lastMarkSlice = false; beginSweepPhase(reason, session); [[fallthrough]]; case State::Sweep: if (storeBuffer().mayHavePointersToDeadCells()) { collectNurseryFromMajorGC(gckind, reason); } if (initialState == State::Sweep) { rt->mainContextFromOwnThread()->traceWrapperGCRooters(&marker); } if (performSweepActions(budget) == NotFinished) { break; } endSweepPhase(destroyingRuntime); incrementalState = State::Finalize; [[fallthrough]]; case State::Finalize: if (waitForBackgroundTask(sweepTask, budget, TriggerSliceWhenFinished) == NotFinished) { break; } assertBackgroundSweepingFinished(); { // Sweep the zones list now that background finalization is finished to // remove and free dead zones, compartments and realms. gcstats::AutoPhase ap1(stats(), gcstats::PhaseKind::SWEEP); gcstats::AutoPhase ap2(stats(), gcstats::PhaseKind::DESTROY); JSFreeOp fop(rt); sweepZones(&fop, destroyingRuntime); } MOZ_ASSERT(!startedCompacting); incrementalState = State::Compact; // Always yield before compacting since it is not incremental. if (isCompacting && !budget.isUnlimited()) { break; } [[fallthrough]]; case State::Compact: if (isCompacting) { if (NeedToCollectNursery(this)) { collectNurseryFromMajorGC(gckind, reason); } storeBuffer().checkEmpty(); if (!startedCompacting) { beginCompactPhase(); } if (compactPhase(reason, budget, session) == NotFinished) { break; } endCompactPhase(); } startDecommit(); incrementalState = State::Decommit; [[fallthrough]]; case State::Decommit: if (waitForBackgroundTask(decommitTask, budget, TriggerSliceWhenFinished) == NotFinished) { break; } incrementalState = State::Finish; [[fallthrough]]; case State::Finish: finishCollection(); incrementalState = State::NotActive; break; } MOZ_ASSERT(safeToYield); MOZ_ASSERT(marker.markColor() == MarkColor::Black); } void GCRuntime::collectNurseryFromMajorGC(const MaybeInvocationKind& gckind, JS::GCReason reason) { collectNursery(gckind.valueOr(GC_NORMAL), reason, gcstats::PhaseKind::EVICT_NURSERY_FOR_MAJOR_GC); } bool GCRuntime::hasForegroundWork() const { switch (incrementalState) { case State::NotActive: // Incremental GC is not running and no work is pending. return false; case State::Prepare: // We yield in the Prepare state after starting unmarking. return !unmarkTask.wasStarted(); case State::Finalize: // We yield in the Finalize state to wait for background sweeping. return !isBackgroundSweeping(); case State::Decommit: // We yield in the Decommit state to wait for background decommit. return !decommitTask.wasStarted(); default: // In all other states there is still work to do. return true; } } IncrementalProgress GCRuntime::waitForBackgroundTask( GCParallelTask& task, const SliceBudget& budget, ShouldTriggerSliceWhenFinished triggerSlice) { // In incremental collections, yield if the task has not finished and request // a slice to notify us when this happens. if (!budget.isUnlimited()) { AutoLockHelperThreadState lock; if (task.wasStarted(lock)) { if (triggerSlice) { requestSliceAfterBackgroundTask = true; } return NotFinished; } } // Otherwise in non-incremental collections, wait here. gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::WAIT_BACKGROUND_THREAD); task.join(); if (triggerSlice) { cancelRequestedGCAfterBackgroundTask(); } return Finished; } GCAbortReason gc::IsIncrementalGCUnsafe(JSRuntime* rt) { MOZ_ASSERT(!rt->mainContextFromOwnThread()->suppressGC); if (!rt->gc.isIncrementalGCAllowed()) { return GCAbortReason::IncrementalDisabled; } return GCAbortReason::None; } inline void GCRuntime::checkZoneIsScheduled(Zone* zone, JS::GCReason reason, const char* trigger) { #ifdef DEBUG if (zone->isGCScheduled()) { return; } fprintf(stderr, "checkZoneIsScheduled: Zone %p not scheduled as expected in %s GC " "for %s trigger\n", zone, JS::ExplainGCReason(reason), trigger); for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { fprintf(stderr, " Zone %p:%s%s\n", zone.get(), zone->isAtomsZone() ? " atoms" : "", zone->isGCScheduled() ? " scheduled" : ""); } fflush(stderr); MOZ_CRASH("Zone not scheduled"); #endif } GCRuntime::IncrementalResult GCRuntime::budgetIncrementalGC( bool nonincrementalByAPI, JS::GCReason reason, SliceBudget& budget) { if (nonincrementalByAPI) { stats().nonincremental(GCAbortReason::NonIncrementalRequested); budget.makeUnlimited(); // Reset any in progress incremental GC if this was triggered via the // API. This isn't required for correctness, but sometimes during tests // the caller expects this GC to collect certain objects, and we need // to make sure to collect everything possible. if (reason != JS::GCReason::ALLOC_TRIGGER) { return resetIncrementalGC(GCAbortReason::NonIncrementalRequested); } return IncrementalResult::Ok; } if (reason == JS::GCReason::ABORT_GC) { budget.makeUnlimited(); stats().nonincremental(GCAbortReason::AbortRequested); return resetIncrementalGC(GCAbortReason::AbortRequested); } if (!budget.isUnlimited()) { GCAbortReason unsafeReason = IsIncrementalGCUnsafe(rt); if (unsafeReason == GCAbortReason::None) { if (reason == JS::GCReason::COMPARTMENT_REVIVED) { unsafeReason = GCAbortReason::CompartmentRevived; } else if (!incrementalGCEnabled) { unsafeReason = GCAbortReason::ModeChange; } } if (unsafeReason != GCAbortReason::None) { budget.makeUnlimited(); stats().nonincremental(unsafeReason); return resetIncrementalGC(unsafeReason); } } GCAbortReason resetReason = GCAbortReason::None; for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { if (!zone->canCollect()) { continue; } if (zone->gcHeapSize.bytes() >= zone->gcHeapThreshold.incrementalLimitBytes()) { checkZoneIsScheduled(zone, reason, "GC bytes"); budget.makeUnlimited(); stats().nonincremental(GCAbortReason::GCBytesTrigger); if (zone->wasGCStarted() && zone->gcState() > Zone::Sweep) { resetReason = GCAbortReason::GCBytesTrigger; } } if (zone->mallocHeapSize.bytes() >= zone->mallocHeapThreshold.incrementalLimitBytes()) { checkZoneIsScheduled(zone, reason, "malloc bytes"); budget.makeUnlimited(); stats().nonincremental(GCAbortReason::MallocBytesTrigger); if (zone->wasGCStarted() && zone->gcState() > Zone::Sweep) { resetReason = GCAbortReason::MallocBytesTrigger; } } if (zone->jitHeapSize.bytes() >= zone->jitHeapThreshold.incrementalLimitBytes()) { checkZoneIsScheduled(zone, reason, "JIT code bytes"); budget.makeUnlimited(); stats().nonincremental(GCAbortReason::JitCodeBytesTrigger); if (zone->wasGCStarted() && zone->gcState() > Zone::Sweep) { resetReason = GCAbortReason::JitCodeBytesTrigger; } } if (isIncrementalGCInProgress() && zone->isGCScheduled() != zone->wasGCStarted()) { budget.makeUnlimited(); resetReason = GCAbortReason::ZoneChange; } } if (resetReason != GCAbortReason::None) { return resetIncrementalGC(resetReason); } return IncrementalResult::Ok; } void GCRuntime::maybeIncreaseSliceBudget(SliceBudget& budget) { if (js::SupportDifferentialTesting()) { return; } // Increase time budget for long-running incremental collections. Enforce a // minimum time budget that increases linearly with time/slice count up to a // maximum. if (budget.isTimeBudget() && !budget.isUnlimited() && isIncrementalGCInProgress()) { // All times are in milliseconds. struct BudgetAtTime { double time; double budget; }; const BudgetAtTime MinBudgetStart{1500, 0.0}; const BudgetAtTime MinBudgetEnd{2500, 100.0}; double totalTime = (ReallyNow() - lastGCStartTime()).ToMilliseconds(); double minBudget = LinearInterpolate(totalTime, MinBudgetStart.time, MinBudgetStart.budget, MinBudgetEnd.time, MinBudgetEnd.budget); if (budget.timeBudget.budget < minBudget) { budget = SliceBudget(TimeBudget(minBudget)); } } } static void ScheduleZones(GCRuntime* gc) { for (ZonesIter zone(gc, WithAtoms); !zone.done(); zone.next()) { if (!zone->canCollect()) { continue; } if (!gc->isPerZoneGCEnabled()) { zone->scheduleGC(); } // To avoid resets, continue to collect any zones that were being // collected in a previous slice. if (gc->isIncrementalGCInProgress() && zone->wasGCStarted()) { zone->scheduleGC(); } // This is a heuristic to reduce the total number of collections. bool inHighFrequencyMode = gc->schedulingState.inHighFrequencyGCMode(); if (zone->gcHeapSize.bytes() >= zone->gcHeapThreshold.eagerAllocTrigger(inHighFrequencyMode) || zone->mallocHeapSize.bytes() >= zone->mallocHeapThreshold.eagerAllocTrigger(inHighFrequencyMode) || zone->jitHeapSize.bytes() >= zone->jitHeapThreshold.startBytes()) { zone->scheduleGC(); } } } static void UnscheduleZones(GCRuntime* gc) { for (ZonesIter zone(gc->rt, WithAtoms); !zone.done(); zone.next()) { zone->unscheduleGC(); } } class js::gc::AutoCallGCCallbacks { GCRuntime& gc_; JS::GCReason reason_; public: explicit AutoCallGCCallbacks(GCRuntime& gc, JS::GCReason reason) : gc_(gc), reason_(reason) { gc_.maybeCallGCCallback(JSGC_BEGIN, reason); } ~AutoCallGCCallbacks() { gc_.maybeCallGCCallback(JSGC_END, reason_); } }; void GCRuntime::maybeCallGCCallback(JSGCStatus status, JS::GCReason reason) { if (!gcCallback.ref().op) { return; } if (isIncrementalGCInProgress()) { return; } if (gcCallbackDepth == 0) { // Save scheduled zone information in case the callback clears it. for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { zone->gcScheduledSaved_ = zone->gcScheduled_; } } gcCallbackDepth++; callGCCallback(status, reason); MOZ_ASSERT(gcCallbackDepth != 0); gcCallbackDepth--; if (gcCallbackDepth == 0) { // Ensure any zone that was originally scheduled stays scheduled. for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { zone->gcScheduled_ = zone->gcScheduled_ || zone->gcScheduledSaved_; } } } /* * We disable inlining to ensure that the bottom of the stack with possible GC * roots recorded in MarkRuntime excludes any pointers we use during the marking * implementation. */ MOZ_NEVER_INLINE GCRuntime::IncrementalResult GCRuntime::gcCycle( bool nonincrementalByAPI, SliceBudget budget, const MaybeInvocationKind& gckind, JS::GCReason reason) { // Assert if this is a GC unsafe region. rt->mainContextFromOwnThread()->verifyIsSafeToGC(); // It's ok if threads other than the main thread have suppressGC set, as // they are operating on zones which will not be collected from here. MOZ_ASSERT(!rt->mainContextFromOwnThread()->suppressGC); // This reason is used internally. See below. MOZ_ASSERT(reason != JS::GCReason::RESET); // Background finalization and decommit are finished by definition before we // can start a new major GC. Background allocation may still be running, but // that's OK because chunk pools are protected by the GC lock. if (!isIncrementalGCInProgress()) { assertBackgroundSweepingFinished(); MOZ_ASSERT(decommitTask.isIdle()); } // Note that GC callbacks are allowed to re-enter GC. AutoCallGCCallbacks callCallbacks(*this, reason); // Increase slice budget for long running collections before it is recorded by // AutoGCSlice. maybeIncreaseSliceBudget(budget); ScheduleZones(this); gcstats::AutoGCSlice agc(stats(), scanZonesBeforeGC(), gckind.valueOr(invocationKind), budget, reason); IncrementalResult result = budgetIncrementalGC(nonincrementalByAPI, reason, budget); if (result == IncrementalResult::ResetIncremental) { if (incrementalState == State::NotActive) { // The collection was reset and has finished. return result; } // The collection was reset but we must finish up some remaining work. reason = JS::GCReason::RESET; } majorGCTriggerReason = JS::GCReason::NO_REASON; MOZ_ASSERT(!stats().hasTrigger()); incGcNumber(); incGcSliceNumber(); gcprobes::MajorGCStart(); incrementalSlice(budget, gckind, reason); gcprobes::MajorGCEnd(); MOZ_ASSERT_IF(result == IncrementalResult::ResetIncremental, !isIncrementalGCInProgress()); return result; } void GCRuntime::waitForBackgroundTasksBeforeSlice() { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::WAIT_BACKGROUND_THREAD); // Background finalization and decommit are finished by definition before we // can start a new major GC. if (!isIncrementalGCInProgress()) { assertBackgroundSweepingFinished(); MOZ_ASSERT(decommitTask.isIdle()); } // We must also wait for background allocation to finish so we can avoid // taking the GC lock when manipulating the chunks during the GC. The // background alloc task can run between slices, so we must wait for it at the // start of every slice. // // TODO: Is this still necessary? allocTask.cancelAndWait(); } inline bool GCRuntime::mightSweepInThisSlice(bool nonIncremental) { MOZ_ASSERT(incrementalState < State::Sweep); return nonIncremental || lastMarkSlice || hasIncrementalTwoSliceZealMode(); } #ifdef JS_GC_ZEAL static bool IsDeterministicGCReason(JS::GCReason reason) { switch (reason) { case JS::GCReason::API: case JS::GCReason::DESTROY_RUNTIME: case JS::GCReason::LAST_DITCH: case JS::GCReason::TOO_MUCH_MALLOC: case JS::GCReason::TOO_MUCH_WASM_MEMORY: case JS::GCReason::TOO_MUCH_JIT_CODE: case JS::GCReason::ALLOC_TRIGGER: case JS::GCReason::DEBUG_GC: case JS::GCReason::CC_FORCED: case JS::GCReason::SHUTDOWN_CC: case JS::GCReason::ABORT_GC: case JS::GCReason::DISABLE_GENERATIONAL_GC: case JS::GCReason::FINISH_GC: case JS::GCReason::PREPARE_FOR_TRACING: return true; default: return false; } } #endif gcstats::ZoneGCStats GCRuntime::scanZonesBeforeGC() { gcstats::ZoneGCStats zoneStats; for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { zoneStats.zoneCount++; zoneStats.compartmentCount += zone->compartments().length(); if (zone->canCollect()) { zoneStats.collectableZoneCount++; if (zone->isGCScheduled()) { zoneStats.collectedZoneCount++; zoneStats.collectedCompartmentCount += zone->compartments().length(); } } } return zoneStats; } // The GC can only clean up scheduledForDestruction realms that were marked live // by a barrier (e.g. by RemapWrappers from a navigation event). It is also // common to have realms held live because they are part of a cycle in gecko, // e.g. involving the HTMLDocument wrapper. In this case, we need to run the // CycleCollector in order to remove these edges before the realm can be freed. void GCRuntime::maybeDoCycleCollection() { const static float ExcessiveGrayRealms = 0.8f; const static size_t LimitGrayRealms = 200; size_t realmsTotal = 0; size_t realmsGray = 0; for (RealmsIter realm(rt); !realm.done(); realm.next()) { ++realmsTotal; GlobalObject* global = realm->unsafeUnbarrieredMaybeGlobal(); if (global && global->isMarkedGray()) { ++realmsGray; } } float grayFraction = float(realmsGray) / float(realmsTotal); if (grayFraction > ExcessiveGrayRealms || realmsGray > LimitGrayRealms) { callDoCycleCollectionCallback(rt->mainContextFromOwnThread()); } } void GCRuntime::checkCanCallAPI() { MOZ_RELEASE_ASSERT(CurrentThreadCanAccessRuntime(rt)); /* If we attempt to invoke the GC while we are running in the GC, assert. */ MOZ_RELEASE_ASSERT(!JS::RuntimeHeapIsBusy()); } bool GCRuntime::checkIfGCAllowedInCurrentState(JS::GCReason reason) { if (rt->mainContextFromOwnThread()->suppressGC) { return false; } // Only allow shutdown GCs when we're destroying the runtime. This keeps // the GC callback from triggering a nested GC and resetting global state. if (rt->isBeingDestroyed() && !IsShutdownReason(reason)) { return false; } #ifdef JS_GC_ZEAL if (deterministicOnly && !IsDeterministicGCReason(reason)) { return false; } #endif return true; } bool GCRuntime::shouldRepeatForDeadZone(JS::GCReason reason) { MOZ_ASSERT_IF(reason == JS::GCReason::COMPARTMENT_REVIVED, !isIncremental); MOZ_ASSERT(!isIncrementalGCInProgress()); if (!isIncremental) { return false; } for (CompartmentsIter c(rt); !c.done(); c.next()) { if (c->gcState.scheduledForDestruction) { return true; } } return false; } struct MOZ_RAII AutoSetZoneSliceThresholds { explicit AutoSetZoneSliceThresholds(GCRuntime* gc) : gc(gc) { // On entry, zones that are already collecting should have a slice threshold // set. for (ZonesIter zone(gc, WithAtoms); !zone.done(); zone.next()) { MOZ_ASSERT((zone->gcState() > Zone::Prepare) == zone->gcHeapThreshold.hasSliceThreshold()); MOZ_ASSERT((zone->gcState() > Zone::Prepare) == zone->mallocHeapThreshold.hasSliceThreshold()); } } ~AutoSetZoneSliceThresholds() { // On exit, update the thresholds for all collecting zones. for (ZonesIter zone(gc, WithAtoms); !zone.done(); zone.next()) { if (zone->gcState() > Zone::Prepare) { zone->setGCSliceThresholds(*gc); } else { MOZ_ASSERT(!zone->gcHeapThreshold.hasSliceThreshold()); MOZ_ASSERT(!zone->mallocHeapThreshold.hasSliceThreshold()); } } } GCRuntime* gc; }; void GCRuntime::collect(bool nonincrementalByAPI, SliceBudget budget, const MaybeInvocationKind& gckindArg, JS::GCReason reason) { MOZ_ASSERT(reason != JS::GCReason::NO_REASON); MaybeInvocationKind gckind = gckindArg; MOZ_ASSERT_IF(!isIncrementalGCInProgress(), gckind.isSome()); // Checks run for each request, even if we do not actually GC. checkCanCallAPI(); // Check if we are allowed to GC at this time before proceeding. if (!checkIfGCAllowedInCurrentState(reason)) { return; } stats().log("GC starting in state %s", StateName(incrementalState)); AutoTraceLog logGC(TraceLoggerForCurrentThread(), TraceLogger_GC); AutoStopVerifyingBarriers av(rt, IsShutdownReason(reason)); AutoEnqueuePendingParseTasksAfterGC aept(*this); AutoMaybeLeaveAtomsZone leaveAtomsZone(rt->mainContextFromOwnThread()); AutoSetZoneSliceThresholds sliceThresholds(this); #ifdef DEBUG if (IsShutdownReason(reason)) { marker.markQueue.clear(); marker.queuePos = 0; } #endif bool repeat; do { IncrementalResult cycleResult = gcCycle(nonincrementalByAPI, budget, gckind, reason); if (reason == JS::GCReason::ABORT_GC) { MOZ_ASSERT(!isIncrementalGCInProgress()); stats().log("GC aborted by request"); break; } /* * Sometimes when we finish a GC we need to immediately start a new one. * This happens in the following cases: * - when we reset the current GC * - when finalizers drop roots during shutdown * - when zones that we thought were dead at the start of GC are * not collected (see the large comment in beginMarkPhase) */ repeat = false; if (!isIncrementalGCInProgress()) { if (cycleResult == ResetIncremental) { repeat = true; } else if (rootsRemoved && IsShutdownReason(reason)) { /* Need to re-schedule all zones for GC. */ JS::PrepareForFullGC(rt->mainContextFromOwnThread()); repeat = true; reason = JS::GCReason::ROOTS_REMOVED; } else if (shouldRepeatForDeadZone(reason)) { repeat = true; reason = JS::GCReason::COMPARTMENT_REVIVED; } } if (repeat) { gckind = Some(invocationKind); } } while (repeat); if (reason == JS::GCReason::COMPARTMENT_REVIVED) { maybeDoCycleCollection(); } #ifdef JS_GC_ZEAL if (hasZealMode(ZealMode::CheckHeapAfterGC)) { gcstats::AutoPhase ap(stats(), gcstats::PhaseKind::TRACE_HEAP); CheckHeapAfterGC(rt); } if (hasZealMode(ZealMode::CheckGrayMarking) && !isIncrementalGCInProgress()) { MOZ_RELEASE_ASSERT(CheckGrayMarkingState(rt)); } #endif stats().log("GC ending in state %s", StateName(incrementalState)); UnscheduleZones(this); } js::AutoEnqueuePendingParseTasksAfterGC:: ~AutoEnqueuePendingParseTasksAfterGC() { if (!OffThreadParsingMustWaitForGC(gc_.rt)) { EnqueuePendingParseTasksAfterGC(gc_.rt); } } SliceBudget GCRuntime::defaultBudget(JS::GCReason reason, int64_t millis) { if (millis == 0) { if (reason == JS::GCReason::ALLOC_TRIGGER) { millis = defaultSliceBudgetMS(); } else if (schedulingState.inHighFrequencyGCMode()) { millis = defaultSliceBudgetMS() * IGC_MARK_SLICE_MULTIPLIER; } else { millis = defaultSliceBudgetMS(); } } return SliceBudget(TimeBudget(millis)); } void GCRuntime::gc(JSGCInvocationKind gckind, JS::GCReason reason) { collect(true, SliceBudget::unlimited(), mozilla::Some(gckind), reason); } void GCRuntime::startGC(JSGCInvocationKind gckind, JS::GCReason reason, int64_t millis) { MOZ_ASSERT(!isIncrementalGCInProgress()); if (!JS::IsIncrementalGCEnabled(rt->mainContextFromOwnThread())) { gc(gckind, reason); return; } collect(false, defaultBudget(reason, millis), Some(gckind), reason); } void GCRuntime::gcSlice(JS::GCReason reason, int64_t millis) { MOZ_ASSERT(isIncrementalGCInProgress()); collect(false, defaultBudget(reason, millis), Nothing(), reason); } void GCRuntime::finishGC(JS::GCReason reason) { MOZ_ASSERT(isIncrementalGCInProgress()); // If we're not collecting because we're out of memory then skip the // compacting phase if we need to finish an ongoing incremental GC // non-incrementally to avoid janking the browser. if (!IsOOMReason(initialReason)) { if (incrementalState == State::Compact) { abortGC(); return; } isCompacting = false; } collect(false, SliceBudget::unlimited(), Nothing(), reason); } void GCRuntime::abortGC() { MOZ_ASSERT(isIncrementalGCInProgress()); checkCanCallAPI(); MOZ_ASSERT(!rt->mainContextFromOwnThread()->suppressGC); collect(false, SliceBudget::unlimited(), Nothing(), JS::GCReason::ABORT_GC); } static bool ZonesSelected(GCRuntime* gc) { for (ZonesIter zone(gc, WithAtoms); !zone.done(); zone.next()) { if (zone->isGCScheduled()) { return true; } } return false; } void GCRuntime::startDebugGC(JSGCInvocationKind gckind, SliceBudget& budget) { MOZ_ASSERT(!isIncrementalGCInProgress()); if (!ZonesSelected(this)) { JS::PrepareForFullGC(rt->mainContextFromOwnThread()); } collect(false, budget, Some(gckind), JS::GCReason::DEBUG_GC); } void GCRuntime::debugGCSlice(SliceBudget& budget) { MOZ_ASSERT(isIncrementalGCInProgress()); if (!ZonesSelected(this)) { JS::PrepareForIncrementalGC(rt->mainContextFromOwnThread()); } collect(false, budget, Nothing(), JS::GCReason::DEBUG_GC); } /* Schedule a full GC unless a zone will already be collected. */ void js::PrepareForDebugGC(JSRuntime* rt) { if (!ZonesSelected(&rt->gc)) { JS::PrepareForFullGC(rt->mainContextFromOwnThread()); } } void GCRuntime::onOutOfMallocMemory() { // Stop allocating new chunks. allocTask.cancelAndWait(); // Make sure we release anything queued for release. decommitTask.join(); nursery().joinDecommitTask(); // Wait for background free of nursery huge slots to finish. sweepTask.join(); AutoLockGC lock(this); onOutOfMallocMemory(lock); } void GCRuntime::onOutOfMallocMemory(const AutoLockGC& lock) { // Release any relocated arenas we may be holding on to, without releasing // the GC lock. releaseHeldRelocatedArenasWithoutUnlocking(lock); // Throw away any excess chunks we have lying around. freeEmptyChunks(lock); // Immediately decommit as many arenas as possible in the hopes that this // might let the OS scrape together enough pages to satisfy the failing // malloc request. decommitFreeArenasWithoutUnlocking(lock); } void GCRuntime::minorGC(JS::GCReason reason, gcstats::PhaseKind phase) { MOZ_ASSERT(!JS::RuntimeHeapIsBusy()); MOZ_ASSERT_IF(reason == JS::GCReason::EVICT_NURSERY, !rt->mainContextFromOwnThread()->suppressGC); if (rt->mainContextFromOwnThread()->suppressGC) { return; } incGcNumber(); collectNursery(GC_NORMAL, reason, phase); #ifdef JS_GC_ZEAL if (hasZealMode(ZealMode::CheckHeapAfterGC)) { gcstats::AutoPhase ap(stats(), phase); CheckHeapAfterGC(rt); } #endif for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { maybeTriggerGCAfterAlloc(zone); maybeTriggerGCAfterMalloc(zone); } } void GCRuntime::collectNursery(JSGCInvocationKind kind, JS::GCReason reason, gcstats::PhaseKind phase) { AutoMaybeLeaveAtomsZone leaveAtomsZone(rt->mainContextFromOwnThread()); // Note that we aren't collecting the updated alloc counts from any helper // threads. We should be but I'm not sure where to add that // synchronisation. uint32_t numAllocs = rt->mainContextFromOwnThread()->getAndResetAllocsThisZoneSinceMinorGC(); for (ZonesIter zone(this, WithAtoms); !zone.done(); zone.next()) { numAllocs += zone->getAndResetTenuredAllocsSinceMinorGC(); } stats().setAllocsSinceMinorGCTenured(numAllocs); gcstats::AutoPhase ap(stats(), phase); nursery().clearMinorGCRequest(); TraceLoggerThread* logger = TraceLoggerForCurrentThread(); AutoTraceLog logMinorGC(logger, TraceLogger_MinorGC); nursery().collect(kind, reason); MOZ_ASSERT(nursery().isEmpty()); startBackgroundFreeAfterMinorGC(); } void GCRuntime::startBackgroundFreeAfterMinorGC() { MOZ_ASSERT(nursery().isEmpty()); { AutoLockHelperThreadState lock; lifoBlocksToFree.ref().transferFrom(&lifoBlocksToFreeAfterMinorGC.ref()); if (lifoBlocksToFree.ref().isEmpty() && buffersToFreeAfterMinorGC.ref().empty()) { return; } } startBackgroundFree(); } JS::AutoDisableGenerationalGC::AutoDisableGenerationalGC(JSContext* cx) : cx(cx) { if (!cx->generationalDisabled) { cx->runtime()->gc.evictNursery(JS::GCReason::DISABLE_GENERATIONAL_GC); cx->nursery().disable(); } ++cx->generationalDisabled; } JS::AutoDisableGenerationalGC::~AutoDisableGenerationalGC() { if (--cx->generationalDisabled == 0 && cx->runtime()->gc.tunables.gcMaxNurseryBytes() > 0) { cx->nursery().enable(); } } JS_PUBLIC_API bool JS::IsGenerationalGCEnabled(JSRuntime* rt) { return !rt->mainContextFromOwnThread()->generationalDisabled; } bool GCRuntime::gcIfRequested() { // This method returns whether a major GC was performed. if (nursery().minorGCRequested()) { minorGC(nursery().minorGCTriggerReason()); } if (majorGCRequested()) { if (majorGCTriggerReason == JS::GCReason::DELAYED_ATOMS_GC && !rt->mainContextFromOwnThread()->canCollectAtoms()) { // A GC was requested to collect the atoms zone, but it's no longer // possible. Skip this collection. majorGCTriggerReason = JS::GCReason::NO_REASON; return false; } if (!isIncrementalGCInProgress()) { startGC(GC_NORMAL, majorGCTriggerReason); } else { gcSlice(majorGCTriggerReason); } return true; } return false; } void js::gc::FinishGC(JSContext* cx, JS::GCReason reason) { // Calling this when GC is suppressed won't have any effect. MOZ_ASSERT(!cx->suppressGC); // GC callbacks may run arbitrary code, including JS. Check this regardless of // whether we GC for this invocation. MOZ_ASSERT(cx->isNurseryAllocAllowed()); if (JS::IsIncrementalGCInProgress(cx)) { JS::PrepareForIncrementalGC(cx); JS::FinishIncrementalGC(cx, reason); } } void js::gc::WaitForBackgroundTasks(JSContext* cx) { cx->runtime()->gc.waitForBackgroundTasks(); } void GCRuntime::waitForBackgroundTasks() { MOZ_ASSERT(!isIncrementalGCInProgress()); MOZ_ASSERT(sweepTask.isIdle()); MOZ_ASSERT(decommitTask.isIdle()); MOZ_ASSERT(markTask.isIdle()); allocTask.join(); freeTask.join(); nursery().joinDecommitTask(); } Realm* js::NewRealm(JSContext* cx, JSPrincipals* principals, const JS::RealmOptions& options) { JSRuntime* rt = cx->runtime(); JS_AbortIfWrongThread(cx); UniquePtr zoneHolder; UniquePtr compHolder; Compartment* comp = nullptr; Zone* zone = nullptr; JS::CompartmentSpecifier compSpec = options.creationOptions().compartmentSpecifier(); switch (compSpec) { case JS::CompartmentSpecifier::NewCompartmentInSystemZone: // systemZone might be null here, in which case we'll make a zone and // set this field below. zone = rt->gc.systemZone; break; case JS::CompartmentSpecifier::NewCompartmentInExistingZone: zone = options.creationOptions().zone(); MOZ_ASSERT(zone); break; case JS::CompartmentSpecifier::ExistingCompartment: comp = options.creationOptions().compartment(); zone = comp->zone(); break; case JS::CompartmentSpecifier::NewCompartmentAndZone: case JS::CompartmentSpecifier::NewCompartmentInSelfHostingZone: break; } if (!zone) { Zone::Kind kind = Zone::NormalZone; const JSPrincipals* trusted = rt->trustedPrincipals(); if (compSpec == JS::CompartmentSpecifier::NewCompartmentInSelfHostingZone) { MOZ_ASSERT(!rt->hasInitializedSelfHosting()); kind = Zone::SelfHostingZone; } else if (compSpec == JS::CompartmentSpecifier::NewCompartmentInSystemZone || (principals && principals == trusted)) { kind = Zone::SystemZone; } zoneHolder = MakeUnique(cx->runtime(), kind); if (!zoneHolder || !zoneHolder->init()) { ReportOutOfMemory(cx); return nullptr; } zone = zoneHolder.get(); } bool invisibleToDebugger = options.creationOptions().invisibleToDebugger(); if (comp) { // Debugger visibility is per-compartment, not per-realm, so make sure the // new realm's visibility matches its compartment's. MOZ_ASSERT(comp->invisibleToDebugger() == invisibleToDebugger); } else { compHolder = cx->make_unique(zone, invisibleToDebugger); if (!compHolder) { return nullptr; } comp = compHolder.get(); } UniquePtr realm(cx->new_(comp, options)); if (!realm || !realm->init(cx, principals)) { return nullptr; } // Make sure we don't put system and non-system realms in the same // compartment. if (!compHolder) { MOZ_RELEASE_ASSERT(realm->isSystem() == IsSystemCompartment(comp)); } AutoLockGC lock(rt); // Reserve space in the Vectors before we start mutating them. if (!comp->realms().reserve(comp->realms().length() + 1) || (compHolder && !zone->compartments().reserve(zone->compartments().length() + 1)) || (zoneHolder && !rt->gc.zones().reserve(rt->gc.zones().length() + 1))) { ReportOutOfMemory(cx); return nullptr; } // After this everything must be infallible. comp->realms().infallibleAppend(realm.get()); if (compHolder) { zone->compartments().infallibleAppend(compHolder.release()); } if (zoneHolder) { rt->gc.zones().infallibleAppend(zoneHolder.release()); // Lazily set the runtime's system zone. if (compSpec == JS::CompartmentSpecifier::NewCompartmentInSystemZone) { MOZ_RELEASE_ASSERT(!rt->gc.systemZone); MOZ_ASSERT(zone->isSystemZone()); rt->gc.systemZone = zone; } } return realm.release(); } void gc::MergeRealms(Realm* source, Realm* target) { JSRuntime* rt = source->runtimeFromMainThread(); rt->gc.mergeRealms(source, target); rt->gc.maybeTriggerGCAfterAlloc(target->zone()); rt->gc.maybeTriggerGCAfterMalloc(target->zone()); } void GCRuntime::mergeRealms(Realm* source, Realm* target) { // The source realm must be specifically flagged as mergable. This // also implies that the realm is not visible to the debugger. MOZ_ASSERT(source->creationOptions().mergeable()); MOZ_ASSERT(source->creationOptions().invisibleToDebugger()); MOZ_ASSERT(!source->hasBeenEnteredIgnoringJit()); MOZ_ASSERT(source->zone()->compartments().length() == 1); JSContext* cx = rt->mainContextFromOwnThread(); MOZ_ASSERT(!source->zone()->wasGCStarted()); JS::AutoAssertNoGC nogc(cx); AutoTraceSession session(rt); // Cleanup tables and other state in the source realm/zone that will be // meaningless after merging into the target realm/zone. source->clearTables(); source->zone()->clearTables(); source->unsetIsDebuggee(); // Release any relocated arenas which we may be holding on to as they might // be in the source zone releaseHeldRelocatedArenas(); // Fixup realm pointers in source to refer to target, and make sure // type information generations are in sync. GlobalObject* global = target->maybeGlobal(); MOZ_ASSERT(global); AssertTargetIsNotGray(global); for (auto group = source->zone()->cellIterUnsafe(); !group.done(); group.next()) { // Replace placeholder object prototypes with the correct prototype in // the target realm. TaggedProto proto(group->proto()); if (proto.isObject()) { JSObject* obj = proto.toObject(); if (GlobalObject::isOffThreadPrototypePlaceholder(obj)) { JSObject* targetProto = global->getPrototypeForOffThreadPlaceholder(obj); MOZ_ASSERT(targetProto->isDelegate()); group->setProtoUnchecked(TaggedProto(targetProto)); } } group->realm_ = target; } // Fixup zone pointers in source's zone to refer to target's zone. bool targetZoneIsCollecting = target->zone()->gcState() > Zone::Prepare; for (auto thingKind : AllAllocKinds()) { for (ArenaIter aiter(source->zone(), thingKind); !aiter.done(); aiter.next()) { Arena* arena = aiter.get(); arena->zone = target->zone(); if (MOZ_UNLIKELY(targetZoneIsCollecting)) { // If we are currently collecting the target zone then we must // treat all merged things as if they were allocated during the // collection. for (ArenaCellIter cell(arena); !cell.done(); cell.next()) { MOZ_ASSERT(!cell->isMarkedAny()); cell->markBlack(); } } } } // The source should be the only realm in its zone. for (RealmsInZoneIter r(source->zone()); !r.done(); r.next()) { MOZ_ASSERT(r.get() == source); } // Merge the allocator, stats and UIDs in source's zone into target's zone. target->zone()->arenas.adoptArenas(&source->zone()->arenas, targetZoneIsCollecting); target->zone()->addTenuredAllocsSinceMinorGC( source->zone()->getAndResetTenuredAllocsSinceMinorGC()); target->zone()->gcHeapSize.adopt(source->zone()->gcHeapSize); target->zone()->adoptUniqueIds(source->zone()); target->zone()->adoptMallocBytes(source->zone()); // Atoms which are marked in source's zone are now marked in target's zone. atomMarking.adoptMarkedAtoms(target->zone(), source->zone()); // The source Realm is a parse-only realm and should not have collected any // zone-tracked metadata. Zone* sourceZone = source->zone(); MOZ_ASSERT(!sourceZone->scriptLCovMap); MOZ_ASSERT(!sourceZone->scriptCountsMap); MOZ_ASSERT(!sourceZone->debugScriptMap); #ifdef MOZ_VTUNE MOZ_ASSERT(!sourceZone->scriptVTuneIdMap); #endif #ifdef JS_CACHEIR_SPEW MOZ_ASSERT(!sourceZone->scriptFinalWarmUpCountMap); #endif // The source realm is now completely empty, and is the only realm in its // compartment, which is the only compartment in its zone. Delete realm, // compartment and zone without waiting for this to be cleaned up by a full // GC. sourceZone->deleteEmptyCompartment(source->compartment()); deleteEmptyZone(sourceZone); } void GCRuntime::runDebugGC() { #ifdef JS_GC_ZEAL if (rt->mainContextFromOwnThread()->suppressGC) { return; } if (hasZealMode(ZealMode::GenerationalGC)) { return minorGC(JS::GCReason::DEBUG_GC); } PrepareForDebugGC(rt); auto budget = SliceBudget::unlimited(); if (hasZealMode(ZealMode::IncrementalMultipleSlices)) { /* * Start with a small slice limit and double it every slice. This * ensure that we get multiple slices, and collection runs to * completion. */ if (!isIncrementalGCInProgress()) { zealSliceBudget = zealFrequency / 2; } else { zealSliceBudget *= 2; } budget = SliceBudget(WorkBudget(zealSliceBudget)); js::gc::State initialState = incrementalState; Maybe gckind = isIncrementalGCInProgress() ? Nothing() : Some(GC_SHRINK); collect(false, budget, gckind, JS::GCReason::DEBUG_GC); /* Reset the slice size when we get to the sweep or compact phases. */ if ((initialState == State::Mark && incrementalState == State::Sweep) || (initialState == State::Sweep && incrementalState == State::Compact)) { zealSliceBudget = zealFrequency / 2; } } else if (hasIncrementalTwoSliceZealMode()) { // These modes trigger incremental GC that happens in two slices and the // supplied budget is ignored by incrementalSlice. budget = SliceBudget(WorkBudget(1)); Maybe gckind = isIncrementalGCInProgress() ? Nothing() : Some(GC_NORMAL); collect(false, budget, gckind, JS::GCReason::DEBUG_GC); } else if (hasZealMode(ZealMode::Compact)) { gc(GC_SHRINK, JS::GCReason::DEBUG_GC); } else { gc(GC_NORMAL, JS::GCReason::DEBUG_GC); } #endif } void GCRuntime::setFullCompartmentChecks(bool enabled) { MOZ_ASSERT(!JS::RuntimeHeapIsMajorCollecting()); fullCompartmentChecks = enabled; } void GCRuntime::notifyRootsRemoved() { rootsRemoved = true; #ifdef JS_GC_ZEAL /* Schedule a GC to happen "soon". */ if (hasZealMode(ZealMode::RootsChange)) { nextScheduled = 1; } #endif } #ifdef JS_GC_ZEAL bool GCRuntime::selectForMarking(JSObject* object) { MOZ_ASSERT(!JS::RuntimeHeapIsMajorCollecting()); return selectedForMarking.ref().get().append(object); } void GCRuntime::clearSelectedForMarking() { selectedForMarking.ref().get().clearAndFree(); } void GCRuntime::setDeterministic(bool enabled) { MOZ_ASSERT(!JS::RuntimeHeapIsMajorCollecting()); deterministicOnly = enabled; } #endif #ifdef DEBUG /* Should only be called manually under gdb */ void PreventGCDuringInteractiveDebug() { TlsContext.get()->suppressGC++; } #endif void js::ReleaseAllJITCode(JSFreeOp* fop) { js::CancelOffThreadIonCompile(fop->runtime()); for (ZonesIter zone(fop->runtime(), SkipAtoms); !zone.done(); zone.next()) { zone->setPreservingCode(false); zone->discardJitCode(fop); } for (RealmsIter realm(fop->runtime()); !realm.done(); realm.next()) { if (jit::JitRealm* jitRealm = realm->jitRealm()) { jitRealm->discardStubs(); } } } void ArenaLists::adoptArenas(ArenaLists* fromArenaLists, bool targetZoneIsCollecting) { // GC may be active so take the lock here so we can mutate the arena lists. AutoLockGC lock(runtime()); fromArenaLists->clearFreeLists(); for (auto thingKind : AllAllocKinds()) { MOZ_ASSERT(fromArenaLists->concurrentUse(thingKind) == ConcurrentUse::None); ArenaList* fromList = &fromArenaLists->arenaList(thingKind); ArenaList* toList = &arenaList(thingKind); fromList->check(); toList->check(); Arena* next; for (Arena* fromArena = fromList->head(); fromArena; fromArena = next) { // Copy fromArena->next before releasing/reinserting. next = fromArena->next; #ifdef DEBUG MOZ_ASSERT(!fromArena->isEmpty()); if (targetZoneIsCollecting) { fromArena->checkAllCellsMarkedBlack(); } else { fromArena->checkNoMarkedCells(); } #endif // If the target zone is being collected then we need to add the // arenas before the cursor because the collector assumes that the // cursor is always at the end of the list. This has the side-effect // of preventing allocation into any non-full arenas until the end // of the next GC. if (targetZoneIsCollecting) { toList->insertBeforeCursor(fromArena); } else { toList->insertAtCursor(fromArena); } } fromList->clear(); toList->check(); } } AutoSuppressGC::AutoSuppressGC(JSContext* cx) : suppressGC_(cx->suppressGC.ref()) { suppressGC_++; } #ifdef DEBUG AutoDisableProxyCheck::AutoDisableProxyCheck() { TlsContext.get()->disableStrictProxyChecking(); } AutoDisableProxyCheck::~AutoDisableProxyCheck() { TlsContext.get()->enableStrictProxyChecking(); } JS_FRIEND_API void JS::AssertGCThingMustBeTenured(JSObject* obj) { MOZ_ASSERT(obj->isTenured() && (!IsNurseryAllocable(obj->asTenured().getAllocKind()) || obj->getClass()->hasFinalize())); } JS_FRIEND_API void JS::AssertGCThingIsNotNurseryAllocable(Cell* cell) { MOZ_ASSERT(cell); MOZ_ASSERT(!cell->is() && !cell->is() && !cell->is()); } JS_FRIEND_API void js::gc::AssertGCThingHasType(js::gc::Cell* cell, JS::TraceKind kind) { if (!cell) { MOZ_ASSERT(kind == JS::TraceKind::Null); return; } MOZ_ASSERT(IsCellPointerValid(cell)); MOZ_ASSERT(cell->getTraceKind() == kind); } #endif #ifdef MOZ_DIAGNOSTIC_ASSERT_ENABLED JS::AutoAssertNoGC::AutoAssertNoGC(JSContext* maybecx) : cx_(maybecx ? maybecx : TlsContext.get()) { if (cx_) { cx_->inUnsafeRegion++; } } JS::AutoAssertNoGC::~AutoAssertNoGC() { if (cx_) { MOZ_ASSERT(cx_->inUnsafeRegion > 0); cx_->inUnsafeRegion--; } } #endif // MOZ_DIAGNOSTIC_ASSERT_ENABLED #ifdef DEBUG AutoAssertNoNurseryAlloc::AutoAssertNoNurseryAlloc() { TlsContext.get()->disallowNurseryAlloc(); } AutoAssertNoNurseryAlloc::~AutoAssertNoNurseryAlloc() { TlsContext.get()->allowNurseryAlloc(); } JS::AutoEnterCycleCollection::AutoEnterCycleCollection(JSRuntime* rt) : runtime_(rt) { MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt)); MOZ_ASSERT(!JS::RuntimeHeapIsBusy()); runtime_->gc.heapState_ = HeapState::CycleCollecting; } JS::AutoEnterCycleCollection::~AutoEnterCycleCollection() { MOZ_ASSERT(JS::RuntimeHeapIsCycleCollecting()); runtime_->gc.heapState_ = HeapState::Idle; } JS::AutoAssertGCCallback::AutoAssertGCCallback() : AutoSuppressGCAnalysis() { MOZ_ASSERT(JS::RuntimeHeapIsCollecting()); } #endif // DEBUG JS_FRIEND_API const char* JS::GCTraceKindToAscii(JS::TraceKind kind) { switch (kind) { #define MAP_NAME(name, _0, _1, _2) \ case JS::TraceKind::name: \ return "JS " #name; JS_FOR_EACH_TRACEKIND(MAP_NAME); #undef MAP_NAME default: return "Invalid"; } } JS_FRIEND_API size_t JS::GCTraceKindSize(JS::TraceKind kind) { switch (kind) { #define MAP_SIZE(name, type, _0, _1) \ case JS::TraceKind::name: \ return sizeof(type); JS_FOR_EACH_TRACEKIND(MAP_SIZE); #undef MAP_SIZE default: return 0; } } JS::GCCellPtr::GCCellPtr(const Value& v) : ptr(0) { switch (v.type()) { case ValueType::String: ptr = checkedCast(v.toString(), JS::TraceKind::String); return; case ValueType::Object: ptr = checkedCast(&v.toObject(), JS::TraceKind::Object); return; case ValueType::Symbol: ptr = checkedCast(v.toSymbol(), JS::TraceKind::Symbol); return; case ValueType::BigInt: ptr = checkedCast(v.toBigInt(), JS::TraceKind::BigInt); return; case ValueType::PrivateGCThing: ptr = checkedCast(v.toGCThing(), v.toGCThing()->getTraceKind()); return; case ValueType::Double: case ValueType::Int32: case ValueType::Boolean: case ValueType::Undefined: case ValueType::Null: case ValueType::Magic: { MOZ_ASSERT(!v.isGCThing()); ptr = checkedCast(nullptr, JS::TraceKind::Null); return; } } ReportBadValueTypeAndCrash(v); } JS::TraceKind JS::GCCellPtr::outOfLineKind() const { MOZ_ASSERT((ptr & OutOfLineTraceKindMask) == OutOfLineTraceKindMask); MOZ_ASSERT(asCell()->isTenured()); return MapAllocToTraceKind(asCell()->asTenured().getAllocKind()); } #ifdef JSGC_HASH_TABLE_CHECKS void GCRuntime::checkHashTablesAfterMovingGC() { /* * Check that internal hash tables no longer have any pointers to things * that have been moved. */ rt->geckoProfiler().checkStringsMapAfterMovingGC(); for (ZonesIter zone(this, SkipAtoms); !zone.done(); zone.next()) { zone->checkUniqueIdTableAfterMovingGC(); zone->checkInitialShapesTableAfterMovingGC(); zone->checkBaseShapeTableAfterMovingGC(); zone->checkAllCrossCompartmentWrappersAfterMovingGC(); zone->checkScriptMapsAfterMovingGC(); JS::AutoCheckCannotGC nogc; for (auto baseShape = zone->cellIterUnsafe(); !baseShape.done(); baseShape.next()) { ShapeCachePtr p = baseShape->getCache(nogc); p.checkAfterMovingGC(); } } for (CompartmentsIter c(this); !c.done(); c.next()) { for (RealmsInCompartmentIter r(c); !r.done(); r.next()) { r->checkObjectGroupTablesAfterMovingGC(); r->dtoaCache.checkCacheAfterMovingGC(); if (r->debugEnvs()) { r->debugEnvs()->checkHashTablesAfterMovingGC(); } } } } #endif #ifdef DEBUG bool GCRuntime::hasZone(Zone* target) { for (AllZonesIter zone(this); !zone.done(); zone.next()) { if (zone == target) { return true; } } return false; } #endif JS_PUBLIC_API void JS::PrepareZoneForGC(JSContext* cx, Zone* zone) { AssertHeapIsIdle(); CHECK_THREAD(cx); MOZ_ASSERT(cx->runtime()->gc.hasZone(zone)); zone->scheduleGC(); } JS_PUBLIC_API void JS::PrepareForFullGC(JSContext* cx) { AssertHeapIsIdle(); CHECK_THREAD(cx); for (ZonesIter zone(cx->runtime(), WithAtoms); !zone.done(); zone.next()) { zone->scheduleGC(); } } JS_PUBLIC_API void JS::PrepareForIncrementalGC(JSContext* cx) { AssertHeapIsIdle(); CHECK_THREAD(cx); if (!JS::IsIncrementalGCInProgress(cx)) { return; } for (ZonesIter zone(cx->runtime(), WithAtoms); !zone.done(); zone.next()) { if (zone->wasGCStarted()) { zone->scheduleGC(); } } } JS_PUBLIC_API bool JS::IsGCScheduled(JSContext* cx) { AssertHeapIsIdle(); CHECK_THREAD(cx); for (ZonesIter zone(cx->runtime(), WithAtoms); !zone.done(); zone.next()) { if (zone->isGCScheduled()) { return true; } } return false; } JS_PUBLIC_API void JS::SkipZoneForGC(JSContext* cx, Zone* zone) { AssertHeapIsIdle(); CHECK_THREAD(cx); MOZ_ASSERT(cx->runtime()->gc.hasZone(zone)); zone->unscheduleGC(); } JS_PUBLIC_API void JS::NonIncrementalGC(JSContext* cx, JSGCInvocationKind gckind, GCReason reason) { AssertHeapIsIdle(); CHECK_THREAD(cx); MOZ_ASSERT(gckind == GC_NORMAL || gckind == GC_SHRINK); cx->runtime()->gc.gc(gckind, reason); MOZ_ASSERT(!IsIncrementalGCInProgress(cx)); } JS_PUBLIC_API void JS::StartIncrementalGC(JSContext* cx, JSGCInvocationKind gckind, GCReason reason, int64_t millis) { AssertHeapIsIdle(); CHECK_THREAD(cx); MOZ_ASSERT(gckind == GC_NORMAL || gckind == GC_SHRINK); cx->runtime()->gc.startGC(gckind, reason, millis); } JS_PUBLIC_API void JS::IncrementalGCSlice(JSContext* cx, GCReason reason, int64_t millis) { AssertHeapIsIdle(); CHECK_THREAD(cx); cx->runtime()->gc.gcSlice(reason, millis); } JS_PUBLIC_API bool JS::IncrementalGCHasForegroundWork(JSContext* cx) { AssertHeapIsIdle(); CHECK_THREAD(cx); return cx->runtime()->gc.hasForegroundWork(); } JS_PUBLIC_API void JS::FinishIncrementalGC(JSContext* cx, GCReason reason) { AssertHeapIsIdle(); CHECK_THREAD(cx); cx->runtime()->gc.finishGC(reason); } JS_PUBLIC_API void JS::AbortIncrementalGC(JSContext* cx) { AssertHeapIsIdle(); CHECK_THREAD(cx); if (IsIncrementalGCInProgress(cx)) { cx->runtime()->gc.abortGC(); } } char16_t* JS::GCDescription::formatSliceMessage(JSContext* cx) const { UniqueChars cstr = cx->runtime()->gc.stats().formatCompactSliceMessage(); size_t nchars = strlen(cstr.get()); UniqueTwoByteChars out(js_pod_malloc(nchars + 1)); if (!out) { return nullptr; } out.get()[nchars] = 0; CopyAndInflateChars(out.get(), cstr.get(), nchars); return out.release(); } char16_t* JS::GCDescription::formatSummaryMessage(JSContext* cx) const { UniqueChars cstr = cx->runtime()->gc.stats().formatCompactSummaryMessage(); size_t nchars = strlen(cstr.get()); UniqueTwoByteChars out(js_pod_malloc(nchars + 1)); if (!out) { return nullptr; } out.get()[nchars] = 0; CopyAndInflateChars(out.get(), cstr.get(), nchars); return out.release(); } JS::dbg::GarbageCollectionEvent::Ptr JS::GCDescription::toGCEvent( JSContext* cx) const { return JS::dbg::GarbageCollectionEvent::Create( cx->runtime(), cx->runtime()->gc.stats(), cx->runtime()->gc.majorGCCount()); } TimeStamp JS::GCDescription::startTime(JSContext* cx) const { return cx->runtime()->gc.stats().start(); } TimeStamp JS::GCDescription::endTime(JSContext* cx) const { return cx->runtime()->gc.stats().end(); } TimeStamp JS::GCDescription::lastSliceStart(JSContext* cx) const { return cx->runtime()->gc.stats().slices().back().start; } TimeStamp JS::GCDescription::lastSliceEnd(JSContext* cx) const { return cx->runtime()->gc.stats().slices().back().end; } JS::UniqueChars JS::GCDescription::sliceToJSONProfiler(JSContext* cx) const { size_t slices = cx->runtime()->gc.stats().slices().length(); MOZ_ASSERT(slices > 0); return cx->runtime()->gc.stats().renderJsonSlice(slices - 1); } JS::UniqueChars JS::GCDescription::formatJSONProfiler(JSContext* cx) const { return cx->runtime()->gc.stats().renderJsonMessage(); } JS_PUBLIC_API JS::UniqueChars JS::MinorGcToJSON(JSContext* cx) { JSRuntime* rt = cx->runtime(); return rt->gc.stats().renderNurseryJson(); } JS_PUBLIC_API JS::GCSliceCallback JS::SetGCSliceCallback( JSContext* cx, GCSliceCallback callback) { return cx->runtime()->gc.setSliceCallback(callback); } JS_PUBLIC_API JS::DoCycleCollectionCallback JS::SetDoCycleCollectionCallback( JSContext* cx, JS::DoCycleCollectionCallback callback) { return cx->runtime()->gc.setDoCycleCollectionCallback(callback); } JS_PUBLIC_API JS::GCNurseryCollectionCallback JS::SetGCNurseryCollectionCallback(JSContext* cx, GCNurseryCollectionCallback callback) { return cx->runtime()->gc.setNurseryCollectionCallback(callback); } JS_PUBLIC_API void JS::SetLowMemoryState(JSContext* cx, bool newState) { return cx->runtime()->gc.setLowMemoryState(newState); } JS_PUBLIC_API void JS::DisableIncrementalGC(JSContext* cx) { cx->runtime()->gc.disallowIncrementalGC(); } JS_PUBLIC_API bool JS::IsIncrementalGCEnabled(JSContext* cx) { GCRuntime& gc = cx->runtime()->gc; return gc.isIncrementalGCEnabled() && gc.isIncrementalGCAllowed(); } JS_PUBLIC_API bool JS::IsIncrementalGCInProgress(JSContext* cx) { return cx->runtime()->gc.isIncrementalGCInProgress(); } JS_PUBLIC_API bool JS::IsIncrementalGCInProgress(JSRuntime* rt) { return rt->gc.isIncrementalGCInProgress() && !rt->gc.isVerifyPreBarriersEnabled(); } JS_PUBLIC_API bool JS::IsIncrementalBarrierNeeded(JSContext* cx) { if (JS::RuntimeHeapIsBusy()) { return false; } auto state = cx->runtime()->gc.state(); return state != gc::State::NotActive && state <= gc::State::Sweep; } JS_PUBLIC_API void JS::IncrementalPreWriteBarrier(JSObject* obj) { if (!obj) { return; } AutoGeckoProfilerEntry profilingStackFrame( TlsContext.get(), "IncrementalPreWriteBarrier(JSObject*)", JS::ProfilingCategoryPair::GCCC_Barrier); PreWriteBarrier(obj); } JS_PUBLIC_API void JS::IncrementalPreWriteBarrier(GCCellPtr thing) { if (!thing) { return; } AutoGeckoProfilerEntry profilingStackFrame( TlsContext.get(), "IncrementalPreWriteBarrier(GCCellPtr)", JS::ProfilingCategoryPair::GCCC_Barrier); CellPtrPreWriteBarrier(thing); } JS_PUBLIC_API bool JS::WasIncrementalGC(JSRuntime* rt) { return rt->gc.isIncrementalGc(); } uint64_t js::gc::NextCellUniqueId(JSRuntime* rt) { return rt->gc.nextCellUniqueId(); } namespace js { namespace gc { namespace MemInfo { static bool GCBytesGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->runtime()->gc.heapSize.bytes())); return true; } static bool MallocBytesGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); double bytes = 0; for (ZonesIter zone(cx->runtime(), WithAtoms); !zone.done(); zone.next()) { bytes += zone->mallocHeapSize.bytes(); } args.rval().setNumber(bytes); return true; } static bool GCMaxBytesGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->runtime()->gc.tunables.gcMaxBytes())); return true; } static bool GCHighFreqGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setBoolean( cx->runtime()->gc.schedulingState.inHighFrequencyGCMode()); return true; } static bool GCNumberGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->runtime()->gc.gcNumber())); return true; } static bool MajorGCCountGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->runtime()->gc.majorGCCount())); return true; } static bool MinorGCCountGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->runtime()->gc.minorGCCount())); return true; } static bool GCSliceCountGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->runtime()->gc.gcSliceCount())); return true; } static bool ZoneGCBytesGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->zone()->gcHeapSize.bytes())); return true; } static bool ZoneGCTriggerBytesGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->zone()->gcHeapThreshold.startBytes())); return true; } static bool ZoneGCAllocTriggerGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); bool highFrequency = cx->runtime()->gc.schedulingState.inHighFrequencyGCMode(); args.rval().setNumber( double(cx->zone()->gcHeapThreshold.eagerAllocTrigger(highFrequency))); return true; } static bool ZoneMallocBytesGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->zone()->mallocHeapSize.bytes())); return true; } static bool ZoneMallocTriggerBytesGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->zone()->mallocHeapThreshold.startBytes())); return true; } static bool ZoneGCNumberGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setNumber(double(cx->zone()->gcNumber())); return true; } #ifdef DEBUG static bool DummyGetter(JSContext* cx, unsigned argc, Value* vp) { CallArgs args = CallArgsFromVp(argc, vp); args.rval().setUndefined(); return true; } #endif } /* namespace MemInfo */ JSObject* NewMemoryInfoObject(JSContext* cx) { RootedObject obj(cx, JS_NewObject(cx, nullptr)); if (!obj) { return nullptr; } using namespace MemInfo; struct NamedGetter { const char* name; JSNative getter; } getters[] = {{"gcBytes", GCBytesGetter}, {"gcMaxBytes", GCMaxBytesGetter}, {"mallocBytes", MallocBytesGetter}, {"gcIsHighFrequencyMode", GCHighFreqGetter}, {"gcNumber", GCNumberGetter}, {"majorGCCount", MajorGCCountGetter}, {"minorGCCount", MinorGCCountGetter}, {"sliceCount", GCSliceCountGetter}}; for (auto pair : getters) { JSNative getter = pair.getter; #ifdef DEBUG if (js::SupportDifferentialTesting()) { getter = DummyGetter; } #endif if (!JS_DefineProperty(cx, obj, pair.name, getter, nullptr, JSPROP_ENUMERATE)) { return nullptr; } } RootedObject zoneObj(cx, JS_NewObject(cx, nullptr)); if (!zoneObj) { return nullptr; } if (!JS_DefineProperty(cx, obj, "zone", zoneObj, JSPROP_ENUMERATE)) { return nullptr; } struct NamedZoneGetter { const char* name; JSNative getter; } zoneGetters[] = {{"gcBytes", ZoneGCBytesGetter}, {"gcTriggerBytes", ZoneGCTriggerBytesGetter}, {"gcAllocTrigger", ZoneGCAllocTriggerGetter}, {"mallocBytes", ZoneMallocBytesGetter}, {"mallocTriggerBytes", ZoneMallocTriggerBytesGetter}, {"gcNumber", ZoneGCNumberGetter}}; for (auto pair : zoneGetters) { JSNative getter = pair.getter; #ifdef DEBUG if (js::SupportDifferentialTesting()) { getter = DummyGetter; } #endif if (!JS_DefineProperty(cx, zoneObj, pair.name, getter, nullptr, JSPROP_ENUMERATE)) { return nullptr; } } return obj; } const char* StateName(State state) { switch (state) { #define MAKE_CASE(name) \ case State::name: \ return #name; GCSTATES(MAKE_CASE) #undef MAKE_CASE } MOZ_CRASH("Invalid gc::State enum value"); } const char* StateName(JS::Zone::GCState state) { switch (state) { case JS::Zone::NoGC: return "NoGC"; case JS::Zone::Prepare: return "Prepare"; case JS::Zone::MarkBlackOnly: return "MarkBlackOnly"; case JS::Zone::MarkBlackAndGray: return "MarkBlackAndGray"; case JS::Zone::Sweep: return "Sweep"; case JS::Zone::Finished: return "Finished"; case JS::Zone::Compact: return "Compact"; } MOZ_CRASH("Invalid Zone::GCState enum value"); } void AutoAssertEmptyNursery::checkCondition(JSContext* cx) { if (!noAlloc) { noAlloc.emplace(); } this->cx = cx; MOZ_ASSERT(cx->nursery().isEmpty()); } AutoEmptyNursery::AutoEmptyNursery(JSContext* cx) : AutoAssertEmptyNursery() { MOZ_ASSERT(!cx->suppressGC); cx->runtime()->gc.stats().suspendPhases(); cx->runtime()->gc.evictNursery(JS::GCReason::EVICT_NURSERY); cx->runtime()->gc.stats().resumePhases(); checkCondition(cx); } } /* namespace gc */ } /* namespace js */ #ifdef DEBUG namespace js { // We don't want jsfriendapi.h to depend on GenericPrinter, // so these functions are declared directly in the cpp. extern JS_FRIEND_API void DumpString(JSString* str, js::GenericPrinter& out); } // namespace js void js::gc::Cell::dump(js::GenericPrinter& out) const { switch (getTraceKind()) { case JS::TraceKind::Object: reinterpret_cast(this)->dump(out); break; case JS::TraceKind::String: js::DumpString(reinterpret_cast(const_cast(this)), out); break; case JS::TraceKind::Shape: reinterpret_cast(this)->dump(out); break; default: out.printf("%s(%p)\n", JS::GCTraceKindToAscii(getTraceKind()), (void*)this); } } // For use in a debugger. void js::gc::Cell::dump() const { js::Fprinter out(stderr); dump(out); } #endif static inline bool CanCheckGrayBits(const Cell* cell) { MOZ_ASSERT(cell); if (!cell->isTenured()) { return false; } auto tc = &cell->asTenured(); auto rt = tc->runtimeFromAnyThread(); if (!CurrentThreadCanAccessRuntime(rt) || !rt->gc.areGrayBitsValid()) { return false; } // If the zone's mark bits are being cleared concurrently we can't depend on // the contents. return !tc->zone()->isGCPreparing(); } JS_PUBLIC_API bool js::gc::detail::CellIsMarkedGrayIfKnown(const Cell* cell) { // We ignore the gray marking state of cells and return false in the // following cases: // // 1) When OOM has caused us to clear the gcGrayBitsValid_ flag. // // 2) When we are in an incremental GC and examine a cell that is in a zone // that is not being collected. Gray targets of CCWs that are marked black // by a barrier will eventually be marked black in the next GC slice. // // 3) When we are not on the runtime's main thread. Helper threads might // call this while parsing, and they are not allowed to inspect the // runtime's incremental state. The objects being operated on are not able // to be collected and will not be marked any color. if (!CanCheckGrayBits(cell)) { return false; } auto tc = &cell->asTenured(); MOZ_ASSERT(!tc->zoneFromAnyThread()->usedByHelperThread()); auto rt = tc->runtimeFromMainThread(); if (rt->gc.isIncrementalGCInProgress() && !tc->zone()->wasGCStarted()) { return false; } return detail::CellIsMarkedGray(tc); } #ifdef DEBUG JS_PUBLIC_API void js::gc::detail::AssertCellIsNotGray(const Cell* cell) { // Check that a cell is not marked gray. // // Since this is a debug-only check, take account of the eventual mark state // of cells that will be marked black by the next GC slice in an incremental // GC. For performance reasons we don't do this in CellIsMarkedGrayIfKnown. if (!CanCheckGrayBits(cell)) { return; } // TODO: I'd like to AssertHeapIsIdle() here, but this ends up getting // called during GC and while iterating the heap for memory reporting. MOZ_ASSERT(!JS::RuntimeHeapIsCycleCollecting()); auto tc = &cell->asTenured(); if (tc->zone()->isGCMarkingBlackAndGray()) { // We are doing gray marking in the cell's zone. Even if the cell is // currently marked gray it may eventually be marked black. Delay checking // non-black cells until we finish gray marking. if (!tc->isMarkedBlack()) { JSRuntime* rt = tc->zone()->runtimeFromMainThread(); AutoEnterOOMUnsafeRegion oomUnsafe; if (!rt->gc.cellsToAssertNotGray.ref().append(cell)) { oomUnsafe.crash("Can't append to delayed gray checks list"); } } return; } MOZ_ASSERT(!tc->isMarkedGray()); } extern JS_PUBLIC_API bool js::gc::detail::ObjectIsMarkedBlack( const JSObject* obj) { return obj->isMarkedBlack(); } #endif js::gc::ClearEdgesTracer::ClearEdgesTracer(JSRuntime* rt) : GenericTracer(rt, JS::TracerKind::ClearEdges, JS::WeakMapTraceAction::TraceKeysAndValues) {} js::gc::ClearEdgesTracer::ClearEdgesTracer() : ClearEdgesTracer(TlsContext.get()->runtime()) {} template inline S* js::gc::ClearEdgesTracer::onEdge(S* thing) { // We don't handle removing pointers to nursery edges from the store buffer // with this tracer. Check that this doesn't happen. MOZ_ASSERT(!IsInsideNursery(thing)); // Fire the pre-barrier since we're removing an edge from the graph. InternalBarrierMethods::preBarrier(thing); // Return nullptr to clear the edge. return nullptr; } JSObject* js::gc::ClearEdgesTracer::onObjectEdge(JSObject* obj) { return onEdge(obj); } JSString* js::gc::ClearEdgesTracer::onStringEdge(JSString* str) { return onEdge(str); } JS::Symbol* js::gc::ClearEdgesTracer::onSymbolEdge(JS::Symbol* sym) { return onEdge(sym); } JS::BigInt* js::gc::ClearEdgesTracer::onBigIntEdge(JS::BigInt* bi) { return onEdge(bi); } js::BaseScript* js::gc::ClearEdgesTracer::onScriptEdge(js::BaseScript* script) { return onEdge(script); } js::Shape* js::gc::ClearEdgesTracer::onShapeEdge(js::Shape* shape) { return onEdge(shape); } js::ObjectGroup* js::gc::ClearEdgesTracer::onObjectGroupEdge( js::ObjectGroup* group) { return onEdge(group); } js::BaseShape* js::gc::ClearEdgesTracer::onBaseShapeEdge(js::BaseShape* base) { return onEdge(base); } js::jit::JitCode* js::gc::ClearEdgesTracer::onJitCodeEdge( js::jit::JitCode* code) { return onEdge(code); } js::Scope* js::gc::ClearEdgesTracer::onScopeEdge(js::Scope* scope) { return onEdge(scope); } js::RegExpShared* js::gc::ClearEdgesTracer::onRegExpSharedEdge( js::RegExpShared* shared) { return onEdge(shared); } JS_PUBLIC_API void js::gc::FinalizeDeadNurseryObject(JSContext* cx, JSObject* obj) { CHECK_THREAD(cx); MOZ_ASSERT(JS::RuntimeHeapIsMinorCollecting()); MOZ_ASSERT(obj); MOZ_ASSERT(IsInsideNursery(obj)); mozilla::DebugOnly prior(obj); MOZ_ASSERT(IsAboutToBeFinalizedUnbarriered(&prior)); MOZ_ASSERT(obj == prior); const JSClass* jsClass = JS::GetClass(obj); jsClass->doFinalize(cx->defaultFreeOp(), obj); } JS_FRIEND_API void js::gc::SetPerformanceHint(JSContext* cx, PerformanceHint hint) { CHECK_THREAD(cx); MOZ_ASSERT(!JS::RuntimeHeapIsCollecting()); cx->runtime()->gc.setPerformanceHint(hint); } void GCRuntime::setPerformanceHint(PerformanceHint hint) { bool wasInPageLoad = inPageLoadCount != 0; if (hint == PerformanceHint::InPageLoad) { inPageLoadCount++; } else { MOZ_ASSERT(inPageLoadCount); inPageLoadCount--; } bool inPageLoad = inPageLoadCount != 0; if (inPageLoad == wasInPageLoad) { return; } AutoLockGC lock(this); schedulingState.inPageLoad = inPageLoad; atomsZone->updateGCStartThresholds(*this, invocationKind, lock); maybeTriggerGCAfterAlloc(atomsZone); }