Source/JavaScriptCore/ChangeLog

 12012-10-09 Mark Hahnenberg <mhahnenberg@apple.com>
 2
 3 Copying collection shouldn't require O(live bytes) memory overhead
 4 https://bugs.webkit.org/show_bug.cgi?id=98792
 5
 6 Reviewed by NOBODY (OOPS!).
 7
 8 Currently our copying collection occurs simultaneously with the marking phase. We'd like
 9 to be able to reuse CopiedBlocks as soon as they become fully evacuated, but this is not
 10 currently possible because we don't know the liveness statistics of each old CopiedBlock
 11 until marking/copying has already finished. Instead, we have to allocate additional memory
 12 from the OS to use as our working set of CopiedBlocks while copying. We then return the
 13 fully evacuated old CopiedBlocks back to the block allocator, thus giving our copying phase
 14 an O(live bytes) overhead.
 15
 16 To fix this, we should instead split the copying phase apart from the marking phase. This
 17 way we have full liveness data for each CopiedBlock during the copying phase so that we
 18 can reuse them the instant they become fully evacuated. With the additional liveness data
 19 that each CopiedBlock accumulates, we can add some additional heuristics to the collector.
 20 For example, we can calculate our global Heap fragmentation and only choose to do a copying
 21 phase if that fragmentation exceeds some limit. As another example, we can skip copying
 22 blocks that are already above a particular fragmentation limit, which allows older objects
 23 to coalesce into blocks that are rarely copied.
 24
 25 * JavaScriptCore.xcodeproj/project.pbxproj:
 26 * heap/CopiedBlock.h:
 27 (CopiedBlock):
 28 (JSC::CopiedBlock::CopiedBlock): Added support for tracking live bytes in a CopiedBlock in a
 29 thread-safe fashion.
 30 (JSC::CopiedBlock::reportLiveBytes): Adds a number of live bytes to the block in a thread-safe
 31 fashion using compare and swap.
 32 (JSC):
 33 (JSC::CopiedBlock::didSurviveGC): Called when a block survives a single GC without being
 34 evacuated. This could be called for a couple reasons: (a) the block was pinned or (b) we
 35 decided not to do any copying. A block can become pinned for a few reasons: (1) a pointer into
 36 the block was found during the conservative scan. (2) the block was deemed full enough to
 37 not warrant any copying. (3) The block is oversize and was found to be live.
 38 (JSC::CopiedBlock::didEvacuateBytes): Called when some number of bytes are copied from this
 39 block. If the number of live bytes ever hits zero, the block will return itself to the
 40 BlockAllocator to be recycled.
 41 (JSC::CopiedBlock::canBeRecycled): Indicates that a block has no live bytes and can be
 42 immediately recycled. This is used for blocks that are found to have zero live bytes at the
 43 beginning of the copying phase.
 44 (JSC::CopiedBlock::shouldEvacuate): This function returns true if the current fragmentation
 45 of the block is above our fragmentation threshold, and false otherwise.
 46 (JSC::CopiedBlock::isPinned): Added an accessor for the pinned flag
 47 (JSC::CopiedBlock::liveBytes):
 48 * heap/CopiedSpace.cpp:
 49 (JSC::CopiedSpace::CopiedSpace):
 50 (JSC::CopiedSpace::doneFillingBlock): Changed so that we can exchange our filled block for a
 51 fresh block. This avoids the situation where a thread returns its borrowed block, it's the last
 52 borrowed block, so CopiedSpace thinks that copying has completed, and it starts doing all of the
 53 copying phase cleanup. In actuality, the thread wanted another block after returning the current
 54 block. So we allow the thread to atomically exchange its block for another block.
 55 (JSC::CopiedSpace::startedCopying): Added the calculation of global Heap fragmentation to
 56 determine if the copying phase should commence. We include the MarkedSpace in our fragmentation
 57 calculation by assuming that the MarkedSpace is 0% fragmented since we can reuse any currently
 58 free memory in it (i.e. we ignore any internal fragmentation in the MarkedSpace). While we're
 59 calculating the fragmentation of CopiedSpace, we also return any free blocks we find along the
 60 way (meaning liveBytes() == 0).
 61 (JSC):
 62 (JSC::CopiedSpace::doneCopying): We still have to iterate over all the blocks, regardless of
 63 whether the copying phase took place or not so that we can reset all of the live bytes counters
 64 and un-pin any pinned blocks.
 65 * heap/CopiedSpace.h:
 66 (CopiedSpace):
 67 (JSC::CopiedSpace::shouldDoCopyPhase):
 68 * heap/CopiedSpaceInlineMethods.h:
 69 (JSC::CopiedSpace::recycleEvacuatedBlock): This function is distinct from recycling a borrowed block
 70 because a borrowed block hasn't been added to the CopiedSpace yet, but an evacuated block is still
 71 currently in CopiedSpace, so we have to make sure we properly remove all traces of the block from
 72 CopiedSpace before returning it to BlockAllocator.
 73 (JSC::CopiedSpace::recycleBorrowedBlock): Renamed to indicate the distinction mentioned above.
 74 * heap/CopyVisitor.cpp: Added.
 75 (JSC):
 76 (JSC::CopyVisitor::CopyVisitor):
 77 (JSC::CopyVisitor::copyFromShared): Main function for any thread participating in the copying phase.
 78 Grabs chunks of MarkedBlocks from the shared list and copies the backing store of anybody who needs
 79 it until there are no more chunks to copy.
 80 * heap/CopyVisitor.h: Added.
 81 (JSC):
 82 (CopyVisitor):
 83 * heap/CopyVisitorInlineMethods.h: Added.
 84 (JSC):
 85 (GCCopyPhaseFunctor):
 86 (JSC::GCCopyPhaseFunctor::GCCopyPhaseFunctor):
 87 (JSC::GCCopyPhaseFunctor::operator()):
 88 (JSC::CopyVisitor::checkIfShouldCopy): We don't have to check shouldEvacuate() because all of those
 89 checks are done during the marking phase.
 90 (JSC::CopyVisitor::allocateNewSpace):
 91 (JSC::CopyVisitor::allocateNewSpaceSlow):
 92 (JSC::CopyVisitor::startCopying): Initialization function for a thread that is about to start copying.
 93 (JSC::CopyVisitor::doneCopying):
 94 (JSC::CopyVisitor::didCopy): This callback is called by an object that has just successfully copied its
 95 backing store. It indicates to the CopiedBlock that somebody has just finished evacuating some number of
 96 bytes from it, and, if the CopiedBlock now has no more live bytes, can be recycled immediately.
 97 * heap/GCThread.cpp: Added.
 98 (JSC):
 99 (JSC::GCThread::GCThread): This is a new class that encapsulates a single thread responsible for participating
 100 in a specific set of GC phases. Currently, that set of phases includes Mark, Copy, and Exit. Each thread
 101 monitors a shared variable in its associated GCThreadSharedData. The main thread updates this m_currentPhase
 102 variable as collection progresses through the various phases. Parallel marking still works exactly like it
 103 has. In other words, the "run loop" for each of the GC threads sits above any individual phase, thus keeping
 104 the separate phases of the collector orthogonal.
 105 (JSC::GCThread::threadID):
 106 (JSC::GCThread::initializeThreadID):
 107 (JSC::GCThread::slotVisitor):
 108 (JSC::GCThread::copyVisitor):
 109 (JSC::GCThread::waitForNextPhase):
 110 (JSC::GCThread::gcThreadMain):
 111 (JSC::GCThread::gcThreadStartFunc):
 112 * heap/GCThread.h: Added.
 113 (JSC):
 114 (GCThread):
 115 * heap/GCThreadSharedData.cpp: The GCThreadSharedData now has a list of GCThread objects rather than raw
 116 ThreadIdentifiers.
 117 (JSC::GCThreadSharedData::resetChildren):
 118 (JSC::GCThreadSharedData::childVisitCount):
 119 (JSC::GCThreadSharedData::GCThreadSharedData):
 120 (JSC::GCThreadSharedData::~GCThreadSharedData):
 121 (JSC::GCThreadSharedData::reset):
 122 (JSC::GCThreadSharedData::didStartMarking): Callback to let the GCThreadSharedData know that marking has
 123 started and updates the m_currentPhase variable and notifies the GCThreads accordingly.
 124 (JSC::GCThreadSharedData::didFinishMarking): Ditto for finishing marking.
 125 (JSC::GCThreadSharedData::didStartCopying): Ditto for starting the copying phase.
 126 (JSC::GCThreadSharedData::didFinishCopying): Ditto for finishing copying.
 127 * heap/GCThreadSharedData.h:
 128 (JSC):
 129 (GCThreadSharedData):
 130 (JSC::GCThreadSharedData::getNextBlocksToCopy): Atomically gets the next chunk of work for a copying thread.
 131 * heap/Heap.cpp:
 132 (JSC::Heap::Heap):
 133 (JSC::Heap::markRoots):
 134 (JSC):
 135 (JSC::Heap::copyBackingStores): Responsible for setting up the copying phase, notifying the copying threads,
 136 and doing any copying work if necessary.
 137 (JSC::Heap::collect):
 138 * heap/Heap.h:
 139 (Heap):
 140 (JSC):
 141 (JSC::CopyFunctor::CopyFunctor):
 142 (CopyFunctor):
 143 (JSC::CopyFunctor::operator()):
 144 * heap/IncrementalSweeper.cpp: Changed the incremental sweeper to have a reference to the list of MarkedBlocks
 145 that need sweeping, since this now resides in the Heap so that it can be easily shared by the GCThreads.
 146 (JSC::IncrementalSweeper::IncrementalSweeper):
 147 (JSC::IncrementalSweeper::startSweeping):
 148 * heap/IncrementalSweeper.h:
 149 (JSC):
 150 (IncrementalSweeper):
 151 * heap/SlotVisitor.cpp:
 152 (JSC::SlotVisitor::setup):
 153 (JSC::SlotVisitor::drainFromShared): We no longer do any copying-related work here.
 154 (JSC):
 155 * heap/SlotVisitor.h:
 156 (SlotVisitor):
 157 * heap/SlotVisitorInlineMethods.h:
 158 (JSC):
 159 (JSC::SlotVisitor::copyLater): Notifies the CopiedBlock that there are some live bytes that may need
 160 to be copied.
 161 * runtime/Butterfly.h:
 162 (JSC):
 163 (Butterfly):
 164 * runtime/ButterflyInlineMethods.h:
 165 (JSC::Butterfly::createUninitializedDuringCollection): Uses the new CopyVisitor.
 166 * runtime/ClassInfo.h:
 167 (MethodTable): Added new "virtual" function copyBackingStore to method table.
 168 (JSC):
 169 * runtime/JSCell.cpp:
 170 (JSC::JSCell::copyBackingStore): Default implementation that does nothing.
 171 (JSC):
 172 * runtime/JSCell.h:
 173 (JSC):
 174 (JSCell):
 175 * runtime/JSObject.cpp:
 176 (JSC::JSObject::copyButterfly): Does the actual copying of the butterfly.
 177 (JSC):
 178 (JSC::JSObject::visitButterfly): Calls copyLater for the butterfly.
 179 (JSC::JSObject::copyBackingStore):
 180 * runtime/JSObject.h:
 181 (JSObject):
 182 (JSC::JSCell::methodTable):
 183 (JSC::JSCell::inherits):
 184 * runtime/Options.h: Added two new constants, maxHeapFragmentation and maxCopiedBlockFragmentation,
 185 to govern the amount of fragmentation we allow before doing copying.
 186 (JSC):
 187
11882012-10-10 Balazs Kilvady <kilvadyb@homejinni.com>
2189
3190 RegisterFile to JSStack rename fix for a struct member.
130954

Source/JavaScriptCore/CMakeLists.txt

@@SET(JavaScriptCore_SOURCES
107107
108108 heap/BlockAllocator.cpp
109109 heap/CopiedSpace.cpp
 110 heap/CopyVisitor.cpp
110111 heap/ConservativeRoots.cpp
111112 heap/DFGCodeBlocks.cpp
 113 heap/GCThread.cpp
112114 heap/GCThreadSharedData.cpp
113115 heap/HandleSet.cpp
114116 heap/HandleStack.cpp
130954

Source/JavaScriptCore/GNUmakefile.list.am

@@javascriptcore_sources += \
256256 Source/JavaScriptCore/heap/CopiedSpace.cpp \
257257 Source/JavaScriptCore/heap/CopiedSpace.h \
258258 Source/JavaScriptCore/heap/CopiedSpaceInlineMethods.h \
 259 Source/JavaScriptCore/heap/CopyVisitor.h \
 260 Source/JavaScriptCore/heap/CopyVisitorInlineMethods.h \
 261 Source/JavaScriptCore/heap/CopyVisitor.cpp \
259262 Source/JavaScriptCore/heap/CardSet.h \
260263 Source/JavaScriptCore/heap/ConservativeRoots.cpp \
261264 Source/JavaScriptCore/heap/ConservativeRoots.h \

@@javascriptcore_sources += \
280283 Source/JavaScriptCore/heap/BlockAllocator.h \
281284 Source/JavaScriptCore/heap/GCThreadSharedData.cpp \
282285 Source/JavaScriptCore/heap/GCThreadSharedData.h \
 286 Source/JavaScriptCore/heap/GCThread.cpp \
 287 Source/JavaScriptCore/heap/GCThread.h \
283288 Source/JavaScriptCore/heap/Heap.cpp \
284289 Source/JavaScriptCore/heap/Heap.h \
285290 Source/JavaScriptCore/heap/HeapStatistics.cpp \
130954

Source/JavaScriptCore/Target.pri

@@SOURCES += \
7373 bytecompiler/BytecodeGenerator.cpp \
7474 bytecompiler/NodesCodegen.cpp \
7575 heap/CopiedSpace.cpp \
 76 heap/CopyVisitor.cpp \
7677 heap/ConservativeRoots.cpp \
7778 heap/DFGCodeBlocks.cpp \
7879 heap/WeakSet.cpp \

@@SOURCES += \
8283 heap/HandleStack.cpp \
8384 heap/BlockAllocator.cpp \
8485 heap/GCThreadSharedData.cpp \
 86 heap/GCThread.cpp \
8587 heap/Heap.cpp \
8688 heap/HeapStatistics.cpp \
8789 heap/HeapTimer.cpp \
130954

Source/JavaScriptCore/JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.def

@@EXPORTS
116116 ?convertLatin1ToUTF8@Unicode@WTF@@YA?AW4ConversionResult@12@PAPBEPBEPAPADPAD@Z
117117 ?convertUTF16ToUTF8@Unicode@WTF@@YA?AW4ConversionResult@12@PAPB_WPB_WPAPADPAD_N@Z
118118 ?convertUTF8ToUTF16@Unicode@WTF@@YA?AW4ConversionResult@12@PAPBDPBDPAPA_WPA_W_N@Z
 119 ?copyBackingStore@JSObject@JSC@@SAXPAVJSCell@2@AAVCopyVisitor@2@@Z
119120 ?create@JSFunction@JSC@@SAPAV12@PAVExecState@2@PAVJSGlobalObject@2@HABVString@WTF@@P6I_J0@ZW4Intrinsic@2@3@Z
120121 ?create@JSGlobalData@JSC@@SA?AV?$PassRefPtr@VJSGlobalData@JSC@@@WTF@@W4ThreadStackType@2@W4HeapType@2@@Z
121122 ?create@RegExp@JSC@@SAPAV12@AAVJSGlobalData@2@ABVString@WTF@@W4RegExpFlags@2@@Z
130954

Source/JavaScriptCore/JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.vcproj

23542354 >
23552355 </File>
23562356 <File
 2357 RelativePath="..\..\heap\CopyVisitor.cpp"
 2358 >
 2359 </File>
 2360 <File
 2361 RelativePath="..\..\heap\CopyVisitor.h"
 2362 >
 2363 </File>
 2364 <File
 2365 RelativePath="..\..\heap\CopyVisitorInlineMethods.h"
 2366 >
 2367 </File>
 2368 <File
23572369 RelativePath="..\..\heap\DFGCodeBlocks.cpp"
23582370 >
23592371 </File>

24502462 >
24512463 </File>
24522464 <File
 2465 RelativePath="..\..\heap\GCThread.cpp"
 2466 >
 2467 </File>
 2468 <File
 2469 RelativePath="..\..\heap\GCThread.h"
 2470 >
 2471 </File>
 2472 <File
24532473 RelativePath="..\..\heap\GCThreadSharedData.cpp"
24542474 >
24552475 </File>
130954

Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

711711 C21122E215DD9AB300790E3A /* GCThreadSharedData.h in Headers */ = {isa = PBXBuildFile; fileRef = C21122DF15DD9AB300790E3A /* GCThreadSharedData.h */; settings = {ATTRIBUTES = (Private, ); }; };
712712 C21122E315DD9AB300790E3A /* MarkStackInlineMethods.h in Headers */ = {isa = PBXBuildFile; fileRef = C21122E015DD9AB300790E3A /* MarkStackInlineMethods.h */; settings = {ATTRIBUTES = (Private, ); }; };
713713 C2160FE715F7E95E00942DFC /* SlotVisitorInlineMethods.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FCB408515C0A3C30048932B /* SlotVisitorInlineMethods.h */; settings = {ATTRIBUTES = (Private, ); }; };
 714 C2239D1716262BDD005AC5FD /* CopyVisitor.cpp in Sources */ = {isa = PBXBuildFile; fileRef = C2239D1216262BDD005AC5FD /* CopyVisitor.cpp */; };
 715 C2239D1816262BDD005AC5FD /* CopyVisitor.h in Headers */ = {isa = PBXBuildFile; fileRef = C2239D1316262BDD005AC5FD /* CopyVisitor.h */; settings = {ATTRIBUTES = (Private, ); }; };
 716 C2239D1916262BDD005AC5FD /* CopyVisitorInlineMethods.h in Headers */ = {isa = PBXBuildFile; fileRef = C2239D1416262BDD005AC5FD /* CopyVisitorInlineMethods.h */; settings = {ATTRIBUTES = (Private, ); }; };
 717 C2239D1A16262BDD005AC5FD /* GCThread.cpp in Sources */ = {isa = PBXBuildFile; fileRef = C2239D1516262BDD005AC5FD /* GCThread.cpp */; };
 718 C2239D1B16262BDD005AC5FD /* GCThread.h in Headers */ = {isa = PBXBuildFile; fileRef = C2239D1616262BDD005AC5FD /* GCThread.h */; };
714719 C225494315F7DBAA0065E898 /* SlotVisitor.cpp in Sources */ = {isa = PBXBuildFile; fileRef = C225494215F7DBAA0065E898 /* SlotVisitor.cpp */; };
715720 C22B31B9140577D700DB475A /* SamplingCounter.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F77008E1402FDD60078EB39 /* SamplingCounter.h */; settings = {ATTRIBUTES = (Private, ); }; };
716721 C240305514B404E60079EB64 /* CopiedSpace.cpp in Sources */ = {isa = PBXBuildFile; fileRef = C240305314B404C90079EB64 /* CopiedSpace.cpp */; };

14941499 C21122DE15DD9AB300790E3A /* GCThreadSharedData.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = GCThreadSharedData.cpp; sourceTree = "<group>"; };
14951500 C21122DF15DD9AB300790E3A /* GCThreadSharedData.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = GCThreadSharedData.h; sourceTree = "<group>"; };
14961501 C21122E015DD9AB300790E3A /* MarkStackInlineMethods.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MarkStackInlineMethods.h; sourceTree = "<group>"; };
 1502 C2239D1216262BDD005AC5FD /* CopyVisitor.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CopyVisitor.cpp; sourceTree = "<group>"; };
 1503 C2239D1316262BDD005AC5FD /* CopyVisitor.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CopyVisitor.h; sourceTree = "<group>"; };
 1504 C2239D1416262BDD005AC5FD /* CopyVisitorInlineMethods.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CopyVisitorInlineMethods.h; sourceTree = "<group>"; };
 1505 C2239D1516262BDD005AC5FD /* GCThread.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = GCThread.cpp; sourceTree = "<group>"; };
 1506 C2239D1616262BDD005AC5FD /* GCThread.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = GCThread.h; sourceTree = "<group>"; };
14971507 C225494215F7DBAA0065E898 /* SlotVisitor.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SlotVisitor.cpp; sourceTree = "<group>"; };
14981508 C240305314B404C90079EB64 /* CopiedSpace.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CopiedSpace.cpp; sourceTree = "<group>"; };
14991509 C24D31E0161CD695002AA4DB /* HeapStatistics.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = HeapStatistics.cpp; sourceTree = "<group>"; };

18271837 142E312A134FF0A600AFADB5 /* heap */ = {
18281838 isa = PBXGroup;
18291839 children = (
 1840 C2239D1216262BDD005AC5FD /* CopyVisitor.cpp */,
 1841 C2239D1316262BDD005AC5FD /* CopyVisitor.h */,
 1842 C2239D1416262BDD005AC5FD /* CopyVisitorInlineMethods.h */,
 1843 C2239D1516262BDD005AC5FD /* GCThread.cpp */,
 1844 C2239D1616262BDD005AC5FD /* GCThread.h */,
18301845 C24D31E0161CD695002AA4DB /* HeapStatistics.cpp */,
18311846 C24D31E1161CD695002AA4DB /* HeapStatistics.h */,
18321847 C225494215F7DBAA0065E898 /* SlotVisitor.cpp */,

25902605 86ADD1450FDDEA980006EEC2 /* ARMv7Assembler.h in Headers */,
25912606 C2EAD2FC14F0249800A4B159 /* CopiedAllocator.h in Headers */,
25922607 C2B916C214DA014E00CBAC86 /* MarkedAllocator.h in Headers */,
 2608 C2239D1816262BDD005AC5FD /* CopyVisitor.h in Headers */,
 2609 C2239D1916262BDD005AC5FD /* CopyVisitorInlineMethods.h in Headers */,
25932610 C24D31E3161CD695002AA4DB /* HeapStatistics.h in Headers */,
25942611 C2A7F688160432D400F76B98 /* JSDestructibleObject.h in Headers */,
25952612 FE20CE9E15F04A9500DF3430 /* LLIntCLoop.h in Headers */,

29822999 862553D216136E1A009F17D0 /* JSProxy.h in Headers */,
29833000 0F5541B21613C1FB00CE3E25 /* SpecialPointer.h in Headers */,
29843001 0FEB3ECD16237F4D00AB67AD /* TypedArrayDescriptor.h in Headers */,
 3002 C2239D1B16262BDD005AC5FD /* GCThread.h in Headers */,
29853003 );
29863004 runOnlyForDeploymentPostprocessing = 0;
29873005 };

35903608 0F5541B11613C1FB00CE3E25 /* SpecialPointer.cpp in Sources */,
35913609 0FEB3ECF16237F6C00AB67AD /* MacroAssembler.cpp in Sources */,
35923610 C24D31E2161CD695002AA4DB /* HeapStatistics.cpp in Sources */,
 3611 C2239D1716262BDD005AC5FD /* CopyVisitor.cpp in Sources */,
 3612 C2239D1A16262BDD005AC5FD /* GCThread.cpp in Sources */,
35933613 );
35943614 runOnlyForDeploymentPostprocessing = 0;
35953615 };
130954

Source/JavaScriptCore/heap/CopiedBlock.h

2929#include "HeapBlock.h"
3030#include "JSValue.h"
3131#include "JSValueInlineMethods.h"
 32#include "Options.h"
 33#include <wtf/Atomics.h>
3234
3335namespace JSC {
3436

@@public:
4143 static CopiedBlock* create(const PageAllocationAligned&);
4244 static CopiedBlock* createNoZeroFill(const PageAllocationAligned&);
4345
 46 bool isPinned();
 47
 48 unsigned liveBytes();
 49 void reportLiveBytes(unsigned);
 50 void didSurviveGC();
 51 bool didEvacuateBytes(unsigned);
 52 bool shouldEvacuate();
 53 bool canBeRecycled();
 54
4455 // The payload is the region of the block that is usable for allocations.
4556 char* payload();
4657 char* payloadEnd();

@@private:
6677
6778 size_t m_remaining;
6879 uintptr_t m_isPinned;
 80 unsigned m_liveBytes;
6981};
7082
7183inline CopiedBlock* CopiedBlock::createNoZeroFill(const PageAllocationAligned& allocation)

@@inline CopiedBlock::CopiedBlock(const Pa
96108 : HeapBlock<CopiedBlock>(allocation)
97109 , m_remaining(payloadCapacity())
98110 , m_isPinned(false)
 111 , m_liveBytes(0)
99112{
100113 ASSERT(is8ByteAligned(reinterpret_cast<void*>(m_remaining)));
101114}
102115
 116inline void CopiedBlock::reportLiveBytes(unsigned bytes)
 117{
 118 unsigned oldValue = 0;
 119 unsigned newValue = 0;
 120 do {
 121 oldValue = m_liveBytes;
 122 newValue = oldValue + bytes;
 123 } while (!WTF::weakCompareAndSwap(&m_liveBytes, oldValue, newValue));
 124}
 125
 126inline void CopiedBlock::didSurviveGC()
 127{
 128 m_liveBytes = 0;
 129 m_isPinned = false;
 130}
 131
 132inline bool CopiedBlock::didEvacuateBytes(unsigned bytes)
 133{
 134 ASSERT(m_liveBytes >= bytes);
 135 unsigned oldValue = 0;
 136 unsigned newValue = 0;
 137 do {
 138 oldValue = m_liveBytes;
 139 newValue = oldValue - bytes;
 140 } while (!WTF::weakCompareAndSwap(&m_liveBytes, oldValue, newValue));
 141 ASSERT(m_liveBytes < oldValue);
 142 return !newValue;
 143}
 144
 145inline bool CopiedBlock::canBeRecycled()
 146{
 147 return !m_liveBytes;
 148}
 149
 150inline bool CopiedBlock::shouldEvacuate()
 151{
 152 return static_cast<double>(m_liveBytes) / static_cast<double>(payloadCapacity()) <= Options::maxCopiedBlockFragmentation();
 153}
 154
 155inline bool CopiedBlock::isPinned()
 156{
 157 return m_isPinned;
 158}
 159
 160inline unsigned CopiedBlock::liveBytes()
 161{
 162 return m_liveBytes;
 163}
 164
103165inline char* CopiedBlock::payload()
104166{
105167 return reinterpret_cast<char*>(this) + ((sizeof(CopiedBlock) + 7) & ~7);
130954

Source/JavaScriptCore/heap/CopiedSpace.cpp

2828
2929#include "CopiedSpaceInlineMethods.h"
3030#include "GCActivityCallback.h"
 31#include "Options.h"
3132
3233namespace JSC {
3334

@@CopiedSpace::CopiedSpace(Heap* heap)
3637 , m_toSpace(0)
3738 , m_fromSpace(0)
3839 , m_inCopyingPhase(false)
 40 , m_shouldDoCopyPhase(false)
3941 , m_numberOfLoanedBlocks(0)
4042{
4143 m_toSpaceLock.Init();

@@CheckedBoolean CopiedSpace::tryReallocat
152154 return true;
153155}
154156
155 void CopiedSpace::doneFillingBlock(CopiedBlock* block)
 157void CopiedSpace::doneFillingBlock(CopiedBlock* block, CopiedBlock** exchange)
156158{
157159 ASSERT(m_inCopyingPhase);
158160
 161 if (exchange)
 162 *exchange = allocateBlockForCopyingPhase();
 163
159164 if (!block)
160165 return;
161166
162167 if (!block->dataSize()) {
163  recycleBlock(block);
 168 recycleBorrowedBlock(block);
164169 return;
165170 }
166171

@@void CopiedSpace::doneFillingBlock(Copie
182187 }
183188}
184189
 190void CopiedSpace::startedCopying()
 191{
 192 std::swap(m_fromSpace, m_toSpace);
 193
 194 m_blockFilter.reset();
 195 m_allocator.resetCurrentBlock();
 196
 197 CopiedBlock* next = 0;
 198 size_t totalLiveBytes = 0;
 199 size_t totalUsableBytes = 0;
 200 for (CopiedBlock* block = m_fromSpace->head(); block; block = next) {
 201 next = block->next();
 202 if (!block->isPinned() && block->canBeRecycled()) {
 203 recycleEvacuatedBlock(block);
 204 continue;
 205 }
 206 totalLiveBytes += block->liveBytes();
 207 totalUsableBytes += block->payloadCapacity();
 208 }
 209
 210 double markedSpaceBytes = m_heap->objectSpace().capacity();
 211 double totalFragmentation = ((double)totalLiveBytes + markedSpaceBytes) / ((double)totalUsableBytes + markedSpaceBytes);
 212 m_shouldDoCopyPhase = totalFragmentation <= Options::maxCopiedSpaceFragmentation();
 213 if (!m_shouldDoCopyPhase)
 214 return;
 215
 216 ASSERT(m_shouldDoCopyPhase);
 217 ASSERT(!m_inCopyingPhase);
 218 ASSERT(!m_numberOfLoanedBlocks);
 219 m_inCopyingPhase = true;
 220}
 221
185222void CopiedSpace::doneCopying()
186223{
187224 {

@@void CopiedSpace::doneCopying()
190227 m_loanedBlocksCondition.wait(m_loanedBlocksLock);
191228 }
192229
193  ASSERT(m_inCopyingPhase);
 230 ASSERT(m_inCopyingPhase == m_shouldDoCopyPhase);
194231 m_inCopyingPhase = false;
 232
195233 while (!m_fromSpace->isEmpty()) {
196234 CopiedBlock* block = m_fromSpace->removeHead();
197  if (block->m_isPinned) {
198  block->m_isPinned = false;
 235 if (block->isPinned() || !m_shouldDoCopyPhase) {
 236 block->didSurviveGC();
199237 // We don't add the block to the blockSet because it was never removed.
200238 ASSERT(m_blockSet.contains(block));
201239 m_blockFilter.add(reinterpret_cast<Bits>(block));

@@void CopiedSpace::doneCopying()
210248 CopiedBlock* curr = m_oversizeBlocks.head();
211249 while (curr) {
212250 CopiedBlock* next = curr->next();
213  if (!curr->m_isPinned) {
 251 if (!curr->isPinned()) {
214252 m_oversizeBlocks.remove(curr);
215253 m_blockSet.remove(curr);
216254 CopiedBlock::destroy(curr).deallocate();
217255 } else {
218256 m_blockFilter.add(reinterpret_cast<Bits>(curr));
219  curr->m_isPinned = false;
 257 curr->didSurviveGC();
220258 }
221259 curr = next;
222260 }

@@void CopiedSpace::doneCopying()
225263 allocateBlock();
226264 else
227265 m_allocator.setCurrentBlock(m_toSpace->head());
 266
 267 m_shouldDoCopyPhase = false;
228268}
229269
230270size_t CopiedSpace::size()
130954

Source/JavaScriptCore/heap/CopiedSpace.h

@@class Heap;
4646class CopiedBlock;
4747
4848class CopiedSpace {
 49 friend class CopyVisitor;
4950 friend class SlotVisitor;
5051 friend class JIT;
5152public:

@@public:
7475 size_t capacity();
7576
7677 bool isPagedOut(double deadline);
 78 bool shouldDoCopyPhase() { return m_shouldDoCopyPhase; }
7779
7880 static CopiedBlock* blockFor(void*);
7981

@@private:
8890 void allocateBlock();
8991 CopiedBlock* allocateBlockForCopyingPhase();
9092
91  void doneFillingBlock(CopiedBlock*);
92  void recycleBlock(CopiedBlock*);
 93 void doneFillingBlock(CopiedBlock*, CopiedBlock**);
 94 void recycleEvacuatedBlock(CopiedBlock*);
 95 void recycleBorrowedBlock(CopiedBlock*);
9396
9497 Heap* m_heap;
9598

@@private:
108111 DoublyLinkedList<CopiedBlock> m_oversizeBlocks;
109112
110113 bool m_inCopyingPhase;
 114 bool m_shouldDoCopyPhase;
111115
112116 Mutex m_loanedBlocksLock;
113117 ThreadCondition m_loanedBlocksCondition;
130954

Source/JavaScriptCore/heap/CopiedSpaceInlineMethods.h

@@inline void CopiedSpace::pinIfNecessary(
9393 pin(block);
9494}
9595
96 inline void CopiedSpace::startedCopying()
 96inline void CopiedSpace::recycleEvacuatedBlock(CopiedBlock* block)
9797{
98  std::swap(m_fromSpace, m_toSpace);
99 
100  m_blockFilter.reset();
101  m_allocator.resetCurrentBlock();
102 
103  ASSERT(!m_inCopyingPhase);
104  ASSERT(!m_numberOfLoanedBlocks);
105  m_inCopyingPhase = true;
 98 ASSERT(block);
 99 ASSERT(block->canBeRecycled());
 100 ASSERT(!block->m_isPinned);
 101 {
 102 SpinLockHolder locker(&m_toSpaceLock);
 103 m_blockSet.remove(block);
 104 m_fromSpace->remove(block);
 105 }
 106 m_heap->blockAllocator().deallocate(CopiedBlock::destroy(block));
106107}
107108
108 inline void CopiedSpace::recycleBlock(CopiedBlock* block)
 109inline void CopiedSpace::recycleBorrowedBlock(CopiedBlock* block)
109110{
110111 m_heap->blockAllocator().deallocate(CopiedBlock::destroy(block));
111112
130954

Source/JavaScriptCore/heap/CopyVisitor.cpp

 1/*
 2 * Copyright (C) 2012 Apple Inc. All rights reserved.
 3 *
 4 * Redistribution and use in source and binary forms, with or without
 5 * modification, are permitted provided that the following conditions
 6 * are met:
 7 * 1. Redistributions of source code must retain the above copyright
 8 * notice, this list of conditions and the following disclaimer.
 9 * 2. Redistributions in binary form must reproduce the above copyright
 10 * notice, this list of conditions and the following disclaimer in the
 11 * documentation and/or other materials provided with the distribution.
 12 *
 13 * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
 14 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 15 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 16 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
 17 * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 18 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 19 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 20 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 21 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 22 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
 23 * THE POSSIBILITY OF SUCH DAMAGE.
 24 */
 25
 26#include "config.h"
 27#include "CopyVisitor.h"
 28
 29#include "CopyVisitorInlineMethods.h"
 30#include "GCThreadSharedData.h"
 31#include "JSCell.h"
 32#include "JSObject.h"
 33#include <wtf/Threading.h>
 34
 35namespace JSC {
 36
 37CopyVisitor::CopyVisitor(GCThreadSharedData& shared)
 38 : m_shared(shared)
 39{
 40}
 41
 42void CopyVisitor::copyFromShared()
 43{
 44 GCCopyPhaseFunctor functor(*this);
 45 Vector<MarkedBlock*>& blocksToCopy = m_shared.m_blocksToCopy;
 46 size_t startIndex, endIndex;
 47
 48 m_shared.getNextBlocksToCopy(startIndex, endIndex);
 49 while (startIndex < endIndex) {
 50 for (size_t i = startIndex; i < endIndex; i++)
 51 blocksToCopy[i]->forEachLiveCell(functor);
 52 m_shared.getNextBlocksToCopy(startIndex, endIndex);
 53 }
 54 ASSERT(startIndex == endIndex);
 55}
 56
 57} // namespace JSC
0

Source/JavaScriptCore/heap/CopyVisitor.h

 1/*
 2 * Copyright (C) 2012 Apple Inc. All rights reserved.
 3 *
 4 * Redistribution and use in source and binary forms, with or without
 5 * modification, are permitted provided that the following conditions
 6 * are met:
 7 * 1. Redistributions of source code must retain the above copyright
 8 * notice, this list of conditions and the following disclaimer.
 9 * 2. Redistributions in binary form must reproduce the above copyright
 10 * notice, this list of conditions and the following disclaimer in the
 11 * documentation and/or other materials provided with the distribution.
 12 *
 13 * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
 14 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 15 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 16 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
 17 * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 18 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 19 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 20 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 21 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 22 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
 23 * THE POSSIBILITY OF SUCH DAMAGE.
 24 */
 25
 26#ifndef CopyVisitor_h
 27#define CopyVisitor_h
 28
 29#include "CopiedSpace.h"
 30
 31namespace JSC {
 32
 33class GCThreadSharedData;
 34
 35class CopyVisitor {
 36public:
 37 CopyVisitor(GCThreadSharedData&);
 38
 39 void copyFromShared();
 40
 41 void startCopying();
 42 void doneCopying();
 43
 44 // Low-level API for copying, appropriate for cases where the object's heap references
 45 // are discontiguous or if the object occurs frequently enough that you need to focus on
 46 // performance. Use this with care as it is easy to shoot yourself in the foot.
 47 bool checkIfShouldCopy(void*, size_t);
 48 void* allocateNewSpace(size_t);
 49 void didCopy(void*, size_t);
 50
 51private:
 52 void* allocateNewSpaceSlow(size_t);
 53
 54 GCThreadSharedData& m_shared;
 55 CopiedAllocator m_copiedAllocator;
 56};
 57
 58} // namespace JSC
 59
 60#endif
0

Source/JavaScriptCore/heap/CopyVisitorInlineMethods.h

 1/*
 2 * Copyright (C) 2012 Apple Inc. All rights reserved.
 3 *
 4 * Redistribution and use in source and binary forms, with or without
 5 * modification, are permitted provided that the following conditions
 6 * are met:
 7 * 1. Redistributions of source code must retain the above copyright
 8 * notice, this list of conditions and the following disclaimer.
 9 * 2. Redistributions in binary form must reproduce the above copyright
 10 * notice, this list of conditions and the following disclaimer in the
 11 * documentation and/or other materials provided with the distribution.
 12 *
 13 * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
 14 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 15 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 16 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
 17 * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 18 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 19 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 20 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 21 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 22 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
 23 * THE POSSIBILITY OF SUCH DAMAGE.
 24 */
 25
 26#ifndef CopyVisitorInlineMethods_h
 27#define CopyVisitorInlineMethods_h
 28
 29#include "ClassInfo.h"
 30#include "CopyVisitor.h"
 31#include "GCThreadSharedData.h"
 32#include "JSCell.h"
 33#include "JSDestructibleObject.h"
 34
 35namespace JSC {
 36
 37class GCCopyPhaseFunctor : public MarkedBlock::VoidFunctor {
 38public:
 39 GCCopyPhaseFunctor(CopyVisitor& visitor)
 40 : m_visitor(visitor)
 41 {
 42 }
 43
 44 void operator()(JSCell* cell)
 45 {
 46 Structure* structure = cell->structure();
 47 if (!structure->outOfLineCapacity() && !hasIndexedProperties(structure->indexingType()))
 48 return;
 49 ASSERT(structure->classInfo()->methodTable.copyBackingStore == JSObject::copyBackingStore);
 50 JSObject::copyBackingStore(cell, m_visitor);
 51 }
 52
 53private:
 54 CopyVisitor& m_visitor;
 55};
 56
 57inline bool CopyVisitor::checkIfShouldCopy(void* oldPtr, size_t bytes)
 58{
 59 if (CopiedSpace::isOversize(bytes)) {
 60 ASSERT(CopiedSpace::oversizeBlockFor(oldPtr)->isPinned());
 61 return false;
 62 }
 63
 64 if (CopiedSpace::blockFor(oldPtr)->isPinned())
 65 return false;
 66
 67 return true;
 68}
 69
 70inline void* CopyVisitor::allocateNewSpace(size_t bytes)
 71{
 72 void* result = 0; // Compilers don't realize that this will be assigned.
 73 if (LIKELY(m_copiedAllocator.tryAllocate(bytes, &result)))
 74 return result;
 75
 76 result = allocateNewSpaceSlow(bytes);
 77 ASSERT(result);
 78 return result;
 79}
 80
 81inline void* CopyVisitor::allocateNewSpaceSlow(size_t bytes)
 82{
 83 CopiedBlock* newBlock = 0;
 84 m_shared.m_copiedSpace->doneFillingBlock(m_copiedAllocator.resetCurrentBlock(), &newBlock);
 85 m_copiedAllocator.setCurrentBlock(newBlock);
 86
 87 void* result = 0;
 88 CheckedBoolean didSucceed = m_copiedAllocator.tryAllocate(bytes, &result);
 89 ASSERT(didSucceed);
 90 return result;
 91}
 92
 93inline void CopyVisitor::startCopying()
 94{
 95 ASSERT(!m_copiedAllocator.isValid());
 96 CopiedBlock* block = 0;
 97 m_shared.m_copiedSpace->doneFillingBlock(m_copiedAllocator.resetCurrentBlock(), &block);
 98 m_copiedAllocator.setCurrentBlock(block);
 99}
 100
 101inline void CopyVisitor::doneCopying()
 102{
 103 if (!m_copiedAllocator.isValid())
 104 return;
 105
 106 m_shared.m_copiedSpace->doneFillingBlock(m_copiedAllocator.resetCurrentBlock(), 0);
 107}
 108
 109inline void CopyVisitor::didCopy(void* ptr, size_t bytes)
 110{
 111 ASSERT(!CopiedSpace::isOversize(bytes));
 112 CopiedBlock* block = CopiedSpace::blockFor(ptr);
 113 ASSERT(!block->isPinned());
 114
 115 if (block->didEvacuateBytes(bytes))
 116 m_shared.m_copiedSpace->recycleEvacuatedBlock(block);
 117}
 118
 119} // namespace JSC
 120
 121#endif
0

Source/JavaScriptCore/heap/GCThread.cpp

 1/*
 2 * Copyright (C) 2012 Apple Inc. All rights reserved.
 3 *
 4 * Redistribution and use in source and binary forms, with or without
 5 * modification, are permitted provided that the following conditions
 6 * are met:
 7 * 1. Redistributions of source code must retain the above copyright
 8 * notice, this list of conditions and the following disclaimer.
 9 * 2. Redistributions in binary form must reproduce the above copyright
 10 * notice, this list of conditions and the following disclaimer in the
 11 * documentation and/or other materials provided with the distribution.
 12 *
 13 * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
 14 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 15 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 16 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
 17 * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 18 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 19 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 20 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 21 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 22 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
 23 * THE POSSIBILITY OF SUCH DAMAGE.
 24 */
 25
 26#include "config.h"
 27#include "GCThread.h"
 28
 29#include "CopyVisitor.h"
 30#include "CopyVisitorInlineMethods.h"
 31#include "GCThreadSharedData.h"
 32#include "SlotVisitor.h"
 33#include <wtf/MainThread.h>
 34#include <wtf/PassOwnPtr.h>
 35
 36namespace JSC {
 37
 38GCThread::GCThread(GCThreadSharedData& shared, SlotVisitor* slotVisitor, CopyVisitor* copyVisitor, size_t index)
 39 : m_threadID(0)
 40 , m_shared(shared)
 41 , m_slotVisitor(WTF::adoptPtr(slotVisitor))
 42 , m_copyVisitor(WTF::adoptPtr(copyVisitor))
 43 , m_index(index)
 44{
 45}
 46
 47ThreadIdentifier GCThread::threadID()
 48{
 49 ASSERT(m_threadID);
 50 return m_threadID;
 51}
 52
 53void GCThread::initializeThreadID(ThreadIdentifier threadID)
 54{
 55 ASSERT(!m_threadID);
 56 m_threadID = threadID;
 57}
 58
 59SlotVisitor* GCThread::slotVisitor()
 60{
 61 ASSERT(m_slotVisitor);
 62 return m_slotVisitor.get();
 63}
 64
 65CopyVisitor* GCThread::copyVisitor()
 66{
 67 ASSERT(m_copyVisitor);
 68 return m_copyVisitor.get();
 69}
 70
 71GCPhase GCThread::waitForNextPhase()
 72{
 73 MutexLocker locker(m_shared.m_phaseLock);
 74 while (m_shared.m_currentPhase == NoPhase)
 75 m_shared.m_phaseCondition.wait(m_shared.m_phaseLock);
 76 return m_shared.m_currentPhase;
 77}
 78
 79void GCThread::gcThreadMain()
 80{
 81 GCPhase currentPhase;
 82#if ENABLE(PARALLEL_GC)
 83 WTF::registerGCThread();
 84#endif
 85 // Wait for the main thread to finish creating us.
 86 {
 87 MutexLocker locker(m_shared.m_markingLock);
 88 }
 89 {
 90 ParallelModeEnabler enabler(*m_slotVisitor);
 91 while ((currentPhase = waitForNextPhase()) != Exit) {
 92 switch (currentPhase) {
 93 case Mark:
 94 m_slotVisitor->drainFromShared(SlotVisitor::SlaveDrain);
 95 break;
 96 case Copy:
 97 // We don't have to call startCopying() because it's called for us on the main thread.
 98 m_copyVisitor->startCopying();
 99 m_copyVisitor->copyFromShared();
 100 m_copyVisitor->doneCopying();
 101 break;
 102 case NoPhase:
 103 ASSERT_NOT_REACHED();
 104 break;
 105 case Exit:
 106 ASSERT_NOT_REACHED();
 107 break;
 108 }
 109 }
 110 }
 111}
 112
 113void GCThread::gcThreadStartFunc(void* data)
 114{
 115 GCThread* thread = static_cast<GCThread*>(data);
 116 thread->gcThreadMain();
 117}
 118
 119} // namespace JSC
0

Source/JavaScriptCore/heap/GCThread.h

 1/*
 2 * Copyright (C) 2012 Apple Inc. All rights reserved.
 3 *
 4 * Redistribution and use in source and binary forms, with or without
 5 * modification, are permitted provided that the following conditions
 6 * are met:
 7 * 1. Redistributions of source code must retain the above copyright
 8 * notice, this list of conditions and the following disclaimer.
 9 * 2. Redistributions in binary form must reproduce the above copyright
 10 * notice, this list of conditions and the following disclaimer in the
 11 * documentation and/or other materials provided with the distribution.
 12 *
 13 * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
 14 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 15 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 16 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
 17 * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 18 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 19 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 20 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 21 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 22 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
 23 * THE POSSIBILITY OF SUCH DAMAGE.
 24 */
 25
 26#ifndef GCThread_h
 27#define GCThread_h
 28
 29#include <GCThreadSharedData.h>
 30#include <wtf/Deque.h>
 31#include <wtf/OwnPtr.h>
 32#include <wtf/Threading.h>
 33
 34namespace JSC {
 35
 36class CopyVisitor;
 37class GCThreadSharedData;
 38class SlotVisitor;
 39
 40class GCThread {
 41public:
 42 GCThread(GCThreadSharedData&, SlotVisitor*, CopyVisitor*, size_t);
 43
 44 SlotVisitor* slotVisitor();
 45 CopyVisitor* copyVisitor();
 46 ThreadIdentifier threadID();
 47 void initializeThreadID(ThreadIdentifier);
 48
 49 static void gcThreadStartFunc(void*);
 50
 51private:
 52 void gcThreadMain();
 53 GCPhase waitForNextPhase();
 54
 55 ThreadIdentifier m_threadID;
 56 GCThreadSharedData& m_shared;
 57 OwnPtr<SlotVisitor> m_slotVisitor;
 58 OwnPtr<CopyVisitor> m_copyVisitor;
 59 size_t m_index;
 60};
 61
 62} // namespace JSC
 63
 64#endif
0

Source/JavaScriptCore/heap/GCThreadSharedData.cpp

2626#include "config.h"
2727#include "GCThreadSharedData.h"
2828
 29#include "CopyVisitor.h"
 30#include "CopyVisitorInlineMethods.h"
 31#include "GCThread.h"
2932#include "JSGlobalData.h"
3033#include "MarkStack.h"
3134#include "SlotVisitor.h"
3235#include "SlotVisitorInlineMethods.h"
33 #include <wtf/MainThread.h>
3436
3537namespace JSC {
3638
3739#if ENABLE(PARALLEL_GC)
3840void GCThreadSharedData::resetChildren()
3941{
40  for (unsigned i = 0; i < m_markingThreadsMarkStack.size(); ++i)
41  m_markingThreadsMarkStack[i]->reset();
 42 for (size_t i = 0; i < m_gcThreads.size(); ++i)
 43 m_gcThreads[i]->slotVisitor()->reset();
4244}
4345
4446size_t GCThreadSharedData::childVisitCount()
4547{
4648 unsigned long result = 0;
47  for (unsigned i = 0; i < m_markingThreadsMarkStack.size(); ++i)
48  result += m_markingThreadsMarkStack[i]->visitCount();
 49 for (unsigned i = 0; i < m_gcThreads.size(); ++i)
 50 result += m_gcThreads[i]->slotVisitor()->visitCount();
4951 return result;
5052}
51 
52 void GCThreadSharedData::markingThreadMain(SlotVisitor* slotVisitor)
53 {
54  WTF::registerGCThread();
55  {
56  ParallelModeEnabler enabler(*slotVisitor);
57  slotVisitor->drainFromShared(SlotVisitor::SlaveDrain);
58  }
59  delete slotVisitor;
60 }
61 
62 void GCThreadSharedData::markingThreadStartFunc(void* myVisitor)
63 {
64  SlotVisitor* slotVisitor = static_cast<SlotVisitor*>(myVisitor);
65 
66  slotVisitor->sharedData().markingThreadMain(slotVisitor);
67 }
6853#endif
6954
7055GCThreadSharedData::GCThreadSharedData(JSGlobalData* globalData)

@@GCThreadSharedData::GCThreadSharedData(J
7459 , m_sharedMarkStack(m_segmentAllocator)
7560 , m_numberOfActiveParallelMarkers(0)
7661 , m_parallelMarkersShouldExit(false)
 62 , m_blocksToCopy(globalData->heap.m_blockSnapshot)
 63 , m_copyIndex(0)
 64 , m_currentPhase(NoPhase)
7765{
 66 m_copyLock.Init();
7867#if ENABLE(PARALLEL_GC)
 68 // Grab the lock so the new GC threads can be properly initialized before they start running.
 69 MutexLocker locker(m_markingLock);
7970 for (unsigned i = 1; i < Options::numberOfGCMarkers(); ++i) {
8071 SlotVisitor* slotVisitor = new SlotVisitor(*this);
81  m_markingThreadsMarkStack.append(slotVisitor);
82  m_markingThreads.append(createThread(markingThreadStartFunc, slotVisitor, "JavaScriptCore::Marking"));
83  ASSERT(m_markingThreads.last());
 72 CopyVisitor* copyVisitor = new CopyVisitor(*this);
 73 size_t index = m_gcThreads.size();
 74 GCThread* newThread = new GCThread(*this, slotVisitor, copyVisitor, index);
 75 ThreadIdentifier threadID = createThread(GCThread::gcThreadStartFunc, newThread, "JavaScriptCore::Marking");
 76 newThread->initializeThreadID(threadID);
 77 m_gcThreads.append(newThread);
8478 }
8579#endif
8680}

@@GCThreadSharedData::~GCThreadSharedData(
9084#if ENABLE(PARALLEL_GC)
9185 // Destroy our marking threads.
9286 {
93  MutexLocker locker(m_markingLock);
 87 MutexLocker markingLocker(m_markingLock);
 88 MutexLocker phaseLocker(m_phaseLock);
 89 ASSERT(m_currentPhase == NoPhase);
9490 m_parallelMarkersShouldExit = true;
95  m_markingCondition.broadcast();
 91 m_currentPhase = Exit;
 92 m_phaseCondition.broadcast();
 93 }
 94 for (unsigned i = 0; i < m_gcThreads.size(); ++i) {
 95 waitForThreadCompletion(m_gcThreads[i]->threadID());
 96 delete m_gcThreads[i];
9697 }
97  for (unsigned i = 0; i < m_markingThreads.size(); ++i)
98  waitForThreadCompletion(m_markingThreads[i]);
9998#endif
10099}
101100
102101void GCThreadSharedData::reset()
103102{
104  ASSERT(!m_numberOfActiveParallelMarkers);
105  ASSERT(!m_parallelMarkersShouldExit);
106103 ASSERT(m_sharedMarkStack.isEmpty());
107104
108105#if ENABLE(PARALLEL_GC)

@@void GCThreadSharedData::reset()
119116 }
120117}
121118
 119void GCThreadSharedData::didStartMarking()
 120{
 121 MutexLocker markingLocker(m_markingLock);
 122 MutexLocker phaseLocker(m_phaseLock);
 123 ASSERT(m_currentPhase == NoPhase);
 124 m_currentPhase = Mark;
 125 m_parallelMarkersShouldExit = false;
 126 m_phaseCondition.broadcast();
 127}
 128
 129void GCThreadSharedData::didFinishMarking()
 130{
 131 MutexLocker markingLocker(m_markingLock);
 132 MutexLocker phaseLocker(m_phaseLock);
 133 ASSERT(m_currentPhase == Mark);
 134 m_currentPhase = NoPhase;
 135 m_parallelMarkersShouldExit = true;
 136 m_markingCondition.broadcast();
 137}
 138
 139void GCThreadSharedData::didStartCopying()
 140{
 141 {
 142 SpinLockHolder locker(&m_copyLock);
 143 m_blocksToCopy = m_globalData->heap.m_blockSnapshot;
 144 m_copyIndex = 0;
 145 }
 146
 147 // We do this here so that we avoid a race condition where the main thread can
 148 // blow through all of the copying work before the GCThreads fully wake up.
 149 // The GCThreads then request a block from the CopiedSpace when the copying phase
 150 // has completed, which isn't allowed.
 151 for (size_t i = 0; i < m_gcThreads.size(); i++)
 152 m_gcThreads[i]->copyVisitor()->startCopying();
 153
 154 MutexLocker locker(m_phaseLock);
 155 ASSERT(m_currentPhase == NoPhase);
 156 m_currentPhase = Copy;
 157 m_phaseCondition.broadcast();
 158}
 159
 160void GCThreadSharedData::didFinishCopying()
 161{
 162 MutexLocker locker(m_phaseLock);
 163 ASSERT(m_currentPhase == Copy);
 164 m_currentPhase = NoPhase;
 165 m_phaseCondition.broadcast();
 166}
 167
122168} // namespace JSC
130954

Source/JavaScriptCore/heap/GCThreadSharedData.h

2828
2929#include "ListableHandler.h"
3030#include "MarkStack.h"
 31#include "MarkedBlock.h"
3132#include "UnconditionalFinalizer.h"
3233#include "WeakReferenceHarvester.h"
3334#include <wtf/HashSet.h>
 35#include <wtf/TCSpinLock.h>
3436#include <wtf/Threading.h>
3537#include <wtf/Vector.h>
3638
3739namespace JSC {
3840
 41class GCThread;
3942class JSGlobalData;
4043class CopiedSpace;
 44class CopyVisitor;
 45
 46enum GCPhase {
 47 NoPhase,
 48 Mark,
 49 Copy,
 50 Exit
 51};
4152
4253class GCThreadSharedData {
4354public:

@@public:
4657
4758 void reset();
4859
 60 void didStartMarking();
 61 void didFinishMarking();
 62 void didStartCopying();
 63 void didFinishCopying();
 64
4965#if ENABLE(PARALLEL_GC)
5066 void resetChildren();
5167 size_t childVisitCount();

@@public:
5369#endif
5470
5571private:
 72 friend class GCThread;
5673 friend class SlotVisitor;
 74 friend class CopyVisitor;
5775
58 #if ENABLE(PARALLEL_GC)
59  void markingThreadMain(SlotVisitor*);
60  static void markingThreadStartFunc(void* heap);
61 #endif
 76 void getNextBlocksToCopy(size_t&, size_t&);
6277
6378 JSGlobalData* m_globalData;
6479 CopiedSpace* m_copiedSpace;

@@private:
6782
6883 bool m_shouldHashConst;
6984
70  Vector<ThreadIdentifier> m_markingThreads;
71  Vector<SlotVisitor*> m_markingThreadsMarkStack;
72 
 85 Vector<GCThread*> m_gcThreads;
 86
7387 Mutex m_markingLock;
7488 ThreadCondition m_markingCondition;
7589 MarkStackArray m_sharedMarkStack;

@@private:
7993 Mutex m_opaqueRootsLock;
8094 HashSet<void*> m_opaqueRoots;
8195
 96 SpinLock m_copyLock;
 97 Vector<MarkedBlock*>& m_blocksToCopy;
 98 size_t m_copyIndex;
 99 static const size_t s_blockFragmentLength = 32;
 100
 101 Mutex m_phaseLock;
 102 ThreadCondition m_phaseCondition;
 103 GCPhase m_currentPhase;
 104
82105 ListableHandler<WeakReferenceHarvester>::List m_weakReferenceHarvesters;
83106 ListableHandler<UnconditionalFinalizer>::List m_unconditionalFinalizers;
84107};
85108
 109inline void GCThreadSharedData::getNextBlocksToCopy(size_t& start, size_t& end)
 110{
 111 SpinLockHolder locker(&m_copyLock);
 112 start = m_copyIndex;
 113 end = std::min(m_blocksToCopy.size(), m_copyIndex + s_blockFragmentLength);
 114 m_copyIndex = end;
 115}
 116
86117} // namespace JSC
87118
88119#endif
130954

Source/JavaScriptCore/heap/Heap.cpp

2121#include "config.h"
2222#include "Heap.h"
2323
24 #include "CopiedSpace.h"
25 #include "CopiedSpaceInlineMethods.h"
2624#include "CodeBlock.h"
2725#include "ConservativeRoots.h"
 26#include "CopiedSpace.h"
 27#include "CopiedSpaceInlineMethods.h"
 28#include "CopyVisitorInlineMethods.h"
2829#include "GCActivityCallback.h"
2930#include "HeapRootVisitor.h"
3031#include "HeapStatistics.h"

@@Heap::Heap(JSGlobalData* globalData, Hea
252253 , m_machineThreads(this)
253254 , m_sharedData(globalData)
254255 , m_slotVisitor(m_sharedData)
 256 , m_copyVisitor(m_sharedData)
255257 , m_handleSet(globalData)
256258 , m_isSafeToCollect(false)
257259 , m_globalData(globalData)

@@void Heap::markRoots(bool fullGC)
464466 m_objectSpace.clearMarks();
465467 }
466468
467  m_storageSpace.startedCopying();
 469 m_sharedData.didStartMarking();
468470 SlotVisitor& visitor = m_slotVisitor;
469471 visitor.setup();
470472 HeapRootVisitor heapRootVisitor(visitor);

@@void Heap::markRoots(bool fullGC)
589591
590592 GCCOUNTER(VisitedValueCount, visitor.visitCount());
591593
592  visitor.doneCopying();
 594 m_sharedData.didFinishMarking();
593595#if ENABLE(OBJECT_MARK_LOGGING)
594596 size_t visitCount = visitor.visitCount();
595597#if ENABLE(PARALLEL_GC)

@@void Heap::markRoots(bool fullGC)
603605 m_sharedData.resetChildren();
604606#endif
605607 m_sharedData.reset();
 608}
 609
 610void Heap::copyBackingStores()
 611{
 612 m_storageSpace.startedCopying();
 613 if (m_storageSpace.shouldDoCopyPhase()) {
 614 m_sharedData.didStartCopying();
 615 CopyVisitor& visitor = m_copyVisitor;
 616 visitor.startCopying();
 617 visitor.copyFromShared();
 618 visitor.doneCopying();
 619 m_sharedData.didFinishCopying();
 620 }
606621 m_storageSpace.doneCopying();
607622}
608623

@@void Heap::collect(SweepToggle sweepTogg
734749 JAVASCRIPTCORE_GC_MARKED();
735750
736751 {
 752 m_blockSnapshot.resize(m_objectSpace.blocks().set().size());
 753 CopyFunctor functor(m_blockSnapshot);
 754 m_objectSpace.forEachBlock(functor);
 755 }
 756
 757 copyBackingStores();
 758
 759 {
737760 GCPHASE(FinalizeUnconditionalFinalizers);
738761 finalizeUnconditionalFinalizers();
739762 }

@@void Heap::collect(SweepToggle sweepTogg
755778 m_objectSpace.shrink();
756779 }
757780
758  m_sweeper->startSweeping(m_objectSpace.blocks().set());
 781 m_sweeper->startSweeping(m_blockSnapshot);
759782 m_bytesAbandoned = 0;
760783
761784 {
130954

Source/JavaScriptCore/heap/Heap.h

2323#define Heap_h
2424
2525#include "BlockAllocator.h"
 26#include "CopyVisitor.h"
2627#include "DFGCodeBlocks.h"
2728#include "GCThreadSharedData.h"
2829#include "HandleSet.h"

@@namespace JSC {
182183 friend class MarkedAllocator;
183184 friend class MarkedBlock;
184185 friend class CopiedSpace;
 186 friend class CopyVisitor;
185187 friend class SlotVisitor;
 188 friend class IncrementalSweeper;
186189 friend class HeapStatistics;
187190 template<typename T> friend void* allocateCell(Heap&);
188191 template<typename T> friend void* allocateCell(Heap&, size_t);

@@namespace JSC {
204207 void markRoots(bool fullGC);
205208 void markProtectedObjects(HeapRootVisitor&);
206209 void markTempSortVectors(HeapRootVisitor&);
 210 void copyBackingStores();
207211 void harvestWeakReferences();
208212 void finalizeUnconditionalFinalizers();
209213 void deleteUnmarkedCompiledCode();

@@namespace JSC {
239243
240244 GCThreadSharedData m_sharedData;
241245 SlotVisitor m_slotVisitor;
 246 CopyVisitor m_copyVisitor;
242247
243248 HandleSet m_handleSet;
244249 HandleStack m_handleStack;

@@namespace JSC {
256261
257262 GCActivityCallback* m_activityCallback;
258263 IncrementalSweeper* m_sweeper;
 264 Vector<MarkedBlock*> m_blockSnapshot;
 265 };
 266
 267 struct CopyFunctor : public MarkedBlock::VoidFunctor {
 268 CopyFunctor(Vector<MarkedBlock*>& blocks)
 269 : m_index(0)
 270 , m_blocks(blocks)
 271 {
 272 }
 273
 274 void operator()(MarkedBlock* block) { m_blocks[m_index++] = block; }
 275
 276 size_t m_index;
 277 Vector<MarkedBlock*>& m_blocks;
259278 };
260279
261280 inline bool Heap::shouldCollect()
130954

Source/JavaScriptCore/heap/IncrementalSweeper.cpp

@@static const double sweepTimeMultiplier
4848IncrementalSweeper::IncrementalSweeper(Heap* heap, CFRunLoopRef runLoop)
4949 : HeapTimer(heap->globalData(), runLoop)
5050 , m_currentBlockToSweepIndex(0)
 51 , m_blocksToSweep(heap->m_blockSnapshot)
5152{
5253}
5354

@@void IncrementalSweeper::sweepNextBlock(
127128 }
128129}
129130
130 void IncrementalSweeper::startSweeping(const HashSet<MarkedBlock*>& blockSnapshot)
 131void IncrementalSweeper::startSweeping(Vector<MarkedBlock*>& blockSnapshot)
131132{
132  m_blocksToSweep.resize(blockSnapshot.size());
133  CopyFunctor functor(m_blocksToSweep);
134  m_globalData->heap.objectSpace().forEachBlock(functor);
 133 m_blocksToSweep = blockSnapshot;
135134 m_currentBlockToSweepIndex = 0;
136135 scheduleTimer();
137136}

@@IncrementalSweeper* IncrementalSweeper::
160159 return new IncrementalSweeper(heap->globalData());
161160}
162161
163 void IncrementalSweeper::startSweeping(const HashSet<MarkedBlock*>&)
 162void IncrementalSweeper::startSweeping(Vector<MarkedBlock*>&)
164163{
165164}
166165
130954

Source/JavaScriptCore/heap/IncrementalSweeper.h

@@namespace JSC {
3737
3838class Heap;
3939
40 struct CopyFunctor : public MarkedBlock::VoidFunctor {
41  CopyFunctor(Vector<MarkedBlock*>& blocks)
42  : m_index(0)
43  , m_blocks(blocks)
44  {
45  }
46 
47  void operator()(MarkedBlock* block) { m_blocks[m_index++] = block; }
48 
49  size_t m_index;
50  Vector<MarkedBlock*>& m_blocks;
51 };
52 
5340class IncrementalSweeper : public HeapTimer {
5441public:
5542 static IncrementalSweeper* create(Heap*);
56  void startSweeping(const HashSet<MarkedBlock*>& blockSnapshot);
 43 void startSweeping(Vector<MarkedBlock*>&);
5744 virtual void doWork();
5845 void sweepNextBlock();
5946 void willFinishSweeping();

@@private:
7158 void cancelTimer();
7259
7360 unsigned m_currentBlockToSweepIndex;
74  Vector<MarkedBlock*> m_blocksToSweep;
 61 Vector<MarkedBlock*>& m_blocksToSweep;
7562#else
7663
7764 IncrementalSweeper(JSGlobalData*);
130954

Source/JavaScriptCore/heap/SlotVisitor.cpp

44#include "ConservativeRoots.h"
55#include "CopiedSpace.h"
66#include "CopiedSpaceInlineMethods.h"
 7#include "GCThread.h"
78#include "JSArray.h"
89#include "JSDestructibleObject.h"
910#include "JSGlobalData.h"

@@void SlotVisitor::setup()
3536 m_shared.m_shouldHashConst = m_shared.m_globalData->haveEnoughNewStringsToHashConst();
3637 m_shouldHashConst = m_shared.m_shouldHashConst;
3738#if ENABLE(PARALLEL_GC)
38  for (unsigned i = 0; i < m_shared.m_markingThreadsMarkStack.size(); ++i)
39  m_shared.m_markingThreadsMarkStack[i]->m_shouldHashConst = m_shared.m_shouldHashConst;
 39 for (unsigned i = 0; i < m_shared.m_gcThreads.size(); ++i)
 40 m_shared.m_gcThreads[i]->slotVisitor()->m_shouldHashConst = m_shared.m_shouldHashConst;
4041#endif
4142}
4243

@@void SlotVisitor::drainFromShared(Shared
181182 while (true) {
182183 // Did we reach termination?
183184 if (!m_shared.m_numberOfActiveParallelMarkers && m_shared.m_sharedMarkStack.isEmpty()) {
184  // Let any sleeping slaves know it's time for them to give their private CopiedBlocks back
 185 // Let any sleeping slaves know it's time for them to return;
185186 m_shared.m_markingCondition.broadcast();
186187 return;
187188 }

@@void SlotVisitor::drainFromShared(Shared
200201 if (!m_shared.m_numberOfActiveParallelMarkers && m_shared.m_sharedMarkStack.isEmpty())
201202 m_shared.m_markingCondition.broadcast();
202203
203  while (m_shared.m_sharedMarkStack.isEmpty() && !m_shared.m_parallelMarkersShouldExit) {
204  if (!m_shared.m_numberOfActiveParallelMarkers && m_shared.m_sharedMarkStack.isEmpty())
205  doneCopying();
 204 while (m_shared.m_sharedMarkStack.isEmpty() && !m_shared.m_parallelMarkersShouldExit)
206205 m_shared.m_markingCondition.wait(m_shared.m_markingLock);
207  }
208206
209  // Is the VM exiting? If so, exit this thread.
210  if (m_shared.m_parallelMarkersShouldExit) {
211  doneCopying();
 207 // Is the current phase done? If so, return from this function.
 208 if (m_shared.m_parallelMarkersShouldExit)
212209 return;
213  }
214210 }
215211
216212 size_t idleThreadCount = Options::numberOfGCMarkers() - m_shared.m_numberOfActiveParallelMarkers;

@@void SlotVisitor::mergeOpaqueRoots()
236232 m_opaqueRoots.clear();
237233}
238234
239 void SlotVisitor::startCopying()
240 {
241  ASSERT(!m_copiedAllocator.isValid());
242 }
243 
244 void* SlotVisitor::allocateNewSpaceSlow(size_t bytes)
245 {
246  m_shared.m_copiedSpace->doneFillingBlock(m_copiedAllocator.resetCurrentBlock());
247  m_copiedAllocator.setCurrentBlock(m_shared.m_copiedSpace->allocateBlockForCopyingPhase());
248 
249  void* result = 0;
250  CheckedBoolean didSucceed = m_copiedAllocator.tryAllocate(bytes, &result);
251  ASSERT(didSucceed);
252  return result;
253 }
254 
255 void* SlotVisitor::allocateNewSpaceOrPin(void* ptr, size_t bytes)
256 {
257  if (!checkIfShouldCopyAndPinOtherwise(ptr, bytes))
258  return 0;
259 
260  return allocateNewSpace(bytes);
261 }
262 
263235ALWAYS_INLINE bool JSString::tryHashConstLock()
264236{
265237#if ENABLE(PARALLEL_GC)

@@ALWAYS_INLINE void SlotVisitor::internal
335307 internalAppend(cell);
336308}
337309
338 void SlotVisitor::copyAndAppend(void** ptr, size_t bytes, JSValue* values, unsigned length)
339 {
340  void* oldPtr = *ptr;
341  void* newPtr = allocateNewSpaceOrPin(oldPtr, bytes);
342  if (newPtr) {
343  size_t jsValuesOffset = static_cast<size_t>(reinterpret_cast<char*>(values) - static_cast<char*>(oldPtr));
344 
345  JSValue* newValues = reinterpret_cast_ptr<JSValue*>(static_cast<char*>(newPtr) + jsValuesOffset);
346  for (unsigned i = 0; i < length; i++) {
347  JSValue& value = values[i];
348  newValues[i] = value;
349  if (!value)
350  continue;
351  internalAppend(&newValues[i]);
352  }
353 
354  memcpy(newPtr, oldPtr, jsValuesOffset);
355  *ptr = newPtr;
356  } else
357  append(values, length);
358 }
359 
360 void SlotVisitor::doneCopying()
361 {
362  if (!m_copiedAllocator.isValid())
363  return;
364 
365  m_shared.m_copiedSpace->doneFillingBlock(m_copiedAllocator.resetCurrentBlock());
366 }
367 
368310void SlotVisitor::harvestWeakReferences()
369311{
370312 for (WeakReferenceHarvester* current = m_shared.m_weakReferenceHarvesters.head(); current; current = current->next())
130954

Source/JavaScriptCore/heap/SlotVisitor.h

2626#ifndef SlotVisitor_h
2727#define SlotVisitor_h
2828
29 #include "CopiedSpace.h"
3029#include "HandleTypes.h"
3130#include "MarkStackInlineMethods.h"
3231

@@public:
8079 void harvestWeakReferences();
8180 void finalizeUnconditionalFinalizers();
8281
83  void startCopying();
 82 void copyLater(void*, size_t);
8483
85  // High-level API for copying, appropriate for cases where the object's heap references
86  // fall into a contiguous region of the storage chunk and if the object for which you're
87  // doing copying does not occur frequently.
88  void copyAndAppend(void**, size_t, JSValue*, unsigned);
89 
90  // Low-level API for copying, appropriate for cases where the object's heap references
91  // are discontiguous or if the object occurs frequently enough that you need to focus on
92  // performance. Use this with care as it is easy to shoot yourself in the foot.
93  bool checkIfShouldCopyAndPinOtherwise(void* oldPtr, size_t);
94  void* allocateNewSpace(size_t);
95 
96  void doneCopying();
97 
9884#if ENABLE(SIMPLE_HEAP_PROFILING)
9985 VTableSpectrum m_visitedTypeCounts;
10086#endif

@@private:
125111 void mergeOpaqueRootsIfNecessary();
126112 void mergeOpaqueRootsIfProfitable();
127113
128  void* allocateNewSpaceOrPin(void*, size_t);
129  void* allocateNewSpaceSlow(size_t);
130 
131114 void donateKnownParallel();
132115
133116 MarkStackArray m_stack;

@@private:
146129 unsigned m_logChildCount;
147130#endif
148131
149  CopiedAllocator m_copiedAllocator;
150 
151132public:
152133#if !ASSERT_DISABLED
153134 bool m_isCheckingForDefaultMarkViolation;
130954

Source/JavaScriptCore/heap/SlotVisitorInlineMethods.h

@@inline void SlotVisitor::mergeOpaqueRoot
136136 mergeOpaqueRoots();
137137}
138138
139 ALWAYS_INLINE bool SlotVisitor::checkIfShouldCopyAndPinOtherwise(void* oldPtr, size_t bytes)
140 {
141  if (CopiedSpace::isOversize(bytes)) {
142  m_shared.m_copiedSpace->pin(CopiedSpace::oversizeBlockFor(oldPtr));
143  return false;
144  }
145 
146  if (m_shared.m_copiedSpace->isPinned(oldPtr))
147  return false;
148 
149  return true;
150 }
151 
152 ALWAYS_INLINE void* SlotVisitor::allocateNewSpace(size_t bytes)
153 {
154  void* result = 0; // Compilers don't realize that this will be assigned.
155  if (LIKELY(m_copiedAllocator.tryAllocate(bytes, &result)))
156  return result;
157 
158  result = allocateNewSpaceSlow(bytes);
159  ASSERT(result);
160  return result;
161 }
162 
163139inline void SlotVisitor::donate()
164140{
165141 ASSERT(m_isInParallelMode);

@@inline void SlotVisitor::donateAndDrain(
175151 drain();
176152}
177153
 154inline void SlotVisitor::copyLater(void* ptr, size_t bytes)
 155{
 156 if (CopiedSpace::isOversize(bytes)) {
 157 m_shared.m_copiedSpace->pin(CopiedSpace::oversizeBlockFor(ptr));
 158 return;
 159 }
 160
 161 CopiedBlock* block = CopiedSpace::blockFor(ptr);
 162 if (block->isPinned())
 163 return;
 164
 165 block->reportLiveBytes(bytes);
 166
 167 if (!block->shouldEvacuate())
 168 m_shared.m_copiedSpace->pin(block);
 169}
 170
178171} // namespace JSC
179172
180173#endif // SlotVisitorInlineMethods_h
130954

Source/JavaScriptCore/runtime/Butterfly.h

3535namespace JSC {
3636
3737class JSGlobalData;
38 class SlotVisitor;
 38class CopyVisitor;
3939struct ArrayStorage;
4040
4141class Butterfly {

@@public:
7373
7474 static Butterfly* create(JSGlobalData&, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, const IndexingHeader&, size_t indexingPayloadSizeInBytes);
7575 static Butterfly* create(JSGlobalData&, Structure*);
76  static Butterfly* createUninitializedDuringCollection(SlotVisitor&, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, size_t indexingPayloadSizeInBytes);
 76 static Butterfly* createUninitializedDuringCollection(CopyVisitor&, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, size_t indexingPayloadSizeInBytes);
7777
7878 IndexingHeader* indexingHeader() { return IndexingHeader::from(this); }
7979 const IndexingHeader* indexingHeader() const { return IndexingHeader::from(this); }
130954

Source/JavaScriptCore/runtime/ButterflyInlineMethods.h

2929#include "ArrayStorage.h"
3030#include "Butterfly.h"
3131#include "CopiedSpaceInlineMethods.h"
 32#include "CopyVisitor.h"
3233#include "JSGlobalData.h"
33 #include "SlotVisitor.h"
3434#include "Structure.h"
3535
3636namespace JSC {

@@inline Butterfly* Butterfly::create(JSGl
5959 return create(globalData, 0, structure->outOfLineCapacity(), hasIndexingHeader(structure->indexingType()), IndexingHeader(), 0);
6060}
6161
62 inline Butterfly* Butterfly::createUninitializedDuringCollection(SlotVisitor& visitor, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, size_t indexingPayloadSizeInBytes)
 62inline Butterfly* Butterfly::createUninitializedDuringCollection(CopyVisitor& visitor, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, size_t indexingPayloadSizeInBytes)
6363{
6464 Butterfly* result = fromBase(
6565 visitor.allocateNewSpace(totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes)),
130954

Source/JavaScriptCore/runtime/ClassInfo.h

@@namespace JSC {
3939 typedef void (*VisitChildrenFunctionPtr)(JSCell*, SlotVisitor&);
4040 VisitChildrenFunctionPtr visitChildren;
4141
 42 typedef void (*CopyBackingStoreFunctionPtr)(JSCell*, CopyVisitor&);
 43 CopyBackingStoreFunctionPtr copyBackingStore;
 44
4245 typedef CallType (*GetCallDataFunctionPtr)(JSCell*, CallData&);
4346 GetCallDataFunctionPtr getCallData;
4447

@@struct MemberCheck##member { \
116119#define CREATE_METHOD_TABLE(ClassName) { \
117120 &ClassName::destroy, \
118121 &ClassName::visitChildren, \
 122 &ClassName::copyBackingStore, \
119123 &ClassName::getCallData, \
120124 &ClassName::getConstructData, \
121125 &ClassName::put, \
130954

Source/JavaScriptCore/runtime/JSCell.cpp

@@void JSCell::destroy(JSCell* cell)
3838 cell->JSCell::~JSCell();
3939}
4040
 41void JSCell::copyBackingStore(JSCell*, CopyVisitor&)
 42{
 43}
 44
4145bool JSCell::getString(ExecState* exec, String& stringValue) const
4246{
4347 if (!isString())
130954

Source/JavaScriptCore/runtime/JSCell.h

3838
3939namespace JSC {
4040
 41 class CopyVisitor;
4142 class JSDestructibleObject;
4243 class JSGlobalObject;
4344 class LLIntOffsetsExtractor;

@@namespace JSC {
100101 JS_EXPORT_PRIVATE JSObject* toObject(ExecState*, JSGlobalObject*) const;
101102
102103 static void visitChildren(JSCell*, SlotVisitor&);
 104 JS_EXPORT_PRIVATE static void copyBackingStore(JSCell*, CopyVisitor&);
103105
104106 // Object operations, with the toObject operation included.
105107 const ClassInfo* classInfo() const;
130954

Source/JavaScriptCore/runtime/JSObject.cpp

2626
2727#include "ButterflyInlineMethods.h"
2828#include "CopiedSpaceInlineMethods.h"
 29#include "CopyVisitor.h"
 30#include "CopyVisitorInlineMethods.h"
2931#include "DatePrototype.h"
3032#include "ErrorConstructor.h"
3133#include "GetterSetter.h"

@@static inline void getClassPropertyNames
9496 }
9597}
9698
97 ALWAYS_INLINE void JSObject::visitButterfly(SlotVisitor& visitor, Butterfly* butterfly, size_t storageSize)
 99ALWAYS_INLINE void JSObject::copyButterfly(CopyVisitor& visitor, Butterfly* butterfly, size_t storageSize)
98100{
99101 ASSERT(butterfly);
100102

@@ALWAYS_INLINE void JSObject::visitButter
111113 preCapacity = 0;
112114 indexingPayloadSizeInBytes = 0;
113115 }
114  size_t capacityInBytes = Butterfly::totalSize(
115  preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
116  if (visitor.checkIfShouldCopyAndPinOtherwise(
117  butterfly->base(preCapacity, propertyCapacity), capacityInBytes)) {
 116 size_t capacityInBytes = Butterfly::totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
 117 if (visitor.checkIfShouldCopy(butterfly->base(preCapacity, propertyCapacity), capacityInBytes)) {
118118 Butterfly* newButterfly = Butterfly::createUninitializedDuringCollection(visitor, preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
119119
120  // Mark and copy the properties.
 120 // Copy the properties.
121121 PropertyStorage currentTarget = newButterfly->propertyStorage();
122122 PropertyStorage currentSource = butterfly->propertyStorage();
123  for (size_t count = storageSize; count--;) {
124  JSValue value = (--currentSource)->get();
125  ASSERT(value);
126  visitor.appendUnbarrieredValue(&value);
127  (--currentTarget)->setWithoutWriteBarrier(value);
128  }
 123 for (size_t count = storageSize; count--;)
 124 (--currentTarget)->setWithoutWriteBarrier((--currentSource)->get());
129125
130126 if (UNLIKELY(hasIndexingHeader)) {
131127 *newButterfly->indexingHeader() = *butterfly->indexingHeader();
132128
133  // Mark and copy the array if appropriate.
 129 // Copy the array if appropriate.
134130
135131 WriteBarrier<Unknown>* currentTarget;
136132 WriteBarrier<Unknown>* currentSource;

@@ALWAYS_INLINE void JSObject::visitButter
150146 currentTarget = newButterfly->arrayStorage()->m_vector;
151147 currentSource = butterfly->arrayStorage()->m_vector;
152148 count = newButterfly->arrayStorage()->vectorLength();
153  if (newButterfly->arrayStorage()->m_sparseMap)
154  visitor.append(&newButterfly->arrayStorage()->m_sparseMap);
155149 break;
156150 }
157151 default:

@@ALWAYS_INLINE void JSObject::visitButter
162156 break;
163157 }
164158
165  while (count--) {
166  JSValue value = (currentSource++)->get();
167  if (value)
168  visitor.appendUnbarrieredValue(&value);
169  (currentTarget++)->setWithoutWriteBarrier(value);
170  }
 159 while (count--)
 160 (currentTarget++)->setWithoutWriteBarrier((currentSource++)->get());
171161 }
172162
173163 m_butterfly = newButterfly;
 164 visitor.didCopy(butterfly->base(preCapacity, propertyCapacity), capacityInBytes);
 165 }
 166}
 167
 168ALWAYS_INLINE void JSObject::visitButterfly(SlotVisitor& visitor, Butterfly* butterfly, size_t storageSize)
 169{
 170 ASSERT(butterfly);
 171
 172 Structure* structure = this->structure();
 173
 174 size_t propertyCapacity = structure->outOfLineCapacity();
 175 size_t preCapacity;
 176 size_t indexingPayloadSizeInBytes;
 177 bool hasIndexingHeader = JSC::hasIndexingHeader(structure->indexingType());
 178 if (UNLIKELY(hasIndexingHeader)) {
 179 preCapacity = butterfly->indexingHeader()->preCapacity(structure);
 180 indexingPayloadSizeInBytes = butterfly->indexingHeader()->indexingPayloadSizeInBytes(structure);
174181 } else {
175  // Mark the properties.
176  visitor.appendValues(butterfly->propertyStorage() - storageSize, storageSize);
177 
178  // Mark the array if appropriate.
179  switch (structure->indexingType()) {
180  case ALL_CONTIGUOUS_INDEXING_TYPES:
181  visitor.appendValues(butterfly->contiguous(), butterfly->publicLength());
182  break;
183  case ALL_ARRAY_STORAGE_INDEXING_TYPES:
184  visitor.appendValues(butterfly->arrayStorage()->m_vector, butterfly->arrayStorage()->vectorLength());
185  if (butterfly->arrayStorage()->m_sparseMap)
186  visitor.append(&butterfly->arrayStorage()->m_sparseMap);
187  break;
188  default:
189  break;
190  }
 182 preCapacity = 0;
 183 indexingPayloadSizeInBytes = 0;
 184 }
 185 size_t capacityInBytes = Butterfly::totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
 186
 187 // Mark the properties.
 188 visitor.appendValues(butterfly->propertyStorage() - storageSize, storageSize);
 189 visitor.copyLater(butterfly->base(preCapacity, propertyCapacity), capacityInBytes);
 190
 191 // Mark the array if appropriate.
 192 switch (structure->indexingType()) {
 193 case ALL_CONTIGUOUS_INDEXING_TYPES:
 194 visitor.appendValues(butterfly->contiguous(), butterfly->publicLength());
 195 break;
 196 case ALL_ARRAY_STORAGE_INDEXING_TYPES:
 197 visitor.appendValues(butterfly->arrayStorage()->m_vector, butterfly->arrayStorage()->vectorLength());
 198 if (butterfly->arrayStorage()->m_sparseMap)
 199 visitor.append(&butterfly->arrayStorage()->m_sparseMap);
 200 break;
 201 default:
 202 break;
191203 }
192204}
193205

@@void JSObject::visitChildren(JSCell* cel
211223#endif
212224}
213225
 226void JSObject::copyBackingStore(JSCell* cell, CopyVisitor& visitor)
 227{
 228 JSObject* thisObject = jsCast<JSObject*>(cell);
 229 ASSERT_GC_OBJECT_INHERITS(thisObject, &s_info);
 230
 231 Butterfly* butterfly = thisObject->butterfly();
 232 if (butterfly)
 233 thisObject->copyButterfly(visitor, butterfly, thisObject->structure()->outOfLineSize());
 234}
 235
214236void JSFinalObject::visitChildren(JSCell* cell, SlotVisitor& visitor)
215237{
216238 JSFinalObject* thisObject = jsCast<JSFinalObject*>(cell);
130954

Source/JavaScriptCore/runtime/JSObject.h

@@namespace JSC {
110110 typedef JSCell Base;
111111
112112 JS_EXPORT_PRIVATE static void visitChildren(JSCell*, SlotVisitor&);
 113 JS_EXPORT_PRIVATE static void copyBackingStore(JSCell*, CopyVisitor&);
113114
114115 JS_EXPORT_PRIVATE static String className(const JSObject*);
115116

@@namespace JSC {
634635 void resetInheritorID(JSGlobalData&);
635636
636637 void visitButterfly(SlotVisitor&, Butterfly*, size_t storageSize);
 638 void copyButterfly(CopyVisitor&, Butterfly*, size_t storageSize);
637639
638640 // Call this if you know that the object is in a mode where it has array
639641 // storage. This will assert otherwise.

@@inline JSValue JSObject::prototype() con
956958 return structure()->storedPrototype();
957959}
958960
959 inline bool JSCell::inherits(const ClassInfo* info) const
 961inline const MethodTable* JSCell::methodTable() const
960962{
961  return classInfo()->isSubClassOf(info);
 963 return &classInfo()->methodTable;
962964}
963965
964 inline const MethodTable* JSCell::methodTable() const
 966inline bool JSCell::inherits(const ClassInfo* info) const
965967{
966  return &classInfo()->methodTable;
 968 return classInfo()->isSubClassOf(info);
967969}
968970
969971// this method is here to be after the inline declaration of JSCell::inherits
130954

Source/JavaScriptCore/runtime/Options.h

@@namespace JSC {
116116 v(unsigned, gcMarkStackSegmentSize, pageSize()) \
117117 v(unsigned, numberOfGCMarkers, computeNumberOfGCMarkers(7)) \
118118 v(unsigned, opaqueRootMergeThreshold, 1000) \
 119 v(double, maxCopiedSpaceFragmentation, 0.8) \
 120 v(double, maxCopiedBlockFragmentation, 0.9) \
119121 \
120122 v(bool, forceWeakRandomSeed, false) \
121123 v(unsigned, forcedWeakRandomSeed, 0) \
130954