219232 2020-11-21 00:44:27 -0800 [GLib] Leaked RunLoop objects on worker threads 2021-02-21 17:01:22 -0800 1 1 1 Unclassified WebKit New Bugs WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=219591 https://bugs.webkit.org/show_bug.cgi?id=219593 https://bugs.webkit.org/show_bug.cgi?id=221115 InRadar P2 Normal --- 1 zan zan aperez benjamin cdumez cgarcia cmarcelo ews-watchlist fujii ggaren webkit-bug-importer ysuzuki oldest_to_newest 1709743 0 zan 2020-11-21 00:44:27 -0800 On JS worker threads, the thread-specific RunLoop object remains alive, along with all its resources, even after the thread is destroyed. I've only observed this with WPE on JetStream2.0, subtest bomb-workers. For GLib implementation of RunLoop, the internal GMainContext manages an eventfd file descriptor. With large amounts of workers and corresponding RunLoops, these file descriptors grow in number until they hit the user limit for the number of allowed file descriptors, at which point the executable is terminated. It could still be a problem on other ports, but like said, I haven't tested there. Once the worker thread is terminated, the thread-specific RunLoop object outlives that thread with one reference remaining. This reference is stuck in a RunLoop::Timer object that is spawned in the per-VM data for JSRunLoopTimer. When the owner VM is being unregistered and the per-VM data destroyed, the unique_ptr owning the timer is taken and pushed into the RunLoop's dispatch queue in order to be destroyed on the thread it is associated with. At that point, the thread is already being shut down, and the dispatch of the clean-up functor is never performed, leaving the RunLoop::Timer dangling inside the queue owned by the RunLoop instance, which never gets destroyed because of the remaining reference on the RunLoop::Timer. https://trac.webkit.org/browser/webkit/trunk/Source/JavaScriptCore/runtime/JSRunLoopTimer.cpp#L61 1709744 1 414759 zan 2020-11-21 00:47:47 -0800 Created attachment 414759 RunLoop::Holder cleanup One possible fix to this issue is explicitly clearing out the dispatch queues in RunLoop::Holder destructor, which is called when the TLS storage for this thread-specific RunLoop is being destroyed. 1710414 2 webkit-bug-importer 2020-11-28 00:45:18 -0800 <rdar://problem/71772277> 1710415 3 414975 zan 2020-11-28 01:16:43 -0800 Created attachment 414975 Patch 1710685 4 414975 ggaren 2020-11-30 11:13:40 -0800 Comment on attachment 414975 Patch Nice catch! This kind of feels like the wrong level at which to fix this bug. I wonder if TimerBase can hold a weak pointer to RunLoop instead. 1710708 5 ggaren 2020-11-30 12:24:13 -0800 I guess we have a few related problems: 1. DispatchTimer retains RunLoop. If a thread exits before a scheduled dispatchAfter fires, we leak. 2. RunLoop::dispatch() lambdas may retain anything. If a thread exits before a dispatch() lambda dequeues, we leak. 3. The function body of a timer or dispatch may be responsible for releasing something. If a thread exits before a function body executes, we leak. (This is the general case of #1.) For #1, maybe RunLoop should keep an explicit retained set of DispatchTimers waiting to fire (instead of DispatchTimer owning itself). dispatchAfter() would add to the set, and firing the timer would remove from the set. That way, if the RunLoop is destroyed, all DispatchTimers in the set are freed. For #2, I guess it's right to clear the queues. But can move that clearing into a member function named something like RunLoop::threadWillExit()? I'm torn about whether we should clear the queues or ASSERT that they're already empty. See below. For #3, maybe there's nothing we can do about it, and it's just programmer error to allow a thread to exit when it has more work to do. 1711038 6 zan 2020-12-01 08:16:40 -0800 I think DispatchTimer problem would be resolved if queues were emptied out upon thread destruction and the DispatchTimer ownership was tied to the lambda lifetime and not its execution. I'm not sure how much people could be blamed for dispatches still pending at the point of thread determination. It's certainly problematic if the queued dispatches were supposed to manually release resources that otherwise weren't tied to the dispatch functor lifetime. There could be a helper class made available, instances of which could be included in lambdas and would have to be manually marked upon dispatch, and asserting upon destruction of those instances if marking was not done. 1711041 7 zan 2020-12-01 08:19:18 -0800 (In reply to Geoffrey Garen from comment #4) > Comment on attachment 414975 [details] > Patch > > Nice catch! > > This kind of feels like the wrong level at which to fix this bug. > > I wonder if TimerBase can hold a weak pointer to RunLoop instead. A WeakPtr would work for the GLib implementation. Beyond the TimerBase constructor, the only place where RunLoop is used is during the dispatch of the RunLoop::dispatch() source, which wouldn't happen after the RunLoop is already destroyed. 1711058 8 ggaren 2020-12-01 09:28:37 -0800 Another option is to remove the m_runLoop member entirely from TimberBase, and instead require the caller to pass in a RunLoop when scheduling a timer into a RunLoop. That seems to be the way that the CoreFoundation API avoids a circular reference between timer and runloop. 1711064 9 ggaren 2020-12-01 09:32:23 -0800 > I think DispatchTimer problem would be resolved if queues were emptied out > upon thread destruction and the DispatchTimer ownership was tied to the > lambda lifetime and not its execution. I don't think emptying WTF::RunLoop queues is sufficient to address DispatchTimer because DispatchTimer is not in any WTF::RunLoop queue; instead it is registered with lower-level OS timer primitives (GSource in the Glib case). 1711292 10 zan 2020-12-02 03:21:45 -0800 (In reply to Geoffrey Garen from comment #9) > > I think DispatchTimer problem would be resolved if queues were emptied out > > upon thread destruction and the DispatchTimer ownership was tied to the > > lambda lifetime and not its execution. > > I don't think emptying WTF::RunLoop queues is sufficient to address > DispatchTimer because DispatchTimer is not in any WTF::RunLoop queue; > instead it is registered with lower-level OS timer primitives (GSource in > the Glib case). Yes, this is correct. I was wrong before. 1711628 11 415284 zan 2020-12-03 01:30:17 -0800 Created attachment 415284 Patch 1711629 12 zan 2020-12-03 01:31:56 -0800 (In reply to Zan Dobersek from comment #11) > Created attachment 415284 [details] > Patch This still only addresses the original bug, i.e. case #2. It now adds and uses RunLoop::threadWillExit(), and a unit test is provided. 1711651 13 zan 2020-12-03 04:48:46 -0800 Removing (In reply to Geoffrey Garen from comment #8) > Another option is to remove the m_runLoop member entirely from TimberBase, > and instead require the caller to pass in a RunLoop when scheduling a timer > into a RunLoop. That seems to be the way that the CoreFoundation API avoids > a circular reference between timer and runloop. At least for GLib, it could be possible to drop the RunLoop reference from RunLoop::TimerBase. We don't need it beyond the GSource creation in the constructor. 1712104 14 415284 ggaren 2020-12-04 12:05:16 -0800 Comment on attachment 415284 Patch r=me Would be nice to do a follow-up for timers. Still, this is clearly an improvement over the status quo. 1712393 15 zan 2020-12-07 01:02:29 -0800 I'll be opening separate bugs for the general dispatchAfter() leaks and the RunLoop ref removal from TimerBase. 1712399 16 ews-feeder 2020-12-07 01:19:56 -0800 Committed r270496: <https://trac.webkit.org/changeset/270496> All reviewed patches have been landed. Closing bug and clearing flags on attachment 415284. 1732014 17 415284 fujii 2021-02-21 17:01:22 -0800 Comment on attachment 415284 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=415284&action=review > Source/WTF/wtf/RunLoop.cpp:177 > + m_nextIteration.clear(); This causes a problem. Filed: Bug 221115 – RunLoop::threadWillExit is doing m_nextIteration.clear() without locking m_nextIterationLock 414759 2020-11-21 00:47:47 -0800 2020-11-28 01:16:36 -0800 RunLoop::Holder cleanup holder.patch text/plain 433 zan ZGlmZiAtLWdpdCBhL1NvdXJjZS9XVEYvd3RmL1J1bkxvb3AuY3BwIGIvU291cmNlL1dURi93dGYv UnVuTG9vcC5jcHAKaW5kZXggYzg1OGExMTZmZTJkLi4xZjBkZTQxOWZmMDUgMTAwNjQ0Ci0tLSBh L1NvdXJjZS9XVEYvd3RmL1J1bkxvb3AuY3BwCisrKyBiL1NvdXJjZS9XVEYvd3RmL1J1bkxvb3Au Y3BwCkBAIC00NSw2ICs0NSwxMSBAQCBwdWJsaWM6CiAgICAgICAgIDogbV9ydW5Mb29wKGFkb3B0 UmVmKCpuZXcgUnVuTG9vcCkpCiAgICAgewogICAgIH0KKyAgICB+SG9sZGVyKCkKKyAgICB7Cisg ICAgICAgIG1fcnVuTG9vcC0+bV9jdXJyZW50SXRlcmF0aW9uLmNsZWFyKCk7CisgICAgICAgIG1f cnVuTG9vcC0+bV9uZXh0SXRlcmF0aW9uLmNsZWFyKCk7CisgICAgfQogCiAgICAgUnVuTG9vcCYg cnVuTG9vcCgpIHsgcmV0dXJuIG1fcnVuTG9vcDsgfQogCg== 414975 2020-11-28 01:16:43 -0800 2020-12-03 01:30:13 -0800 Patch bug-219232-20201128101642.patch text/plain 1816 zan U3VidmVyc2lvbiBSZXZpc2lvbjogMjcwMjE1CmRpZmYgLS1naXQgYS9Tb3VyY2UvV1RGL0NoYW5n ZUxvZyBiL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCmluZGV4IDJiNDhhY2UyZWU1ODcyY2UyZTZjZmVk YjMyNmRmZDc3ZjhlZGJhMDIuLmU3MTJmY2JmMjA2NmJkOTgzMzRjNDY2ZDFkM2Q3Nzk1OTYwMTE5 YmMgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCisrKyBiL1NvdXJjZS9XVEYvQ2hh bmdlTG9nCkBAIC0xLDMgKzEsMjUgQEAKKzIwMjAtMTEtMjggIFphbiBEb2JlcnNlayAgPHpkb2Jl cnNla0BpZ2FsaWEuY29tPgorCisgICAgICAgIFtHTGliXSBMZWFrZWQgUnVuTG9vcCBvYmplY3Rz IG9uIHdvcmtlciB0aHJlYWRzCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3df YnVnLmNnaT9pZD0yMTkyMzIKKyAgICAgICAgPHJkYXI6Ly9wcm9ibGVtLzcxNzcyMjc3PgorCisg ICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIER1cmluZyB0aGUg dGhyZWFkLWxvY2FsIFJ1bkxvb3A6OkhvbGRlciBkZXN0cnVjdGlvbiwgZXhwbGljaXRseSBjbGVh ciBvdXQKKyAgICAgICAgdGhlIGl0ZXJhdGlvbiBEZXF1ZSBvYmplY3RzLCBkZXN0cm95aW5nIGFu eSBGdW5jdGlvbiBvYmplY3RzIHRoYXQgbmV2ZXIKKyAgICAgICAgZ290IHRvIGV4ZWN1dGUgb24g dGhpcyB0aHJlYWQuCisKKyAgICAgICAgVGhpcyB3b3VsZCBhbGxvdyBmb3IgYWN0dWFsIGRlc3Ry dWN0aW9uIG9mIHRoZSBSdW5Mb29wOjpUaW1lciBvYmplY3QgdGhhdCdzCisgICAgICAgIHF1ZXVl ZCB1cCBpbiB0aGUgSlNSdW5Mb29wVGltZXI6Ok1hbmFnZXI6OlBlclZNRGF0YSBkZXN0cnVjdG9y IGJ1dCBuZXZlcgorICAgICAgICBnZXRzIGRpc3BhdGNoZWQgYmVjYXVzZSB0aGUgdGhyZWFkIChh IEpTIHdvcmtlcikgaXMgc2h1dCBkb3duIGJlZm9yZSB0aGF0CisgICAgICAgIGhhcHBlbnMuIERl c3RydWN0aW9uIG9mIHRoZSB0aW1lciB3aWxsIHJlbGVhc2UgdGhlIHJlZmVyZW5jZSBvZiB0aGUg UnVuTG9vcAorICAgICAgICB0aGF0J3MgaGVsZCBieSB0aGUgUnVuTG9vcDo6SG9sZGVyLCBmaW5h bGx5IGVuYWJsaW5nIHRoZSBSdW5Mb29wIG9iamVjdAorICAgICAgICBpdHNlbGYgYmUgZGVzdHJv eWVkIG9uY2UgdGhlIFJ1bkxvb3A6OkhvbGRlciByZWZlcmVuY2UgaXMgbGV0IGdvLgorCisgICAg ICAgICogd3RmL1J1bkxvb3AuY3BwOgorICAgICAgICAoV1RGOjpSdW5Mb29wOjpIb2xkZXI6On5I b2xkZXIpOgorCiAyMDIwLTExLTI2ICBGdWppaSBIaXJvbm9yaSAgPEhpcm9ub3JpLkZ1amlpQHNv bnkuY29tPgogCiAgICAgICAgIFtXaW5DYWlyb10gRW5hYmxlIEdQVSBwcm9jZXNzCmRpZmYgLS1n aXQgYS9Tb3VyY2UvV1RGL3d0Zi9SdW5Mb29wLmNwcCBiL1NvdXJjZS9XVEYvd3RmL1J1bkxvb3Au Y3BwCmluZGV4IGM4NThhMTE2ZmUyZDQ0YTZlZTE1YWQ1YzdlZjUzZDcwMGRmMGI1ZDQuLmQ3MWMy MzlkYmQ4YTQ4MjBjYzY5NDEzZDczZjliZjBmZjk2NDVlNDUgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9X VEYvd3RmL1J1bkxvb3AuY3BwCisrKyBiL1NvdXJjZS9XVEYvd3RmL1J1bkxvb3AuY3BwCkBAIC00 Niw2ICs0NiwxMiBAQCBwdWJsaWM6CiAgICAgewogICAgIH0KIAorICAgIH5Ib2xkZXIoKQorICAg IHsKKyAgICAgICAgbV9ydW5Mb29wLT5tX2N1cnJlbnRJdGVyYXRpb24uY2xlYXIoKTsKKyAgICAg ICAgbV9ydW5Mb29wLT5tX25leHRJdGVyYXRpb24uY2xlYXIoKTsKKyAgICB9CisKICAgICBSdW5M b29wJiBydW5Mb29wKCkgeyByZXR1cm4gbV9ydW5Mb29wOyB9CiAKIHByaXZhdGU6Cg== 415284 2020-12-03 01:30:17 -0800 2020-12-07 01:19:57 -0800 Patch bug-219232-20201203103016.patch text/plain 4682 zan U3VidmVyc2lvbiBSZXZpc2lvbjogMjcwMzg3CmRpZmYgLS1naXQgYS9Tb3VyY2UvV1RGL0NoYW5n ZUxvZyBiL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCmluZGV4IDdlOTRlYmY1ZGNkN2QzNjZiNmZkOGI3 YjAwYmJmMjY3NTkwODY4MWIuLjY2Y2JlYjJlNmRmZmFlNWQ5NDE5MmQ1ZWRkZTRiNjVmYzA2Y2Vk NDkgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCisrKyBiL1NvdXJjZS9XVEYvQ2hh bmdlTG9nCkBAIC0xLDMgKzEsMjkgQEAKKzIwMjAtMTItMDMgIFphbiBEb2JlcnNlayAgPHpkb2Jl cnNla0BpZ2FsaWEuY29tPgorCisgICAgICAgIFtHTGliXSBMZWFrZWQgUnVuTG9vcCBvYmplY3Rz IG9uIHdvcmtlciB0aHJlYWRzCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3df YnVnLmNnaT9pZD0yMTkyMzIKKyAgICAgICAgPHJkYXI6Ly9wcm9ibGVtLzcxNzcyMjc3PgorCisg ICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIER1cmluZyB0aGUg dGhyZWFkLWxvY2FsIFJ1bkxvb3A6OkhvbGRlciBkZXN0cnVjdGlvbiwgZXhwbGljaXRseSBjbGVh ciBvdXQKKyAgICAgICAgdGhlIGl0ZXJhdGlvbiBEZXF1ZSBvYmplY3RzIG9uIHRoZSBoZWxkIFJ1 bkxvb3AsIGRlc3Ryb3lpbmcgYW55IEZ1bmN0aW9uCisgICAgICAgIG9iamVjdHMgdGhhdCBuZXZl ciBnb3QgdG8gZXhlY3V0ZSBvbiB0aGlzIHRocmVhZC4gR2VuZXJhbGx5LCB0aGlzIGFsbG93cwor ICAgICAgICBmb3IgYW55IFJ1bkxvb3AgcmVmZXJlbmNlIHN0b3JlZCBpbiB0aGVzZSBvYmplY3Rz IHRvIGJlIHJlbGVhc2VkLgorCisgICAgICAgIFNwZWNpZmljYWxseSwgdGhpcyB3b3VsZCBhbGxv dyBmb3IgZGVzdHJ1Y3Rpb24gb2YgdGhlIFJ1bkxvb3A6OlRpbWVyCisgICAgICAgIG9iamVjdCB0 aGF0J3MgcXVldWVkIHVwIGluIHRoZSBKU1J1bkxvb3BUaW1lcjo6TWFuYWdlcjo6UGVyVk1EYXRh CisgICAgICAgIGRlc3RydWN0b3IgYnV0IG5ldmVyIGdldHMgZGlzcGF0Y2hlZCBiZWNhdXNlIHRo ZSB0aHJlYWQgKGEgSlMgd29ya2VyKSBpcworICAgICAgICBzaHV0IGRvd24gYmVmb3JlIHRoYXQg aGFwcGVucy4gRGVzdHJ1Y3Rpb24gb2YgdGhlIHRpbWVyIHdpbGwgcmVsZWFzZSB0aGUKKyAgICAg ICAgcmVmZXJlbmNlIG9mIHRoZSBSdW5Mb29wIHRoYXQncyBoZWxkIGJ5IHRoZSBSdW5Mb29wOjpI b2xkZXIsIGZpbmFsbHkKKyAgICAgICAgZW5hYmxpbmcgdGhlIFJ1bkxvb3Agb2JqZWN0IGl0c2Vs ZiBiZSBkZXN0cm95ZWQgb25jZSB0aGUgUnVuTG9vcDo6SG9sZGVyCisgICAgICAgIHJlZmVyZW5j ZSBpcyBsZXQgZ28uCisKKyAgICAgICAgKiB3dGYvUnVuTG9vcC5jcHA6CisgICAgICAgIChXVEY6 OlJ1bkxvb3A6OkhvbGRlcjo6fkhvbGRlcik6CisgICAgICAgIChXVEY6OlJ1bkxvb3A6OnRocmVh ZFdpbGxFeGl0KToKKyAgICAgICAgKiB3dGYvUnVuTG9vcC5oOgorCiAyMDIwLTEyLTAyICBNaWNo YWVsIENhdGFuemFybyAgPG1jYXRhbnphcm9AZ25vbWUub3JnPgogCiAgICAgICAgIGFhcmNoNjQg bGxpbnQgZG9lcyBub3QgYnVpbGQgd2l0aCBKSVQgZGlzYWJsZWQKZGlmZiAtLWdpdCBhL1NvdXJj ZS9XVEYvd3RmL1J1bkxvb3AuY3BwIGIvU291cmNlL1dURi93dGYvUnVuTG9vcC5jcHAKaW5kZXgg Yzg1OGExMTZmZTJkNDRhNmVlMTVhZDVjN2VmNTNkNzAwZGYwYjVkNC4uNmZmOGYzNGU0NTE2Zjhj ZTg5ZWRlODkxZjRhZjE3NmFmNGE1YzIxZiAxMDA2NDQKLS0tIGEvU291cmNlL1dURi93dGYvUnVu TG9vcC5jcHAKKysrIGIvU291cmNlL1dURi93dGYvUnVuTG9vcC5jcHAKQEAgLTQ2LDYgKzQ2LDEx IEBAIHB1YmxpYzoKICAgICB7CiAgICAgfQogCisgICAgfkhvbGRlcigpCisgICAgeworICAgICAg ICBtX3J1bkxvb3AtPnRocmVhZFdpbGxFeGl0KCk7CisgICAgfQorCiAgICAgUnVuTG9vcCYgcnVu TG9vcCgpIHsgcmV0dXJuIG1fcnVuTG9vcDsgfQogCiBwcml2YXRlOgpAQCAtMTY2LDQgKzE3MSwx MCBAQCB2b2lkIFJ1bkxvb3A6OnN1c3BlbmRGdW5jdGlvbkRpc3BhdGNoRm9yQ3VycmVudEN5Y2xl KCkKICAgICB3YWtlVXAoKTsKIH0KIAordm9pZCBSdW5Mb29wOjp0aHJlYWRXaWxsRXhpdCgpCit7 CisgICAgbV9jdXJyZW50SXRlcmF0aW9uLmNsZWFyKCk7CisgICAgbV9uZXh0SXRlcmF0aW9uLmNs ZWFyKCk7Cit9CisKIH0gLy8gbmFtZXNwYWNlIFdURgpkaWZmIC0tZ2l0IGEvU291cmNlL1dURi93 dGYvUnVuTG9vcC5oIGIvU291cmNlL1dURi93dGYvUnVuTG9vcC5oCmluZGV4IDFmODhiMjI3Mjll Mjg0MjcxNjg3OWVmNjU0NmFiZjJiMjFiNmRiMjEuLjA0NjZkNDQ1OWQ2Yjc0YmNjMWNlNGQ4NDIw MzczNjg3MjYwN2NhMzQgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XVEYvd3RmL1J1bkxvb3AuaAorKysg Yi9Tb3VyY2UvV1RGL3d0Zi9SdW5Mb29wLmgKQEAgLTk3LDYgKzk3LDggQEAgcHVibGljOgogICAg IGVudW0gY2xhc3MgQ3ljbGVSZXN1bHQgeyBDb250aW51ZSwgU3RvcCB9OwogICAgIFdURl9FWFBP UlRfUFJJVkFURSBDeWNsZVJlc3VsdCBzdGF0aWMgY3ljbGUoUnVuTG9vcE1vZGUgPSBEZWZhdWx0 UnVuTG9vcE1vZGUpOwogCisgICAgV1RGX0VYUE9SVF9QUklWQVRFIHZvaWQgdGhyZWFkV2lsbEV4 aXQoKTsKKwogI2lmIFVTRShHTElCX0VWRU5UX0xPT1ApCiAgICAgV1RGX0VYUE9SVF9QUklWQVRF IEdNYWluQ29udGV4dCogbWFpbkNvbnRleHQoKSBjb25zdCB7IHJldHVybiBtX21haW5Db250ZXh0 LmdldCgpOyB9CiAgICAgZW51bSBjbGFzcyBFdmVudCB7IFdpbGxEaXNwYXRjaCwgRGlkRGlzcGF0 Y2ggfTsKZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBiL1Rvb2xzL0NoYW5nZUxvZwppbmRl eCBjNWU2OTZhNmJmYmEzYzdmYmVjMjM0MGY5ZmIzNThhZTkzZjQwOGZjLi41NjNkYzMxNTMwYzg2 ZDZkM2QyNzM2NGQ4OWFiYWNhZWZmMWU4MTY1IDEwMDY0NAotLS0gYS9Ub29scy9DaGFuZ2VMb2cK KysrIGIvVG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTcgQEAKKzIwMjAtMTItMDMgIFphbiBE b2JlcnNlayAgPHpkb2JlcnNla0BpZ2FsaWEuY29tPgorCisgICAgICAgIFtHTGliXSBMZWFrZWQg UnVuTG9vcCBvYmplY3RzIG9uIHdvcmtlciB0aHJlYWRzCisgICAgICAgIGh0dHBzOi8vYnVncy53 ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMTkyMzIKKyAgICAgICAgPHJkYXI6Ly9wcm9ibGVt LzcxNzcyMjc3PgorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAg ICAgIEFkZCBhIHVuaXQgdGVzdCBjb3ZlcmluZyBwcm9wZXIgUnVuTG9vcCB0ZWFyZG93biB1cG9u IHRocmVhZCBkZXN0cnVjdGlvbgorICAgICAgICBldmVuIGlmIFJ1bkxvb3AgcmVmZXJlbmNlcyBh cmUgc3RvcmVkIGluIHRoZSBkaXNwYXRjaCBxdWV1ZXMuCisKKyAgICAgICAgKiBUZXN0V2ViS2l0 QVBJL1Rlc3RzL1dURi9SdW5Mb29wLmNwcDoKKyAgICAgICAgKFRlc3RXZWJLaXRBUEk6OlRFU1Qp OgorCiAyMDIwLTEyLTAyICBUaW0gSG9ydG9uICA8dGltb3RoeV9ob3J0b25AYXBwbGUuY29tPgog CiAgICAgICAgIE1hbnkgZGlmZmVyZW50IGFzc2VydGlvbiBmYWlsdXJlcyBvbiB0aGUgR1BVIHBy b2Nlc3MgYm90IGFmdGVyIHIyNzAzNjYKZGlmZiAtLWdpdCBhL1Rvb2xzL1Rlc3RXZWJLaXRBUEkv VGVzdHMvV1RGL1J1bkxvb3AuY3BwIGIvVG9vbHMvVGVzdFdlYktpdEFQSS9UZXN0cy9XVEYvUnVu TG9vcC5jcHAKaW5kZXggNTQxNjEwYTY4MTkyZDI0MmNiMjU0OTEzMDNkNWMwYTA2M2YyMDFiMy4u ZDM4OGJmODA1NzJmMDI4M2VmZmRhMDQ2OWRiMDQzNjlkOTliN2I1ZiAxMDA2NDQKLS0tIGEvVG9v bHMvVGVzdFdlYktpdEFQSS9UZXN0cy9XVEYvUnVuTG9vcC5jcHAKKysrIGIvVG9vbHMvVGVzdFdl YktpdEFQSS9UZXN0cy9XVEYvUnVuTG9vcC5jcHAKQEAgLTIxNSw0ICsyMTUsMjEgQEAgVEVTVChX VEZfUnVuTG9vcCwgTWFueVRpbWVzKQogICAgIH0pLT53YWl0Rm9yQ29tcGxldGlvbigpOwogfQog CitURVNUKFdURl9SdW5Mb29wLCBUaHJlYWRUZXJtaW5hdGlvblNlbGZSZWZlcmVuY2VDbGVhbnVw KQoreworICAgIFJlZlB0cjxSdW5Mb29wPiBydW5Mb29wOworCisgICAgVGhyZWFkOjpjcmVhdGUo IlJ1bkxvb3BUaHJlYWRUZXJtaW5hdGlvblNlbGZSZWZlcmVuY2VDbGVhbnVwIiwgWyZdIHsKKyAg ICAgICAgcnVuTG9vcCA9IG1ha2VSZWZQdHIoUnVuTG9vcDo6Y3VycmVudCgpKTsKKworICAgICAg ICAvLyBUaGlzIHN0b3JlcyBhIFJ1bkxvb3AgcmVmZXJlbmNlIGluIHRoZSBkaXNwYXRjaCBxdWV1 ZSB0aGF0IHdpbGwgbm90IGJlIHJlbGVhc2VkCisgICAgICAgIC8vIHZpYSB0aGUgdXN1YWwgZGlz cGF0Y2gsIGJ1dCBzaG91bGQgc3RpbGwgYmUgcmVsZWFzZWQgdXBvbiB0aHJlYWQgdGVybWluYXRp b24uCisgICAgICAgIC8vIEFmdGVyIHRoYXQsIHRoZSBvYnNlcnZpbmcgUmVmUHRyIHNob3VsZCBi ZSB0aGUgb25seSBvbmUgaG9sZGluZyBhIHJlZmVyZW5jZQorICAgICAgICAvLyB0byB0aGUgUnVu TG9vcCBvYmplY3QuCisgICAgICAgIHJ1bkxvb3AtPmRpc3BhdGNoKFtyZWYgPSBydW5Mb29wLmNv cHlSZWYoKV0geyB9KTsKKyAgICB9KS0+d2FpdEZvckNvbXBsZXRpb24oKTsKKworICAgIEVYUEVD VF9UUlVFKHJ1bkxvb3AtPmhhc09uZVJlZigpKTsKK30KKwogfSAvLyBuYW1lc3BhY2UgVGVzdFdl YktpdEFQSQo=