171159 2017-04-22 01:02:10 -0700 REGRESSION(r215638): [Linux] Several worker tests are crashing in Thread::signalHandlerSuspendResume after r215638 2017-04-23 09:45:20 -0700 1 1 1 Unclassified WebKit Web Template Framework WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=170976 Gtk, LayoutTestFailure, Regression P2 Normal --- 170976 1 cgarcia ysuzuki benjamin buildbot cdumez cgarcia clopez cmarcelo dbates fpizlo jfbastien keith_miller mark.lam mcatanzaro saam ysuzuki oldest_to_newest 1300160 0 cgarcia 2017-04-22 01:02:10 -0700 You can see backtraces in GTK+ bots: https://build.webkit.org/results/GTK Linux 64-bit Release (Tests)/r215663 (693)/results.html For example: https://build.webkit.org/results/GTK Linux 64-bit Release (Tests)/r215663 (693)/fast/workers/worker-document-leak-crash-log.txt 1300171 1 ysuzuki 2017-04-22 02:19:23 -0700 Super curious thing is that there is no user of calling Thread::suspend / Thread::resume. I'm still investigating. 1300172 2 ysuzuki 2017-04-22 02:23:04 -0700 I guess this is related to bug 170976. Before starting using signaling APIs, WTF in Linux uses SIGUSR2 to suspend and resume threads. I guess this suspend/resume handler is incorrectly routed from jscSignalHandler. Investigating. 1300173 3 ysuzuki 2017-04-22 02:25:16 -0700 We should use Signaling APIs to set up suspend / resume signals I guess. 1300176 4 ysuzuki 2017-04-22 02:28:00 -0700 OK, signaling API uses SIGUSR2 for there purpose, but it is already used in Linux to suspend and resume threads. 1300179 5 ysuzuki 2017-04-22 03:20:49 -0700 (In reply to Yusuke Suzuki from comment #4) > OK, signaling API uses SIGUSR2 for there purpose, but it is already used in > Linux to suspend and resume threads. suspend / resume is a bit tricky operation. We should use dedicated signal for this operation. Now SIGUSR2 is used by ThreadMessage. But SIGUSR1 becomes empty. We should use it instead. 1300182 6 307904 ysuzuki 2017-04-22 03:33:08 -0700 Created attachment 307904 Patch 1300183 7 307905 ysuzuki 2017-04-22 03:35:04 -0700 Created attachment 307905 Patch 1300187 8 307905 mcatanzaro 2017-04-22 06:28:07 -0700 Comment on attachment 307905 Patch I don't think it's possible to use UNIX signals for any purpose without introducing subtle race conditions that are extremely hard to detect or reason about. Is it possible to eventually eliminate our use of UNIX signals for thread suspension, or is there just no other way to accomplish this than by using signals? 1300191 9 ysuzuki 2017-04-22 06:54:12 -0700 (In reply to Michael Catanzaro from comment #8) > Comment on attachment 307905 [details] > Patch > > I don't think it's possible to use UNIX signals for any purpose without > introducing subtle race conditions that are extremely hard to detect or > reason about. Is it possible to eventually eliminate our use of UNIX signals > for thread suspension, or is there just no other way to accomplish this than > by using signals? If we want to have the identical mechanism (ability to suspend and resume threads), I believe we do not have a non-signal way in Linux. For non Linux env, sometimes an alternative is offered, like pthread_suspend_np. But it also has a problem that we cannot retrieve register sets from a suspended thread. If we do not want to have the current strong mechanism, the simplest way is something like safepoint approach. Maybe current CheckTrap op can be used. safepoint repeatedly appears in JIT code and JS mutators will park if the ccollector requires. Then, the stop the world op works. But to make it efficiently implemented, yet it involves SEGV signal handler :P Wihout it, I think it's not as efficient as is... 1300192 10 ysuzuki 2017-04-22 07:07:38 -0700 Committed r215665: <http://trac.webkit.org/changeset/215665> 1300332 11 fpizlo 2017-04-23 08:37:36 -0700 I have some thoughts: - MachineStackMarker should switch to using this new mechanism. You would use the functional message API to send a Function that scans the stack. We should test whether this is perf-neutral on macOS/iOS/etc, and if it is, then we should get rid of the thread_suspend/resume code and just use this. It's better if all OSes use the same approach because then we share test coverage, ideas, etc. We should only use different APIs on different OSes if it gives us some measurable progression. - You don't need the full CheckTraps mechanism, since that's more of a safepoint - you stop at a place where the JIT is able to vend additional information. We don't really need safepoints for stack scanning. One thread holds the JSLock, and that thread does need to be safepointed but it's already at a safepoint anytime the GC requests a stack scan. None of the other threads that MachineStackMarker looks at hold the JSLock and so they don't need to be safepointed. In fact, all we really need is a ragged safepoint, where the thread requesting scanning signals all of the threads and then waits for them to return with stack scan results. (In reply to Michael Catanzaro from comment #8) > Comment on attachment 307905 [details] > Patch > > I don't think it's possible to use UNIX signals for any purpose without > introducing subtle race conditions that are extremely hard to detect or > reason about. That's not true. Two sound uses are: - pthread_kill, as we are doing. - handling a signal that originated as a CPU trap, like SEGV, BUS, TRAP, ILL, FPE, etc. We use signal handling for those purposes. > Is it possible to eventually eliminate our use of UNIX signals > for thread suspension, or is there just no other way to accomplish this than > by using signals? It's not possible. 1300337 12 ysuzuki 2017-04-23 09:45:20 -0700 Interesting. (In reply to Filip Pizlo from comment #11) > I have some thoughts: > > - MachineStackMarker should switch to using this new mechanism. You would > use the functional message API to send a Function that scans the stack. We > should test whether this is perf-neutral on macOS/iOS/etc, and if it is, > then we should get rid of the thread_suspend/resume code and just use this. > It's better if all OSes use the same approach because then we share test > coverage, ideas, etc. We should only use different APIs on different OSes > if it gives us some measurable progression. From the point of cross-platform view, one obstacle is that ThreadMessage is not implemented in Windows since it does not have identical mechanism to pthread_kill. I'm not sure there is a good mechanism in Windows to implement such semantics. > - You don't need the full CheckTraps mechanism, since that's more of a > safepoint - you stop at a place where the JIT is able to vend additional > information. We don't really need safepoints for stack scanning. One > thread holds the JSLock, and that thread does need to be safepointed but > it's already at a safepoint anytime the GC requests a stack scan. None of > the other threads that MachineStackMarker looks at hold the JSLock and so > they don't need to be safepointed. In fact, all we really need is a ragged > safepoint, where the thread requesting scanning signals all of the threads > and then waits for them to return with stack scan results. Right, make sense. Mutator already has JSLock when performing GC, and other ones do not have JSLock for that VM. It sounds like similar to TLB shootdown implementation. One thread broadcasts the request for the stack scan, threads perform it and return the result. The requester waits for the results, then, all the threads start executing. > (In reply to Michael Catanzaro from comment #8) > > Comment on attachment 307905 [details] > > Patch > > > > I don't think it's possible to use UNIX signals for any purpose without > > introducing subtle race conditions that are extremely hard to detect or > > reason about. > > That's not true. Two sound uses are: > > - pthread_kill, as we are doing. > > - handling a signal that originated as a CPU trap, like SEGV, BUS, TRAP, > ILL, FPE, etc. > > We use signal handling for those purposes. > > > Is it possible to eventually eliminate our use of UNIX signals > > for thread suspension, or is there just no other way to accomplish this than > > by using signals? > > It's not possible. 307904 2017-04-22 03:33:08 -0700 2017-04-22 03:35:02 -0700 Patch bug-171159-20170422193307.patch text/plain 2734 ysuzuki U3VidmVyc2lvbiBSZXZpc2lvbjogMjE1NjY0CmRpZmYgLS1naXQgYS9Tb3VyY2UvV1RGL0NoYW5n ZUxvZyBiL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCmluZGV4IGM1ZTQyYTY4NzEzMDI0OTBjNWNjMjJi ZmFjNmJiM2ZjZTg0ZDY1MWEuLmVlMjI3NDgxNjZhMDJmOTcyM2UxMjRiNmQxZjE0NDcwYmRmM2Q3 OGMgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCisrKyBiL1NvdXJjZS9XVEYvQ2hh bmdlTG9nCkBAIC0xLDMgKzEsMjUgQEAKKzIwMTctMDQtMjIgIFl1c3VrZSBTdXp1a2kgIDx1dGF0 YW5lLnRlYUBnbWFpbC5jb20+CisKKyAgICAgICAgUkVHUkVTU0lPTihyMjE1NjM4KTogW0xpbnV4 XSBTZXZlcmFsIHdvcmtlciB0ZXN0cyBhcmUgY3Jhc2hpbmcgaW4gVGhyZWFkOjpzaWduYWxIYW5k bGVyU3VzcGVuZFJlc3VtZSBhZnRlciByMjE1NjM4CisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJr aXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xNzExNTkKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JP RFkgKE9PUFMhKS4KKworICAgICAgICBOb3csIFNJR1VTUjIgaXMgdXNlZCBieSBUaHJlYWRNZXNz YWdlLiBJbnN0ZWFkIFNJR1VTUjEgYmVjb21lcyBlbXB0eS4KKyAgICAgICAgV2Ugc2hvdWxkIHVz ZSBzb21lIHNpZ25hbCwgdGhhdCBpcyBub3QgbWFuYWdlZCBieSBXVEYgU2lnbmFsIG1lY2hhbmlz bQorICAgICAgICBiZWNhdXNlIHRoZSBiZWhhaXZvciBvZiB0aGlzIHN1c3BlbmQgYW5kIHJlc3Vt ZSBpcyBhIGJpdCB0cmlja3kuCisgICAgICAgIEZvciBleGFtcGxlLCB3ZSBzZXQgYSBiaXQgdHJp Y2t5IHNpZ25hbCBtYXNrIHRvIHNpZ2FjdGlvbiB0byB0ZW1wb3JhcnkKKyAgICAgICAgYmxvY2sg U0lHVVNSMiBpbiB0aGUgaGFuZGxlciB0byBhdm9pZCBuZXN0ZWQgU0lHVVNSMi4gQW5kIHdlIGNh bm5vdAorICAgICAgICB1c2UgVGhyZWFkTWVzc2FnZSB0byBpbXBsZW1lbnQgdGhpcyBtZWNoYW5p c20gYmVjYXVzZSB0aGlzIHN1c3BlbmQKKyAgICAgICAgYW5kIHJlc3VtZSBoYW5kbGVyIHdpbGwg c3RvcCBpbiB0aGUgbWlkZGxlIG9mIHRoZSBoYW5kbGVyIGJ5IHNpZ3N1c3BlbmQuCisgICAgICAg IEl0IGlzIG5vdCB0aGUgZXhwZWN0ZWQgc2VtYW50aWNzIGJ5IFRocmVhZE1lc3NhZ2UuCisKKyAg ICAgICAgU28gbm93LCB3ZSB1c2UgU0lHVVNSMSBpbnN0ZWFkLiBBbmQgd2Ugbm90ZSB0aGF0IFNJ R1VTUjEgaXMgdXNlZCBmb3IKKyAgICAgICAgcHRocmVhZCBwb3J0cy4KKworICAgICAgICAqIHd0 Zi9UaHJlYWRpbmdQdGhyZWFkcy5jcHA6CisgICAgICAgICogd3RmL3RocmVhZHMvU2lnbmFscy5o OgorCiAyMDE3LTA0LTIwICBTYW0gV2VpbmlnICA8c2FtQHdlYmtpdC5vcmc+CiAKICAgICAgICAg U3BsaXQgY3J5cHRvZ3JhcGhpYyBkaWdlc3QgY29tcHV0YXRpb24gYW5kIHBhcnNpbmcgb3V0IG9m IENTUCBjb2RlIHNvIGl0IGNhbiBiZSByZXVzZWQKZGlmZiAtLWdpdCBhL1NvdXJjZS9XVEYvd3Rm L1RocmVhZGluZ1B0aHJlYWRzLmNwcCBiL1NvdXJjZS9XVEYvd3RmL1RocmVhZGluZ1B0aHJlYWRz LmNwcAppbmRleCAxYTVkOTlmNjE0ZjVkOTVhY2Y2MTcxZWYwMWQwM2QxNGMyMWM3ZmQwLi4xMzc4 NDNjZTM0MDIwMWJmYjg1ZjcwODYzNTc1Y2VkZTBhNzY5MDE5IDEwMDY0NAotLS0gYS9Tb3VyY2Uv V1RGL3d0Zi9UaHJlYWRpbmdQdGhyZWFkcy5jcHAKKysrIGIvU291cmNlL1dURi93dGYvVGhyZWFk aW5nUHRocmVhZHMuY3BwCkBAIC04OSw4ICs4OSw4IEBAIFRocmVhZDo6flRocmVhZCgpCiAKICNp ZiAhT1MoREFSV0lOKQogCi0vLyBXZSB1c2UgU0lHVVNSMiB0byBzdXNwZW5kIGFuZCByZXN1bWUg bWFjaGluZSB0aHJlYWRzIGluIEphdmFTY3JpcHRDb3JlLgotc3RhdGljIGNvbnN0ZXhwciBjb25z dCBpbnQgU2lnVGhyZWFkU3VzcGVuZFJlc3VtZSA9IFNJR1VTUjI7CisvLyBXZSB1c2UgU0lHVVNS MSB0byBzdXNwZW5kIGFuZCByZXN1bWUgbWFjaGluZSB0aHJlYWRzIGluIEphdmFTY3JpcHRDb3Jl Lgorc3RhdGljIGNvbnN0ZXhwciBjb25zdCBpbnQgU2lnVGhyZWFkU3VzcGVuZFJlc3VtZSA9IFNJ R1VTUjE7CiBzdGF0aWMgc3RkOjphdG9taWM8VGhyZWFkKj4gdGFyZ2V0VGhyZWFkIHsgbnVsbHB0 ciB9Owogc3RhdGljIFN0YXRpY1dvcmRMb2NrIGdsb2JhbFN1c3BlbmRMb2NrOwogCmRpZmYgLS1n aXQgYS9Tb3VyY2UvV1RGL3d0Zi90aHJlYWRzL1NpZ25hbHMuaCBiL1NvdXJjZS9XVEYvd3RmL3Ro cmVhZHMvU2lnbmFscy5oCmluZGV4IDAzYTFkMmU1ODA4ODFiYWU5MzM5ZTE2NzgxOGE3YjllZjFi ODBmYzIuLjNmYmI0MDA2OGQ4NjgxZjgyMWIyMmZkZTc5M2Y2NGE5YTBhMzY0ZjYgMTAwNjQ0Ci0t LSBhL1NvdXJjZS9XVEYvd3RmL3RocmVhZHMvU2lnbmFscy5oCisrKyBiL1NvdXJjZS9XVEYvd3Rm L3RocmVhZHMvU2lnbmFscy5oCkBAIC0zMiw2ICszMiw3IEBACiAKIG5hbWVzcGFjZSBXVEYgewog CisvLyBOb3RlIHRoYXQgU0lHVVNSMSBpcyB1c2VkIGluIFB0aHJlYWQtYmFzZWQgcG9ydHMgb3Ro ZXIgdGhhbiBEYXJ3aW4gdG8gc3VzcGVuZCBhbmQgcmVzdW1lIHRocmVhZHMuCiBlbnVtIGNsYXNz IFNpZ25hbCB7CiAgICAgLy8gVXNyIHdpbGwgYWx3YXlzIGNoYWluIHRvIGFueSBub24tZGVmYXVs dCBoYW5kbGVyIGluc3RhbGwgYmVmb3JlIHVzLiBTaW5jZSB0aGVyZSBpcyBubyB3YXkgdG8ga25v dwogICAgIC8vIGlmIGEgc2lnbmFsIHdhcyBpbnRlbmRlZCBleGNsdXNpdmVseSBmb3IgdXMuCg== 307905 2017-04-22 03:35:04 -0700 2017-04-22 06:28:07 -0700 Patch bug-171159-20170422193503.patch text/plain 2734 ysuzuki U3VidmVyc2lvbiBSZXZpc2lvbjogMjE1NjY0CmRpZmYgLS1naXQgYS9Tb3VyY2UvV1RGL0NoYW5n ZUxvZyBiL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCmluZGV4IGM1ZTQyYTY4NzEzMDI0OTBjNWNjMjJi ZmFjNmJiM2ZjZTg0ZDY1MWEuLmVlMjI3NDgxNjZhMDJmOTcyM2UxMjRiNmQxZjE0NDcwYmRmM2Q3 OGMgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCisrKyBiL1NvdXJjZS9XVEYvQ2hh bmdlTG9nCkBAIC0xLDMgKzEsMjUgQEAKKzIwMTctMDQtMjIgIFl1c3VrZSBTdXp1a2kgIDx1dGF0 YW5lLnRlYUBnbWFpbC5jb20+CisKKyAgICAgICAgUkVHUkVTU0lPTihyMjE1NjM4KTogW0xpbnV4 XSBTZXZlcmFsIHdvcmtlciB0ZXN0cyBhcmUgY3Jhc2hpbmcgaW4gVGhyZWFkOjpzaWduYWxIYW5k bGVyU3VzcGVuZFJlc3VtZSBhZnRlciByMjE1NjM4CisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJr aXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xNzExNTkKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JP RFkgKE9PUFMhKS4KKworICAgICAgICBOb3csIFNJR1VTUjIgaXMgdXNlZCBieSBUaHJlYWRNZXNz YWdlLiBJbnN0ZWFkIFNJR1VTUjEgYmVjb21lcyBlbXB0eS4KKyAgICAgICAgV2Ugc2hvdWxkIHVz ZSBzb21lIHNpZ25hbCwgdGhhdCBpcyBub3QgbWFuYWdlZCBieSBXVEYgU2lnbmFsIG1lY2hhbmlz bQorICAgICAgICBiZWNhdXNlIHRoZSBiZWhhaXZvciBvZiB0aGlzIHN1c3BlbmQgYW5kIHJlc3Vt ZSBpcyBhIGJpdCB0cmlja3kuCisgICAgICAgIEZvciBleGFtcGxlLCB3ZSBzZXQgYSBiaXQgdHJp Y2t5IHNpZ25hbCBtYXNrIHRvIHNpZ2FjdGlvbiB0byB0ZW1wb3JhcnkKKyAgICAgICAgYmxvY2sg U0lHVVNSMiBpbiB0aGUgaGFuZGxlciB0byBhdm9pZCBuZXN0ZWQgU0lHVVNSMi4gQW5kIHdlIGNh bm5vdAorICAgICAgICB1c2UgVGhyZWFkTWVzc2FnZSB0byBpbXBsZW1lbnQgdGhpcyBtZWNoYW5p c20gYmVjYXVzZSB0aGlzIHN1c3BlbmQKKyAgICAgICAgYW5kIHJlc3VtZSBoYW5kbGVyIHdpbGwg c3RvcCBpbiB0aGUgbWlkZGxlIG9mIHRoZSBoYW5kbGVyIGJ5IHNpZ3N1c3BlbmQuCisgICAgICAg IEl0IGlzIG5vdCB0aGUgZXhwZWN0ZWQgc2VtYW50aWNzIGJ5IFRocmVhZE1lc3NhZ2UuCisKKyAg ICAgICAgU28gbm93LCB3ZSB1c2UgU0lHVVNSMSBpbnN0ZWFkLiBBbmQgd2Ugbm90ZSB0aGF0IFNJ R1VTUjEgaXMgdXNlZCBmb3IKKyAgICAgICAgcHRocmVhZCBwb3J0cy4KKworICAgICAgICAqIHd0 Zi9UaHJlYWRpbmdQdGhyZWFkcy5jcHA6CisgICAgICAgICogd3RmL3RocmVhZHMvU2lnbmFscy5o OgorCiAyMDE3LTA0LTIwICBTYW0gV2VpbmlnICA8c2FtQHdlYmtpdC5vcmc+CiAKICAgICAgICAg U3BsaXQgY3J5cHRvZ3JhcGhpYyBkaWdlc3QgY29tcHV0YXRpb24gYW5kIHBhcnNpbmcgb3V0IG9m IENTUCBjb2RlIHNvIGl0IGNhbiBiZSByZXVzZWQKZGlmZiAtLWdpdCBhL1NvdXJjZS9XVEYvd3Rm L1RocmVhZGluZ1B0aHJlYWRzLmNwcCBiL1NvdXJjZS9XVEYvd3RmL1RocmVhZGluZ1B0aHJlYWRz LmNwcAppbmRleCAxYTVkOTlmNjE0ZjVkOTVhY2Y2MTcxZWYwMWQwM2QxNGMyMWM3ZmQwLi4xMzc4 NDNjZTM0MDIwMWJmYjg1ZjcwODYzNTc1Y2VkZTBhNzY5MDE5IDEwMDY0NAotLS0gYS9Tb3VyY2Uv V1RGL3d0Zi9UaHJlYWRpbmdQdGhyZWFkcy5jcHAKKysrIGIvU291cmNlL1dURi93dGYvVGhyZWFk aW5nUHRocmVhZHMuY3BwCkBAIC04OSw4ICs4OSw4IEBAIFRocmVhZDo6flRocmVhZCgpCiAKICNp ZiAhT1MoREFSV0lOKQogCi0vLyBXZSB1c2UgU0lHVVNSMiB0byBzdXNwZW5kIGFuZCByZXN1bWUg bWFjaGluZSB0aHJlYWRzIGluIEphdmFTY3JpcHRDb3JlLgotc3RhdGljIGNvbnN0ZXhwciBjb25z dCBpbnQgU2lnVGhyZWFkU3VzcGVuZFJlc3VtZSA9IFNJR1VTUjI7CisvLyBXZSB1c2UgU0lHVVNS MSB0byBzdXNwZW5kIGFuZCByZXN1bWUgbWFjaGluZSB0aHJlYWRzIGluIEphdmFTY3JpcHRDb3Jl Lgorc3RhdGljIGNvbnN0ZXhwciBjb25zdCBpbnQgU2lnVGhyZWFkU3VzcGVuZFJlc3VtZSA9IFNJ R1VTUjE7CiBzdGF0aWMgc3RkOjphdG9taWM8VGhyZWFkKj4gdGFyZ2V0VGhyZWFkIHsgbnVsbHB0 ciB9Owogc3RhdGljIFN0YXRpY1dvcmRMb2NrIGdsb2JhbFN1c3BlbmRMb2NrOwogCmRpZmYgLS1n aXQgYS9Tb3VyY2UvV1RGL3d0Zi90aHJlYWRzL1NpZ25hbHMuaCBiL1NvdXJjZS9XVEYvd3RmL3Ro cmVhZHMvU2lnbmFscy5oCmluZGV4IDAzYTFkMmU1ODA4ODFiYWU5MzM5ZTE2NzgxOGE3YjllZjFi ODBmYzIuLjFlNGViNDZlYWE3NzRhMWVlODlmNmZiMmRhZWYzMzI2OGNlNTY3NzkgMTAwNjQ0Ci0t LSBhL1NvdXJjZS9XVEYvd3RmL3RocmVhZHMvU2lnbmFscy5oCisrKyBiL1NvdXJjZS9XVEYvd3Rm L3RocmVhZHMvU2lnbmFscy5oCkBAIC0zMiw2ICszMiw3IEBACiAKIG5hbWVzcGFjZSBXVEYgewog CisvLyBOb3RlIHRoYXQgU0lHVVNSMSBpcyB1c2VkIGluIFB0aHJlYWQtYmFzZWQgcG9ydHMgZXhj ZXB0IGZvciBEYXJ3aW4gdG8gc3VzcGVuZCBhbmQgcmVzdW1lIHRocmVhZHMuCiBlbnVtIGNsYXNz IFNpZ25hbCB7CiAgICAgLy8gVXNyIHdpbGwgYWx3YXlzIGNoYWluIHRvIGFueSBub24tZGVmYXVs dCBoYW5kbGVyIGluc3RhbGwgYmVmb3JlIHVzLiBTaW5jZSB0aGVyZSBpcyBubyB3YXkgdG8ga25v dwogICAgIC8vIGlmIGEgc2lnbmFsIHdhcyBpbnRlbmRlZCBleGNsdXNpdmVseSBmb3IgdXMuCg==