167626 2017-01-30 19:31:40 -0800 REGRESSION(r182916) run-perf-tests never timeouts 2017-02-10 22:03:02 -0800 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 clopez clopez ap bugs-noreply cdumez commit-queue dbates glenn lforschler mcatanzaro ossy rniwa oldest_to_newest 1271553 0 clopez 2017-01-30 19:31:40 -0800 Not sure when this started happening, but it seems that the timeout feature of run-perf-tests is broken. The Script run-perf-tests accepts a parameter --time-out-ms with default value of 600 seconds (600 * 1000). But this parameter is completely ignored because in webkitpy/performance_tests/perftest.py the driver is created with no_timeout=True unconditionally. def _create_driver(self): return self._port.create_driver(worker_number=0, no_timeout=True) And this causes that the tests run without any timeout deadline: See Driver.run_test() at webkitpy/port/driver.py when self._no_timeout is true. So, I think we should: * Make the --time-out-ms parameter work back as expected. * Add a --no-timeout parameter for testing purposes. * Keep the default 600 second time out value for for each performance (sub)test. This means that: * Some tests on the perf bots may break if we do what I propose, as they are running now without any timeout value. But I think this tests should be fixed to not take so much time to run. * There is a general 1200 second timeout value for the whole perf-test step (command timed out: 1200 seconds without output), but when this triggers it kills the whole step rather than stopping only the running test. * If some port (Apple/EFL/GTK+) has any reason to want different defaults, they can override this for their own Perf bots in the master.cfg file at Tools/BuildSlaveSupport/build.webkit.org-config/ 1271556 1 300181 clopez 2017-01-30 19:49:43 -0800 Created attachment 300181 Patch 1271613 2 300181 rniwa 2017-01-31 00:36:52 -0800 Comment on attachment 300181 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=300181&action=review > Tools/ChangeLog:10 > + The --time-out-ms value (defaults to 600 seconds) of run-perf-tests > + was beeing ignored because the driver was created with no_timeout=True > + unconditionally. This is intentional. The timeout is enforced by Python code in _run_with_driver. In fact, if we don't use no timeout option, DRT/WTR would timeout after 10 minutes or so. 1271663 3 clopez 2017-01-31 04:24:40 -0800 (In reply to comment #2) > Comment on attachment 300181 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=300181&action=review > > > Tools/ChangeLog:10 > > + The --time-out-ms value (defaults to 600 seconds) of run-perf-tests > > + was beeing ignored because the driver was created with no_timeout=True > > + unconditionally. > > This is intentional. The timeout is enforced by Python code in > _run_with_driver. > In fact, if we don't use no timeout option, DRT/WTR would timeout after 10 > minutes or so. I don't think that is the case. I have been testing this and I'm pretty sure that the tests are all ran without any timeout value (actually a 7 days one). See a more detailed explanation of the issue below: At Tools/Scripts/webkitpy/performance_tests/perftest.py line 177 you have. def _run_with_driver(self, driver, time_out_ms): output = self.run_single(driver, self.test_path(), time_out_ms) So PerfTest.run_single() is effectively called with the timeout value. This is defined as: def run_single(self, driver, test_path, time_out_ms, should_run_pixel_test=False): return driver.run_test(DriverInput(test_path, time_out_ms, image_hash=None, should_run_pixel_test=should_run_pixel_test), stop_when_done=False) So it ends calling Driver.run_test(). And this function is defined as follows at Tools/Scripts/webkitpy/port/driver.py line 174 def run_test(self, driver_input, stop_when_done): """Run a single test and return the results. Note that it is okay if a test times out or crashes and leaves the driver in an indeterminate state. The upper layers of the program are responsible for cleaning up and ensuring things are okay. Returns a DriverOutput object. """ start_time = time.time() self.start(driver_input.should_run_pixel_test, driver_input.args) test_begin_time = time.time() self._driver_timed_out = False self._crash_report_from_driver = None self.error_from_test = str() self.err_seen_eof = False command = self._command_from_driver_input(driver_input) # Certain timeouts are detected by the tool itself; tool detection is better, # because results contain partial output in this case. Make script timeout longer # by 5 seconds to avoid racing for which timeout is detected first. # FIXME: It's not the job of the driver to decide what the timeouts should be. # Move the additional timeout to driver_input. if self._no_timeout: deadline = test_begin_time + 60 * 60 * 24 * 7 # 7 days. Using sys.maxint causes a hang. else: deadline = test_begin_time + int(driver_input.timeout) / 1000.0 + 5 [..... ] See the issue above? The driver class that is called was created at Tools/Scripts/webkitpy/performance_tests/perftest.py line 125 def _create_driver(self): return self._port.create_driver(worker_number=0, no_timeout=True) So Driver().run_test() will enter into this: if self._no_timeout: deadline = test_begin_time + 60 * 60 * 24 * 7 # 7 days. Using Causing the driver_input.timeout value to be ignored, and all the tests to run with no timeout (a 7 days one actually!) 1271672 4 clopez 2017-01-31 04:56:58 -0800 (In reply to comment #2) > In fact, if we don't use no timeout option, DRT/WTR would timeout after 10 > minutes or so. If that were the case, then I wonder how the following can be happening: 1) The test IndexedDB/large-binary-keys.html is taking 1048 seconds (~17 minutes) to run on the Apple perf bot: Apple perf bot https://build.webkit.org/builders/Apple%20El%20Capitan%20Release%20WK2%20%28Perf%29/builds/4138/steps/perf-test/logs/stdio [.............] Running IndexedDB/large-binary-keys.html (78 of 162) RESULT IndexedDB: large-binary-keys: Time= 12710.29875 ms median= 12767.6 ms, stdev= 192.085687956 ms, min= 12049.8 ms, max= 12873.7 ms RESULT IndexedDB: large-binary-keys: JSHeap= 690339155.65 bytes median= 663655063.0 bytes, stdev= 248329564.206 bytes, min= 248397774.0 bytes, max= 1206834295.0 bytes RESULT IndexedDB: large-binary-keys: Malloc= 847334707.2 bytes median= 872071168.0 bytes, stdev= 258591471.735 bytes, min= 369184768.0 bytes, max= 1255882752.0 bytes Finished: 1048.817416 s 2) In both the GTK+ and EFL perf bots, some of this IndexedDBs are causing the whole step to abort because the 1200 second timeout value for the whole perf-test step (command timed out: 1200 seconds without output) triggers before any individual perf test timeout? GTK perf bot https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Perf%29/builds/7761/steps/perf-test/logs/stdio [.............] Running IndexedDB/index-get.html (75 of 161) command timed out: 1200 seconds without output, killing pid 12302 process killed by signal 9 program finished with exit code -1 elapsedTime=7711.394962 EFL perf bot https://build.webkit.org/builders/EFL%20Linux%2064-bit%20Release%20WK2%20%28Perf%29/builds/11181/steps/perf-test/logs/stdio [.............] Running IndexedDB/large-number-of-inserts-responsiveness.html (79 of 162) command timed out: 1200 seconds without output running ['python', './Tools/Scripts/run-perf-tests', '--output-json-path', 'perf-test-results.json', '--slave-config-json-path', '../../perf-test-config.json', '--no-show-results', '--reset-results', '--test-results-server', 'perf.webkit.org', '--builder-name', 'EFL Linux 64-bit Release WK2 (Perf)', '--build-number', '11181', '--platform', 'efl', '--no-build', '--release'], attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=8274.328622 1271872 5 rniwa 2017-01-31 13:54:48 -0800 That got regressed in https://trac.webkit.org/changeset/182916. We need to differentiate the timeout in DRT/WTR from the timeout in the tool. I think https://trac.webkit.org/changeset/182916 is trying to disable both, but for perf testing, we need to only disable the watchdog timer in DRT/WTR, not the one in the tool. 1271884 6 ap 2017-01-31 14:07:55 -0800 To only disable DRT timers, --additional-drt-flag may be best. 1271890 7 rniwa 2017-01-31 14:16:43 -0800 (In reply to comment #6) > To only disable DRT timers, --additional-drt-flag may be best. Well, this is what run-perf-tests always uses but I suppose we can re-use the same facility used to add additional DRT flags. 1272206 8 clopez 2017-02-01 11:16:58 -0800 So... How do you propose we fix this? Is the patch I uploaded not good enough? 1272251 9 rniwa 2017-02-01 13:27:43 -0800 (In reply to comment #8) > How do you propose we fix this? We need to pass --no-timeout to DRT using the same facility as --additional-drt-flag's > Is the patch I uploaded not good enough? Unfortunately not. 1272277 10 clopez 2017-02-01 14:17:27 -0800 (In reply to comment #9) > (In reply to comment #8) > > How do you propose we fix this? > > We need to pass --no-timeout to DRT using the same facility as > --additional-drt-flag's > I'm confused. How passing --no-timeout to DRT will fix the issue? The issue is that the driver that runs the tests from run-perf-tests ignores the timeout value. Tools/Scripts/run-perf-tests --time-out-ms=${value} This ${value} defaults to 10 minutes if not specified, but Driver().run_test() ignores any ${value} because no_timeout=True is passed. 1272335 11 rniwa 2017-02-01 17:02:17 -0800 (In reply to comment #10) > (In reply to comment #9) > > (In reply to comment #8) > > > How do you propose we fix this? > > > > We need to pass --no-timeout to DRT using the same facility as > > --additional-drt-flag's > > > > I'm confused. > > How passing --no-timeout to DRT will fix the issue? > > The issue is that the driver that runs the tests from run-perf-tests ignores > the timeout value. What I'm saying is that we need to make DRT take --no-timeout, and then make Python code still use the timeout set to run-perf-tests. > This ${value} defaults to 10 minutes if not specified, but > Driver().run_test() ignores any ${value} because no_timeout=True is passed. We need to 1. fix this code, and 2. making sure DRT/WTR is ran with --no-timeout inside run-perf-tests. Your patch only does (1) and not (2). 1272995 12 300553 clopez 2017-02-03 11:51:19 -0800 Created attachment 300553 Patch 1275729 13 mcatanzaro 2017-02-10 20:06:53 -0800 Hi Ryosuke, does this look good? 1275750 14 300553 rniwa 2017-02-10 21:52:01 -0800 Comment on attachment 300553 Patch Yup, this looks correct. Thanks for the fix! 1275755 15 300553 commit-queue 2017-02-10 22:02:55 -0800 Comment on attachment 300553 Patch Clearing flags on attachment: 300553 Committed r212182: <http://trac.webkit.org/changeset/212182> 1275756 16 commit-queue 2017-02-10 22:03:02 -0800 All reviewed patches have been landed. Closing bug. 300181 2017-01-30 19:49:43 -0800 2017-02-03 11:50:43 -0800 Patch bug-167626-20170131044732.patch text/plain 3806 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjExMzgyCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggMzljYTdlYTdiZTk2MWE5NmU3MDQ3YjA4NzUyOTRhOTQ1 YWE3NzRhZS4uMTg0M2QxY2VkZmIyYTQ4NTNiZmNlMzg0ODNjZWQzNzAyYzlhNzFmZSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDIz IEBACisyMDE3LTAxLTMwICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIHJ1bi1wZXJmLXRlc3RzIHRpbWVvdXQgc2V0dGluZ3MgYnJva2Vu CisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xNjc2MjYK KworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBUaGUgLS10 aW1lLW91dC1tcyB2YWx1ZSAoZGVmYXVsdHMgdG8gNjAwIHNlY29uZHMpIG9mIHJ1bi1wZXJmLXRl c3RzCisgICAgICAgIHdhcyBiZWVpbmcgaWdub3JlZCBiZWNhdXNlIHRoZSBkcml2ZXIgd2FzIGNy ZWF0ZWQgd2l0aCBub190aW1lb3V0PVRydWUKKyAgICAgICAgdW5jb25kaXRpb25hbGx5LgorICAg ICAgICBUaGlzIGFkZHMgYSBuZXcgcGFyYW1ldGVyIC0tbm8tdGltZW91dCAoZGlzYWJsZWQgYnkg ZGVmYXVsdCksIHRoYXQKKyAgICAgICAgYWxsb3dzIHRvIHVzZSB0aGUgZHJpdmVyIHdpdGggdGhl IGV4cGVjdGVkIHRpbWVvdXQgdmFsdWVzLgorCisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9w ZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdC5weToKKyAgICAgICAgKFBlcmZUZXN0Ll9jcmVhdGVf ZHJpdmVyKTogT25seSBkaXNhYmxlIHRpbWVvdXQgaWYgLS1uby10aW1lb3V0IGlzIHBhc3NlZC4K KyAgICAgICAgKFBlcmZUZXN0LnJ1bik6CisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9wZXJm b3JtYW5jZV90ZXN0cy9wZXJmdGVzdHNydW5uZXIucHk6CisgICAgICAgIChQZXJmVGVzdHNSdW5u ZXIuX3BhcnNlX2FyZ3MpOiBBZGQgYSAtLW5vLXRpbWVvdXQgcGFyYW1ldGVyLgorICAgICAgICAo X3J1bl90ZXN0c19zZXQpOgorCiAyMDE3LTAxLTMwICBNeWxlcyBDLiBNYXhmaWVsZCAgPG1tYXhm aWVsZEBhcHBsZS5jb20+CiAKICAgICAgICAgQ29ycmVjdCBzcGFjaW5nIHJlZ3Jlc3Npb24gb24g aW50ZXItZWxlbWVudCBjb21wbGV4IHBhdGggc2hhcGluZyBvbiBzb21lIGZvbnRzCmRpZmYgLS1n aXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0LnB5 IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdC5weQpp bmRleCBmYzE5NmJjNjRmNzQ1YzMyOGU1NjA2MDVkZmM1OTRjMjIwMWFmOGM1Li5kMWNlY2U4N2Fj OTJiYzg4ZTA1ZjAwYzg3NzRiNTQ2MWYyOTc0YzdiIDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRz L3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0LnB5CisrKyBiL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvcGVyZm9ybWFuY2VfdGVzdHMvcGVyZnRlc3QucHkKQEAgLTEyMiwxMiArMTIy LDEyIEBAIGNsYXNzIFBlcmZUZXN0KG9iamVjdCk6CiAgICAgZGVmIHByZXBhcmUoc2VsZiwgdGlt ZV9vdXRfbXMpOgogICAgICAgICByZXR1cm4gVHJ1ZQogCi0gICAgZGVmIF9jcmVhdGVfZHJpdmVy KHNlbGYpOgotICAgICAgICByZXR1cm4gc2VsZi5fcG9ydC5jcmVhdGVfZHJpdmVyKHdvcmtlcl9u dW1iZXI9MCwgbm9fdGltZW91dD1UcnVlKQorICAgIGRlZiBfY3JlYXRlX2RyaXZlcihzZWxmLCBu b190aW1lb3V0KToKKyAgICAgICAgcmV0dXJuIHNlbGYuX3BvcnQuY3JlYXRlX2RyaXZlcih3b3Jr ZXJfbnVtYmVyPTAsIG5vX3RpbWVvdXQ9bm9fdGltZW91dCkKIAotICAgIGRlZiBydW4oc2VsZiwg dGltZV9vdXRfbXMpOgorICAgIGRlZiBydW4oc2VsZiwgdGltZV9vdXRfbXMsIG5vX3RpbWVvdXQ9 RmFsc2UpOgogICAgICAgICBmb3IgXyBpbiB4cmFuZ2Uoc2VsZi5fdGVzdF9ydW5uZXJfY291bnQp OgotICAgICAgICAgICAgZHJpdmVyID0gc2VsZi5fY3JlYXRlX2RyaXZlcigpCisgICAgICAgICAg ICBkcml2ZXIgPSBzZWxmLl9jcmVhdGVfZHJpdmVyKG5vX3RpbWVvdXQpCiAgICAgICAgICAgICB0 cnk6CiAgICAgICAgICAgICAgICAgaWYgbm90IHNlbGYuX3J1bl93aXRoX2RyaXZlcihkcml2ZXIs IHRpbWVfb3V0X21zKToKICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIE5vbmUKZGlmZiAtLWdp dCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvcGVyZm9ybWFuY2VfdGVzdHMvcGVyZnRlc3RzcnVu bmVyLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVz dHNydW5uZXIucHkKaW5kZXggZTAwMzQyMDI0ODI0MWRlMTYwZDFlNjI0MjU1OTQ2YjM2OTkxYjVi OC4uN2ViNWYzMWViNmI3NjdjNDQ2Njk3YjZlNTE2OTg4NGZmZmUxOTMyMiAxMDA2NDQKLS0tIGEv VG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdHNydW5uZXIu cHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVz dHNydW5uZXIucHkKQEAgLTEwMCw2ICsxMDAsOCBAQCBjbGFzcyBQZXJmVGVzdHNSdW5uZXIob2Jq ZWN0KToKICAgICAgICAgICAgICAgICBoZWxwPSJQYXRoIHRvIHRoZSBkaXJlY3RvcnkgdW5kZXIg d2hpY2ggYnVpbGQgZmlsZXMgYXJlIGtlcHQgKHNob3VsZCBub3QgaW5jbHVkZSBjb25maWd1cmF0 aW9uKSIpLAogICAgICAgICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24oIi0tdGltZS1vdXQtbXMi LCBkZWZhdWx0PTYwMCAqIDEwMDAsCiAgICAgICAgICAgICAgICAgaGVscD0iU2V0IHRoZSB0aW1l b3V0IGZvciBlYWNoIHRlc3QiKSwKKyAgICAgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCIt LW5vLXRpbWVvdXQiLCBhY3Rpb249InN0b3JlX3RydWUiLCBkZWZhdWx0PUZhbHNlLAorICAgICAg ICAgICAgICAgIGhlbHA9IkRpc2FibGUgdGVzdCB0aW1lb3V0cyIpLAogICAgICAgICAgICAgb3B0 cGFyc2UubWFrZV9vcHRpb24oIi0tbm8tcmVzdWx0cyIsIGFjdGlvbj0ic3RvcmVfZmFsc2UiLCBk ZXN0PSJnZW5lcmF0ZV9yZXN1bHRzIiwgZGVmYXVsdD1UcnVlLAogICAgICAgICAgICAgICAgIGhl bHA9IkRvIG5vIGdlbmVyYXRlIHJlc3VsdHMgSlNPTiBhbmQgcmVzdWx0cyBwYWdlLiIpLAogICAg ICAgICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24oIi0tb3V0cHV0LWpzb24tcGF0aCIsIGFjdGlv bj0nY2FsbGJhY2snLCBjYWxsYmFjaz1fZXhwYW5kX3BhdGgsIHR5cGU9InN0ciIsCkBAIC0zNzUs NyArMzc3LDggQEAgY2xhc3MgUGVyZlRlc3RzUnVubmVyKG9iamVjdCk6CiAgICAgICAgIGZvciBp LCB0ZXN0IGluIGVudW1lcmF0ZSh0ZXN0cyk6CiAgICAgICAgICAgICBfbG9nLmluZm8oJ1J1bm5p bmcgJXMgKCVkIG9mICVkKScgJSAodGVzdC50ZXN0X25hbWUoKSwgaSArIDEsIGxlbih0ZXN0cykp KQogICAgICAgICAgICAgc3RhcnRfdGltZSA9IHRpbWUudGltZSgpCi0gICAgICAgICAgICBtZXRy aWNzID0gdGVzdC5ydW4oc2VsZi5fb3B0aW9ucy50aW1lX291dF9tcykKKyAgICAgICAgICAgIG1l dHJpY3MgPSB0ZXN0LnJ1bihzZWxmLl9vcHRpb25zLnRpbWVfb3V0X21zLCBzZWxmLl9vcHRpb25z Lm5vX3RpbWVvdXQpCisKICAgICAgICAgICAgIGlmIG1ldHJpY3M6CiAgICAgICAgICAgICAgICAg c2VsZi5fcmVzdWx0cyArPSBtZXRyaWNzCiAgICAgICAgICAgICBlbHNlOgo= 300553 2017-02-03 11:51:19 -0800 2017-02-10 22:02:55 -0800 Patch bug-167626-20170203204901.patch text/plain 7406 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjExNjMyCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZjQ4MjExMDQyNzkzZTc0ODM0NmVlMzliZjlkZjI2MGJi MDcyMjcwNS4uMTZkYzFlMzA0NzY2ZWFlNzNiZTExMzhkZTFjODQyZDQ4MTZhZGUzMiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDMw IEBACisyMDE3LTAyLTAzICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIFJFR1JFU1NJT04ocjE4MjkxNikgcnVuLXBlcmYtdGVzdHMgbmV2 ZXIgdGltZW91dHMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dp P2lkPTE2NzYyNgorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAg ICAgIFRoZSAtLXRpbWUtb3V0LW1zIHZhbHVlIChkZWZhdWx0cyB0byA2MDAgc2Vjb25kcykgb2Yg cnVuLXBlcmYtdGVzdHMKKyAgICAgICAgd2FzIGJlZWluZyBpZ25vcmVkIGJlY2F1c2UgdGhlIGRy aXZlciB3YXMgY3JlYXRlZCB3aXRoIG5vX3RpbWVvdXQ9VHJ1ZQorICAgICAgICB1bmNvbmRpdGlv bmFsbHkuCisgICAgICAgIFRoaXMgYWRkcyBhIG5ldyBwYXJhbWV0ZXIgLS1uby10aW1lb3V0IChk aXNhYmxlZCBieSBkZWZhdWx0KSwgdGhhdAorICAgICAgICBhbGxvd3MgdG8gdXNlIHRoZSBkcml2 ZXIgd2l0aCB0aGUgZXhwZWN0ZWQgdGltZW91dCB2YWx1ZXMuCisgICAgICAgIEl0IGFsc28gcGFz c2VzIC0tbm8tdGltZW91dCB0byBEUlQvV1RSIHRvIGVuc3VyZSB0aGF0IHRpbWVvdXRzCisgICAg ICAgIGFyZSBhbHdheXMgY29udHJvbGxlZCBieSB0aGUgUHl0aG9uIERyaXZlci4KKworICAgICAg ICAqIFNjcmlwdHMvd2Via2l0cHkvcGVyZm9ybWFuY2VfdGVzdHMvcGVyZnRlc3QucHk6CisgICAg ICAgIChQZXJmVGVzdC5fY3JlYXRlX2RyaXZlcik6IE9ubHkgZGlzYWJsZSB0aW1lb3V0IGlmIC0t bm8tdGltZW91dCBpcyBwYXNzZWQuCisgICAgICAgIChQZXJmVGVzdC5ydW4pOiBEaXNhYmxpbmcg dGltZW91dCBzaG91bGQgYmUgZG9uZSB3aGVuIHRoZSBEcml2ZXIgaXMgY3JlYXRlZC4KKyAgICAg ICAgKiBTY3JpcHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0c3J1bm5lci5w eToKKyAgICAgICAgKFBlcmZUZXN0c1J1bm5lci5fX2luaXRfXyk6IEVuc3VyZSAtLW5vLXRpbWVv dXQgaXMgcGFzc2VkIHRvIERSVC9XVFIKKyAgICAgICAgKFBlcmZUZXN0c1J1bm5lci5fcGFyc2Vf YXJncyk6IEFkZCBhIC0tbm8tdGltZW91dCBwYXJhbWV0ZXIuCisgICAgICAgIChfcnVuX3Rlc3Rz X3NldCk6IFBhc3MgdGhlIG5vLXRpbWVvdXQgcGFyYW1ldGVyLgorICAgICAgICAqIFNjcmlwdHMv d2Via2l0cHkvcGVyZm9ybWFuY2VfdGVzdHMvcGVyZnRlc3RzcnVubmVyX3VuaXR0ZXN0LnB5Ogor ICAgICAgICAoTWFpblRlc3QudGVzdF9kcnRfbm90aW1lb3V0KTogVGVzdCAtLW5vLXRpbWVvdXQg aXMgcGFzc2VkIHRvIERSVC9XVFIKKyAgICAgICAgKE1haW5UZXN0LnRlc3RfZGVmYXVsdF9hcmdz KTogVGVzdCBkZWZhdWx0IHZhbHVlcyBmb3IgYWRkaXRpb25hbF9kcnRfZmxhZyBhbmQgbm9fdGlt ZW91dC4KKyAgICAgICAgKE1haW5UZXN0LnRlc3RfcGFyc2VfYXJncyk6IFRlc3QgdGhhdCB0aGUg YXJndW1lbnQgbm8tdGltZW91dC4KKwogMjAxNy0wMi0wMiAgQWxleCBDaHJpc3RlbnNlbiAgPGFj aHJpc3RlbnNlbkB3ZWJraXQub3JnPgogCiAgICAgICAgIFVSTFBhcnNlcjogRml4IHBhcnNpbmcg aW52YWxpZCBJUHY0IGFkZHJlc3NlcyB3aXRoIG5vbi1BU0NJSSBjaGFyYWN0ZXJzCmRpZmYgLS1n aXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0LnB5 IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdC5weQpp bmRleCBmYzE5NmJjNjRmNzQ1YzMyOGU1NjA2MDVkZmM1OTRjMjIwMWFmOGM1Li5kMWNlY2U4N2Fj OTJiYzg4ZTA1ZjAwYzg3NzRiNTQ2MWYyOTc0YzdiIDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRz L3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0LnB5CisrKyBiL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvcGVyZm9ybWFuY2VfdGVzdHMvcGVyZnRlc3QucHkKQEAgLTEyMiwxMiArMTIy LDEyIEBAIGNsYXNzIFBlcmZUZXN0KG9iamVjdCk6CiAgICAgZGVmIHByZXBhcmUoc2VsZiwgdGlt ZV9vdXRfbXMpOgogICAgICAgICByZXR1cm4gVHJ1ZQogCi0gICAgZGVmIF9jcmVhdGVfZHJpdmVy KHNlbGYpOgotICAgICAgICByZXR1cm4gc2VsZi5fcG9ydC5jcmVhdGVfZHJpdmVyKHdvcmtlcl9u dW1iZXI9MCwgbm9fdGltZW91dD1UcnVlKQorICAgIGRlZiBfY3JlYXRlX2RyaXZlcihzZWxmLCBu b190aW1lb3V0KToKKyAgICAgICAgcmV0dXJuIHNlbGYuX3BvcnQuY3JlYXRlX2RyaXZlcih3b3Jr ZXJfbnVtYmVyPTAsIG5vX3RpbWVvdXQ9bm9fdGltZW91dCkKIAotICAgIGRlZiBydW4oc2VsZiwg dGltZV9vdXRfbXMpOgorICAgIGRlZiBydW4oc2VsZiwgdGltZV9vdXRfbXMsIG5vX3RpbWVvdXQ9 RmFsc2UpOgogICAgICAgICBmb3IgXyBpbiB4cmFuZ2Uoc2VsZi5fdGVzdF9ydW5uZXJfY291bnQp OgotICAgICAgICAgICAgZHJpdmVyID0gc2VsZi5fY3JlYXRlX2RyaXZlcigpCisgICAgICAgICAg ICBkcml2ZXIgPSBzZWxmLl9jcmVhdGVfZHJpdmVyKG5vX3RpbWVvdXQpCiAgICAgICAgICAgICB0 cnk6CiAgICAgICAgICAgICAgICAgaWYgbm90IHNlbGYuX3J1bl93aXRoX2RyaXZlcihkcml2ZXIs IHRpbWVfb3V0X21zKToKICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIE5vbmUKZGlmZiAtLWdp dCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvcGVyZm9ybWFuY2VfdGVzdHMvcGVyZnRlc3RzcnVu bmVyLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVz dHNydW5uZXIucHkKaW5kZXggZTAwMzQyMDI0ODI0MWRlMTYwZDFlNjI0MjU1OTQ2YjM2OTkxYjVi OC4uNGNmNThjNTdlM2U0Y2Y0YTIyODQzNTYwYmI3MzMwOTNjYTc4OTZlNSAxMDA2NDQKLS0tIGEv VG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdHNydW5uZXIu cHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVz dHNydW5uZXIucHkKQEAgLTY2LDYgKzY2LDkgQEAgY2xhc3MgUGVyZlRlc3RzUnVubmVyKG9iamVj dCk6CiAgICAgICAgICAgICBzZWxmLl9ob3N0ID0gSG9zdCgpCiAgICAgICAgICAgICBzZWxmLl9w b3J0ID0gc2VsZi5faG9zdC5wb3J0X2ZhY3RvcnkuZ2V0KHNlbGYuX29wdGlvbnMucGxhdGZvcm0s IHNlbGYuX29wdGlvbnMpCiAKKyAgICAgICAgIyBUaW1lb3V0cyBhcmUgY29udHJvbGxlZCBieSB0 aGUgUHl0aG9uIERyaXZlciwgc28gRFJUL1dUUiBydW5zIHdpdGggbm8tdGltZW91dC4KKyAgICAg ICAgc2VsZi5fb3B0aW9ucy5hZGRpdGlvbmFsX2RydF9mbGFnLmFwcGVuZCgnLS1uby10aW1lb3V0 JykKKwogICAgICAgICAjIFRoZSBHVEsrIGFuZCBFRkwgcG9ydHMgb25seSBzdXBwb3J0cyBXZWJL aXQyLCBzbyB0aGV5IGFsd2F5cyB1c2UgV0tUUi4KICAgICAgICAgaWYgc2VsZi5fcG9ydC5uYW1l KCkuc3RhcnRzd2l0aCgiZ3RrIikgb3Igc2VsZi5fcG9ydC5uYW1lKCkuc3RhcnRzd2l0aCgiZWZs Iik6CiAgICAgICAgICAgICBzZWxmLl9vcHRpb25zLndlYmtpdF90ZXN0X3J1bm5lciA9IFRydWUK QEAgLTEwMCw2ICsxMDMsOCBAQCBjbGFzcyBQZXJmVGVzdHNSdW5uZXIob2JqZWN0KToKICAgICAg ICAgICAgICAgICBoZWxwPSJQYXRoIHRvIHRoZSBkaXJlY3RvcnkgdW5kZXIgd2hpY2ggYnVpbGQg ZmlsZXMgYXJlIGtlcHQgKHNob3VsZCBub3QgaW5jbHVkZSBjb25maWd1cmF0aW9uKSIpLAogICAg ICAgICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24oIi0tdGltZS1vdXQtbXMiLCBkZWZhdWx0PTYw MCAqIDEwMDAsCiAgICAgICAgICAgICAgICAgaGVscD0iU2V0IHRoZSB0aW1lb3V0IGZvciBlYWNo IHRlc3QiKSwKKyAgICAgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCItLW5vLXRpbWVvdXQi LCBhY3Rpb249InN0b3JlX3RydWUiLCBkZWZhdWx0PUZhbHNlLAorICAgICAgICAgICAgICAgIGhl bHA9IkRpc2FibGUgdGVzdCB0aW1lb3V0cyIpLAogICAgICAgICAgICAgb3B0cGFyc2UubWFrZV9v cHRpb24oIi0tbm8tcmVzdWx0cyIsIGFjdGlvbj0ic3RvcmVfZmFsc2UiLCBkZXN0PSJnZW5lcmF0 ZV9yZXN1bHRzIiwgZGVmYXVsdD1UcnVlLAogICAgICAgICAgICAgICAgIGhlbHA9IkRvIG5vIGdl bmVyYXRlIHJlc3VsdHMgSlNPTiBhbmQgcmVzdWx0cyBwYWdlLiIpLAogICAgICAgICAgICAgb3B0 cGFyc2UubWFrZV9vcHRpb24oIi0tb3V0cHV0LWpzb24tcGF0aCIsIGFjdGlvbj0nY2FsbGJhY2sn LCBjYWxsYmFjaz1fZXhwYW5kX3BhdGgsIHR5cGU9InN0ciIsCkBAIC0zNzUsNyArMzgwLDggQEAg Y2xhc3MgUGVyZlRlc3RzUnVubmVyKG9iamVjdCk6CiAgICAgICAgIGZvciBpLCB0ZXN0IGluIGVu dW1lcmF0ZSh0ZXN0cyk6CiAgICAgICAgICAgICBfbG9nLmluZm8oJ1J1bm5pbmcgJXMgKCVkIG9m ICVkKScgJSAodGVzdC50ZXN0X25hbWUoKSwgaSArIDEsIGxlbih0ZXN0cykpKQogICAgICAgICAg ICAgc3RhcnRfdGltZSA9IHRpbWUudGltZSgpCi0gICAgICAgICAgICBtZXRyaWNzID0gdGVzdC5y dW4oc2VsZi5fb3B0aW9ucy50aW1lX291dF9tcykKKyAgICAgICAgICAgIG1ldHJpY3MgPSB0ZXN0 LnJ1bihzZWxmLl9vcHRpb25zLnRpbWVfb3V0X21zLCBzZWxmLl9vcHRpb25zLm5vX3RpbWVvdXQp CisKICAgICAgICAgICAgIGlmIG1ldHJpY3M6CiAgICAgICAgICAgICAgICAgc2VsZi5fcmVzdWx0 cyArPSBtZXRyaWNzCiAgICAgICAgICAgICBlbHNlOgpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0 cy93ZWJraXRweS9wZXJmb3JtYW5jZV90ZXN0cy9wZXJmdGVzdHNydW5uZXJfdW5pdHRlc3QucHkg Yi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0c3J1bm5l cl91bml0dGVzdC5weQppbmRleCA2MDJhMjg4NjRjMGViMGUyZmJiMDM2MTc3OTc0ZDhiNjU2Yjcz YTY3Li4zNmQzZGRlNDU1MWUyZmY1MDEyOGE1YThlZDVlNjNkY2Y4NzA3MjZlIDEwMDY0NAotLS0g YS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rlc3RzL3BlcmZ0ZXN0c3J1bm5l cl91bml0dGVzdC5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3BlcmZvcm1hbmNlX3Rl c3RzL3BlcmZ0ZXN0c3J1bm5lcl91bml0dGVzdC5weQpAQCAtNTUsNiArNTUsMTAgQEAgY2xhc3Mg TWFpblRlc3QodW5pdHRlc3QuVGVzdENhc2UpOgogICAgICAgICBydW5uZXIuX2hvc3QuZmlsZXN5 c3RlbS5tYXliZV9tYWtlX2RpcmVjdG9yeShkaXJuYW1lKQogICAgICAgICBydW5uZXIuX2hvc3Qu ZmlsZXN5c3RlbS5maWxlc1tydW5uZXIuX2hvc3QuZmlsZXN5c3RlbS5qb2luKGRpcm5hbWUsIGZp bGVuYW1lKV0gPSBjb250ZW50CiAKKyAgICBkZWYgdGVzdF9kcnRfbm90aW1lb3V0KHNlbGYpOgor ICAgICAgICBydW5uZXIsIHBvcnQgPSBzZWxmLmNyZWF0ZV9ydW5uZXIoKQorICAgICAgICBzZWxm LmFzc2VydEVxdWFsKHJ1bm5lci5fb3B0aW9ucy5hZGRpdGlvbmFsX2RydF9mbGFnLCBbJy0tbm8t dGltZW91dCddKQorCiAgICAgZGVmIHRlc3RfY29sbGVjdF90ZXN0cyhzZWxmKToKICAgICAgICAg cnVubmVyLCBwb3J0ID0gc2VsZi5jcmVhdGVfcnVubmVyKCkKICAgICAgICAgc2VsZi5fYWRkX2Zp bGUocnVubmVyLCAnaW5zcGVjdG9yJywgJ2FfZmlsZS5odG1sJywgJ2EgY29udGVudCcpCkBAIC0x MTYsMTEgKzEyMCwxMyBAQCBjbGFzcyBNYWluVGVzdCh1bml0dGVzdC5UZXN0Q2FzZSk6CiAgICAg ICAgIG9wdGlvbnMsIGFyZ3MgPSBQZXJmVGVzdHNSdW5uZXIuX3BhcnNlX2FyZ3MoW10pCiAgICAg ICAgIHNlbGYuYXNzZXJ0VHJ1ZShvcHRpb25zLmJ1aWxkKQogICAgICAgICBzZWxmLmFzc2VydEVx dWFsKG9wdGlvbnMudGltZV9vdXRfbXMsIDYwMCAqIDEwMDApCisgICAgICAgIHNlbGYuYXNzZXJ0 RXF1YWwob3B0aW9ucy5hZGRpdGlvbmFsX2RydF9mbGFnLCBbXSkKICAgICAgICAgc2VsZi5hc3Nl cnRUcnVlKG9wdGlvbnMuZ2VuZXJhdGVfcmVzdWx0cykKICAgICAgICAgc2VsZi5hc3NlcnRUcnVl KG9wdGlvbnMuc2hvd19yZXN1bHRzKQogICAgICAgICBzZWxmLmFzc2VydFRydWUob3B0aW9ucy51 c2Vfc2tpcHBlZF9saXN0KQogICAgICAgICBzZWxmLmFzc2VydEVxdWFsKG9wdGlvbnMucmVwZWF0 LCAxKQogICAgICAgICBzZWxmLmFzc2VydEVxdWFsKG9wdGlvbnMudGVzdF9ydW5uZXJfY291bnQs IC0xKQorICAgICAgICBzZWxmLmFzc2VydEVxdWFsKG9wdGlvbnMubm9fdGltZW91dCwgRmFsc2Up CiAKICAgICBkZWYgdGVzdF9wYXJzZV9hcmdzKHNlbGYpOgogICAgICAgICBydW5uZXIsIHBvcnQg PSBzZWxmLmNyZWF0ZV9ydW5uZXIoKQpAQCAtMTM5LDcgKzE0NSw4IEBAIGNsYXNzIE1haW5UZXN0 KHVuaXR0ZXN0LlRlc3RDYXNlKToKICAgICAgICAgICAgICAgICAnLS1hZGRpdGlvbmFsLWRydC1m bGFnPS0tYXdlc29tZXNhdWNlJywKICAgICAgICAgICAgICAgICAnLS1yZXBlYXQ9NScsCiAgICAg ICAgICAgICAgICAgJy0tdGVzdC1ydW5uZXItY291bnQ9NScsCi0gICAgICAgICAgICAgICAgJy0t ZGVidWcnXSkKKyAgICAgICAgICAgICAgICAnLS1kZWJ1ZycsCisgICAgICAgICAgICAgICAgJy0t bm8tdGltZW91dCddKQogICAgICAgICBzZWxmLmFzc2VydFRydWUob3B0aW9ucy5idWlsZCkKICAg ICAgICAgc2VsZi5hc3NlcnRFcXVhbChvcHRpb25zLmJ1aWxkX2RpcmVjdG9yeSwgJ2ZvbGRlcjQy JykKICAgICAgICAgc2VsZi5hc3NlcnRFcXVhbChvcHRpb25zLnBsYXRmb3JtLCAncGxhdGZvcm00 MicpCkBAIC0xNTUsNiArMTYyLDcgQEAgY2xhc3MgTWFpblRlc3QodW5pdHRlc3QuVGVzdENhc2Up OgogICAgICAgICBzZWxmLmFzc2VydEVxdWFsKG9wdGlvbnMuYWRkaXRpb25hbF9kcnRfZmxhZywg WyctLWVuYWJsZS10aHJlYWRlZC1wYXJzZXInLCAnLS1hd2Vzb21lc2F1Y2UnXSkKICAgICAgICAg c2VsZi5hc3NlcnRFcXVhbChvcHRpb25zLnJlcGVhdCwgNSkKICAgICAgICAgc2VsZi5hc3NlcnRF cXVhbChvcHRpb25zLnRlc3RfcnVubmVyX2NvdW50LCA1KQorICAgICAgICBzZWxmLmFzc2VydEVx dWFsKG9wdGlvbnMubm9fdGltZW91dCwgVHJ1ZSkKIAogICAgIGRlZiB0ZXN0X3VwbG9hZF9qc29u KHNlbGYpOgogICAgICAgICBydW5uZXIsIHBvcnQgPSBzZWxmLmNyZWF0ZV9ydW5uZXIoKQo=