<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugs.webkit.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.4.1"
          urlbase="https://bugs.webkit.org/"
          
          maintainer="admin@webkit.org"
>

    <bug>
          <bug_id>51802</bug_id>
          
          <creation_ts>2011-01-02 12:01:54 -0800</creation_ts>
          <short_desc>Move LayoutTestResults over to new-run-webkit-tests TestResult architecture</short_desc>
          <delta_ts>2011-01-05 18:15:20 -0800</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>1</classification_id>
          <classification>Unclassified</classification>
          <product>WebKit</product>
          <component>New Bugs</component>
          <version>528+ (Nightly build)</version>
          <rep_platform>Other</rep_platform>
          <op_sys>OS X 10.5</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>P2</priority>
          <bug_severity>Normal</bug_severity>
          <target_milestone>---</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Eric Seidel (no email)">eric</reporter>
          <assigned_to name="Eric Seidel (no email)">eric</assigned_to>
          <cc>abarth</cc>
    
    <cc>commit-queue</cc>
    
    <cc>dpranke</cc>
    
    <cc>ojan</cc>
          

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>328410</commentid>
    <comment_count>0</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-02 12:01:54 -0800</bug_when>
    <thetext>Move LayoutTestResults over to new-run-webkit-tests TestResult architecture</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328413</commentid>
    <comment_count>1</comment_count>
      <attachid>77778</attachid>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-02 12:16:15 -0800</bug_when>
    <thetext>Created attachment 77778
Patch</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328416</commentid>
    <comment_count>2</comment_count>
      <attachid>77778</attachid>
    <who name="Adam Barth">abarth</who>
    <bug_when>2011-01-02 12:30:47 -0800</bug_when>
    <thetext>Comment on attachment 77778
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=77778&amp;action=review

&gt; Tools/Scripts/webkitpy/common/net/buildbot/buildbot_unittest.py:34
&gt; +from webkitpy.layout_tests.layout_package import test_results
&gt; +from webkitpy.layout_tests.layout_package import test_failures

We should move these classes to common.   Common shouldn&apos;t depend on layout_tests.

&gt; Tools/Scripts/webkitpy/tool/commands/queries.py:275
&gt; -        chosen_name = User.prompt_with_list(&quot;Which builder to diagnose:&quot;, builder_choices)
&gt; +        chosen_name = self._tool.user.prompt_with_list(&quot;Which builder to diagnose:&quot;, builder_choices)

These changes seem separate, but ok.

&gt; Tools/Scripts/webkitpy/tool/commands/rebaseline.py:38
&gt; +from webkitpy.layout_tests.layout_package import test_failures
&gt;  from webkitpy.layout_tests.port import factory

Another bad dependency....  :(</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328417</commentid>
    <comment_count>3</comment_count>
      <attachid>77778</attachid>
    <who name="Adam Barth">abarth</who>
    <bug_when>2011-01-02 12:33:16 -0800</bug_when>
    <thetext>Comment on attachment 77778
Patch

You might want to fix the dependencies in a follow up patch, but please don&apos;t let them languish.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328712</commentid>
    <comment_count>4</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-03 12:26:59 -0800</bug_when>
    <thetext>It&apos;s not clear that any of this layout test stuff belongs in common.

So maybe I should move LayoutTestResults over net to the rest of the stuff?  Except then we have BuildBot depending on stuff outside of common. :)</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328713</commentid>
    <comment_count>5</comment_count>
      <attachid>77778</attachid>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-03 12:27:45 -0800</bug_when>
    <thetext>Comment on attachment 77778
Patch

I would like to clean up the dependencies, yes.  I think that&apos;s in the next patch.  I&apos;d also like to chat with you some in person about how we should lay this all out.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328719</commentid>
    <comment_count>6</comment_count>
      <attachid>77778</attachid>
    <who name="WebKit Commit Bot">commit-queue</who>
    <bug_when>2011-01-03 12:59:25 -0800</bug_when>
    <thetext>Comment on attachment 77778
Patch

Clearing flags on attachment: 77778

Committed r74927: &lt;http://trac.webkit.org/changeset/74927&gt;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328720</commentid>
    <comment_count>7</comment_count>
    <who name="WebKit Commit Bot">commit-queue</who>
    <bug_when>2011-01-03 12:59:32 -0800</bug_when>
    <thetext>All reviewed patches have been landed.  Closing bug.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328746</commentid>
    <comment_count>8</comment_count>
    <who name="Adam Barth">abarth</who>
    <bug_when>2011-01-03 13:27:45 -0800</bug_when>
    <thetext>(In reply to comment #4)
&gt; It&apos;s not clear that any of this layout test stuff belongs in common.

Yeah, it&apos;s possible we need three layers instead of two.

&gt; So maybe I should move LayoutTestResults over net to the rest of the stuff?  Except then we have BuildBot depending on stuff outside of common. :)

LayoutTestResults isn&apos;t really net-specific in the sense that you can care about LayoutTestResults without a network.

The goal of having different layers is the constraint the effects of changes.  We&apos;d like to be able to work on new-run-webkit-tests without worrying about breaking webkit-patch.  maybe the thing to do is to pull the new-run-webkit-tests-specific parts out of layout_tests?  We could then move the remaining parts of layout_tests into common?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328793</commentid>
    <comment_count>9</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-03 14:39:11 -0800</bug_when>
    <thetext>(In reply to comment #8)
&gt; (In reply to comment #4)
&gt; &gt; It&apos;s not clear that any of this layout test stuff belongs in common.
&gt; 
&gt; Yeah, it&apos;s possible we need three layers instead of two.

I&apos;m not sure what that would look like.

I see at least these pieces of webkitpy:

- common - stuff which is shared, but not necessarily to webkit.
- config - things specific to webkit which is used by common
- tool - webkit-patch specific code
- layout_tests (which probably should be renamed), is NRWT specific code.
- thirdparty - used by various places, is third-party code.

&gt; &gt; So maybe I should move LayoutTestResults over net to the rest of the stuff?  Except then we have BuildBot depending on stuff outside of common. :)
&gt; 
&gt; LayoutTestResults isn&apos;t really net-specific in the sense that you can care about LayoutTestResults without a network.

Yes.  It&apos;s ORWT specific.  Buildbot needs something that&apos;s not ORWT specific eventually.  Ideally (at least in my mind) it needs some sort of generalized &quot;results&quot; abstraction which knows how to vend some object (like LayoutTestResults) which knows how to handle results from whatever run-webkit-tests we&apos;re using.

Ideally LayoutTestResults, etc, would be outside of common, possibly in Config, or at least the buildbot results class would be in config, or provided by some webkit-specific config.  (I still have this -- likely overengineered -- pipedream that much of webkitpy need not be specific to webkit, and be re-usable by things like chromium.org or andriod or Apple (for Radar, etc.).

&gt; The goal of having different layers is the constraint the effects of changes.  We&apos;d like to be able to work on new-run-webkit-tests without worrying about breaking webkit-patch.  maybe the thing to do is to pull the new-run-webkit-tests-specific parts out of layout_tests?

Yes, I think that&apos;s a good goal.  Some of that is tricky right now due to the fact that webkit-patch deals only in ORWT concepts and NRWT deals only with NRWT concepts. :)

&gt; We could then move the remaining parts of layout_tests into common?
Maybe.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>328796</commentid>
    <comment_count>10</comment_count>
    <who name="Adam Barth">abarth</who>
    <bug_when>2011-01-03 14:48:32 -0800</bug_when>
    <thetext>Right, the three layers could be:

1) application-specific (webkit-patch, new-run-webkit-tests, etc)
2) shared, but webkit-specific
3) shared, and non-webkit specific

Model classes about test results seem to go in layer 2.  Classes for talking to buildbot (generally) seem to go in layer 3.  Controller classes for executing DumpRenderTree and creating models of the results seems to go in layer 1.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329267</commentid>
    <comment_count>11</comment_count>
      <attachid>77778</attachid>
    <who name="Dirk Pranke">dpranke</who>
    <bug_when>2011-01-04 15:52:07 -0800</bug_when>
    <thetext>Comment on attachment 77778
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=77778&amp;action=review

&gt; Tools/Scripts/webkitpy/common/net/layouttestresults.py:98
&gt; +        return test_results.TestResult(test_name, failures, test_run_time=None, total_time_for_all_diffs=None, time_for_diffs=None)

Technically, test names are port-specific, and you should be getting the name of the test from the port object. In practice, all of the real ports use the same names, and they should not include the &quot;LayoutTests&quot; prefix.

&gt; Tools/Scripts/webkitpy/common/net/layouttestresults.py:138
&gt; +        return self.tests_matching_failure_types(failure_types)

Do you want to include any tests that had missing or incorrect expectations files?

Also, you shouldn&apos;t be looking at test_failures values at all. You should look at the &quot;type&quot; field on the TestResult object and compare it to one of the test_expectations constants (e.g., TEXT, IMAGE, CRASH, etc.).

I&apos;m not sure what you&apos;re intended definition of &quot;failing&quot; is, but we should probably push the definition into the TestResult object itself and add a test_failed() method or something on it instead of you having the logic here.

&gt; Tools/Scripts/webkitpy/layout_tests/layout_package/test_failures.py:40
&gt; +# collects them all from the failure list and returns the worst one.

I&apos;m not sure what you&apos;re trying to say here. The mapping from test_expectation type to TestFailure isn&apos;t 1:1, and I would argue that it shouldn&apos;t be, given the current design. So I wouldn&apos;t add this FIXME at all. But perhaps I&apos;m misunderstanding you?

&gt; Tools/Scripts/webkitpy/layout_tests/layout_package/test_results.py:65
&gt; +

See the comment above. I think we want a test_failed() or failed() method here, rather than something taking a list of failure types. I.e., this class should own the definition of &quot;failing&quot;.

&gt; Tools/Scripts/webkitpy/tool/commands/rebaseline.py:92
&gt; +        failing_tests = build.layout_test_results().tests_matching_failure_types([test_failures.FailureTextMismatch])

I don&apos;t know the context of this routine but (a) it seems like &quot;FailureTextMismatch&quot; is probably the wrong value to be looking for and (b) you should be checking against result.type == test_expectations.TEXT instead of looking at the failure types directly (as per above).</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329270</commentid>
    <comment_count>12</comment_count>
    <who name="Dirk Pranke">dpranke</who>
    <bug_when>2011-01-04 16:01:02 -0800</bug_when>
    <thetext>(In reply to comment #10)
&gt; Right, the three layers could be:
&gt; 
&gt; 1) application-specific (webkit-patch, new-run-webkit-tests, etc)
&gt; 2) shared, but webkit-specific
&gt; 3) shared, and non-webkit specific
&gt; 
&gt; Model classes about test results seem to go in layer 2.  Classes for talking to buildbot (generally) seem to go in layer 3.  Controller classes for executing DumpRenderTree and creating models of the results seems to go in layer 1.

This way of looking at things is a little weird to me. I would rather see stuff grouped by functionality rather than by visibility. i.e., layout_tests/public/foo rather than public/layout_tests/foo. Although either of these things seems like a bit of a hack, and we should just be using python&apos;s normal visibility mechanisms to control things.

Also, I&apos;m not sure what the current (not intended) definition of &quot;common&quot; is - whether it is (1), (2), (3), or some mixture of all of them?

I would be inclined to make the buildbot stuff its own top-level package rather than putting it under &quot;common&quot;, as I probably would with some of the other things in common.net.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329271</commentid>
    <comment_count>13</comment_count>
    <who name="Dirk Pranke">dpranke</who>
    <bug_when>2011-01-04 16:03:02 -0800</bug_when>
    <thetext>(In reply to comment #6)
&gt; (From update of attachment 77778 [details])
&gt; Clearing flags on attachment: 77778
&gt; 
&gt; Committed r74927: &lt;http://trac.webkit.org/changeset/74927&gt;

Note that if it wasn&apos;t clear from my comments in #11, I would have R-&apos;ed this patch. Assuming my comments make sense and you agree with them, can you please file another bug to fix/undo some of the changes? (And if they don&apos;t make sense, or you don&apos;t agree, let me know ;).</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329279</commentid>
    <comment_count>14</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-04 16:18:37 -0800</bug_when>
    <thetext>(In reply to comment #11)
&gt; (From update of attachment 77778 [details])
&gt; View in context: https://bugs.webkit.org/attachment.cgi?id=77778&amp;action=review
&gt; 
&gt; &gt; Tools/Scripts/webkitpy/common/net/layouttestresults.py:98
&gt; &gt; +        return test_results.TestResult(test_name, failures, test_run_time=None, total_time_for_all_diffs=None, time_for_diffs=None)
&gt; 
&gt; Technically, test names are port-specific, and you should be getting the name of the test from the port object. In practice, all of the real ports use the same names, and they should not include the &quot;LayoutTests&quot; prefix.

I think we should add an assert() into the TestResult constructor if names are expected not to be passed with LayoutTests/.  (They&apos;re currently not in this patch, but that would make the class easier to use.

&gt; &gt; Tools/Scripts/webkitpy/common/net/layouttestresults.py:138
&gt; &gt; +        return self.tests_matching_failure_types(failure_types)
&gt; 
&gt; Do you want to include any tests that had missing or incorrect expectations files?

No.  The previous failing_tests() logic didn&apos;t, and I didn&apos;t want to change it.

&gt; Also, you shouldn&apos;t be looking at test_failures values at all. You should look at the &quot;type&quot; field on the TestResult object and compare it to one of the test_expectations constants (e.g., TEXT, IMAGE, CRASH, etc.).

That&apos;s much less specific, but that could be easier.

&gt; I&apos;m not sure what you&apos;re intended definition of &quot;failing&quot; is, but we should probably push the definition into the TestResult object itself and add a test_failed() method or something on it instead of you having the logic here.

The definition of &quot;failing&quot; here is something specific to LayoutTestResults and is a historical hold-over.  I tried to be more specific in new code.

&gt; &gt; Tools/Scripts/webkitpy/layout_tests/layout_package/test_failures.py:40
&gt; &gt; +# collects them all from the failure list and returns the worst one.
&gt; 
&gt; I&apos;m not sure what you&apos;re trying to say here. The mapping from test_expectation type to TestFailure isn&apos;t 1:1, and I would argue that it shouldn&apos;t be, given the current design. So I wouldn&apos;t add this FIXME at all. But perhaps I&apos;m misunderstanding you?

This may be a mis-understanding on my part.  Maybe I wouldn&apos;t have felt the need for this if I had tried comparing .type to a test_expectations enum.

&gt; &gt; Tools/Scripts/webkitpy/layout_tests/layout_package/test_results.py:65
&gt; &gt; +
&gt; 
&gt; See the comment above. I think we want a test_failed() or failed() method here, rather than something taking a list of failure types. I.e., this class should own the definition of &quot;failing&quot;.

Again, just a historical hold-over.  But I wasn&apos;t ready to convert all the places to be more specific.

&gt; &gt; Tools/Scripts/webkitpy/tool/commands/rebaseline.py:92
&gt; &gt; +        failing_tests = build.layout_test_results().tests_matching_failure_types([test_failures.FailureTextMismatch])
&gt; 
&gt; I don&apos;t know the context of this routine but (a) it seems like &quot;FailureTextMismatch&quot; is probably the wrong value to be looking for and (b) you should be checking against result.type == test_expectations.TEXT instead of looking at the failure types directly (as per above).

I can try that in a later patch.  But I don&apos;t think that will work as you expect.   .type is only one value.  What happens when a test fails both image and text?  .type only reflects one of those.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329283</commentid>
    <comment_count>15</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-04 16:20:32 -0800</bug_when>
    <thetext>(In reply to comment #12)
&gt; (In reply to comment #10)
&gt; &gt; Right, the three layers could be:
&gt; &gt; 
&gt; &gt; 1) application-specific (webkit-patch, new-run-webkit-tests, etc)
&gt; &gt; 2) shared, but webkit-specific
&gt; &gt; 3) shared, and non-webkit specific
&gt; &gt; 
&gt; &gt; Model classes about test results seem to go in layer 2.  Classes for talking to buildbot (generally) seem to go in layer 3.  Controller classes for executing DumpRenderTree and creating models of the results seems to go in layer 1.
&gt; 
&gt; This way of looking at things is a little weird to me. I would rather see stuff grouped by functionality rather than by visibility. i.e., layout_tests/public/foo rather than public/layout_tests/foo. Although either of these things seems like a bit of a hack, and we should just be using python&apos;s normal visibility mechanisms to control things.

I&apos;m not sure what you mean.

&gt; Also, I&apos;m not sure what the current (not intended) definition of &quot;common&quot; is - whether it is (1), (2), (3), or some mixture of all of them?

common was stuff that was pulled out of the original bugzilla-tool to be shared by other python code.  bugzilla-tool became webkit-patch and thus the tool/ directory.  :)

I think common is a mixture of a lot of things these days.

&gt; I would be inclined to make the buildbot stuff its own top-level package rather than putting it under &quot;common&quot;, as I probably would with some of the other things in common.net.

buildbot depends on stuff in common.net (and is already a package, although not a very useful one).</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329285</commentid>
    <comment_count>16</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-04 16:24:00 -0800</bug_when>
    <thetext>(In reply to comment #13)
&gt; (In reply to comment #6)
&gt; &gt; (From update of attachment 77778 [details] [details])
&gt; &gt; Clearing flags on attachment: 77778
&gt; &gt; 
&gt; &gt; Committed r74927: &lt;http://trac.webkit.org/changeset/74927&gt;
&gt; 
&gt; Note that if it wasn&apos;t clear from my comments in #11, I would have R-&apos;ed this patch. Assuming my comments make sense and you agree with them, can you please file another bug to fix/undo some of the changes? (And if they don&apos;t make sense, or you don&apos;t agree, let me know ;).

This stuff definitely needs more iteration.

I find the TestResults stuff very cumbersome to use.  The fact that I have to import 3 different packages from layout_tests just to use it is one problem.

I think LayoutTestResults needs to move to being in the same package as the rest of this stuff and then it will make more sense.

I think .type is not a complete solution for replacing teh type(FailureClass) comparisons I had to do in this code (I did this based on other examples in the codebase).

I like the idea of passing around Failure classes, since they can contain more than just a name/number.  however I would like to shorten the names of the various failure types so they&apos;re easier to use. (Similar to how we use the steps module).  I also think that it should be possible to do a one-to-many (or many-to-one?) mapping from test expectations enums to failure classes.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329313</commentid>
    <comment_count>17</comment_count>
    <who name="Dirk Pranke">dpranke</who>
    <bug_when>2011-01-04 17:21:01 -0800</bug_when>
    <thetext>(In reply to comment #14)
&gt; &gt; Technically, test names are port-specific, and you should be getting the name of the test from the port object. In practice, all of the real ports use the same names, and they should not include the &quot;LayoutTests&quot; prefix.
&gt; 
&gt; I think we should add an assert() into the TestResult constructor if names are expected not to be passed with LayoutTests/.  (They&apos;re currently not in this patch, but that would make the class easier to use.
&gt;

The design is the way it is intentionally, to allow for situations like the one Chrome used to have where you might have multiple directories of test files. Apart from the &quot;test&quot; port using a directory that doesn&apos;t have &quot;LayoutTests&quot; in the name, this feature is currently unused AFAIK, but I&apos;m a bit reluctant to remove it and lose the functionality since there are clear uses for it.
 
&gt; &gt; &gt; Tools/Scripts/webkitpy/common/net/layouttestresults.py:138
&gt; &gt; &gt; +        return self.tests_matching_failure_types(failure_types)
&gt; &gt; 
&gt; &gt; Do you want to include any tests that had missing or incorrect expectations files?
&gt; 
&gt; No.  The previous failing_tests() logic didn&apos;t, and I didn&apos;t want to change it.
&gt; 

Okay. If you are trying to map some sort of legacy notion of failing, I can see how a generic failed() method on TestResult might not be helpful.

&gt; &gt; Also, you shouldn&apos;t be looking at test_failures values at all. You should look at the &quot;type&quot; field on the TestResult object and compare it to one of the test_expectations constants (e.g., TEXT, IMAGE, CRASH, etc.).
&gt; 
&gt; That&apos;s much less specific, but that could be easier.
&gt;

It depends on what you&apos;re trying to do. It matches arguably the most common case, which is &quot;how should this failure be suppressed&quot; (or classified).

&gt; &gt; &gt; Tools/Scripts/webkitpy/tool/commands/rebaseline.py:92
&gt; &gt; &gt; +        failing_tests = build.layout_test_results().tests_matching_failure_types([test_failures.FailureTextMismatch])
&gt; &gt; 
&gt; &gt; I don&apos;t know the context of this routine but (a) it seems like &quot;FailureTextMismatch&quot; is probably the wrong value to be looking for and (b) you should be checking against result.type == test_expectations.TEXT instead of looking at the failure types directly (as per above).
&gt; 
&gt; I can try that in a later patch.  But I don&apos;t think that will work as you expect.   .type is only one value.  What happens when a test fails both image and text?  .type only reflects one of those.

No, type is classified based on all of the failures that happened. So, if it failed both image and text diffs, it gets classified as IMAGE_PLUS_TEXT. If it failed the text diff and the image expectations were missing, it gets classified as MISSING (since who knows if the text result is even meaningful)? Similarly, TIMEOUT and CRASH trump everything else.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329315</commentid>
    <comment_count>18</comment_count>
    <who name="Dirk Pranke">dpranke</who>
    <bug_when>2011-01-04 17:23:31 -0800</bug_when>
    <thetext>(In reply to comment #15)
&gt; (In reply to comment #12)
&gt; &gt; (In reply to comment #10)
&gt; &gt; &gt; Right, the three layers could be:
&gt; &gt; &gt; 
&gt; &gt; &gt; 1) application-specific (webkit-patch, new-run-webkit-tests, etc)
&gt; &gt; &gt; 2) shared, but webkit-specific
&gt; &gt; &gt; 3) shared, and non-webkit specific
&gt; &gt; &gt; 
&gt; &gt; &gt; Model classes about test results seem to go in layer 2.  Classes for talking to buildbot (generally) seem to go in layer 3.  Controller classes for executing DumpRenderTree and creating models of the results seems to go in layer 1.
&gt; &gt; 
&gt; &gt; This way of looking at things is a little weird to me. I would rather see stuff grouped by functionality rather than by visibility. i.e., layout_tests/public/foo rather than public/layout_tests/foo. Although either of these things seems like a bit of a hack, and we should just be using python&apos;s normal visibility mechanisms to control things.
&gt; 
&gt; I&apos;m not sure what you mean.

Meaning you use the __init__.py files to control what symbols are exported outside of a directory, and you use naming conventions like single and double underscores otherwise to indicate visibility.

&gt; 
&gt; &gt; Also, I&apos;m not sure what the current (not intended) definition of &quot;common&quot; is - whether it is (1), (2), (3), or some mixture of all of them?
&gt; 
&gt; common was stuff that was pulled out of the original bugzilla-tool to be shared by other python code.  bugzilla-tool became webkit-patch and thus the tool/ directory.  :)
&gt; 
&gt; I think common is a mixture of a lot of things these days.
&gt; 
&gt; &gt; I would be inclined to make the buildbot stuff its own top-level package rather than putting it under &quot;common&quot;, as I probably would with some of the other things in common.net.
&gt; 
&gt; buildbot depends on stuff in common.net (and is already a package, although not a very useful one).

I think that&apos;s fine, since webkitpy/buildbot should have no trouble importing webkitpy/common/net .</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329321</commentid>
    <comment_count>19</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-04 17:27:16 -0800</bug_when>
    <thetext>(In reply to comment #17)
&gt; &gt; I can try that in a later patch.  But I don&apos;t think that will work as you expect.   .type is only one value.  What happens when a test fails both image and text?  .type only reflects one of those.
&gt; 
&gt; No, type is classified based on all of the failures that happened. So, if it failed both image and text diffs, it gets classified as IMAGE_PLUS_TEXT. If it failed the text diff and the image expectations were missing, it gets classified as MISSING (since who knows if the text result is even meaningful)? Similarly, TIMEOUT and CRASH trump everything else.

We&apos;d still have to write some sort fo comparison function which knew how to take TEXT and compare it with IMAGE_PLUS_TEXT.  Perhaps such already exists, but the code I was dealing with wanted to be able to check for various types of failures.  passing TEXT would be fine, but having to compare type against TEXT and IMAGE_PLUS_TEXT seems error-prone.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329324</commentid>
    <comment_count>20</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-04 17:28:28 -0800</bug_when>
    <thetext>(In reply to comment #18)
&gt; &gt; &gt; This way of looking at things is a little weird to me. I would rather see stuff grouped by functionality rather than by visibility. i.e., layout_tests/public/foo rather than public/layout_tests/foo. Although either of these things seems like a bit of a hack, and we should just be using python&apos;s normal visibility mechanisms to control things.
&gt; &gt; 
&gt; &gt; I&apos;m not sure what you mean.
&gt; 
&gt; Meaning you use the __init__.py files to control what symbols are exported outside of a directory, and you use naming conventions like single and double underscores otherwise to indicate visibility.

Yes, I support such.  Although we&apos;ve had some confusion with such, see the recent removal of import api from Checkout/__init__.py.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329332</commentid>
    <comment_count>21</comment_count>
    <who name="Dirk Pranke">dpranke</who>
    <bug_when>2011-01-04 17:38:09 -0800</bug_when>
    <thetext>(In reply to comment #16)
&gt; 
&gt; This stuff definitely needs more iteration.
&gt; 
&gt; I find the TestResults stuff very cumbersome to use.  The fact that I have to import 3 different packages from layout_tests just to use it is one problem.
&gt;

I agree completely. It&apos;s not designed to be used outside of the way it was used in NRWT. The troubles you&apos;re running into are small compared to the issues James ran into with the rebaselining work. It&apos;s not yet clear what a better solution would be, though. (There are other bugs open for this).

&gt; I think LayoutTestResults needs to move to being in the same package as the rest of this stuff and then it will make more sense.
&gt; 

Probably. It&apos;s probably not a good thing that the code parsing the HTML is separate from the code generating the HTML.

&gt; I think .type is not a complete solution for replacing teh type(FailureClass) comparisons I had to do in this code (I did this based on other examples in the codebase).
&gt;

Perhaps.  I haven&apos;t really looked at the pre-NRWT code yet, so I can&apos;t say how much of this is just remapping old concepts versus trying to handle multiple different needs. The rebaselining tools may need some additional information, but even then that information might be best provided by methods on the TestResult rather than by exposing the underlying failure instances.

&gt; I like the idea of passing around Failure classes, since they can contain more than just a name/number.

They can contain more than a name/number, but I have yet to see a compelling example for how that additional data would be programmatically used (as opposed to just displayed to the user).

&gt;  however I would like to shorten the names of the various failure types so they&apos;re easier to use.

I would be happy to do this as a part of restructuring this code and figuring out what things should be public/shared.

&gt; I also think that it should be possible to do a one-to-many (or many-to-one?) mapping from test expectations enums to failure classes.

The current mapping, depending on how you look at it, is many-to-many. In order to make it 1:many for enum-&gt;failure classes you have to make some assumptions. For example, you should ignore the text diff if the image expectation was missing. I don&apos;t think there are any alternatives that would get to you to 1:many in that sense. 

In a different sense, the relation of (set of failures) -&gt; enum (the overall relation, if you will), is currently many:1, in that a given set of failures will only ever produce one unique result. 

In all of the uses I&apos;ve seen so far, this has been fine and simplifies the client code greatly, since the caller almost always only cares about the overall classification of the test, rather than the specifics of how each check ran.

I am certainly open to other designs, however.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329334</commentid>
    <comment_count>22</comment_count>
    <who name="Dirk Pranke">dpranke</who>
    <bug_when>2011-01-04 17:39:45 -0800</bug_when>
    <thetext>(In reply to comment #19)
&gt; We&apos;d still have to write some sort fo comparison function which knew how to take TEXT and compare it with IMAGE_PLUS_TEXT.  Perhaps such already exists, but the code I was dealing with wanted to be able to check for various types of failures.  passing TEXT would be fine, but having to compare type against TEXT and IMAGE_PLUS_TEXT seems error-prone.

Why are you trying to compare TEXT to IMAGE_PLUS_TEXT ? What is the larger problem you&apos;re trying to solve?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329335</commentid>
    <comment_count>23</comment_count>
    <who name="Dirk Pranke">dpranke</who>
    <bug_when>2011-01-04 17:41:05 -0800</bug_when>
    <thetext>(In reply to comment #20)
&gt; (In reply to comment #18)
&gt; &gt; &gt; &gt; This way of looking at things is a little weird to me. I would rather see stuff grouped by functionality rather than by visibility. i.e., layout_tests/public/foo rather than public/layout_tests/foo. Although either of these things seems like a bit of a hack, and we should just be using python&apos;s normal visibility mechanisms to control things.
&gt; &gt; &gt; 
&gt; &gt; &gt; I&apos;m not sure what you mean.
&gt; &gt; 
&gt; &gt; Meaning you use the __init__.py files to control what symbols are exported outside of a directory, and you use naming conventions like single and double underscores otherwise to indicate visibility.
&gt; 
&gt; Yes, I support such.  Although we&apos;ve had some confusion with such, see the recent removal of import api from Checkout/__init__.py.

Yes. The problem there (as far as I understand it) was that we were mixing both models. You shouldn&apos;t both export stuff in __init__.py and allow people to import modules in the package directly.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329758</commentid>
    <comment_count>24</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-05 13:35:01 -0800</bug_when>
    <thetext>Committed r75099: &lt;http://trac.webkit.org/changeset/75099&gt;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329844</commentid>
    <comment_count>25</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-05 15:57:17 -0800</bug_when>
    <thetext>Committed r75113: &lt;http://trac.webkit.org/changeset/75113&gt;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>329952</commentid>
    <comment_count>26</comment_count>
    <who name="Eric Seidel (no email)">eric</who>
    <bug_when>2011-01-05 18:15:20 -0800</bug_when>
    <thetext>Committed r75125: &lt;http://trac.webkit.org/changeset/75125&gt;</thetext>
  </long_desc>
      
          <attachment
              isobsolete="0"
              ispatch="1"
              isprivate="0"
          >
            <attachid>77778</attachid>
            <date>2011-01-02 12:16:15 -0800</date>
            <delta_ts>2011-01-04 15:52:07 -0800</delta_ts>
            <desc>Patch</desc>
            <filename>bug-51802-20110102121607.patch</filename>
            <type>text/plain</type>
            <size>18261</size>
            <attacher name="Eric Seidel (no email)">eric</attacher>
            
              <data encoding="base64">ZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBiL1Rvb2xzL0NoYW5nZUxvZwppbmRleCAxNmZi
OGU4MDAxNDRkOWEzZjA1ODM2MDdhNDZkNDg2OGMzZjE4YjE1Li5mZDI5OGFmYWRkMTEyZTVlNmE2
MDAzY2Q5NDc0MjliYzcyNTI0ZWZjIDEwMDY0NAotLS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIv
VG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsNDMgQEAKKzIwMTEtMDEtMDIgIEVyaWMgU2VpZGVs
ICA8ZXJpY0B3ZWJraXQub3JnPgorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEp
LgorCisgICAgICAgIE1vdmUgTGF5b3V0VGVzdFJlc3VsdHMgb3ZlciB0byBuZXctcnVuLXdlYmtp
dC10ZXN0cyBUZXN0UmVzdWx0IGFyY2hpdGVjdHVyZQorICAgICAgICBodHRwczovL2J1Z3Mud2Vi
a2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9NTE4MDIKKworICAgICAgICBJJ20gbm90IHRoZSBiaWdn
ZXN0IGZhbiBvZiB0aGUgdGVzdF9mYWlsdXJlcyBvciB0ZXN0X3Jlc3VsdHMgY2xhc3NlcywKKyAg
ICAgICAgYnV0IGl0J3MgYmV0dGVyIHRvIGhhdmUgb25lIHNoYXJlZCBzZXQgb2YgY2xhc3Nlcywg
dGhhbiBzZXBhcmF0ZSBvbmVzCisgICAgICAgIGZvciBuZXcgdnMuIG9sZCBydW4td2Via2l0LXRl
c3QgcmVzdWx0IGhhbmRsaW5nLgorCisgICAgICAgIFRoaXMgbW92ZXMgdGhlIE9SV1QgcmVzdWx0
cyBjbGFzcyAiTGF5b3V0VGVzdFJlc3VsdHMiIG92ZXIgdG8gdXNpbmcKKyAgICAgICAgVGVzdFJl
c3VsdCBhbmQgVGVzdEZhaWx1cmUgY2xhc3NlcywgbWFraW5nIGl0IGVhc3kgZm9yIHVzIHRvCisg
ICAgICAgIG1ha2UgYWxsIG91ciBzaGVyaWZmLWJvdCBhbmQgb3RoZXIgd2Via2l0cHkgY29kZSBO
UldUIHJlYWR5LgorCisgICAgICAgIFRoaXMgYWxzbyBtYWtlcyBpdCBhIHRyaXZpYWwgcGF0Y2gg
dG8gZ2VuZXJhdGUgcmVzdWx0cy5qc29uIGluZm9ybWF0aW9uCisgICAgICAgIGZyb20gT1JXVCBy
ZXN1bHRzLmh0bWwgZmlsZXMgKGZvciBmbGFreSB0ZXN0IGFuYWx5c2lzLCBldGMuKSBhcyB3ZWxs
CisgICAgICAgIGFzIG1ha2luZyBpdCBhIG9uZS1saW5lciB0byByZXBvcnQgdGVzdCBmYWlsdXJl
IHR5cGVzIHdoZW4gdGhlCisgICAgICAgIGNvbW1pdC1xdWV1ZSBzZWVzIGZsYWt5IHRlc3RzLgor
CisgICAgICAgIFRoaXMgcGF0Y2ggdHJpZWQgbm90IHRvIGFkZCBuZXcgZnVuY3Rpb25hbGl0eSwg
YnV0IG9ubHkgdG8gcmVwbGFjZQorICAgICAgICB0aGUgZ3V0cyBvZiBMYXlvdXRUZXN0UmVzdWx0
cywgd2hpbGUgYWRkaW5nIHVuaXQgdGVzdHMgYW5kIGhvcGluZworICAgICAgICBub3QgdG8gYnJl
YWsgYW55dGhpbmcuCisKKyAgICAgICAgSSBhbHNvIG1vdmVkIGNhbGxlcnMgd2hpY2ggYXNzdW1l
ZCBVc2VyLnByb21wdCogd2VyZSBzdGF0aWMvY2xhc3MgbWV0aG9kcworICAgICAgICB0byB1c2lu
ZyB0aGVtIGFzIGluc3RhbmNlIG1ldGhvZHMgKHNpbmNlIHdlJ2xsIGV2ZW50dWFsbHkgd2FudCB0
byBtYWtlIHRoZW0gc3VjaCkuCisKKyAgICAgICAgSW4gdGhlIHByb2Nlc3Mgb2YgcmUtd3JpdGlu
ZyB0aGluZ3MsIEkgYnJva2UgdGhlIHJlYmFzZWxpbmUgY29tbWFuZCwgc28gSSB3cm90ZQorICAg
ICAgICBhIHVuaXQgdGVzdCB0byBjYXRjaCBteSBicmVha2FnZSB3ZXJlIEkgZG8gZG8gc28gYWdh
aW4gaW4gdGhlIGZ1dHVyZS4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvY29tbW9uL25l
dC9idWlsZGJvdC9idWlsZGJvdF91bml0dGVzdC5weToKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtp
dHB5L2NvbW1vbi9uZXQvbGF5b3V0dGVzdHJlc3VsdHMucHk6CisgICAgICAgICogU2NyaXB0cy93
ZWJraXRweS9jb21tb24vbmV0L2xheW91dHRlc3RyZXN1bHRzX3VuaXR0ZXN0LnB5OgorICAgICAg
ICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfZmFp
bHVyZXMucHk6CisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0
X3BhY2thZ2UvdGVzdF9yZXN1bHRzLnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvdG9v
bC9jb21tYW5kcy9xdWVyaWVzLnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvdG9vbC9j
b21tYW5kcy9yZWJhc2VsaW5lLnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvdG9vbC9j
b21tYW5kcy9yZWJhc2VsaW5lX3VuaXR0ZXN0LnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0
cHkvdG9vbC9tb2NrdG9vbC5weToKKwogMjAxMC0xMi0zMSAgQWRhbSBCYXJ0aCAgPGFiYXJ0aEB3
ZWJraXQub3JnPgogCiAgICAgICAgIFJldmlld2VkIGJ5IEFyaXlhIEhpZGF5YXQuCmRpZmYgLS1n
aXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9uZXQvYnVpbGRib3QvYnVpbGRib3Rf
dW5pdHRlc3QucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9uZXQvYnVpbGRib3Qv
YnVpbGRib3RfdW5pdHRlc3QucHkKaW5kZXggYTEwZTQzMmU2N2VjMjVlYzQ1NDY3YTZhMzZhNjRk
YTNkMzYyNDc1NC4uZDlmYzYwNWE5NDgyMTgxZTA3NTdmOTFjNDIyZGU4NGQwNWFiMzIxZCAxMDA2
NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9jb21tb24vbmV0L2J1aWxkYm90L2J1aWxk
Ym90X3VuaXR0ZXN0LnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvY29tbW9uL25ldC9i
dWlsZGJvdC9idWlsZGJvdF91bml0dGVzdC5weQpAQCAtMzAsMTAgKzMwLDE2IEBAIGltcG9ydCB1
bml0dGVzdAogCiBmcm9tIHdlYmtpdHB5LmNvbW1vbi5uZXQubGF5b3V0dGVzdHJlc3VsdHMgaW1w
b3J0IExheW91dFRlc3RSZXN1bHRzCiBmcm9tIHdlYmtpdHB5LmNvbW1vbi5uZXQuYnVpbGRib3Qg
aW1wb3J0IEJ1aWxkQm90LCBCdWlsZGVyLCBCdWlsZAorZnJvbSB3ZWJraXRweS5sYXlvdXRfdGVz
dHMubGF5b3V0X3BhY2thZ2UgaW1wb3J0IHRlc3RfcmVzdWx0cworZnJvbSB3ZWJraXRweS5sYXlv
dXRfdGVzdHMubGF5b3V0X3BhY2thZ2UgaW1wb3J0IHRlc3RfZmFpbHVyZXMKIGZyb20gd2Via2l0
cHkudGhpcmRwYXJ0eS5CZWF1dGlmdWxTb3VwIGltcG9ydCBCZWF1dGlmdWxTb3VwCiAKIAogY2xh
c3MgQnVpbGRlclRlc3QodW5pdHRlc3QuVGVzdENhc2UpOgorICAgIGRlZiBfbW9ja190ZXN0X3Jl
c3VsdChzZWxmLCB0ZXN0bmFtZSk6CisgICAgICAgIGZhaWx1cmVzID0gW3Rlc3RfZmFpbHVyZXMu
RmFpbHVyZVRleHRNaXNtYXRjaCgpXQorICAgICAgICByZXR1cm4gdGVzdF9yZXN1bHRzLlRlc3RS
ZXN1bHQodGVzdG5hbWUsIGZhaWx1cmVzLCB0ZXN0X3J1bl90aW1lPU5vbmUsIHRvdGFsX3RpbWVf
Zm9yX2FsbF9kaWZmcz1Ob25lLCB0aW1lX2Zvcl9kaWZmcz1Ob25lKQorCiAgICAgZGVmIF9pbnN0
YWxsX2ZldGNoX2J1aWxkKHNlbGYsIGZhaWx1cmUpOgogICAgICAgICBkZWYgX21vY2tfZmV0Y2hf
YnVpbGQoYnVpbGRfbnVtYmVyKToKICAgICAgICAgICAgIGJ1aWxkID0gQnVpbGQoCkBAIC00Miw4
ICs0OCw4IEBAIGNsYXNzIEJ1aWxkZXJUZXN0KHVuaXR0ZXN0LlRlc3RDYXNlKToKICAgICAgICAg
ICAgICAgICByZXZpc2lvbj1idWlsZF9udW1iZXIgKyAxMDAwLAogICAgICAgICAgICAgICAgIGlz
X2dyZWVuPWJ1aWxkX251bWJlciA8IDQKICAgICAgICAgICAgICkKLSAgICAgICAgICAgIHBhcnNl
ZF9yZXN1bHRzID0ge0xheW91dFRlc3RSZXN1bHRzLmZhaWxfa2V5OiBmYWlsdXJlKGJ1aWxkX251
bWJlcil9Ci0gICAgICAgICAgICBidWlsZC5fbGF5b3V0X3Rlc3RfcmVzdWx0cyA9IExheW91dFRl
c3RSZXN1bHRzKHBhcnNlZF9yZXN1bHRzKQorICAgICAgICAgICAgcmVzdWx0cyA9IFtzZWxmLl9t
b2NrX3Rlc3RfcmVzdWx0KHRlc3RuYW1lKSBmb3IgdGVzdG5hbWUgaW4gZmFpbHVyZShidWlsZF9u
dW1iZXIpXQorICAgICAgICAgICAgYnVpbGQuX2xheW91dF90ZXN0X3Jlc3VsdHMgPSBMYXlvdXRU
ZXN0UmVzdWx0cyhyZXN1bHRzKQogICAgICAgICAgICAgcmV0dXJuIGJ1aWxkCiAgICAgICAgIHNl
bGYuYnVpbGRlci5fZmV0Y2hfYnVpbGQgPSBfbW9ja19mZXRjaF9idWlsZAogCmRpZmYgLS1naXQg
YS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9uZXQvbGF5b3V0dGVzdHJlc3VsdHMucHkg
Yi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9uZXQvbGF5b3V0dGVzdHJlc3VsdHMucHkK
aW5kZXggMTVlOTVjZWFjMjVlNDI2YWY2NTJmODY5ZmMwY2E3OTlmMDY4ZTcxNC4uZmE4YzM5ZGFk
OTg4YTYwMzJlZTRhMjM0MGE4MWY1N2I3MTc2MTc4NCAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0
cy93ZWJraXRweS9jb21tb24vbmV0L2xheW91dHRlc3RyZXN1bHRzLnB5CisrKyBiL1Rvb2xzL1Nj
cmlwdHMvd2Via2l0cHkvY29tbW9uL25ldC9sYXlvdXR0ZXN0cmVzdWx0cy5weQpAQCAtMjcsOCAr
MjcsMTEgQEAKICMgT0YgVEhJUyBTT0ZUV0FSRSwgRVZFTiBJRiBBRFZJU0VEIE9GIFRIRSBQT1NT
SUJJTElUWSBPRiBTVUNIIERBTUFHRS4KICMKICMgQSBtb2R1bGUgZm9yIHBhcnNpbmcgcmVzdWx0
cy5odG1sIGZpbGVzIGdlbmVyYXRlZCBieSBvbGQtcnVuLXdlYmtpdC10ZXN0cworIyBUaGlzIGNs
YXNzIGlzIG9uZSBiaWcgaGFjayBhbmQgb25seSBuZWVkcyB0byBleGlzdCB1bnRpbCB3ZSB0cmFu
c2l0aW9uIHRvIG5ldy1ydW4td2Via2l0LXRlc3RzLgogCiBmcm9tIHdlYmtpdHB5LnRoaXJkcGFy
dHkuQmVhdXRpZnVsU291cCBpbXBvcnQgQmVhdXRpZnVsU291cCwgU291cFN0cmFpbmVyCitmcm9t
IHdlYmtpdHB5LmxheW91dF90ZXN0cy5sYXlvdXRfcGFja2FnZSBpbXBvcnQgdGVzdF9yZXN1bHRz
Citmcm9tIHdlYmtpdHB5LmxheW91dF90ZXN0cy5sYXlvdXRfcGFja2FnZSBpbXBvcnQgdGVzdF9m
YWlsdXJlcwogCiAKICMgRklYTUU6IFRoaXMgc2hvdWxkIGJlIHVuaWZpZWQgd2l0aCBhbGwgdGhl
IGxheW91dCB0ZXN0IHJlc3VsdHMgY29kZSBpbiB0aGUgbGF5b3V0X3Rlc3RzIHBhY2thZ2UKQEAg
LTU3LDM2ICs2MCw3OSBAQCBjbGFzcyBMYXlvdXRUZXN0UmVzdWx0cyhvYmplY3QpOgogICAgIF0K
IAogICAgIEBjbGFzc21ldGhvZAorICAgIGRlZiBfZmFpbHVyZXNfZnJvbV9mYWlsX3JvdyhzZWxm
LCByb3cpOgorICAgICAgICAjIFRoZSBmaXJzdCBhbmNob3Igc2hvdWxkIGhhdmUgYWxyZWFkeSBi
ZWVuIGZldGNoZWQgYnkgdGhlIGNhbGxlciwKKyAgICAgICAgIyB3ZSBsb29rIGF0IGFsbCByZW1h
aW5pbmcgYW5jaG9ycyBpbiB0aGlzIHJvdywgYW5kIGd1ZXNzIHdoYXQgdHlwZQorICAgICAgICAj
IG9mIG5ldy1ydW4td2Via2l0LXRlc3QgZmFpbHVyZXMgdGhleSBlcXVhdGUgdG8uCisgICAgICAg
IGZhaWx1cmVzID0gc2V0KCkKKyAgICAgICAgZm9yIGFuY2hvciBpbiByb3cuZmluZEFsbCgiYSIp
OgorICAgICAgICAgICAgYW5jaG9yX3RleHQgPSBhbmNob3Iuc3RyaW5nCisgICAgICAgICAgICBp
ZiBhbmNob3JfdGV4dCBpbiBbImV4cGVjdGVkIGltYWdlIiwgImltYWdlIGRpZmZzIl0gb3IgYW5j
aG9yX3RleHQuY29udGFpbnMoJyUnKToKKyAgICAgICAgICAgICAgICBmYWlsdXJlcy5hZGQodGVz
dF9mYWlsdXJlcy5GYWlsdXJlSW1hZ2VIYXNoTWlzbWF0Y2goKSkKKyAgICAgICAgICAgIGVsaWYg
YW5jaG9yX3RleHQgaW4gWyJleHBlY3RlZCIsICJhY3R1YWwiLCAiZGlmZiIsICJwcmV0dHkgZGlm
ZiJdOgorICAgICAgICAgICAgICAgIGZhaWx1cmVzLmFkZCh0ZXN0X2ZhaWx1cmVzLkZhaWx1cmVU
ZXh0TWlzbWF0Y2goKSkKKyAgICAgICAgICAgIGVsc2U6CisgICAgICAgICAgICAgICAgbG9nKCJV
bmhhbmRsZWQgbGluayB0ZXh0IGluIHJlc3VsdHMuaHRtbCBwYXJzaW5nOiAlcy4gIFBsZWFzZSBm
aWxlIGEgYnVnIGFnYWluc3Qgd2Via2l0cHkuIiAlIGFuY2hvcl90ZXh0KQorICAgICAgICAjIEZJ
WE1FOiBJdHMgcG9zc2libGUgdGhlIHJvdyBjb250YWluZWQgbm8gbGlua3MgZHVlIHRvIE9SV1Qg
YnJva2VuZXNzLgorICAgICAgICAjIFdlIHNob3VsZCBwcm9iYWJseSBhc3N1bWUgc29tZSB0eXBl
IG9mIGZhaWx1cmUgYW55d2F5LgorCisgICAgQGNsYXNzbWV0aG9kCisgICAgZGVmIF9mYWlsdXJl
c19mcm9tX3JvdyhjbHMsIHJvdywgdGFibGVfdGl0bGUpOgorICAgICAgICBpZiB0YWJsZV90aXRs
ZSA9PSBjbHMuZmFpbF9rZXk6CisgICAgICAgICAgICByZXR1cm4gY2xzLl9mYWlsdXJlc19mcm9t
X2ZhaWxfcm93KHJvdykKKyAgICAgICAgaWYgdGFibGVfdGl0bGUgPT0gY2xzLmNyYXNoX2tleToK
KyAgICAgICAgICAgIHJldHVybiBbdGVzdF9mYWlsdXJlcy5GYWlsdXJlQ3Jhc2goKV0KKyAgICAg
ICAgaWYgdGFibGVfdGl0bGUgPT0gY2xzLnRpbWVvdXRfa2V5OgorICAgICAgICAgICAgcmV0dXJu
IFt0ZXN0X2ZhaWx1cmVzLkZhaWx1cmVUaW1lb3V0KCldCisgICAgICAgIGlmIHRhYmxlX3RpdGxl
ID09IGNscy5taXNzaW5nX2tleToKKyAgICAgICAgICAgIHJldHVybiBbdGVzdF9mYWlsdXJlcy5G
YWlsdXJlTWlzc2luZ1Jlc3VsdCgpLCB0ZXN0X2ZhaWx1cmVzLkZhaWx1cmVNaXNzaW5nSW1hZ2VI
YXNoKCksIHRlc3RfZmFpbHVyZXMuRmFpbHVyZU1pc3NpbmdJbWFnZSgpXQorICAgICAgICByZXR1
cm4gTm9uZQorCisgICAgQGNsYXNzbWV0aG9kCisgICAgZGVmIF90ZXN0X3Jlc3VsdF9mcm9tX3Jv
dyhjbHMsIHJvdywgdGFibGVfdGl0bGUpOgorICAgICAgICB0ZXN0X25hbWUgPSB1bmljb2RlKHJv
dy5maW5kKCJhIikuc3RyaW5nKQorICAgICAgICBmYWlsdXJlcyA9IGNscy5fZmFpbHVyZXNfZnJv
bV9yb3cocm93LCB0YWJsZV90aXRsZSkKKyAgICAgICAgIyBUZXN0UmVzdWx0IGlzIGEgY2xhc3Mg
ZGVzaWduZWQgdG8gd29yayB3aXRoIG5ldy1ydW4td2Via2l0LXRlc3RzLgorICAgICAgICAjIG9s
ZC1ydW4td2Via2l0LXRlc3RzIGRvZXMgbm90IHNhdmUgcXVpdGUgZW5vdWdoIGluZm9ybWF0aW9u
IGluIHJlc3VsdHMuaHRtbCBmb3IgdXMgdG8gcGFyc2UuCisgICAgICAgICMgRklYTUU6IEl0J3Mg
dW5jbGVhciBpZiB0ZXN0X25hbWUgc2hvdWxkIGluY2x1ZGUgTGF5b3V0VGVzdHMgb3Igbm90Lgor
ICAgICAgICByZXR1cm4gdGVzdF9yZXN1bHRzLlRlc3RSZXN1bHQodGVzdF9uYW1lLCBmYWlsdXJl
cywgdGVzdF9ydW5fdGltZT1Ob25lLCB0b3RhbF90aW1lX2Zvcl9hbGxfZGlmZnM9Tm9uZSwgdGlt
ZV9mb3JfZGlmZnM9Tm9uZSkKKworICAgIEBjbGFzc21ldGhvZAorICAgIGRlZiBfcGFyc2VfcmVz
dWx0c190YWJsZShjbHMsIHRhYmxlKToKKyAgICAgICAgdGFibGVfdGl0bGUgPSB1bmljb2RlKHRh
YmxlLmZpbmRQcmV2aW91c1NpYmxpbmcoInAiKS5zdHJpbmcpCisgICAgICAgIGlmIHRhYmxlX3Rp
dGxlIG5vdCBpbiBjbHMuZXhwZWN0ZWRfa2V5czoKKyAgICAgICAgICAgICMgVGhpcyBFeGNlcHRp
b24gc2hvdWxkIG9ubHkgZXZlciBiZSBoaXQgaWYgcnVuLXdlYmtpdC10ZXN0cyBjaGFuZ2VzIGl0
cyByZXN1bHRzLmh0bWwgZm9ybWF0LgorICAgICAgICAgICAgcmFpc2UgRXhjZXB0aW9uKCJVbmhh
bmRsZWQgdGl0bGU6ICVzIiAlIHRhYmxlX3RpdGxlKQorICAgICAgICAjIElnbm9yZSBzdGRlcnIg
ZmFpbHVyZXMuICBFdmVyeW9uZSBpZ25vcmVzIHRoZW0gYW55d2F5LgorICAgICAgICBpZiB0YWJs
ZV90aXRsZSA9PSBjbHMuc3RkZXJyX2tleToKKyAgICAgICAgICAgIHJldHVybiBbXQorICAgICAg
ICAjIEZJWE1FOiBXZSBtaWdodCBlbmQgd2l0aCB0d28gVGVzdFJlc3VsdHMgb2JqZWN0IGZvciB0
aGUgc2FtZSB0ZXN0IGlmIGl0IGFwcGVhcnMgaW4gbW9yZSB0aGFuIG9uZSByb3cuCisgICAgICAg
IHJldHVybiBbY2xzLl90ZXN0X3Jlc3VsdF9mcm9tX3Jvdyhyb3csIHRhYmxlX3RpdGxlKSBmb3Ig
cm93IGluIHRhYmxlLmZpbmRBbGwoInRyIildCisKKyAgICBAY2xhc3NtZXRob2QKICAgICBkZWYg
X3BhcnNlX3Jlc3VsdHNfaHRtbChjbHMsIHBhZ2UpOgotICAgICAgICBpZiBub3QgcGFnZToKLSAg
ICAgICAgICAgIHJldHVybiBOb25lCi0gICAgICAgIHBhcnNlZF9yZXN1bHRzID0ge30KICAgICAg
ICAgdGFibGVzID0gQmVhdXRpZnVsU291cChwYWdlKS5maW5kQWxsKCJ0YWJsZSIpCi0gICAgICAg
IGZvciB0YWJsZSBpbiB0YWJsZXM6Ci0gICAgICAgICAgICB0YWJsZV90aXRsZSA9IHVuaWNvZGUo
dGFibGUuZmluZFByZXZpb3VzU2libGluZygicCIpLnN0cmluZykKLSAgICAgICAgICAgIGlmIHRh
YmxlX3RpdGxlIG5vdCBpbiBjbHMuZXhwZWN0ZWRfa2V5czoKLSAgICAgICAgICAgICAgICAjIFRo
aXMgRXhjZXB0aW9uIHNob3VsZCBvbmx5IGV2ZXIgYmUgaGl0IGlmIHJ1bi13ZWJraXQtdGVzdHMg
Y2hhbmdlcyBpdHMgcmVzdWx0cy5odG1sIGZvcm1hdC4KLSAgICAgICAgICAgICAgICByYWlzZSBF
eGNlcHRpb24oIlVuaGFuZGxlZCB0aXRsZTogJXMiICUgdGFibGVfdGl0bGUpCi0gICAgICAgICAg
ICAjIFdlIG1pZ2h0IHdhbnQgdG8gdHJhbnNsYXRlIHRhYmxlIHRpdGxlcyBpbnRvIGlkZW50aWZp
ZXJzIGJlZm9yZSBzdG9yaW5nLgotICAgICAgICAgICAgcGFyc2VkX3Jlc3VsdHNbdGFibGVfdGl0
bGVdID0gW3VuaWNvZGUocm93LmZpbmQoImEiKS5zdHJpbmcpIGZvciByb3cgaW4gdGFibGUuZmlu
ZEFsbCgidHIiKV0KLQotICAgICAgICByZXR1cm4gcGFyc2VkX3Jlc3VsdHMKKyAgICAgICAgcmV0
dXJuIHN1bShbY2xzLl9wYXJzZV9yZXN1bHRzX3RhYmxlKHRhYmxlKSBmb3IgdGFibGUgaW4gdGFi
bGVzXSwgW10pCiAKICAgICBAY2xhc3NtZXRob2QKICAgICBkZWYgcmVzdWx0c19mcm9tX3N0cmlu
ZyhjbHMsIHN0cmluZyk6Ci0gICAgICAgIHBhcnNlZF9yZXN1bHRzID0gY2xzLl9wYXJzZV9yZXN1
bHRzX2h0bWwoc3RyaW5nKQotICAgICAgICBpZiBub3QgcGFyc2VkX3Jlc3VsdHM6CisgICAgICAg
IGlmIG5vdCBzdHJpbmc6CisgICAgICAgICAgICByZXR1cm4gTm9uZQorICAgICAgICB0ZXN0X3Jl
c3VsdHMgPSBjbHMuX3BhcnNlX3Jlc3VsdHNfaHRtbChzdHJpbmcpCisgICAgICAgIGlmIG5vdCB0
ZXN0X3Jlc3VsdHM6CiAgICAgICAgICAgICByZXR1cm4gTm9uZQotICAgICAgICByZXR1cm4gY2xz
KHBhcnNlZF9yZXN1bHRzKQorICAgICAgICByZXR1cm4gY2xzKHRlc3RfcmVzdWx0cykKIAotICAg
IGRlZiBfX2luaXRfXyhzZWxmLCBwYXJzZWRfcmVzdWx0cyk6Ci0gICAgICAgIHNlbGYuX3BhcnNl
ZF9yZXN1bHRzID0gcGFyc2VkX3Jlc3VsdHMKKyAgICBkZWYgX19pbml0X18oc2VsZiwgdGVzdF9y
ZXN1bHRzKToKKyAgICAgICAgc2VsZi5fdGVzdF9yZXN1bHRzID0gdGVzdF9yZXN1bHRzCiAKLSAg
ICBkZWYgcGFyc2VkX3Jlc3VsdHMoc2VsZik6Ci0gICAgICAgIHJldHVybiBzZWxmLl9wYXJzZWRf
cmVzdWx0cworICAgIGRlZiB0ZXN0X3Jlc3VsdHMoc2VsZik6CisgICAgICAgIHJldHVybiBzZWxm
Ll90ZXN0X3Jlc3VsdHMKIAotICAgIGRlZiByZXN1bHRzX21hdGNoaW5nX2tleXMoc2VsZiwgcmVz
dWx0X2tleXMpOgotICAgICAgICByZXR1cm4gc29ydGVkKHN1bShbdGVzdHMgZm9yIGtleSwgdGVz
dHMgaW4gc2VsZi5fcGFyc2VkX3Jlc3VsdHMuaXRlbXMoKSBpZiBrZXkgaW4gcmVzdWx0X2tleXNd
LCBbXSkpCisgICAgZGVmIHRlc3RzX21hdGNoaW5nX2ZhaWx1cmVfdHlwZXMoc2VsZiwgZmFpbHVy
ZV90eXBlcyk6CisgICAgICAgIHJldHVybiBbcmVzdWx0LmZpbGVuYW1lIGZvciByZXN1bHQgaW4g
c2VsZi5fdGVzdF9yZXN1bHRzIGlmIHJlc3VsdC5oYXNfZmFpbHVyZV9tYXRjaGluZ190eXBlcyhm
YWlsdXJlX3R5cGVzKV0KIAogICAgIGRlZiBmYWlsaW5nX3Rlc3RzKHNlbGYpOgotICAgICAgICBy
ZXR1cm4gc2VsZi5yZXN1bHRzX21hdGNoaW5nX2tleXMoW3NlbGYuZmFpbF9rZXksIHNlbGYuY3Jh
c2hfa2V5LCBzZWxmLnRpbWVvdXRfa2V5XSkKKyAgICAgICAgIyBUaGVzZSBzaG91bGQgbWF0Y2gg
dGhlICJmYWlsIiwgImNyYXNoIiwgYW5kICJ0aW1lb3V0IiBrZXlzLgorICAgICAgICBmYWlsdXJl
X3R5cGVzID0gW3Rlc3RfZmFpbHVyZXMuRmFpbHVyZVRleHRNaXNtYXRjaCwgdGVzdF9mYWlsdXJl
cy5GYWlsdXJlSW1hZ2VIYXNoTWlzbWF0Y2gsIHRlc3RfZmFpbHVyZXMuRmFpbHVyZUNyYXNoLCB0
ZXN0X2ZhaWx1cmVzLkZhaWx1cmVUaW1lb3V0XQorICAgICAgICByZXR1cm4gc2VsZi50ZXN0c19t
YXRjaGluZ19mYWlsdXJlX3R5cGVzKGZhaWx1cmVfdHlwZXMpCmRpZmYgLS1naXQgYS9Ub29scy9T
Y3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9uZXQvbGF5b3V0dGVzdHJlc3VsdHNfdW5pdHRlc3QucHkg
Yi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9uZXQvbGF5b3V0dGVzdHJlc3VsdHNfdW5p
dHRlc3QucHkKaW5kZXggODQ5MGVhZTNmYjUyYjc4MzdlNjQwZjgzOWYxNmZjMmI2Yzg4ZTc0NS4u
ZTBhMzVlODQ2M2M5YTA5OWIyM2UzMjljYTYyNmZlOGFiMWE1ZDU2NiAxMDA2NDQKLS0tIGEvVG9v
bHMvU2NyaXB0cy93ZWJraXRweS9jb21tb24vbmV0L2xheW91dHRlc3RyZXN1bHRzX3VuaXR0ZXN0
LnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvY29tbW9uL25ldC9sYXlvdXR0ZXN0cmVz
dWx0c191bml0dGVzdC5weQpAQCAtMjksNyArMjksOCBAQAogaW1wb3J0IHVuaXR0ZXN0CiAKIGZy
b20gd2Via2l0cHkuY29tbW9uLm5ldC5sYXlvdXR0ZXN0cmVzdWx0cyBpbXBvcnQgTGF5b3V0VGVz
dFJlc3VsdHMKLQorZnJvbSB3ZWJraXRweS5sYXlvdXRfdGVzdHMubGF5b3V0X3BhY2thZ2UgaW1w
b3J0IHRlc3RfcmVzdWx0cworZnJvbSB3ZWJraXRweS5sYXlvdXRfdGVzdHMubGF5b3V0X3BhY2th
Z2UgaW1wb3J0IHRlc3RfZmFpbHVyZXMKIAogY2xhc3MgTGF5b3V0VGVzdFJlc3VsdHNUZXN0KHVu
aXR0ZXN0LlRlc3RDYXNlKToKICAgICBfZXhhbXBsZV9yZXN1bHRzX2h0bWwgPSAiIiIKQEAgLTU3
LDE4ICs1OCwxMyBAQCBjbGFzcyBMYXlvdXRUZXN0UmVzdWx0c1Rlc3QodW5pdHRlc3QuVGVzdENh
c2UpOgogPC9odG1sPgogIiIiCiAKLSAgICBfZXhwZWN0ZWRfbGF5b3V0X3Rlc3RfcmVzdWx0cyA9
IHsKLSAgICAgICAgJ1Rlc3RzIHRoYXQgaGFkIHN0ZGVyciBvdXRwdXQ6JzogWwotICAgICAgICAg
ICAgJ2FjY2Vzc2liaWxpdHkvYXJpYS1hY3RpdmVkZXNjZW5kYW50LWNyYXNoLmh0bWwnLAotICAg
ICAgICBdLAotICAgICAgICAnVGVzdHMgdGhhdCBoYWQgbm8gZXhwZWN0ZWQgcmVzdWx0cyAocHJv
YmFibHkgbmV3KTonOiBbCi0gICAgICAgICAgICAnZmFzdC9yZXBhaW50L25vLWNhcmV0LXJlcGFp
bnQtaW4tbm9uLWNvbnRlbnQtZWRpdGFibGUtZWxlbWVudC5odG1sJywKLSAgICAgICAgXSwKLSAg
ICB9Ci0KICAgICBkZWYgdGVzdF9wYXJzZV9sYXlvdXRfdGVzdF9yZXN1bHRzKHNlbGYpOgorICAg
ICAgICBmYWlsdXJlcyA9IFt0ZXN0X2ZhaWx1cmVzLkZhaWx1cmVNaXNzaW5nUmVzdWx0KCksIHRl
c3RfZmFpbHVyZXMuRmFpbHVyZU1pc3NpbmdJbWFnZUhhc2goKSwgdGVzdF9mYWlsdXJlcy5GYWls
dXJlTWlzc2luZ0ltYWdlKCldCisgICAgICAgIHRlc3RuYW1lID0gJ2Zhc3QvcmVwYWludC9uby1j
YXJldC1yZXBhaW50LWluLW5vbi1jb250ZW50LWVkaXRhYmxlLWVsZW1lbnQuaHRtbCcKKyAgICAg
ICAgZXhwZWN0ZWRfcmVzdWx0cyA9IFt0ZXN0X3Jlc3VsdHMuVGVzdFJlc3VsdCh0ZXN0bmFtZSwg
ZmFpbHVyZXMsIHRlc3RfcnVuX3RpbWU9Tm9uZSwgdG90YWxfdGltZV9mb3JfYWxsX2RpZmZzPU5v
bmUsIHRpbWVfZm9yX2RpZmZzPU5vbmUpXQorCiAgICAgICAgIHJlc3VsdHMgPSBMYXlvdXRUZXN0
UmVzdWx0cy5fcGFyc2VfcmVzdWx0c19odG1sKHNlbGYuX2V4YW1wbGVfcmVzdWx0c19odG1sKQot
ICAgICAgICBzZWxmLmFzc2VydEVxdWFsKHNlbGYuX2V4cGVjdGVkX2xheW91dF90ZXN0X3Jlc3Vs
dHMsIHJlc3VsdHMpCisgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWwoZXhwZWN0ZWRfcmVzdWx0cywg
cmVzdWx0cykKIAogICAgIGRlZiB0ZXN0X3Jlc3VsdHNfZnJvbV9zdHJpbmcoc2VsZik6CiAgICAg
ICAgIHNlbGYuYXNzZXJ0RXF1YWwoTGF5b3V0VGVzdFJlc3VsdHMucmVzdWx0c19mcm9tX3N0cmlu
ZyhOb25lKSwgTm9uZSkKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0
X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfZmFpbHVyZXMucHkgYi9Ub29scy9TY3JpcHRzL3dl
YmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS90ZXN0X2ZhaWx1cmVzLnB5CmluZGV4
IDZkNTU3NjEwZmJmMWI2MzY0ZTQ4ZTgyNGQwMjFiNTJmZjc5YzcxZjguLmQ1MzdkNTBmYTk0NzI2
NDExZjhlNzU1NWVlN2I2Yjk5ZDUzZmUyOGIgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Vi
a2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfZmFpbHVyZXMucHkKKysrIGIv
VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2thZ2UvdGVzdF9m
YWlsdXJlcy5weQpAQCAtMzUsNiArMzUsOSBAQCBpbXBvcnQgdGVzdF9leHBlY3RhdGlvbnMKIGlt
cG9ydCBjUGlja2xlCiAKIAorIyBGSVhNRTogVGhpcyBpcyBiYWNrd2FyZHMuICBFYWNoIFRlc3RG
YWlsdXJlIHN1YmNsYXNzIHNob3VsZCBrbm93IHdoYXQKKyMgdGVzdF9leHBlY3RhdGlvbiB0eXBl
IGl0IGNvcnJlc3BvbmRzIHRvby4gIFRoZW4gdGhpcyBtZXRob2QganVzdAorIyBjb2xsZWN0cyB0
aGVtIGFsbCBmcm9tIHRoZSBmYWlsdXJlIGxpc3QgYW5kIHJldHVybnMgdGhlIHdvcnN0IG9uZS4K
IGRlZiBkZXRlcm1pbmVfcmVzdWx0X3R5cGUoZmFpbHVyZV9saXN0KToKICAgICAiIiJUYWtlcyBh
IHNldCBvZiB0ZXN0X2ZhaWx1cmVzIGFuZCByZXR1cm5zIHdoaWNoIHJlc3VsdCB0eXBlIGJlc3Qg
Zml0cwogICAgIHRoZSBsaXN0IG9mIGZhaWx1cmVzLiAiQmVzdCBmaXRzIiBtZWFucyB3ZSB1c2Ug
dGhlIHdvcnN0IHR5cGUgb2YgZmFpbHVyZS4KZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Vi
a2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcmVzdWx0cy5weSBiL1Rvb2xz
L1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcmVzdWx0
cy5weQppbmRleCAyNDE3ZmI3NTVjMjAyYjU5NzUzYmU0MjA0NjMyYTlmMDdlOTM0ZWEzLi44NjMw
NDM3OTk0MWE0ZjM4NGQzOWY1MWEzMmI1OWQ2YWZmOWUxMWY4IDEwMDY0NAotLS0gYS9Ub29scy9T
Y3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS90ZXN0X3Jlc3VsdHMu
cHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2th
Z2UvdGVzdF9yZXN1bHRzLnB5CkBAIC01Nyw1ICs1NywxMSBAQCBjbGFzcyBUZXN0UmVzdWx0KG9i
amVjdCk6CiAgICAgZGVmIF9fbmVfXyhzZWxmLCBvdGhlcik6CiAgICAgICAgIHJldHVybiBub3Qg
KHNlbGYgPT0gb3RoZXIpCiAKKyAgICBkZWYgaGFzX2ZhaWx1cmVfbWF0Y2hpbmdfdHlwZXMoc2Vs
ZiwgdHlwZXMpOgorICAgICAgICBmb3IgZmFpbHVyZSBpbiBzZWxmLmZhaWx1cmVzOgorICAgICAg
ICAgICAgaWYgdHlwZShmYWlsdXJlKSBpbiB0eXBlczoKKyAgICAgICAgICAgICAgICByZXR1cm4g
VHJ1ZQorICAgICAgICByZXR1cm4gRmFsc2UKKwogICAgIGRlZiBkdW1wcyhzZWxmKToKICAgICAg
ICAgcmV0dXJuIGNQaWNrbGUuZHVtcHMoc2VsZikKZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMv
d2Via2l0cHkvdG9vbC9jb21tYW5kcy9xdWVyaWVzLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRw
eS90b29sL2NvbW1hbmRzL3F1ZXJpZXMucHkKaW5kZXggZjA0ZjM4NDA4ZWVjMGQxZTRjZTY1ZGUz
YjI3YzExZTI4ODk3YmU0MS4uNzMzNzUxZWM3MzhkNjlkOTRhYTUwZTgxZDlkYmVmZGQxMDJiYTll
MCAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS90b29sL2NvbW1hbmRzL3F1ZXJp
ZXMucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS90b29sL2NvbW1hbmRzL3F1ZXJpZXMu
cHkKQEAgLTI3Miw3ICsyNzIsNyBAQCBjbGFzcyBGYWlsdXJlUmVhc29uKEFic3RyYWN0RGVjbGFy
YXRpdmVDb21tYW5kKToKICAgICAgICAgcHJpbnQgIiVzIGZhaWxpbmciICUgKHBsdXJhbGl6ZSgi
YnVpbGRlciIsIGxlbihyZWRfc3RhdHVzZXMpKSkKICAgICAgICAgYnVpbGRlcl9jaG9pY2VzID0g
W3N0YXR1c1sibmFtZSJdIGZvciBzdGF0dXMgaW4gcmVkX3N0YXR1c2VzXQogICAgICAgICAjIFdl
IGNvdWxkIG9mZmVyIGFuICJBbGwiIGNob2ljZSBoZXJlLgotICAgICAgICBjaG9zZW5fbmFtZSA9
IFVzZXIucHJvbXB0X3dpdGhfbGlzdCgiV2hpY2ggYnVpbGRlciB0byBkaWFnbm9zZToiLCBidWls
ZGVyX2Nob2ljZXMpCisgICAgICAgIGNob3Nlbl9uYW1lID0gc2VsZi5fdG9vbC51c2VyLnByb21w
dF93aXRoX2xpc3QoIldoaWNoIGJ1aWxkZXIgdG8gZGlhZ25vc2U6IiwgYnVpbGRlcl9jaG9pY2Vz
KQogICAgICAgICAjIEZJWE1FOiBwcm9tcHRfd2l0aF9saXN0IHNob3VsZCByZWFsbHkgdGFrZSBh
IHNldCBvZiBvYmplY3RzIGFuZCBhIHNldCBvZiBuYW1lcyBhbmQgdGhlbiByZXR1cm4gdGhlIG9i
amVjdC4KICAgICAgICAgZm9yIHN0YXR1cyBpbiByZWRfc3RhdHVzZXM6CiAgICAgICAgICAgICBp
ZiBzdGF0dXNbIm5hbWUiXSA9PSBjaG9zZW5fbmFtZToKQEAgLTM0NSw3ICszNDUsNyBAQCBjbGFz
cyBGaW5kRmxha3lUZXN0cyhBYnN0cmFjdERlY2xhcmF0aXZlQ29tbWFuZCk6CiAgICAgZGVmIF9i
dWlsZGVyX3RvX2FuYWx5emUoc2VsZik6CiAgICAgICAgIHN0YXR1c2VzID0gc2VsZi5fdG9vbC5i
dWlsZGJvdC5idWlsZGVyX3N0YXR1c2VzKCkKICAgICAgICAgY2hvaWNlcyA9IFtzdGF0dXNbIm5h
bWUiXSBmb3Igc3RhdHVzIGluIHN0YXR1c2VzXQotICAgICAgICBjaG9zZW5fbmFtZSA9IFVzZXIu
cHJvbXB0X3dpdGhfbGlzdCgiV2hpY2ggYnVpbGRlciB0byBhbmFseXplOiIsIGNob2ljZXMpCisg
ICAgICAgIGNob3Nlbl9uYW1lID0gc2VsZi5fdG9vbC51c2VyLnByb21wdF93aXRoX2xpc3QoIldo
aWNoIGJ1aWxkZXIgdG8gYW5hbHl6ZToiLCBjaG9pY2VzKQogICAgICAgICBmb3Igc3RhdHVzIGlu
IHN0YXR1c2VzOgogICAgICAgICAgICAgaWYgc3RhdHVzWyJuYW1lIl0gPT0gY2hvc2VuX25hbWU6
CiAgICAgICAgICAgICAgICAgcmV0dXJuIChzZWxmLl90b29sLmJ1aWxkYm90LmJ1aWxkZXJfd2l0
aF9uYW1lKGNob3Nlbl9uYW1lKSwgc3RhdHVzWyJidWlsdF9yZXZpc2lvbiJdKQpkaWZmIC0tZ2l0
IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS90b29sL2NvbW1hbmRzL3JlYmFzZWxpbmUucHkgYi9U
b29scy9TY3JpcHRzL3dlYmtpdHB5L3Rvb2wvY29tbWFuZHMvcmViYXNlbGluZS5weQppbmRleCA4
YzRiOTk3ZWU1ZGJiMmQ1NTE4MDVjOWZhZTU2ZjI2NGUxNDk1NjFlLi4zNGEzOThhODc1ODcwODU3
MjFkYTBkNjUxYTAzOGVmZmI3MWM1NWE2IDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtp
dHB5L3Rvb2wvY29tbWFuZHMvcmViYXNlbGluZS5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtp
dHB5L3Rvb2wvY29tbWFuZHMvcmViYXNlbGluZS5weQpAQCAtMzQsNiArMzQsNyBAQCBpbXBvcnQg
dXJsbGliCiBmcm9tIHdlYmtpdHB5LmNvbW1vbi5uZXQuYnVpbGRib3QgaW1wb3J0IEJ1aWxkQm90
CiBmcm9tIHdlYmtpdHB5LmNvbW1vbi5uZXQubGF5b3V0dGVzdHJlc3VsdHMgaW1wb3J0IExheW91
dFRlc3RSZXN1bHRzCiBmcm9tIHdlYmtpdHB5LmNvbW1vbi5zeXN0ZW0udXNlciBpbXBvcnQgVXNl
cgorZnJvbSB3ZWJraXRweS5sYXlvdXRfdGVzdHMubGF5b3V0X3BhY2thZ2UgaW1wb3J0IHRlc3Rf
ZmFpbHVyZXMKIGZyb20gd2Via2l0cHkubGF5b3V0X3Rlc3RzLnBvcnQgaW1wb3J0IGZhY3RvcnkK
IGZyb20gd2Via2l0cHkudG9vbC5ncmFtbWFyIGltcG9ydCBwbHVyYWxpemUKIGZyb20gd2Via2l0
cHkudG9vbC5tdWx0aWNvbW1hbmR0b29sIGltcG9ydCBBYnN0cmFjdERlY2xhcmF0aXZlQ29tbWFu
ZApAQCAtODgsNyArODksNyBAQCBjbGFzcyBSZWJhc2VsaW5lKEFic3RyYWN0RGVjbGFyYXRpdmVD
b21tYW5kKToKICAgICAgICAgc2h1dGlsLm1vdmUoZG93bmxvYWRlZF9maWxlLCBsb2NhbF9maWxl
KQogCiAgICAgZGVmIF90ZXN0c190b191cGRhdGUoc2VsZiwgYnVpbGQpOgotICAgICAgICBmYWls
aW5nX3Rlc3RzID0gYnVpbGQubGF5b3V0X3Rlc3RfcmVzdWx0cygpLnJlc3VsdHNfbWF0Y2hpbmdf
a2V5cyhbTGF5b3V0VGVzdFJlc3VsdHMuZmFpbF9rZXldKQorICAgICAgICBmYWlsaW5nX3Rlc3Rz
ID0gYnVpbGQubGF5b3V0X3Rlc3RfcmVzdWx0cygpLnRlc3RzX21hdGNoaW5nX2ZhaWx1cmVfdHlw
ZXMoW3Rlc3RfZmFpbHVyZXMuRmFpbHVyZVRleHRNaXNtYXRjaF0pCiAgICAgICAgIHJldHVybiBz
ZWxmLl90b29sLnVzZXIucHJvbXB0X3dpdGhfbGlzdCgiV2hpY2ggdGVzdChzKSB0byByZWJhc2Vs
aW5lOiIsIGZhaWxpbmdfdGVzdHMsIGNhbl9jaG9vc2VfbXVsdGlwbGU9VHJ1ZSkKIAogICAgIGRl
ZiBfcmVzdWx0c191cmxfZm9yX3Rlc3Qoc2VsZiwgYnVpbGQsIHRlc3QpOgpkaWZmIC0tZ2l0IGEv
VG9vbHMvU2NyaXB0cy93ZWJraXRweS90b29sL2NvbW1hbmRzL3JlYmFzZWxpbmVfdW5pdHRlc3Qu
cHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3Rvb2wvY29tbWFuZHMvcmViYXNlbGluZV91bml0
dGVzdC5weQppbmRleCBkNjU4MmE3MTVlNThiYWJmOTJiMWJiNzE1ZWIxODFiMzk5MzI1ZTY0Li43
OWU0Y2Y0YTk3YjYwMWY3MGRjZjYwZWE5NDM4Y2U4YTM3OWU2OWQ4IDEwMDY0NAotLS0gYS9Ub29s
cy9TY3JpcHRzL3dlYmtpdHB5L3Rvb2wvY29tbWFuZHMvcmViYXNlbGluZV91bml0dGVzdC5weQor
KysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3Rvb2wvY29tbWFuZHMvcmViYXNlbGluZV91bml0
dGVzdC5weQpAQCAtMjgsNyArMjgsMTkgQEAKIAogaW1wb3J0IHVuaXR0ZXN0CiAKLWZyb20gd2Vi
a2l0cHkudG9vbC5jb21tYW5kcy5yZWJhc2VsaW5lIGltcG9ydCBCdWlsZGVyVG9Qb3J0Citmcm9t
IHdlYmtpdHB5LmNvbW1vbi5zeXN0ZW0ub3V0cHV0Y2FwdHVyZSBpbXBvcnQgT3V0cHV0Q2FwdHVy
ZQorZnJvbSB3ZWJraXRweS50aGlyZHBhcnR5Lm1vY2sgaW1wb3J0IE1vY2sKK2Zyb20gd2Via2l0
cHkudG9vbC5jb21tYW5kcy5yZWJhc2VsaW5lIGltcG9ydCBCdWlsZGVyVG9Qb3J0LCBSZWJhc2Vs
aW5lCitmcm9tIHdlYmtpdHB5LnRvb2wubW9ja3Rvb2wgaW1wb3J0IE1vY2tUb29sCisKKworY2xh
c3MgUmViYXNlbGluZVRlc3QodW5pdHRlc3QuVGVzdENhc2UpOgorICAgICMgVGhpcyBqdXN0IG1h
a2VzIHN1cmUgdGhlIGNvZGUgcnVucyB3aXRob3V0IGV4Y2VwdGlvbnMuCisgICAgZGVmIHRlc3Rf
dGVzdHNfdG9fdXBkYXRlKHNlbGYpOgorICAgICAgICBjb21tYW5kID0gUmViYXNlbGluZSgpCisg
ICAgICAgIGNvbW1hbmQuYmluZF90b190b29sKE1vY2tUb29sKCkpCisgICAgICAgIGJ1aWxkID0g
TW9jaygpCisgICAgICAgIE91dHB1dENhcHR1cmUoKS5hc3NlcnRfb3V0cHV0cyhzZWxmLCBjb21t
YW5kLl90ZXN0c190b191cGRhdGUsIFtidWlsZF0pCiAKIAogY2xhc3MgQnVpbGRlclRvUG9ydFRl
c3QodW5pdHRlc3QuVGVzdENhc2UpOgpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRw
eS90b29sL21vY2t0b29sLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS90b29sL21vY2t0b29s
LnB5CmluZGV4IDMwYTRiYzNlOTgzZDM5ZTZiMDJmMGIwMWViZjg5NzFhMjNjYTJjNTIuLmViN2My
NDhmN2Q2NzNlYmNmZWM4NmU3Zjc0NzJiN2JhNDc2YTI1MmEgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1Nj
cmlwdHMvd2Via2l0cHkvdG9vbC9tb2NrdG9vbC5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtp
dHB5L3Rvb2wvbW9ja3Rvb2wucHkKQEAgLTU0MCwxMCArNTQwLDE0IEBAIGNsYXNzIE1vY2tDaGVj
a291dChvYmplY3QpOgogCiBjbGFzcyBNb2NrVXNlcihvYmplY3QpOgogCi0gICAgQHN0YXRpY21l
dGhvZAotICAgIGRlZiBwcm9tcHQobWVzc2FnZSwgcmVwZWF0PTEsIHJhd19pbnB1dD1yYXdfaW5w
dXQpOgorICAgIEBjbGFzc21ldGhvZAorICAgIGRlZiBwcm9tcHQoY2xzLCBtZXNzYWdlLCByZXBl
YXQ9MSwgcmF3X2lucHV0PXJhd19pbnB1dCk6CiAgICAgICAgIHJldHVybiAiTW9jayB1c2VyIHJl
c3BvbnNlIgogCisgICAgQGNsYXNzbWV0aG9kCisgICAgZGVmIHByb21wdF93aXRoX2xpc3QoY2xz
LCBsaXN0X3RpdGxlLCBsaXN0X2l0ZW1zLCBjYW5fY2hvb3NlX211bHRpcGxlPUZhbHNlLCByYXdf
aW5wdXQ9cmF3X2lucHV0KToKKyAgICAgICAgcGFzcworCiAgICAgZGVmIGVkaXQoc2VsZiwgZmls
ZXMpOgogICAgICAgICBwYXNzCiAK
</data>

          </attachment>
      

    </bug>

</bugzilla>