Bug 202928 – layout step not summarizes results to stdout when running with "--report"

• Component: Tools / Tests
• Status: NEW

[Carlos Alberto Lopez Perez]

Since r250966 the layout step for GTK and WPE is not longer printing the results in stdout.

Previously, when this step finished it printed something like this: http://sprunge.us/XZjjH4 than then the build master parsed and put on the step for the layout tests as " 68 failures 80 new passes 11 flakes 14 missing results " <-- https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11560

But since r250966 the layout test step runs with a new parameter "--report https://results.webkit.org" and it not longer prints this to stdout. So in the results of the buildbot the summary for the step is empty.
For example: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11603 <-- it not longer has the summary of the number of failures in the layout test step.

[Carlos Alberto Lopez Perez]

Seems the bug happens when the layout step fails to upload.

Our bots are giving this error:

05:10:27.819 123196 Uploading to https://results.webkit.org ...
05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443
05:10:28.207 123196     Failed to upload to https://results.webkit.org, results server not online

(likely caused because results.webkit.org has an invalid SSL certificate, signed by an unknown CA)

$ gnutls-cli  results.webkit.org:443
Processed 124 CA certificate(s).
Resolving 'results.webkit.org:443'...
Connecting to '17.32.208.166:443'...
- Certificate type: X.509
- Got a certificate list of 1 certificates.
- Certificate[0] info:
 - subject `C=US,ST=California,O=Apple Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc="
	Public Key ID:
		sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279
		sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87
	Public Key PIN:
		pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=

- Status: The certificate is NOT trusted. The certificate issuer is unknown. 
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.

[Jonathan Bedard]

webkitpy tests are probably the right example for getting to the bottom of this:

https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11603/steps/webkitpy-test/logs/stdio

Looks like GTK machines can't reach results.webkit.org. WinCario has the same problem:

https://build.webkit.org/builders/WPE%20Linux%2064-bit%20Release%20%28Tests%29/builds/15688/steps/webkitpy-test/logs/stdio

Can those machines reach https://results.webkit.org? You should be able to reproduce this with 'test-webkitpy --report https://results.webkit.org' (note that without the API key, which the buildbot master servers, this request will fail, but it should explicitly say that it failed because of the API key, rather than 'results server not online'.

I did some work last week, and I had a personal machine off the Apple network that was able to report, so I'm surprised we're having this issue.

[Jonathan Bedard]

(In reply to Carlos Alberto Lopez Perez from comment #1)
> Seems the bug happens when the layout step fails to upload.
> 
> Our bots are giving this error:
> 
> 05:10:27.819 123196 Uploading to https://results.webkit.org ...
> 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443
> 05:10:28.207 123196     Failed to upload to https://results.webkit.org,
> results server not online
> 
> (likely caused because results.webkit.org has an invalid SSL certificate,
> signed by an unknown CA)
> 
> $ gnutls-cli  results.webkit.org:443
> Processed 124 CA certificate(s).
> Resolving 'results.webkit.org:443'...
> Connecting to '17.32.208.166:443'...
> - Certificate type: X.509
> - Got a certificate list of 1 certificates.
> - Certificate[0] info:
>  - subject `C=US,ST=California,O=Apple
> Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer
> `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1',
> serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using
> RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02
> 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc="
> 	Public Key ID:
> 		sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279
> 		sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87
> 	Public Key PIN:
> 		pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=
> 
> - Status: The certificate is NOT trusted. The certificate issuer is unknown. 
> *** PKI verification of server certificate failed...
> *** Fatal error: Error in the certificate.

I temporarily turned off the verification in r250997 because our Catalina bots where encountering a similar problem.

[Carlos Alberto Lopez Perez]

(In reply to Jonathan Bedard from comment #2)
> webkitpy tests are probably the right example for getting to the bottom of
> this:
> 
> https://build.webkit.org/builders/GTK%20Linux%2064-
> bit%20Release%20%28Tests%29/builds/11603/steps/webkitpy-test/logs/stdio
> 
> Looks like GTK machines can't reach results.webkit.org. WinCario has the
> same problem:
> 
> https://build.webkit.org/builders/WPE%20Linux%2064-
> bit%20Release%20%28Tests%29/builds/15688/steps/webkitpy-test/logs/stdio
> 
> Can those machines reach https://results.webkit.org? You should be able to
> reproduce this with 'test-webkitpy --report https://results.webkit.org'
> (note that without the API key, which the buildbot master servers, this
> request will fail, but it should explicitly say that it failed because of
> the API key, rather than 'results server not online'.
> 
> I did some work last week, and I had a personal machine off the Apple
> network that was able to report, so I'm surprised we're having this issue.

They can reach it, but they can't recognized the certificate.

It seems that the server hosting results.webkit.org is missing to send the intermediate certificates of the chain.

See: https://www.sslshopper.com/ssl-checker.html#hostname=results.webkit.org

[Carlos Alberto Lopez Perez]

(In reply to Jonathan Bedard from comment #3)
> (In reply to Carlos Alberto Lopez Perez from comment #1)
> > Seems the bug happens when the layout step fails to upload.
> > 
> > Our bots are giving this error:
> > 
> > 05:10:27.819 123196 Uploading to https://results.webkit.org ...
> > 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443
> > 05:10:28.207 123196     Failed to upload to https://results.webkit.org,
> > results server not online
> > 
> > (likely caused because results.webkit.org has an invalid SSL certificate,
> > signed by an unknown CA)
> > 
> > $ gnutls-cli  results.webkit.org:443
> > Processed 124 CA certificate(s).
> > Resolving 'results.webkit.org:443'...
> > Connecting to '17.32.208.166:443'...
> > - Certificate type: X.509
> > - Got a certificate list of 1 certificates.
> > - Certificate[0] info:
> >  - subject `C=US,ST=California,O=Apple
> > Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer
> > `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1',
> > serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using
> > RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02
> > 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc="
> > 	Public Key ID:
> > 		sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279
> > 		sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87
> > 	Public Key PIN:
> > 		pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=
> > 
> > - Status: The certificate is NOT trusted. The certificate issuer is unknown. 
> > *** PKI verification of server certificate failed...
> > *** Fatal error: Error in the certificate.
> 
> I temporarily turned off the verification in r250997 because our Catalina
> bots where encountering a similar problem.


Digging more into this:

 - By default Debian 10 (which our bots run) only accepts(In reply to Jonathan Bedard from comment #3)
> (In reply to Carlos Alberto Lopez Perez from comment #1)
> > Seems the bug happens when the layout step fails to upload.
> > 
> > Our bots are giving this error:
> > 
> > 05:10:27.819 123196 Uploading to https://results.webkit.org ...
> > 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443
> > 05:10:28.207 123196     Failed to upload to https://results.webkit.org,
> > results server not online
> > 
> > (likely caused because results.webkit.org has an invalid SSL certificate,
> > signed by an unknown CA)
> > 
> > $ gnutls-cli  results.webkit.org:443
> > Processed 124 CA certificate(s).
> > Resolving 'results.webkit.org:443'...
> > Connecting to '17.32.208.166:443'...
> > - Certificate type: X.509
> > - Got a certificate list of 1 certificates.
> > - Certificate[0] info:
> >  - subject `C=US,ST=California,O=Apple
> > Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer
> > `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1',
> > serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using
> > RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02
> > 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc="
> > 	Public Key ID:
> > 		sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279
> > 		sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87
> > 	Public Key PIN:
> > 		pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=
> > 
> > - Status: The certificate is NOT trusted. The certificate issuer is unknown. 
> > *** PKI verification of server certificate failed...
> > *** Fatal error: Error in the certificate.
> 
> I temporarily turned off the verification in r250997 because our Catalina
> bots where encountering a similar problem.



Digging more into this:

 - By default Debian 10 (which our bots run) only accept SSL connections
 with at least TLSv1.2
 
 - results.webkit.org is not able to do anything better than TLSv1.0.
 Check: https://www.ssllabs.com/ssltest/analyze.html?d=results.webkit.org
 
 - So even when running with Verify=false it fails because it isn't able
 to even do the handshake, check:
 
 $ python
Python 2.7.16 (default, Apr  6 2019, 01:42:57) 
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> p=requests.post('https://results.webkit.org/api/upload',headers={'Content-type': 'application/json'},data="",verify=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='results.webkit.org', port=443): Max retries exceeded with url: /api/upload (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl_choose_client_version', 'unsupported protocol')],)",),))
>>> 


 - We already had a similar issue on our EWS bots, where we had to downgrade
 the security settings of Debian to accept TLSv1.0 because the server hosting
 the EWS master was also providing SSL with deprecated encryption settings.
 
 
 - We can do something similar here, I can set our buildbot bots to accept SSL with
 TLSv1.0 if upgrading that server to provide more modern SSL settings is
 difficult for you. But I guess this is only going to keep causing issues
 in the future, upgrading that server in your end to support TLSv1.2 would
 be nice.

[Carlos Alberto Lopez Perez]

In any case I think we maybe should keep printing the results to stdout even when that upload step fails

[Jonathan Bedard]

(In reply to Carlos Alberto Lopez Perez from comment #6)
> In any case I think we maybe should keep printing the results to stdout even
> when that upload step fails

This is reasonable, and also pretty easy, I'll put together the patch to do so.

[Jonathan Bedard]

(In reply to Carlos Alberto Lopez Perez from comment #5)
> ...
>
>  - We can do something similar here, I can set our buildbot bots to accept
> SSL with
>  TLSv1.0 if upgrading that server to provide more modern SSL settings is
>  difficult for you. But I guess this is only going to keep causing issues
>  in the future, upgrading that server in your end to support TLSv1.2 would
>  be nice.

Actually, looking through our configuration, I had explicitly downgraded our TLS version in a misguided attempt to fix our Catalina bots. I just changed those configurations, we should be using TLSv1.2 now.

[Jonathan Bedard]

Created attachment 380891 [details]
Patch

[Carlos Alberto Lopez Perez]

(In reply to Jonathan Bedard from comment #8)
> (In reply to Carlos Alberto Lopez Perez from comment #5)
> > ...
> >
> >  - We can do something similar here, I can set our buildbot bots to accept
> > SSL with
> >  TLSv1.0 if upgrading that server to provide more modern SSL settings is
> >  difficult for you. But I guess this is only going to keep causing issues
> >  in the future, upgrading that server in your end to support TLSv1.2 would
> >  be nice.
> 
> Actually, looking through our configuration, I had explicitly downgraded our
> TLS version in a misguided attempt to fix our Catalina bots. I just changed
> those configurations, we should be using TLSv1.2 now.

Thanks.

Now it is able to connect to the server, but gives other error:

10:54:12.318 35166 Uploading to https://results.webkit.org ...
10:54:12.387 35166 Starting new HTTPS connection (1): results.webkit.org:443
/home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
10:54:16.429 35166 https://results.webkit.org:443 "POST /api/upload HTTP/1.1" 400 79
10:54:16.430 35166     Error uploading to https://results.webkit.org
10:54:16.430 35166         Invalid configuration
10:54:16.430 35166 Uploads completed!

full log here: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11605/steps/layout-test/logs/stdio

Is there something we have to configure on the bots for this upload to happen as expected?

[Jonathan Bedard]

(In reply to Carlos Alberto Lopez Perez from comment #10)
> ...
> Is there something we have to configure on the bots for this upload to
> happen as expected?

Not expected, although that error is actually coming from the server, so you are making a POST request, it's just getting rejected.

My guess is that we are creating a configuration that's missing something...I don't have a GTK machine handy, if we log the dictionary returned by Tools/Scripts/webkitpy/results/upload.py Upload.create_configuration, what does it look like? (this is called from Tools/Scripts/port/base.py for GTK)

[Jonathan Bedard]

(In reply to Jonathan Bedard from comment #11)
> (In reply to Carlos Alberto Lopez Perez from comment #10)
> > ...
> > Is there something we have to configure on the bots for this upload to
> > happen as expected?
> 
> Not expected, although that error is actually coming from the server, so you
> are making a POST request, it's just getting rejected.
> 
> My guess is that we are creating a configuration that's missing
> something...I don't have a GTK machine handy, if we log the dictionary
> returned by Tools/Scripts/webkitpy/results/upload.py
> Upload.create_configuration, what does it look like? (this is called from
> Tools/Scripts/port/base.py for GTK)

Side note: You can test this by running test-webkitpy --report https://results.webkit.org locally on a GTK machine. You won't have an API key, so it will get rejected, but we can at least see what GTK machines try and upload

[Carlos Alberto Lopez Perez]

(In reply to Jonathan Bedard from comment #12)
> (In reply to Jonathan Bedard from comment #11)
> > (In reply to Carlos Alberto Lopez Perez from comment #10)
> > > ...
> > > Is there something we have to configure on the bots for this upload to
> > > happen as expected?
> > 
> > Not expected, although that error is actually coming from the server, so you
> > are making a POST request, it's just getting rejected.
> > 
> > My guess is that we are creating a configuration that's missing
> > something...I don't have a GTK machine handy, if we log the dictionary
> > returned by Tools/Scripts/webkitpy/results/upload.py
> > Upload.create_configuration, what does it look like? (this is called from
> > Tools/Scripts/port/base.py for GTK)
> 
> Side note: You can test this by running test-webkitpy --report
> https://results.webkit.org locally on a GTK machine. You won't have an API
> key, so it will get rejected, but we can at least see what GTK machines try
> and upload

How the bots are supposed to obtain the API key? I think nobody installed that on the GTK or WPE bots (if that was supposed to be done)

[Jonathan Bedard]

(In reply to Carlos Alberto Lopez Perez from comment #13)
> ...
> 
> How the bots are supposed to obtain the API key? I think nobody installed
> that on the GTK or WPE bots (if that was supposed to be done)

Buildbot passes it as an environment variable.

The error message indicates your bots are getting past that point, you would have gotten something like 'Missing API key' or 'Incorrect API key'

[Carlos Alberto Lopez Perez]

(In reply to Jonathan Bedard from comment #11)
> (In reply to Carlos Alberto Lopez Perez from comment #10)
> > ...
> > Is there something we have to configure on the bots for this upload to
> > happen as expected?
> 
> Not expected, although that error is actually coming from the server, so you
> are making a POST request, it's just getting rejected.
> 
> My guess is that we are creating a configuration that's missing
> something...I don't have a GTK machine handy, if we log the dictionary
> returned by Tools/Scripts/webkitpy/results/upload.py
> Upload.create_configuration, what does it look like? (this is called from
> Tools/Scripts/port/base.py for GTK)

This is how that dictionary looks on GTK (and I guess on WPE it would look the same):

{'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version': 'None', 'architecture': 'x86'}

[Jonathan Bedard]

(In reply to Carlos Alberto Lopez Perez from comment #15)
> (In reply to Jonathan Bedard from comment #11)
> > (In reply to Carlos Alberto Lopez Perez from comment #10)
> > > ...
> > > Is there something we have to configure on the bots for this upload to
> > > happen as expected?
> > 
> > Not expected, although that error is actually coming from the server, so you
> > are making a POST request, it's just getting rejected.
> > 
> > My guess is that we are creating a configuration that's missing
> > something...I don't have a GTK machine handy, if we log the dictionary
> > returned by Tools/Scripts/webkitpy/results/upload.py
> > Upload.create_configuration, what does it look like? (this is called from
> > Tools/Scripts/port/base.py for GTK)
> 
> This is how that dictionary looks on GTK (and I guess on WPE it would look
> the same):
> 
> {'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version':
> 'None', 'architecture': 'x86'}

This is actually a pretty straight-forward problem. platform.os_version isn
't defined for GTK.

Curious, what is the output of this python program:

import platform
print(platform.version())

[Carlos Alberto Lopez Perez]

(In reply to Jonathan Bedard from comment #16)
> (In reply to Carlos Alberto Lopez Perez from comment #15)
> > (In reply to Jonathan Bedard from comment #11)
> > > (In reply to Carlos Alberto Lopez Perez from comment #10)
> > > > ...
> > > > Is there something we have to configure on the bots for this upload to
> > > > happen as expected?
> > > 
> > > Not expected, although that error is actually coming from the server, so you
> > > are making a POST request, it's just getting rejected.
> > > 
> > > My guess is that we are creating a configuration that's missing
> > > something...I don't have a GTK machine handy, if we log the dictionary
> > > returned by Tools/Scripts/webkitpy/results/upload.py
> > > Upload.create_configuration, what does it look like? (this is called from
> > > Tools/Scripts/port/base.py for GTK)
> > 
> > This is how that dictionary looks on GTK (and I guess on WPE it would look
> > the same):
> > 
> > {'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version':
> > 'None', 'architecture': 'x86'}
> 
> This is actually a pretty straight-forward problem. platform.os_version isn
> 't defined for GTK.
> 
> Curious, what is the output of this python program:
> 
> import platform
> print(platform.version())

This is the output:
#1 SMP Debian 4.19.37-5+deb10u2~bpo9+1 (2019-08-16)