Bug 239341 – Enable --treat-failing-as-flaky for the JSC post-commit bots

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Angelos Oikonomopoulos]

Enable --treat-failing-as-flaky for the JSC post-commit bots

[Angelos Oikonomopoulos]

Created attachment 457634 [details]
Patch

[Angelos Oikonomopoulos]

Created attachment 457638 [details]
Patch

[Ryan Haddad]

rdar://92099399

[Angelos Oikonomopoulos]

Created attachment 458261 [details]
Patch

[Angelos Oikonomopoulos]

Created attachment 458262 [details]
Patch

[EWS]

Committed r293430 (249990@main): <https://commits.webkit.org/249990@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 458262 [details].

[Angelos Oikonomopoulos]

This has been picked up by the build.webkit.org after yesterday's restart, and there's already been a run with a flaky test:

https://build.webkit.org/#/builders/31/builds/3567

This is correctly captured in the per-test results:

❯ curl -sL 'http://results.webkit.org/api/results/javascriptcore-tests/stress/new-largeish-contiguous-array-with-size.js.ram-size-10000000?architecture=mips&uuid=165109908000'  | jq .
[
  {
    "configuration": {
      "architecture": "mips",
      "is_simulator": false,
      "platform": "linux",
      "style": "release",
      "version": 4019000
    },
    "results": [
      {
        "actual": "PASS",
        "expected": "PASS",
        "flakiness_num_passes": 7,
        "flakiness_num_tries": 8,
        "start_time": 1651099268,
        "uuid": 165109908000
      }
    ]
  }
]

However, when I look at the "suite" results, I get:

❯ curl -sL 'http://results.webkit.org/api/results/javascriptcore-tests?architecture=mips&limit=40' | jq -C . | grep build-number | grep 356
          "build-number": "3560",
          "build-number": "3561",
          "build-number": "3563",
          "build-number": "3564",
          "build-number": "3568",
          "build-number": "3569",

I.e. the result for the build number (3567) is missing.

That said, I see that e.g. https://build.webkit.org/#/builders/31/builds/3565, which didn't have any flaky tests, is also missing, so I'm not sure what to think.

FWIW I've locally verified again that when I submit a report with the current ToT (+ some flakiness injection) my local resultsdb can return the result just fine.

[Jonathan Bedard]

This feels like we dropped processing the build, but I would need to investigate to verify that.

Since we have an example, I can put this in the staging instance to check.