Bug 214087 – webkitdbpy: Partial loading of results from configurations with same version_name but different version

• Status: NEW
• Component: Tools / Tests

[Lauro Moura]

Created attachment 403792 [details]
Screenshot of wpe/gtk configurations

Looking through the results api from `https://results.webkit.org/api/results/layout-tests/accessibility/textarea-selected-text-range.html?platform=GTK&platform=WPE`, the following configurations are returned:

GTK debug (Xvfb/4.19.0)
GTK release (Wayland/4.19.0)
GTK release (Xvfb/5.4.0)
GTK release (Xvfb/5.5.0)
WPE debug (-/5.4.0)
WPE debug (-/5.5.0)
WPE release (-/4.19.0)

As the attached screenshot shows, WPE debug configurations results into two separate series while GTK's Release-Xvfb gets grouped together under `Xvfb wk2 Release`. 

While initially not a big issue, the problem is that only the results from one of the series is shown for both series, instead of the expected behavior of both results separated.

As the change in version happened between r263965/r263966, this means that either results after r263966 (5.5.0) are shown or results before it. (Refreshing the page often changes the results displayed)

Link to example search: https://results.webkit.org/?suite=layout-tests&test=accessibility%2Ftextarea-selected-text-range.html&platform=WPE&platform=GTK

[Jonathan Bedard]

I'm not super familiar with how we should be thinking about GTK configuration differences, so it's certainly possible we need a special UI case for GTK.

I think there is also a lack of standardization among the GTK bots, though. https://results.webkit.org/api/suites?platform=GTK may shed more light on the issue. Looks like we're running 4.19, 5.4 and 5.5. Is that correct?

What's going on here is that I've assumed that version_name and version are related (because they are on Apple's platform, ie, Catalina refers to 10.15.*). That may not be true for other ports. It would just be a UI change to expose the version number in GTK configurations.

[Lauro Moura]

(In reply to Jonathan Bedard from comment #1)
> I'm not super familiar with how we should be thinking about GTK
> configuration differences, so it's certainly possible we need a special UI
> case for GTK.
> 
> I think there is also a lack of standardization among the GTK bots, though.
> https://results.webkit.org/api/suites?platform=GTK may shed more light on
> the issue. Looks like we're running 4.19, 5.4 and 5.5. Is that correct?
> 

Yeah, as it happens to get the version from the platform, in this case, the Linux kernel of the host. We are in the process of standardizing the workers as much as possible but I'm not sure if we'll be able to use the same kernel for all workers.

> What's going on here is that I've assumed that version_name and version are
> related (because they are on Apple's platform, ie, Catalina refers to
> 10.15.*). That may not be true for other ports. It would just be a UI change
> to expose the version number in GTK configurations.

For GTK, nowadays version name is used to distinguish the Xvfb and Wayland results when uploading, and version is kinda "ignored", but soon we'll be enabling the GTK4 bots so this probably will need more thought.

As you said, this causes both Xvfb configurations to have the same key, writing to the same slot in the resultsByKey object in timeline.js and thus "filtering" the results.

There would be any problem to change the toKey behavior to include the version?

[Radar WebKit Bug Importer]

<rdar://problem/65853095>

[Jonathan Bedard]

(In reply to Lauro Moura from comment #2)
> (In reply to Jonathan Bedard from comment #1)
> > I'm not super familiar with how we should be thinking about GTK
> > configuration differences, so it's certainly possible we need a special UI
> > case for GTK.
> > 
> > I think there is also a lack of standardization among the GTK bots, though.
> > https://results.webkit.org/api/suites?platform=GTK may shed more light on
> > the issue. Looks like we're running 4.19, 5.4 and 5.5. Is that correct?
> > 
> 
> Yeah, as it happens to get the version from the platform, in this case, the
> Linux kernel of the host. We are in the process of standardizing the workers
> as much as possible but I'm not sure if we'll be able to use the same kernel
> for all workers.
> 
> > What's going on here is that I've assumed that version_name and version are
> > related (because they are on Apple's platform, ie, Catalina refers to
> > 10.15.*). That may not be true for other ports. It would just be a UI change
> > to expose the version number in GTK configurations.
> 
> For GTK, nowadays version name is used to distinguish the Xvfb and Wayland
> results when uploading, and version is kinda "ignored", but soon we'll be
> enabling the GTK4 bots so this probably will need more thought.
> 
> As you said, this causes both Xvfb configurations to have the same key,
> writing to the same slot in the resultsByKey object in timeline.js and thus
> "filtering" the results.
> 
> There would be any problem to change the toKey behavior to include the
> version?

Changing toKey to include version is probably something we want to do for non-mac ports (or maybe even GTK ports specifically) rather than everywhere, because we attempt to collapse major MacOS/iOS versions into a single group (although it gets weird because we want to consider the mid-year release a major release, which is why we have the whole "xxxE" thing going on)

We also definitely want https://bugs.webkit.org/show_bug.cgi?id=214511, because machine with different kernel versions would be considered different configurations still.