Bug 201053 – results.webkit.org: Make dot colors more accessible

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[Jonathan Bedard]

We need to put a symbol in the dots, but the success-failed-timeout-crash idiom is common enough that we should have colors for these states that are decernible in grayscale.

[Jonathan Bedard]

<rdar://problem/54586444>

[Jonathan Bedard]

Created attachment 377046 [details]
Patch

[Jonathan Bedard]

Created attachment 377050 [details]
light-mode

[Jonathan Bedard]

Created attachment 377051 [details]
dark-mode

[Jonathan Bedard]

Created attachment 377052 [details]
dark-mode-grayscale

[Jonathan Bedard]

Created attachment 377053 [details]
light-mode-grayscale

[Ryosuke Niwa]

The difference between all tests passed & some tests failed are still too subtle.

Really, there isn't a really way to provide enough contrast between four difference colors so we shouldn't be relying on colors to provide any information here.

[Zhifei Fang]

(In reply to Ryosuke Niwa from comment #7)
> The difference between all tests passed & some tests failed are still too
> subtle.
> 
> Really, there isn't a really way to provide enough contrast between four
> difference colors so we shouldn't be relying on colors to provide any
> information here.

How about adding some label in the dots ?

Like  X for tests failed. ! for crashes?

[Ryosuke Niwa]

See examples of colors in:
https://www.w3.org/WAI/WCAG20/Techniques/working-examples/G183/link-contrast.html

e.g.
https://accessibility.oit.ncsu.edu/tools/color-contrast/accessible-color-palette.php?&colors=79df8f,ff6666,665200,884ac5&main=79df8f&level=AA

shows that #f66 doesn't have enough contrast with either 665200, 79DF8F, or 884AC5.

[Ryosuke Niwa]

(In reply to Zhifei Fang from comment #8)
> (In reply to Ryosuke Niwa from comment #7)
> > The difference between all tests passed & some tests failed are still too
> > subtle.
> > 
> > Really, there isn't a really way to provide enough contrast between four
> > difference colors so we shouldn't be relying on colors to provide any
> > information here.
> 
> How about adding some label in the dots ?
> 
> Like  X for tests failed. ! for crashes?

Yeah, that's what I've been suggesting all along. There needs to be an indicator that doesn't rely on any color given we need to convey four different states. Putting three colors is already pushing the boundary of what's being accessible. Four color is really not possible if we wanted to use otherwise pleasant colors.

[Ryosuke Niwa]

Comment on attachment 377046 [details]
Patch

r- because I don't think this patch would address my concern with the website.

[Jonathan Bedard]

(In reply to Ryosuke Niwa from comment #11)
> Comment on attachment 377046 [details]
> Patch
> 
> r- because I don't think this patch would address my concern with the
> website.

I'm not sure that alone is a good reason to abandon the change.

If it isn't an improvement at all, then fair enough, we should abandon the patch. But I think our problem here goes beyond this particular case. The pass/fail/timeout/crash is a pretty common idiom in infrastructure, and while feature-complete services should definitely be using symbols, it's likely that contributors will use continue to use color based schemes while developing new sites, especially since we have a number of existing sites which portray meaning with color (EWS, buildbot, https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html).

In this case, we have a good alternative (symbols). But there are definitely schemes I've heard floated where symbols wouldn't be enough (for example, Zhifei has an idea for displaying a mini-map for long timelines). That's why I titled the bug 'Make dot colors more accessible' and not 'Make dots accessible'.

[Ryosuke Niwa]

(In reply to Jonathan Bedard from comment #12)
> (In reply to Ryosuke Niwa from comment #11)
> > Comment on attachment 377046 [details]
> > Patch
> > 
> > r- because I don't think this patch would address my concern with the
> > website.
> 
> I'm not sure that alone is a good reason to abandon the change.
> 
> If it isn't an improvement at all, then fair enough, we should abandon the
> patch.

I don't think change will make a material difference in my ability to discern passing from failing.

> But I think our problem here goes beyond this particular case. The
> pass/fail/timeout/crash is a pretty common idiom in infrastructure, and
> while feature-complete services should definitely be using symbols, it's
> likely that contributors will use continue to use color based schemes while
> developing new sites, especially since we have a number of existing sites
> which portray meaning with color (EWS, buildbot,
> https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html).

I agree, and that's a serious issue. I've been complaining this issue as long as I've been contributing to the WebKit project (in buildbot, EWS bubbles, bot watcher dashboard, etc...), and we seem to keep introducing new tools or websites with a similar issue. And I'm not the only one who is colorblind in the WebKit team. I'm just the vocal one because I'm fed up with this situation.

It's one thing for the project to create these unfriendly UI once or twice. It's entirely another when the project continues to make similar mistakes year after year, time after time after the affected people had complained about it.

> In this case, we have a good alternative (symbols). But there are definitely
> schemes I've heard floated where symbols wouldn't be enough (for example,
> Zhifei has an idea for displaying a mini-map for long timelines).

I'm fine with such a tool coming into an existence so long as I don't ever have to use it. In the case of the flakiness dashboard, it's a part of WebKit contributor's daily routine to investigate & fix flaky tests or newly failing tests.

It's not okay for tools to be unusable by those who need to use them.

[Alexey Proskuryakov]

Yes, no one questions the need to make this tool accessible to everyone who uses it. It isn't live on webkit.org yet, so this is work in progress.

Most contributors are not UI design experts, so they rely on example and guidance from you. Since the existing flakiness dashboard only relies on color to convey information, and so do multiple tools that you personally created, I'm also surprised that this patch isn't a step in the right direction.

[Ryosuke Niwa]

(In reply to Alexey Proskuryakov from comment #14)
> Yes, no one questions the need to make this tool accessible to everyone who
> uses it. It isn't live on webkit.org yet, so this is work in progress.

Except it’s already being used internally at Apple.

> Since the existing flakiness dashboard only relies on
> color to convey information, and so do multiple tools that you personally
> created,

I don’t know which tool you’re talking about. If you’re talking about the old results UI, I was merely matching the UI of the currently deployed flakiness dashboard. Also, colors were way more saturated and differed in contrast than what’s being proposed here.

Perf dashboard doesn’t rely on any color. Baseline & current graph lines are distinguishable from luminance difference and the relative location or shape of lines. All other uses of color are supplementary. Summary page bar graphs for example purposely extend to opposite directions based on whether it’s progression or regression. All labels explicitly convey the information communicated by color in text.

> I'm also surprised that this patch isn't a step in the right direction.

It could be for some people. I personally don’t see this set of colors as any improvement over what’s currently being used. It’s possible another set of colors (particularly ones used on the current flakiness dashboard) would be distinguishable at least to me.

The trouble really is that pleasant set of colors and accessible set of colors are mutually exclusive as far as I can tell. Whenever people use “pleasant colors”, it becomes unusable to me and others with color blindness.

[Zhifei Fang]

(In reply to Ryosuke Niwa from comment #15)
> (In reply to Alexey Proskuryakov from comment #14)
> > Yes, no one questions the need to make this tool accessible to everyone who
> > uses it. It isn't live on webkit.org yet, so this is work in progress.
> 
> Except it’s already being used internally at Apple.
> 
> > Since the existing flakiness dashboard only relies on
> > color to convey information, and so do multiple tools that you personally
> > created,
> 
> I don’t know which tool you’re talking about. If you’re talking about the
> old results UI, I was merely matching the UI of the currently deployed
> flakiness dashboard. Also, colors were way more saturated and differed in
> contrast than what’s being proposed here.
> 
> Perf dashboard doesn’t rely on any color. Baseline & current graph lines are
> distinguishable from luminance difference and the relative location or shape
> of lines. All other uses of color are supplementary. Summary page bar graphs
> for example purposely extend to opposite directions based on whether it’s
> progression or regression. All labels explicitly convey the information
> communicated by color in text.
> 
> > I'm also surprised that this patch isn't a step in the right direction.
> 
> It could be for some people. I personally don’t see this set of colors as
> any improvement over what’s currently being used. It’s possible another set
> of colors (particularly ones used on the current flakiness dashboard) would
> be distinguishable at least to me.
> 
> The trouble really is that pleasant set of colors and accessible set of
> colors are mutually exclusive as far as I can tell. Whenever people use
> “pleasant colors”, it becomes unusable to me and others with color blindness.

I am trying to find a set of color that works for most people, and still have a good looking.

Besides, there are also something other than color we can do, alt text (for people who can not see), keyboard short cut(for people who can not use mouse) etc, are also very important.

I think accessibility test will help us to address all those issues, adding those test s will help us don't make the same mistake again and again.

[Ryosuke Niwa]

(In reply to Zhifei Fang from comment #16)
>
> I am trying to find a set of color that works for most people, and still
> have a good looking.
> 
> Besides, there are also something other than color we can do, alt text (for
> people who can not see), keyboard short cut(for people who can not use
> mouse) etc, are also very important.

Keyboard accessibility is important for people who use keyboard as the primary input device as well.

In perf dashboard, the menu to add a new chart is keyboard focusable, and menu item can be picked via arrow keys. Similarly, graph’s current data point can be moved with arrow keys although there isn’t currently a way to focus a point in the graph in the first place or change between current and baseline points so that need to be fixed.

> I think accessibility test will help us to address all those issues, adding
> those test s will help us don't make the same mistake again and again.

Yeah, a good guideline on how to make website accessible along with some audit / tests would help here.