Maximum element size is too restrictive for virtual scrolling

• Component: Layout and Rendering
• Status: RESOLVED WONTFIX

[Andy E]

We are using a common technique for virtual scrolling in a list (1-dimensional) and a grid (2-dimensional). The technique involves using a container element of fixed dimensions, with a child that is the size of the area you want to scroll, then the element that updates as you scroll sticky-positioned to the maximum size of the container.

Here's a rudimentary example:

https://jsfiddle.net/AndyE/2ev53nuq/56/

This works well, until you have customers with over 10 million items in a list, and those are transposed into columns on a grid.

Safari tops out at 33,554,428px for the width, height, scrollWidth or scrollHeight of an element. This limits us to 838,860 rows at the 40 pixels per row given in the example. If you update the jsfiddle example to display more than that, Safari caps to that maximum height. If the list items are transposed into column labels, then the displayable amount reduces drastically.

Naturally, our product has searching and filtering to provide more sensible navigation of large datasets. However, we still want this to work as expected if we scroll the user to a section of the grid or list based on their search results. About 64 of our customer's lists have more than 10 million items, about 470 have between 1 and 10 million and there are about 35,000 lists in total.

At the moment, we're using a custom scrollbar, but this gets too complex as you try to match the features and styling of native scrollbars.

I have heard, anecdotally whilst chasing people in my company and asking them, that we have a customer with a list of 35 million items. To virtually scroll over that many items in columns, at an average of 250px per column, we would need a scrollWidth of nearly 9 billion pixels, more than 500 times the current maximum in Firefox and almost 270 times the maximum in Safari/Chrome.

My preference is that the standards provide a better way of having virtual scrollbars than resorting to jumping through hoops like creating ridiculously large elements; I have proposed a CSSOM feature[1] and attempted to start a discussion around web components[2]. In the latter ticket, Domenic Denicola from the Google Chrome team recommended I file tickets with the major browsers to see if I could get the limits increased instead, and this is what I am doing as an interim solution.

[1]: https://github.com/w3c/csswg-drafts/issues/3397
[2]: https://github.com/w3c/webcomponents/issues/791

[Simon Fraser (smfr)]

What are the limits in other browsers?

[Brent Fulgham]

If I understood Andy's comment:

Safari: 3.3e7 px
Chrome: 9e9 / 270 = 3.3e7 px
Firefox: 9e9 / 500 = 1,4e7 px

[Radar WebKit Bug Importer]

<rdar://problem/48553451>

[Andy E]

#2 is correct-ish on those estimates.

AnneVK of Mozilla suggested I show that others are also experiencing this issue, so here's a bunch of links to open source project issue trackers where people have encountered this problem:

SlickGrid — [Virtualization fails above 50k records in IE](https://github.com/mleibman/SlickGrid/issues/22) (title mentions IE but other browsers are also discussed)
Clusterize.js — [Cannot scroll past 1,342,180th line](https://github.com/NeXTs/Clusterize.js/issues/161)
ngx-virtual-scroller — [What in case for example 1 million records](https://github.com/rintoj/ngx-virtual-scroller/issues/68)

Another project, react-virtualized (probably the most popular virtual scrolling library), uses the same workaround we are using to get around the limit, but the author [created an issue](https://github.com/bvaughn/react-virtualized/issues/396) to discuss the drawbacks of the workaround and potential solutions.

[Andy E]

I'd be really grateful for some technical feedback on this issue, so that I can further build the case for a spec addition if it is not technically feasible to increase these limits. I appreciate any time spared to investigate and respond and understand that this may not be a priority for WebKit. 

Mozilla and Chrome have signalled that the change would be large for them since it requires using 64-bit numbers for layout object sizes (or sacrificing other data like sub-pixel precision, which they are not prepared to do).

https://bugs.chromium.org/p/chromium/issues/detail?id=932109
https://bugzilla.mozilla.org/show_bug.cgi?id=1527883

[Simon Fraser (smfr)]

(In reply to Andy E from comment #5)
> I'd be really grateful for some technical feedback on this issue, so that I
> can further build the case for a spec addition if it is not technically
> feasible to increase these limits. I appreciate any time spared to
> investigate and respond and understand that this may not be a priority for
> WebKit. 
> 
> Mozilla and Chrome have signalled that the change would be large for them
> since it requires using 64-bit numbers for layout object sizes (or
> sacrificing other data like sub-pixel precision, which they are not prepared
> to do).
> 
> https://bugs.chromium.org/p/chromium/issues/detail?id=932109
> https://bugzilla.mozilla.org/show_bug.cgi?id=1527883

Our position aligns with Blink and Gecko. We have no intention of making the wide-spread changes that would be required to fix this bug.

Instead, if you need this kind of scaling, you'll have to make some sort of virtual scroller that keeps the element size within reasonable limits.