Bug 301003 – [Tools][CMake] Avoid ccache misses when building in different directories

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Carlos Alberto Lopez Perez]

ccache by default stores the full path of the files as part of the hash, so it is not possible to share the cache when building the same project on different paths.
This is an issue because we are using a remote ccache server and every user now has a different path since we stopped using flatpak (previously it was the same for everyone with the /app bind mount)

https://ccache.dev/manual/4.12.1.html#_compiling_in_different_directories

But we can configure ccache by default to use relative paths for the webkit source files, enabling the cache to hit even when the path on disk to the webkit repository changes.

[Carlos Alberto Lopez Perez]

I did this test:

- Start with a clean ccache
- Clean build of webkit (wpe) (do not use flatpak)
- Rename the source directory where it is webkit
- Clean build of webkit again (wpe) (do not use flatpak)


1) With 301701@main:
Stats updated:              Fri Oct 17 17:47:26 2025
Cacheable calls:             7696 / 15268 (50.41%)
  Hits:                         6 /  7696 ( 0.08%)
    Direct:                     6 /     6 (100.0%)
    Preprocessed:               0 /     6 ( 0.00%)
  Misses:                    7690 /  7696 (99.92%)
Uncacheable calls:           7572 / 15268 (49.59%)
  Called for linking:         424 /  7572 ( 5.60%)
  Called for preprocessing:  7092 /  7572 (93.66%)
  Compilation failed:          12 /  7572 ( 0.16%)
  No input file:               14 /  7572 ( 0.18%)
  Preprocessing failed:        30 /  7572 ( 0.40%)
Successful lookups:
  Direct:                       6 /  7738 ( 0.08%)
  Preprocessed:                 0 /  7702 ( 0.00%)
Local storage:
  Cache size (GB):           0.64 / 30.00 ( 2.14%)
  Files:                    15362
  Hits:                         6 /  7708 ( 0.08%)
  Misses:                    7702 /  7708 (99.92%)
  Reads:                    15446
  Writes:                   15362

2) Repeat the test with the patch of the PR below

Stats updated:              Fri Oct 17 18:46:59 2025
Cacheable calls:             7700 / 15276 (50.41%)
  Hits:                      3670 /  7700 (47.66%)
    Direct:                  2838 /  3670 (77.33%)
    Preprocessed:             832 /  3670 (22.67%)
  Misses:                    4030 /  7700 (52.34%)
Uncacheable calls:           7576 / 15276 (49.59%)
  Called for linking:         428 /  7576 ( 5.65%)
  Called for preprocessing:  7092 /  7576 (93.61%)
  Compilation failed:          12 /  7576 ( 0.16%)
  No input file:               14 /  7576 ( 0.18%)
  Preprocessing failed:        30 /  7576 ( 0.40%)
Successful lookups:
  Direct:                    2838 /  7742 (36.66%)
  Preprocessed:               832 /  4874 (17.07%)
Local storage:
  Cache size (GB):           0.33 / 30.00 ( 1.09%)
  Files:                     8171
  Hits:                      3670 /  7712 (47.59%)
  Misses:                    4042 /  7712 (52.41%)
  Reads:                    15454
  Writes:                    8874


Time to build WebKit on the second attempt: 55 minutes first case, 4 minutes second case.

So this patch clearly enables ccache to hit as if the directory was not renamed

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #1)

> Time to build WebKit on the second attempt: 55 minutes first case, 4 minutes
> second case.
> 

I mean: on the second rebuild (the one that is supposed to hit on the cache)
it took 55 minutes on the second rebuild (for the first test on 301701@main)
with 100% misses, but when this second rebuild was done with the patch
attached below it took 4 minutes and the cache was hitting like if the directory
was not renamed


> So this patch clearly enables ccache to hit as if the directory was not
> renamed

[Carlos Alberto Lopez Perez]

Pull request: https://github.com/WebKit/WebKit/pull/52593

[EWS]

Committed 301772@main (8fe054e11988): <https://commits.webkit.org/301772@main>

Reviewed commits have been landed. Closing PR #52593 and removing active labels.

[Radar WebKit Bug Importer]

<rdar://problem/162942426>

[Michael Catanzaro]

It broke debuginfo extraction in Fedora, https://bugzilla.redhat.com/show_bug.cgi?id=2408418. Any objection to limiting this to DEVELOPER_MODE only?

We should also prepend the flag rather than append it. Prepending allows overriding it at build time by changing CFLAGS/CXXFLAGS. Appending means the build system gets final say, so that's more suitable for stuff that really absolutely has to always be used, like -fno-exceptions, -fno-rtti, -fno-strict-aliasing.

[Carlos Alberto Lopez Perez]

(In reply to Michael Catanzaro from comment #6)
> It broke debuginfo extraction in Fedora,
> https://bugzilla.redhat.com/show_bug.cgi?id=2408418. Any objection to
> limiting this to DEVELOPER_MODE only?
> 

The flag -ffile-prefix-map not only helps with ccache hits, but also makes the build more reproducible.
In Debian this flag is enabled by default https://wiki.debian.org/ReproducibleBuilds/GCC-build-path and that helps webkit2gtk to be reproducible https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/webkit2gtk.html

However, if this is breaking debuginfo in other distros, then I think it is reasonable to only enable this by default on DEVELOPER_MODE builds

> We should also prepend the flag rather than append it. Prepending allows
> overriding it at build time by changing CFLAGS/CXXFLAGS. Appending means the
> build system gets final say, so that's more suitable for stuff that really
> absolutely has to always be used, like -fno-exceptions, -fno-rtti,
> -fno-strict-aliasing.

Yes, good point. We should do that.

[Michael Catanzaro]

I will follow up on this:

 * will change to prepend rather than append
 * will move it to WebKitCompilerFlags.cmake rather than OptionsCommon.cmake (and investigate moving several other things in this file as well)
 * will probably make it conditional on DEVELOPER_MODE so that it's used for build-webkit builds but not for distro builds

Maybe Fedora's find-debuginfo script should be able to handle relative paths.

Reproducibility is important, but most distros won't require this flag for reproducibility because most distros build packages in chroots with standardized path names. I suppose it doesn't matter, since Debian has its own infrastructure for injecting this flag.

Trying to share a ccache seems like a pretty good use case to me. Anybody using build-webkit will have DEVELOPER_MODE enabled, though, and I think that's the case you're concerned with.

[Michael Catanzaro]

Follow-up: bug #302021

[Michael Catanzaro]

So in bug #302021 I changed this to only be enabled in DEVELOPER_MODE. Unfortunately this means debug sources are broken in DEVELOPER_MODE. For example,  if I hit a breakpoint in gdb it prints "No such file or directory":

Thread 1 "epiphany" hit Breakpoint 2, webkit_web_view_load_uri (webView=0x10df980, uri=0x14cf360 "http://example.com/")
    at ./Source/WebKit/UIProcess/API/glib/WebKitWebView.cpp:3431
warning: 3431	./Source/WebKit/UIProcess/API/glib/WebKitWebView.cpp: No such file or directory

And if I attempt to print the code manually using l, it fails:

(gdb) l
3426	in ./Source/WebKit/UIProcess/API/glib/WebKitWebView.cpp

So I think this is not worth it even in DEVELOPER_MODE. Maybe developers who are using remote ccache can just configure it manually?

[Nikolas Zimmermann]

I agree Michael, that we need to do something about this - it's actively breaking things. However, the idea is that all WebKit devs using the SDK (which are at least all Igalians) use the remote ccache, since it's speeding up builds a lot. So we need a solution that works in all cases.

[Michael Catanzaro]

Probably the only correct fix here would be to build in a container (or chroot) to guarantee consistent paths.

Where is the remote ccache configured? We could start by adding the flag there instead of in WebKitCompilerFlags.cmake, so at least only developers who actually use the remote ccache would be affected.

Presumably only trusted infrastructure is able to populate the remote cache, right?

I'm also curious: where is this SDK? I've never used it.