Bug 278362 – All Tests crashes when running `run-webkit-tests`

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Fady Farag]

Summary:

When trying to reproduce test failures locally I always get a crash error message.

Steps to reproduce:

Download a build product of WebKit: python3 Tools/CISupport/download-built-product --platform=mac --debug 'https://s3-us-west-2.amazonaws.com/archives.webkit.org/mac-ventura-x86_64 arm64-debug/[e-mail]'
Extract the built product: python3 Tools/CISupport/built-product-archive --platform=mac-ventura --debug extract
Run run-webkit-tests with any test 
The test crash with this error message: 

`dyld[6890]: Symbol not found: _NSTextCompletionAttributeName Referenced from: <89A995EB-08FD-3D70-A2F0-D3D2F49FCFCF> /Users/iidmsa/Documents/usr/WebKit/WebKitBuild/Release/WebKitTestRunner (built for macOS 14.2 which is newer than running OS)
Expected in:   <F66BD32D-B52D-3EFF-8D5C-48D0F7189785> /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit`

Environment

Does crash on:

System Version:	macOS 13.6.9 (22G830)
Kernel Version:	Darwin 22.6.0
Architecture:   x86_64

Crash log:
{"app_name":"WebKitTestRunner","timestamp":"2024-08-19 19:14:18.00 -0500","app_version":"","slice_uuid":"28272774-64e6-32b9-bc01-fa47c0cd7e8f","build_version":"","platform":1,"share_with_app_devs":0,"is_first_party":1,"bug_type":"309","os_version":"macOS 13.6.9 (22G830)","roots_installed":0,"incident_id":"B0F10F51-BBC9-4613-8028-9CE7B6E8145C","name":"WebKitTestRunner"}
{
  "uptime" : 78000,
  "procRole" : "Unspecified",
  "version" : 2,
  "userID" : 0,
  "deployVersion" : 210,
  "modelCode" : "MacBookPro14,1",
  "coalitionID" : 689,
  "osVersion" : {
    "train" : "macOS 13.6.9",
    "build" : "22G830",
    "releaseType" : "User"
  },
  "captureTime" : "2024-08-19 19:14:18.2722 -0500",
  "incident" : "B0F10F51-BBC9-4613-8028-9CE7B6E8145C",
  "pid" : 20320,
  "cpuType" : "X86-64",
  "roots_installed" : 0,
  "bug_type" : "309",
  "procLaunch" : "2024-08-19 19:14:15.6930 -0500",
  "procStartAbsTime" : 78378329568869,
  "procExitAbsTime" : 78380908356323,
  "procName" : "WebKitTestRunner",
  "procPath" : "\/Users\/USER\/Documents\/*\/WebKitTestRunner",
  "parentProc" : "Python",
  "parentPid" : 20198,
  "coalitionName" : "com.googlecode.iterm2",
  "crashReporterKey" : "14D06113-A758-DB98-FE14-6A3FBE3327F2",
  "responsiblePid" : 18174,
  "responsibleProc" : "iTerm2",
  "codeSigningID" : "com.apple.WebKit.WebKitTestRunner",
  "codeSigningTeamID" : "",
  "codeSigningFlags" : 570425349,
  "codeSigningValidationCategory" : 10,
  "codeSigningTrustLevel" : 0,
  "wakeTime" : 4429,
  "sleepWakeUUID" : "86BFB286-F59A-4981-AAE1-65CAF2D7B023",
  "sip" : "enabled",
  "exception" : {"codes":"0x0000000000000000, 0x0000000000000000","rawCodes":[0,0],"type":"EXC_CRASH","signal":"SIGABRT"},
  "termination" : {"code":4,"flags":518,"namespace":"DYLD","indicator":"Symbol missing","details":["(terminated at launch; ignore backtrace)"],"reasons":["Symbol not found: _NSTextCompletionAttributeName","Referenced from: <28272774-64E6-32B9-BC01-FA47C0CD7E8F> \/Users\/USER\/Documents\/*\/WebKitTestRunner (built for macOS 14.2 which is newer than running OS)","Expected in:     <F66BD32D-B52D-3EFF-8D5C-48D0F7189785> \/System\/Library\/Frameworks\/AppKit.framework\/Versions\/C\/AppKit"]},
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":531602,"threadState":{"r13":{"value":140702028151696},"rax":{"value":33554953},"rflags":{"value":582},"cpu":{"value":0},"r14":{"value":4},"rsi":{"value":4},"r8":{"value":140702028150672},"cr2":{"value":0},"rdx":{"value":140702028151696},"r10":{"value":183},"r9":{"value":0},"r15":{"value":6},"rbx":{"value":0},"trap":{"value":133},"err":{"value":33554953},"r11":{"value":582},"rip":{"value":140703156714214,"matchesCrashFrame":1},"rbp":{"value":140702028150640},"rsp":{"value":140702028150568},"r12":{"value":183},"rcx":{"value":140702028150568},"flavor":"x86_THREAD_STATE","rdi":{"value":6}},"frames":[{"imageOffset":441062,"symbol":"__abort_with_payload","symbolLocation":10,"imageIndex":10},{"imageOffset":548263,"symbol":"abort_with_payload_wrapper_internal","symbolLocation":82,"imageIndex":10},{"imageOffset":548313,"symbol":"abort_with_payload","symbolLocation":9,"imageIndex":10},{"imageOffset":43072,"symbol":"dyld4::halt(char const*)","symbolLocation":375,"imageIndex":10},{"imageOffset":31590,"symbol":"dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*)","symbolLocation":4526,"imageIndex":10},{"imageOffset":25523,"symbol":"start","symbolLocation":1795,"imageIndex":10}]}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4320628736,
    "size" : 1392640,
    "uuid" : "28272774-64e6-32b9-bc01-fa47c0cd7e8f",
    "path" : "\/Users\/USER\/Documents\/*\/WebKitTestRunner",
    "name" : "WebKitTestRunner"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4652146688,
    "CFBundleShortVersionString" : "620+",
    "CFBundleIdentifier" : "com.apple.JavaScriptCore",
    "size" : 57688064,
    "uuid" : "c606b023-b89f-3ad9-a1e1-0bb706803f77",
    "path" : "\/Users\/USER\/Documents\/*\/JavaScriptCore.framework\/Versions\/A\/JavaScriptCore",
    "name" : "JavaScriptCore",
    "CFBundleVersion" : "620.1.5+"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 5923721216,
    "CFBundleShortVersionString" : "620+",
    "CFBundleIdentifier" : "com.apple.WebKit",
    "size" : 92766208,
    "uuid" : "dcdede38-5d0e-3549-ac25-2a2588c44595",
    "path" : "\/Users\/USER\/Documents\/*\/WebKit.framework\/Versions\/A\/WebKit",
    "name" : "WebKit",
    "CFBundleVersion" : "620.1.5+"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4379394048,
    "size" : 6799360,
    "uuid" : "519e5950-d5a4-3f23-9cee-db4b81ae149b",
    "path" : "\/Users\/USER\/Documents\/*\/libWebCoreTestSupport.dylib",
    "name" : "libWebCoreTestSupport.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4815904768,
    "CFBundleShortVersionString" : "620+",
    "CFBundleIdentifier" : "com.apple.WebCore",
    "size" : 153321472,
    "uuid" : "a2f9675d-e119-389e-b176-ff9bb94e74ae",
    "path" : "\/Users\/USER\/Documents\/*\/WebCore.framework\/Versions\/A\/WebCore",
    "name" : "WebCore",
    "CFBundleVersion" : "620.1.5+"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4406542336,
    "CFBundleShortVersionString" : "620+",
    "CFBundleIdentifier" : "com.apple.WebKitLegacy",
    "size" : 5144576,
    "uuid" : "2ec303d9-7d64-3cc7-b8ce-7d7af1b02a41",
    "path" : "\/Users\/USER\/Documents\/*\/WebKitLegacy.framework\/Versions\/A\/WebKitLegacy",
    "name" : "WebKitLegacy",
    "CFBundleVersion" : "620.1.5+"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 5549350912,
    "size" : 30949376,
    "uuid" : "656ec349-3a81-3c5a-86b8-73695352bcde",
    "path" : "\/Users\/USER\/Documents\/*\/libwebrtc.dylib",
    "name" : "libwebrtc.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4325486592,
    "CFBundleShortVersionString" : "620+",
    "CFBundleIdentifier" : "com.apple.WebInspectorUI",
    "size" : 16384,
    "uuid" : "86ab32b0-76f1-35ee-9213-916941c2ddb5",
    "path" : "\/Users\/USER\/Documents\/*\/WebInspectorUI.framework\/Versions\/A\/WebInspectorUI",
    "name" : "WebInspectorUI",
    "CFBundleVersion" : "620.1.5+"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4325535744,
    "size" : 11075584,
    "uuid" : "a4c0bcf3-62de-39ad-8fa5-acf015a35d84",
    "path" : "\/Users\/USER\/Documents\/*\/libANGLE-shared.dylib",
    "name" : "libANGLE-shared.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4478627840,
    "CFBundleShortVersionString" : "620+",
    "CFBundleIdentifier" : "com.apple.WebGPU",
    "size" : 6209536,
    "uuid" : "9c243e6b-7195-39e5-b298-32e7b084123d",
    "path" : "\/Users\/USER\/Documents\/*\/WebGPU.framework\/Versions\/A\/WebGPU",
    "name" : "WebGPU",
    "CFBundleVersion" : "620.1.5+"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703156273152,
    "size" : 623536,
    "uuid" : "2b649d59-89d8-3db6-9ba4-a6aecba42f6e",
    "path" : "\/usr\/lib\/dyld",
    "name" : "dyld"
  }
],
  "sharedCache" : {
  "base" : 140703155650560,
  "size" : 21474836480,
  "uuid" : "774caf66-b420-37ec-a8ec-8d92014cbfbc"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=2.0G resident=0K(0%) swapped_out_or_unallocated=2.0G(100%)\nWritable regions: Total=9332K written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=9332K(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nSTACK GUARD                       56.0M        1 \nStack                             8192K        1 \nVM_ALLOCATE                          8K        2 \n__CTF                               824        1 \n__DATA                            22.7M      370 \n__DATA_CONST                      20.8M      221 \n__DATA_DIRTY                       761K      120 \n__FONT_DATA                        2352        1 \n__LINKEDIT                         1.4G       11 \n__OBJC_RO                         66.3M        1 \n__OBJC_RW                         2013K        2 \n__TEXT                           634.2M      392 \ndyld private memory                260K        2 \nshared memory                        4K        1 \n===========                     =======  ======= \nTOTAL                              2.2G     1126 \n",
  "legacyInfo" : {
  "threadTriggered" : {

  }
},
  "logWritingSignature" : "5a059de395bab8fb7de0f334c133ff9aafcd9e23",
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "64c17a9925d75a7281053d4c",
      "factorPackIds" : {
        "SIRI_AUDIO_DISABLE_MEDIA_ENTITY_SYNC" : "64d29746ad29a465b3bbeace"
      },
      "deploymentId" : 240000002
    },
    {
      "rolloutId" : "6410af69ed1e1e7ab93ed169",
      "factorPackIds" : {

      },
      "deploymentId" : 240000011
    }
  ],
  "experiments" : [

  ]
}
}

[Ben Schwartz]

The answer is this:

`dyld[6890]: Symbol not found: _NSTextCompletionAttributeName Referenced from: <89A995EB-08FD-3D70-A2F0-D3D2F49FCFCF> /Users/iidmsa/Documents/usr/WebKit/WebKitBuild/Release/WebKitTestRunner (built for macOS 14.2 which is newer than running OS)
Expected in:   <F66BD32D-B52D-3EFF-8D5C-48D0F7189785> /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit`

Specifically, "built for macOS 14.2". I don't think that shouldn't be happening... is this affecting testing on the bots on macOS Ventura / x86_64?

[Ben Schwartz]

Thought about this a bit more. Can you post your Xcode version? You can get it by running this in terminal: `xcodebuild -version`.

Also, when running WebKit tests, try to pass `--no-build` as a flag.

[Fady Farag]

(In reply to Ben Schwartz from comment #1)
> The answer is this:
> 
> `dyld[6890]: Symbol not found: _NSTextCompletionAttributeName Referenced
> from: <89A995EB-08FD-3D70-A2F0-D3D2F49FCFCF>
> /Users/iidmsa/Documents/usr/WebKit/WebKitBuild/Release/WebKitTestRunner
> (built for macOS 14.2 which is newer than running OS)
> Expected in:   <F66BD32D-B52D-3EFF-8D5C-48D0F7189785>
> /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit`
> 
> Specifically, "built for macOS 14.2". I don't think that shouldn't be
> happening... is this affecting testing on the bots on macOS Ventura / x86_64?

Thank you for your response. That's indeed what's confusing me, it doesn't affect the test bots running `macOS Ventura / x86_64`. Even though the OS version on the bots is very close:
macOS 13.6.7 (22G720)

[Fady Farag]

(In reply to Ben Schwartz from comment #2)
> Thought about this a bit more. Can you post your Xcode version? You can get
> it by running this in terminal: `xcodebuild -version`.
> 
> Also, when running WebKit tests, try to pass `--no-build` as a flag.

Running `xcodebuild -version` results:
Xcode 15.2
Build version 15C500b

I get the same crashing result when running using the `--no-build` option.

[Ben Schwartz]

You're probably seeing issues because you're using Xcode 15. Try downgrading to the same version that's on the bot that's running tests and try again.

[Fady Farag]

(In reply to Ben Schwartz from comment #5)
> You're probably seeing issues because you're using Xcode 15. Try downgrading
> to the same version that's on the bot that's running tests and try again.

Downgrading Xcode dev tools to the same version that's on the bot(14.3) and running `DevToolsSecurity -enable` resolved the issue. Thank you.