Bug 161433 – [CMake] CMake does not support the dep files for implicit dependency

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Fujii Hironori]

[CMake] CMake does not support the dep file of preprocess-idls.pl --supplementalMakefileDeps

preprocess-idls.pl has a switch --supplementalMakefileDeps to generate dynamic dependency file (SupplementalDependencies.dep).
And, Source/WebCore/DerivedSources.make includes the file:

> -include $(SUPPLEMENTAL_MAKEFILE_DEPS)

But, CMake does not support the dynamic custom dependencies.

For example, we have a following problem of incremental build:

1) Build
>   ./Tools/Scripts/build-webkit --gtk --debug --64-bit
2) Rename webkitBattery to webkitBattery2
>   sed -i s/webkitBattery/webkitBattery2/ Source/WebCore/Modules/battery/NavigatorBattery.*
3) Build again
>    ./Tools/Scripts/build-webkit --gtk --debug --64-bit
4) But, WebKitBuild/Debug/DerivedSources/WebCore/JSNavigator.cpp is not updated
5) We need to manually touch supplemented IDL files.
>   touch Source/WebCore/page/Navigator.idl

There are three solutions:

A) Use DerivedSources.make (Bug 130339)
B) Generate Makefile for preprocess-idls.pl and generate-bindings.pl and invoke 'make'
C) Implement Make-like tool in Python or Perl

[Fujii Hironori]

(In reply to comment #0)
> There are three solutions:
> 
> A) Use DerivedSources.make (Bug 130339)
> B) Generate Makefile for preprocess-idls.pl and generate-bindings.pl and
> invoke 'make'
> C) Implement Make-like tool in Python or Perl


According to Ninja documentation, it seems to support dep file.
https://ninja-build.org/manual.html#ref_rule
We seem to have fourth choice:

D) Generate bulid.ninja for preprocess-idls.pl and generate-bindings.pl and invoke 'ninja' or 'ninja-build'

[Fujii Hironori]

I've finished a experimental prototyping.

> https://github.com/fujii/webkit/pull/1

There are several commits of refrigerating CMake scripts. This is the
important part:

> https://github.com/fujii/webkit/commit/fffdb707fe123742a5440d5ec4a9c074f1c82d7c

I've got well understanding thought this experiment.

> C) Implement Make-like tool in Python or Perl

This approach is better than others because more precise dependency
updating checking is possible.  For example, my patches solve the
problem updating supplemental_dependency.tmp triggers regenerating all
bindings.

And, Perl is better than Python because generate-bindings.pl is
written in Perl.  If generate-bindings.pl will be converted to Perl
Module, I can stop using fork&exec for speed and avoiding Windows
quoting command problem.

Ninja deletes all output files before executing a command.
So, I need to use add_custom_target.

Can I proceed?

[Carlos Alberto Lopez Perez]

(In reply to comment #2)
> I've finished a experimental prototyping.
> 
> > https://github.com/fujii/webkit/pull/1
> 

You have to upload the patch here, with the r? flag set so a reviewer can take a look into it.

The patch should also contain changelogs (we have a tool to generate them).

Check: https://webkit.org/contributing-code/


Very summarized the process for git users is the following:

1. Commit to a local git branch your patch.
2. Generate a changelog for it as follows:
$ Tools/Scripts/prepare-ChangeLog -b $bugnumer -g $sha1hashofthecommit
3. Edit the changelogs and commit them amending the previous commit
4. Upload the patch here using this:
$ Tools/Scripts/webkit-patch upload -g $sha1hashofthenewammendedcommit --suggest-reviewers --request-commit

[Carlos Alberto Lopez Perez]

(In reply to comment #3)
> (In reply to comment #2)
> > I've finished a experimental prototyping.
> > 
> > > https://github.com/fujii/webkit/pull/1
> > 
> 
> You have to upload the patch here, with the r? flag set so a reviewer can
> take a look into it.
> 
> The patch should also contain changelogs (we have a tool to generate them).
> 
> Check: https://webkit.org/contributing-code/
> 
> 
> Very summarized the process for git users is the following:
> 
> 1. Commit to a local git branch your patch.
> 2. Generate a changelog for it as follows:
> $ Tools/Scripts/prepare-ChangeLog -b $bugnumer -g $sha1hashofthecommit
> 3. Edit the changelogs and commit them amending the previous commit
> 4. Upload the patch here using this:
> $ Tools/Scripts/webkit-patch upload -g $sha1hashofthenewammendedcommit
> --suggest-reviewers --request-commit

If any of the previous commands fails, ensure that you have configured git-svn to track also the svn repository as documented here https://trac.webkit.org/wiki/UsingGitWithWebKit#Checkout and retry.

[Fujii Hironori]

Thank you for the advice, Carlos. I'll do that.

[Fujii Hironori]

Created attachment 290773 [details]
WIP Patch

[Fujii Hironori]

Created attachment 290803 [details]
Patch

[Fujii Hironori]

win EWS fails:

> UIScriptController.idl
>   C:/cygwin/home/buildbot/WebKit/Tools/TestRunnerShared/UIScriptContext/Bindings/UIScriptController.idl
>   : No such file or directory at /home/buildbot/WebKit/Source/WebCore/bindings/scripts/generate-bindings.pl line 79.
>   Thread 1 terminated abnormally: Died at C:/cygwin/home/buildbot/WebKit/Source/WebCore/bindings/scripts/generate-bindings-all.pl line 157.
>   	...propagated at C:/cygwin/home/buildbot/WebKit/Source/WebCore/bindings/scripts/generate-bindings-all.pl line 162.
> C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" exited with code 1. 

EWS uses Cygwin Perl. This problem seems a filename with trailing CR.

> -- Found Perl: C:/cygwin/bin/perl.exe (found suitable version "5.22.2", minimum required is "5.10.0")

[Fujii Hironori]

Created attachment 290810 [details]
Patch

[Fujii Hironori]

Created attachment 290919 [details]
Patch

Rebased the patch for r206904.

[Brent Fulgham]

Comment on attachment 290919 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=290919&action=review

I think this looks good, but I'd like Carlos to give final approval.

> Source/WebCore/ChangeLog:24
> +        * bindings/scripts/generate-bindings-all.pl: Added.

Is the original "generate-bindings.pl" still need after this change?

[Fujii Hironori]

Thank you for reviewing my patch.

(In reply to comment #11)
> Is the original "generate-bindings.pl" still need after this change?

generate-bindings.pl is still used in non-CMake port (Mac port)
and run-bindings-tests.  And, I can not combine these two scripts
without tricks because generate-bindings.pl uses global variables
which prevents multi-threading.

I think it's good idea to do refactoring.
If generate-bindings.pl will be converted to Perl
Module, I can stop using fork&exec for speed and avoiding Windows
quoting command problem.

[Brent Fulgham]

Comment on attachment 290919 [details]
Patch

Since Carlos hasn't objected, I think we should move forward. We can always fix up afterwards if there are any issues.

[WebKit Commit Bot]

Comment on attachment 290919 [details]
Patch

Clearing flags on attachment: 290919

Committed r206972: <http://trac.webkit.org/changeset/206972>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Michael Catanzaro]

Hm, it's very noisy:

[96/3651] Generate bindings (WebCoreBindings)
Preprocess IDL
WebKitPlaybackTargetAvailabilityEvent.idl
BatteryManager.idl
NavigatorBattery.idl
DOMWindowFetch.idl
FetchBody.idl
FetchHeaders.idl
FetchRequest.idl
FetchResponse.idl
WorkerGlobalScopeFetch.idl
Coordinates.idl
Geolocation.idl
Geoposition.idl
NavigatorGeolocation.idl
PositionCallback.idl
PositionError.idl
PositionErrorCallback.idl
DOMWindowIndexedDatabase.idl
IDBCursor.idl
IDBCursorWithValue.idl
IDBDatabase.idl
IDBFactory.idl
IDBIndex.idl
IDBKeyRange.idl
IDBObjectStore.idl
IDBOpenDBRequest.idl
IDBRequest.idl
IDBTransaction.idl
IDBVersionChangeEvent.idl
WorkerGlobalScopeIndexedDatabase.idl
UIRequestEvent.idl
MediaControlsHost.idl
HTMLMediaElementMediaSession.idl
MediaRemoteControls.idl
MediaSession.idl
DOMURLMediaSource.idl
MediaSource.idl
SourceBuffer.idl
SourceBufferList.idl
VideoPlaybackQuality.idl
DOMURLMediaStream.idl
HTMLMediaElementMediaStream.idl
MediaDeviceInfo.idl
MediaDevices.idl
MediaStream.idl
MediaStreamEvent.idl
MediaStreamTrack.idl
MediaStreamTrackEvent.idl
MediaTrackConstraint.idl
MediaTrackConstraintSet.idl
MediaTrackConstraints.idl
MediaTrackSupportedConstraints.idl
NavigatorMediaDevices.idl
NavigatorUserMedia.idl
OverconstrainedError.idl
OverconstrainedErrorEvent.idl
RTCConfiguration.idl
RTCDTMFSender.idl
RTCDTMFToneChangeEvent.idl
RTCDataChannel.idl
RTCDataChannelEvent.idl
RTCIceCandidate.idl
RTCIceCandidateEvent.idl
RTCIceServer.idl
RTCPeerConnection.idl
RTCRtpReceiver.idl
RTCRtpSender.idl
RTCRtpTransceiver.idl
RTCSessionDescription.idl
RTCStatsReport.idl
RTCStatsResponse.idl
RTCTrackEvent.idl
NavigatorContentUtils.idl
DOMWindowNotifications.idl
Notification.idl
NotificationCenter.idl
NotificationPermissionCallback.idl
WorkerGlobalScopeNotifications.idl
DeviceProximityEvent.idl
DOMWindowSpeechSynthesis.idl
SpeechSynthesis.idl
SpeechSynthesisEvent.idl
SpeechSynthesisUtterance.idl
SpeechSynthesisVoice.idl
ByteLengthQueuingStrategy.idl
CountQueuingStrategy.idl
ReadableStream.idl
ReadableStreamDefaultController.idl
ReadableStreamDefaultReader.idl
ReadableStreamSource.idl
WritableStream.idl
NavigatorVibration.idl
AnalyserNode.idl
AudioBuffer.idl
AudioBufferCallback.idl
AudioBufferSourceNode.idl
AudioContext.idl
AudioDestinationNode.idl
AudioListener.idl
AudioNode.idl
AudioParam.idl
AudioProcessingEvent.idl
BiquadFilterNode.idl
ChannelMergerNode.idl
ChannelSplitterNode.idl
ConvolverNode.idl
DelayNode.idl
DynamicsCompressorNode.idl
GainNode.idl
MediaElementAudioSourceNode.idl
MediaStreamAudioDestinationNode.idl
MediaStreamAudioSourceNode.idl
OfflineAudioCompletionEvent.idl
OfflineAudioContext.idl
OscillatorNode.idl
PannerNode.idl
PeriodicWave.idl
ScriptProcessorNode.idl
WaveShaperNode.idl
DOMWindowWebDatabase.idl
Database.idl
DatabaseCallback.idl
SQLError.idl
SQLException.idl
SQLResultSet.idl
SQLResultSetRowList.idl
SQLStatementCallback.idl
SQLStatementErrorCallback.idl
SQLTransaction.idl
SQLTransactionCallback.idl
SQLTransactionErrorCallback.idl
CloseEvent.idl
WebSocket.idl
Animatable.idl
AnimationEffect.idl
AnimationTimeline.idl
DocumentAnimation.idl
DocumentTimeline.idl
KeyframeEffect.idl
WebAnimation.idl
CryptoKey.idl
CryptoKeyPair.idl
SubtleCrypto.idl
WebKitSubtleCrypto.idl
CSSCharsetRule.idl
CSSFontFaceLoadEvent.idl
CSSFontFaceRule.idl
CSSImportRule.idl
CSSKeyframeRule.idl
CSSKeyframesRule.idl
CSSMediaRule.idl
CSSPageRule.idl
CSSPrimitiveValue.idl
CSSRule.idl
CSSRuleList.idl
CSSStyleDeclaration.idl
CSSStyleRule.idl
CSSStyleSheet.idl
CSSSupportsRule.idl
CSSUnknownRule.idl
CSSValue.idl
CSSValueList.idl
Counter.idl
DOMCSSNamespace.idl
FontFace.idl
FontFaceSet.idl
MediaList.idl
MediaQueryList.idl
MediaQueryListListener.idl
RGBColor.idl
Rect.idl
StyleMedia.idl
StyleSheet.idl
StyleSheetList.idl
WebKitCSSFilterValue.idl
WebKitCSSMatrix.idl
WebKitCSSRegionRule.idl
WebKitCSSTransformValue.idl
WebKitCSSViewportRule.idl
AnimationEvent.idl
Attr.idl
BeforeLoadEvent.idl
BeforeUnloadEvent.idl
CDATASection.idl
CharacterData.idl
ChildNode.idl
ClientRect.idl
ClientRectList.idl
ClipboardEvent.idl
Comment.idl
CompositionEvent.idl
CustomElementRegistry.idl
CustomEvent.idl
DOMCoreException.idl
DOMError.idl
DOMImplementation.idl
DOMNamedFlowCollection.idl
DOMStringList.idl
DOMStringMap.idl
DataTransfer.idl
DataTransferItem.idl
DataTransferItemList.idl
DeviceMotionEvent.idl
DeviceOrientationEvent.idl
Document.idl
DocumentFragment.idl
DocumentOrShadowRoot.idl
DocumentType.idl
Element.idl
ErrorEvent.idl
Event.idl
EventInit.idl
EventModifierInit.idl
EventTarget.idl
FocusEvent.idl
GlobalEventHandlers.idl
HashChangeEvent.idl
InputEvent.idl
KeyboardEvent.idl
MessageChannel.idl
MessageEvent.idl
MessagePort.idl
MouseEvent.idl
MutationEvent.idl
MutationObserver.idl
MutationRecord.idl
NamedNodeMap.idl
Node.idl
NodeFilter.idl
NodeIterator.idl
NodeList.idl
NonDocumentTypeChildNode.idl
NonElementParentNode.idl
OverflowEvent.idl
PageTransitionEvent.idl
ParentNode.idl
PopStateEvent.idl
ProcessingInstruction.idl
ProgressEvent.idl
Range.idl
RequestAnimationFrameCallback.idl
SecurityPolicyViolationEvent.idl
ShadowRoot.idl
Slotable.idl
StringCallback.idl
Text.idl
TextEvent.idl
Touch.idl
TouchEvent.idl
TouchList.idl
TransitionEvent.idl
TreeWalker.idl
UIEvent.idl
UIEventInit.idl
WebKitAnimationEvent.idl
WebKitNamedFlow.idl
WebKitTransitionEvent.idl
WheelEvent.idl
XMLDocument.idl
Blob.idl
File.idl
FileError.idl
FileException.idl
FileList.idl
FileReader.idl
FileReaderSync.idl
DOMFormData.idl
DOMTokenList.idl
DOMURL.idl
HTMLAllCollection.idl
HTMLAnchorElement.idl
HTMLAppletElement.idl
HTMLAreaElement.idl
HTMLAttachmentElement.idl
HTMLAudioElement.idl
HTMLBRElement.idl
HTMLBaseElement.idl
HTMLBodyElement.idl
HTMLButtonElement.idl
HTMLCanvasElement.idl
HTMLCollection.idl
HTMLDListElement.idl
HTMLDataElement.idl
HTMLDataListElement.idl
HTMLDetailsElement.idl
HTMLDirectoryElement.idl
HTMLDivElement.idl
HTMLDocument.idl
HTMLElement.idl
HTMLEmbedElement.idl
HTMLFieldSetElement.idl
HTMLFontElement.idl
HTMLFormControlsCollection.idl
HTMLFormElement.idl
HTMLFrameElement.idl
HTMLFrameSetElement.idl
HTMLHRElement.idl
HTMLHeadElement.idl
HTMLHeadingElement.idl
HTMLHtmlElement.idl
HTMLHyperlinkElementUtils.idl
HTMLIFrameElement.idl
HTMLImageElement.idl
HTMLInputElement.idl
HTMLKeygenElement.idl
HTMLLIElement.idl
HTMLLabelElement.idl
HTMLLegendElement.idl
HTMLLinkElement.idl
HTMLMapElement.idl
HTMLMarqueeElement.idl
HTMLMediaElement.idl
HTMLMenuElement.idl
HTMLMetaElement.idl
HTMLMeterElement.idl
HTMLModElement.idl
HTMLOListElement.idl
HTMLObjectElement.idl
HTMLOptGroupElement.idl
HTMLOptionElement.idl
HTMLOptionsCollection.idl
HTMLOutputElement.idl
HTMLParagraphElement.idl
HTMLParamElement.idl
HTMLPictureElement.idl
HTMLPreElement.idl
HTMLProgressElement.idl
HTMLQuoteElement.idl
HTMLScriptElement.idl
HTMLSelectElement.idl
HTMLSlotElement.idl
HTMLSourceElement.idl
HTMLSpanElement.idl
HTMLStyleElement.idl
HTMLTableCaptionElement.idl
HTMLTableCellElement.idl
HTMLTableColElement.idl
HTMLTableElement.idl
HTMLTableRowElement.idl
HTMLTableSectionElement.idl
HTMLTemplateElement.idl
HTMLTextAreaElement.idl
HTMLTimeElement.idl
HTMLTitleElement.idl
HTMLTrackElement.idl
HTMLUListElement.idl
HTMLUnknownElement.idl
HTMLVideoElement.idl
ImageData.idl
MediaController.idl
MediaError.idl
RadioNodeList.idl
TextMetrics.idl
TimeRanges.idl
URLSearchParams.idl
URLUtils.idl
ValidityState.idl
VoidCallback.idl
CanvasGradient.idl
CanvasPath.idl
CanvasPattern.idl
CanvasProxy.idl
CanvasRenderingContext2D.idl
DOMPath.idl
CommandLineAPIHost.idl
InspectorFrontendHost.idl
DOMApplicationCache.idl
BarProp.idl
Crypto.idl
DOMSelection.idl
DOMWindow.idl
EventSource.idl
GlobalCrypto.idl
History.idl
Location.idl
Navigator.idl
NavigatorConcurrentHardware.idl
NavigatorID.idl
NavigatorLanguage.idl
NavigatorOnLine.idl
Performance.idl
PerformanceEntry.idl
PerformanceMark.idl
PerformanceMeasure.idl
PerformanceNavigation.idl
PerformanceResourceTiming.idl
PerformanceTiming.idl
Screen.idl
ScrollToOptions.idl
WebKitPoint.idl
WindowEventHandlers.idl
WindowOrWorkerGlobalScope.idl
WorkerNavigator.idl
DOMMimeTypeArray.idl
DOMMimeType.idl
DOMPlugin.idl
DOMPluginArray.idl
Storage.idl
StorageEvent.idl
AbstractWorker.idl
DedicatedWorkerGlobalScope.idl
Worker.idl
WorkerGlobalScope.idl
WorkerLocation.idl
DOMParser.idl
XMLHttpRequest.idl
XMLHttpRequestEventTarget.idl
XMLHttpRequestProgressEvent.idl
XMLHttpRequestUpload.idl
XMLSerializer.idl
XPathEvaluator.idl
XPathException.idl
XPathExpression.idl
XPathNSResolver.idl
XPathResult.idl
XSLTProcessor.idl
SVGAElement.idl
SVGAltGlyphDefElement.idl
SVGAltGlyphElement.idl
SVGAltGlyphItemElement.idl
SVGAngle.idl
SVGAnimateColorElement.idl
SVGAnimateElement.idl
SVGAnimateMotionElement.idl
SVGAnimateTransformElement.idl
SVGAnimatedAngle.idl
SVGAnimatedBoolean.idl
SVGAnimatedEnumeration.idl
SVGAnimatedInteger.idl
SVGAnimatedLength.idl
SVGAnimatedLengthList.idl
SVGAnimatedNumber.idl
SVGAnimatedNumberList.idl
SVGAnimatedPreserveAspectRatio.idl
SVGAnimatedRect.idl
SVGAnimatedString.idl
SVGAnimatedTransformList.idl
SVGAnimationElement.idl
SVGCircleElement.idl
SVGClipPathElement.idl
SVGColor.idl
SVGComponentTransferFunctionElement.idl
SVGCursorElement.idl
SVGDefsElement.idl
SVGDescElement.idl
SVGDocument.idl
SVGElement.idl
SVGEllipseElement.idl
SVGException.idl
SVGExternalResourcesRequired.idl
SVGFEBlendElement.idl
SVGFEColorMatrixElement.idl
SVGFEComponentTransferElement.idl
SVGFECompositeElement.idl
SVGFEConvolveMatrixElement.idl
SVGFEDiffuseLightingElement.idl
SVGFEDisplacementMapElement.idl
SVGFEDistantLightElement.idl
SVGFEDropShadowElement.idl
SVGFEFloodElement.idl
SVGFEFuncAElement.idl
SVGFEFuncBElement.idl
SVGFEFuncGElement.idl
SVGFEFuncRElement.idl
SVGFEGaussianBlurElement.idl
SVGFEImageElement.idl
SVGFEMergeElement.idl
SVGFEMergeNodeElement.idl
SVGFEMorphologyElement.idl
SVGFEOffsetElement.idl
SVGFEPointLightElement.idl
SVGFESpecularLightingElement.idl
SVGFESpotLightElement.idl
SVGFETileElement.idl
SVGFETurbulenceElement.idl
SVGFilterElement.idl
SVGFilterPrimitiveStandardAttributes.idl
SVGFitToViewBox.idl
SVGFontElement.idl
SVGFontFaceElement.idl
SVGFontFaceFormatElement.idl
SVGFontFaceNameElement.idl
SVGFontFaceSrcElement.idl
SVGFontFaceUriElement.idl
SVGForeignObjectElement.idl
SVGGElement.idl
SVGGlyphElement.idl
SVGGlyphRefElement.idl
SVGGradientElement.idl
SVGGraphicsElement.idl
SVGHKernElement.idl
SVGImageElement.idl
SVGLength.idl
SVGLengthList.idl
SVGLineElement.idl
SVGLinearGradientElement.idl
SVGMPathElement.idl
SVGMarkerElement.idl
SVGMaskElement.idl
SVGMatrix.idl
SVGMetadataElement.idl
SVGMissingGlyphElement.idl
SVGNumber.idl
SVGNumberList.idl
SVGPaint.idl
SVGPathElement.idl
SVGPathSeg.idl
SVGPathSegArcAbs.idl
SVGPathSegArcRel.idl
SVGPathSegClosePath.idl
SVGPathSegCurvetoCubicAbs.idl
SVGPathSegCurvetoCubicRel.idl
SVGPathSegCurvetoCubicSmoothAbs.idl
SVGPathSegCurvetoCubicSmoothRel.idl
SVGPathSegCurvetoQuadraticAbs.idl
SVGPathSegCurvetoQuadraticRel.idl
SVGPathSegCurvetoQuadraticSmoothAbs.idl
SVGPathSegCurvetoQuadraticSmoothRel.idl
SVGPathSegLinetoAbs.idl
SVGPathSegLinetoHorizontalAbs.idl
SVGPathSegLinetoHorizontalRel.idl
SVGPathSegLinetoRel.idl
SVGPathSegLinetoVerticalAbs.idl
SVGPathSegLinetoVerticalRel.idl
SVGPathSegList.idl
SVGPathSegMovetoAbs.idl
SVGPathSegMovetoRel.idl
SVGPatternElement.idl
SVGPoint.idl
SVGPointList.idl
SVGPolygonElement.idl
SVGPolylineElement.idl
SVGPreserveAspectRatio.idl
SVGRadialGradientElement.idl
SVGRect.idl
SVGRectElement.idl
SVGRenderingIntent.idl
SVGSVGElement.idl
SVGScriptElement.idl
SVGSetElement.idl
SVGStopElement.idl
SVGStringList.idl
SVGStyleElement.idl
SVGSwitchElement.idl
SVGSymbolElement.idl
SVGTRefElement.idl
SVGTSpanElement.idl
SVGTests.idl
SVGTextContentElement.idl
SVGTextElement.idl
SVGTextPathElement.idl
SVGTextPositioningElement.idl
SVGTitleElement.idl
SVGTransform.idl
SVGTransformList.idl
SVGURIReference.idl
SVGUnitTypes.idl
SVGUseElement.idl
SVGVKernElement.idl
SVGViewElement.idl
SVGViewSpec.idl
SVGZoomAndPan.idl
SVGZoomEvent.idl
ANGLEInstancedArrays.idl
EXTBlendMinMax.idl
EXTFragDepth.idl
EXTShaderTextureLOD.idl
EXTTextureFilterAnisotropic.idl
EXTsRGB.idl
OESElementIndexUint.idl
OESStandardDerivatives.idl
OESTextureFloat.idl
OESTextureFloatLinear.idl
OESTextureHalfFloat.idl
OESTextureHalfFloatLinear.idl
OESVertexArrayObject.idl
WebGL2RenderingContext.idl
WebGLActiveInfo.idl
WebGLBuffer.idl
WebGLCompressedTextureATC.idl
WebGLCompressedTexturePVRTC.idl
WebGLCompressedTextureS3TC.idl
WebGLContextAttributes.idl
WebGLContextEvent.idl
WebGLDebugRendererInfo.idl
WebGLDebugShaders.idl
WebGLDepthTexture.idl
WebGLDrawBuffers.idl
WebGLFramebuffer.idl
WebGLLoseContext.idl
WebGLProgram.idl
WebGLQuery.idl
WebGLRenderbuffer.idl
WebGLRenderingContext.idl
WebGLRenderingContextBase.idl
WebGLSampler.idl
WebGLShader.idl
WebGLShaderPrecisionFormat.idl
WebGLSync.idl
WebGLTexture.idl
WebGLTransformFeedback.idl
WebGLUniformLocation.idl
WebGLVertexArrayObject.idl
WebGLVertexArrayObjectOES.idl
AudioTrack.idl
AudioTrackList.idl
DataCue.idl
TextTrack.idl
TextTrackCue.idl
TextTrackCueList.idl
TextTrackList.idl
TrackEvent.idl
VTTCue.idl
VTTRegion.idl
VTTRegionList.idl
VideoTrack.idl
VideoTrackList.idl
UserMessageHandler.idl
UserMessageHandlersNamespace.idl
WebKitNamespace.idl
[110/3651] Generate bindings (WebKitTestRunnerInjectedBundleBindings)
AccessibilityController.idl
AccessibilityTextMarker.idl
AccessibilityTextMarkerRange.idl
AccessibilityUIElement.idl
EventSendingController.idl
GCController.idl
TestRunner.idl
TextInputController.idl
[779/3651] Generate bindings (WebKitTestRunnerBindings)
UIScriptController.idl
[2995/3651] Generate bindings (WebCoreTestSupportBindings)
GCObservation.idl
InternalSettings.idl
Internals.idl
MallocStatistics.idl
MemoryInfo.idl
MockContentFilterSettings.idl
MockPageOverlay.idl
TypeConversions.idl
InternalSettingsGenerated.idl

We previously had a nice clean build. Can we make it not print, please?

[Carlos Garcia Campos]

It also seems to be doing some commands now always, even on incremental builds when nothing changed

[0/935] Generate bindings (WebCoreBindings)
[2/38] Generate bindings (WebKitTestRunnerBindings)
[3/36] Generate bindings (WebCoreTestSupportBindings)
[4/24] Generate bindings (WebKitTestRunnerInjectedBundleBindings)

[Brent Fulgham]

We are seeing some EWS flakiness on Windows as well. Could you please look into this ASAP?

Here's the EWS error:
Thread 2 terminated abnormally: Died at C:/cygwin/home/buildbot/WebKit/Source/WebCore/bindings/scripts/generate-bindings-all.pl line 157.

Here's that line in the perl:
                system($perl, "$scriptDir/generate-bindings.pl", @args, $file) == 0 or die;

I'm not sure why this is failing.

I got this to happen in a local Windows build, but when I built a second time it seemed to bypass the error.

[WebKit Commit Bot]

Re-opened since this is blocked by bug 163227

[Brent Fulgham]

I'm (regretfully) rolling this change out, because it has made EWS flaky and is causing some unwanted noise for GTK builds.

I don't have enough deep CMake skills to understand why this is happening, but I'd like to back it out until Fujii has chance to review the issue.

[Fujii Hironori]

(In reply to comment #20)
> I'm (regretfully) rolling this change out, because it has made EWS flaky and
> is causing some unwanted noise for GTK builds.

I'm sorry for causing the serious problem and thank you for
rolling back, Brent.

(In reply to comment #16)
> Hm, it's very noisy:
> 
> [96/3651] Generate bindings (WebCoreBindings)
> Preprocess IDL
(..)
> We previously had a nice clean build. Can we make it not print, please?

I think it should output something because it takes long time to
generate all bindings.  I'll improve the progress output as Ninja
does.


(In reply to comment #17)
> It also seems to be doing some commands now always, even on incremental
> builds when nothing changed
> 
> [0/935] Generate bindings (WebCoreBindings)
> [2/38] Generate bindings (WebKitTestRunnerBindings)
> [3/36] Generate bindings (WebCoreTestSupportBindings)
> [4/24] Generate bindings (WebKitTestRunnerInjectedBundleBindings)

This is intended. It checks which IDL needs to be regenerated by
checking timestamp as well as generate-forwarding-headers.pl
does.

[Fujii Hironori]

I've succeeded to reproduce this flaky problem.  The reproduction
ratio of this problem depends on which Perl is used and which version
of Cygwin is used.

64bit ActivePerl : never happen
64bit latest Cygwin Perl: never happen
32bit latest Cygwin Perl: I saw the problem only once
32bit 5-month-old Cygwin Perl: about 10%
32bit 3-year-old Cygwin Perl: never happen

I dumped $! and $?.

>  Can't exec "/usr/bin/perl": Bad address at C:/home/0000134108/work/webkit/webkit1/Source/WebCore/bindings/scripts/generate-bindings-all.pl line 159.
>  Thread 8 terminated abnormally: generate-bindings.pl fails: ($!=Bad address)($?=-1) at C:/home/0000134108/work/webkit/webkit1/Source/WebCore/bindings/scripts/generate-bindings-all.pl line 159.
>  	...propagated at C:/home/0000134108/work/webkit/webkit1/Source/WebCore/bindings/scripts/generate-bindings-all.pl line 164.
>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" exited with code 1.

Sometimes different error message and no error message:

>  SVGScriptElement.idl
>  SVGStringList.idl
>        1 [main] perl 3588 cygwin_exception::open_stackdumpfile: Dumping stack trace to perl.exe.stackdump
>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" exited with code 1536.

This seems Cygwin fork failure problem.
https://cygwin.com/faq/faq.html#faq.using.fixing-fork-failures

I'd like to request installing ActiveState Perl on BuildBot as
described in this page <https://webkit.org/webkit-on-windows/>.
And, Uninstall Cygwin because cmake script append a path to cygwin.
<https://trac.webkit.org/browser/trunk/Source/cmake/WebKitCommon.cmake?rev=207060#L13>

[Fujii Hironori]

Hi Alex, Blent:

You added Cygwin path to CMAKE_PROGRAM_PATH in Bug 152937 for
Apple internal Windows build. Is there a reason to need to use
Cygwin tools yet? Why don't you use Windows native tools? I have
a trouble with Cygwin Perl.

[Fujii Hironori]

CygwinDownloader has perl-libwin32.
  <https://trac.webkit.org/browser/trunk/Tools/CygwinDownloader/cygwin-downloader.py?rev=167430#L63>

It seems to be able to use Win32::Job perl module for the workaround.

  Win32::Job
  http://search.cpan.org/~gsar/libwin32-0.191/Job/Job.pm

I'll try.

[Fujii Hironori]

Created attachment 291471 [details]
Patch

* Use Win32::Job for the workaround
* Improve the progress output such like Ninja does

[Fujii Hironori]

Comment on attachment 291471 [details]
Patch

Win EWS failed:

>  Can't locate Win32/Job.pm in @INC (you may need to install the Win32::Job module) (@INC contains: /home/buildbot/WebKit/Source/WebCore/bindings/scripts /usr/lib/perl5/site_perl/5.22/i686-cygwin-threads-64int /usr/lib/perl5/site_perl/5.22 /usr/lib/perl5/vendor_perl/5.22/i686-cygwin-threads-64int /usr/lib/perl5/vendor_perl/5.22 /usr/lib/perl5/5.22/i686-cygwin-threads-64int /usr/lib/perl5/5.22 .) at /usr/lib/perl5/5.22/if.pm line 13.

No luck.

[Alex Christensen]

(In reply to comment #23)
> Hi Alex, Blent:
> 
> You added Cygwin path to CMAKE_PROGRAM_PATH in Bug 152937 for
> Apple internal Windows build. Is there a reason to need to use
> Cygwin tools yet? Why don't you use Windows native tools? I have
> a trouble with Cygwin Perl.

Our internal build needs to continue to use Cygwin.  The WinCairo bot doesn't even have Cygwin installed.  I agree that's the preferred way to do development on Windows, but we need to continue to support Cygwin builds.

[Fujii Hironori]

Created attachment 291573 [details]
Minimal test case for Cygwin Perl

This is a minimal test case for Cygwin Perl.  Cygwin Perl fails
to execute a external command with 'system' function in
multi-threading.  This problem seems that 'system' of Cygwin Perl
is not thread-safe.  Not a Cygwin fork problem.  It works fine if
I use fork&exec instread of 'system' even in Cygwin Perl.  I'll
rewrite the patch.

[Fujii Hironori]

Created attachment 291589 [details]
Patch

* Use fork&exec if Cygwin Perl is used
* Fix a bug of getting TTY width

[Fujii Hironori]

Created attachment 291790 [details]
Patch

* Merged r207311

[Brent Fulgham]

Comment on attachment 291790 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=291790&action=review

Looks good. I'm running it on a local Windows build to make sure it works on our standard setup.

> Source/WebCore/bindings/scripts/generate-bindings-all.pl:83
> +        executeCommand($perl, "$scriptDir/preprocess-idls.pl", @args) == 0 or die;

This is fine, but I prefer "File::Spec->catfile" and related so we don't have to worry about path separator differences for WIndows versus Unix.

> Source/WebCore/bindings/scripts/generate-bindings-all.pl:146
> +            # print "$_ is newer than $oldestObject\n";

Should remove this debugging statement.

[Brent Fulgham]

This worked very nicely on my local Cygwin-based build!

Fujii, could you please correct those two minor comments I made when you land this?

[Fujii Hironori]

Created attachment 292147 [details]
Patch

Thank you for r+ and testing my patch.
I updated the patch.  Please give me cq+.

* Use File::Spec->catfile for preprocess-idls.pl and generate-bindings.pl
* Removed the debugging statement
* Fix the problem 'stty is not recognized' is output in Windows Ninja Win32 Perl build.

[WebKit Commit Bot]

Comment on attachment 292147 [details]
Patch

Clearing flags on attachment: 292147

Committed r207617: <http://trac.webkit.org/changeset/207617>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Fujii Hironori]

Coming CMake 3.7 will finally have DEPFILE argument of add_custom_command only for Ninja generator.
https://cmake.org/cmake/help/v3.7/command/add_custom_command.html