Bug 55236 – [Chromium] .checksum files without .pngs and vice-versa

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Mihai Parparita]

We seem to have a bunch of .checksum expectation files that don't have a .png (and vice-versa). This seems like a bad idea since it can lead to inconsistent/confusing results (e.g. .checksum mismatch even through the .pngs would match).

In directory LayoutTests/platform/chromium-mac/animations
  No PNG:
    state-at-end-event-expected
In directory LayoutTests/platform/chromium-mac/css2.1
  No checksum:
    t0905-c414-flt-01-d-g-expected
    t100801-c544-valgn-03-d-agi-expected
    t100801-c544-valgn-02-d-agi-expected
    t100801-c544-valgn-04-d-agi-expected
In directory LayoutTests/platform/chromium-mac/editing/deleting
  No PNG:
    5390681-2-expected
In directory LayoutTests/platform/chromium-mac/editing/execCommand
  No PNG:
    5481523-expected
    indent-selection-expected
    5138441-expected
In directory LayoutTests/platform/chromium-mac/editing/selection
  No checksum:
    replaced-boundaries-1-expected
    image-before-linebreak-expected
    extend-by-word-002-expected
    replaced-boundaries-2-expected
    4402375-expected
  No PNG:
    select-text-overflow-ellipsis-expected
    4960116-expected
In directory LayoutTests/platform/chromium-mac/fast/backgrounds
  No PNG:
    quirks-mode-line-box-backgrounds-expected
    svg-as-background-2-expected
In directory LayoutTests/platform/chromium-mac/fast/block/positioning
  No PNG:
    absolute-positioning-no-scrollbar-expected
In directory LayoutTests/platform/chromium-mac/fast/blockflow
  No checksum:
    background-vertical-rl-expected
    english-lr-text-expected
    japanese-rl-text-expected
In directory LayoutTests/platform/chromium-mac/fast/borders
  No PNG:
    borderRadiusSolid04-expected
    borderRadiusSolid03-expected
    borderRadiusGroove02-expected
    borderRadiusGroove01-expected
    borderRadiusOutset01-expected
    fieldsetBorderRadius-expected
    borderRadiusSolid01-expected
    borderRadiusDouble02-expected
    borderRadiusDouble03-expected
    borderRadiusArcs01-expected
    borderRadiusInset01-expected
    borderRadiusSolid02-expected
    borderRadiusRidge01-expected
In directory LayoutTests/platform/chromium-mac/fast/css
  No PNG:
    font-face-woff-expected
    css3-space-in-nth-and-lang-expected
    compare-content-style-expected
    apple-prefix-expected
In directory LayoutTests/platform/chromium-mac/fast/css/namespaces
  No PNG:
    namespaces-comments-expected
    namespaces-invalid-at-expected
    namespaces-empty-expected
    namespaces-escapes-expected
In directory LayoutTests/platform/chromium-mac/fast/dom/HTMLMeterElement
  No checksum:
    meter-appearances-capacity-expected
    meter-styles-expected
    meter-styles-changing-pseudo-expected
    meter-optimums-expected
    meter-element-expected
In directory LayoutTests/platform/chromium-mac/fast/forms
  No checksum:
    preserveFormDuringResidualStyle-expected
    input-appearance-bkcolor-expected
    input-align-image-expected
    input-type-change-expected
  No PNG:
    textarea-scrollbar-expected
In directory LayoutTests/platform/chromium-mac/fast/images
  No checksum:
    icon-decoding-expected
    imagemap-focus-ring-expected
In directory LayoutTests/platform/chromium-mac/fast/js
  No PNG:
    missing-title-end-tag-js-expected
In directory LayoutTests/platform/chromium-mac/fast/lists
  No PNG:
    anonymous-items-expected
    marker-before-empty-inline-expected
In directory LayoutTests/platform/chromium-mac/fast/media
  No checksum:
    media-query-invalid-value-expected
In directory LayoutTests/platform/chromium-mac/fast/multicol
  No PNG:
    column-count-with-rules-expected
In directory LayoutTests/platform/chromium-mac/fast/overflow
  No PNG:
    line-clamp-expected
In directory LayoutTests/platform/chromium-mac/fast/reflections
  No PNG:
    opacity-reflection-transform-expected
In directory LayoutTests/platform/chromium-mac/fast/replaced
  No checksum:
    image-tag-expected
In directory LayoutTests/platform/chromium-mac/fast/runin
  No PNG:
    generated3-expected
    generated4-expected
    generated2-expected
In directory LayoutTests/platform/chromium-mac/fast/text
  No PNG:
    stripNullFromText-expected
In directory LayoutTests/platform/chromium-mac/http/tests/misc
  No PNG:
    slow-loading-mask-expected
In directory LayoutTests/platform/chromium-mac/media
  No PNG:
    video-transformed-expected
In directory LayoutTests/platform/chromium-mac/scrollbars
  No checksum:
    overflow-scrollbar-combinations-expected
    listbox-scrollbar-combinations-expected
In directory LayoutTests/platform/chromium-mac/svg/batik/text
  No PNG:
    textProperties-expected
In directory LayoutTests/platform/chromium-mac/svg/css
  No checksum:
    text-gradient-shadow-expected
  No PNG:
    css-box-min-width-expected
In directory LayoutTests/platform/chromium-mac/svg/custom
  No PNG:
    fill-opacity-rgba-expected
    gradient-stop-corner-cases-expected
    container-opacity-clip-viewBox-expected
    fill-opacity-hsl-expected
    mask-invalidation-expected
    massive-coordinates-expected
    transformedMaskFails-expected
In directory LayoutTests/platform/chromium-mac/svg/filters
  No checksum:
    filterRes-expected
    filteredImage-expected
  No PNG:
    feColorMatrix-values-expected
In directory LayoutTests/platform/chromium-mac/svg/text
  No checksum:
    foreignObject-text-clipping-bug-expected
In directory LayoutTests/platform/chromium-mac/svg/W3C-SVG-1.1
  No checksum:
    struct-image-01-t-expected
In directory LayoutTests/platform/chromium-mac/tables/mozilla/bugs
  No checksum:
    bug12908-1-expected
    bug56563-expected
    bug10565-expected
    bug13169-expected
    bug4527-expected
    bug50695-2-expected
    bug4284-expected
    bug17138-expected
    bug120107-expected
    bug196870-expected
    bug12908-2-expected
    bug15544-expected
    bug1271-expected
    bug2981-2-expected
    bug6404-expected
    bug86708-expected
    bug4093-expected
    bug11026-expected
    bug1188-expected
    bug1318-expected
  No PNG:
    bug6674-expected
In directory LayoutTests/platform/chromium-mac/tables/mozilla/core
  No checksum:
    misc-expected
    col_widths_auto_autoFix-expected
In directory LayoutTests/platform/chromium-mac/tables/mozilla/marvin
  No checksum:
    th_valign_middle-expected
    thead_valign_baseline-expected
    tbody_valign_middle-expected
    tfoot_valign_baseline-expected
    thead_valign_middle-expected
    tr_valign_top-expected
    tbody_valign_bottom-expected
    td_valign_baseline-expected
    tr_valign_middle-expected
    tbody_valign_top-expected
    tbody_valign_baseline-expected
    th_valign_top-expected
    thead_valign_top-expected
    th_valign_baseline-expected
    td_valign_top-expected
    td_valign_middle-expected
    tfoot_valign_bottom-expected
    th_valign_bottom-expected
    td_valign_bottom-expected
    tr_valign_bottom-expected
    tfoot_valign_middle-expected
    tr_valign_baseline-expected
    thead_valign_bottom-expected
    tfoot_valign_top-expected
In directory LayoutTests/platform/chromium-mac/tables/mozilla_expected_failures/bugs
  No checksum:
    bug6933-expected
    97619-expected
In directory LayoutTests/platform/chromium-mac/transitions
  No PNG:
    transition-end-event-rendering-expected
In directory LayoutTests/platform/chromium-mac/webarchive
  No PNG:
    test-link-rel-icon-expected

In directory LayoutTests/platform/chromium-win/compositing/repaint
  No checksum:
    become-overlay-composited-layer-expected
In directory LayoutTests/platform/chromium-win/fast/backgrounds/size
  No PNG:
    backgroundSize03-expected
In directory LayoutTests/platform/chromium-win/fast/block/float
  No PNG:
    float-on-zero-height-line-expected
    br-with-clear-expected
    negative-margin-clear-expected
    009-expected
    overhanging-after-height-decrease-offsets-expected
    nowrap-clear-min-width-expected
    034-expected
    narrow-after-wide-expected
    4145535Crash-expected
In directory LayoutTests/platform/chromium-win/fast/block/positioning
  No PNG:
    010-expected
    003-expected
    038-expected
    015-expected
    018-expected
    049-expected
    029-expected
    030-expected
    relative-overconstrained-expected
    028-expected
    021-expected
    017-expected
    013-expected
    006-expected
    035-expected
    005-expected
    042-expected
    039-expected
    041-expected
    024-expected
    move-with-auto-width-expected
    044-expected
    033-expected
    023-expected
    025-expected
    027-expected
    020-expected
    032-expected
    036-expected
    016-expected
    050-expected
    009-expected
    045-expected
    008-expected
    007-expected
    043-expected
    048-expected
    062-expected
    019-expected
    014-expected
    004-expected
    012-expected
    022-expected
    040-expected
    037-expected
    026-expected
    034-expected
    011-expected
    031-expected
    046-expected
In directory LayoutTests/platform/chromium-win/fast/borders
  No PNG:
    outline-offset-min-assert-expected
In directory LayoutTests/platform/chromium-win/fast/canvas
  No PNG:
    canvas-transform-identity-expected
    canvas-transform-infinity-expected
    canvas-transform-multiply-expected
    canvas-transform-nan-expected
    canvas-transform-non-invertible-expected
In directory LayoutTests/platform/chromium-win/fast/clip
  No PNG:
    017-expected
In directory LayoutTests/platform/chromium-win/fast/css
  No PNG:
    last-child-style-sharing-expected
    transition-color-unspecified-expected
    content-dynamic-expected
    affected-by-hover-after-style-change-expected
    006-expected
In directory LayoutTests/platform/chromium-win/fast/dom
  No PNG:
    css-cached-import-rule-expected
    delete-contents-expected
    css-insert-import-rule-expected
In directory LayoutTests/platform/chromium-win/fast/dom/HTMLStyleElement
  No PNG:
    insert-parser-generated-expected
In directory LayoutTests/platform/chromium-win/fast/dynamic
  No PNG:
    staticY-expected
    positioned-movement-with-positioned-children-expected
    subtree-boundary-percent-height-expected
    001-expected
    floating-to-positioned-expected
    subtree-table-cell-height-expected
    005-expected
    float-withdrawal-2-expected
    float-no-longer-overhanging-expected
    genContentDestroyChildren-expected
    floating-to-positioned-2-expected
    subtree-no-common-root-static-y-expected
In directory LayoutTests/platform/chromium-win/fast/events
  No PNG:
    onloadFrameCrash-expected
In directory LayoutTests/platform/chromium-win/fast/forms
  No checksum:
    searchfield-heights-expected
In directory LayoutTests/platform/chromium-win/fast/frames
  No PNG:
    calculate-relative-expected
    empty-cols-attribute-expected
    empty-frame-src-expected
    002-expected
    frame-length-fractional-expected
    calculate-fixed-expected
    calculate-percentage-expected
    calculate-order-expected
    no-frame-borders-expected
    calculate-round-expected
    frame-set-whitespace-attributes-expected
In directory LayoutTests/platform/chromium-win/fast/images
  No PNG:
    pdf-as-tiled-background-expected
    svg-width-100p-as-background-expected
    svg-as-image-expected
    svg-as-relative-image-expected
    svg-as-background-expected
    image-in-map-expected
In directory LayoutTests/platform/chromium-win/fast/inline-block
  No PNG:
    005-expected
    004-expected
In directory LayoutTests/platform/chromium-win/fast/invalid
  No PNG:
    table-residual-style-crash-expected
    015-expected
In directory LayoutTests/platform/chromium-win/fast/js
  No PNG:
    missing-title-end-tag-js-expected
    missing-style-end-tag-js-expected
    exception-linenums-in-html-3-expected
In directory LayoutTests/platform/chromium-win/fast/layers
  No PNG:
    positioned-inside-root-with-margins-expected
    zindex-ridonkulous-expected
In directory LayoutTests/platform/chromium-win/fast/lists
  No PNG:
    outSideListMarkers-expected
In directory LayoutTests/platform/chromium-win/fast/media
  No checksum:
    media-query-invalid-value-expected
In directory LayoutTests/platform/chromium-win/fast/overflow
  No PNG:
    position-relative-expected
    unreachable-content-test-expected
In directory LayoutTests/platform/chromium-win/fast/parser
  No PNG:
    style-script-head-test-expected
In directory LayoutTests/platform/chromium-win/fast/reflections
  No PNG:
    table-cell-expected
In directory LayoutTests/platform/chromium-win/fast/repaint
  No PNG:
    button-spurious-layout-hint-expected
    float-in-new-block-with-layout-delta-expected
In directory LayoutTests/platform/chromium-win/fast/replaced
  No PNG:
    008-expected
In directory LayoutTests/platform/chromium-win/fast/table
  No PNG:
    vertical-align-baseline-readjust-expected
    growCellForImageQuirk-expected
    024-expected
    max-width-integer-overflow-expected
    percent-widths-stretch-expected
    giantCellspacing-expected
    031-expected
    caption-relayout-expected
    div-as-col-span-expected
In directory LayoutTests/platform/chromium-win/fast/text
  No PNG:
    align-center-rtl-spill-expected
    apply-start-width-after-skipped-text-expected
    selection-painted-separately-expected
    soft-hyphen-3-expected
    stripNullFromText-expected
    fixed-pitch-control-characters-expected
In directory LayoutTests/platform/chromium-win/fast/tokenizer
  No PNG:
    missing-style-end-tag-2-expected
    missing-style-end-tag-1-expected
    script-after-frameset-expected
In directory LayoutTests/platform/chromium-win/fast/xsl
  No PNG:
    xslt-mismatched-tags-in-xslt-expected
In directory LayoutTests/platform/chromium-win/printing
  No checksum:
    return-from-printing-mode-expected
In directory LayoutTests/platform/chromium-win/svg/custom
  No PNG:
    transform-ignore-after-invalid-expected
    use-recalcStyle-crash-expected
    inner-svg-hit-test-expected
    fill-update-expected
    inner-percent-expected
    transformedMaskFails-expected
    use-empty-reference-expected
    invalid-transforms-expected
    gradient-add-stops-expected
    clip-path-with-transform-expected
    create-metadata-element-expected
    percentage-rect2-expected
    visibility-override-expected
    polyline-invalid-points-expected
    resource-client-removal-expected
    visibility-override-mask-expected
    evt-onload-expected
    transform-scale-parse-expected
    image-with-aspect-ratio-stretch-expected
    viewport-update2-expected
    dynamic-viewBox-expected
    transform-with-ending-space-expected
    repaint-on-image-bounds-change-expected
    SVGRect-interface-expected
    mask-excessive-malloc-expected
    pattern-in-defs-expected
    viewport-no-width-height-expected
    viewport-clip-expected
    svg-absolute-children-expected
    scroll-hit-test-expected
    xhtml-no-svg-renderer-expected
    SVGNumber-interface-expected
    clip-path-href-changes-expected
    polyline-setattribute-points-null-expected
    SVGPoint-interface-expected
    xml-stylesheet-expected
    css-pixels-dpi-expected
    polyline-hittest-expected
    rgbcolor-syntax-expected
    gradient-attr-update-expected
    visibility-override-clip-expected
    viewport-update-expected
    hover-default-fill-expected
    clip-path-child-changes-expected
    rootelement-expected
    class-selector-expected
    transform-removeAttributeNS-expected
    gradient-stop-style-change-expected
    text-gradient-no-content-expected
    transform-invalid-expected
    display-none-expected
    prevent-default-expected
    sheet-title-expected
    filter-source-alpha-expected
    mask-inside-defs-expected
    createelement-expected
    conditional-processing-outside-switch-expected
    clip-path-units-changes-expected
    poly-identify-expected
    attribute-namespace-check-expected
    use-symbol-overflow-expected
In directory LayoutTests/platform/chromium-win/svg/hixie/dynamic
  No PNG:
    003-expected
    004-expected
    006-expected
In directory LayoutTests/platform/chromium-win/svg/hixie/error
  No PNG:
    007-expected
    001-expected
    015-expected
    014-test-expected
    016-expected
    006-expected
    009-expected
    008-expected
    014-expected
In directory LayoutTests/platform/chromium-win/svg/hixie/painting
  No PNG:
    001-expected
In directory LayoutTests/platform/chromium-win/svg/hixie/processing-model
  No PNG:
    005-expected
In directory LayoutTests/platform/chromium-win/svg/hixie/rendering-model
  No PNG:
    003a-expected
In directory LayoutTests/platform/chromium-win/svg/hixie/use
  No PNG:
    002-expected
    002-test-expected
    001-expected
In directory LayoutTests/platform/chromium-win/svg/hixie/viewbox
  No PNG:
    003-expected
    002-expected
    004-expected
    001-expected
In directory LayoutTests/platform/chromium-win/svg/W3C-SVG-1.1
  No PNG:
    struct-frag-01-t-expected
In directory LayoutTests/platform/chromium-win/svg/webarchive
  No PNG:
    svg-script-subresouces-expected
In directory LayoutTests/platform/chromium-win/tables/mozilla/bugs
  No PNG:
    bug222336-expected
    bug222467-expected
    bug8032-2-expected
    bug9024-expected
    bug269566-expected
    adforce_imgis_com-expected
    bug1818-6-expected
    bug277062-expected
    bug103533-expected
    bug2954-expected
    bug28933-expected
In directory LayoutTests/platform/chromium-win/tables/mozilla/core
  No PNG:
    col_widths_fix_autoFixPer-expected
In directory LayoutTests/platform/chromium-win/tables/mozilla_expected_failures/bugs
  No PNG:
    bug3517-expected
    bug128876-expected
In directory LayoutTests/platform/chromium-win/transitions
  No PNG:
    transition-end-event-rendering-expected
In directory LayoutTests/platform/chromium-win/webarchive
  No PNG:
    test-td-background-expected
    test-img-src-expected
    test-link-rel-icon-expected
    test-body-background-expected
    test-input-src-expected
    test-script-src-expected
    test-object-data-expected
    archive-with-unencoded-url-expected
    test-xml-stylesheet-expected
    test-table-background-expected


In directory LayoutTests/platform/chromium-linux/animations
  No PNG:
    suspend-transform-animation-expected
In directory LayoutTests/platform/chromium-linux/fast/block/float
  No checksum:
    clear-element-too-wide-for-containing-block-expected
In directory LayoutTests/platform/chromium-linux/fast/blockflow
  No checksum:
    english-bt-text-expected
    english-rl-text-expected
    english-lr-text-expected
In directory LayoutTests/platform/chromium-linux/fast/lists
  No PNG:
    outSideListMarkers-expected
In directory LayoutTests/platform/chromium-linux/fast/multicol
  No checksum:
    margin-collapse-expected
    border-padding-pagination-expected
In directory LayoutTests/platform/chromium-linux/svg/text
  No PNG:
    text-hkern-expected
    text-vkern-expected
In directory LayoutTests/platform/chromium-linux/tables/mozilla/bugs
  No checksum:
    bug16012-expected
In directory LayoutTests/platform/chromium-linux/transitions
  No PNG:
    suspend-transform-transition-expected

[Mihai Parparita]

Created attachment 83842 [details]
find-mismatched-results.py

Script that I used to find these

[Tony Chang]

I have a script that generates .checksum from .png files, but it's probably not useful.  Odds are if there's a png and no checksum, we should delete the png.

If there's a checksum and no png, we can probably run the rebaseline script to grab the pngs.

If there's a mismatch, we probably want to manually verify the differences.  The rebaseline script can still help with this.

A pre-submit hook would be useful for catching the missing file cases.

[Tony Chang]

Created attachment 83846 [details]
generate checksums from pngs

For reference, here's how to calculate the checksum in python.  I think it only works for win/linux since mac does something weird with opacity.

[Tony Chang]

Created attachment 84809 [details]
Patch

[Tony Chang]

(In reply to comment #4)
> Created an attachment (id=84809) [details]
> Patch

May as well get find-mismatched-results.py checked in.  This is based on Mihai's code, but I wanted something I could pipe to xargs rm.

[Dirk Pranke]

Comment on attachment 84809 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=84809&action=review

Looks fine otherwise.

> Tools/Scripts/find-mismatched-results:51
> +        option_parser.error("Please --missing-pngs or --missing-checksums.")

Nit. "Please specify either ..."

> Tools/Scripts/find-mismatched-results:55
> +            '..', '..', 'LayoutTests')]

Nit. Should replace this with a webkitpy routine to find the top of the tree. Unfortunately, common.checkout.scm.find_checkout_root() doesn't really do the right thing. layout_tests.port.get().path_from_webkit_base('LayoutTests') should, I think. Maybe add a FIXME?

[Tony Chang]

Created attachment 84985 [details]
Patch

[Mihai Parparita]

Comment on attachment 84985 [details]
Patch

Nit: The name find-mismatched-results is rather generic, maybe find-mismatched-layout-test-results?

[Dirk Pranke]

Comment on attachment 84985 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=84985&action=review

> Tools/ChangeLog:5
> +        [Chromium] .checksum files without .pngs and vice-versa

Nit (if it's not too late): "Find .checksum files ..." ?

[Tony Chang]

Committed r80504: <http://trac.webkit.org/changeset/80504>

[Tony Chang]

I'm going to go ahead and delete pngs without checksums (all ports) and try grabbing pngs from the bots for checksums without pngs (chromium ports only).

We should probably also have a presubmit check for this, but that can be tracked in a different bug.

[Tony Chang]

(In reply to comment #11)
> I'm going to go ahead and delete pngs without checksums (all ports) and try grabbing pngs from the bots for checksums without pngs (chromium ports only).

I deleted pngs without checksums, so `find-mismatched-layout-test-results --missing-checksums` should return nothing now.

I was unable to grab pngs from the bots because passing tests don't have archived results.  I went ahead and landed pngs for chromium-linux, but most of the missing pngs are in chromium-mac.