Bug 218519 – [GTK] Can't paste image data (like image/png) from the clipboard in WebKit browsers

• Component: Platform
• Status: NEW

[Miikka]

I'm unable to paste any images from my clipboard into browser windows using WebKitGtk (like Epiphany or WebKitGtk MiniBrowser).

I've verified that the data certainly is in the clipboard because following commands:
xclip -selection clipboard -t image/png -o > image.png
xclip -selection clipboard -t image/bmp -o > image.bmp
outputs proper bitmap or PNG files. And I'm also able to paste the data in Gimp, Firefox, LibreOffice and Chromium. Only the WebKitGtk based browser does not work properly so I assume the problem is with my WebKitGtk. I compiled webkitgtk-2.30.2 and Epiphany from git (2020-10-29) manually but the problem still exists in Epiphany and in the MiniBrowser.

Could there be some library I'm missing or something?

I found one bug that could relate: https://bugs.webkit.org/show_bug.cgi?id=211519 but in there there is broken image area when the data has been pasted. With my problem the browser does nothing when I hit Ctrl+V (like in chat/email window or in imgur.com).

I compiled WebKitGtk with the following arguments:
cmake .. -DCMAKE_BUILD_TYPE=Release -DPORT=GTK -DENABLE_MINIBROWSER=ON -DCMAKE_EXE_LINKER_FLAGS="-Wl,--as-needed -Wl,-z,now -pthread" -DCMAKE_MODULE_LINKER_FLAGS="-Wl,--as-needed -Wl,-z,now -pthread" -DCMAKE_SHARED_LINKER_FLAGS="-Wl,--as-needed -Wl,-z,now -pthread" -DUSE_WOFF2=false -DENABLE_MEDIA_SOURCE=OFF -DPYTHON_EXECUTABLE=/usr/bin/python3:
-- Enabled features:
--  ENABLE_ACCELERATED_2D_CANVAS .................. OFF
--  ENABLE_BUBBLEWRAP_SANDBOX                       ON
--  ENABLE_DRAG_SUPPORT ........................... ON
--  ENABLE_GLES2                                    OFF
--  ENABLE_GRAPHICS_CONTEXT_GL .................... ON
--  ENABLE_GTKDOC                                   OFF
--  ENABLE_INTROSPECTION .......................... ON
--  ENABLE_MEDIA_SOURCE                             OFF
--  ENABLE_MINIBROWSER ............................ ON
--  ENABLE_QUARTZ_TARGET                            OFF
--  ENABLE_SHAREABLE_RESOURCE ..................... ON
--  ENABLE_SPELLCHECK                               ON
--  ENABLE_TOUCH_EVENTS ........................... ON
--  ENABLE_VIDEO                                    ON
--  ENABLE_WAYLAND_TARGET ......................... ON
--  ENABLE_WEBDRIVER                                ON
--  ENABLE_WEB_AUDIO .............................. ON
--  ENABLE_WEB_CRYPTO                               ON
--  ENABLE_X11_TARGET ............................. ON
--  USE_GTK4                                        OFF
--  USE_LIBHYPHEN ................................. ON
--  USE_LIBNOTIFY                                   ON
--  USE_LIBSECRET ................................. ON
--  USE_OPENJPEG                                    ON
--  USE_SYSTEMD ................................... ON
--  USE_WOFF2                                       false
--  USE_WPE_RENDERER .............................. ON
-- Configuring done
-- Generating done

Distro: openSUSE Leap 15.2
DE: KDE/Plasma 5.18.5

[Carlos Garcia Campos]

Could you try pasting to MiniBrowser started with -e (editor mode). I guess this is on xorg?

[Miikka]

Yes, it's Xorg I'm using. In the editor mode the pasting of image data from the clipboard works!

I also created a separate bug report to Epiphany but it wasn't correct place to submit that: https://gitlab.gnome.org/GNOME/epiphany/-/issues/1388

I've also noticed that pasting works on Evolution email client when I'm writing/composing email. I can embed images from clipboard directly into email.

Way to reproduce: Select and copy some image data from Gimp into clipboard for example, in Epiphany go to imgur.com and create new post, hit CTRL+V: nothing happens. If I do this in Chromium or Firefox the browser delivers the image data to imgur and the new post gets created.

[Miikka]

Is there anything I should try? I've been able to reproduce this on openSUSE 15.2 Leap and on Debian testing (Bullseye).

[Miikka]

Bump!

[Sonny Piers]

I ran some tests

Everything works fine on MiniBrowser -e (both Wayland and Xorg)

However, on non-editable context it is broken on both Wayland and Xorg.

The paste event https://w3c.github.io/clipboard-apis/#clipboard-event-paste is emitted with a DataTransferItemList length of 0.

I used https://codepen.io/tmrDevelops/pen/YxGQaW to reproduce.

To clarify, file:// pasting works fine - only pasting [Image] data is broken.

[Sonny Piers]

Actually a much better test case is to run 

window.addEventListener('paste', function(e){
  console.log(e.clipboardData.items.length)
});

in the JavaScript console and then paste data.

Both on editable and non-editable contexts it logs 0 when pasting [Image].

[Miikka]

Hi Sonny! Thanks, I'm glad to hear that someone else was able to reproduce this. So I can be sure that it does not happen only with my setup.

[Carlos Garcia Campos]

I can reproduce it now. I don't think this is specific to the GTK port. DataTransfer::items() only returns items that are considered to be safe for DOM, see:

bool Pasteboard::isSafeTypeForDOMToReadAndWrite(const String& type)
{
    return type == "text/plain" || type == "text/html" || type == "text/uri-list";
}

So, images are not allowed. I don't know if that's WebKit specific or part of a spec, though.

[Miikka]

Thanks Carlos.

I tried to modify that particular code in Source/WebCore/platform/Pasteboard.cpp:
bool Pasteboard::isSafeTypeForDOMToReadAndWrite(const String& type)
{   
    return type == "text/plain" || type == "text/html" || type == "text/uri-list";
}
to:
bool Pasteboard::isSafeTypeForDOMToReadAndWrite(const String& type)
{   
    return type == "text/plain" || type == "text/html" || type == "text/uri-list" || type == "image/apng" || type == "image/avif" || type == "image/gif" || type == "image/jpeg" || type == "image/png" || type == "image/svg+xml" || type == "image/webp";
}

And built the WebKitGtk but that didn't fix this issue at least (still unable to paste any image data from the clipboard into WebKit MiniBrowser or Epiphany).

Could this issue be related to these changes?:
https://webkit.org/blog/8170/clipboard-api-improvements/
https://webkit.org/blog/10855/async-clipboard-api/

What do you think, should I move this issue to some other Component or create completely new issue and ref to this?

[Miikka]

Actually I found also something interesting from the file: Source/WebCore/Modules/async-clipboard/Clipboard.cpp:

    if (type == "text/uri-list"_s) {
        String title;
        resultAsString = activePasteboard().readURL(itemIndex, title).string();
    }

    if (type == "text/plain"_s) {
        PasteboardPlainText plainTextReader;
        activePasteboard().read(plainTextReader, PlainTextURLReadingPolicy::IgnoreURL, itemIndex);
        resultAsString = WTFMove(plainTextReader.text);
    }

    if (type == "text/html"_s) {
        WebContentMarkupReader markupReader { *frame };
        activePasteboard().read(markupReader, WebContentReadingPolicy::OnlyRichTextTypes, itemIndex);
        resultAsString = WTFMove(markupReader.markup);
    }   

    // FIXME: Support reading custom data.
    if (updateSessionValidity() == SessionIsValid::No || resultAsString.isNull()) {
        promise->reject(NotAllowedError);
        return;
    }

Could that be related to this issue?

[Miikka]

And this:

File: Source/WebCore/platform/gtk/PasteboardGtk.cpp:
void Pasteboard::read(PasteboardPlainText& text, PlainTextURLReadingPolicy, Optional<size_t>):
    static const ASCIILiteral imageTypes[] = { "image/png"_s, "image/jpeg"_s, "image/gif"_s, "image/bmp"_s, "image/vnd.microsoft.icon"_s, "image/x-icon"_s };
    for (const auto& imageType : imageTypes) {
        if (types.contains(imageType)) {
            auto buffer = platformStrategies()->pasteboardStrategy()->readBufferFromClipboard(m_name, imageType);
            if (!buffer->isEmpty() && reader.readImage(buffer.releaseNonNull(), imageType))
                return;
        }
    }

[Carlos Garcia Campos]

What is probably failing after allowing images is getAsFile(), I don't know why that should work for images, but it's returning null I'm afraid.

[Miikka]

Should I move this bug report from WebKitGTK Component to something else?

[Miikka]

Hmm, is there any progress with this? I think all of my WebKit based browsers on Linux are lacking the support of pasting images from the clipboard because of this issue?

[荒野無燈]

I already found a way to fix up this, and it works good. Need sometime to figure out how to create a patch, seems that submit a patch to webkit is a little bit difficult to me. (I mean, in github I only need one click and submit the Pull Request)

[Michael Catanzaro]

Hi, there are guidelines for contributing patches here: https://webkit.org/contributing-code/

It would also be good to mention what exactly you've discovered....

[Miikka]

(In reply to 荒野無燈 from comment #15)
> I already found a way to fix up this, and it works good. Need sometime to
> figure out how to create a patch, seems that submit a patch to webkit is a
> little bit difficult to me. (I mean, in github I only need one click and
> submit the Pull Request)

Sounds good! Could you please share the patch somewhere so I can test it too?

[荒野無燈]

Created attachment 433288 [details]
fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get no files, this fix makes ctrl + v paste image under GTK  works

in Source/WebCore/dom/DataTransfer.cpp  DataTransfer::filesFromPasteboardAndItemList

```cpp
 WebCorePasteboardFileReader reader(context);
 m_pasteboard->read(reader);
 files = WTFMove(reader.files);
```

it only calls `void Pasteboard::read(PasteboardFileReader& reader, std::optional<size_t>)`, 

and `Pasteboard::read(PasteboardWebContentReader& reader, WebContentReadingPolicy policy, std::optional<size_t>)` got no chances, so the image data dit not got read.

this patch is a workaround to fixup the problem (tested under webkitgtk 2.32.0 and 2.32.1).

by patching `void Pasteboard::read(PasteboardFileReader& reader, std::optional<size_t>)`,  and call `WebCorePasteboardFileReader::readBuffer`, we can make `DataTransferItem::getAsFile()` get the image file data.

[荒野無燈]

I have some working builds if you need quick verify if it works.

-----------------------------------------------------------

libwebkit2gtk-4.0-37_2.32.0 for Ubuntu 20.04.2:

curl -LO https://github.com/ttys3/lark-for-linux/releases/download/v0.9.4/libwebkit2gtk-4.0-37_2.32.0-patched-ubuntu-20.04.2-deb.tar.xz

tar xvf libwebkit2gtk-4.0-37_2.32.0-patched-ubuntu-20.04.2-deb.tar.xz

cd libwebkit2gtk-4.0-37_2.32.0*

sudo apt install -y --allow-downgrades ./*.deb


use https://github.com/ttys3/lark-for-linux/releases/download/v0.9.4/libwebkit2gtk-4.0-37_2.32.0-patched-ubuntu-21.04-gnome40-deb.tar.xz
if you are Ubuntu 21.04 and has GNOME 40 installed.
-------------------------------------------------------------

download and install https://github.com/ttys3/lark-for-linux/releases/download/v0.9.4/webkit2gtk-2.32.1-1-x86_64.pkg.tar.zst if you are a ArchLinux user.

[Michael Catanzaro]

(In reply to 荒野無燈 from comment #18)
> Created attachment 433288 [details]
> fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> no files, this fix makes ctrl + v paste image under GTK  works

I have no doubt that it works, but that can't be the correct fix. readFilePathsFromClipboard should do what it says, not more. I haven't looked closely, but you might be one level too far down from the right place to fix this.

[荒野無燈]

(In reply to Michael Catanzaro from comment #20)
> (In reply to 荒野無燈 from comment #18)
> > Created attachment 433288 [details]
> > fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> > no files, this fix makes ctrl + v paste image under GTK  works
> 
> I have no doubt that it works, but that can't be the correct fix.
> readFilePathsFromClipboard should do what it says, not more. I haven't
> looked closely, but you might be one level too far down from the right place
> to fix this.

Thanks for the quick reply. I think you are right, this is not a correct fix.

I'm not familiar with the WebKit code at all. And I'm just a normal user which wait the correct patch for about 8 months or maybe longer.

I can not do further investigation to finish the right patch.

And hope someone will continue the work.

Thank you all guys.

[Miikka]

(In reply to 荒野無燈 from comment #18)
> Created attachment 433288 [details]
> fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> no files, this fix makes ctrl + v paste image under GTK  works
> 
> in Source/WebCore/dom/DataTransfer.cpp 
> DataTransfer::filesFromPasteboardAndItemList
> 
> ```cpp
>  WebCorePasteboardFileReader reader(context);
>  m_pasteboard->read(reader);
>  files = WTFMove(reader.files);
> ```
> 
> it only calls `void Pasteboard::read(PasteboardFileReader& reader,
> std::optional<size_t>)`, 
> 
> and `Pasteboard::read(PasteboardWebContentReader& reader,
> WebContentReadingPolicy policy, std::optional<size_t>)` got no chances, so
> the image data dit not got read.
> 
> this patch is a workaround to fixup the problem (tested under webkitgtk
> 2.32.0 and 2.32.1).
> 
> by patching `void Pasteboard::read(PasteboardFileReader& reader,
> std::optional<size_t>)`,  and call
> `WebCorePasteboardFileReader::readBuffer`, we can make
> `DataTransferItem::getAsFile()` get the image file data.

Thanks. I applied the patch you linked (https://bugs.webkit.org/attachment.cgi?id=433288&action=diff) against webkitgtk 2.32.2 and the compiled it and then compiled Epiphany 40.2 against that webkitgtk build but pasting image data from the clipboard still does not work for me.

Is this https://bugs.webkit.org/attachment.cgi?id=433288&action=diff enough or should I apply something else too? Waht do you mean with this: "by patching `void Pasteboard::read(PasteboardFileReader& reader, std::optional<size_t>)`,  and call `WebCorePasteboardFileReader::readBuffer`, we can make `DataTransferItem::getAsFile()` get the image file data."?

[荒野無燈]

(In reply to Miikka from comment #22)
> (In reply to 荒野無燈 from comment #18)
> > Created attachment 433288 [details]
> > fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> > no files, this fix makes ctrl + v paste image under GTK  works
> > 
> > in Source/WebCore/dom/DataTransfer.cpp 
> > DataTransfer::filesFromPasteboardAndItemList
> > 
> > ```cpp
> >  WebCorePasteboardFileReader reader(context);
> >  m_pasteboard->read(reader);
> >  files = WTFMove(reader.files);
> > ```
> > 
> > it only calls `void Pasteboard::read(PasteboardFileReader& reader,
> > std::optional<size_t>)`, 
> > 
> > and `Pasteboard::read(PasteboardWebContentReader& reader,
> > WebContentReadingPolicy policy, std::optional<size_t>)` got no chances, so
> > the image data dit not got read.
> > 
> > this patch is a workaround to fixup the problem (tested under webkitgtk
> > 2.32.0 and 2.32.1).
> > 
> > by patching `void Pasteboard::read(PasteboardFileReader& reader,
> > std::optional<size_t>)`,  and call
> > `WebCorePasteboardFileReader::readBuffer`, we can make
> > `DataTransferItem::getAsFile()` get the image file data.
> 
> Thanks. I applied the patch you linked
> (https://bugs.webkit.org/attachment.cgi?id=433288&action=diff) against
> webkitgtk 2.32.2 and the compiled it and then compiled Epiphany 40.2 against
> that webkitgtk build but pasting image data from the clipboard still does
> not work for me.
> 
> Is this https://bugs.webkit.org/attachment.cgi?id=433288&action=diff enough
> or should I apply something else too? Waht do you mean with this: "by
> patching `void Pasteboard::read(PasteboardFileReader& reader,
> std::optional<size_t>)`,  and call
> `WebCorePasteboardFileReader::readBuffer`, we can make
> `DataTransferItem::getAsFile()` get the image file data."?

just use this patch will works fine.

please note that this patch does not related to GNOME Web browser Epiphany.

I compared from webkitgtk-2.32.0 to webkitgtk-2.32.2, the `Source/WebCore/platform/gtk/PasteboardGtk.cpp` file are all the same, 
I tested the patch under webkitgtk 2.32.0 and 2.32.1, so I do think it should works for 2.32.2

by the way, I use this page (thanks Sonny Piers) to test: https://codepen.io/tmrDevelops/pen/YxGQaW

and I also tested it under Epiphany 40.1 on Ubuntu 21.04 (https://i.ibb.co/FXhhSYZ/Epiphany-40-1-on-Ubuntu-21-042021-07-12-21-07.png), and Epiphany Technology Preview 41.alpha-4-g70730e4dd+ does not work. but I recommend you not to verify the patch via Epiphany browser, this will introduce more variable.

Using a simple code (in your favorite lang) to create a webview, then loading the page: https://codepen.io/tmrDevelops/pen/YxGQaW and then see if it works.

In my situation, I use the WebKitGtk lib via https://github.com/gtk-rs/webkit2gtk-rs

hope this will help you.

[Michael Catanzaro]

*** Bug 211519 has been marked as a duplicate of this bug. ***

[Miikka]

(In reply to 荒野無燈 from comment #23)
> (In reply to Miikka from comment #22)
> > (In reply to 荒野無燈 from comment #18)
> > > Created attachment 433288 [details]
> > > fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> > > no files, this fix makes ctrl + v paste image under GTK  works
> > > 
> > > in Source/WebCore/dom/DataTransfer.cpp 
> > > DataTransfer::filesFromPasteboardAndItemList
> > > 
> > > ```cpp
> > >  WebCorePasteboardFileReader reader(context);
> > >  m_pasteboard->read(reader);
> > >  files = WTFMove(reader.files);
> > > ```
> > > 
> > > it only calls `void Pasteboard::read(PasteboardFileReader& reader,
> > > std::optional<size_t>)`, 
> > > 
> > > and `Pasteboard::read(PasteboardWebContentReader& reader,
> > > WebContentReadingPolicy policy, std::optional<size_t>)` got no chances, so
> > > the image data dit not got read.
> > > 
> > > this patch is a workaround to fixup the problem (tested under webkitgtk
> > > 2.32.0 and 2.32.1).
> > > 
> > > by patching `void Pasteboard::read(PasteboardFileReader& reader,
> > > std::optional<size_t>)`,  and call
> > > `WebCorePasteboardFileReader::readBuffer`, we can make
> > > `DataTransferItem::getAsFile()` get the image file data.
> > 
> > Thanks. I applied the patch you linked
> > (https://bugs.webkit.org/attachment.cgi?id=433288&action=diff) against
> > webkitgtk 2.32.2 and the compiled it and then compiled Epiphany 40.2 against
> > that webkitgtk build but pasting image data from the clipboard still does
> > not work for me.
> > 
> > Is this https://bugs.webkit.org/attachment.cgi?id=433288&action=diff enough
> > or should I apply something else too? Waht do you mean with this: "by
> > patching `void Pasteboard::read(PasteboardFileReader& reader,
> > std::optional<size_t>)`,  and call
> > `WebCorePasteboardFileReader::readBuffer`, we can make
> > `DataTransferItem::getAsFile()` get the image file data."?
> 
> just use this patch will works fine.
> 
> please note that this patch does not related to GNOME Web browser Epiphany.
> 
> I compared from webkitgtk-2.32.0 to webkitgtk-2.32.2, the
> `Source/WebCore/platform/gtk/PasteboardGtk.cpp` file are all the same, 
> I tested the patch under webkitgtk 2.32.0 and 2.32.1, so I do think it
> should works for 2.32.2
> 
> by the way, I use this page (thanks Sonny Piers) to test:
> https://codepen.io/tmrDevelops/pen/YxGQaW
> 
> and I also tested it under Epiphany 40.1 on Ubuntu 21.04
> (https://i.ibb.co/FXhhSYZ/Epiphany-40-1-on-Ubuntu-21-042021-07-12-21-07.png),
> and Epiphany Technology Preview 41.alpha-4-g70730e4dd+ does not work. but I
> recommend you not to verify the patch via Epiphany browser, this will
> introduce more variable.
> 
> Using a simple code (in your favorite lang) to create a webview, then
> loading the page: https://codepen.io/tmrDevelops/pen/YxGQaW and then see if
> it works.
> 
> In my situation, I use the WebKitGtk lib via
> https://github.com/gtk-rs/webkit2gtk-rs
> 
> hope this will help you.

Yes, I tested this by using the WebKit4GTK MiniBrowser now (/usr/lib/x86_64-linux-gnu/webkit2gtk-4.0/MiniBrowser) and the image data pasting works there (both in normal and in editor mode: --editor-mode). But in other WebKit based browsers, like Epiphany it still does not work :(

[荒野無燈]

(In reply to Miikka from comment #25)
> (In reply to 荒野無燈 from comment #23)
> > (In reply to Miikka from comment #22)
> 
> Yes, I tested this by using the WebKit4GTK MiniBrowser now
> (/usr/lib/x86_64-linux-gnu/webkit2gtk-4.0/MiniBrowser) and the image data
> pasting works there (both in normal and in editor mode: --editor-mode). But
> in other WebKit based browsers, like Epiphany it still does not work :(

I have binary build for ArchLinux ( if you are using),

I use it for my everyday work. it is here: https://github.com/ttys3/archlinux-webkit2gtk/releases/tag/v2.34.2-1

[Miikka]

(In reply to 荒野無燈 from comment #26)
> I use it for my everyday work. it is here:
> https://github.com/ttys3/archlinux-webkit2gtk/releases/tag/v2.34.2-1

Thanks. I assume you're using this with webkit2gtk-rs where your patch is working, right? But does it work with Gnome Epiphany or other modern/GUI WebKitGTK browser? Have you tested?

[荒野無燈]

(In reply to Miikka from comment #27)
> (In reply to 荒野無燈 from comment #26)
> > I use it for my everyday work. it is here:
> > https://github.com/ttys3/archlinux-webkit2gtk/releases/tag/v2.34.2-1
> 
> Thanks. I assume you're using this with webkit2gtk-rs where your patch is
> working, right? But does it work with Gnome Epiphany or other modern/GUI
> WebKitGTK browser? Have you tested?

No. you are totally wrong. I tested with all, works like a charm.

including Web 41.0 (Powered by WebKitGTK 2.34.2), yes it is what your ref "Gnome Epiphany"

Michael Catanzaro  also confirmed that this patch will work.

The problem is not it work or not work.

It is because this patch is fixing the issue in way which maybe not elegant enough.

[Miikka]

(In reply to 荒野無燈 from comment #28)
> 
> No. you are totally wrong. I tested with all, works like a charm.
> 
> including Web 41.0 (Powered by WebKitGTK 2.34.2), yes it is what your ref
> "Gnome Epiphany"
> 
> Michael Catanzaro  also confirmed that this patch will work.
> 
> The problem is not it work or not work.
> 
> It is because this patch is fixing the issue in way which maybe not elegant
> enough.

Yes, you were right! I had some time now to test and verify this by compiling the WebKitGTK v2.34.3 (with your patches included: https://github.com/ttys3/archlinux-webkit2gtk/tree/main/gtk4.0) and Gnome Web (Epiphany web browser) v3.38.4 on my Debian 11 box and the pasting of images (like PNGs and JPEGs) works now on Gnome Web browser as well.

[Michael Catanzaro]

*** Bug 255505 has been marked as a duplicate of this bug. ***