Bug 218519 - Can't paste image data (like image/png) from the clipboard in WebKit browsers
Summary: Can't paste image data (like image/png) from the clipboard in WebKit browsers
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Platform (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-11-03 09:06 PST by Miikka
Modified: 2021-07-12 07:39 PDT (History)
7 users (show)

See Also:


Attachments
fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get no files, this fix makes ctrl + v paste image under GTK works (1.57 KB, patch)
2021-07-11 06:33 PDT, 荒野無燈
mcatanzaro: review-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Miikka 2020-11-03 09:06:39 PST
I'm unable to paste any images from my clipboard into browser windows using WebKitGtk (like Epiphany or WebKitGtk MiniBrowser).

I've verified that the data certainly is in the clipboard because following commands:
xclip -selection clipboard -t image/png -o > image.png
xclip -selection clipboard -t image/bmp -o > image.bmp
outputs proper bitmap or PNG files. And I'm also able to paste the data in Gimp, Firefox, LibreOffice and Chromium. Only the WebKitGtk based browser does not work properly so I assume the problem is with my WebKitGtk. I compiled webkitgtk-2.30.2 and Epiphany from git (2020-10-29) manually but the problem still exists in Epiphany and in the MiniBrowser.

Could there be some library I'm missing or something?

I found one bug that could relate: https://bugs.webkit.org/show_bug.cgi?id=211519 but in there there is broken image area when the data has been pasted. With my problem the browser does nothing when I hit Ctrl+V (like in chat/email window or in imgur.com).

I compiled WebKitGtk with the following arguments:
cmake .. -DCMAKE_BUILD_TYPE=Release -DPORT=GTK -DENABLE_MINIBROWSER=ON -DCMAKE_EXE_LINKER_FLAGS="-Wl,--as-needed -Wl,-z,now -pthread" -DCMAKE_MODULE_LINKER_FLAGS="-Wl,--as-needed -Wl,-z,now -pthread" -DCMAKE_SHARED_LINKER_FLAGS="-Wl,--as-needed -Wl,-z,now -pthread" -DUSE_WOFF2=false -DENABLE_MEDIA_SOURCE=OFF -DPYTHON_EXECUTABLE=/usr/bin/python3:
-- Enabled features:
--  ENABLE_ACCELERATED_2D_CANVAS .................. OFF
--  ENABLE_BUBBLEWRAP_SANDBOX                       ON
--  ENABLE_DRAG_SUPPORT ........................... ON
--  ENABLE_GLES2                                    OFF
--  ENABLE_GRAPHICS_CONTEXT_GL .................... ON
--  ENABLE_GTKDOC                                   OFF
--  ENABLE_INTROSPECTION .......................... ON
--  ENABLE_MEDIA_SOURCE                             OFF
--  ENABLE_MINIBROWSER ............................ ON
--  ENABLE_QUARTZ_TARGET                            OFF
--  ENABLE_SHAREABLE_RESOURCE ..................... ON
--  ENABLE_SPELLCHECK                               ON
--  ENABLE_TOUCH_EVENTS ........................... ON
--  ENABLE_VIDEO                                    ON
--  ENABLE_WAYLAND_TARGET ......................... ON
--  ENABLE_WEBDRIVER                                ON
--  ENABLE_WEB_AUDIO .............................. ON
--  ENABLE_WEB_CRYPTO                               ON
--  ENABLE_X11_TARGET ............................. ON
--  USE_GTK4                                        OFF
--  USE_LIBHYPHEN ................................. ON
--  USE_LIBNOTIFY                                   ON
--  USE_LIBSECRET ................................. ON
--  USE_OPENJPEG                                    ON
--  USE_SYSTEMD ................................... ON
--  USE_WOFF2                                       false
--  USE_WPE_RENDERER .............................. ON
-- Configuring done
-- Generating done

Distro: openSUSE Leap 15.2
DE: KDE/Plasma 5.18.5
Comment 1 Carlos Garcia Campos 2020-11-06 01:07:17 PST
Could you try pasting to MiniBrowser started with -e (editor mode). I guess this is on xorg?
Comment 2 Miikka 2020-11-17 09:37:18 PST
Yes, it's Xorg I'm using. In the editor mode the pasting of image data from the clipboard works!

I also created a separate bug report to Epiphany but it wasn't correct place to submit that: https://gitlab.gnome.org/GNOME/epiphany/-/issues/1388

I've also noticed that pasting works on Evolution email client when I'm writing/composing email. I can embed images from clipboard directly into email.

Way to reproduce: Select and copy some image data from Gimp into clipboard for example, in Epiphany go to imgur.com and create new post, hit CTRL+V: nothing happens. If I do this in Chromium or Firefox the browser delivers the image data to imgur and the new post gets created.
Comment 3 Miikka 2020-11-30 04:21:42 PST
Is there anything I should try? I've been able to reproduce this on openSUSE 15.2 Leap and on Debian testing (Bullseye).
Comment 4 Miikka 2020-12-08 10:37:38 PST
Bump!
Comment 5 Sonny Piers 2021-01-14 08:04:08 PST
I ran some tests

Everything works fine on MiniBrowser -e (both Wayland and Xorg)

However, on non-editable context it is broken on both Wayland and Xorg.

The paste event https://w3c.github.io/clipboard-apis/#clipboard-event-paste is emitted with a DataTransferItemList length of 0.

I used https://codepen.io/tmrDevelops/pen/YxGQaW to reproduce.

To clarify, file:// pasting works fine - only pasting [Image] data is broken.
Comment 6 Sonny Piers 2021-01-14 08:15:26 PST
Actually a much better test case is to run 

window.addEventListener('paste', function(e){
  console.log(e.clipboardData.items.length)
});

in the JavaScript console and then paste data.

Both on editable and non-editable contexts it logs 0 when pasting [Image].
Comment 7 Miikka 2021-01-14 09:20:28 PST
Hi Sonny! Thanks, I'm glad to hear that someone else was able to reproduce this. So I can be sure that it does not happen only with my setup.
Comment 8 Carlos Garcia Campos 2021-01-21 05:33:37 PST
I can reproduce it now. I don't think this is specific to the GTK port. DataTransfer::items() only returns items that are considered to be safe for DOM, see:

bool Pasteboard::isSafeTypeForDOMToReadAndWrite(const String& type)
{
    return type == "text/plain" || type == "text/html" || type == "text/uri-list";
}

So, images are not allowed. I don't know if that's WebKit specific or part of a spec, though.
Comment 9 Miikka 2021-01-21 22:11:55 PST
Thanks Carlos.

I tried to modify that particular code in Source/WebCore/platform/Pasteboard.cpp:
bool Pasteboard::isSafeTypeForDOMToReadAndWrite(const String& type)
{   
    return type == "text/plain" || type == "text/html" || type == "text/uri-list";
}
to:
bool Pasteboard::isSafeTypeForDOMToReadAndWrite(const String& type)
{   
    return type == "text/plain" || type == "text/html" || type == "text/uri-list" || type == "image/apng" || type == "image/avif" || type == "image/gif" || type == "image/jpeg" || type == "image/png" || type == "image/svg+xml" || type == "image/webp";
}

And built the WebKitGtk but that didn't fix this issue at least (still unable to paste any image data from the clipboard into WebKit MiniBrowser or Epiphany).

Could this issue be related to these changes?:
https://webkit.org/blog/8170/clipboard-api-improvements/
https://webkit.org/blog/10855/async-clipboard-api/

What do you think, should I move this issue to some other Component or create completely new issue and ref to this?
Comment 10 Miikka 2021-01-21 22:18:14 PST
Actually I found also something interesting from the file: Source/WebCore/Modules/async-clipboard/Clipboard.cpp:

    if (type == "text/uri-list"_s) {
        String title;
        resultAsString = activePasteboard().readURL(itemIndex, title).string();
    }

    if (type == "text/plain"_s) {
        PasteboardPlainText plainTextReader;
        activePasteboard().read(plainTextReader, PlainTextURLReadingPolicy::IgnoreURL, itemIndex);
        resultAsString = WTFMove(plainTextReader.text);
    }

    if (type == "text/html"_s) {
        WebContentMarkupReader markupReader { *frame };
        activePasteboard().read(markupReader, WebContentReadingPolicy::OnlyRichTextTypes, itemIndex);
        resultAsString = WTFMove(markupReader.markup);
    }   

    // FIXME: Support reading custom data.
    if (updateSessionValidity() == SessionIsValid::No || resultAsString.isNull()) {
        promise->reject(NotAllowedError);
        return;
    }

Could that be related to this issue?
Comment 11 Miikka 2021-01-21 22:43:32 PST
And this:

File: Source/WebCore/platform/gtk/PasteboardGtk.cpp:
void Pasteboard::read(PasteboardPlainText& text, PlainTextURLReadingPolicy, Optional<size_t>):
    static const ASCIILiteral imageTypes[] = { "image/png"_s, "image/jpeg"_s, "image/gif"_s, "image/bmp"_s, "image/vnd.microsoft.icon"_s, "image/x-icon"_s };
    for (const auto& imageType : imageTypes) {
        if (types.contains(imageType)) {
            auto buffer = platformStrategies()->pasteboardStrategy()->readBufferFromClipboard(m_name, imageType);
            if (!buffer->isEmpty() && reader.readImage(buffer.releaseNonNull(), imageType))
                return;
        }
    }
Comment 12 Carlos Garcia Campos 2021-01-22 00:41:05 PST
What is probably failing after allowing images is getAsFile(), I don't know why that should work for images, but it's returning null I'm afraid.
Comment 13 Miikka 2021-02-10 00:20:03 PST
Should I move this bug report from WebKitGTK Component to something else?
Comment 14 Miikka 2021-06-03 02:05:38 PDT
Hmm, is there any progress with this? I think all of my WebKit based browsers on Linux are lacking the support of pasting images from the clipboard because of this issue?
Comment 15 荒野無燈 2021-07-09 06:25:23 PDT
I already found a way to fix up this, and it works good. Need sometime to figure out how to create a patch, seems that submit a patch to webkit is a little bit difficult to me. (I mean, in github I only need one click and submit the Pull Request)
Comment 16 Michael Catanzaro 2021-07-09 07:54:00 PDT
Hi, there are guidelines for contributing patches here: https://webkit.org/contributing-code/

It would also be good to mention what exactly you've discovered....
Comment 17 Miikka 2021-07-11 02:34:49 PDT
(In reply to 荒野無燈 from comment #15)
> I already found a way to fix up this, and it works good. Need sometime to
> figure out how to create a patch, seems that submit a patch to webkit is a
> little bit difficult to me. (I mean, in github I only need one click and
> submit the Pull Request)

Sounds good! Could you please share the patch somewhere so I can test it too?
Comment 18 荒野無燈 2021-07-11 06:33:22 PDT
Created attachment 433288 [details]
fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get no files, this fix makes ctrl + v paste image under GTK  works

in Source/WebCore/dom/DataTransfer.cpp  DataTransfer::filesFromPasteboardAndItemList

```cpp
 WebCorePasteboardFileReader reader(context);
 m_pasteboard->read(reader);
 files = WTFMove(reader.files);
```

it only calls `void Pasteboard::read(PasteboardFileReader& reader, std::optional<size_t>)`, 

and `Pasteboard::read(PasteboardWebContentReader& reader, WebContentReadingPolicy policy, std::optional<size_t>)` got no chances, so the image data dit not got read.

this patch is a workaround to fixup the problem (tested under webkitgtk 2.32.0 and 2.32.1).

by patching `void Pasteboard::read(PasteboardFileReader& reader, std::optional<size_t>)`,  and call `WebCorePasteboardFileReader::readBuffer`, we can make `DataTransferItem::getAsFile()` get the image file data.
Comment 19 荒野無燈 2021-07-11 06:39:53 PDT
I have some working builds if you need quick verify if it works.

-----------------------------------------------------------

libwebkit2gtk-4.0-37_2.32.0 for Ubuntu 20.04.2:

curl -LO https://github.com/ttys3/lark-for-linux/releases/download/v0.9.4/libwebkit2gtk-4.0-37_2.32.0-patched-ubuntu-20.04.2-deb.tar.xz

tar xvf libwebkit2gtk-4.0-37_2.32.0-patched-ubuntu-20.04.2-deb.tar.xz

cd libwebkit2gtk-4.0-37_2.32.0*

sudo apt install -y --allow-downgrades ./*.deb


use https://github.com/ttys3/lark-for-linux/releases/download/v0.9.4/libwebkit2gtk-4.0-37_2.32.0-patched-ubuntu-21.04-gnome40-deb.tar.xz
if you are Ubuntu 21.04 and has GNOME 40 installed.
-------------------------------------------------------------

download and install https://github.com/ttys3/lark-for-linux/releases/download/v0.9.4/webkit2gtk-2.32.1-1-x86_64.pkg.tar.zst if you are a ArchLinux user.
Comment 20 Michael Catanzaro 2021-07-11 07:13:50 PDT
(In reply to 荒野無燈 from comment #18)
> Created attachment 433288 [details]
> fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> no files, this fix makes ctrl + v paste image under GTK  works

I have no doubt that it works, but that can't be the correct fix. readFilePathsFromClipboard should do what it says, not more. I haven't looked closely, but you might be one level too far down from the right place to fix this.
Comment 21 荒野無燈 2021-07-11 07:46:57 PDT
(In reply to Michael Catanzaro from comment #20)
> (In reply to 荒野無燈 from comment #18)
> > Created attachment 433288 [details]
> > fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> > no files, this fix makes ctrl + v paste image under GTK  works
> 
> I have no doubt that it works, but that can't be the correct fix.
> readFilePathsFromClipboard should do what it says, not more. I haven't
> looked closely, but you might be one level too far down from the right place
> to fix this.

Thanks for the quick reply. I think you are right, this is not a correct fix.

I'm not familiar with the WebKit code at all. And I'm just a normal user which wait the correct patch for about 8 months or maybe longer.

I can not do further investigation to finish the right patch.

And hope someone will continue the work.

Thank you all guys.
Comment 22 Miikka 2021-07-12 04:48:05 PDT
(In reply to 荒野無燈 from comment #18)
> Created attachment 433288 [details]
> fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> no files, this fix makes ctrl + v paste image under GTK  works
> 
> in Source/WebCore/dom/DataTransfer.cpp 
> DataTransfer::filesFromPasteboardAndItemList
> 
> ```cpp
>  WebCorePasteboardFileReader reader(context);
>  m_pasteboard->read(reader);
>  files = WTFMove(reader.files);
> ```
> 
> it only calls `void Pasteboard::read(PasteboardFileReader& reader,
> std::optional<size_t>)`, 
> 
> and `Pasteboard::read(PasteboardWebContentReader& reader,
> WebContentReadingPolicy policy, std::optional<size_t>)` got no chances, so
> the image data dit not got read.
> 
> this patch is a workaround to fixup the problem (tested under webkitgtk
> 2.32.0 and 2.32.1).
> 
> by patching `void Pasteboard::read(PasteboardFileReader& reader,
> std::optional<size_t>)`,  and call
> `WebCorePasteboardFileReader::readBuffer`, we can make
> `DataTransferItem::getAsFile()` get the image file data.

Thanks. I applied the patch you linked (https://bugs.webkit.org/attachment.cgi?id=433288&action=diff) against webkitgtk 2.32.2 and the compiled it and then compiled Epiphany 40.2 against that webkitgtk build but pasting image data from the clipboard still does not work for me.

Is this https://bugs.webkit.org/attachment.cgi?id=433288&action=diff enough or should I apply something else too? Waht do you mean with this: "by patching `void Pasteboard::read(PasteboardFileReader& reader, std::optional<size_t>)`,  and call `WebCorePasteboardFileReader::readBuffer`, we can make `DataTransferItem::getAsFile()` get the image file data."?
Comment 23 荒野無燈 2021-07-12 06:30:08 PDT
(In reply to Miikka from comment #22)
> (In reply to 荒野無燈 from comment #18)
> > Created attachment 433288 [details]
> > fix: fallback to readBufferFromClipboard if readFilePathsFromClipboard get
> > no files, this fix makes ctrl + v paste image under GTK  works
> > 
> > in Source/WebCore/dom/DataTransfer.cpp 
> > DataTransfer::filesFromPasteboardAndItemList
> > 
> > ```cpp
> >  WebCorePasteboardFileReader reader(context);
> >  m_pasteboard->read(reader);
> >  files = WTFMove(reader.files);
> > ```
> > 
> > it only calls `void Pasteboard::read(PasteboardFileReader& reader,
> > std::optional<size_t>)`, 
> > 
> > and `Pasteboard::read(PasteboardWebContentReader& reader,
> > WebContentReadingPolicy policy, std::optional<size_t>)` got no chances, so
> > the image data dit not got read.
> > 
> > this patch is a workaround to fixup the problem (tested under webkitgtk
> > 2.32.0 and 2.32.1).
> > 
> > by patching `void Pasteboard::read(PasteboardFileReader& reader,
> > std::optional<size_t>)`,  and call
> > `WebCorePasteboardFileReader::readBuffer`, we can make
> > `DataTransferItem::getAsFile()` get the image file data.
> 
> Thanks. I applied the patch you linked
> (https://bugs.webkit.org/attachment.cgi?id=433288&action=diff) against
> webkitgtk 2.32.2 and the compiled it and then compiled Epiphany 40.2 against
> that webkitgtk build but pasting image data from the clipboard still does
> not work for me.
> 
> Is this https://bugs.webkit.org/attachment.cgi?id=433288&action=diff enough
> or should I apply something else too? Waht do you mean with this: "by
> patching `void Pasteboard::read(PasteboardFileReader& reader,
> std::optional<size_t>)`,  and call
> `WebCorePasteboardFileReader::readBuffer`, we can make
> `DataTransferItem::getAsFile()` get the image file data."?

just use this patch will works fine.

please note that this patch does not related to GNOME Web browser Epiphany.

I compared from webkitgtk-2.32.0 to webkitgtk-2.32.2, the `Source/WebCore/platform/gtk/PasteboardGtk.cpp` file are all the same, 
I tested the patch under webkitgtk 2.32.0 and 2.32.1, so I do think it should works for 2.32.2

by the way, I use this page (thanks Sonny Piers) to test: https://codepen.io/tmrDevelops/pen/YxGQaW

and I also tested it under Epiphany 40.1 on Ubuntu 21.04 (https://i.ibb.co/FXhhSYZ/Epiphany-40-1-on-Ubuntu-21-042021-07-12-21-07.png), and Epiphany Technology Preview 41.alpha-4-g70730e4dd+ does not work. but I recommend you not to verify the patch via Epiphany browser, this will introduce more variable.

Using a simple code (in your favorite lang) to create a webview, then loading the page: https://codepen.io/tmrDevelops/pen/YxGQaW and then see if it works.

In my situation, I use the WebKitGtk lib via https://github.com/gtk-rs/webkit2gtk-rs

hope this will help you.