Bug 179411 – getUserMedia echoCancellation constraint has no affect

• Component: WebRTC
• Status: RESOLVED FIXED

[Kerry Davis]

Created attachment 326306 [details]
AirBridge Chrome extension for above demo...drag into chrome://extensions on a chrome browser running on a mac

Objective is to turn off all filters on microphone audio sample input, however, the getUserMedia call seems to ignore removing what appear to be default filters affecting a low pass (anything above around 12 KHz and maybe lower is greatly attenuated)

Desktop Chrome seems to not have these filters turned on but turning them off works in both desktop versions of Firefox and Opera on my MacBook Air running Safari 11. However, removing the filters to get raw audio data for both mobile and desktop Safari 11 does not seem to have any affect to remove the filters. The result is a low pass of frequencies below 12K but they still seem to be negatively affected even in the lower ranges.

Example:

 navigator.mediaDevices.getUserMedia({audio: { 
                                                echoCancellation: false,
                                                noiseSuppression: false,
                                                autoGainControl: false,
                                                mozNoiseSuppression: false,
                                                mozAutoGainControl: false
                                                }, 
                                            video: false})
    .then(handleABSuccess);

results in a raw input audio microphone sample on the same MacBook Air for FireFox, Opera and Chrome browsers BUT NOT Safari 11 (desktop or mobile on iPhone running IOS/Safari 11)

This is critical because there Safari 11 also does not support FFT bins above 2048, WebRTC on Save to Home Screen mobile apps (at all) or ability to change sample rate (from 48 KHz down to 44.1 KHz in my case). This means for the case where hardware is static at sampling the microphone audio at 48KHz I then need to oversample the frequency domain at 65K bins, which means I need to send raw audio samples into my own third party FFT analyzer. Once again, this works now for Firefox, Opera and Chrome on the same hardware it does not work on Safari 11.

If any of this is in question, you can open the page at https://www.airbridgelabs.com/s/0/app.html?sd=158 on all of the above named browsers, including Safari 11 (on a mac of course), , click on the star looking icon near the bottom right of that page which should turn green indicating the microphone is streaming input to the code, then browse to my test pages at is.gd/abdemo in a separate window (I use Chrome but I don't think it matters) and click on some of the images in the coupon or youtube demo. Clicking these images will send a high frequency digitally modulated pulse which will be picked up in all but the safari 11 browser and display the images respectively in the other browsers.

Likewise, you can also install my Chrome browser extension and google images (using any search key), set the transmit band in the drop down of the Chrome extension. lowest band is clearly audible and is the only one that works in Safari 11 but all bands work in the other browsers running the same code (once again I mean the code on https://www.airbridgelabs.com/s/0/app.html?sd=158 when the green star is activated the audio stream input)

[youenn fablet]

getUserMedia with echoCancellation false should work on both MacOs and iOS.
Can you verify whether this is working in latest iOS/MacOS Safari versions?

[Kerry Davis]

Responding to : "getUserMedia with echoCancellation false should work on both MacOs and iOS. Can you verify whether this is working in latest iOS/MacOS Safari versions?"

Here is what I have found.

First, yes there is still a problem in that the current code I am using (which includes echoCancellation false) does not appear to turn echo cancellation off. MacOS for sure but since I really have nothing to compare it against on iOS (since everything runs on top of the iOS Safari webkit) I have nothing to compare it with but what results I have are at the end of this report.

Second, I have worked on/with many telecom standards bodies. It is my opinion and experience that allocation of a streaming resource of any kind should by default result in a filterless (aka raw) stream and never default with filters to the stream on. I have always assumed that this is because on bring up (another area i have quite a bit of embedded software experience in) you always want the hardware to come up in a known state for the software (even if, by some weird chance, the hardware actually enables the filter as it is a HW assisted filter...in which case the lowest level firmware should immediate disable all filters). I don't sit or have any real influence on the webRTC standard but I believe presenting a stream in a filterless initial/default state has real benefits in both software and HW debugging and operation (i.e. unforeseen cold resets)

Third, with respect to the second point above, evidently someone else (actually everybody else) either agrees or accidently agrees with allocating a audio stream raw by default. This is how Chrome, Opera, and Firefox are working now (as of nov 2018) on my macbook running MacOS.

How to reproduce: 

First: you need to get my airbridge communication extension for chrome from the chrome extension store. https://chrome.google.com/webstore/detail/airbridge/mblkfpphiadhncblckopllohojmjdoee

Second: AirBridge digital pulses will operate across 5 separate audio bands roughly placed at 3,8,14,18,20 KHz. Select the HIGH, or MEDIUM transmit frequency band by clicking on the extension icon (which should be green when the extension is active). Then do a google IMAGE search on any topic (cute puppies is my goto favorite). When you mouse over a puppy image in the search results you should now see my airbridge logo pop up and when you click the logo you should hear a click which is the default sound that provides feedback to the user that a pulse was sent (useful for the higher bands and us old people)


Now open up 4 instances of a test page I created containing my airbridge receiver code (https://www.airbridgelabs.com/s/sbc2.html?sd=5 .. this page should come up with the AirBridge receiver active...green star flashing on lower screen) in desktop browsers (Safari, Opera, Firefox and Chrome) all running on the same MacBook Air and the same code without conditionals.

The following code is used for getUserMedia()

navigator.mediaDevices.getUserMedia({audio: { 
            echoCancellation: false,
            noiseSuppression: false,
            autoGainControl: false,
            googEchoCancellation: false,
            googAutoGainControl: false,
            googExperimentalAutoGainControl: false,
            googNoiseSuppression: false,
            googExperimentalNoiseSuppression: false,
            googHighpassFilter: false,
            googTypingNoiseDetection: false,
            googBeamforming: false,
            googArrayGeometry: false,
            googAudioMirroring: false,
            googAudioMirroring: false,
            googNoiseReduction: false,
            mozNoiseSuppression: false,
            mozAutoGainControl: false
            }, 
        video: false})
    .then(handleABSuccess)
    .catch(function(err) {
       
        testlog (err.message+": Access to the microphone is necessary to receive AirBridge command pulses.\n\nHowever, microphone access was denied by the user (or the operating system) upon installation.\n\nTry resetting browser settings and redownloading this app or extension.");
        return (1);
    });

Now click on an image from the google search and look at each of the pages you just opened. All of them EXCEPT for Safari should now be displaying the image in a popup window. If not then tweak the volume up or down a bit but for me it usually works even down at one volume bar and all the way up to max volume (make sure you hear the confirmation beep/click and you can change to a couple other or no confirmation sound in the extension settings). If you don't hear the confirmation then refresh the search page and click a different image.

Why do I think this is a problem with audio filters not being disabled? Simply because if you repeat the same experiment from above except you change the aibridge transmit frequency to either 8K or LOW you will then see that the image starts also appearing in the Safari browser. This is because many audio filters, including and maybe especially echo cancellation and certainly compression algorithms (but compression is not the case here) will attenuate the upper frequencies but leave the lower ones (and my FFT analyser node) almost untouched.

In an additional test I actually removed the filters in my code and ran the above test again. EX.

navigator.mediaDevices.getUserMedia({audio: { 
        
            
            }, 
        video: false})
    .then(handleABSuccess)
    .catch(function(err) {
       
        testlog (err.message+": Access to the microphone is necessary to receive AirBridge command pulses.\n\nHowever, microphone access was denied by the user (or the operating system) upon installation.\n\nTry resetting browser settings and redownloading this app or extension.");
        return (1);
    });

What I found was that desktop Chrome && Opera both appear to still work at the high frequency bands while Safari only works at the LOW and 8K bands and Firefox only works at the LOW frequency bands (meaning its audio filters are more aggressive than Safari but at least I can turn them OFF). This makes sense if you consider that someone at both Chrome and Opera believe as I do that the default state of the newly created stream should have filters disabled, Firefox comes up with filters enabled (unless we explicitly disable them), and Safari has filters enabled whether you turn them off explicitly or not.

iPhone results.
Running iOS 12.1 on iPhone 7 I can't even fully load my code in Chrome, Firefox, Opera or Edge. I suspect this is because they all told me they don't support webRTC getUserMedia and my code does not handle that well so it gets stuck. As for Safari, my code does come up and everything appears to work but no pulses, even in the low bands, are received. In the debug I can see that the problem is simple that the analyser data is all 0's. This is exactly the same code that is working in the desktop Safari and I am getting no errors so I have no idea why it is not working in mobile Safari yet works in desktop version???

And interestingly, when I originally wrote this bug, I remember that mobile chrome on Android did NOT default to filters disabled as is the case in desktop Chrome. So much for object inheritance and cross compilers lol.

[Kerry Davis]

Safari PWA version of this code still indicates that navigator.mediaDevices and thus navigator.mediaDevices.getUserMedia are both null, Oddly enough creating and analyser node seems to work (albeit useless without getusermedia

if (!noanalyser){
        try{
            //analyser = new AnalyserNode(acontext);
            analyser = acontext.createAnalyser();
            try{
                analyser.fftSize = 32768; // Maximum FFT size is 32768 by the standard; Desktop Safari handles this but apparently not mobile
            }catch(err){
                analyser = null;
                testlog("can't create analyser.fftSize == 32768");
            }
        }catch(err){
            testlog("can't create analyser");
        }
    }

[Kerry Davis]

As opposed to using the chrome extension for testing using different AirBridge Pulse bands, it should also be possible to more simply use this test page https://is.gd/abbtest I just created along with any airbridge enabled web app (i.e. https://www.airbridgelabs.com/s/sbc2.html?sd=5).

This should work on any browser and any desktop HW (and maybe mobile as well but that is a bit more risky at this untested point) for the transport side and the native apple store mobile app or airbridge enabled web app (like the one above) for the receive side testing.

[Kerry Davis]

This comment is just specific to iOS and iPhone browsers Safari, Firefox, Opera, Chrome and Edge:

After a bit more testing I am able to confirm that mobile Safari is the only browser that even defines getUserMedia on iOS12 in an iPhone. Firefox, Opera, Chrome and Edge all allow allocation of analyser nodes up to 16K FFT bin sizes yet Safari is the only one where getUserMedia is not null.

This allows my code to work inside of a Safari browser to receive and digitally demodulate my audio pulses in my two frequency bands below 10KHz. But above that does not work and as stated in my prior comments relating to desktop Safari, my guess (and more of a guess here on mobile than it was on desktop simply because I have no other browsers to compare it with) is that the problem is simply that the mic audio is being filtered. Keep in mind that my apple store native app is able to do this which removes (or should) any concern that this is a HW capability issue.

But this also confuses me as to why all of the other browser vendors running in iOS have told me they can not support getUserMedia in their iPhone browsers until it webkit (actually they said "until apple allows us" but I assume that is what they meant)

So my immediate question is whether they currently have access to webkit provided getUserMedia as mobile Safari does? Because, at the very least, I should be able to use the lower two bands in all of the other browsers as I have now done in mobile Safari for iOS 12. Is this a correct assumption?

[Ryan Clough]

Hello, I'd like to confirm the bug reported in this ticket. I've been working on detecting high frequency content in JS, and found this ticket trying to find confirmation that there's no way to remove the low pass filter on Safari/webkit WebAudio mic inputs.

I first noticed it heuristically when live graphing an FFT of my mic input, that when I play white noise, theres a low pass that cuts off at about 12khz. No other browser with constraints "noiseSupression" and "echoCancellation" set to false. 

For example my code looks as such:

        navigator.mediaDevices.getUserMedia({
          "audio": {
            echoCancellation: false,
            autoGainControl: false,
            noiseSuppression: false
          },
        }).then(...).catch(...)

Upon further investigation, you can verify that webkit simply does not support the constraints by running:

console.log(navigator.mediaDevices.getSupportedConstraints());

Which will output something like this:

aspectRatio: true
deviceId: true
echoCancellation: false
facingMode: true
frameRate: true
groupId: false
height: true
sampleRate: false
sampleSize: false
volume: true
width: true

As you can see, noiseSupression and echoCancellation are simply not in the list.

Obviously I'm writing because I really wish this would be added to bring webkit to parity with other browsers

[youenn fablet]

Applying echoCancellation does not work with getUserMedia.
A workaround is to applyConstraints on the track:

const stream = await navigator.mediaDevices.getUserMedia({ audio : true });
stream.getAudioTracks()[0].applyConstraints({echoCancellation: false});

[youenn fablet]

Created attachment 383726 [details]
Patch

[youenn fablet]

https://jsfiddle.net/bnowkth4/ disables echo cancellation on both MacOS and iOS.

[youenn fablet]

Created attachment 383926 [details]
Patch

[Eric Carlson]

Comment on attachment 383926 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=383926&action=review

> Source/WebCore/platform/mediastream/mac/MockAudioSharedUnit.mm:236
> +    if (!enableEchoCancellation())
> +        addHum(NoiseVolume, NoiseFrequency, rate, 0, m_bipBopBuffer.data(), sampleCount);

Great idea!

> LayoutTests/webrtc/routines.js:123
> +           if (!results.heardNoise)
> +                results.heardNoise = hasFrequency(3000);
> +
> +            if (results.heardHum && results.heardBip && results.heardBop && results.heardNoise)
>                  done();

Nit: the indentation is wrong here.

[youenn fablet]

Created attachment 383935 [details]
Patch for landing

[youenn fablet]

Created attachment 383938 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 383938 [details]
Patch

Clearing flags on attachment: 383938

Committed r252681: <https://trac.webkit.org/changeset/252681>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/57347253>

[Kerry Davis]

(In reply to youenn fablet from comment #1)
> getUserMedia with echoCancellation false should work on both MacOs and iOS.
> Can you verify whether this is working in latest iOS/MacOS Safari versions?

In my tests it is still not working in iOS/MacOS Safari. However, I just today got the  MacOS tech preview and it seems to work for frequencies below 9KHz and maybe a bit higher but certainly not above 13KHz.

I am still testing with my Chrome extension (sender/receiver) of digital modulated pulses and a PWA (capable of receiving only) which works in both Chrome and Firefox but not at all in mobile or desktop Safari and only below 9KHz in the tech preview (running the PWA enabled receiver)

I need to run more tests to try to isolate what exactly is the reason but I suspect it is the same as when I wrote this original bug report where one or more filters are on by default and I am not able to turn them off with any constraints.

Have also not confirmed whether Opera is still working or not as it was before.

[Ryan Clough]

I suppose it's worth noting that while this bug report title specifically calls out Echo Cancellation, its not clear if the low pass filter is being enabled due to echoCancellation or noiseSuppression or some other constraint unnamed or otherwise. My worry is that it will get closed simply for enabling echo cancellation and not implementing the other constraints or tackling the underlying issue which is the inability to access high frequency content from a users mic through webRTC 

BTW I can confirm that opera implements all the constraints, but thats because they are a glorified chromium wrapper.

[youenn fablet]

When setting echoCancellation to false, we both disable AGC and echo cancellation.
Please file a follow-up bug if you would like to have a specific autoGainControl handling.
Please also file a follow-up bug if you see some filtering you would like to disable as well.

[Ryan Clough]

My point is that despite the (potentially) poor titling of the issue, the crux of the issue was raised in the original description:

"Objective is to turn off all filters on microphone audio sample input, however, the getUserMedia call seems to ignore removing what appear to be default filters affecting a low pass (anything above around 12 KHz and maybe lower is greatly attenuated)"

with seemingly the assumption that what was causing the filtering was echo cancellation. Seems that is not the case.

I suppose we can open another ticket with a better title. Just disheartening that the actual concern raised in the original issue was not addressed and it took 2 years just to get there.

[Kerry Davis]

I certainly admit it wasn't the best title which is why I clarified with the objective statement of turning all filters OFF to get a raw audio stream. I am used to the default state of any stream being without filters (background is telecom). There was actually no way of knowing exactly which filter (and it turns out maybe more than one) is causing the problem.

In my humble opinion, the default state of the audio (or video stream) should be unfiltered but none of the browser did that. So my second hope would be a single constraint to turn raw (unfiltered audio) ON or OFF. I understand that is a bit confusing in light of the fact that by default some browsers turn filters ON.

As a last resort, I would merely request that Safari work in the same way as Chrome. which is in fact working for both android mobile and MacOS desktop Chrome. But as I remember, Desktop Chrome by default DID actually have filters OFF (but don't hold me to that now)

I can certainly open another bug but I think the detail of this one still describes the problem. Which simply put..we need a way to get a raw audio stream from the desktop or mobile mic. If for nothing else, this should be obvious for the reason that not being able to do that decreases or even eliminates the value of having an FFT AnalyserNode at all since any sort of audio filter distorts audio FFT results.

[youenn fablet]

The additional issue probably needs more investigation.
It might be that some filters are still active.
Another possibility is that our WebAudio implementation has some issues.
Your help is of course welcome.

[youenn fablet]

Phil, thibauld, I added the ability to update mock audio sources if echoCancellation is on.
If you want to pass the newly added test, you would need to update GStreamer audio mock source accordingly.

[youenn fablet]

I am closing this bug.
I filed bug 204444 to keep track of adding support for the autoGainControl constraint.
Please open a new bug if the current getUserMedia-without-echoCancellation + web audio support does not meet some of your requirements.

[Kerry Davis]

(In reply to youenn fablet from comment #24)
> I am closing this bug.
> I filed bug 204444 to keep track of adding support for the autoGainControl
> constraint.
> Please open a new bug if the current getUserMedia-without-echoCancellation +
> web audio support does not meet some of your requirements.

Hi Youenn,

I will file another bug but as you can probably tell, it seems to be a moving target. I didn't then and even now have no way of knowing which audio filter is turned on and causing the low pass filter issue. As you can see from the original filing, I am attempting to turn off all filters (I know of) and that includes echo and AGC.

Besides the potential for the problem being a rogue filter, when this bug was originally created against mobile Safari 11, only 2048 bins were allowed for an Analyzer node and I needed 32K bins (max allowed today). So a year ago, I must have been passing the 42100 or 48000 audio samples thru my own FFT (which is pretty good but no match for the HW assisted analyser node.

Point being that for all I know it could be the analyser not. However looking at the frequency domain output from the Safari Technology Preview, this is unlikely because I can see the 16K analyzer node FFT output go lower and lower until they flat line to all 0's at just above 9 KHz (which indicates the raw audio is filtered or compressed prior to the sample (which again seems unlikely for a live mic audio stream).

Since I am already setting AGC to false, I can't see how following that bug can help me.

So, when I create the new bug I have no choice but to be very vague with the title and pretty much duplicate the description of this bug. Which is mainly that I need a raw audio microphone audio stream that can be fed into an analyzer node of fftsize = 32768 where I actually get frequency domain all the way up to the full microphone response (based on sample rate of either 44.1K or 48K samples per second). And optimally, I need it to work with the exact same code that already works in the other major browsers (namely the code below).

And it should be extremely simple for you guys to test. Just play any any CD quality music into the mic and see if you get non zero output where the time domain sample rate is 44.1 or 48K and the FFT size is 32768.

And if that works turn the music off and play a set of tones in some interval starting at 16KHz up to at least 21Khz and look for those spikes in the frequency domain output of the analyser node.


....
try{
            analyser = acontext.createAnalyser();
            try{
                analyser.fftSize = 32768; // Maximum FFT size is 32768 by the standard; Desktop Safari handles this but apparently not mobile
            }catch(err){
                analyser = null;
                acontext = null;
                
                //alert("can't create analyser.fftSize == 32768");
                
                testlog("can't create analyser.fftSize == 32768");
            }
        }catch(err){
            //alert("can't create analyser");
            testlog("can't create analyser");
        }
.....

navigator.mediaDevices.getUserMedia({audio: { 
                echoCancellation: false,
                noiseSuppression: false,
                autoGainControl: false,
                googEchoCancellation: false,
                googAutoGainControl: false,
                googExperimentalAutoGainControl: false,
                googNoiseSuppression: false,
                googExperimentalNoiseSuppression: false,
                googHighpassFilter: false,
                googTypingNoiseDetection: false,
                googBeamforming: false,
                googArrayGeometry: false,
                googAudioMirroring: false,
                googNoiseReduction: false,
                mozNoiseSuppression: false,
                mozAutoGainControl: false
                }, 
            video: false})