Bug 190644 – onaudioprocess Occasionally Has Buffers With Zeroes as Data

• Status: RESOLVED
• Resolution: FIXED
• Component: Web Audio

I have a scriptProccessorNode that I use to capture audio off of a user's mic. I do this by getting the input buffer, like so:

audioProcessEvent.inputBuffer.getChannelData(0);

On Safari 12 on MacOS 10.13, occasionally this buffer has empty data, IE all the values in the array are 0, as if the mic has been muted. I have so far only encountered this behavior on MacOS Safari 12. It does not happen every time I run my application either, occurring about %20 of the time on my machine. However, my tester can reproduce it every time he uses the application. It occurs with every headset we have available in the office as well.

Here is a dump from the console:
> ape.inputBuffer.getChannelData(0)
< Float32Array (2048) = $1
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
10 0
11 0
12 0
13 0
14 0
15 0
16 0
17 0
18 0
19 0
20 0
21 0
22 0
23 0
24 0
25 0
26 0
27 0
28 0
29 0
30 0
31 0
32 0
33 0
34 0
35 0
36 0
37 0
38 0
39 0
40 0
41 0
42 0
43 0
44 0
45 0
46 0
47 0
48 0
49 0
50 0
51 0
52 0
53 0
54 0
55 0
56 0
57 0
58 0
59 0
60 0
61 0
62 0
63 0
64 0
65 0
66 0
67 0
68 0
69 0
70 0
71 0
72 0
73 0
74 0
75 0
76 0
77 0
78 0
79 0
80 0
81 0
82 0
83 0
84 0
85 0
86 0
87 0
88 0
89 0
90 0
91 0
92 0
93 0
94 0
95 0
96 0
97 0
98 0
99 0

Float32Array Prototype

Please let me know if I am doing something wrong or if you need any more info. 
Thanks.

[Radar WebKit Bug Importer]

<rdar://problem/45342184>

[Eric Carlson]

Can you please attach, or give the url of, a sample that we can use to reproduce this?

The application I am working on is subscription based, and we just released the product this week. I have procured a demo account login for you to use. I do not want to post the credentials here, however. Can you email me at the email I have here (stoldney.mcstoldneyson at gmail)?

I just sent the account credentials via email. The application is a reading assessment for early elementary students, so please bear that in mind. 

The application will run through an introduction with a character and then proceed to a mic check, where you get 2 attempts. This is where the issue is easiest to spot.

The issue does not occur every time, unfortunately. On my mini, I get it about 1 in 7 tries. Others here can get more often, about 1 in 4 tries. The most consistent way I've found to reproduce is to not say anything into the mic for the first mic check attempt. You can click "done" immediately when the visualizer shows. The mic check instructions will play again and then occasionally you will see no data coming off of the mic for the second attempt.

The script handling the audio data is Recorder.js. Please let me know if you need any additional information.

[Jer Noble]

So first things first, I see that you're using a ScriptProcessorNode to pull data out of the AnalyserNode. This is going to be incredibly inefficient. The entire purpose of the AnalyserNode is to collect frequency information and make it available outside of the high-priority audio thread.  You're also using a SPN to draw to canvas! That's crazy pants!  Both of these things are the kind of thing you should be doing in a requestAnimationFrame() callback.  By adding a ScriptProcessorNode, you're degrading the performance of the entire audio graph, and you're making upstream nodes susceptible to main thread starvation and drop-outs.  First, try eliminating the ScriptProcessorNodes from Recorder.js; I think that might solve your mic dropouts.

Thanks for the feedback, I will give these suggestions a shot. 

As you can tell, I'm not overly familiar with the web audio API. Are there any resources you would recommend to learn more and get better?

[Jer Noble]

(In reply to stoldney.mcstoldneyson from comment #6)
> Thanks for the feedback, I will give these suggestions a shot. 
> 
> As you can tell, I'm not overly familiar with the web audio API. Are there
> any resources you would recommend to learn more and get better?

I don't have experience directly with this book, but O'Reilly's tech books have a good reputation: <http://shop.oreilly.com/product/0636920025948.do> "Web Audio API - Advanced Sound for Games and Interactive Apps".

That said, this is a pretty advanced (though common) pain point; garbage collection on the main thread can cause audible hiccups in Web Audio graphs containing scriptProcessorNodes(). There's work being done in the Web Audio spec to allow js processing to move to a Worker, but you'll still have GC problems there. Luckily for your particular use case, it appears you can move all your SPN code into requestAnimationFrame() with minimal changes.

My apologies, I forgot to close this bug report. Moving the visualization out of a SPN and updating the canvas on a requestAnimationFrame callback resolved this issue.

Thank you very much for the help.