Bug 132691 – HTMLAudioElement doesn't play with Javascript on mobile WebKit (Safari and Chrome on iPad/iPhone/Android) 7)

• Component: New Bugs
• Status: RESOLVED WONTFIX

[Hugo]

I am trying to create a game that pre-loads Audio elements in JavaScript and HTML5 using the <audio> tag i.e. HTMLAudioElement 

After defining the audio source the Javascript attempts to play the audio so it can pre-buffer the sound with HTMLAudioElement.play() 
This code wworks great on Windows, and on all browsers, except it doesn't play the sound on iOS (iPad, iPhone).
The Audio.play() can only be triggered by a user interactive click. 

This may logically be done because of bandwidth limitations, but the when developing a game, the pre-loading strategy can not properly work. 

Can this be changed in the WebKit ? Is there a Javascript workaround to this?

Thank you in advance
Hugo

PS: I also assume this is the same behavior for HTMLVideoElement 

This is a sample code
var LETTERS={
    L1:{value:1, wavfile:"res/letters/c.wav", mp3file:"res/letters/c.mp3", description:"c"},
    L2:{value:2, wavfile:"res/letters/h.wav", mp3file:"res/letters/h.mp3", description:"h"},
    L3:{value:3, wavfile:"res/letters/k.wav", mp3file:"res/letters/k.mp3", description:"k"},
    L4:{value:4, wavfile:"res/letters/l.wav", mp3file:"res/letters/l.mp3", description:"l"},
    L5:{value:5, wavfile:"res/letters/q.wav", mp3file:"res/letters/q.mp3", description:"q"},
    L6:{value:6, wavfile:"res/letters/r.wav", mp3file:"res/letters/r.mp3", description:"r"},
    L7:{value:7, wavfile:"res/letters/s.wav", mp3file:"res/letters/s.mp3", description:"s"},
    L8:{value:8, wavfile:"res/letters/t.wav", mp3file:"res/letters/t.mp3", description:"t"},
};

   for (var src in letters){

        //var audio=new Audio(); // Tried with this, doesn't change anything 
       
        var audio = document.createElement("audio");
         
        audio.controls=true;
        audio.preload="auto";
        audio.autoplay=true;
        audio.volume=0;
        audio.id=src; 
        audio.style.display="none";
        
        if (audio.canPlayType("audio/mpeg")) {
            var src1 = document.createElement("source");
            src1.type="audio/mpeg";
            src1.src = letters[src].mp3file;
            audio.appendChild(src1);
        }
        if (audio.canPlayType("audio/wav")) {
            var src2 = document.createElement("source");
            src2.type="audio/mpeg";
            src2.src = letters[src].mp3file;
            audio.appendChild(src2);
        }
        
        sounds[letters[src].value]=audio;

        // below code is here to try to play/preload the audio (volume is set to 0)     
        try{
            audio.load();
        }catch (err){};
        
        try{
            audio.click();
        }catch (err){};

        try{
            audio.play();
        }catch (err){};
        
        
        audio.addEventListener("canplaythrough", function(){
            letterCount ++;
            if (letterCount>=numLetters) {
                callback(sounds);
            }
        });
        
    }
    if (letterCount==0){
        callback(sounds);
    }

[Hugo]

More colors on the issue, 

I intend to deliver the game through HTML5 over a browser window, and not a native App. 

I tested the code (see below) on Google Android 4.4.2 Nexus 7 tablet and this platform has the same problem. I.e. the audio would not preload or beeen played by Javascript, unless the user click on the Audio element. 

The same testing works fine with WebKits on Chrome and Safari (ironically), and is perfect with Firefox and IE on Windows 7 (yes, you read correctly IE11, ??#..#?)  

The various HTML5 game framework that I found at http://html5gameengine.com/ are trying to do the same thing (preload video and audio HTML5 assets so the game can present it when it transitions from scene to scene), but all are reporting the same problem, and have to fallback on browser plugins.

Why wouldn't the mobile WebKit let the Javascript page code play the sound without requiring the user to click on the HTMLAudioElement? It does on Windows...

I understand Tablets users may be concerned with bandwidth usage and associated costs over 3/4G, but rather than blocking the code in the WebKit, why not prompting the user if it would allow the application to pre-load the sounds and interacting with the Audio Element? Why not make this feature work when connected over WIFI?

I respect the talents of the many of you that invest time and energy in the development of the WebKit. Please let me know what your thoughts on the subject. This is a show stopper for me and many others. Let me know if you need more code sample. 

Thank you 

Hugo

[Jer Noble]

Hugo, you're correct that iOS WebKit requires user interaction to load or play media, but this policy will not change, so I'm closing this bug as WONTFIX.

However, you have some options.

1) You can try the WebAudio API instead.  It has a similar restriction, but it requires user interaction only once per AudioContext, so playing sounds after an initial click shouldn't be a problem. Media data is loaded over XHR, so pre-loading your media is unrestricted.

2) You can create all your <audio> elements at once in response to a "click" event and require your users to click the screen to begin your presentation.

But if sites like http://html5gameengine.com/ are trying to use <audio> elements for game sounds, they're going to have a bad time.  <audio> is not intended for low-latency short sounds.  They should move over to the WebAudio API instead.

[Hugo]

(In reply to comment #2)
> Hugo, you're correct that iOS WebKit requires user interaction to load or play media, but this policy will not change, so I'm closing this bug as WONTFIX.
> 
> However, you have some options.
> 
> 1) You can try the WebAudio API instead.  It has a similar restriction, but it requires user interaction only once per AudioContext, so playing sounds after an initial click shouldn't be a problem. Media data is loaded over XHR, so pre-loading your media is unrestricted.
> 
> 2) You can create all your <audio> elements at once in response to a "click" event and require your users to click the screen to begin your presentation.
> 
> But if sites like http://html5gameengine.com/ are trying to use <audio> elements for game sounds, they're going to have a bad time.  <audio> is not intended for low-latency short sounds.  They should move over to the WebAudio API instead.

Thank you for your feedback. 

I have followed instructions at https://developer.apple.com/library/safari/documentation/audiovideo/conceptual/using_html5_audio_video/PlayingandSynthesizingSounds/PlayingandSynthesizingSounds.html and http://www.html5rocks.com/en/tutorials/webaudio/intro/ 

I implemented a function that creates a new AudioContext(), load the audio file asynchronously using XHR (arraybuffer) create a new source, set the buffer and play(0).

I attached this function to an onClick() event of a TD tag. It still will not play the sound. Although it play fine on Windows. 

What am I doing wrong ?

Thanks
Hugo

[Jer Noble]

(In reply to comment #3)
> (In reply to comment #2)
> > Hugo, you're correct that iOS WebKit requires user interaction to load or play media, but this policy will not change, so I'm closing this bug as WONTFIX.
> > 
> > However, you have some options.
> > 
> > 1) You can try the WebAudio API instead.  It has a similar restriction, but it requires user interaction only once per AudioContext, so playing sounds after an initial click shouldn't be a problem. Media data is loaded over XHR, so pre-loading your media is unrestricted.
> > 
> > 2) You can create all your <audio> elements at once in response to a "click" event and require your users to click the screen to begin your presentation.
> > 
> > But if sites like http://html5gameengine.com/ are trying to use <audio> elements for game sounds, they're going to have a bad time.  <audio> is not intended for low-latency short sounds.  They should move over to the WebAudio API instead.
> 
> Thank you for your feedback. 
> 
> I have followed instructions at https://developer.apple.com/library/safari/documentation/audiovideo/conceptual/using_html5_audio_video/PlayingandSynthesizingSounds/PlayingandSynthesizingSounds.html and http://www.html5rocks.com/en/tutorials/webaudio/intro/ 
> 
> I implemented a function that creates a new AudioContext(), load the audio file asynchronously using XHR (arraybuffer) create a new source, set the buffer and play(0).
> 
> I attached this function to an onClick() event of a TD tag. It still will not play the sound. Although it play fine on Windows. 
> 
> What am I doing wrong ?

Try to call AudioContext.startRendering() from within your click handler.

[Hugo]

I got it to work on Android., My issue was a codec on iPad. 
It's working now. Thank you for your help
Hugo

[Hugo]

Hi Jer,

Thank you for your patience with me on this. I have implemented the WebAudio for Webkit, and could successfully run it on Android+Chrome, but can not run it on iPad/iPhone with Chrome and Safari (i spoke too fast :-( )


I trigger the javascript code upon a user interaction (touch on the screen). The function does
1- creates the audio context (only once) 
  if (window.webkitAudioContext) {
    myaudioContext = new webkitAudioContext();
  } else if (window.AudioContext) {
    myaudioContext = new AudioContext();
  }

2- on the same call the call loads a collection of 4 sounds (only once, and using a BufferLoader class (XHR, base64 MP3 files from local URL) onto an array called sounds. 

3- The function sets a repeatable timer (3 seconds) and returns from the function.   

4- In parallel, the timer selects a song randomly from the pre-loaded songs list and play the song using 
  var source = myaudioContext.createBufferSource();
  source.buffer = songs[random].buffer;
  source.connect(myaudioContext.destination);

  if (!source.start) source.start = source.noteOn;
  source.start(0);

I tried adding
  try{
    myaudioContext.startRendering();
  }catch (ex){};
but it did not help 

The code executes just fine, but no sound comes out of the speaker. 

The use case is fairly trivial. This works on all windows browsers (FF, Safari,), on Mac browsers (Chrome, Safari and Firefox) and mobile tablets like Chrome on Nexus/Android OS. => But it doesn't work on WebKit on iOS (iPad and iPhone).

The parsing of the MP3 files work just fine, in fact, I can create an <audio> element in the HTML, and when I click on it, play the song just fine, so I believe the MP3 is encoded properly, and decoded by the context just fine.

I am puzzled by this problem.  

Thank you for taking the time to help me. I can give you an url for testing privately upon request.

Best regards

Hugo

[Jer Noble]

Hugo, please file a new Bugzilla bug and attach a testcase, and CC me.

[Hugo]

Created attachment 231234 [details]
Testcase that shows the issue on WebKit on iPad/iPhone 

I can't log a new Bug, the interface doesn't work for me. So I'm attaching the test case here. 

Thank you for your support. 

Hugo

[Hugo]

Re-open the bug for consideration

[Hugo]

restore to original status/ won't fix