Developing a Custom Audio Backend for Love.js
⚠️ This English content was machine-translated from Japanese.
Audio Playback Challenges in Love.js
I've been a bit busy and haven't been able to develop much, but I continued working on improvements to my Love.js fork from before. Previously I worked on WebGL2 support, but another issue I wanted to solve was audio playback.
Web Audio is quite tricky, and with a design that streams data from the Wasm side, there is an inherent instability in playback depending on devices and browsers such as mobile devices. Even when using the latest APIs like Audio Worklet, if streaming raw audio buffers from the WASM side, dependency on the main thread is unavoidable, making it difficult to guarantee completely uninterrupted audio playback.
Love.js's default audio backend goes through OpenAL, which is SDL's audio backend dependency. SDL can be used because Emscripten provides a browser implementation by default, but as mentioned above, there are fundamental limitations to the process of passing raw audio buffers from the WASM side itself.
Solution: Utilizing Two Web Audio APIs
As far as I've confirmed, the most stable approaches for playing audio using the Web Audio API are two methods: createBufferSource and createMediaElementSource.
createBufferSource doesn't stream raw audio buffers from the WASM side, but rather loads the entire thing into memory, so there are no latency or streaming issues. However, due to its nature of loading everything, it's not suitable for long music tracks and is intended for use with short sounds like sound effects.
createMediaElementSource creates a normal HTML5 Audio element and connects and controls the audio to the Web Audio API via createMediaElementSource. Since browsers optimize HTML5 Audio elements, playback works smoothly in various environments. Since audio data needs to be made available via HTTP, some ingenuity is required at build time.
Implementation Example and Testing
This time I created a backend using these two APIs and made an example to test a simple audio API.
When testing on low-performance Android devices, you can confirm that audio cuts out in the example using the existing Love.js build (/projects/comprehensive-audio-example-reference).
Also, by pressing the Heavy Process button in the lower right, heavy processing is executed for several seconds, allowing you to check how frame rate degradation affects audio playback.