ALSA buffer underrun

[Henrik Danielsson]

I'm on Arch Linux, running Enlightenment 0.19.99.20260 as my DE with PulseAudio 6.0 as the sound system.

Music is very distorted and this issue seems to make it difficuly to quit the game as well (as indicated by the log continuing after the game apperently shutting down). Nothing happened when I tried exiting by various methods until I switched to another TTY and back, at which point the game was gone.

ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
Window closed, exiting
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
Exiting at player's request.
Have a nice day!
Player ship destroyed.
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
Player stats: deaths                        10
Player stats: shots fired / hit / accuracy  497 / 138 / 27.7666%
Player stats: worlds completed              3
Player stats: portals entered / backtracked 1 / 0
Player stats: weapons collected / killed    0 / 0
Player stats: powerups collected / killed   2 / 2
InputStorm: Shutting down.
SoundStorm: Stopping streamer thread...
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred
ALSA lib pcm.c:7905:(snd_pcm_recover) underrun occurred

[Henrik Danielsson]

Managed to get complete and separate logs of stderr and stdout using

./SphereFace > >(tee stdout.log) 2> >(tee stderr.log >&2)

However, had to end the game with a kill signal.

[Riot]

Hi Henrik, this isn't our issue but a known problem with pulseaudio.

You need to run the game with pasuspender under linux.

The steam version will have a startup script that does this for you automatically.

I'm leaving this open as a bug, however, since it would be nice for us to add that functionality directly, rather than via a startup script.

[Henrik Danielsson]

Can you link to a bug report? Using pasuspender does not sound like a fix but a workaround for not interfacing with PulseAudio correctly.

Also, running under pasuspender gives me no sound at all.

[Henrik Danielsson]

If I run

pasuspender -- ./SphereFace

I get no sound at all, but exiting works.

If I run

pasuspender -s alsa -- ./SphereFace

I have sound, but still the exact same issues as without pasuspender.

[Norgg]

The cause of the problem is a little complicated, but it's due to us using using portaudio to be cross-platform, which then tries to use alsa, which has been replaced by a pulseaudio emulation layer. All these layers sometimes lead to underruns on systems under load. I don't think there's a specific bug reported to pulse about it but if you search for the error it comes up in a bunch of places.

We're looking into a better fix/workaround for setups where this happens, but could you try running with "PULSE_LATENCY_MSEC=30 ./SphereFace"? Potentially adding to that value if it's still causing problems?

[Norgg]

I think the problems you're having with pasuspender not producing sound is probably due to the wrong device then being selected by default, we'll have an audio device selection menu in later versions.

[Henrik Danielsson]

It does get better with the latency setting, and I'm able to quit (once actually in-game, not while in the post-death zoom). Still a few underruns immediately as the game launches, betweein Initializing graphics and looking for the Oculus Rift.

[Riot]

First of all, I just want to mention how helpful it is having the full logs - thank you! I wish all our bug reports were like that.

One thing to note, the underrun messages don't come from us - they come from pulse.

Regarding pasuspender - that is indeed a workaround, but it's a workaround to underruns that are caused by multiple sound mixers running on top of each other, in a CPU-starved environment, which is inside an interrupt-level kernel callback. Our mixer is already fairly heavy, but if you're running it on top of pulse which is running its own alsa emulation layer and emulating an alsa mixer, as well as running its own mixer, and then mixing that with other system sounds - such as from your browser which may have the device open constantly even if it's sending zeros - then it can get too much for lower powered systems.

It's really a symptom of the linux standards of make-everything-compatible-with-everything-else-by-emulating-everyone-else's-competing-protocols creeping out of control, which is especially insidious in linux sound - but that's a rant for another day.

What pasuspender actually does: Calling pasuspender asks pulse nicely to stop all of that mixing nonsense and just pass this program's audio through directly, and it does this by connecting to the pulse server and sending

            pa_operation_unref(pa_context_suspend_sink_by_index(c, PA_INVALID_INDEX, 1, suspend_complete, NULL));
            pa_operation_unref(pa_context_suspend_source_by_index(c, PA_INVALID_INDEX, 1, suspend_complete, NULL));

In other words it just dumps all other sources and sinks, and this lowers the mixing overheads enough that your cpu can now keep up again, and no underruns occur. Problem is the default sound device is not always any longer the correct one to output on in that situation, and since we don't yet have a menu in sphereFACE, we can't let the user choose their output device if a sensible default isn't chosen; this will be resolved later.

Either way though, to get around it, some workaround is necessary, because there's no bug in our behaviour - the system just can't provide enough resources to run our sound engine on top of multiple layers of emulation and mixing, on your hardware. Altering the latency as John suggests may alleviate that to some degree, and it's an altrnative to pasuspender that should allow the default device to keep working, since a higher latency means the mixer is run less frequently and with a bigger buffer, saving some overhead... but neither is a perfect solution. I'm currently looking into talking to pulse directly, if it's present on the system, and seeing if we can ask it ourselves to play nice rather than relying on external fixes.

[Henrik Danielsson]

You're welcome, I know how difficult it is to debug something with very limited information so I try to provide everything I think is relevant.

I'm aware of the varying methods of forcing sound out of Linux boxes, though I've had much less problems with PA since switching to Arch instead of relying on other distros to configure it (often older versions) for me.

The selected device appears to be the same in all cases judging by the game logs, just with different numbering depending on which method used. (Unless it gets redirected somewhere.) Using pasuspender has the downside of completely killing any other audio on the machine, because ALSA needs to grabs the device instead.

I just noticed that if I run the game without any of the tweaks, it only briefly shows up in pavucontrol and flashes in and out now and then, as if PA thinks the program is not generating any sound. If I set the latency, it shows up and is stable. The entry looks like: LSA plug-in [SphereFACE]: Alsa playback on [Internal Analog Stereo] when visible.

Just in case you were interested in some performance stats: CPU-usage (AMD Phenom II X4) bounces around a bit on the 4 cores with several programs, Chromium, Steam and several servers running in the backround, but no more than it does when running other games. Each core bounces between 70-100% with the game running. Total CPU usage for SphereFace is around 10-50%, usually around 20-30% while running in the background as I type this. Machine RAM usage is around 5/16G total, with SF taking up 0.6%. Suspending the servers, closing Steam, the browser, and all other programs does not improve the sound performance without any of the tweaks applied.

Talking directly to PulseAudio sounds like a good idea. I have everything else I can think of set up to do that. PortAudio looks like cool project with good documentation. But it's a bit scary that their mailing list already had discussions about problems integrating with the emulated ALSA layer under PulseAudio back in 2011, and there's still no "native" PulseAudio backend being worked on, as far as I can tell.

[Henrik Danielsson]

Just for fun, I tested running the Windows version under the latest WINE release. No sound issues at all. =P The game runs smoothly and quitting works too. Just don't try to fire a rocket or it'll try to use a bad pointer hehe.

[Riot]

Yeah all our games work very well in Wine, says something about the state of sound on Linux when the WinMME emulation in Wine performs better than the native Alsa emulation of Pulse... ahem

[Riot]

Regarding a native pulse backend for PortAudio - that would be the ideal. The reason we're using (and in fact maintaining our own fork of) PortAudio is so we can keep the same code fully cross-platform, and we'd lose that if we started writing our own specialised back-ends just to cope with one operating system or another. The ideal solution would be to extend PortAudio and make it play nicer with Pulse, but it's potentially a big job, and outside of our area of expertise. With limited development time available, it's hard to give it a high priority unfortunately.

[Henrik Danielsson]

WINE's WinMME implementation (MMDevAPI for newer versions) passes sound back to ALSA, so it should get picked up by PA's emulation layer the same way the native game's output would, just with one more redirection layer.

Yeah, using existing abstraction layers like that are a good way to take advantage of other people's work across several platforms at once. I maintain an abstraction module (web related) myself so I understand why you use it.

Are you using the Linux-specific ALSA extensions in PortAudio, pa_linux_alsa.h? It should allow turning on realtime scheduling. I tried enabling PA's realtime ALSA scheduling and priorities globally but it didn't help. The best I could do was to get the first few seconds to play normally before getting distorted.

The buffer underruns are making it really difficult to quit the game, feels like it's waiting for the audio as it says it's stopping the streamer thread but still getting underruns until I kill it manually. Input seems a bit unresponsive during the zoom sequences too when not using a workaround. It's fine when playing but I can't skip past the zooms reliably.

[Riot]

Indeed it is waiting for the streamer to end, for which it has to clear its buffers; this is to avoid having a nasty situation with clicks and pops (which can equal blown speakers if the soundsystem is big enough) because of a truncated waveform, as well as the fact some platforms stop producing sound correctly for the rest of the user session if the callbacks aren't properly completed at program exit time (OS X used to have this problem, if i recall). Might be good to try to detect underruns and skip forward in the buffers, but I'm not sure there's any interface exposed by portaudio in a cross-platform way that would let me do that without talking to pulse directly. And pulse likes to report its errors by just spamming to stderr, which isn't great.

We have pa_linux_alsa.h but haven't done any poking with it directly - I'm trying to stick to using the C++ wrapper, of which I seem to also be the only user these days judging by some of the bugs I've had to fix in it, so if it has useful functionality I can add a matching wrapper extension.

The only functions it exposes though are:

/** Initialize host API specific structure, call this before setting relevant attributes. */
void PaAlsa_InitializeStreamInfo( PaAlsaStreamInfo *info );

/** Instruct whether to enable real-time priority when starting the audio thread.
 *
 * If this is turned on by the stream is started, the audio callback thread will be created
 * with the FIFO scheduling policy, which is suitable for realtime operation.
 **/
void PaAlsa_EnableRealtimeScheduling( PaStream *s, int enable );

#if 0
void PaAlsa_EnableWatchdog( PaStream *s, int enable );
#endif

/** Get the ALSA-lib card index of this stream's input device. */
PaError PaAlsa_GetStreamInputCard( PaStream *s, int *card );

/** Get the ALSA-lib card index of this stream's output device. */
PaError PaAlsa_GetStreamOutputCard( PaStream *s, int *card );

/** Set the number of periods (buffer fragments) to configure devices with.
 *
 * By default the number of periods is 4, this is the lowest number of periods that works well on
 * the author's soundcard.
 * @param numPeriods The number of periods.
 */
PaError PaAlsa_SetNumPeriods( int numPeriods );

/** Set the maximum number of times to retry opening busy device (sleeping for a
 * short interval inbetween).
 */
PaError PaAlsa_SetRetriesBusy( int retries );

/** Set the path and name of ALSA library file if PortAudio is configured to load it dynamically (see
 *  PA_ALSA_DYNAMIC). This setting will overwrite the default name set by PA_ALSA_PATHNAME define.
 * @param pathName Full path with filename. Only filename can be used, but dlopen() will lookup default
 *                 searchable directories (/usr/lib;/usr/local/lib) then.
 */
void PaAlsa_SetLibraryPathName( const char *pathName );

I could try tweaking the PaAlsa_SetNumPeriods result to see what happens. Bit tricky for me to experiment with that value because I don't have a linux machine slow enough to reproduce the problem at present, I could expose it in the menu settings for user adjustment once we have a menu though.

If you want to have a chat about this in more detail come join us on IRC, #voxelstorm on Freenode :)

[Henrik Danielsson]

Sounds like you could try calling PaAlsa_EnableRealtimeScheduling(stream, 1) on the stream before starting it.

I'm not sure why you think my machine is being slow? It's apparently capable of playing the sounds just fine, as long as the buffer isn't running empty, as proven by just running the Windows version under WINE.

I'll jump on IRC right away. :)

[Riot]

Removing milestone: Beta (automated comment)