Rhythm Game Lag Compensation

Today I finally got around to implementing latency calibration for Frog In The Pocket. This has been bothering me for awhile since there's a noticeable lag through the speakers of my Macbook Air.

To measure this I ran this test in Logic: Press record with the metronome on and speakers turned up, recording the speaker's output through the built in microphone. (I set the buffer size to as low as possible, 32 samples)

If latency=0 then the transients should line up exactly with downbeats. As you can see there's a noticeable delay, measured out to ~20ms. What does this sound like? Very noticeable flamming 😱

Loopback Metronome Recording

Since this is a roundtrip measurement, it's unclear what proportion is the output latency from the speakers versus input latency from the microphone. My guess is the speakers actually play a big part because I don't experience noticeable latency through my headphones. Macbooks must be doing some kind of DSP on the speakers that is introducing lag.

Unfortunately this makes realtime instruments difficult without requiring the player to use dedicated hardware.

Lag Compensation

One observation: if the player's inputs are triggering sounds in realtime, theoretically no lag compensation is needed for fair grading. To line up their hits (audio-wise) with the reference track the player would need to anticipate such that the resultant hit lines up with the track.

For example: say our latency is 1s. That means when the player presses the space button, the cowbell sample is heard a full second later.

Say we schedule a track to start playing at t=5. The player's job is to have their cowbell sample align with the start of the track.

When should the player actually hit the space bar? The answer is t=5, which seems obvious but from the player's perspective this is one second before they hear the start of the track!

For low enough latency musicians automatically compensate by anticipating, but a 1s delay would basically be impossible.

Zero latency?

There is one always available zero-latency hack and that is to hear the clack of the keys and to mute the cowbell lol. As long as we know what the latency is the player can tap along blissfully to what they actually hear. Behind the scenes we just need to account for the delay on the grading side.

This is probably why most rhythm games do not trigger sound on player inputs, the latency is just too high.

In Frog In The Pocket I want the player to hear themselves play, but if the latency is above say 10ms I think it's better to mute.

Calculating latency

To compensate for latency we need to know what the latency actually is, which is more difficult than I'd like. One way to guesstimate is to have the player do a loopback test like I did but this is too high effort.

The typical method is to have the player tap along to a beat and calculate the average error. This is how I've implemented it so far as well, except instead of averaging the error I'm using autocorrelation to detect the optimal lag. I'm not sure if it's actually better but it feels right haha.

Try the demo out and let me know what kind of numbers you get on your device!

Frog In the Pocket Lag Compensation Demo