Add audio processing module

[ladvoc]

This PR adds support for the WebRTC audio processing module and enables AEC for microphone tracks.

[holofermes]

Hey all, I've been keeping an eye on this PR, and just give this a spin, and while this works on mac/windows, it fails on android arm64 ( I assume it would be the same for the other arm architectures ).

I think the latest android .so does is not up to date:

NullReferenceException: Object reference not set to an instance of an object.
 at LiveKit.AudioProcessingModule..ctor (System.Boolean echoCancellationEnabled, System.Boolean noiseSuppressionEnabled, System.Boolean highPassFilterEnabled, System.Boolean gainControllerEnabled) [0x00000] in <00000000000000000000000000000000>:0 
 at LiveKit.RtcAudioSource..ctor (System.Int32 channels, LiveKit.RtcAudioSourceType audioSourceType) [0x00000] in <00000000000000000000000000000000>:0 

I noticed the install.py does not have android listed in the platforms, but I manually downloaded and replaced the ffi-android-arm64/liblivekit_ffi.so and ran a build with it but that doesn't work either as it fails to load the .so, which I suspect this is why the install.py does not grab the android builds.

[ladvoc]

Hi @holofermes, thank you for reporting this. Android should definitely be included as one of the platforms in install.py. I will look into this and make the necessary changes in a separate PR.

[theomonnom]

I think we're likely going to have a skew here. (this will impact the AEC a lot in long room duration)
Is there a way to directly process the frames as we receive them?

[theomonnom]

We could directly use ProcessReverseStream here?

[theomonnom]

We will also get a skew here, so as soon as we're a bit late, we're going to hear bad quality input (jittery audio).

It's OK to push faster than realtime, the Rust-SDKs will handle it in a high precision queue

[theomonnom]

I see the TASK_DELAY is 5ms.
We're sending 10ms, this work because Rust will buffer, so maybe it's fine?

[ladvoc]

Hi @theomonnom, thank you for your feedback. Yes, there does appear to be a skew for longer room durations. I've moved the calls to the APM methods directly into the audio filter callbacks, however, this seems to introduce some audio artifacts that I haven't been able to explain yet. I think the issue is related to the forward stream being processed before the reverse stream, however, I need to do some more investigation to see if this is the case.

[theomonnom]

Hi @theomonnom, thank you for your feedback. Yes, there does appear to be a skew for longer room durations. I've moved the calls to the APM methods directly into the audio filter callbacks, however, this seems to introduce some audio artifacts that I haven't been able to explain yet. I think the issue is related to the forward stream being processed before the reverse stream, however, I need to do some more investigation to see if this is the case.

I think it's most likely because this function is too slow:

client-sdk-unity/Runtime/Scripts/RtcAudioSource.cs

Line 96 in 89437aa

private void OnAudioRead(float[] data, int channels, int sampleRate)

You could also try to increase the default DSP buffer of Unity

[theomonnom]

Maybe this one too?