LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models
PDF: https://mjenrungrot.github.io/chrome-media-audio-papers/publications/lmcodec/static/pdf/lmcodec.pdf
We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test was conducted and shows that the quality is comparable to reference codecs at higher bitrates.
![notion image](https://www.notion.so/image/https%3A%2F%2Fmjenrungrot.github.io%2Fchrome-media-audio-papers%2Fpublications%2Flmcodec%2Fstatic%2Fimages%2Foverall.png?table=block&id=a561fa1e-7e85-44ac-86c9-dd89fb7f7e9f&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F94c9fc29-0744-46fb-b453-dd8f1f57786d%2FUntitled.png?table=block&id=00131207-48dc-43bd-ac7a-2703278a4700&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F9b1ef4b1-19dc-4e06-9891-502ded068514%2FUntitled.png?table=block&id=853322a4-a394-46ce-b308-811b4c0d4bf1&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fdaa2aa49-dae6-44b3-9979-56f243c98acc%2FUntitled.png?table=block&id=d5ccb791-1f09-4dc8-9539-6dc89518221b&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F88f2dd0f-e264-4def-ad71-2897e3f6ba2c%2FUntitled.png?table=block&id=ba548be2-e8bf-48f1-9ed2-fc58d79fcf4b&cache=v2)