Voice cloning is an exciting area of development in the field of generative AI. It involves replicating a person’s vocal stylings, including pitch, timbre, rhythms, mannerisms, and unique pronunciations, through the use of technology. Startups like ElevenLabs have garnered significant funding for their dedicated pursuit of voice cloning. Now, Meta Platforms, the parent company of popular platforms like Facebook, Instagram, WhatsApp, and Oculus VR, is making its mark with a voice cloning program called Audiobox. However, there’s a catch.
Meta recently unveiled Audiobox on its website, developed by researchers from the Facebook AI Research (FAIR) lab. Described as a “new foundation research model for audio generation,” Audiobox builds upon Meta’s previous work in this domain, Voicebox. This program can generate voices and sound effects using voice inputs and natural language text prompts. With Audiobox, users can easily create custom audio for a multitude of use cases. They simply need to input a sentence for the cloned voice to say or describe a sound they want to generate, and Audiobox does the rest. Additionally, users can even record their own voice and have it cloned by Audiobox.
Meta has developed a family of models under Audiobox, with one specifically focused on speech mimicry and another for generating ambient sounds and sound effects like barking dogs, sirens, or children playing. All these models are built on the shared self-supervised model Audiobox SSL. Self-supervised learning (SSL) is a deep learning technique in which AI algorithms generate their own labels for unlabeled data. By training the foundation model using unsupervised audio data, Meta aims to achieve generalization and tackle the challenges posed by the unavailability or low quality of labeled data.
Like many generative AI models, Audiobox relies heavily on human-generated data for training. The FAIR researchers utilized around 160K hours of speech (primarily in English), 20K hours of music, and 6K hours of sound samples. This dataset covers a wide range of content, including audiobooks, podcasts, speeches, conversations, and recordings under different acoustic conditions. To ensure diversity and representation, speakers from over 150 countries and more than 200 primary languages were included. However, the research paper does not specify the exact sources of this data or whether it is from the public domain, raising important questions about copyright infringement and the rights of creators and owners.
To demonstrate the capabilities of Audiobox, Meta has released several interactive demos. One such demo allows users to record their own voice speaking a sentence’s worth of text and then replicates that voice when the user enters additional text to be read back in their cloned voice. The AI-generated cloned audio is remarkably similar to the original voice, although not identical. Meta also enables users to generate entirely new voices based on text descriptions such as “deep feminine voice” or “high pitched masculine speaker from the U.S.” Additionally, users can restyle their recorded voices or generate entirely new sounds by entering text prompts. For example, typing “dogs barking” results in realistic dog barking sounds that are indistinguishable from the real thing.
Unfortunately, there is a major catch when it comes to Audiobox. Meta includes a disclaimer with its interactive demos, stating that they are for research purposes only and may not be used for any commercial use. Furthermore, usage of Audiobox is restricted in the states of Illinois and Texas, due to state laws related to audio collection. Although Meta has not made Audiobox open source, it deviates from the company’s usual commitment to openness. This departure raises questions about the future of Audiobox and whether it will eventually become a commercial product or be made open source like other Meta projects.
As AI continues to advance rapidly, it is only a matter of time before voice cloning technology becomes more widely available and commercialized. While Audiobox may have limitations in terms of usage and accessibility, it represents a significant step forward in the field of generative AI. Whether from Meta or other innovators in the space, voice cloning is poised to redefine various industries and offer new opportunities for customization and creativity. However, it’s imperative to address important legal and ethical considerations, such as copyright concerns and the protection of intellectual property, as voice cloning becomes more prevalent and accessible.
Leave a Reply