Last month, Microsoft announced that Teams, its competitor to Slack, Facebook’s Workplace, and Google’s Hangouts Chat, had passed 44 million daily active users. The milestone overshadowed its unveiling of a few new features coming “later this year.” Most were straightforward: a hand-raising feature to indicate you have something to say, offline and low-bandwidth support to read chat messages and write responses even if you have poor or no internet connection, and an option to pop chats out into a separate window. But one feature, real-time noise suppression, stood out — Microsoft demoed how the AI minimized distracting background noise during a call.
We’ve all been there. How many times have you asked someone to mute themselves or to relocate from a noisy area? Real-time noise suppression will filter out someone typing on their keyboard while in a meeting, the rustling of a bag of chips (as you can see in the video above), and a vacuum cleaner running in the background. AI will remove the background noise in real time so you can hear only speech on the call. But how exactly does it work? We talked to Robert Aichner, Microsoft Teams group program manager, to find out.
The use of collaboration and video conferencing tools is exploding as the coronavirus crisis forces millions to learn and work from home. Microsoft is pushing Teams as the solution for businesses and consumers as part of its Microsoft 365 subscription suite. The company is leaning on its machine learning expertise to ensure AI features are one of its big differentiators. When it finally arrives, real-time background noise suppression will be a boon for businesses and households full of distracting noises. Additionally, how Microsoft built the feature is also instructive to other companies tapping machine learning.
Stationary vs. non-stationary noises
Of course, noise suppression has existed in the Microsoft Teams, Skype, and Skype for Business apps for years. Other communication tools and video conferencing apps have some form of noise suppression as well. But that noise suppression covers stationary noise, such as a computer fan or air conditioner running in the background. The traditional noise suppression method is to look for speech pauses, estimate the baseline of noise, assume that the continuous background noise doesn’t change over time, and filter it out.
Going forward, Microsoft Teams will suppress non-stationary noises like a dog barking or somebody shutting a door. “That is not stationary,” Aichner explained. “You cannot estimate that in speech pauses. What machine learning now allows you to do is to create this big training set, with a lot of representative noises.”
In fact, Microsoft open-sourced its training set earlier this year on GitHub “to advance the research community in that field.” While the first version is publicly available, Microsoft is actively working on extending the data sets. A company spokesperson confirmed that as part of the real-time noise suppression feature, certain categories of noises in the data sets will not be filtered out on calls, including musical instruments, laughter, and singing.
Microsoft can’t simply isolate the sound of human voices because other noises also happen at the same frequencies. On a spectrogram of speech signal, unwanted noise appears in the gaps between speech and overlapping with the speech. It’s thus next to impossible to filter out the noise — if your speech and noise overlap, you can’t distinguish the two. Instead, you need to train a neural network beforehand on what noise looks like and speech looks like.
Speech recognition vs. noise suppression
To get his points across, Aichner compared machine learning models for noise suppression to machine learning models for speech recognition. For speech recognition, you need to record a large corpus of users talking into the microphone and then have humans label that speech data by writing down what was said. Instead of mapping microphone input to written words, in noise suppression you’re trying to get from noisy speech to clean speech.
“We train a model to understand the difference between noise and speech, and then the model is trying to just keep the speech,” Aichner said. “We have training data sets. We took thousands of diverse speakers and more than 100 noise types. And then what we do is we mix the clean speech without noise with the noise. So we simulate a microphone signal. And then you also give the model the clean speech as the ground truth. So you’re asking the model, ‘From this noisy data, please extract this clean signal, and this is how it should look like.’ That’s how you train neural networks [in] supervised learning, where you basically have some ground truth.”
For speech recognition, the ground truth is what was said into the microphone. For real-time noise suppression, the ground truth is the speech without noise. By feeding a large enough data set — in this case hundreds of hours of data — Microsoft can effectively train its model. “It’s able to generalize and reduce the noise with my voice even though my voice wasn’t part of the training data,” Aichner said. “In real time, when I speak, there is noise that the model would be able to extract the clean speech [from] and just send that to the remote person.”
Challenges
Comparing the functionality to speech recognition makes noise suppression sound much more achievable, even though it’s happening in real time. So why has it not been done before? Can Microsoft’s competitors quickly recreate it? Aichner listed challenges for building real-time noise suppression, including finding representative data sets, building and shrinking the model, and leveraging machine learning expertise.
Representative data sets
We already touched on the first challenge: representative data sets. The team spent a lot of time figuring out how to produce sound files that exemplify what happens on a typical call.
They used audio books for representing male and female voices, since “speech characteristics do differ between male and female voices.” They used YouTube data sets with labeled data that specify that a recording includes, say, typing and music. Aichner’s team then combined the speech data and noises data using a synthesizer script at different signal to noise ratios. By amplifying the noise, they could imitate different realistic situations that can happen on a call.
But audiobooks are drastically different than conference calls. Would that not affect the model, and thus the noise suppression?
“That is a good point,” Aichner conceded. “Our team did make some recordings as well to make sure that we are not just training on synthetic data we generate ourselves, but that it also works on actual data. But it’s definitely harder to get those real recordings.”
Privacy restrictions
Aichner’s team is not allowed to look at any customer data. Additionally, Microsoft has strict privacy guidelines internally. “I can’t just simply say, ‘Now I record every meeting.’”
So the team couldn’t use Microsoft Teams calls. Even if they could — say, if some Microsoft employees opted-in to have their meetings recorded — someone would still have to mark down when exactly distracting noises occurred.
“And so that’s why we right now have some smaller-scale effort of making sure that we collect some of these real recordings with a variety of devices and speakers and so on,” said Aichner. “What we then do is we make that part of the test set. So we have a test set which we believe is even more representative of real meetings. And then, we see if we use a certain training set, how well does that do on the test set? So ideally yes, I would love to have a training set, which is all Teams recordings and have all types of noises people are listening to. It’s just that I can’t easily get the same number of the same volume of data that I can by grabbing some other open source data set.”
I pushed the point once more: How would an opt-in program to record Microsoft employees using Teams impact the feature?
“You could argue that it gets better,” Aichner said. “If you have more representative data, it could get even better. So I think that’s a good idea to potentially in the future see if we can improve even further. But I think what we are seeing so far is even with just taking public data, it works really well.”
Cloud and edge
The next challenge is to figure out how to build the neural network, what the model architecture should be, and iterate. The machine learning model went through a lot of tuning. That required a lot of compute. Aichner’s team was of course relying on Azure, using many GPUs. Even with all that compute, however, training a large model with a large data set could take multiple days.
“A lot of the machine learning happens in the cloud,” Aichner said. “So, for speech recognition for example, you speak into the microphone, that’s sent to the cloud. The cloud has huge compute, and then you run these large models to recognize your speech. For us, since it’s real-time communication, I need to process every frame. Let’s say it’s 10 or 20 millisecond frames. I need to now process that within that time, so that I can send that immediately to you. I can’t send it to the cloud, wait for some noise suppression, and send it back.”
For speech recognition, leveraging the cloud may make sense. For real-time noise suppression, it’s a nonstarter. Once you have the machine learning model, you then have to shrink it to fit on the client. You need to be able to run it on a typical phone or computer. A machine learning model only for people with high-end machines is useless.
Pushing processing to the edge
There’s another reason why the machine learning model should live on the edge rather than the cloud. Microsoft wants to limit server use. Sometimes, there isn’t even a server in the equation to begin with. For one-to-one calls in Microsoft Teams, the call setup goes through a server, but the actual audio and video signal packets are sent directly between the two participants. For group calls or scheduled meetings, there is a server in the picture, but Microsoft minimizes the load on that server. Doing a lot of server processing for each call increases costs, and every additional network hop adds latency. It’s more efficient from a cost and latency perspective to do the processing on the edge.
“You want to make sure that you push as much of the compute to the endpoint of the user because there isn’t really any cost involved in that. You already have your laptop or your PC or your mobile phone, so now let’s do some additional processing. As long as you’re not overloading the CPU, that should be fine,” Aichner said.
I pointed out there is a cost, especially on devices that aren’t plugged in: battery life. “Yeah, battery life, we are obviously paying attention to that too,” he said. “We don’t want you now to have much lower battery life just because we added some noise suppression. That’s definitely another requirement we have when we are shipping. We need to make sure that we are not regressing there.”
Download size and future-proofing
It’s not just regression that the team has to consider, but progression in the future as well. Because we’re talking about a machine learning model, the work never ends.
“We are trying to build something which is flexible in the future because we are not going to stop investing in noise suppression after we release the first feature,” Aichner said. “We want to make it better and better. Maybe for some noise tests we are not doing as good as we should. We definitely want to have the ability to improve that. The Teams client will be able to download new models and improve the quality over time whenever we think we have something better.”
The model itself will clock in at a few megabytes, but it won’t affect the size of the client itself. He said, “That’s also another requirement we have. When users download the app on the phone or on the desktop or laptop, you want to minimize the download size. You want to help the people get going as fast as possible.”
Adding megabytes to that download “just for some model” isn’t going to fly, Aichner said. After you install Microsoft Teams, later in the background it will download that model. “That’s what also allows us to be flexible in the future that we could do even more, have different models.”
Machine learning expertise
All the above requires one final component: talent.
“You also need to have the machine learning expertise to know what you want to do with that data,” Aichner said. “That’s why we created this machine learning team in this intelligent communications group. You need experts to know what they should do with that data. What are the right models? Deep learning has a very broad meaning. There are many different types of models you can create. We have several centers around the world in Microsoft Research, and we have a lot of audio experts there too. We are working very closely with them because they have a lot of expertise in this deep learning space.”
The data is open source and can be improved upon. A lot of compute is required, but any company can simply leverage a public cloud, including the leaders Amazon Web Services, Microsoft Azure, and Google Cloud. So if another company with a video chat tool had the right machine learners, could they pull this off?
“The answer is probably yes, similar to how several companies are getting speech recognition,” Aichner said. “They have a speech recognizer where there’s also lots of data involved. There’s also lots of expertise needed to build a model. So the large companies are doing that.”
Aichner believes Microsoft still has a heavy advantage because of its scale. “I think that the value is the data,” he said. “What we want to do in the future is like what you said, have a program where Microsoft employees can give us more than enough real Teams Calls so that we have an even better analysis of what our customers are really doing, what problems they are facing, and customize it more towards that.”
Author: Emil Protalinski.
Source: Venturebeat
