How an NLP pioneer’s new company is evolving conversational AI

Natural Language Processing (NLP) is a cornerstone of modern AI and can be used to enable any number of different business use cases, from speech recognition to chatbots and sentiment analysis. NLP can be used to help AI applications better understand what a given user wants and then as part of a larger platform, actually help the user to execute whatever action or operation they need.

Jason Flaks understands what NLP is capable of more than most: He was the leading inventor of the conversational technology behind Microsoft’s Kinect and Hololens.

Now, Flaks is the CTO of startup Xembly, which is officially launching today. Flaks leads Xembly alongside CEO Pete Christothoulou, with the goal to use NLP and conversational AI to enable what the company refers to as an “AI chief of staff.”

Helping business people and individuals to better organize and schedule their work lives is a complicated task. A top level business executive might be lucky enough to have a chief of staff to help schedule meetings, take notes and determine action items for follow up. Xembly aims to fill a similar role, with an automated NLP-powered AI platform.

“Having a great week is actually pretty straightforward – you have to plan to have a great week,” Christothoulou told VentureBeat. “We developed Xembly to be a chief of staff for every knowledge worker, to handle tasks for them just as an executive assistant or chief of staff would for their CEO.”

Xena is Xembly’s conversational agent – not a warrior princess

One of Xembly’s foundational elements is Xena, and no –  it’s not a reference to the fictional warrior princess from the 90s TV show. Rather, Xena is the conversational AI agent in Xembly’s platform that users can interact with. Xena is able to listen in on meetings to take notes, pull out action items, as well as schedule meetings via email or Slack.

Flaks explained that the Xembly system operates across multiple points of presence. “We have to be present where the worker is,” he told VentureBeat. “We are where the worker is during their day, either on their calendar, email or on Slack is where most workers spend probably 90% of their time.”

The ability to transcribe notes is not a new thing for NLP. What Xembly is doing is going beyond just spoken dialogue to producing well-written prose for action items and meeting summaries. Flaks said that it takes a really complex set of machine learning models to be able to achieve that. He explained that Xembly runs its own machine learning models to detect action items from a conversation or interaction with Xena. The action items can include items like pulling out due dates and order requests. The Xembly system is also able to conduct topic segmentation for conversations to help users better understand the flow of a meeting.

Flaks emphasized that Xembly is not using an off-the-shelf NLP model. The reason why is because in his view many models have been trained on internet data, and not necessarily data coming directly from meetings.

Xembly takes advantage of the NLP evolution

Since Microsoft and Hololens, conversational AI has undergone a dramatic transformation, Flaks said.

“Conversational AI is going from being a passive listener to an active participant,” he explained, recounting that when he was at Microsoft working on the original Kinect camera, his team built what it thought was revolutionary, with the first open microphone, no push to talk, speech recognition system. 

With Kinect, a user could just say “Xbox, start a chat with Bob,” and it worked. While that system was remarkable at the time, Flaks explained it really still was a passive solution, as a user still needed to actually tell the system what to do.

“We’ve achieved the point now where we can sit in a long-form dialogue, and actually maintain state and understand what’s happening across the dialogue,” he said.

Modern conversational AI technologies and NLP can listen into a roomful of people and understand when more than just one person is talking and who is talking to whom. In Flaks’ view, modern conversational AI technologies can actually be an active listener, even in a room full of people, rather than just waiting for a single voice to request a single action.

Christothoulou said that today Xembly is able to help with scheduling, delivering notes, taking action items and putting those action items on that user’s calendar at times, so they can get their work done helping them plan their week. In the future the plan is to be able to take even more actions on behalf of the user, but further making conversational AI more active.

Flaks commented that when he thinks about NLP and the latest conversational AI technologies, one part is being able to take either the written or spoken word and understand what the user’s intent was. The other part is all about taking action on what the NLP understands.

“The real magic is, ultimately, what you can do with that,” Flaks said. “I think that’s where we continue to want to expand. The question is, how do we take that data and do meaningful things with it?”