When enterprise data software company Cloudera looked into using conversational AI to improve its customer support question-and-answer experience, it didn’t want to go slow, said senior director of engineering Adam Warrington in a conversation at Transform 2020. When your company is new to conversational AI, conventional wisdom says you might gradually ease into it with a simple use case and an off-the-shelf chatbot that learns over time.
But Cloudera is a data company, which gives it a head start. “We were kind of interested in how we could possibly use our own data sets and technologies that we had internally to do something a little bit more than just dipping our toes into the water,” Warrington said. “We were more interested in getting off-the-shelf chatbot software that was extensible through APIs,” he added. Warrington said Cloudera already had an internally stored “wealth” of data in the form of customer interactions, support cases, community posts, and so on. The idea was to answer customer support questions with a high degree of accuracy without having to wait for the chatbot to acquire domain knowledge.
Because Cloudera maintained records — again, this is a data company — of past customer issues and solutions, it had its own corpus to feed the chatbot. In order to teach the chatbot, the company wanted to extract the semantic context of things like the back-and-forth chatter between a support person and customer, as well as the specifics of the actual problem being solved.
To ensure that they knew what was relevant, the Cloudera team relied on their own subject experts to manually label and classify the data set. “The work can be a little bit tedious, as is the case with many machine learning projects, but you don’t need — in this particular case — millions and millions of things categorized and labeled,” Warrington said. He added that after about a week of work, they ended up with a labeled data set they could use for training and testing. And, Warrington said, they achieved their goal of 90% accuracy.
The company now had models that could understand which words and sentences within a given support case were technically relevant to that case. Then the models could extract the right solution from the best source, be it a knowledge base article, product documentation, community post, or what have you.
But the team needed to go a step further. “Now there’s the derivative problem downstream, which is [that] what we actually want to do is … provide answers to the customers that are relevant to their problems. It’s not just about understanding what’s relevant and what’s not,” Warrington said. Here again, the team relied on subject matter experts — specifically, support engineers — to ensure customers were receiving the best solutions.
Warrington said that although Cloudera is currently using its subject matter experts internally, more data is coming in from real interactions. “As this project continues to go on in the public space, we expect to get more signals from our customers that are actually using the chatbot,” he said. “And so we’ll start to use those inputs, those signals, from our customers to really expand on our test sets and our training set, to improve the quality from where it’s at today.”
What’s perhaps most surprising is the short time to market. “From inception of the problem statement — of trying to use our own data sets and our own technology to augment chatbot software to return relevant results based on customer problem descriptions — this took under a month,” Warrington said. Why so fast? It certainly helped that Cloudera has its data already set up in its own data lake. “All of our processing capabilities already exist on top of this, so everything from analytics to operational databases to our machine learning systems and things like Spark — we’re able to access these data sets through these different technologies.”
More to the point, Warrington said in the course of researching chatbot software they could use, the team discovered they already had some pertinent models. They had previously built models to help their internal engineers more efficiently find and address customer support issues. “It turns out when you’re running all these machine learning projects on an architecture like this, you can share work that has been done in the past that you didn’t necessarily expect to use in this way,” Warrington noted. He also said the fact that they had a modern data structure, meaning the data was already unsiloed, was a huge advantage.
In addition to the wisdom of relying on subject matter experts, focusing on a specific problem or set of problems, and starting with data architectures that grant you agility, Warrington’s advice is to keep things simple. “As we grow and mature, this particular approach in this particular implementation — we very well could go and explore more advanced techniques [and] more advanced models as we add more types of signals into the system,” he said. “But out of the gate, to hit the ground running, use something simple. We found that you can actually provide very useful results to the customers, very quickly, using these kinds of approaches.”
Author: Seth Colaner.
Source: Venturebeat
