AI-powered deep neural nets increase accuracy for credit score predictions

Credit Karma has more than 110 million users and a customer approval rate of 90%, but that wasn’t always the case. When the company launched 14 years ago, its approval percentages were in the single digits, chief technical officer Ryan Graciano said during VentureBeat’s virtual Transform 2021 conference last week.

The reason for this turnaround? Big data and machine learning.

When Credit Karma launched in 2007, the company relied on traditional datacenters because the cloud wasn’t yet part of the conversation. There would have been trouble with banking partners and credit bureaus, and “compliance people wouldn’t even let you in the door,” Graciano said.

The company got very proficient at hardware procurement and systems management but realized the physical hardware was limiting.

“The thing about big data and cloud is that big data moves really quickly, [and] the technologies change very rapidly,” Graciano said. “If you’re needing to do a six-to-nine-month hardware procurement cycle [and] a significant platform change, you’re going to be pretty far behind the curve.”

That was the first issue Credit Karma sought to resolve — the company needed more elasticity. It wasn’t just the time required to set up the hardware, but the fact that the hardware requirements were changing rapidly to keep up with new capabilities and the technology stack couldn’t keep up.

Credit Karma wound up picking Google Cloud and its machine learning offerings because BigQuery and TensorFlow made it easier to handle big data.

The machine learning evolution

The machine learning attempts were initially very straightforward. The company applied simple linear regression models to the anonymized data from its databases. Later, Credit Karma moved on to using gradient boosted trees. Nowadays, the company relies on wide and deep neural nets to predict which banks will approve customers, and at what rates. This technique runs about 80% of Credit Karma’s methods and helps facilitate Darwin, an internal system of experimentation and problem-solving.

The platform Credit Karma built is reusable, Graciano said. There was a recommendation engine on top of the machine learning platform, and everything else connected to it. Anything that happened with Credit Karma came from the system, whether it was receiving an email from Credit Karma, a push notification, or badges on the site.

“All of those things are powered by this one single system. And so that gave us the ability to spend a lot of time on the nuts and bolts of how our data scientists would work in the system,” Graciano said.

It is far easier to add new data sources and clean up the data than it is to define new algorithms. One way to improve the system is to add orthogonal data, rather than innovating on the algorithm, Graciano said. The company’s prediction capabilities expanded as more data sources were added.

“Getting those additional elements is actually a lot more powerful than the 32nd iteration on our algorithm can ever be,” Graciano said.

Graciano acknowledged it took a while to figure out what Credit Karma needed — such as a platform that allowed data scientists to automate retraining models.

“I would say we stumbled through many, many issues,” Graciano said.

Cloud was the way forward

Graciano recommends businesses move toward the cloud because it increases interoperability within the external ecosystem.

“If you’re looking for uplift, you’ll usually get more uplift by adding orthogonal data than you will by innovating on your algorithm,” he said. For Credit Karma, this was a strategic decision that paid off for the longevity of the platform, allowing it to amass useful data and making the company able to leverage it.

“Nothing is more strategic to us than data, and having a lot of power over our data,” Graciano said. Many businesses are likely going to make this move for the very same reasons, shifting from a deterministic way of developing software to a more experimental framework.