Zillow, an online marketplace that facilitates the buying, selling, renting, financing, and remodeling of homes, employs lots of AI technologies to do things like estimate home prices. But the output of AI systems like these can be opaque, creating a “black box” problem where practitioners and customers can’t audit the systems properly. Without transparency, serious problems like algorithmic bias can persist undetected, and trust in the models becomes impossible.
For obvious ethical reasons, this is why explainable AI (XAI) is so crucial to the creation and deployment of AI systems, but pragmatically, it’s also key to the success of AI-powered products and services from companies like Zillow.
David Fagnan, director of applied science on the Zillow Offers team, discussed with VentureBeat how and why XAI is indispensable for the company. When Fagnan joined Zillow four years ago, the company was working on its “Zestimate” product that estimated home prices. Earlier this year, the company crowdsourced help improving its model for home valuations, handing $1 million to a trio of data scientists who improved the Zestimate algorithm by 13%. “What attracted me to that problem was not just that it was an interesting machine learning challenge. It was really about bringing power to the people and helping them make smarter decisions about their homes.”
About a year and half ago, the company started up Zillow Offers, and that pulled him in even further. “Zestimate was [about] helping change the way people shop, ‘turning on the lights’ to what used to be a pretty opaque space. But now we’re really trying to move deeper into the transaction, so changing the way people actually do this process,” he said.
Zillow Offers is a relatively new offering, but it’s growing. Launched first in Phoenix in April 2018, it’s now available in 22 markets including Los Angeles, and Zillow expects to light up five more by the middle of 2020. In Q3 2019 alone, Zillow said, more than 80,000 homeowners requested an offer from the company. The number of conversions was much lower: Within the quarter, Zillow bought 2,291 homes and sold 1,211, meaning it’s holding almost half of its purchases from Q3.
“There was an enormous breadth of different challenges with Zillow Offers. The focus is really on helping home sellers have a less stressful process.” That’s especially poignant, he said, if sellers are motivated by specific life events that are already stressful, such as a death in the family or a job relocation.
But it’s not a simple process.
The challenge of offer accuracy
Selling your home and buying another is usually an exhausting and nerve-wracking experience. There are inspections on both ends, inevitable repairs and renovations that need done, and worst of all, timing issues. It’s almost a chicken-and-egg problem, especially if you’re planning to use profits from the sale of one home as a down payment on another: If you find the home you want, you need to make an offer before someone else does, but in order to close the sale, you may need to sell your home first. There are usually no guarantees on that timeline. Sometimes you end up carrying two mortgages, one for your new home and one for the home you’re trying to sell.
Zillow Offers is meant to be an intermediary that obviates this problem. Instead of waiting and hoping for someone to make an offer on your house, agreeing to a price, scheduling the inspection, performing any necessary repairs and/or agreeing to an adjusted price, and waiting for the realtors and title office to schedule a closing date, Zillow can make you a cash offer within days.
Determining the amount of that offer is the central challenge that Zillow Offers has to meet. Housing prices are notoriously hard to pin down, and they range wildly — not just from region to region and state to state, but neighborhood by neighborhood and even street to street. Prices are tied to “comps,” or comparative analysis of similar properties, but within that there’s a great deal of subjectivity depending on specific amenities like the size of the yard, or whether it has a new kitchen, and on and on.
In other words, it’s not just that housing prices are affected by an interconnected web of interdependent variables, it’s that many of those variables are of subjective value and importance.
Humans in the loop
Zillow has a lot of data. Much of it has come from shoppers on its website, so the company has some idea of what people are looking for — “…and we have a lot of analytics behind that,” said Fagnan.
“What’s new about Zillow Offers now, when we’re actually buying a house, helping to transact — it’s fairly high stakes. You know, we’re not just putting a number on the website that helps people shop, we’re buying a $300,000 asset,” he said.
Zillow Offers has its own internal analysts but also local partners, like realtors. And they combine the expertise and judgment of those humans with AI tools — a human-in-the-loop approach.
Getting humans and machines to work together is a practical challenge. First, they need to be “speaking” the same language. Every industry has its own domain-specific lingo, nomenclature, and processes, and housing is no different, so any models need to be trained accordingly so that both humans and machines are performing reasonably similar evaluations of the same sorts of factors to create comparative market analyses.
This need for XAI shows up right away. Fagnan likens it to showing your work in math classes in school.
That’s especially important because often, you can only get explainability at the cost of some accuracy. However, Fagnan said that in a human-in-the-loop system, doing can actually have the net effect of greater accuracy on the whole. “If the human is more easily able to audit it, validate it — now, in that setting, the combined output of the human and machine may be even more optimal.”
Fagnan mentioned a deep learning technique for image classification called “this looks like that,” which was detailed in a paper that was presented at NeurIPS 2019. It involves a deep learning network architecture called prototypical part network (ProtoPNet) that “dissects the image by finding prototypical parts, and combines evidence from the prototypes to make a final classification.” In other words, it outputs “prototypes” of images, and when taken together, the prototypes offer parts that can classify the whole. It’s an intermediate output, Fagnan explained, that is the sum of the prototypes. For Zillow Offers, the “prototypes” are similar home sales.
“It is a fairly general technique, but we’re…using a specific definition of explainability that’s motivated by industry,” he said.
Part of the challenge of Zillow Offers’ human-in-the-loop system — any such system, really — is finding the balance between humans and machines. “In order to optimize this human-in-the-loop system, we’d like to figure out when the human is best, when an assistive situation is best, and […] when a machine is best,” said Fagnan.
An art and a science
The practical application of this process for Zillow Offers relies on local real estate agents, which is crucial, because those particular humans in the loop happen to have not just industry expertise, but knowledge that’s germane to specific locales, like knowing that a certain neighborhood is on the upswing with an influx of young families, or that street noise is too strong near a given house, or what have you. Fagnan refers to this as a sort of “art,” compared to the “science” that machine learning brings.
In addition to bringing valuable but intangible or unquantifiable knowledge, humans can serve as auditors of the data. They may be able to spot glaring data errors, like maybe the listed square footage seems really far off, and they can look at the property photos and cross reference with county records and figure it out. They can also provide data labeling that isn’t available at scale, take a close in-person look at a home to evaluate potential repairs or check the quality of past renovations, and more.
Some things just don’t show up in reports or even in photos. Photos, especially, can be misleading or obfuscate problems. Perhaps the photos look great, but a person taking one step into the house will discover that a floor is slanted. That’s the sort of thing computer vision is unlikely to catch.
As human auditors catch errors and improve the data being fed into the system, the system’s accuracy improves. Taking the example of the slanted floor: If the human flags it, tracks what the underlying problem is, and notes the resulting repair cost, the improved model can factor that cost into its estimate, and surface it the next time the house goes on the market.
“So that’s where over time, I think the combination of the human’s art and the machine’s science, and speaking the same language, can really take things to the next level,” said Fagnan.
As with any AI-assisted tool that improves over time, it’s fair to wonder if realtors who participate with Zillow Offers are helping to perfect a tool that could replace their job someday. Although Fagnan wouldn’t say for certain that machines will never replace the human component of Zillow Offers, he came close to it. “It doesn’t make sense to me that having more options would be more optimal,” he said. “I think having both pieces is always going to be more optimal.”
Author: Seth Colaner
Source: Venturebeat
