Addressing bias in AI: Opportunities to even the playing field for consumers

(00:00) Kareem Saleh:

The challenge with using the AI where you don’t have good data, where the data’s messy, missing or wrong, means that you run the risk of learning the wrong thing.

(00:13) Narrator:

Welcome to 22 Minutes in Lending, your go-to podcast for insights on all things lending. From lending practices, regulatory updates, how to enhance lending efforts and more. In each episode, Vince Passione connects with industry leaders to discuss the latest trends and happenings around the lending industry. Let’s dive in to the latest in lending.

(00:36) Vince Passione:

Welcome back to the second part of our conversation with Kareem Saleh. I’m your host, Vince Passione. I’m excited to continue our conversation with Kareem, founder of FairPlay.ai. In our first episode, we took a deep dive on AI and underwriting, where it’s fallen short in its promise to remove bias from lending decisioning and how fairness as a service could get the industry back on track. Today we’ll build on that foundation and cover further insights from Kareem’s work and vision. So you’ve said this in prior podcasts. I want to go back, and you mentioned it here, but I want to make sure everyone understands. You’ve made the statement “Historical data is inherently biased”, correct?

(01:13) Kareem Saleh:

Sounds like a gotcha question is about to come true.

(01:18) Vince Passione:

But you have. Yeah.

(01:19) Kareem Saleh:

Yeah. I have. Sure. Yes, for all kinds of historical reasons, the data reflects the decisions that we made as a society and not all of those decisions were fair to everybody.

(01:30) Vince Passione:

Right. So, I’m a simple guy and I live in a simple world when it comes to credit. I always tell people when they ask me “What’s credit like?”, and I say, well, if your neighbor came by and they asked you to lend you money, you’d want to know two things. You’d want to know their ability to pay you back, which typically you would do it personally or we do it when we lend money is we take a look at your income, we look at your outstanding debts, we take a look and make sure that you got some excess cash there and say, “Yeah, you have ability to pay this back.” And the second thing we’re going to do is if it was someone I didn’t know, my neighbor, I’m going to talk around the community and say, “Hey, they borrowed money from you, did he pay you back?”

(02:06):

So, what is his willingness or her willingness to pay me back? So, when I think about ability to pay, yeah, I’ve heard you say that real-time information is better. We think about cash flow underwriting, we could talk about Nova Credit. We had Misha on the show, I want to hear about that later. But then when I think about your willingness to pay, isn’t it about looking at your historical performance and isn’t your historical performance some indicator of future performance? How do you solve for that? Am I missing your point when you say that, “Look, historical data is biased and let’s look to the future and be smarter about it”?

(02:44) Kareem Saleh:

You’re 100% right about that. The most difficult-

(02:48) Vince Passione:

Let’s stop the show. Okay, you can say that, no, I’m only kidding.

(02:53) Kareem Saleh:

You’re 100% right about that. One of the most difficult questions in all of this fairness stuff is effectively how do you quantify the value of a swap-in? Because there are going to be folks that you declined and you don’t know anything about their performance and maybe they weren’t approved anywhere else, and there are going to be folks who maybe were approved for products that weren’t right for them or that were predatory and their riskiness might be overstated. And so fundamentally what you’re saying is how do I quantify the value of a swap-in? And the truth is, there are some tests that you can do like reject inference and others.

(03:32):

In my experience working in emerging markets, and I don’t think anybody wants to hear this, and this may not make me popular at cocktail parties with your audience, but sometimes you have to test and learn into these populations. If you want 100% certainty about what the credit performance of a population is going to be, you either have to estimate it and get comfortable with the estimates or you have to go test and learn into that population. Which invariably means that sometimes you’re going to lose money because you have to understand what a bad loan in that population looks like.

(04:13) Vince Passione:

Looks like. Yeah, no, I always say you need a whole bunch of test data that shows you bad behavior if you’re going to train an algorithm.

(04:23) Kareem Saleh:

I wonder if in your experience, that can be a hard pill for people to swallow sometimes.

(04:28) Vince Passione:

It absolutely is, right? They want to use someone else’s data, right? No one wants to take the hit. And we saw what happens when you do that. Many fintechs that launched as lenders, they went to venture capitalists and said, “Let me have your money. I’m going to lend a bunch of money out. I’m probably going to lose a lot of it, but that learning data is more valuable than the money you gave me because now I’ve got the corpus of something to build upon.” I think the challenge sometimes what many of those VCs didn’t realize is the speed at which you grow a portfolio is not a sign of success. I always tell people, “It’s easy to give people money. It’s hard to get it back.” That’s where the data becomes valuable is whether you get it back or not. But you touched again on learning and losing money and taking risks on new populations.

(05:19):

Many of our clients are ITIN lenders. They’re starting to get into that business, and it can be risky. We had Victor Corro on from Coopera Consulting, and he consults with our clients about, hey, this is how you attract a minority population to your credit union. You need to look like them. You have to be able to be trusted by that community. And I, like most people when they listen to Victor, make the mistake of believing that all these minorities, we were talking about Hispanics, well, they must be so that underserved community. He said, “No, Vince, they’re not. These are second, third generation individuals. They’re doing well. Some of them want wealth management.”

(05:58):

But having said that, I go back to that bias piece, and I look at ITIN lending and how my clients are doing it. Are you interacting with any lenders now that are going into those populations where today the way my clients get their test data is they go; they certify themselves as either low-income designated or CDFIs. They then apply for a grant to cover the losses and then they go into that community and start lending. But I think about it and I go, that sounds antiquated. Why do I need to go get a grant if potentially I could use AI? Can AI solve that problem? Will we mitigate the need for these grants to promote people lending into riskier populations?

(06:45) Kareem Saleh:

I did a lot of work in emerging markets where we use those kinds of credit enhancements. And I think that, again, as we were just saying, the challenge with using the AI where you don’t have good data, where the data’s messy, missing or wrong means that you run the risk of learning the wrong thing. And so I actually, just as we were saying earlier, talking about the VC-funded companies, it’s not clear to me that that’s altogether different from what you were describing a moment ago, which is we’re going to provide some credit enhancement to acquire a bunch of data about the distributional properties of this population. And frankly, I think we need more of that to really achieve financial inclusion. The way that these subpopulations, we need an on-ramp for these subpopulations, which are poorly represented in the data to become well-represented. And so, I think that there’s probably a lot of solutions in that area that can do a lot of good, and that I know from my experience doing lending in that segment can also be profitable for lenders.

(07:58) Margie Click:

Hello, this is Margie Click, CEO and President of Agriculture Federal Credit Union. As a $360 million credit union, we’re always looking for ways to innovate and expand our financial solution offerings to attract new members. That’s why for nearly a decade we’ve been partnering with Lenti to attract and acquire new credit union members.

(08:21) Vince Passione:

So, we’re at that point where I have to ask you, what’s the secret sauce? How do you do what you do?

(08:26) Kareem Saleh:

Yeah. So happy to talk about the secret sauce. We use a variety of algorithmic fairness techniques and let me try to explain them in ways that your audience might understand. So the first technique that we use comes from the world of self-driving cars. And it might be useful to take a step back and remember that every algorithm, every predictive model must be given a target, an objective that it seeks to relentlessly maximize. So in financial services or a credit model, that target might be predict who’s ever going to go 60 days or 90 days delinquent. But if you think about it for a second, it will occur to you that giving a machine or a robot one objective to relentlessly maximize might cause it to behave in all kinds of unintended ways. Imagine if Tesla gave the neural networks that power self-driving cars, the mere objective of getting you from point A to point B.

(09:25):

Well, the self-driving car might do that while blowing through red lights or driving on the sidewalk or causing mayhem to pedestrians. So what does Tesla do? It has to give the neural networks that power self-driving cars two objectives, get the passenger from point A to point B while also respecting the rules of the road. And we can do that in financial services, predict who’s going to default while also minimizing the differences in outcomes for one group relative to another. And it works. We see that it allows lenders to increase their approval rates on the order of 10%, increase their take rates through optimized pricing on the order of 13% and increase their positive outcomes for historically disadvantaged groups by 20%. So good for profits, good for people, good for progress.

(10:19) Vince Passione:

Wow.

(10:20) Kareem Saleh:

Another method that we use involves seeing whether you can take samples of the data, small samples, small subsets, and find alternative learning paths to the target that require different subsets of information, which we call anti proxies. So if you think that the thing that you’re looking for when you’re doing fair lending compliance of models is, are the variables serving as a proxy for protected status? And so what you can do is effectively take samples of the data and see if they predict the opposite of protected status, which is unable to make a prediction about protected status. You can think of those variables as anti proxies. And so, we’ve come up with a really smart way of sampling the data and then searching for anti-proxies, which are both predictive of the target, but not at all predictive of protected status.

(11:20) Vince Passione:

Now that’s a great explanation for how you do what you do, Kareem, let’s go back to data and training data. So how do you avoid your algorithms being contaminated the same way those credit algorithms… I say ‘contaminated’, but biased by the data. Is it because you have this derivative data that tests it? Because I look at training data and to me, sometimes you have to create your own because the future is different. It’s like I always look at ‘buy now, pay later’, and I go, is there enough information to really figure out if anybody’s ever going to pay that back? These are just anomalies and maybe we need to make up the data.

(11:59) Kareem Saleh:

Yes, you’re absolutely right. We have to maintain really rigorous model governance practices to make sure that our models don’t fall prey to the very threats that we started our company to combat. But you’re also right that in order to assess a model’s fairness, you need to know who’s a man, who’s a woman, who is Black, who’s over the age of 62. And because we see all of that information across a wide range of asset classes and across the credit spectrum, it allows us to be able to make judgments about where a given outcome is relative to the average outcome or the most commonly occurring outcome, et cetera.

(12:41) Vince Passione:

Okay. Let’s change topics. Climate change. You spent a large part of your career there. You did some work for the Obama administration. And it’s interesting, even today I got a couple of phone calls from the credit unions. We do lending in the construction space, they were wanting to know some of the demographic information because of what happened down in Florida. So there was a Filene study that said about 60% of credit unions have about $1.2 trillion in assets where they’re at physical risk for climate shifts. It seems like a high number, but what’s your take on that risk and how our clients should be thinking about when they think about climate change?

(13:24) Kareem Saleh:

I think your clients should be thinking very seriously about it. I can tell you I live in California, we have very severe climate events here. I used to live in the Hollywood area, and I have friends who have homes in that area who are unable to insure those homes because the insurance companies are terrified of the climate risk. They have to go all the way to Lloyd’s of London and get specialty policies on their homes and that is both difficult and expensive. And that’s just brush fires, right? Haven’t talked about sea level rise. I am fortunate to live by Venice Beach, and if there’s not very much sea level rise, I won’t be sitting here right now, and there are loans on this property. So I think that that means that my lender is also at risk. And so I think that lenders have to be taking climate risk in their portfolios very seriously.

(14:28):

It is the canonical, arguably black swan that a thing that is a low frequency, but potentially very high impact event. And it feels to me like the smart thing to do in those cases is to try to hedge that risk somehow. But I also think that climate change represents a really powerful opportunity for credit unions, right? There’s a bunch of money that’s been made as a part of this IRA, this infrastructure legislation that was passed to finance things like energy efficiency projects, electric vehicles, clean energy generation and storage, all kinds of resilience and adaptation projects. I did a lot of project finance work earlier in my career when I was in the Obama administration. That stuff is profitable and does a lot of good in the world, which I think is what credit unions exist to do.

(15:32) Vince Passione:

Yeah, we saw Inclusive gave out $2 billion, right? They were awarded $2 billion for this greenhouse gas reduction fund. So where do you think credit unions fit in this space? We helped them years ago do some energy efficiency lending for HVAC upgrades, but what types of loans, based on your experience, do you think the credit unions should be getting involved in? Because look, I think a lot of folks are very concerned. There was an interesting article in the Times just two weeks ago about people who bought homes in St. Petersburg, Florida, right? Can’t get insurance, and now property values are going down. I wouldn’t want to be the lender on those properties. So where do you think our credit unions should go with those dollars when they’re lending?

(16:15) Kareem Saleh:

I think energy efficiency projects. I think clean energy generation, clean energy storage, grid enhancements, all of the stuff to promote distributed generation and clean generation of power. I think that the future of our grid is more digital, more distributed and cleaner.

(16:38) Vince Passione:

That’s great. So, let’s talk about the future. I always like to end up here talking about your views on the future and trends. What do you think our credit unions should be alerted for in 2025?

(16:48) Kareem Saleh:

Yeah. Well, so you don’t have to take my word for it. The NCUA gave a webinar, I think it was in December of last year. They were very clear that fair lending issues were really high on the agenda. They’re super focused on auto, they’re super focused on auto pricing. If you’re not performing ongoing fair lending testing of your auto portfolios, I think you’re putting your institution at some regulatory and reputational risk. I also think that increasingly credit unions are buying either loans that are generated by third parties using AI scores or buying the AI scores themselves, or maybe AI created attributes. I think you need to be very careful to understand those AI generated scores, loans and attributes, including doing independent testing and validation. Because ultimately, it’s your charter, your institution whose reputation and business are at risk.

(17:52) Vince Passione:

That’s great. Good insight. So, what’s next for Fairplay? New products, new partners? What’s going on?

(17:57) Kareem Saleh:

Yeah, we’ve got a lot of very cool announcements coming up this fall, including with some big technology companies, some big telecom companies, some big credit bureaus. So hopefully if you’ll have me back, Vince, there’ll be more I can say about it. But we’ve got a wind at our back and we’re fortunate to have partners who really care about getting fairness in AI right. And I can’t wait to tell you more about what we’re doing the next time we talk.

(18:29) Vince Passione:

Well, everyone, that’s all the time that we have for today. Kareem, thank you so much for joining me today. I appreciate your time and your insights. And thank you to our listeners for tuning in to this two-part series. Don’t forget to subscribe so you can enjoy future episodes, and I’ll meet you back here for our next 22 Minutes in Lending.

(18:46) Narrator:

Thank you for listening to the 22 Minutes in Lending podcast. We hope you enjoyed today’s episode. You’ll find links to any resources mentioned in the show notes. If you’re enjoying our show, be sure to subscribe and leave us a five-star review.