Back in February 2024, the UK government published its response for AI regulation, to adopt a non-statutory, contextual, cross-sectoral principles-based approach. This might be old news for some, but if you work in financial services, it’s crucial to pay attention. The Financial Conduct Authority (FCA) and other regulators also responded. As an engineer in financial services developing machine learning models for determining creditworthiness, this grabbed my attention…

“We are focused on how firms can safely and responsibly adopt the technology as well as understanding what impact AI innovations are having on consumers and markets. This includes close scrutiny of the systems and processes firms have in place to ensure our regulatory expectations are met. And we believe an evidence-based view, one that balances both the benefits and risks of AI, will ensure a proportionate, effective and pro-innovation approach to the use of AI in financial services.” FCA AI Update.

Despite these efforts, it’s surprising how many firms seem to have adopted a “carry on and do nothing” strategy. We have a world where mandated open APIs like UK Open Banking (CMA9) have unleashed a wealth of data, and powerful tools to model this data have proliferated to the point of saturation. While the FCA and Prudential Regulation Authority (PRA) are developing sector-specific guidelines to ensure AI applications are safe, firms are racing ahead using this technology without adequate safety measures in place (UK Government AI Regulation Approach).

Accountability and governance are top priorities for regulators like the FCA, and firms should not wait for regulation to catch up. It is our duty to act responsibly now.

Talk is cheap, so let’s delve into the problems and provide real examples of why this is so important and yes, I’m aware that different sectors have unique needs, and the financial sector is no exception to this.

Why AI is scary: Bias and Discrimination

Machine learning models can reflect the biases present in their training data. For instance, if historical data is biased, the model might perpetuate these patterns, leading to unfair lending practices. Let’s consider a platform that decides you’re a credit risk because it learned from biased historical data that didn’t favor your demographic.

A notable example of this type of bias can been seen in Google Vision Cloud, a computer vision model that labeled an image of a dark-skinned individual holding a thermometer as a “gun” while a similar image of a light-skinned individual was labeled “electronic device.” Unfair, right?

In another example the University of Washington researchers found that when searching for various occupations, such as “CEO,” women were significantly underrepresented in the image results due to how data was labled.

I’m not just picking on Google, these are just a few examples highlighting the potential for bias in AI. In financial services, such biases can significantly impact a person’s livelihood, leading to discriminatory lending practices where certain demographics might be unfairly denied credit.

You may well be thinking that in financial services, bias in the data sets is necessary because these factors matter (and you might be right but that is where transparency and explainability come in later). However, imagine a model trained on data from 2020–2022, during the peak of COVID-19, when we encountered outliers and patterns never seen before, at least in my lifetime. Would we see the same results using the same model on data from periods before or after the pandemic? Likely, the results would be very different.

Let’s consider another example, the rise of the gig economy and how people earn money now versus post-2015. This paints a very different picture compared to earlier periods.

So how can we solve this? The answer is in first understanding why a model is making the decisions it makes and ensuring it aligns with our expectations. We achieve this through transparency and explainability and in my opinion, this is precisely the problem the FCA is trying to solve via a non-statutory, contextual, cross-sectoral principles-based approach we will see over the next few years. By promoting these principles, the FCA aims to ensure that models in financial services are both fair and reliable, addressing biases without overcorrecting and maintaining accuracy.

It comes down to the infamous “black box” problem. If we can’t understand how our models arrived at a decision, it’s hard to trust them. This lack of transparency can erode trust and make regulatory compliance challenging.

Preparing for the Future: Explainable AI for Regulatory Compliance

Explainable AI (XAI) is highly relevant and crucial for models predicting creditworthiness. Here’s why and how it applies to the work I’m doing.

Regulatory Compliance: Financial regulations, such as those set by the FCA, require transparency in decision-making processes. Explainable AI ensures that credit models comply with these regulations by providing clear, understandable reasons for decisions made by the AI.
Trust and Accountability: Explainable AI helps build trust with consumers and stakeholders by making the decision-making process transparent. When customers understand why they were granted or denied credit, it fosters trust and reduces the likelihood of disputes.
Bias Detection and Mitigation: Explainable AI aids in identifying and correcting biases in the model. By understanding how decisions are made, engineers can detect patterns that may indicate discriminatory practices and take steps to mitigate them.
Improving Model Performance: Understanding the decision-making process can highlight areas where the model might be improved. It allows for a more nuanced analysis of model performance, leading to better, more accurate predictions.

Explainability Tools and Approach

SHAP (SHapley Additive exPlanations): This tool provides a way to understand the contribution of each feature to the final decision. For credit models, SHAP values can show how factors like income, credit history, and employment status impact the credit score. Read more: https://github.com/shap/shap
LIME (Local Interpretable Model-agnostic Explanations): LIME explains the predictions of any classifier by perturbing the input and seeing how the predictions change. This can help in understanding local model behavior and the importance of different features.
Continuous Monitoring and Feedback: Implement a system for continuous monitoring of the model’s performance and the explanations it provides. Collect feedback from real users to understand if the explanations are clear and helpful, and adjust the model or explanation methods accordingly.

Example of Explainable AI in Action

Consider a creditworthiness model that uses features like income, credit history, employment status, and number of dependents. Using SHAP, you might find that income has the highest impact on the model’s decision, followed by credit history. If a customer is denied credit, SHAP values can show that the primary reasons were insufficient income and a short credit history. This explanation could also be communicated to the customer, providing transparency and an opportunity for them to improve their credit profile.

Focus on Fairness and Bias Mitigation

Regularly audit your models for biases. This means using diverse datasets and implementing fairness constraints to ensure your trained model treats all users equitably.

The Other Extreme: Reverse bias!

I guess if we are talking about biases in data we should add a few words on over overcorrection which can lead to equally problematic outcomes. For instance, in a bid to avoid reinforcing stereotypes or underrepresentation, some AI image generation tools have stopped producing accurate images of historical figures based on factual historical data, instead opting to diversify their appearance. While the goal is commendable, the outcome can distort historical understanding and educational content, leading to a form of “reverse bias.”

In the realm of financial services, overcorrecting bias can have serious consequences as well. Consider a credit scoring model overcorrected to avoid any potential bias against somone who is unemployed for example. In an effort to ensure fairness, the model might overcompensate and incorrectly classify high-risk applicants as low-risk, simply to avoid perceived discrimination. This could result in lending money to individuals who are likely to default, thereby increasing the financial risk to the lender.

Conclusion

While the UK government’s discussions on AI regulation are a positive step, it’s clear that the FCA is actively engaged in understanding and regulating AI. They have set priorities for the next 12 months, including continued diagnostic work, international cooperation, and enhancing their regulatory frameworks to better oversee AI applications in financial services (Skadden, Arps, Slate, Meagher & Flom LLP) (Goodwin), while the FCA is making significant efforts, the pace of regulatory adaptation can indeed be slower compared to the rapid advancements in AI technology.

For firms with regulatory permissions, there are indeed stringent requirements to provide detailed information and maintain transparency, but the effectiveness of these regulations in fully understanding and mitigating AI risks is still evolving. The FCA is focused on promoting safe and responsible AI use and is aware of the challenges posed by rapid technological change (FCA) (Goodwin). Therefore, firms should not rely solely on regulatory directives but should take proactive steps to ensure their models or applications are fair, transparent, and well-governed. It’s up to us as engineers and data scientists to lead the way in responsible AI adoption, setting the standard for the industry.

Written by Glyn Jackson