Operationalizing AI: A game of trust and ethics

Explainability

Explainability for AI systems has taken a center stage in policy debates across research, business forums, and regulatory bodies. In all these discussions, the expectations from the explanations are that they should provide explainability, interpretability, transparency, and contestability. 

The most prevalent form of explanations in the industry today are feature importance and saliency maps. Various techniques are available to generate feature importance or relevance for a specific decision or for global behavior of the model.

Likewise, the ability to provide counterfactual explanation to end users or individuals about how an unfavorable outcome can be converted to a favorable one is crucial for trustworthy AI systems.  

Bias and fairness

Biases can creep into modeling processes at various stages and in various forms. The training data used might have inherent bias due to historical reasons; or bias can get into modeling if data sampling is not uniform across different classes or does not represent different groups fairly. 

Reproducibility

The ability to reproduce the entire AI model development cycle instills trust in the overall AI solution and encourages AI adoption. Documenting steps such as data processing, model training and tuning, and model testing and validation, allows us to replicate the model development, as well as helps lower risks, and trace and reproduce bugs. 

Transparency

Factors such as publishing details of models in the decision system, information about how they work together to make the final prediction, and insights on how the model performs under different conditions, are vital. Along with the expected behavior of the model, like variability of the outcomes under different conditions, is critical for end users. This aspect of transparency adds credibility to AI deployment. 

Sustainability

Developing AI solutions could lead to a large carbon footprint due to the processing of large volumes of data, use of large compute instances, and the energy needed to cool such data centers. There is a need to optimally use resources, monitor resource consumption, and optimize AI solutions to reduce the carbon footprint and be sustainable. 

Lack of understanding and ways to bridge the gap

Most of the explanations for AI model predictions are in numeric values, force plots and graphs, saliency, or heat maps, which are understood only by data scientists, and mostly remain opaque to end users. This results in lack of understanding and the inability to act on AI decisions and increases reluctance in consuming the AI outcomes. 

Communicating the explanations:

It is vital to communicate the explainer outputs to end users, business stakeholders, and regulators in simple form and language. An explanation in human perceivable form can go a long way in making AI decision far more consumable. 

Here’s an example: Consider an AI system that predicts medical insurance premiums. A chat interface to this system not only provides a mechanism to interact in natural language but also provides an explanation in human perceivable form. A user can inquire about their premium charges, where an AI model predicts and communicates the outcome in the chat. Further, the user can contest and demand an explanation for higher medical premium. The force plot explanation provided by known explainers is then translated and relayed back to user in natural language.

For instance, a message such as this would be displayed: “The premium charges are higher because your age (51 years) and weight (91 kg) are on the higher side, and you had one major surgery in the recent past.”

Generating and communicating counterfactuals

The ability of the system to recommend a minimum change that is actionable and achievable and allows one to change an unfavorable decision into a favorable one, is equally critical in establishing trust in AI systems. In case of the medical insurance premium prediction, the ability to recommend to the user that reducing their weight by five to seven kilograms or giving up smoking can significantly reduce the premium charges, highlights the transparency of the decision system. 

Uncertainty quantification with explanations

Uncertainty quantification is critical in AI modeling, especially in domains like financial services, life sciences, and healthcare, where variability of prediction or distributions of outcomes is far more important compared to a point prediction. The uncertainty quantification not only tells us the model behavior but also points out gaps in data that would lead to higher variability in model predictions. 

Computing the model uncertainty with respect to the important features identified by the explainer provides meaningful insights into the overall model behavior, including model prediction variability. It also identifies the gap in training data. 

This is illustrated with an example of medical insurance premium estimator model with the help of a SHAP (SHapley Additive exPlanations) explainer.

Caption: The SHAP explainer identifies age, weight, and previous transplant surgeries as most influential factors in predicting premium charges. It also takes into account the patient’s height, known issues such as allergies, diabetes, and chronic diseases, and family history of cancer and other problems, while calculating the premium amount.

Such mechanisms would enable us to provide an analytic conclusion such as, “Age is the most influential factor in predicting medical insurance premium. However, model uncertainty is high for the age group (29-39 years) and high variability is expected in model predictions for this age group. Augmenting the training data for the age group (29-39 years) would help lower uncertainty.”

Bias mitigation

Bias mitigating techniques such as reweighing and sampling allows us to remove bias in preprocessing stage by calibrating data, while techniques such as adversarial debiasing which is an in-processing or in-training method, allows us to make predictions free from carrying any information which can be exploited by an adversary for group discrimination. 

Several effective metrics are available to measure bias, such as the disparate impact ratio that gives us a fair view of distribution of favorable decisions between two groups—underprivileged and privileged. A disparate impact ratio closer to one indicates an unbiased system and goes a long way in building faith into AI system.

It is recommended that bias is managed in all stages of modeling. In the data pre-processing and profiling stage, protected attributes can be detected automatically, and reweighing and resampling techniques can be applied to overcome the historic or representation bias in the data. Likewise, bias in the modelling needs to be identified in the testing and assurance stages, before deployment.