For Otherwise We Don’t Know What They Are Doing: Why AI Systems Need to Be Explainable

Intelligent systems increasingly support or take over human decisions. This development influences all areas of human life: Intelligent systems recommend new shows on streaming portals, support online searches by giving recommendations, automatically screen thousands of applicants, predict crime, or support medical diagnosis. Systems support human decision-making and work tasks by filtering and analyzing information, providing recommendations for decisions, or automatically implementing actions. In many cases, the human is still in charge of the final decision – ‘in the decision loop’ – as when intelligent systems analyze applications and filter the most promising applicants for human resources managers. But current developments suggest that completely autonomous systems such as autonomous cars – in which the human is ‘out of the loop’ – will become more prevalent in the future.

There is broad national and international agreement (at least within Europe) that the successful and ethical use of intelligent systems corresponds with trust in these systems, that there will be challenges regarding attribution of responsibility and accountability concerning decisions based on intelligent systems, and that such decisions might easily interfere with fundamental (ethical and legal) rights. Accordingly, calls for policy-makers, scientists, or developers to address these three issues are growing louder. Plausibly, these societal desiderata – enabling reasonable trust, responsible decision-making, and respect for fundamental rights – can best be met if the relevant intelligent systems are “comprehensible”, “transparent”, “interpretable”, or “understandable”. (Notably, these are just some of the terms used in this context.) We propose that these properties can be delivered by the vehicle of explanation: Transparency properties require explainability. Others have defended similar claims. For instance, articles in research and white papers from policymakers (e.g., recommendations from the German Federal Government or the European Union) call for explainability as a foundation of human-centered or humane AI, as a means of adhering to judicial standards and ethical norms. On the operative side of things, computer scientists try to follow these calls by providing ways of making intelligent systems interpretable, which has become a serious challenge in the age of advanced machine-learning. Accordingly, the research field of explainable AI (XAI), which aims to render intelligent systems explainable, is booming. 

However, there is a lack of connection between the calls for transparency or explainability from theorists and representatives of society, on the one hand, and the work done by developers on concrete methods for achieving interpretability, on the other. Overstating the claim a bit, up to now it is unclear whether the two worlds can be connected successfully, whether they talk about the same things, and whether the endpoint of this development will turn out to be a suitable leverage point for judicial regulations, norms, and ethical standards. Computer science investigates interpretability of systems mostly focusing on interpretability of the outputs, but typically with little effort to increase understanding of how the system processed the inputs to get to the outputs. However, understanding what inside the system actually led up to the output seems to be highly relevant for guaranteeing genuine transparency, in particular the kind of transparency that can serve as a foundation for reasonable trust, responsible decision-making, or respect for fundamental rights. At the same time, stakeholders call for explainability without clearly specifying what they mean, or how one could create explainable systems that adhere to their standards of explainability. This raises the worry that their expectations simply cannot be met by intelligent systems.

The project Explainable Intelligent Systems (EIS) operates on the assumption that bringing together societal demands and their active implementation requires theoretical background work on two fronts, on the aforementioned societal demands and on explanations. Additionally, EIS will combine insights from both areas to gain a better understanding of how explanations of intelligent systems need to work to provide for transparency and thus to meet the societal demands. Work on the societal demands examines what features of intelligent systems are desirable from the perspective of society, i.e. what moral, legal, or psychological demands can appropriately be placed on such systems. It compares these with current legislation, highlighting which demands are already met (and which ones aren’t). It investigates how these demands can motivate a call for transparency.

Work on the nature of explanation and understanding focuses on how explanations function and are able to provide understanding for actual people, and on the requirements placed on explanations from a legal perspective. It connects these insights with interpretability methods developed by computer science, and even plans to make a novel proposal as to how black box systems may be opened.

Finally, drawing the strands together, EIS will develop an account of adequate explanation of intelligent systems and their outputs. This account can be expected to be context-sensitive, seeing as explanations need to cater to the needs of the relevant users and affected parties in particular contexts. Further, we will test empirically whether our proposed explanations provide users and affected parties with the required understanding, so that the systems can meet the societal demands. Moreover, the proposed method of opening the black box will be tested for its implementability. In the end, EIS will have the materials on hand for developing concrete policy recommendations.



An Interdisciplinary Project (or: the More, the Merrier)

The project as described calls for a highly interdisciplinary research effort. In short, Computer Science explores what is technically possible, while Philosophy, Psychology, and Law Studies formulate and/or test desiderata and requirements, and apply theoretical work on explanations and understanding to the case of intelligent systems. The combination of these different perspectives adds feedback loops, informs each of the fields, and guides further research. Here is a more detailed overview regarding the research of the different fields as we plan to tackle them in EIS.

Whenever an intelligent system recommends something (or someone) to someone, or makes an autonomous decision, this has moral dimensions and thus raises ethical questions. Is the decision morally permissible? Does a system’s recommendation help us make (morally) better decisions? Are we allowed to act upon it and what are our obligations in the role of decision-maker? Vice versa, if we are subject to such decisions, what are our (moral) rights? This is the perspective from Practical Philosophy.

For instance, it seems essential to a liberal democracy that people have a right to contest decisions made by the government or administrative authorities. In order to be in a position to make an autonomous decision about a therapy, I must be in a position to give my informed consent. And in order to send someone to jail instead of putting them on parole, on the basis of a recidivism prediction, a judge ought to trust the system that yields such a prediction. Who is responsible for which decision in the context of automatization and supported decision making? Is it sufficient to keep a human in the loop in order to prevent responsibility gaps?  

All these aspects can be understood as closely related to some kind of explainability of the relevant intelligent system. This leads us to another question: What exactly is it that explanations have to deliver in particular situations in order to respect stakeholders’ rights to contest, to enable informed consent, or to ground justified trust?

XAI further gives rise to questions traditionally discussed in the context of philosophy of science, action theory, and epistemology (i.e. central areas of Theoretical Philosophy). To begin with, everybody is talking about ‘explanations’, but there is little reflection in the field on what precisely explanations are, and how they provide understanding. Philosophy of science can provide the remedy: It has been concerned with the nature of explanations for a long time, and our project will bring its insights to XAI. Discussions about how to individuate an explanandum phenomenon and the pragmatics of explanations will help us illuminate the context-sensitivity of XAI; especially why different levels or kinds of explanations may be required in different circumstances and for different addressees, but also where to draw the boundaries of explanandum and explanans in different cases. A slightly different perspective comes from the philosophy of action. Humans naturally explain the actions of others by appeal to the reasons they have for so acting. We will investigate whether reason explanations can be transferred to intelligent systems and explain their behavior. 

Moreover, XAI gives rise to important epistemological questions which the project will address. For instance, it seems natural to think of intelligent systems which provide recommendations as a kind of ‘artificial experts’, who provide us with expert testimony. But how far does this analogy go? Should we hold intelligent systems to the same standards as human experts? Can we trust and hold them responsible for their utterances in the same way?

Psychology examines human feelings, thoughts, and behavior, all of which are influenced by intelligent systems. For an efficient and effective application of intelligent systems, trust in such systems is crucial. If a system provides a recommendation or if it helps to fulfill work tasks, adequately examining the trustworthiness of a system, and thus also building appropriate trust in a system, requires comprehensibility of its processes and recommendations. Furthermore, non-explainable systems might lead to problems with the attribution of responsibility, as humans cannot make an informed decision about following or rejecting the recommendation without an explanation. Within the EIS project, the Department of Industrial and Organizational Psychology therefore examines – among other things – (a) the relation of explainability and trust, (b) the adequate presentation of explanations in order to achieve comprehensibility and assessment of trustworthiness of systems, (c) the boundary conditions regarding calls for explainability and (d) the relation between explainability and attribution of responsibility. All of these questions will be examined using qualitative and quantitative empirical research.

The research progress in Computer Science not only dictates which kinds of intelligent systems are available, it also has to investigate whether these systems can be made explainable at all and if so, to what degree. So, while the other research fields are needed to determine which kind of explainability would be required, e.g. for responsible decision-making, computer science is in the position to declare whether a requirement is realizable at all and if so, how this can be achieved. The computer scientists involved in EIS will contribute an overview over – and evaluation of – interpretability techniques that are already available, but also develop a novel method for interpreting intelligent systems, taking into account input from Philosophy, Psychology, and Law. Further questions to be addressed include these: Does the achievement of explainability necessitate a drop in predictive power, efficiency, or accuracy? If so, how great a drop are we willing to accept in light of the gain of explainability? Further, is it possible to provide accurate, model-specific explanations even for systems that are black boxes for principled reasons? Alternatively, can model-agnostic explanations be reliable, accurate, and thus adequate to meet society’s demands? If not, what are the best approximations we can hope for?

Intelligent systems raise numerous legal questions: On what legal basis are courts, public authorities, or private bodies allowed to use such systems? For whom and to what extent need the decisions of intelligent systems be explainable? How does the (legal) context of a machine-supported decision influence the requirements for the explanations to be provided by the system? How to make sure that algorithms operate without discriminating against those affected by their outputs?

From a legal perspective, duties of explanation already exist in similar contexts, but most of them do not explicitly take into account the increasing use of intelligent systems as partial replacement for human experts. While there are many legal instruments for ensuring that a (human-made) decision can be understood by its addressees, it is as yet unexamined which of these instruments might be appropriate to ensure that machine-made decisions can be understood or at least be interpreted by qualified users.

The legal part of the EIS project will examine (a) which legal requirements already exist for decisions that might be supported by intelligent systems, (b) the impact of the legal context of a decision for machine-made decisions, (c) the existing legal instruments to ensure explainability of a decision, (d) how the legal instruments can be used to ensure the explainability of a machine-supported decision.

To develop an adequate picture of the explanations that are needed to meet society’s demands – and to develop a proposal of how the corresponding explainability might be implemented in a system – the contributions of the four disciplines need to be brought together. To gain a full understanding of the societal demands – such as a demand for reasonable trust or responsible decision making – an ethical perspective needs to be combined with a legal one, but also with a (psychological) grasp of how explanations influence trust and responsibility ascription. To gain a view of how explanations and understanding can be applied to intelligent systems in a relevant context, theorizing from the philosophy of science and from action theory needs to be combined with psychological insights into how understanding is generated by explanation. But moreover, computer scientific expertise is needed with respect to how explanations of a system’s behavior can be generated. Finally, any proposal regarding how explainability might actually be implemented in a system needs to be tested: Does this kind of explainability help generate understanding, trust, responsible decisions? Does it meet societal demands? In a nutshell, EIS can only succeed as a deeply interdisciplinary project.


Where Do We Go From Here?

Since March 2019, EIS has been operating on a generous seed fund provided by the Volkswagen Foundation’s “AI and the Society of the Future” funding track. Since then, we have established and achieved a lot. We have built a strong core team, with at least one Post-Doc or PhD student from each field and a wider team of associated EIS members, four further PhD students, an additional PI from the Philosophy of Science and a number of Master’s students. We have established weekly meetings and, most importantly, a highly productive interdisciplinary colloquium targeted at finding a common language. After all, terms like “explanation”, “justification”, “cause”, “accountability”, “trust(worthiness)”, “responsibility”, “guilt”, “autonomy”, “fairness”, “rights” play a role in the four participating disciplines, even though typically with slightly different interpretations of these terms. We have thereby achieved a deeper understanding on all sides.

As a result, we have already been able to do some serious research. We have contributed papers to a number of conferences, both interdisciplinary and discipline-specific. Some of our first research results have been accepted for publication in proceedings. We have visited the Leverhulme Center for the Future of Intelligence in Cambridge and set up a cooperation with it as well as with the TU Delft. We have brought aboard the German Research Center for Artificial Intelligence (DFKI) and the new collaborative research center on the Foundations of Perspicuous Software Systems as collaborators. On top of this, we have delivered a number of talks in the spirit of science communication to the interested public, from doctors and entrepreneurs to politicians and the general public

In the next couple of months, we will present panels at a number of further conferences and present some of our research at others. At the end of our funding phase, we are organizing an interdisciplinary workshop on “Issues in Explainable AI: Black Boxes, Recommendations, and Levels of Explanation”, where we will host a number of national and international guests. The workshop will be accompanied by a talk and panel discussion addressed at the general public, entitled “Explainable Intelligent Systems: Understanding. Responsibility. Trust”. The seed funding phase of EIS will culminate in the submission of our application to the Volkswagen Foundation’s full grant track in October 2019.

We have started the project with the certain conviction that XAI is not only an important, but also an interesting and rewarding research topic – and as it turns out, we were right. Everyone in our team is passionate about XAI, EIS and, in general, about the opportunity to help shape the beneficial development and application of AI and the society of the future. We are optimistic about the future and we strongly believe that the real journey of EIS has only just begun. 


Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top