MIPRO 2018, May 21-25, 2018, Opatija Croatia

Explainable Artificial Intelligence: A Survey

Filip Karlo Dosilovi6*, Mario Brci6** and Nikica Hlupi6**

Abstract - In the last decade, with availability of large more unfair wealth distribution. Content recommendation
datasets and more com puting power, machine learning [4] and generation of fake content [5] coupled with other
systems have achieved (super)human performance in a wide technologies will deeply impact the social dynamics [6].
variety of tasks. Examples of this rapid development can be Many things will be prescribed by such algorithms and
seen in image recognition, speech analysis, strategic game that will affect human lives in ways maybe now
planning and many more. The problem with many state-of- unimagined so people will need to trust them in order to
the-art models is a lack of transparency and interpretability. accept those prescriptions. Systems must, like humans,
The lack of thereof is a major drawback in many satisfY many criteria (assurances) in order to boost trust
applications, e.g. health care and finance, where rationale for
[2]: unbiasedness (fairness), reliability, safety, explanatory
model's decision is a requirement for trust. In the light of
justifiability, privacy, usability etc. Among humans such
these issues, explainable artificial intelligence (XAI) has
become an area of interest in research community. This
assurances are assumed due to bias towards human
paper summarizes recent developments in XAI in decision making, since people are social creatures
supervised learning, starts a discussion on its connection accustomed to life in human communities. Artificial
with artificial general intelligence, and gives proposals for intelligence, due to its novel status in our lives, as well as
further research directions. being of the human making, causes much skepticism -
rightfully so. Deep learning, as one of the most successful
Keywords explainable artificial intelligence; machine learning approaches in supervised learning has
interpretability; explainability; comprehensibility been criticized in [7] for working well as approximations
where answers often not to be fully trusted, pointing out to
I. INTRODUCTION model vulnerabilities in language and vision models.
Spoofability and biasedness have been demonstrated for
In the last decade, especially since 2012, artificial visual recognition in [8], [9] and natural language
intelligence (AI) and machine learning (ML) systems have processing [10], [11]. No robust solution has been found
achieved (super)human performance in many tasks that to these problems so far. The potential ethical pitfalls
were previously thought to be computationally should be addressed as soon as possible since inactivity
unattainable [1]. Advances in the field were achieved due could lead to unforeseeable splits and differences in the
to the rise of available information, and major hardware future society. European Union introduced a right to
improvements combined with new optimization explanation in General Data Protection Regulation
algorithms. We can also attribute these advancements to (GDPR) [12] as an attempt to remedy the potential
high-quality open-source libraries which allowed problems with the rising importance of algorithms. Since
developers and researchers to quickly code and test the aforementioned trust criteria are hard to formalize and
models. Improvements in speech recognition, image quantifY, usually criteria of interpretability and
classification, object detection, classical (board) games, explainability are used as intermediate goals. Afterwards
Texas Hol'em, and many more have led to their in the following stage, system's explanations can be
proliferation and percolation to the real-world applications checked if they satisfY other desirable trust criteria.
outside research labs, mostly in the area of supervised
learning. More generally, abstracted explanations can be utilized
for finding useful properties and generating hypotheses
We also saw advances in application critical areas, e.g. about data-generating processes, such as causal
medicine, [mance, self-driving cars, government, relationships - which is crucial application in science as
bioinformatics, churn prediction and content well as in future Artificial General Intelligence (AGI).
recommendation but these applications also brought Generated hypotheses can be basis for further automated
attention to the crucial trust-related problems. Future or manual experimentation, knowledge discovery, and
applications will, in addition to extensions in optimization. This view is supported by [13] and
aforementioned areas, also include cognitive assistance, encompasses: checking for satisfaction of trust-criteria,
interpretable science and reliable ML [2], [3] The all- optimization of ethical outcomes due to technology,
pervasive utilization of these systems will significantly assisted(automated) scientific discovery, transferring
transform the social landscape of the world. These skills, etc. mentioned in[3], [14]-[16]. Previous
changes include many ethical challenges to which society overviews and surveys of interpretability in machine
will have to adapt fast in order to steer the developments learning are given in [2], [3], [14], [15], [17]-[19].
to the favorable directions and outcomes. Automation will
significantly change the job market [2], which may lead to

210
 In this paper we survey the advances in the III. METHODS FOR INTERPRETABILITY AND
interpretability and explainability of machine learning EXPLAINABILITY
models under the supervised learning paradigm. Much of
There are two categories of approaches to
the recent work is in the area of deep learning, due to
interpretability and explainability: integrated
remarkable performance gains of these models on the one
(transparency-based) and post-hoc.
hand and intrinsic opaqueness on other hand. The paper is
organized as follows: in section 2 we deal with the Transparency [3] is one of the properties that can
preliminaries and definitions. In section 3 we categorize enable interpretability. Transparency was a traditional first
the work in methods for interpretability. Section 4 offers step to protection of rights in human-based institutions
discussion into the current state of the research field and and by analogy it is ported to algorithmic concerns such as
lists future research ideas. Section 5 concludes the paper unfairness and discrimination [20]. But, models in AI are
becoming much more complex than human-based
II. PRELIMINARIES AND DEFINITIONS institutions and it becomes hard to find meaningful
explanation that users might be able to understand. Also,
In this section we offer defmitions of: trust, human thinking, including our own, is not transparent to
interpretability, comprehensibility, and explainability. us and justifications in the form of explanations and
Trust is defmed in [2] as a psychological state in interpretations may differ from the actual decision
which an agent willingly and securely becomes mechanism. In addition, predictive performance and
vulnerable, or depends on, a trustee, having taken into transparency are conflicting objectives and they have to be
consideration the characteristics of the trustee. traded-off in a model [21], [22]. In [7] it is stated that it is
not clear how much transparency matters in the long run.
Authors in [15] claim that unlike normal ML objective If the systems are robust and self-contained it may not be
functions, it is hard to formalize the defmitions of criteria necessary. But, if they are part of other systems, then
that are crucial for trust and acceptance, view backed by transparency can be good for debuggability.
[2], [3]. In those cases of incomplete problem
formalization, interpretability is used as a fallback or Post-hoc interpretability extracts informations from
proxy for other criteria. already learned model and it does not precisely depend on
how the model works. The advantage of this approach is
However, there is no unique defmition of that it does not impact performance of the model which is
interpretability [3], [15]. In [3] interpretability is found not treated as a black-box (BB). This is similar mode to how
to be a monolithic concept, but in fact it reflects several people make justifications for their own decisions, without
distinct ideas and that in many papers interpretability is fully knowing the real functioning of their decision-
proclaimed axiomatically. Authors in [15] define that to making mechanisms. However, special care must be taken
interpret means to explain or to present in understandable in order to avoid systems that generate plausible but
terms. Then, interpretability in the context of ML systems misleading explanations. Such explanations could satisfy
is the ability to explain or to present in understandable laws like GDPR, but there is a problem of checking their
terms to humans. veracity.
Interpretability and explainability are often used
interchangeably in literature, but some papers make
distinction. In [17] interpretation is the mapping of Tree based models
abstract concept into a domain humans can make sense of, Rule based models
while explanation is the collection of features of Linear models
interpretable domain that have contributed for a given
example to produce a decision. Edwards and Veale in
[20] split explanations into model-centric and subject-
centric, notions which correspond to definitions of
interpretability and explainability from [17]. Similar roles Intrinsically linear model
in [15] take up global and local interpretability,
respectively. In that view, we can see that GDPR covers
only explainability. Comprehensibility [14] is used in a
literature as a synonym for interpretability. Transparency
[3] is used as a synonym for model interpretability, that is /SVM
some sense of understanding the working logic of the
model.
None of the aforementioned definitions is specific or Figure I Performance-transparency trade-off [I 8]
restrictive enough to enable formalization. They implicitly
depend on user's expertise, preferences and other
contextual variables. A. Integrated intepretability
The best explanation of a simple model is the model
itself; it perfectly represents itself and is easy to
understand [23]. This approach is limited to the model
families with lower complexity (flexibility), such as linear
models, decision trees and rules. On the other hand, other

211
model families such as artificial neural networks (ANN) B. Post-hoc methods
and support vector machines (SVM), boosted trees, and With hardware improvements and increased
random forests are considered opaque and their availability of data, predictive performance benefits of
complexity prevents users from tracing the logic behind using complex, opaque models are increased. However,
predictions. The latter are most often considered as black- interpretability and explainability are issues that have to
boxes and they are dealt with in a post-hoc manner. This be properly addressed. In these approaches we start with
trade-off between transparency and performance is trained black-box predictor and, sometimes, the used
conceptually depicted in Fig 1. The majority of work is training data. Some methods deal with interpretability
done for classification tasks. Different constraints can be while others with explainability, according to the
imposed on models in order to increase their definitions in section 2. Methods are model-agnostic if
interpretability. Model size, sparsity, and monotonicity they work only with the inputs and outputs of BB model,
([14], [18]) are some of the constraints used in the and model-specific if they use idiosyncrasies of some
literature. Even the choice of model family representation.
(representation) can be considered a constraint on models
that affects the interpretability. Transparent models are I) Interpretability
both interpretable and explainable. Transparent proxy model approach fmds interpretable
model that globally approximates the predictions of the
User-based studies on interpretability of decision- black-box model. This approach offers both
rules tables and tree algorithms in classification were interpretability and explainability. In [32] a model-specific
cond~cted ~ [24] and [25]. In the latter paper, decision method was used to learn single decision tree from the
tables were found to be the easiest to use for ensemble of decision trees. The learned model was more
inexperienced users and that the model size in general has accurate than the decision tree learned directly from the
negative impact on interpretability, answer time and data. Rules were extracted from SVM in [33] to make
confidence. Freitas in [14] gives an overview of work in more interpretable model for credit scoring.
comprehensible classification models where transparency Interpretability was gained at only a small loss in
plays the major role. Interpretability of decision trees, performance compared to SVMs. Symbolic rules were
classification rules, decision tables, nearest neighbors, and extracted from neural network ensembles in [34].
Bayesian network classifiers is discussed and Bayesian regression mixture with multiple elastic nets was
monotonicity constraints are advocated for improving the proposed [35] and used on DNN, SVM, and random
model transparency. forests to explain individual decisions and look for model
There are two sub-approaches: pure transparent and vulnerabilities in image recognition and text classification.
hybrid. Model vulnerabilities were tested with adversarial
examples. Adversarial training scheme was used on DNNs
In pure transparent approaches we are restricted to use in order to increase their interpretability [36]. Using
model families that are considered transparent. adversarial examples, it was found that in normally trained
Evolutionary programming was used in [26] to search for DNNs neurons do not detect semantic parts but only
sets of interpretable classification rules with small number discriminative part patches. Also, representations are not
of rules and conditions. Interpretable decision sets [27] are robust codes of visual concepts. After adversarial training
sets of independent if-then rules. Since each rule can be scheme, representation is more interpretable, enabling to
applied independently, interpretation is simple. The model trace the outcomes to influential neurons. This is more
is found by optimizing objective that takes into account transparent way of making predictions.
both accuracy and interpretability. Oblique treed sparse
additive models, region-specific predictive models, were Indicative approaches give weaker notion of
proposed in [28]. They achieved competitive performance interpretability to BB model than the above methods.
with kernel SVMs while providing interpretable model. Some aspects of model functioning are elicited with
Prototype selection [29] uses set covering optimization conceptual representations. Visualization techniques for
problem to achieve sparsity in samples for classification. high-dimensional data like [37] can be used. In [38],
Minimal set of prototypes is selected in order to get good visualization technique based on trained deconvolutional
nearest neighbor classifier. It was tested on recognizing network was created for visualization of intermediate
handwritten digits and it showed reasonable performance. layers of convolutional neural networks. The insights
through these visualizations enabled creation of improved
Hybrid approaches combine transparent model architectures that outperformed existing approaches at the
families with black-box methods in order to get time. Similar was done for recurrent neural networks in
appropriate trade-off between the model interpretability [39], where character-level language models were used as
and the predictive performance. Combination of logistic an interpretable benchmark. Experiments revealed the
regression and SVMs was used for credit scoring in [30] existence of interpretable cells that keep track of long-
in order to improve accuracy of the initial interpretable range dependencies in text. Further work was done in [40]
model. Multi-objective learning of hybrid classifiers was on interpreting neural models in natural language
utilized in [31] to learn hybrid trees where certain leafs processing through visualization of unit's salience.
were substituted with black-box classifiers for boosting
accuracy at the expense of interpretabil ity. Model-agnostic visualization method based on
sensitivity analysis was proposed in [41 ].lnput effects on
model responses are inferred and visualizations of barplots
for feature importance and Variable Effect Characteristic
curves. The experiments were performed on neural

212
network ensemble and SVM models. Model-agnostic enable easier transfer of results and information. Factors
method for auditing indirect influences in BB models was of trust-inducing criteria should be formalized, since these
presented in [42]. The procedure finds indirect influences concepts are ambiguous and need to be split into smaller,
of attributes to output through related features, even when more specific constituents. Model framework of human
attributes don't have the direct influence. Tamagnini et al. trust should be found, as an extension of work in [2]. This
[43] created pedagogical visualization-based system can be done in a similar data-driven approach as proposed
Rivelo for interpretation of BB classifiers using instance- in [15] for interpretability. There is a lack of empirical
level explanations. User can interactively explore the set studies on user-based measures of interpretability. Also,
of instance-level explanations to create a mental model. richer loss functions need to be developed that take into
account more criteria of performance in the real world.
2) Explainability Most of the machine learning work has revolved around
These methods mostly output for each prediction also the scalar objectives, while the problems we are talking
an explanation in the form of feature importances for about are multicriteria with some criteria not even
making that decision. Layer-wise relevance propagation explicitly given in the form of optimization objective.
and sensitivity analysis were presented in [16] to explain Research into the extensions of interpretability research to
predictions for deep learning models in the terms of input reinforcement learning is another potential venue.
variables. In [44], deep Taylor decomposition propagates Pedagogical interactive post-hoc approaches such as [43]
explanations from the outputs ofDNN to the contributions are promising to enable people create their mental models
of its inputs. Model-agnostic method for capturing the of complex algorithms [20] which can boost trust by
degree of influence of inputs on outputs of the system was increasing the familiarity with models' decisions.
presented in [45]. Local approximation method, SHapley
Additive exPlanations (SHAP), was used in [23] to
explain prediction f(x) for a single input x. SHAP is a V. CONCLUSION
unified framework for value estimation of additive feature There are evident problems with the ethical- as well as
attributions that generalizes several works from the quality-of-life implications of using AI in their current
literature [45]-[49]. Both model-agnostic and more form in real-world scenarios. On several fronts, people are
efficient model-specific variant were proposed. DNN that yet to see major impact applications such as in judicial,
identifies contents in the image and generates caption was governmental, fmancial, and autonomous transport. But,
described in [50]. For each word in caption, an for some time already, human lives have been influenced
explanation is generated in a form of highlighted relevant by algorithmic content recommendation which shapes
regions of the input image. opinions and tastes. With a greater spread of AI
applications, trust-related problems are likely to become
Other methods give explanations in other forms, such
more pressing issues. Trust is boosted with specific
as visualization, text, examples, etc. DNNs were trained
criteria, but there are prominent problems with the
for visual question answering and explaining human
incompleteness in problem formalization. This is a barrier
activities in [51]. Justifications for decisions were given
to straightforward optimization approaches - some
textually and evidence in images was emphasized using
notions are so complex, multidimensional, and ambiguous
the attention mechanism. Textual explanations were
that they are hard to put down in a formal way.
generated together with visual classification using DNN
Interpretability and explainability offer abstracted
[52]. Reinforcement learning based discriminative loss
explanations for finding, checking, and reasoning over
function was used for explanation model.
useful properties. This can be used not only for verifying
trust criteria, but also for scientific discovery and in future
IV. DISCUSSION AGI systems.
In the introduction we mentioned the utility of
abstracted explanations for finding useful properties and REFERENCES
generating hypotheses about data-generating processes
[1] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature,
which is important for science as well as for future vol. 521, no. 7553, pp. 436-444, May 2015.
Artificial General Intelligence (AGI) systems. Generated [2] B. W. Israel sen, "'1 can assure you ... that it's going to be all right'
hypotheses can be further reasoned and experimented -- A definition, case for, and survey of algorithmic assurances in
with, leading to the bootstrapped process of iterative human-autonomy trust relationships," ArXiv170800495 Cs Stat,
improvement. However, as the authors in [19] point out, Aug. 2017.
approaches listed in this paper only enable explanations of [3] Z. C. Lipton, "The Mythos of Model Interpretability,"
ArXiv 160603490 Cs Stat, Jun. 2016.
decisions, instead of actually generating them. It is up to
the user to do the hypothesis generation, experimentation [4] T. T. Nguyen, P.-M. Hui, F. M. Harper, L. Terveen, and 1. A.
Konstan, "Exploring the Filter Bubble: The Effect of Using
and reasoning. Research direction interesting for science Recommender Systems on Content Diversity," in Proceedings of
and AGI is bridging neural-symbolic gap for seamless the 23rd International Conference on World Wide Web, New
integration of learning and reasoning [53], [54]. York, NY, USA, 2014, pp. 677-686.
Successful implementation would enable automatic [5] R. Kumar, J. Sotelo, K. Kumar, A. de Brebisson, and Y. Bengio,
generation of interpretations, explanations, and reasoning "ObamaNet: Photo-realistic lip-sync from text," ArXivI80IOI442
over them. Cs, Dec. 2017.
[6] A. Kucharski, "Study epidemiology of fake news," Nature, vol.
We have also seen that researchers use different names 540, p. 525, Dec. 2016.
for similar or identical concepts. Definitions and [7] G. Marcus, "Deep Learning: A Critical Appraisal,"
vocabulary should be fixed in the community in order to ArXivi 80 10063 1 Cs Stat, Jan. 2018.

213
[8] C. Szegedy et aI., "Intriguing properties of neural networks," [29] J. Bien and R. Tibshirani, "Prototype selection for interpretable
presented at the International Conference on Learning classification," Ann. Appl. Stat., vol. 5, no. 4, pp. 2403-2424,
Representations, 2014, pp. 1-10. Dec. 201 I.
[9] K. Crawford, "Opinion I Artificial Intelligence's White Guy [30] T. Van Gestel, B. Baesens, P. Van Dijcke, J. Suykens, J. Garcia,
Problem," The New York Times, 25-Jun-2016. and T. Alderweireld, "Linear and Non-linear Credit Scoring by
[10] T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Combining Logistic Regression and Support Vector Machines," J.
Kalai, "Man is to Computer Programmer as Woman is to Credit Risk, vol. I, Jan. 2005.
Homemaker? Debiasing Word Embeddings," in Advances in [3 I] R. Piltaver, M. Lustrek, and M. Gams, "Multi-objective learning
Neural Information Processing Systems 29, Barcelona, Spain, of accurate and comprehensible classifiers - a case study," 2014.
2016,pp.4349-4357. [32] A. V. Assche and H. Blockeel, "Seeing the Forest Through the
[II] M. J. Wolf, K. Miller, and F. S. Grodzinsky, "Why We Should Trees: Learning a Comprehensible Model from an Ensemble," in
Have Seen That Coming: Comments on Microsoft's Tay Machine Learning: ECML 2007, 2007, pp. 418-429.
'Experiment,' and Wider Implications," SIGCAS Comput Soc, [33] D. Martens, B. Baesens, T. Van Gestel, and J. Vanthienen,
vol. 47, no. 3, pp. 54-64, Sep. 2017. "Comprehensible credit scoring models using rule extraction from
[12] B. Goodman and S. Flaxman, "European Union regulations on support vector machines," Eur. J. Oper. Res., vol. 183, no. 3, pp.
algorithmic decision-making and a 'right to explanation,''' 1466-1476, Dec. 2007.
ArXiv 160608813 Cs Stat, Jun. 2016. [34] Z.-H. Zhou, Y. Jiang, and S.-F. Chen, "Extracting symbolic rules
[13] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. 1. Gershman, from trained neural network ensembles," Al Commun., vol. 16,
"Building machines that learn and think like people," Behav. no. I, pp. 3-15, Jan. 2003.
Brain Sci., vol. 40, ed 2017. [35] W. Guo, K. Zhang, L. Lin, S. Huang, and X. Xing, "Towards
[14] A. A. Freitas, "Comprehensible Classification Models: A Position Interrogating Discriminative Machine Learning Models,"
Paper," SIGKDD Explor Newsl, vol. 15, no. I, pp. 1-10, Mar. ArXiv 170508564 Cs Stat, May 2017.
2014. [36] Y. Dong, H. Su, 1. Zhu, and F. Bao, "Towards Interpretable Deep
[15] F. Doshi-Velez and B. Kim, "Towards A Rigorous Science of Neural Networks by Leveraging Adversarial Examples,"
Interpretable Machine Learning," ArXiv170208608 Cs Stat, Feb. ArXiv170805493 Cs, Aug. 2017.
2017. [37] L. van der Maaten and G. Hinton, "Visualizing Data using t-
[16] W. Samek, T. Wiegand, and K.-R. Muller, "Explainable Artificial SNE," 1. Mach. Learn. Res., vol. 9, no. Nov, pp. 2579-2605,
Intelligence: Understanding, Visualizing and Interpreting Deep 2008.
Learning Models," ITU J. ICT Discov. - Spec. Issue I - Impact [38] M. D. Zeiler and R. Fergus, "Visualizing and Understanding
Artif. Intell. AI Commun. Netw. Serv., vol. I, pp. 1-10, Dec. Convolutional Networks," in Computer Vision - ECCV 2014,
2017. 2014,pp.818-833.
[17] G. Montavon, W. Samek, and K.-R. Muller, "Methods for [39] A. Karpathy, 1. Johnson, and L. Fei-Fei, "Visualizing and
interpreting and understanding deep neural networks," Digit. Understanding Recurrent Networks," ArXivl50602078 Cs, Jun.
Signal Process., vol. 73, pp. 1-15, Feb. 2018. 2015.
[18] D. Martens, 1. Vanthienen, W. Verbeke, and B. Baesens, [40] J. Li, X. Chen, E. Hovy, and D. Jurafsky, "Visualizing and
"Performance of classification models from a user perspective," Understanding Neural Models in NLP," in Proceedings of the
Decis. Support Syst., vol. 5 I, no. 4, pp. 782-793, Nov. 20 I I. 20 I6 Conference of the North American Chapter of the
[19] D. Doran, S. Schulz, and T. R. Besold, "What Does Explainable Association for Computational Linguistics: Human Language
AI Really Mean? A New Conceptualization of Perspectives," Technologies, 2016, pp. 681-691.
ArXiv 171 000794 Cs, Oct. 2017. [41] P. Cortez and M. J. Embrechts, "Using sensitivity analysis and
[20] L. Edwards and M. Veale, "Slave to the Algorithm? Why a 'Right visualization techniques to open black box data mining models,"
to an Explanation' Is Probably Not the Remedy You Are Looking Inf Sci., vol. 225, pp. 1-17, Mar. 2013.
For," Social Science Research Network, Rochester, NY, SSRN [42] P. Adler et aI., "Auditing Black-box Models for Indirect
Scholarly Paper ID 2972855, May 20 I 7. Influence," Knowl InfSyst, vol. 54, no. I, pp. 95-122, Jan. 2018.
[21] Y. Jin and B. Sendhoff, "Pareto-Based Multiobjective Machine [43] P. Tamagnini, J. Krause, A. Dasgupta, and E. Bertini,
Learning: An Overview and Case Studies," IEEE Trans. Syst. "Interpreting Black-Box Classifiers Using Instance-Level Visual
Man Cybern. Part C Appl. Rev., vol. 38, no. 3, pp. 397-415, May Explanations," in Proceedings of the 2Nd Workshop on Human-
2008. In-the-Loop Data Analytics, New York, NY, USA, 2017, p. 6:1-
[22] A. A. Freitas, "A Critical Review of Multi-objective Optimization 6:6.
in Data Mining: A Position Paper," SIGKDD Explor Newsl, vol. [44] G. Montavon, S. Lapuschkin, A. Binder, W. Samek, and K.-R.
6, no. 2, pp. 77-86, Dec. 2004. Muller, "Explaining nonlinear classification decisions with deep
[23] S. Lundberg and S.-I. Lee, "A Unified Approach to Interpreting Taylor decomposition," Pattern Recognit., vol. 65, pp. 21 1-222,
Model Predictions," ArXivl 70507874 Cs Stat, May 2017. May 2017.
[24] H. Allahyari and N. Lavesson, "User-oriented Assessment of [45] A. Datta, S. Sen, and Y. Zick, "Algorithmic Transparency via
Classification Model Understandability," in DI VA, 20 I I. Quantitative Input Influence: Theory and Experiments with
[25] J. Huysmans, K. Dejaeger, C. Mues, 1. Vanthienen, and B. Learning Systems," in 2016 IEEE Symposium on Security and
Baesens, "An empirical evaluation of the comprehensibility of Privacy (SP), 2016, pp. 598-617.
decision table, tree and rule based predictive models," Decis. [46] M. T. Ribeiro, S. Singh, and C. Guestrin, '''Why Should I Trust
Support Syst., vol. 51, no. I, pp. 141-154, Apr. 20 I I. You?': Explaining the Predictions of Any Classifier," in
[26] A. Cano, A. Zafra, and S. Ventura, "An interpretable classification Proceedings of the 22Nd ACM SIGKDD International Conference
rule mining algorithm," Inf. Sci., vol. 240, pp. 1-20, Aug. 2013. on Knowledge Discovery and Data Mining, New York, NY, USA,
2016,pp.1135-1144.
[27] H. Lakkaraju, S. H. Bach, and J. Leskovec, "Interpretable
Decision Sets: A Joint Framework for Description and [47] A. Shrikumar, P. Greenside, and A. Kundaje, "Learning Important
Prediction," in Proceedings of the 22Nd ACM SIGKDD Features Through Propagating Activation Differences,"
International Conference on Knowledge Discovery and Data ArXivl70402685 Cs, Apr. 2017.
Mining, New York, NY, USA, 2016, pp. 1675-1684. [48] E. Strumbelj and I. Kononenko, "Explaining Prediction Models
[28] J. Wang, R. Fujimaki, and Y. Motohashi, "Trading Interpretability and Individual Predictions with Feature Contributions," Knowllnf
for Accuracy: Oblique Treed Sparse Additive Models," in Syst, vol. 41, no. 3, pp. 647-665, Dec. 2014.
Proceedings of the 21th ACM SIGKDD International Conference [49] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Muller, and
on Knowledge Discovery and Data Mining, New York, NY, USA, W. Samek, "On Pixel-Wise Explanations for Non-Linear
2015, pp. 1245-1254. Classifier Decisions by Layer-Wise Relevance Propagation,"
PLOS ONE, vol. 10, no. 7, p. e0130140, Jul. 2015.

214
[50] K. Xu et aI., "Show, Attend and Tell: Neural Image Caption
Generation with Visual Attention," in International Conference on
Machine Learning, 2015, pp. 2048-2057.
[51] D. H. Park, L. A. Hendricks, Z. Akata, B. Schiele, T. Darrell, and
M. Rohrbach, "Attentive Explanations: JustifYing Decisions and
Pointing to the Evidence," ArXivl61204757 Cs, Dec. 2016.
[52] L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele,
and T. Darrell, "Generating Visual Explanations," in Computer
Vision - ECCV 2016, 2016, pp. 3-19.
[53] T. R. Besold and K.-U. Kiihnberger, "Towards integrated neural-
symbolic systems for human-level AJ: Two research programs
helping to bridge the gaps," BioI. Inspired Cogn. Archil., vol. 14,
pp. 97-110, Ocl. 2015.
[54] T. R. Besold et aI., "Neural-Symbolic Learning and Reasoning: A
Survey and Interpretation," ArXivl71103902 Cs, Nov. 2017.

215