In pursuit of interpreting black-box models such as deep image classifiers, a number of techniques have been developed that attribute and visualize the importance of input features with respect to the output of a model. ExplainableAI.jl brings several of these methods to Julia, building on top of primitives from the Flux ecosystem. In this talk, we will give an overview of current features and show how the package can easily be extended, allowing users to implement their own methods and rules.