in recent years, there has been a lot of focus on explaining deep neural networks. In the computer vision domain, one of the most important explanation techniques is the heatmap approach [9, 5, 10], which focuses on generating heatmaps that highlight parts of the input image that are most important to the decision of the deep networks on a particular classification target.