Our contributions of this paper are two-fold. First, wepresent a method to predict the similarity of an image pair based on deep neural network using whole-image similarity measure; second, we apply our approach on unseen data to examine the generalization characteristics, showing that it out- performs a state-of-the-art CNN trained for the classification task.