Representation Learning for Image Retrieval through 3D CNN and Manifold Ranking

Abstract

Despite of the substantial success of Convolutional Neural Networks (CNNs) on many recognition and representation tasks, such models are very reliant on huge amount of data to allow effective training. In order to improve the generalization ability of CNNs, several approaches have been proposed, including variations of data augmentation strategies. With the goal of achieving more effective retrieval results on unsupervised learning scenarios, we propose a representation learning approach which exploits a rank-based formulation to build a more comprehensive data representation. The proposed model uses 2D and 3D CNNs trained by transfer learning and fuse both representations through a rank-based formulation based on manifold learning algorithms. Our approach was evaluated on an unsupervised image retrieval scenario applied to action recognition datasets. The experimental results indicated that significant effectiveness gains can be obtained on various datasets, reaching +56.93% of relative gains on MAP scores.