Deep learning has enabled machines to be used for many tasks, including computer vision, image recognition, natural language processing, self-driving cars, and predicting the contingencies of certain business transactions. Effective application of deep learning techniques can help develop intelligent applications. Many web-based tools get their information from generic search engines such as Google but also specialized search engines such as PubMed (for retrieving information on biomedical publications). In this study, we evaluate the performance of different deep learning architectures for the task of document retrieval. Our goal was to identify the optimal architecture for the document retrieval task. We therefore compare the deep architecture of feed-forward neural networks with recurrent neural networks, in particular the long short-term memory neural networks. We observe that the local context is not sufficient as it does not provide reliable predictions, which are, however, provided by long distance dependencies where long short-term memory neural networks are used. We first describe the algorithms used for our evaluation and then compare their performance. Our dataset contained two publicly available document sets from the 20-Newsgroup and BBC, respectively.