The recent years have witnessed massive development in the fields of computer vision, object recognition, and natural language processing (NLP). Artificial Intelligence (AI) applications use NLP to provide the computer with facilities, such as question-answering models. An extension of this method is to integrate NLP with computer vision to perform the Visual Question Answering (VQA) mission, which is to construct systems that can respond to image questions in natural language. In this paper, the application of Convolutional Neural Network (CNN) for the image Question Answering (QA) task is proposed. The proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations, but also their inter-modal interactions to produce the answer. More specifically, this model consists of three CNNs: image CNN to encode the image content, sentence CNN to compose the words of the question, and multimodal convolution layer to learn their joint representation for the classification in the space of candidate answer words.
Research Article
Answer Me: Pictorial Interrogation on Categorized Queries
https://doi.org/10.21203/rs.3.rs-3086224/v1
This work is licensed under a CC BY 4.0 License
Version 1
posted
You are reading this latest preprint version
Natural Language Processing
Computer Vision
Convolutional Neural Network
Visual Question and Answering
Visual Question Answering (VQA) is an inspiring task that includes two major fields of AI namely Natural Language Processing and Computer Vision in which an image is given with a question in natural language and the task is to find the answer to that question. A VQA task is defined as answering a question about a provided image. The Visual Question Answering (VQA) task involves challenges in which the data is processed with visual as well as linguistic processing to find the answer of basic common-sense questions related to a given image. This task requires reasoning capabilities on visual features and objects of the image along with that general knowledge to predict the correct answer to the given question. VQA is considered as a field of research that develops a well-established system that helps in answering the questions regarding the given image in natural language [1-4]. It is an AI-complete problem that integrates multiple techniques like natural language processing and Computer Vision (CV). With the tremendous success of the computer vision (e.g., object detection[5]) used for image analysis and natural language processing (e.g., word representation[6]) used for language understanding, VQA models have been further developed to great extent. There are many potential applications of a VQA system e.g., assistance to blind users, in medical field, summarizing video data, human–machine interface and automatic customer service.
In the proposed approach, for given an image-question combination, the algorithm not only provides a response, but also a collection of explanations (as text) and visual attention maps. A new dataset containing open ended questions about images (need to mention how it is created) that contains open-ended image-related questions is used. The new VQA data set used is larger than the COCO dataset. The dataset includes 614,163 questions and 7,984,119 answers for 204,721 images from the COCO dataset. Given the visual complexity of these images, they are well-suited for our VQA task. The more diverse collection of images, the more diverse, comprehensive, and interesting the resultant set of questions and their answers. To answer these questions, a strong grasp of vision, language, and common sense is required. VQA models can reason over unstructured external knowledge sources (in this example, text detected in a test picture) and tolerating several sources of the flow of information.
In short, the contributions of the proposed VQA model are: (need to mention few):
- Introduce the task of free-form and open-ended Visual Question Answering (VQA). We collect a large-scale dataset and make it available for VQA
- A long short-term memory convolution neural networks for answering the questions is proposed.
- An image-region selection mechanism that learns to identify image regions relevant to questions and then compare it with MLIM model which uses COCO dataset is presented.
The remaining sections of the manuscript are structured as follows: the second section of this paper includes a detailed survey of related papers, detailed evaluation of VQA methodologies, which are classified into four groups based on the nature of their primary contribution. In the next part of investigation, datasets that train and evaluate VQA systems [2] and the methodology is studied. The experimental details are given in the fourth section. The fifth section discusses about the results. The last section is about the conclusions and future enhancements.
Visual question answering is a technique that unifies multiple modern techniques like natural language processing (NLP) and machine vision to improve research and get better result out of all fields. Computer vision or machine vision is a technique used for collecting, processing, analyzing the images that are to be used in the system [7] [8]. It seeks to teach machines how to sight in a nutshell. NLP, on the other hand, is a field focused with facilitating natural language exchanges between computers and people, which includes teaching robots to read. Artificial intelligence encompasses both natural language processing (NLP) and machine vision, and both use machine learning to achieve their goals [8].
According to survey, recent trends in Visual Question Answering, the system is given with an image and a question in a Natural language, as inputs. Then the relationship between image and the question is processed thoroughly. Then, the system will analyze through potential answers which may be of different types like multiple choice (A/B/C), Yes/No type of questions. Fill in the blank is a similar activity in which one or more missing words must be added to an affirmation characterizing the sight. These affirmations are just affirmative answers to declarative queries. Unlike traditional methods, where questions were taken at the time of execution, VQA sets apart from that category. Visual question answering may look simple but is of higher complexity where the information about the question is not necessarily present every time in the given image unlike picture captioning. The required information can be anywhere and anything from common things to complex equations regarding visual elements. This proves that Visual Question Answering is a complete AI problem as it requires multimodal information’s and not just of a single field. It should be noted that image captioning might theoretically be used to evaluate picture understanding just as efficiently. In practice, however, VQA has the advantage of having fewer review criteria. Answers are usually limited to a few words [9]. Longer real-world image captions are more difficult to match to anticipated captions. Despite the fact that sophisticated assessment measures have been investigated, this is still an unresolved research topic.
Since the emergence of the VQA dataset towards the end of 2014, there have been several active research works on the topic to date. According to Singh et al. (2019) [10], VQA methods are categorized into joint embedding methods, attention mechanisms, and compositional models. This study categorized VQA approaches into joint embedding, attention mechanism, and compositional, external knowledge, and graph-based approaches. In joint embedding approaches, text features can be extracted using bag-of-words (BOW) or long short term memory (LSTM) (Ben-Younes et al. 2017; Fukui et al. 2016; Shih et al. 2016) [11–13]. Then, CNN can be used to extract image features. The respective features are further combined into common feature space using either concatenation or element-wise multiplication methods. Finally, the combined feature vector is passed into a classifier to predict an answer to the input question. The joint embedding methods focus on the entire region of the image, which poses a challenge of understanding question specific semantic information of an image. Some of the early research in VQA concentrate on joint embedding approaches and they are considered as common practice in vision and language research communities (Ben-Younes et al. 2017; Fukui et al. 2016; Shih et al. 2016). Although these approaches are mostly used for open-end questions and multiple-choice answers, yet they can only generate answers that are observed during training.
The attention mechanism based approaches improve further by considering part of the input space (Wang et al. [14]; Shah et al. [15]). They extract the global features of an image to answer a question focusing on only a specific region of the image that might confuse the VQA system. For instance, given the question ‘what size is the book’, the salient part of the image contains the book. Also, ‘size’ and ‘book’ are the most relevant words in the sentence. The attention mechanism is guided by an algorithm that represents feature vectors corresponding to each region in an image at a more local level which are then ranked based on their similarity with the features of the question asked (Kallooriyakath et al. [16]). The global image features (like the last hidden layer of a CNN) and global text features (bag-of-words) may not be capable of addressing region specific questions. More promising results were obtained on the attention-based VQA methods using benchmark datasets. But, they still have issues with complex questions that contain reasoning and counting.
In compositional models, questions involve a series of reasoning steps to infer proper answers. For instance, a question like ‘what is beside the cup’ entails finding the cup and naming the object beside it. These approaches are mostly restricted to visual reasoning only. In these approaches, two compositional systems have been proposed in an attempt to solve VQA in a series of sub-steps. The first framework is Neural Module Network (NMN), and the second structure is Recurrent Answering Units (RAU). The NMN structure utilizes external question parsers to find the subtask in the question, whereas RAU is prepared end-to-end, and subtask can be implicitly learned (Andreas et al. [17]; Noh et al. [18]).
In more recent scenarios, the above-mentioned VQA approaches underperform on more complex questions like ‘why is the baby smiling’ and ‘can we get water here’. These types of questions require common sense reasoning and knowledge about the spatial relationship among objects. Consequently, there is a need for robust VQA systems capable of solving more natural questions. Wu et al. [19] proposed an external knowledge base VQA approach. This approach is useful when some common sense or additional background knowledge is required to answer questions correctly. The advantage of this method is that it answers more general questions and understand the reasoning behind how it arrives at the answer by viewing supporting facts generated in the process. But, lack of more precise semantic or visual attention applied to the question and image before querying the knowledge bases is a disadvantage.
Recently, most external knowledge or attention-based VQA approaches do not adequately examine or utilize how the words in the question interact with each other. Hence, there are still issues in answering complex or reasoning questions. The evolution of graph-based approaches (Teney et al. [20]; Kipf and Welling [21]; Narasimhan et al. [22]; Zhu et al. [23]) create new opportunities to overcome still existing challenges in VQA, with better performance on standard datasets.
The objective is to develop a system which uses long short-term memory convolution neural networks, and multilayer perception which will identify the correct response to a question. This is accomplished by response by extracting the necessary data from the input. Its purpose is to use digitalized data to improve the model's accuracy and boost the system's speed [24].
3.1. Methodology
Data collection, pre-processing procedures, categorization are the important steps that are discussed here. The data set is acquired from a resource and thoroughly analyzed. The image is chosen for training/testing functions only when it meets the required criteria and is not repeated. The overall architecture is shown in Fig. 1.
3.2. Proposed Approach
One of the primary goals is to use long short-term memory, convolutional neural networks, and multilayer perception; identify the correct response to a question asked based on an image [24]. Then, construct a natural language response by extracting the necessary data from the input. Its purpose is to use digitalized data to improve the model's accuracy and boost the system's speed.
3.2.1 Data Acquisition and Preprocessing
Data Acquisition and Data Preprocessing is considered to be one of the important steps used for processing of data in an efficient way. Preprocessing of the images is done by converting it from RGB to gray scale image. An averaging filter is used to filter out the noise global basic thresholding is used to eliminate the backdrop and consider only the image, and a high-pass filter is used to emphasize the finer features in the image to sharpen it [25].
3.2.2 Conversion from RGB to Grayscale
The image is converted from RGB to Grayscale as the first stage in pre-processing. The RGB image can be used to derive it using the following formula [26]. Figure 2 depicts the process of conversion of RGB into grayscale. The formula used is: .299R + .587G + .114B
3.2.3 Noise Removal
The technique of removing or reducing noise from an image is known as a noise removal algorithm. By smoothing the entire image and leaving sections near contrast boundaries, noise reduction algorithms diminish or eliminate the visibility of noise. The second phase in image pre-processing is noise reduction. The grayscale image obtained from the previous phase is used as input here. The Median Filter is used in this case, which is a Noise Removal Technique [27].
3.2.4 Median Filtering
A non-linear digital filtering method called the median filter is used to reduce noise in a signal or image. The matrix, which represents the grey scale image, has 0s added to its corners and edges in this instance. Then, for each 3*3 matrix, arrange the elements in ascending order, find the median/middle element of those 9 elements, and assign that median value to that pixel position. The illustration in Fig. 3 demonstrates noise filtering with the Median Filter.
3.2.5 Basic Global Thresholding
Thresholding is a technique of image segmentation in which the pixels of a picture are altered to make them simpler to evaluate. If A(i,j) is larger than or equal to the threshold T, the value is replaced by 0. In this case, the value of T may be changed in the front end to meet the various demands of different pictures [28]. Here, we apply the trial-and-error technique to determine the appropriate threshold value for us. The graphic depicts thresholding using simple global thresholding.
3.2. 5 High Pass Filtering
A high-pass filter can be used to enhance the sharpness of an image. These filters draw attention to the image's finer elements. Here, the thresholding output is used as the input. Here, we're using a filter, and to the pixels at the boundary, we first append the closest values. Image sharpening using a high-pass filter is seen in Fig. 4. The value for a certain pixel position is obtained by multiplying the 3*3 input matrix's elements by the filter matrix, which may be written as A (1,1) * B (1,1). All of the 3*3 input matrix's components are so multiplied, and their sum is divided by 9, giving the result. The values of all pixel places are determined in the same manner. Negative numbers are treated as zero because there can be no negative illumination
3.3. Data Classification and Detection
Convolutional Neural Networks is used for data classification, as CNN accepts the output of the high-pass filter as input while disregarding feature extraction. The resulting comparison matrix is then classified by classifying algorithms like SoftMax.
Algorithms used in VQA are:
-
Convolutional Neural Network (CNN)
-
Recurrent Neural Network (RNN)
Convolutional Neural Network: A convolutional neural network (CNN) is a sub-branch of deep learning in the category of the neural network which has enabled us to assess visual aspects of the image in a systematic way. CNN performs feature extraction. If a weapon is visible, the preprocessed picture is directly submitted to the CNN classifier [29]. By taking into account all of the characteristics in the output layer; the result has some predictive value. SoftMax activation function is used to calculate these values. SoftMax activation yields predicted results. The final product will be designated as a weapon based on the prediction value [11].
Classification using CNN
A convolutional neural network (CNN) is a sub-branch of deep learning in the category of the neural network which has enabled us to assess visual aspects of the image in a systematic way. The layers in CNN help to achieve the tasks which are mentioned below.
3.3.1 Convolutional Layer
The Convolutional Layer is the initial stage in CNN, and it receives 3*3 of the provided matrix obtained from the High-pass filter as input. The result of multiplying the 3*3 matrix by the filter matrix for the proper location is written in the position. The figure below depicts this. This data is passed to the pooling layer, which decreases the matrix even further. Figure 5 depicts the Convolutional Layer and Fig. 6 shows the output of the convolution layer. Convolution is preceded by rectification of negative values to 0s prior to pooling. Because all values are positive, it is not verifiable in this scenario. Many cycles of each are required before pooling.
3.3.2 Pooling Layer
A 3*3 matrix is reduced to a 2*2 matrix in the Pooling layer by selecting the maxima of the particular 2*2 matrix at the exact position. The Pooling Layer is shown in the Fig. 7.
3.3.3 Fully Connected Layer and Output Layer
The pooling layer's output is flattened, and this flat matrix shown in Fig. 8 is passed to the Fully Connected Layer [13]. The fully connected layer has several layers, including the Input layer, Hidden layer, and Output layer. This is shown in Fig. 9.
3.4. Data Extraction
To extract the features from the preprocessed image received as input, we utilize a method called Histogram Orientation Gradient (HOG). It requires a number of procedures, such as figuring out Gx and Gy, the gradients mostly on x and y axes that surround each pixel. Then, by adding these gradients to the required formulas, the magnitude and gradient of the pixel's position are determined. The output of the module is a histogram that is created by displaying the angles and their frequencies. Feature extraction is a dimensionality reduction procedure that reduces a large set of raw data into smaller groupings for processing.
3.4.1Histogram Orientation Gradient
At the matrix's edges and corners, 0's are inserted. Gx and Gy are then computed. Gx is computed as Gx = value on the right value on the left, while Gy is computed as Gy = value on the top value on the left. Gx and Gy in HOG are depicted in Fig. 10.
Magnitude and direction are then computed using the formula shown in figure. Figure depicts Feature Extraction Using HOG. Magnitude represents illumination, while degree of orientation represents the angle of orientation.
After calculating the angle of orientation, the frequency of the angles for the specific intervals are recorded and used as input for the classifier. In this case, the zeroes are not taken into account while calculating the frequency. For example, there are two occurrences in the interval from 40 to 59, therefore we write the frequency as
3.5. Recurrent Neural Network
A category of artificial neural networks where chronological connection of nodes is created to form a graph is called as recurrent neural network (RNN). As a result, it's possible for it to show temporal dynamic behavior. Recurrent neural network (RNN) can process variable length sequences of inputs with the help of their internal state memory. They can therefore be used for applications like speech synthesis or connected, unsegmented character recognition [30].
3.5.1 Long Short-Term Memory
Long Short-Term Memory networks, abbreviated "LSTMs," are a type of RNN smart enough to learn long-term dependencies. They were introduced, refined, and popularized by various others in subsequent works. They work exceptionally effectively on a wide range of issues and are now widely used. Long-term dependency is a problem that LSTMs are specifically meant to prevent. They have a natural ability to remember information for long periods of time; it is not something they have to strive hard to master [30]. A series of recurring neural network modules comprises all recurrent neural networks.
The experimental details are listed below:
-
Get the input image: This step is considered to be the first input, where the image that contains a certain object considered is inserted. The image can be either colored or gray image file. We don’t directly use the image as input into the model. The picture is scaled to 224 by 224, and then the activations are removed from the last VGGNet19 [31] layer of CONV. These [512 x 7 x 7] shape activations are used as input features for the images.
-
Generate the caption from the input image: A common body of basic understanding linking communication and perception is needed for visual question answering (VQA) and image captioning. We propose a new approach to improving the efficiency of VQA that takes advantage of this relation by jointly producing captions aimed at helping to address a particular visual query. The model is trained using an existing caption dataset by automatically determining question-relevant captions using an online gradient-based method
-
Get the question from the user: Get the input image-related question from the user. And we send the input question to the trained dataset. A single image can contain various answers to a related user question. This model has been very faster because of using the neural network concept.
-
Answering to the user from generated caption: Next, we discuss the distinction between using captions models that can be created as well as using given by human annotators. In particular, we train our model with generated question-agnostic captions using the Up-Down caption, question-relevant captions from our caption generation module, and human-annotated captions from the COCO dataset.
Figure 11 shows the experimental model of the proposed system. Figure 12 depicts the complete process of VQA. The sample data sets and results are depicted in Figs. 13 and 14 respectively.
5.1.1 Datasets and Evaluation Metrics
Experiments on COCO-QA [15] and VQA are conducted. The COCO-QA dataset is based on Microsoft COCO image data [16]. There are 78,736 training questions and 38,948 test ones, based on a total of 123,287 images. Four types of questions are provided, including Object, Number, Color and Location. Each type takes 70%, 7%, 17% and 6% of the whole dataset respectively. In the VQA dataset, each image from the COCO data is annotated by Amazon Mechanical Turk (AMT) with three questions. It is the largest for VQA benchmark so far. There are 248,349, 121,512 and 244,302 questions for training, validation and testing, respectively. For each question, ten answers are provided to take consensus of annotators. Following [3], we choose the top 1,000 most frequent answers as candidate outputs, which constitutes 82:67% of the train + val answers.
The proposed approach leverages on question categories to solve the VQA problem. An ablation analysis is conducted to show the efficacy of the proposed approach. In this analysis, a baseline model is constructed by removing Question Categorization and Answer Predictor components of VQA. However, the baseline uses the same set of attended and fused features as CQ-VQA. Results shown in Table-1 shows a relative improvement of 1:45% by CQ-VQA in terms of overall accuracy. CQ-VQA shows improved performance on the other two evaluation metrics as well.
The proposed method achieved competitive performance on benchmark VQA dataset with much smaller message passing times. Table 1 presents the accuracy of different methods and Fig. 15 gives the comparative analysis of the same. The overall accuracy obtained by baseline method [17] and proposed VQA is compared.
| Model | Overall Accuracy | Color | Location | Number | Object |
|---|---|---|---|---|---|
| Baseline | 87.60 | 59.81 | 56.82 | 46.65 | 64.52 |
| Proposed VQA | 87.85 | 60.20 | 56.86 | 46.70 | 64.66 |
The proposed model for categorized questions is the best model for visual question answering since it has more than 85% accuracy. This work presents a model which is end to end and completely hierarchical for visual question answering. This makes use of categorization of questions to reduce the solution space and search time. The results acquired from the implementation of this on the given dataset have shown better results than traditional methods. This section compares the effectiveness of VQA and other conventional approaches. To understand the efficiency of the suggested model, we run several analyses.
In this paper, a very powerful yet very simple approach is used for image processing using CNN architecture of various kinds [32]. More focus is on various techniques for the prediction and the output in natural language. Different methods for image processing are also having been discussed in this work. Any kind of good modification can be done on the current set of code and better results can be found. This helps in finding out the colors for blind people and can be implemented accurately
The major hindrance of the proposed approach is the presence of question categories which are explicit. Extending this model to the questions for which ground level truth does not exist is the next goal. Along with this performance and accuracy of the system can be increased by better feature extraction, attention mechanism and some complex prediction module for the visual questions.
In this work, a novel hierarchical end-to-end model VQA is presented which uses the CNN model. VQA leverages over question categorization to reduce the potential answer search space. Our proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations, but also their inter-modal interactions to produce the answer. More specifically, our model consists of three CNNs: image CNN to encode the image content, sentence CNN to compose the words of the question, and multimodal convolution layer to learn their joint representation for the classification in the space of candidate answer words.
The performance of the proposed model can be further enhanced by using better feature extractor proposals, attention mechanisms and more complex question/answer prediction networks. Also, a challenge remains for datasets where question category ground truth is not available. We plan to work in that direction as a natural extension of the present proposal.
- Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola, “Stacked attention networks for image question answering”, In CVPR, 2016.
- Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh, “Hierarchical question-image co-attention for visual question answering”, In NIPS, 2016.
- Jung-Woo Ha Hyeonseob Nam and Jeonghee Kim, “Dual attention networks for multimodal reasoning and matching”, In CVPR, 2017.
- Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, and Xiaogang Wang, “Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering”, In AAAI, 2018
- Ren S, He K, Girshick R, Sun J, “Faster r-cnn: Towards real-time object detection with region proposal networks”, Advances in neural information processing systems, pp. 91–99, 2015.
- Pennington J, Socher R, Manning C, “Glove: Global vectors for word representation”, Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543,2014.
- Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913, 2017.
- Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., & Fei-Fei, L, “Visual genome: Connecting language and vision using crowdsourced dense image annotations”, International journal of computer vision, 123(1), 32-73, 2017.
- Yash Srivastava, Vaishnav Murali, Shiv Ram Dubey, and Snehasis Mukherjee, “Visual question answering using deep learning: A survey and performance analysis”, arXiv preprint arXiv:1909.01860, 2019.
- Singh AK, Mishra A, Shekhar S, Chakraborty A, “From strings to things: knowledge-enabled VQA model that can read and reason”, Proceedings of the IEEE/CVF international conference on computer vision, pp 4602–4612,2019.
- Ben-Younes H, Cadene R, Cord M, Thome N, “Mutan: multimodal tucker fusion for visual question answering”, Proceedings of the IEEE international conference on computer vision, pp 2612–2620,2017.
- Fukui A, Park DH, Yang D, Rohrbach A, Darrell T, Rohrbach M, “Multimodal compact bilinear pooling for visual question answering and visual grounding”, Conference on empirical methods in natural language processing, pp 457–468, AC, 2016.
- Shih KJ, Singh S, Hoiem D, “Where to look: focus regions for visual question answering”, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4613–4621, 2016.
- Wang P, Wu Q, Shen C, Dick A, van den Hengel A, “FVQA: fact-based visual question answering”, IEEE Trans Pattern Anal Mach Intell 40(10):2413–2427,2018.
- Shah S, Mishra A, Yadati N, Talukdar PP, “Kvqa: knowledge-aware visual question answering”, Proceedings of the AAAI conference on artificial intelligence, vol 33, no 01, pp 8876–8884, 2019
- Kallooriyakath LS, Jithin MV, Bindu PV, Adith PP, “Visual question answering: methodologies and challenges”, 2020 international conference on smart technologies in computing, electrical and electronics (ICSTCEE). IEEE, pp 402–407, 2020.
- Andreas J, Rohrbach M, Darrell T, Klein D, “Neural module networks”, Proceedings of the IEEE conference on computer vision and pattern recognition, pp 39–48, 2016.
- Noh H, Han B, “Training recurrent answering units with joint loss minimization for vqa” Preprint http://arxiv.org/abs/1606.03647, 2016.
- Wu Q, Wang P, Shen C, Dick A, Van Den Hengel A, “Ask me anything: free-form visual question answering based on knowledge from external sources”, Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4622–4630, 2016.
- Teney D, Liu L, van Den Hengel A, “Graph-structured representations for visual question answering”, Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9, 2017.
- Kipf TN, Welling M, “Semi-supervised classification with graph convolutional networks”, Preprint http://arxiv.org/abs/1609.02907. ICLR 2017, 2017.
- Narasimhan M, Lazebnik S, Schwing AG, “Out of the box: reasoning with graph convolution nets for factual visual question answering”, Proceedings of the 32nd international conference on neural information processing systems, pp 2659–2670, 2018.
- Zhu X, Mao Z, Chen Z, Li Y, Wang Z, Wang B, “Object-difference drived graph convolutional networks for visual question answering”, Multimed Tools Appl 80(11):16247–16265, 2021.
- Remi Cadene, Hedi Ben-Younes, Matthieu Cord, and Nico- las Thome. Murel, “Multimodal relational reasoning for visual question answering”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1989–1998, 2019.
- Kevin J Shih, Saurabh Singh, and Derek Hoiem, “Where to look: Focus regions for visual question answering”, Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4613–4621, 2016.
- Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S., Wang, X., & Li, H, “Dynamic Fusion with Intra- and Inter-Modality Attention Flow for Visual Question Answering”, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6632-6641, 2019.
- Lin Ma, Zhengdong Lu, and Hang Li, “Learning to answer questions from image using convolutional neural network”, In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI'16). AAAI Press, 3567–3573, 2016.
- Pennington, Jeffrey & Socher, Richard & Manning, Christopher, “Glove: Global Vectors for Word Representation”, EMNLP. 14. 1532-1543. 10.3115/v1/D14-1162, 2014.
- Aakansha Mishra, Ashish Anand, Prithwijit Guha, “CQ-VQA: Visual Question Answering on Categorized Questions”, 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, 2020.
- D. Kanakamedala, T. Veeranki, R. Bitla, S. Vangalapudi and S. T, "Visual Question Answering Using Deep Learning," 2021 Innovations in Power and Advanced Computing Technologies (i-PACT), Kuala Lumpur, Malaysia, 2021, pp. 1-5, doi: 10.1109/i-PACT52855.2021.9696665.
- M. H. Rafi, S. Islam, S. M. Hasan Imtiaz Labib, S. S. Hasan, F. M. Shah and S. Ahmed, "A Deep Learning-Based Bengali Visual Question Answering System," 2022 25th International Conference on Computer and Information Technology (ICCIT), Cox's Bazar, Bangladesh, 2022, pp. 114-119, doi: 10.1109/ICCIT57492.2022.10055205.
- H. J. Singh, G. Bathla, M. Mehta, G. Chhabra and P. Singh, "Visual Questions Answering Developments, Applications, Datasets and Opportunities: A State-of-the-Art Survey," 2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS), Erode, India, 2023, pp. 778-785, doi: 10.1109/ICSCDS56580.2023.10104870.
No competing interests reported.