Visual Analysis of Scene-Graph-Based Visual Question Answering

In this paper, we introduce a visual analytics approach aimed at helping machine learning experts analyze the hidden states of layers in recurrent neural networks. Our technique allows the user to interactively inspect how hidden states store and process information throughout the feeding of an input sequence into the network. The technique can help answer questions, such as which parts of the input data have a higher impact on the prediction and how the model correlates each hidden state configuration with a certain output. Our visual analytics approach comprises several components: First, our input visualization shows the input sequence and how it relates to the output (using color coding). In addition, hidden states are visualized through a nonlinear projection into a 2-D visualization space using t-distributed stochastic neighbor embedding to understand the shape of the space of the hidden states. Trajectories are also employed to show the details of the evolution of the hidden state configurations. Finally, a time-multi-class heatmap matrix visualizes the evolution of the expected predictions for multi-class classifiers, and a histogram indicates the distances between the hidden states within the original space. The different visualizations are shown simultaneously in multiple views and support brushing-and-linking to facilitate the analysis of the classifications and debugging for misclassified input sequences. To demonstrate the capability of our approach, we discuss two typical use cases for long short-term memory models applied to two widely used natural language processing datasets.

Project – Article - PDF - Kudos - GitHub - Source code (DaRUS) Model Parameters and Evaluation Data (DaRUS) -

@InProceedings{vqa2023,
  author = {Schäfer, Noel and Künzel, Sebastian and Munz-Körner, Tanja and Tilli, Pascal and Vidyapu, Sandeep and Vu, Ngoc Thang and Weiskopf, Daniel},
  title = {Visual analysis of scene-graph-based visual question answering},
  year = {2023},
  booktitle = {Proceedings of the 16th International Symposium on Visual Information Communication and Interaction},
  publisher = {Association for Computing Machinery},
  series = {VINCI '23},
  doi = {10.1145/3615522.3615547},
}

Download BibTeX