Visual explainable AI for Graph-Based VQA and Scene Graph Curation

We present a novel visualization approach to explainable AI for graph-based Visual Question Answering (VQA) systems. Our method focuses on identifying false answer predictions by the model and offers users the opportunity to directly correct mistakes in the input space, thus facilitating datset curation. The decision-making process of the model is visualized by highlighting certain internal states of the Graph Neural Network (GNN). We build our system on top of a GraphVQA framework that implements various GNN-based models for VQA trained on the GQA dataset. We evaluated our tool with a demonstration of identified use cases, quantitative measures, and a user study conducted on experts from the machine learning, visualization, and natural language processing domains. Our findings highlight the prominence of our implemented features in not only supporting the users with incorrect predictions' identification but also identifying the underlying issues. Additionally, our approach is also easily extendable to similar models aiming at graph-based question answering.

Artikel - GitHub - Quellcode (DaRUS) - Model Parameters and Evaluation Data (DaRUS)

@article{vqa2024,
  author = {Künzel, Sebastian and Munz-Körner, Tanja and Tilli, Pascal and Schäfer, Noel and Vidyapu, Sandeep and Vu, Ngoc Thang and Weiskopf, Daniel},
  title = {Visual Explainable {AI} for Graph-Based {VQA} and Scene Graph Curation},
  year = {2024},
  journal = {Visual Computing for Industry, Biomedicine, and Art},
  doi = {TODO}
}

BibTeX herunterladen