Scene-Graph-Based Visual Question Answering

Project Overview

This project introduces a novel visual analysis approach aimed at enhancing the interpretability of scene-graph-based visual question answering. The system is designed to assist users in identifying and addressing issues with predictions originating from inadequate scene graphs. Through interactive visualizations, users can gain insights into the model's decision-making processes. This project was developed in collaboration with the Institute for Natural Language Processing (IMS) at the University of Stuttgart and started as a master's thesis: Visual Analytics für Visual-Reasoning-Aufgaben. The student developed the initial version of our approach and authored most sections of our first paper.

My Contribution

I significantly revised the paper, contributed to the related work section, and provided guidance throughout the project. For an extended paper, I took charge of describing use cases and detailing additional features of our interface.

Publications

Results of this project are published in the paper Visual Analysis of Scene-Graph-Based Visual Question Answering. We were subsequently invited to submit an extended version of our VINCI paper to the journal Visual Computing for Industry, Biomedicine, and Art.