Few-Shot Multimodal Instruction Tuning for Vision-Language Models
Students & Supervisors
Student Authors
Supervisors
Abstract
Vision-language models (VLMs) exhibit excellent multimodal reasoning and instruction-following abilities across various vision and language tasks. Nev- ertheless, adapting these models to specialized domains usually necessitates large-scale instruction sets, which are often impractical to obtain. In extreme few-shot scenarios, standard parameter-efficient adaptation methods may fail to generalize or suffer from catastrophic forgetting of the original model’s capabilities. In this paper, we propose a novel few-shot multimodal instruction tuning framework designed to adapt VLMs using minimal data. Our framework relies on instruction-centric supervision and low-rank parameter-efficient adaptation, modifying less than 1% of the parameters, while maintaining general multimodal alignment through stability regularization. We apply the proposed framework to medical visual question answering (VQA) tasks, including VQA-RAD, in extreme few-shot scenarios using only 16–32 training sam- ples. Experimental results demonstrate the framework’s efficiency in improving both accuracy and semantic alignment of the generated answers.
Keywords
Publication Details
- Type of Publication:
- Conference Name: International Conference on Electrical, Computer and Communication Technologies (ECCT 2026)
- Date of Conference: 05/07/2026 - 05/07/2026
- Venue: Dhaka International University, Bangladesh
- Organizer: Prof. Dr. Md. Abdul Based