Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representati
Li, Dongxu, Li, Junnan, Hoi, Steven C. H.
Research paper by Seyed Ahmad Hosseini Miangoleh and collaborators. arXiv record dated 2025-10-25. Research area: Artificial intelligence.
Seyed Ahmad Hosseini Miangoleh, Amin Jalal Aghdasian, Farzaneh Abdollahi
Research paper by Anas Awadalla and collaborators. Published 2024-11-12. Topics: vision-language models, knowledge-augmented captions, multimodal models.
Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, Etash Guha, Silvio Savarese, Ludwig Schmidt, Yejin Choi, Caiming Xiong, Ran Xu
Research paper by Jiuhai Chen and collaborators. Published 2025-05-14. Topics: diffusion transformer, CLIP image features, VAE-based representations, sequential pretraining, image understanding.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, Ran Xu
Research paper by Jiuhai Chen and collaborators. Published 2025-10-17. Topics: Autoregressive, Diffusion, text-to-image generation, image editing, discrete image tokens.
Jiuhai Chen, Le Xue, Zhiyang Xu, Xichen Pan, Shusheng Yang, Can Qin, An Yan, Honglu Zhou, Zeyuan Chen, Lifu Huang, Tianyi Zhou, Junnan Li, Silvio Savarese, Caiming Xiong, Ran Xu
Research paper by Dario Coscia and collaborators. arXiv record dated 2026-01-16. Research area: Machine learning.
Dario Coscia, Pim de Haan, Max Welling
Research paper by Raza Imam and collaborators. Published 2024-07-10. Topics: contrastive learning, CLIP model, SpaceNet, BLIP, FLARE.
Raza Imam, Mohammed Talha Alam, Umaima Rahman, Mohsen Guizani, Fakhri Karray
Research paper by Shyam Marjit and collaborators. Published 2024-02-27. Topics: subject-driven text-to-image, DreamBooth, BLIP-Diffusion, low-rank adaptation, Kronecker product.
Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, Pin-Yu Chen
Our study delves into Multimodal Entity Linking, aligning the mention in multimodal information with entities in knowledge base. Existing methods are still facing challenges like ambiguous entity representations and limited image information utilization. Thus, we propose dynamic entity extraction using ChatGPT, which dynamically extracts entities and enhances datasets. We also propose a method: Dynamically Integrate Multimodal information with knowledge base (DIM), employing the capability of the Large Language Model (LLM) for visual understanding. The LLM, such as BLIP-2, extracts information relevant to entities in the image, which can faci
Song, Shezheng, Li, Shasha, Yu, Jie, Zhao, Shan, Li, Xiaopeng, Ma, Jun, Liu, Xiaodong, Li, Zhuo, Mao, Xiaoguang
Research paper by Weizhi Wang and collaborators. Published 2024-03-05. Topics: Multimodal Language Models, MLM, CLIPScore, CLIP, BLIP2.
Weizhi Wang, Khalil Mrini, Linjie Yang, Sateesh Kumar, Yu Tian, Xifeng Yan, Heng Wang
This project proposes the development of an intelligent smoke and fire detection system using computer vision integrated with the Internet of Things (IoT). The solution aims to improve the efficiency and reliability of traditional fire detection methods, reducing false alarms and enabling faster and more accurate responses. The system is implemented in Python and integrates several libraries, including OpenCV for image capture and processing, the BLIP model for automatic caption generation and visual content analysis, as well as Flask and Blynk Cloud for real-time remote monitoring and control. The architecture was designed in a modular fashi
Silva Soares, Gian Luca, Augusto moura da Silva, Luciano, Matheus, Souza, Rhaissa Julliani, Souza, Vitor Amadeu
Research paper by Boyuan Zheng and collaborators. Published 2024-01-03. Topics: GPT-4V, multimodal models, SEEACT, MIND2WEB benchmark, web agents.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su
