| --- |
| tags: |
| - model_hub_mixin |
| - pytorch_model_hub_mixin |
| language: |
| - en |
| base_model: |
| - google/vit-base-patch16-224 |
| - google-bert/bert-base-uncased |
| --- |
| |
| # 🖼️📝 OneEncoder: A Unified Text & Image Model |
|
|
| **OneEncoder** is a lightweight framework for cross-modal alignment, focusing on efficiently integrating **text and images** (with future extensions to other modalities). Unlike traditional methods relying on massive modality-specific encoders, OneEncoder progressively aligns different data types, making it cost-effective and performant even on small paired datasets. |
|
|
| ## 🚀 Key Features |
| ✅ **Multimodal Alignment**: Initially supports **text & image**, with extension to other modalities. |
| ✅ **Lightweight & Efficient**: Avoids full retraining when adding new modalities. |
| ✅ **Superior Performance**: Outperforms models that require large specialized datasets. |
|
|
| ## 🎯 Applications |
| - **Visual Question Answering (VQA)** |
| - **Image-Text Retrieval** |
| - **Multimodal Content Understanding** |
|
|
| ## 📄 Research Paper |
| 📜 **arXiv**: [OneEncoder: Progressive Cross-Modal Alignment](https://arxiv.org/abs/2409.11059) |
|
|
| ## 📌 Resources |
| 🔗 **GitHub Repo**: [OneEncoder](https://github.com/b-faye/OneEncoder) |
| 🚀 **Hugging Face Demo**: [OneEncoder Retriever](https://huggingface.co/spaces/bilalfaye/OneEncoder-retriever) |
| 📓 **Demo Notebook**: [OneEncoder Demos](https://github.com/b-faye/OneEncoder/tree/main/demo) |
| 🔊 **OneEncoder for Text, Image with temperature=2.5**: [HF Model](https://huggingface.co/bilalfaye/OneEncoder-text-image) |
| 🔊 **OneEncoder for Text, Image & Audio**: [HF Model](https://huggingface.co/bilalfaye/OneEncoder-text-image-audio) |
| 🔊 **OneEncoder for Text, Image & Video**: [HF Model](https://huggingface.co/bilalfaye/OneEncoder-text-image-video) |
| 🔊 **OneEncoder for Text, Image & X-ray**: [HF Model](https://huggingface.co/bilalfaye/OneEncoder-text-image-xray) |
|
|
| ## 📝 Authors |
| 📌 **Bilal FAYE**, Hanane AZZAG, Mustapha LEBBAH, Djamel BOUCHAFFRA |
|
|
|
|
| **Note: This model is training with temperature=1.0 and addition as fusion operation** |
|
|
