README_en.md

Visual-Auditory Fusion Perception AI Platform

     

📕 中文版 README | 📗 English README

📻 Installation Guide

Before using our model, you need to ensure that all necessary dependencies are installed in your environment. These dependencies cover various libraries and tools required for the model's operation, ensuring a smooth inference process.

Please follow these steps for the installation:

1. Open the Terminal or Command Prompt: Depending on your operating system, open the corresponding command-line interface.
2. Install dependencies using pip: Enter the following command to install the required Python packages and libraries using pip.

pip install -r requirements.txt

🚀 Inference Guide

After installing all the necessary dependencies, you can start using our model for inference. We provide two ways of performing inference: using the terminal and using the interactive inference.

Here, we will use the example image asserts/demo.jpg for illustration:

1. Inference using the Terminal

If you want to directly run the inference script in the terminal, you can use the following command:

python chatme.py --image asserts/demo.jpg --question "How many apples are there on the shelf?"

This command will load the pre-trained model and perform inference using the provided image (demo.jpg) and question ("How many apples are there on the shelf?").

The model will analyze the image and attempt to answer the question. The inference result will be output to the terminal in text form, for example:

Xiaochuan: There are three apples on the shelf.

2. Interactive Inference

In addition to using the terminal for inference, you can also use the interactive inference feature to interact with the large model in real-time. To start the interactive terminal, run the following command:

python main.py

This command will launch an interactive terminal that waits for you to enter the image path. You can type the image path (e.g., asserts/demo.jpg) in the terminal and press Enter.

The model will perform inference based on the provided image and wait for you to enter a question.

Once you enter a question (e.g., "How many apples are there on the shelf?"), the model will analyze the image and attempt to answer it. The inference result will be output to the terminal in text form, for example:

Image Path >>>>> asserts/demo.jpg
User: How many apples are there on the shelf?
Xiaochuan: There are three apples on the shelf.

Using this approach, you can easily interact with the model and ask it various questions.

🧾 References

📈 Benchmark

• AGE Challenge Dataset

• COVID-DA Dataset

• Visually Aligned Sound (VAS) Dataset

📷 Visual Perception

• 2D Perception

  • Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation

  • Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clustering Regularizad Self-Training

  • Towards Real-World Test-Time Adaptation: Tri-Net Self-Training with Balanced Normalization

  • Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clustering

  • Distillation Using Oracle Queries for Transformer-based Human-Object Interaction Detection

  • High-resolution networks (HRNets) for Image classification

  • Intra- and Inter-Slice Contrastive Learning for Point Supervised OCT Fluid Segmentation

  • Partitioning Stateful Data Stream Applications in Dynamic Edge Cloud Environments

  • Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution

  • Dense Regression Network for Video Grounding

  • Graph Convolutional Networks for Temporal Action Localization

  • NAT: Neural Architecture Transformer for Accurate and Compact Architectures

  • Efficient Test-Time Model Adaptation without Forgetting

  • Breaking the Curse of Space Explosion: Towards Effcient NAS with Curriculum Search

  • Contrastive Neural Architecture Search with Neural Architecture Comparators

  • Domain-Symnetric Networks for Adversarial Domain Adaptation

  • RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning

  • Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation

  • Self-Supervised Gait Encoding with Locality-Aware Attention for Person Re-Identification

  • Towards Stable Test-Time Adaptation in Dynamic Wild World

  • Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score

  • Masked Motion Encoding for Self-Supervised Video Representation Learning

  • Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation

  • Dynamic Extension Nets for Few-shot Semantic Segmentation

  • Densely-Anchored Sampling for Deep Metric Learning

  • Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation

  • Attention Guided Network for Retinal Image Segmentation

  • Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detection/Object Detection

  • Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection

  • Polysemy Deciphering Network for Human-Object Interaction Detection

  • Bidirectional Posture-Appearance Interaction Network for Driver Behavior Recognition

  • Improving Generative Adversarial Networks with Local Coordinate Coding

  • Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis

• 3D Perception

  • SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation

  • Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection

  • Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks

  • VISTA: Boosting 3D Object Detection via Dual Cross-VIew SpaTial Attention

  • A Self-Supervised Gait Encoding Approach with Locality-Awareness for 3D Skeleton Based Person Re-Identification

  • Deep Multi-View Learning Using Neuron-Wise Correlation-Maximizing Regularizers

  • A Skeleton-Bridged Deep Learning Approach for Generating Meshes of Complex Topologies From Single RGB Images

  • Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation

  • CPEM: Consistent Parameter Estimation Model

  • CR-NeRF: Cross-Ray Neural Radiance Fields for Novel-view Synthesis from Unconstrained Image Collections

  • Contextual Point Cloud Modeling for Weakly-supervised Point Cloud Semantic Segmentation

  • Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap

  • HelixSurf: A Robust and Efficient Neural Implicit Surface Learning of Indoor Scenes with Iterative Intertwined Regularization

🎧 Audio

• Automatic Speech Recognition

• Dialogue System

• Text To Speech

• Audio Anti-spoofing

• Blizzard_Challenge

• Voice Activity Detection

• RegNet

💬 NLP

• How to Train Your Agent to Read and Write

• CogVLM

• Qwen

🔮 Multi-Modal

• Test-Time Model Adaptation for Visual Question Answering with Debiased Self-Supervisions

• Debiased Visual Question Answering from Feature and Sample Perspectives

• Intelligent Home 3D: Automatic 3D-House Design from Linguistic Descriptions Only

• Cross-Modal Relation-Aware Networks for Audio-Visual Event Localization

• Cascade Reasoning Network for Text-based Visual Question Answering

• Length-Controllable Image Captioning

• V2C: Visual Voice Cloning

🤖 Robotic

• Learning Active Camera for Multi-Object Navigation

• Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation

• Learning Vision-and-Language Navigation from YouTube Videos