Introduction

Multimodal learning aims to create models that can simultaneously process information from various data types or modalities. However, the challenge of bridging the modality gap—differing characteristics between text, images, and audio—remains significant.

Recent advancements in vision-language pretraining have shown promise, yet extending this approach to more modalities without paired training data continues to be a hurdle. This study investigates the application of transformer architectures for unified representation learning across a spectrum of data types.

The core concept revolves around the Meta-Transformer framework, which includes (1) modality-specific tokenization, (2) a general-purpose transformer encoder, and (3) simple task heads. The framework is validated extensively on 12 modalities, demonstrating strong cross-modal transfer learning after pretraining on LAION-2B images. This illustrates the potential of transformers for developing generalized multimodal intelligence through a unified model.

The human brain adeptly integrates information from various sensory inputs—visual, auditory, and tactile—where insights from one modality can enhance understanding in another. However, the significant modality gap complicates the design of a unified network capable of handling diverse data formats. While recent strides in multimodal learning have been made through paired vision-language data pretraining, challenges remain when addressing additional modalities with unpaired data.

Objective

This paper delves into the use of standard transformers for unified multimodal learning. It emphasizes the creation of a framework capable of encoding text, images, point clouds, audio, and other modalities using a shared encoder and sparse fine-tuning, paving the way for unified multimodal intelligence with transformers.

• Proposes the Meta-Transformer, which allows a unified encoder to handle 12 modalities using the same parameters.
• Conducts a comprehensive examination of transformer components across various modalities.
• Achieves robust performance on 12 tasks, supporting the potential for unified learning.

Related Work

1. Single Modality Perception: Various networks facilitate machine perception, including MLPs, CNNs, RNNs, and more recently, transformers. However, each network is tailored to specific modalities, and a unified architecture for multimodal learning is still absent.
2. Transformer-Based Multimodal Perception: Progress has been made in vision-language tasks through cross-attention mechanisms. However, the ability to extend this to additional modalities without paired data remains limited. Meta-Transformer stands out as the first model to enable parameter sharing across 12 modalities.

What is Meta-Transformer?

Meta-Transformer integrates multiple data processing pipelines and is capable of encoding texts, images, point clouds, audio, and eight other modalities using a shared encoder. It consists of a data-to-sequence tokenizer that maps data to a shared embedding space, a modality-agnostic encoder for embedding different modalities, and task-specific heads for downstream predictions.

Meta-Transformer comprises three components:

• Data-to-Sequence Tokenizer: Converts raw inputs into token sequences mapped to a shared manifold space.
• Modality-Shared Encoder: Extracts representations across different modalities.
• Task-Specific Heads: Executes predictions for specific tasks.

It standardizes multimodal data into a common embedding space. A frozen encoder extracts features, adapting them to tasks by updating only lightweight tokenizers and task heads.

The model represents the input space of n modalities as {X1, X2, · · · , Xn}, with corresponding label spaces {Y1, Y2, · · · , Yn}. Each modality has an effective parameter space ?i, enabling the processing of data xi ? Xi from that modality. The essence of Meta-Transformer lies in discovering a shared ? that satisfies: ? ? ? ?1 ? ?2 ? ?3 ? · · · ?n. The multimodal neural networks can be articulated as a unified mapping function F: x ? X ? yˆ ? Y, where x signifies input data from any modality, and yˆ represents the network's prediction.

Data-to-Sequence Tokenization

A novel meta-tokenization scheme is introduced to manage various modalities:

• Text: Utilizes WordPiece embeddings with a vocabulary of 30K.
• Images: Reshaped into flattened patches and projected to embeddings.
• Point Clouds: Sampled skeletons with constructed adjacency matrices, projected into tokens.
• Audio: Splits spectrograms into patches, flattened and projected.

This approach transforms inputs into sequential token embeddings within a shared space.

Unified Encoder

A transformer encoder with fixed parameters encodes the token sequences. It is pretrained on LAION-2B images using contrastive learning to enable versatile encoding. The final CLS token summarizes the sequence for recognition, with position embeddings included to maintain positional information. This unified encoding facilitates the extraction of high-level features across modalities.

Task-Specific Heads

Encoded representations are input into task heads, consisting of MLPs for predictions. The primary aim is to minimize the loss between predictions and ground truth by updating only lightweight tokenizers and heads.

Experiments

Extensive evaluations were conducted across 12 datasets, encompassing text, images, point clouds, audio, video, infrared, X-ray, IMU, tabular, graph, time-series, and hyperspectral data. With pretraining solely on LAION-2B images, Meta-Transformer exhibits remarkable performance across diverse modalities, affirming its viability for unified multimodal learning.

Natural Language Understanding on GLUE Benchmark

Meta-Transformer achieves competitive results in sentiment analysis, paraphrasing, duplication, inference, and answering tasks, with significant performance gains observed post fine-tuning of lightweight components.

Image Understanding

In ImageNet classification tasks, Meta-Transformer records an accuracy ranging from 69.3% to 75.3% in zero-shot scenarios, and 85.4% to 88.1% following tuning, surpassing Swin Transformers in object detection and semantic segmentation tasks.

Infrared, X-ray, and Hyperspectral Image Recognition

For infrared recognition on the RegDB dataset, Meta-Transformer achieves 73.5% Rank-1 accuracy and 65.2% mean average precision (mAP). In hyperspectral image classification, it delivers competitive results with significantly fewer parameters, and 94.1% accuracy is attained for X-ray recognition.

Point Cloud Understanding

In classification tasks on the ModelNet-40 dataset, Meta-Transformer achieves an accuracy of 93.6%, comparable to state-of-the-art methods but with six times fewer parameters. It also excels in segmentation tasks on S3DIS and ShapeNetPart, outperforming alternative techniques.

Audio Recognition

On the Speech Commands V2 dataset, Meta-Transformer attains 97.0% accuracy, competing effectively with audio-specific AST models but with significantly fewer trainable parameters.

Video Recognition

In video understanding tasks on the UCF101 dataset, Meta-Transformer demonstrates an accuracy of 46.6% with only 1.1 million trainable parameters, while other advanced methods require around 86.9 million parameters. Although it does not surpass other state-of-the-art video understanding models, Meta-Transformer's significantly reduced parameter count highlights its potential for unified multimodal learning and decreased architectural complexity.

Graph and IMU Data Understanding

The performance of Meta-Transformer in graph understanding is detailed in the results, comparing it with various graph neural network models on the PCQM4M-LSC dataset. While Graphormer achieves the best performance with the lowest train and validation MAE scores, Meta-Transformer reveals limitations in current architecture for structural data learning, yielding higher MAE scores. Future improvements are anticipated. Additionally, in experiments following ImageBind, Meta-Transformer achieves 73.9% accuracy on the Ego4D dataset.

Limitations

The limitations of Meta-Transformer can be summarized as follows:

• Complexity: The model requires O(n^2 × D) computation for token embeddings, resulting in high memory costs and computational demands that hinder scalability.
• Methodology: Unlike Axial Attention mechanisms in TimeSformer and Graphormer, Meta-Transformer lacks awareness of temporal and structural elements, which could impact its performance in tasks necessitating these aspects, such as video understanding and visual tracking.
• Application: While Meta-Transformer excels in multimodal perception, its capabilities for cross-modal generation remain to be explored.

Conclusion

This research demonstrates that transformers can facilitate unified multimodal learning without the need for modality-specific components or paired training data. Meta-Transformer effectively extracts unified representations across 12 modalities using a shared encoder, showcasing impressive performance. This indicates a promising future for the development of unified multimodal intelligence utilizing transformers.

Paper Link:

2307.10802.pdf (arxiv.org)

Thanks for being a part of our community! Before you go: - Clap for the story and follow the author. - View more content in the Level Up Coding publication.

Follow us: Twitter | LinkedIn | Newsletter

AI Tools: Become an AI prompt engineer