MoFu — Scale-Aware Modulation and Fourier Fusion for Multi-Subject Video Generation

Multi-subject video generation aims to synthesize videos from textual prompts and multiple reference images, ensuring that each subject preserves natural scale and visual fidelity. However, current methods face two challenges: scale inconsistency, where variations in subject size lead to unnatural generation, and permutation sensitivity, where the order of reference inputs causes subject distortion.

In this paper, we propose MoFu, a unified framework that tackles both challenges. For scale inconsistency, we introduce Scale-Aware Modulation (SMO), an LLM-guided module that extracts implicit scale cues from the prompt and modulates features to ensure consistent subject sizes. To address permutation sensitivity, we present a simple yet effective Fourier Fusion strategy that processes the frequency information of reference features via the Fast Fourier Transform to produce a unified representation. Besides, we design a Scale-Permutation Stability Loss to jointly encourage scale-consistent and permutation-invariant generation.

To further evaluate these challenges, we establish a dedicated benchmark with controlled variations in subject scale and reference permutation. Extensive experiments demonstrate that MoFu significantly outperforms existing methods in preserving natural scale, subject fidelity, and overall visual quality.

Multi-subject video generation aims to synthesize videos from textual prompts and multiple reference images, producing temporally consistent results where each subject preserves visual fidelity, appears at a natural scale, and remains unaffected by permutations of reference images.

Despite recent progress, multi-subject video generation faces two challenges. The first is scale inconsistency: reference images often vary significantly in subject scales due to different zoom levels, causing subjects in generated videos to appear unnaturally large or small. Another is permutation sensitivity: existing methods process images sequentially, inserting them one by one along the frame or channel dimension. This leads to subject distortion, disappearance, or physically implausible interactions that depend on the input order.

Several recent works attempt to address scale inconsistency by fusing textual prompts into the reference image representation. However, without an explicit mechanism to interpret and enforce natural scale, these methods produce subjects with unnatural scales that contradict the prompt, especially when the reference images themselves vary significantly in scale. To address permutation sensitivity, some methods concatenate multiple reference images onto a single blank canvas. While this avoids sequential processing, it introduces spatial biases determined by subject placement in the composite image.

To address these challenges, we propose MoFu, a unified framework that tackles both scale inconsistency and permutation sensitivity. For the former, we introduce Scale-Aware Modulation (SMO), an LLM-based module that extracts implicit scale relationships from the prompt and injects the condition into the model via modulation. For the latter, we propose a simple yet effective Fourier Fusion strategy that aggregates reference features in the frequency domain via FFT, leveraging near-orthogonality to produce a permutation-invariant representation. We also introduce the Scale-Permutation Stability Loss and construct MoFu-1M and MoFu-Bench.

To train and evaluate MoFu, we construct a high-quality multi-subject video dataset, MoFu-1M, and a dedicated benchmark, MoFu-Bench, specifically designed to assess scale consistency and permutation invariance.

• Video Preprocessing and Captioning. Large-scale raw videos are segmented into coherent clips using scene detection. Low-quality clips are filtered via aesthetic and motion metrics, and the remaining clips are automatically captioned with Qwen2.5-VL.
• Subject Processing and Filtering. We first filter out videos with poor text–video alignment using GmeScore. Next, we extract entities (e.g., people, animals, objects) from captions using LLM-based parsing and localize them in video frames with Grounded-SAM. Finally, we filter out videos where the extracted subjects are either too large or too small based on their relative area in the frames.
• Face Processing. Faces, which require higher fidelity, undergo additional detection and filtering to preserve identity clarity, completing the final MoFu-1M dataset.
• MoFu-Bench Construction. We build MoFu-Bench by introducing controlled variations to the reference images: subject-centric zoom operations to create scale inconsistencies and randomly shuffled reference order to simulate permutation changes.

We train our model on a high-quality, self-curated video dataset. Starting from 15M clips obtained after scene detection, we perform a two-stage filtering process to ensure semantic alignment and appropriate subject visibility. This yields 2.5M high-quality clips, each paired with multiple reference images stored as RGBA. Each clip consists of 81 frames. From these, we select 1M unique clips to form the final training set MoFu-1M.

To evaluate scale consistency and permutation invariance, we construct MoFu-Bench, a dedicated benchmark with 1,000 subject-text pairs. Cases are partially adapted from ConsisID, A2-Bench, and OpenS2V-Bench, with the remainder curated for broad subject diversity. Each case includes up to three reference images with systematic scale variations and randomized permutations, paired with a high-quality prompt for semantic alignment. Compared to existing benchmarks, MoFu-Bench is the first to explicitly assess both scale consistency and permutation invariance.

The purpose of SMO is to explicitly capture scale relationships from the textual prompt and inject them into the generation process. We first encode the prompt $p$ using a frozen LLM to obtain a semantic embedding $\mathbf{e}_p = \text{LLM}(p)$, then pass it through a lightweight Scale Control Adapter (SCA) to predict modulation parameters $\gamma, \beta, \eta = f_{\text{SCA}}(\mathbf{e}_p)$.

Given a DiT block feature map $\mathbf{F}$, SMO performs adaptive feature modulation: $\hat{\mathbf{F}} = \gamma \odot \mathbf{F} + \beta$, $\mathbf{F}_{out} = \mathbf{F} + \eta \cdot \text{Layer}(\hat{\mathbf{F}})$, where $\text{Layer}$ represents either the MHA or FFN module. This modulation is inserted at MHA and FFN layers, ensuring natural subject scales.

To address permutation sensitivity, we propose the Fourier Fusion strategy, which aggregates multiple reference images without introducing positional or order bias. Given $N$ reference images, we apply Grounded-SAM to segment each image, crop and resize subject regions, and encode each with a $3 \times 3$ CNN to obtain feature maps $\mathbf{F}_i$.

Each feature map is transformed into the frequency domain using FFT and decomposed into high-frequency (HF) and low-frequency (LF) components using a radial frequency mask $M_{\text{freq}}$. Leveraging the near-orthogonality of high-dimensional vectors, we aggregate these components across all references by simple summation, then reconstruct via IFFT. The fused representation is concatenated with the video features to condition the generation process.

Scale loss. The scale loss leverages reference masks and their relative area ratios to adaptively re-weight the standard flow-matching loss:

$$\mathbf{M} = \sum_{r=1}^{R} w_r \cdot \text{Resize}(m_r),\quad w_r = \frac{\exp(a_r)}{\sum_{r'} \exp(a_{r'})}$$

$$\mathcal{L}_{\text{scale}} = \frac{\sum (\mathcal{L}_{\text{mse}} \odot \mathbf{M})}{\sum \mathbf{M} + \epsilon}$$

Permutation loss. To ensure that the change in the permutation of references does not affect the generation, we compute an MSE loss in the frequency domain across $P$ permutations of reference inputs:

$$\mathcal{L}_{\text{perm}} = \frac{1}{P} \sum_{p=1}^{P} \left\| F(\mathcal{R}_p) - F(\mathcal{R}_{\text{ref}}) \right\|_2^2$$

$$\mathcal{L}_{\text{SPSL}} = \mathcal{L}_{\text{scale}} + \mathcal{L}_{\text{perm}}$$

We train our model using the AdamW optimizer, configured with $\beta_1 = 0.9$, $\beta_2 = 0.999$, and a weight decay of 0.01. The learning rate is initialized at $1 \times 10^{-5}$ and follows a cosine annealing schedule with periodic restarts. Model training is conducted on 16 NVIDIA H800 GPUs for 7 days. Input videos are processed at a resolution of 480P, with a sequence length of 81 frames.

As shown in Tab., MoFu achieves state-of-the-art performance across nearly all metrics on MoFu-Bench. It attains the highest Aesthetics and FaceSim scores, producing visually appealing videos with strong identity preservation. MoFu also leads on GmeScore, reflecting superior text–video alignment, and achieves balanced Motion results by maintaining temporal coherence without sacrificing fidelity. Moreover, it yields substantial gains on ScaleScore and SubjectSim, demonstrating robust scale consistency and subject stability across permutations.

Aesthetics: We evaluate the perceptual quality and artistic appeal of the generated frames with Aesthetics Score, using an aesthetics prediction model trained on large-scale image quality datasets. Higher scores indicate visually pleasing and artifact-free results.

FaceSim: For videos containing human subjects, we employ FaceSim to measure the similarity between generated faces and reference faces using a face recognition network (e.g., ArcFace).

GmeScore: We use GmeScore, a retrieval-based metric built on a Qwen2-VL model fine-tuned for vision-language alignment, to quantify how accurately the model maintains subject shapes and key structural features.

Motion Score: We assess temporal coherence by Motion Score, implemented by OpenCV, measuring inter-frame motion stability using optical flow smoothness and frame-level warping error.

ScaleScore: To evaluate ScaleScore, we leverage GPT-4o to judge whether the relative sizes of subjects in the generated video match the relationships implied by the input prompt. We provide GPT-4o with the text description and 10 sampled frames from the video, and ask it to output a scale consistency score from 1 to 5.

SubjectSim: SubjectSim measures the overall similarity between generated subjects and their reference images, combining accurate subject localization and feature similarity. We obtain subject masks in reference images using GroundSAM; for sampled generated frames, candidate regions are detected and matched to the reference subject via CLIP feature similarity.

SubjectSim matching (frame $t$, subject $i$):

$$j^{*} = \arg\max_j \frac{f(b_{t,j}) \cdot f(S^{\text{ref}}_i)}{\|f(b_{t,j})\| \|f(S^{\text{ref}}_i)\|}$$

where $b_{t,j}$ is the $j$-th detected region at frame $t$ and $S^{\text{ref}}_i$ is the reference subject. We then compute the cosine similarity of the matched regions' CLIP embeddings and average over all frames and subjects:

$$\text{SubjectSim} = \frac{1}{N} \sum_{i=1}^{N} \frac{1}{T} \sum_{t=1}^{T} \frac{f(S^{\text{gen}}_{i,t}) \cdot f(S^{\text{ref}}_i)}{\|f(S^{\text{gen}}_{i,t})\| \|f(S^{\text{ref}}_i)\|}$$

Higher SubjectSim values indicate better preservation of subject identity and appearance. Together, these metrics provide a holistic evaluation of spatial coherence, subject identity preservation, temporal stability, and scale robustness.

We proposed MoFu, a unified framework that addresses scale inconsistency and permutation sensitivity in multi-subject video generation. By introducing Scale-Aware Modulation and the Fourier Fusion strategy, MoFu explicitly enforces natural subject scales and permutation-invariant conditioning. Besides, we build MoFu-1M for training, and establish a dedicated benchmark MoFu-Bench for assessing scale consistency and permutation-invariance.