Concept Conductor:
Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis

(a) Attribute leakage denotes the application of one concept’s attributes to another (e.g., a cat acquiring the fur and eyes of a dog).

(b) Concept omission indicates one or more target concepts not appearing in the image (e.g., the absence of the target cat).

(c) Subject redundancy refers to the appearance of extra subjects similar to the target concept (e.g., an extra cat).

(d) Appearance truncation signifies the target concept’s appearance being observed only in a partial area of the subject (e.g., the upper half of a dog and the lower half of a cat).

Our method comprises three key components: Multipath Sampling, Layout Alignment, and Concept Injection. At each denoising step, the input latent vector $z_{\mathrm{t}}$ is first corrected to $z_{\mathrm{t}}$′ by the Layout Alignment module. $z_{\mathrm{t}}$′ is then sent to the Concept Injection module for denoising. Both Layout Alignment and Concept Injection utilize the Multipath Sampling structure. After denoising, our method can generate images that align with the given text prompt and visual concepts.

Custom models ${ϵ}_{\theta }^{V_1}$ and ${ϵ}_{\theta }^{V_2}$ are created by adding ED-LoRA to the base model ${ϵ}_{\theta }^{\mathrm{base}}$. The base prompt and edited prompts are sent to the base model and custom models, respectively. Different models receive the same latent input $z_t$ and predict different noises. Self-attention features $F_t^{\mathrm{base}}$, $F_t^{V_1}$, $F_t^{V_2}$, and the output feature maps of the attention layers $h_t^{\mathrm{base}}$, $h_t^{V_1}$, $h_t^{V_2}$ are recorded during this process.

The self-attention feature $F_t^{\mathrm{ref}}$ of the layout reference image is extracted through DDIM inversion, which is then used to compute the loss with $F_t^{\mathrm{base}}$, $F_t^{V_1}$ and $F_t^{V_2}$, updating the input latent vector $z_t$. For simplicity, the conversion from pixel space to latent space is omitted.

Concept Injection consists of two parts: (1) Feature Fusion. The output feature maps of the attention layers from different models are multiplied by their corresponding masks and summed to obtain the fused feature map $h_t$, which is used to replace the original feature map $h_t^{\mathrm{base}}$. (2) Mask Refinement. Segmentation maps are obtained by clustering on the self-attention, and the masks required for feature fusion are extracted from these maps.

Qualitative comparison on two concepts

Qualitative comparison on more than two concepts

More qualitative comparisons on multi-concept customization

Collage-to-Image Generation. Concept Conductor can also utilize a user-created collage as a layout reference and generate images following the given layout.

Object Placement. Concept Conductor can also replace objects in a given scene or add new objects to it.