DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes

ICLR 2025 Spotlight

Hengwei Bian1,2,*, Lingdong Kong1,3, Haozhe Xie4, Liang Pan1,†,‡, Yu Qiao1, Ziwei Liu4
1Shanghai AI Laboratory    2Carnegie Mellon University    3National University of Singapore
4S-Lab, Nanyang Technological University
*Work done during an internship at Shanghai AI Laboratory    †Corresponding author    ‡Project lead
Arxiv Code Talk @ AnySyn3D
Teaser Image

We introduce a new occupancy generation model that generates diverse 4D scenes of large spatial scales (80×80×6.4 m³) and long sequential modeling (up to 128 frames), enabling a diverse set of downstream applications.

Abstract

Urban scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D occupancy generation framework capable of generating large-scale, high-quality dynamic 4D scenes with semantics. DynamicCity mainly consists of two key models: 1. A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2. A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D occupancy generation methods across multiple metrics. The code and models have been released to facilitate future research.

Method

Pipeline Image

Our DynamicCity framework consists of two key procedures: (a) Encoding HexPlane with an VAE architecture, and (b) 4D Scene Generation with HexPlane DiT.

Dynamic Scene Generation Results

1. Unconditional Generation

Unconditional Generation 1 Unconditional Generation 2 Unconditional Generation 3
Unconditional Generation 4 Unconditional Generation 5 Unconditional Generation 6
Unconditional Generation 7 Unconditional Generation 8 Unconditional Generation 9
Unconditional Generation 10 Unconditional Generation 11 Unconditional Generation 12
Unconditional Generation 13 Unconditional Generation 14 Unconditional Generation 15

2. HexPlane Conditional Generation

HexPlane Conditional Generation 1 HexPlane Conditional Generation 2 HexPlane Conditional Generation 3
HexPlane Conditional Generation 4 HexPlane Conditional Generation 5 HexPlane Conditional Generation 6

3. Command/Trajectory-Driven Generation

Command-Driven Generation 1 Command-Driven Generation 2 Command-Driven Generation 3
Command-Driven Generation 4 Command-Driven Generation 5 Command-Driven Generation 6

4. Layout-Conditioned Generation

Layout-Conditioned Generation 1
Layout condition
Result
Layout-Conditioned Generation 2
Layout condition
Result
Layout-Conditioned Generation 3
Layout condition
Result
Layout-Conditioned Generation 4
Layout condition
Result
Layout-Conditioned Generation 5
Layout condition
Result
Layout-Conditioned Generation 6
Layout condition
Result
Layout-Conditioned Generation 7
Layout condition
Result
Layout-Conditioned Generation 8
Layout condition
Result
Layout-Conditioned Generation 9
Layout condition
Result
Layout-Conditioned Generation 10
Layout condition
Result

5. Dynamic Scene Inpainting

Dynamic Scene Inpainting 1
Before inpainting
After inpainting
Dynamic Scene Inpainting 2
Before inpainting
After inpainting
Dynamic Scene Inpainting 3
Before inpainting
After inpainting
Dynamic Scene Inpainting 4
Before inpainting
After inpainting
Dynamic Scene Inpainting 5
Before inpainting
After inpainting
Dynamic Scene Inpainting 6
Before inpainting
After inpainting

6. Dynamic Scene Outpainting

Dynamic Scene Outpainting 1
Before outpainting
After outpainting
Dynamic Scene Outpainting 2
Before outpainting
After outpainting

7. Single Occupancy Conditional Generation

Single Occupancy Conditional Generation 1
Occupancy condition
Result
Single Occupancy Conditional Generation 2
Occupancy condition
Result

BibTeX

@inproceedings{bian2025dynamiccity,
  title={DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes},
  author={Bian, Hengwei and Kong, Lingdong and Xie, Haozhe and Pan, Liang and Qiao, Yu and Liu, Ziwei},
  booktitle={Proceedings of the International Conference on Learning Representations (ICLR)},
  year={2025},
}