To Infinity and Beyond 🚀: a GPU-driven memory sharing pipeline to generate and process infinite synthetic data

Daniele Della Pietra1, Gino Lanzo Hahn1, Nicola Garau1,2
1University of Trento, 2CNIT
Paper Short Paper Poster Code
Teaser image.

Overview of a typical synthetic data generation pipeline (a) and data processing via an algorithm or neural network (b), versus our fast unified synthetic data generation and processing pipeline (c).

Abstract

What if data generation, manipulation, and training could all happen entirely on the GPU, without ever touching the RAM or the CPU?

In this work, we present a novel pipeline based on Unreal Engine 5, which allows us to generate, render, and process graphics data entirely on the GPU.

By keeping the data stored in GPU memory throughout all the steps, we bypass the traditional bottlenecks related to CPU-GPU transfers, significantly accelerating data manipulation and enabling fast training of deep learning algorithms.

Demo Video

Motivation

The typical workflow—(1) build 3D scenes in a game engine, (2) render and write its data to disk, and finally (3) use that static data as input to an algorithm or neural network—suffers from repeated CPU–GPU round-trips and I/O stalls.

The main bottleneck of this approach is usually the storage, which can contain a finite amount of static data, with a reduced access speed from the graphics hardware.

Our contribution is twofold: (i) generating an infinite amount of data in real-time on the GPU, and (ii) consuming it without storing it anywhere, bypassing the delays due to storage access.

Use Cases

Multi-camera sharing of images and character rig.

Qualitative results 1.

Real-time rendering and sharing of a complex scenario.

Qualitative results 1.

Results

Quantitative results on three separate tasks, averaged over 1000 iterations.

Results.

BibTeX

@inproceedings{10.1145/3721250.3742983,
    author = {Della Pietra, Daniele and Lanzo Hahn, Gino and Garau, Nicola},
    title = {To Infinity and Beyond: a GPU-driven memory sharing pipeline to generate and process infinite synthetic data},
    year = {2025},
    isbn = {9798400715495},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3721250.3742983},
    doi = {10.1145/3721250.3742983},
    abstract = {What if data generation, manipulation, and training could all happen entirely on the GPU, without ever touching the RAM or the CPU? In this work, we present a novel pipeline based on Unreal Engine 5, which allows us to generate, render, and process graphics data entirely on the GPU. By keeping the data stored in GPU memory throughout all the steps, we bypass the traditional bottlenecks related to CPU-GPU transfers, significantly accelerating data manipulation and enabling fast training of deep learning algorithms. Traditional storage systems impose latency and capacity limitations, which become increasingly problematic as data volume increases. Our method demonstrates substantial performance improvements on multiple benchmarks, offering a new paradigm for integrating game engines with data-driven applications. More information on our project page: https://mmlab-cv.github.io/Infinity/},
    booktitle = {Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Posters},
    articleno = {75},
    numpages = {2},
    keywords = {Computer Graphics, GPU, Datasets, AI, Deep Learning},
    location = {
    },
    series = {SIGGRAPH Posters '25}
}