3D VisualizationAIVirtual Reality

New AI tool by Nvidia can automatically generate realistic virtual worlds

The GAN-based AI technology was presented at the NeurIPS conference in Montreal this week

Stay up to date with everything that is happening in the wonderful world of AM via our LinkedIn community.

As one might expect, creating a virtual 3D world is a laborious process that requires not only time and effort but also a high level of skill. Look at any video game, and the graphics will have required thousands and thousands of hours of work on the part of developers specialized in 3D rendering, lighting, textures and more.

That level of work to create virtual environments may not be necessary for much longer, however, as graphics processing unit company Nvidia has developed a powerful AI algorithm capable of generating photorealistic details for a virtual world. The AI tool could not only impact the video game development industry, but could also have important applications in generating environments for virtual reality as well as for training self-driving cars and robots.

Bryan Catanzaro, the vice president of applied deep learning at Nvidia, explained: “We can create new sketches that have never been seen before and render those. We’re actually teaching the model how to draw based on real video.”

If the AI tool sounds impressive, that’s because it is. Nvidia’s team has utilized a conventional machine-learning approach to identify various objects appearing in a video scene, such as cars, trees, homes, etc. From there, it has leveraged a more sophisticated generative adversarial network (GAN) to train a computer to generate photorealistic 3D imagery to fill the outlines of objects.

In other words, users can input the outline of a scene (indicating where different objects should be), and the algorithm will fill in the outlines with realistic details. At this stage, the graphics generated by the algorithm are not perfect—as some objects can appear lightly warped, almost dreamlike—but the output is already highly advanced.

“Classical computer graphics render by building up the way light interacts with objects,” added Catanzaro. “We wondered what we could do with artificial intelligence to change the rendering process.”

The innovative approach was presented at NeurIPS, an AI conference hosted in Montreal this week and has gained significant attention already, largely from the gaming sector, which could drastically improve (in terms of ease) game design.


“NVIDIA has been inventing new ways to generate interactive graphics for 25 years, and this is the first time we can do so with a neural network,” said Catanzaro. “Neural networks — specifically generative models — will change how graphics are created. This will enable developers to create new scenes at a fraction of the traditional cost.”

Located in Santa Clara, California, Nvidia is the latest company to unveil a GAN-based algorithm. Invented by Google, GANs have only been around for a few years but are already demonstrating their ability to produce realistic imagery and sounds. You might recall a viral video of Trump from May 2018 calling on the Belgian people to withdraw from the Paris climate agreement. The video—though clearly fake—was created using GANs and demonstrated the potentially dangerous ramifications of the technology on creating realistic and (actually fake) fake news.

Still, from a technology point of view, Nvidia’s algorithm is mighty impressive and we’re eager to see what its first real world (virtual world?) applications are.

Composites AM 2024

746 composites AM companies individually surveyed and studied. Core composites AM market generated over $785 million in 2023. Market expected to grow to $7.8 billion by 2033 at 25.8% CAGR. This new...

Tess Boissonneault

Tess Boissonneault is a Montreal-based content writer and editor with five years of experience covering the additive manufacturing world. She has a particular interest in amplifying the voices of women working within the industry and is an avid follower of the ever-evolving AM sector. Tess holds a master's degree in Media Studies from the University of Amsterdam.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button
Close Popup
Privacy Settings saved!
Privacy Settings

When you visit any web site, it may store or retrieve information on your browser, mostly in the form of cookies. Control your personal Cookie Services here.

These cookies are necessary for the website to function and cannot be switched off in our systems.

Technical Cookies
In order to use this website we use the following technically required cookies
  • wordpress_test_cookie
  • wordpress_logged_in_
  • wordpress_sec

Decline all Services
Accept all Services


Join our 12,000+ Professional community and get weekly AM industry insights straight to your inbox. Our editor-curated newsletter equips executives, engineers, and end-users with crucial updates, helping you stay ahead.