Nvidia unveils GeForce RTX enhancements for AI PC digital assistants

The big AI and graphics tech company unveiled Project G-Assist — an RTX-powered AI assistant tech demo that provides context-aware help for PC games and apps. The Project G-Assist tech demo debuted with ARK: Survival Ascended from Studio Wildcard.

Nvidia also introduced the first PC-based Nvidia NIM (Nvidia inference microservices) for the Nvidia Ace digital human platform. Nvidia made the announcements during CEO Jensen Huang’s keynote at the Computex trade show in Taiwan.

These technologies are enabled by the Nvidia RTX AI Toolkit, a new suite of tools and SDKs that aid developers in optimizing and deploying large generative AI models on Windows PCs. They join Nvidia’s full-stack RTX AI innovations accelerating over 500 PC applications and games and 200 OEM laptop designs. Nvidia is trying to spread AI around the world, in data centers, at the edge and in homes.

In addition, newly announced RTX AI PC laptops from ASUS and MSI feature up to GeForce RTX 4070 GPUs and power-efficient systems-on-a-chip with Windows 11 AI PC capabilities.

“Nvidia launched the era of AI PCs in 2018, with the release of RTX Tensor Core GPUs and DLSS technology,” said Jason Paul, vice president of consumer AI at Nvidia, in a statement. “Now, with Project G-Assist and Nvidia ACE, we’re unlocking the next generation of AI-powered experiences for over 100 million RTX AI PC users.”

Project G-Assist, a GeForce AI Assistant

AI assistants are set to transform gaming and in-app experiences — from offering gaming strategies and analyzing multiplayer replays to assisting with complex creative workflows. Project G-Assist is a glimpse into this future.

PC games offer vast universes to explore and intricate mechanics to master, which are challenging and time-consuming feats even for the most dedicated gamers. Project G-Assist aims to put game knowledge at players’ fingertips using generative AI.

Project G-Assist takes voice or text inputs from the player, along with contextual information from the game screen, and runs the data through AI vision models. These models enhance the contextual awareness and app-specific understanding of a large language model (LLM) linked to a game knowledge database, and then generates a tailored response delivered as text or speech.

Nvidia partnered with Studio Wildcard to demo the technology with ARK: Survival Ascended. Project G-Assist can help answer questions about creatures, items, lore, objectives, difficult bosses and more. Because Project G-Assist is context-aware, it personalizes its responses to the player’s game session.

In addition, Project G-Assist can configure the player’s gaming system for optimal performance and efficiency. It can provide insights into performance metrics, optimize graphics settings depending on the user’s hardware, apply a safe overclock, and even intelligently reduce power consumption while maintaining a performance target.

First ACE PC NIM Debuts

Nvidia ACE technology for powering digital humans is now coming to RTX AI PCs and workstations with NVIDIA NIM — inference microservices that enable developers to reduce deployment times from weeks to minutes. ACE NIMs deliver high-quality inference running locally on devices for natural language understanding, speech synthesis, facial animation and more.

At Computex, the gaming debut of Nvidia ACE NIM on the PC will be featured in the Covert Protocol tech demo, developed in collaboration with Inworld AI. It now showcases NVIDIA Audio2FaceTM and NVIDIA Riva automatic speech recognition running locally on devices.

Windows Copilot Runtime to Add GPU Acceleration for Local PC SLMs Microsoft and NVIDIA are collaborating to help developers bring new generative AI capabilities to their Windows native and web apps. The collaboration will provide application developers with easy application programming interface (API) access to GPU-accelerated small language models (SLMs) that enable retrieval-augmented generation (RAG) capabilities that run on-device powered by Windows Copilot Runtime.

SLMs provide tremendous possibilities for Windows developers including content summarization, content generation, and task automation. RAG capabilities augment SLMs by giving the AI model access to domain-specific information not well represented in the base models. RAG APIs enable developers to harness application-specific data sources and tune SLM behavior and capabilities to application needs.

These AI capabilities will be accelerated by Nvidia RTX GPUs, as well as AI accelerators from other hardware vendors, providing end users with fast, responsive AI experiences across the breadth of the Windows ecosystem.

The API will be released in developer preview later this year.

Faster and smaller models with the RTX AI toolkit

The AI ecosystem has built hundreds of thousands of open-source models for app developers to leverage, but most models are pretrained for general purposes and built to run in a data center.

To help developers build application-specific AI models that run on PCs, Nvidia is introducing RTX AI Toolkit — a suite of tools and SDKs for model customization, optimization and deployment on RTX AI PCs. RTX AI Toolkit will be available in June for broader developer access.

Developers can customize a pretrained model with open-source QLoRa tools. Then, they can use the Nvidia TensorRT model optimizer to quantize models to consume up to three times less RAM. Nvidia TensorRT Cloud then optimizes the model for peak performance across the RTX GPU lineups. The result is up to 4x faster performance compared with the pretrained model.

The Nvidia AI Inference Manager (AIM) software development kit (SDK), now available in early access, simplifies the complexity of AI integration for PC application developers by orchestrating AI inference seamlessly across PCs and the cloud. It also preconfigures the PC with the necessary AI models, engines and dependencies in a unified NIM format and supports all major inference backends — including TensorRT, DirectML, Llama.cpp and PyTorch-CUDA across different processors, including GPUs, NPUs and CPUs.

Software partners such as Adobe, Blackmagic Design and Topaz are integrating components of the RTX AI Toolkit within their popular creative apps to accelerate AI performance on RTX PCs.

“Adobe and Nvidia continue to collaborate to deliver breakthrough customer experiences across all creative workflows, from video to imaging, design, 3D and beyond,” said Deepa Subramaniam, vice president of product marketing for Creative Cloud at Adobe, in a statement. “TensorRT 10.0 on RTX PCs delivers unprecedented performance and AI-powered capabilities for creators, designers and developers, unlocking new creative possibilities for content creation in industry-leading creative tools like Photoshop.”

Components of the RTX AI Toolkit, such as TensorRT-LLM, are integrated in popular developer frameworks and applications for generative AI, including Automatic1111, ComfyUI, Jan.AI, Langchain, LlamaIndex, Oobabooga and Sanctum.AI.

AI for content creation

Nvidia is also integrating RTX AI acceleration into apps for creators, modders and video enthusiasts.

Last year, Nvidia introduced RTX acceleration using TensorRT for one of the most popular Stable Diffusion user interfaces, Automatic1111. Starting this week, RTX will also accelerate the highly popular ComfyUI, delivering up to a 60% improvement in performance over the currently shipping version, and seven times faster performance compared with the MacBook Pro M3 Max.

Nvidia RTX Remix is a modding platform for remastering classic DirectX 8 and DirectX 9 games with full ray tracing, NVIDIA DLSS 3.5 and physically accurate materials. RTX Remix includes a runtime renderer and the RTX Remix Toolkit app, which facilitates the modding of game assets and materials.

Last year, Nvidia made RTX Remix Runtime open source, allowing modders to expand game compatibility and advance rendering capabilities.

Since RTX Remix Toolkit launched earlier this year, 20,000 modders have used it to mod classic games, resulting in over 130 RTX remasters in development on the RTX Remix Showcase Discord.

This month, Nvidia will make the RTX Remix Toolkit open source, allowing modders to streamline how assets are replaced and scenes are relit, increase supported file formats for RTX Remix’s asset ingestor, and bolster RTX Remix’s AI Texture Tools with new models.

In addition, Nvidia is making the capabilities of RTX Remix Toolkit accessible via a REST API, allowing modders to livelink RTX Remix to digital content creation tools such as Blender, modding tools such as Hammer and generative AI apps such as ComfyUI. Nvidia is also providing an SDK for RTX Remix Runtime to allow modders to deploy RTX Remix’s renderer into other applications and games beyond DirectX 8 and 9 classics.

With more of the RTX Remix platform being made open source, modders across the globe can build even more stunning RTX remasters.

Nvidia RTX Video, the popular AI-powered super-resolution feature supported in the Google Chrome, Microsoft Edge and Mozilla Firefox browsers, is now available as an SDK to all developers, helping them natively integrate AI for upscaling, sharpening, compression artifact reduction and high-dynamic range (HDR) conversion.

Coming soon to video editing software Blackmagic Design’s DaVinci Resolve and Wondershare Filmora, RTX Video will enable video editors to upscale lower-quality video files to 4K resolution, as well as convert standard dynamic range source files into HDR. In addition, the free media player VLC media will soon add RTX Video HDR to its existing super-resolution capability.