Xbox Series X’s machine-learning auto-HDR renders stunning results

The Xbox Series X has a number of display-related technologies and features that we haven’t seen in any console before. This includes support for 4K at 120Hz over HDMI 2.1 and variable refresh rate (like AMD’s FreeSync). But the upcoming PlayStation 5 and most new graphics cards will also have HDMI 2.1. One feature that Microsoft’s console does have a monopoly on for now, however, is something called auto-HDR. The company is using a machine-learning process to add support for high-dynamic range, which — as the name suggests — expands the display range for both contrast and color. Even more than a higher resolution, this is the tech that makes most modern TVs look better than your old flat panels.

Now Xbox Series X and S can apply this effect to games that launched before television HDR was even invented.

I’ve tested auto-HDR in a number of backward compatible games on a preview version of the Xbox Series X hardware. And as long as your TV supports HDR, you don’t have to do anything else. There’s an option to turn it off in the Xbox’s display settings, but otherwise, most of your backward compatible games (Microsoft turns it off for some games if it clashes with an artistic vision) will get this benefit.

And it is a benefit.

Every game I tested maxes out at 1,000 Nits, which is a measure of luminance. When it comes to HDR, 1,000 Nits is a gold standard for the bright highlights in an image. HDR expert Adam Fairclough, EvilBoris on Twitter (who you should follow), helped out with this by grabbing a heatmap of some of my HDR footage. Let’s take a look at that.

Xbox Series X’s auto-HDR makes games pop

Microsoft hasn’t shared the exact process for applying HDR to older games, but the company did confirm that it is something that the Xbox is doing.

“In partnership with the Xbox Advanced Technology Group, Xbox Series X delivers a new, innovative HDR reconstruction technique which enables the platform to automatically add HDR support to games,” Xbox Series X program director Jason Ronald wrote in a blog post back in May. “As this technique is handled by the platform itself, it allows us to enable HDR with zero impact to the game’s performance and we can also apply it to Xbox 360 and original Xbox titles developed almost 20 years ago, well before the existence of HDR.”

The company has also said that this technique uses machine learning. That means a piece of software is running through games to learn what an HDR image should look like. Then, when the program sees similar objects in a non-HDR game, it knows what image processing to apply to get the desired effect.

This works how you would expect it to a lot of the time. The sun is almost always a very bright source, but so are the neon signs in Yakuza 0.

You can see that Yakuza’s lights peak at 1,000 Nits but cover the entire HDR range.

To get the most out of HDR, you should calibrate the Xbox through its system settings. The Xbox Series X has a tab for HDR calibration, and this handles it across all games. You should then avoid messing with the in-game brightness settings. That could potentially spoil the effect.

The HDR machine knows when to apply HDR

Light sources are an obvious target for AI HDR, and Xbox Series X nails that consistently. What it also nails is livening up visual and lighting effects for objects and attacks.

This is noticeable in Dead Cells, where attacks and defensive maneuvers are supposed to explode off the screen. And now moves like a parry crackle more than ever when the auto-HDR brings it up to 1,000 Nits of peak brightness.

In a game like Dead Cells, where the world is drab and muted, the game communicates to the player through these kinds of visual cues. And HDR amplifies that.

You can see more examples of the AI getting the HDR right in the gallery below. I especially like that the reflection of the sun in the mirrors in GTA IV hit a full 1,000 Nits.

Gallery: Auto-HDR on Xbox Series X

Above: The sun lights up the entire sky, but the brightest spots are the car mirrors directly reflecting the sun.

But sometimes, the algorithm doesn’t quite know how to handle a particular aesthetic. In Banjo-Kazooie: Nuts & Bolts, which looks great on Xbox Series X, the auto-HDR adds extra luminosity to a lot of white objects. Sometimes that includes the character’s eyes.

If you look close, you can see Kazooie’s eyes shine bright. She has “crazy torch eyes,” like Fairclough told me.

But overall, the effect is great, and it’s really awesome to go back and play all-time greats like Titanfall 2 in HDR.

I also hope that this is only the first we see of machine learning in consoles this generation. Fans already remastered the original Final Fantasy VII using an AI upscaling tool to rescue its low-quality textures. And Nvidia’s deep-learning resolution reconstruction is enabling PC games to run at super high framerates.

All of this tech is promising and could help push gaming forward over the next few years.