The AI PC era has a benchmarking problem

I like numbers. Data feels sure. What could be better than measurable progress, a way of quantifying the world to stop arguments before they start?

But as we all know, people find plenty of reason still to fight about performance, even in spite of PC benchmarks. (Half the time, it’s because of the benchmarks.) Neither the humans referring to the results nor the companies producing the hardware in question have much interest in tidy interpretations. And now we have Nvidia’s RTX Spark in the mix.

I’ve heard more than one person express frustration about Nvidia, Microsoft, and other companies pushing AI-focused hardware on consumers during Computex. This YouTube comment from user @vr0k3n sums up the vibe pretty well: “Referring to anything consumer related when presenting these clearly AI B2B products is complete deception….the only reason they did that was to frame [these] as a consumer product so people would still be interested in it when it launches.”

But I’m not so certain this take is quite on the mark. At Microsoft Build, which ran concurrently with Computex this year, one Surface Laptop Ultra demo showed off a split workload—the generation of a 3D art asset through use of local AI and cloud AI tools, each handling different tasks. When I asked about this hybrid work style in an interview shortly thereafter with Andrew Hill, corporate vice president of Surface, he became notably animated and spoke more at length, telling me that such an approach is “exactly what we’re trying to give people options for.” I genuinely believe that Nvidia and Microsoft see a future where “people evolve how they think about what work happens where,” as Hill put it.

Consumers have already begun stepping in this direction, splitting workloads between their local system and the cloud. (For example, I game off my local hardware, but I write using an online document editor.) It’s in part the reason why Chromebooks and aged hardware have become not just viable, but common solutions for everyday computing. So if that’s the case, what’s the performance we should be measuring?

When Nvidia’s RTX Spark CPU launches, people will put it through its paces in all manners of ways. AI workloads, gaming, common productivity tasks, content creation—the whole nine yards and then some. But for me, the point will be less Nvidia’s chip and its specific audience. Instead I’ll be looking at it and wondering what precedent it will set for benchmarking such hardware, meant for tasks split between online and offline tools.

We can’t stop the vision the RTX Spark represents. The companies will push such chips on us. What we can do is thoughtfully respond with how we evaluate them, especially if more and more of consumer computing shifts to the cloud—because new chip production also ends up centering AI more and more.

We may have to let go of certain benchmarks we’re accustomed to, or demand new ones. We may need to adjust the way we form opinions based on the numbers, and what we focus on. Ultimately, testing can answer a million granular queries, yet also to fail the broadest, most important anyone can ask about performance: Is this right for me?

Adam asked more than once recently if we’ve reached a point where PC computing has become good enough for most people, where a need for more performance doesn’t truly exist. I don’t think so, personally. For enthusiasts, we’re bottomless pits when it comes to seeing tech evolve. But we may be in danger of losing attention if we assume that we’ve gotten everything we can get—or react as we always have, using the same approach we have.

As I said, I love numbers. But I’m reminded of my dad, ever the practical person. (If I have any claim to sense, it’s the little that rubbed off on me from him.) He’s the one who always asks, “What are you going to do with that?” So as much as I find joy in poring over charts and how else a piece of hardware will respond, I know it’s most important to answer that with my efforts.

I hope everyone else will find their way there, too.

In this episode of The Full Nerd, Adam Patrick Murray, Brad Chacos, and Alaina Yee chat about their final thoughts on Computex—and their guesses at what will come next. Not on the list of predicted trends: This Montech case, which caught Adam’s attention for reasons that surprised neither me or Brad. I fully expect to be subjected to an in-person sniff test at some point.

Adam also casually revealed that he and his wife are fostering kittens right now. Lindsay sent me photos. THEY ARE SO SMOL.

Missed our live show? Subscribe now to The Full Nerd Network YouTube channel, and activate notifications. We also answer viewer questions in real time!

Don’t miss out on our other shows too—you can catch episodes of Dual Boot Diaries, The Full Nerd: Extra Edition, and Expedition: Handheld through our channel!

And if you need more hardware talk during the rest of the week, come join our Discord community—it’s full of cool, laid-back nerds.

It’s just me this time around—Alex caught a cold. That means more random science news for you. And terrifying anecdotes from Reddit, like a PC supposedly frying through a lightning strike.

But it’s not all gloom: Adam had an interesting chat with AMD during Computex. The Steam Machine is actually on its way soon(ish). And animal retirement communities are apparently a thing. (I wish there were a dedicated daily live stream.)

And with that, The Full Nerd newsletter now belongs to Alex. (Hopefully this hand-off is not a complete surprise, given all the hints in recent weeks.)

Don’t worry—I’m still on The Full Nerd show. You can also still catch me on PCWorld.com and through Safe Mode, a newsletter I’m launching next week on security and privacy. I may even pop back up here from time to time.

It’s been fun sharing my thoughts on PC enthusiast topics over the past year with you all. Thanks for being here through all the twists and turns so far. It’s going to stay wild for a while longer.

Alaina

(P.S. – I said I wouldn’t field requests for an audio podcast based on this newsletter that’s based on a podcast. Alex, however, has not said such a thing. Yet.)

This newsletter is dedicated to the memory of Gordon Mah Ung, founder and host of The Full Nerd, and executive editor of hardware at PCWorld.