At NeurIPS 2022, generative AI and LLMs are hot topics

Generative AI and LLMs were two of the hottest topics at NeurIPS 2022, which brought the AI and ML community back in-person for the first time since 2019 and has offered “a lot of excitement,” said Alice Oh, professor at the Korea Advanced Institute of Science and Technology and the conference’s lead program chair. 

Some of that excitement may have been the sound of thousands of keyboards trying out OpenAI’s ChatGPT demo, which was released on Wednesday and has been the talk of Twitter, if not NeurIPS, since then. 

But the Conference and Workshop on Neural Information Processing Systems, a machine learning and computational neuroscience conference held every December, this year in New Orleans, certainly had plenty of its own buzz going on. According to conference leaders, over 10,000 were in attendance in person, with another 3,000 tuning in online. Just a decade ago, the event drew under 2,000. 

In addition, over 2,900 papers were accepted at NeurIPS from a whopping 9,634 submissions, on topics ranging from neural networks and vision transformation to federated learning and offline reinforcement. 

NeurIPS was, as usual, focused heavily on theoretical aspects of machine learning, said Oh, including building more efficient, accurate machine learning algorithms. But large language models (LLMs), diffusion models and generative AI were also hot, trendy topics, along with reinforcement learning – which has been a top focus for several years, she explained. 

“When you go to the poster sessions, the generative AI models do tend to attract people, I think, not only because the research is good but also because they’re fun to look at and talk about,” she said. 

Winning NeurIPS papers included MineDojo and Google Imagen

There were thirteen outstanding paper recipients at NeurIPS, including two from Nvidia — one of which described MineDojo, a generalist AI agent that can perform actions from written prompts in Minecraft. 

Google AI won for a paper around its large text-to-image and super-resolution diffusion model, Imagen, which generates photorealistic images. 

The Allen Institute for AI got attention for its paper on ProcTHOR, a framework that generates interactive 3D environments used to train embodied AI. It takes the time and resource-intensive process of building new virtual environments in which to train intelligent machines, and procedurally creates 1000s of them — offering implications for the future of in-home robots, for example. 

One of the winning papers, Gradient Descent: The Ultimate Optimizer, was authored by MIT CSAIL and Meta researchers including Erik Meijer, who headed Meta’s 50-person Probability team that was laid off in early November. 

Geoffrey Hinton says the future of computing is analog

In a closing virtual keynote at NeurIPS on Thursday, Geoffrey Hinton told the audience — as he told VentureBeat in September — that the future of computing is analog. 

“What I think is that we’re going to see a completely different type of computer, not for a few years, but there’s every reason for investigating this completely different type of computer,” he said. 

These new “mortal” computers won’t replace traditional digital computers. “It won’t be the computer that is in charge of your bank account and knows exactly how much money you’ve got,” he said. “It’ll be used for putting something like GPT-3 in your toaster for one dollar, so running on a few watts, you can have a conversation with your toaster.”

Hinton was asked to speak as part of the conference’s “Test of Time” award, which recognized the 10th anniversary and “huge impact” of “ImageNet Classification with Deep Convolutional Neural Networks,” written with his grad students Alex Krizhevsky and Ilya Sutskever and published in 2012. The paper, which is regarded as the beginning of the deep learning “revolution,” was the first time a convolutional neural network competed at a human level on the ImageNet image recognition competition.

Hinton also released a new paper, The Forward-Forward Algorithm, which he explained offers a new approach to neural networks, called a forward-forward network, which moves away from the backpropagation used in almost all neural networks. 

“I think it can be something really novel and significant,” said Oh of Hinton’s new paper. “We’ll have to see, we’ll have to give it some time for people to digest it and try to replicate and make improvements.” 

Ethical AI was ‘really big,’ plus industry takeaways

Ethical AI topics were also front and center at the first week of NeurIPS, said Oh, with most invited speakers touching on the ethical and social impacts of AI. These included Alondra Nelson, who leads the White House Office of Science and Technology Policy, speaking on the Blueprint for an AI Bill of Rights. 

Overall, there were several topics that were particularly relevant for industry, said Oh. “One is reinforcement learning, which has been pretty slow to be picked up by industry because of data or efficiency issues, but I think it’s slowly but surely going to be applied in industry — to self-driving cars, robots, or even natural language processing — where reinforcement learning could be used to better predict states.” 

She also said that smaller, efficient models are an important NeurIPS topic, which will be important for companies that don’t have the compute resources of Google, Meta or Amazon.  

“I think one really interesting aspect of research that can be applied is the distillation of models, like compressing the models to make them small, or transfer learning — to take a domain with lots of data and then apply it to a domain where you don’t have that large amount of data,” she said. “That’s something that’s been going on for a while that will be very relevant.”