A firm randomly assigned its scientists AI: here’s what happened

Artificial intelligence (AI) is becoming ubiquitous in applied research, but can it actually invent useful materials faster than humans can? It is still too early to tell, but a massive study suggests that it might.
How close is AI to human-level intelligence?
Aidan Toner-Rodgers, an economist at the Massachusetts Institute of Technology (MIT) in Cambridge, followed the deployment of a machine-learning tool at an unnamed corporate laboratory employing more than 1,000 researchers. Teams that were randomly assigned to use the tool discovered 44% more new materials and filed 39% more patent applications than did the ones that stuck to their standard workflow, he found. Toner-Rodgers posted the results online last month, and has submitted them to a peer-reviewed journal.
“It is a very interesting paper,” says Robert Palgrave, a solid-state chemist at University College London, adding that the limited disclosure of the trial’s details makes the results of the AI deployment hard to evaluate. “It maybe doesn’t surprise me that AI can come up with a lot of suggestions,” Palgrave says. “What we’re kind of missing is whether those suggestions were good suggestions or not.”
Toner-Rodgers had access to internal data from the lab and interviewed the researchers under the condition that he would not disclose the name of the company or the specific products it designed. He writes that it is a US firm that develops new inorganic materials — including molecular compounds, crystal structures, glasses and metal alloys — for use in “healthcare, optics, and industrial manufacturing”.
Researchers built an ‘AI Scientist’ — what can it do?
Starting in 2022, the company systematically adopted an AI tool that it had customized to fit its needs. According to Toner-Rodgers, the tool combines graph neural networks — a popular approach in materials discovery that has been used by DeepMind, Google’s London-based AI firm, among others — with reinforcement learning. The neural network was pre-trained using data from vast existing databases, including crystal structures and their properties from the Materials Project and molecular structures from the Alexandria Materials Database.
Researchers input requirements for a material’s desired properties into the neural network, and the system suggests structures for new materials that could have those properties. The teams then weed out potential duds — such as formulas that would not lead to a stable compound — using their own specialist knowledge and computer simulations. They then attempt to synthesize the candidate structures and, if successful, test them in experiments and even in prototypes of finished products. The results are fed back into the neural network — the ‘reinforcement’ stage that helps it to improve its predictive abilities.
Overall, the teams that used the AI tools designed more new materials than did the ones following the standard workflow. But Toner-Rodgers also found unevenness among the AI-powered teams. Researchers who had been ranked as the company’s top performers got even better, whereas the bottom ones did not seem to get much benefit. “Top scientists leverage their domain knowledge to prioritize promising AI suggestions, while others waste significant resources testing false positives,” he writes.
Could AI help you to write your next paper?
“The finding that high-performing researchers get the most out of AI, to me, is the most interesting result, regardless of whether this is just perception or reality,” says Jevin West, a computational social scientist at the University of Washington in Seattle.
It usually takes years, or even decades, for a new material to be developed and incorporated into a product and enter mass production, so a two-year study cannot measure the ultimate success of the inventions. Instead, Toner-Rodgers used various objective metrics, such as the occurrences of certain pairs of words in the text of patent applications, to find that the AI-designed materials were ‘more novel’ — meaning farther away from the ones in the original databases — than the human-designed ones. This was the most surprising finding, he says, showing that the AI tool did not simply regurgitate knowledge.
But existing metrics, such as those used by Toner-Rodgers, have limitations, West says, and the company’s secrecy “really limits the potential engagement of this study by other researchers”. Palgrave agrees, saying that the lack of any details about how exactly these materials perform better than others makes it difficult to evaluate even the initial results. Still, he adds, “it seems like their teams would not bother filing patents and making prototypes unless they thought so — but we don’t have that direct evidence”. And both West and Palgrave commend the company for having set up a randomized study in early 2022, before the release of the AI-powered chatbot ChatGPT caused an explosion in interest in machine learning.
In a follow-up questionnaire, the researchers using AI-powered workflows reported less satisfaction with their jobs: the tool had taken away some of the more creative steps in their work, and left the scientists mostly to select which suggested materials to take to the next stage. “Evaluating AI suggestions is important, but it’s less enjoyable,” says Toner-Rodgers.