Research Results: What AI can do for work.
A look at four studies.
How will AI affect our jobs? Will it replace us? Will it make us more productive? The research has been coming in since May, and we may know the answer. I want to call your attention to four academic studies that provide insight into what happens when humans use Generative AI/ChatGPT for work.
“Experimental evidence on the productivity effects of generative artificial intelligence”
One of the first studies about Generative AI/ChatGPT’s impact on productivity was from researchers at MIT and was published in Science in July (the pre-publication came out in May). The researchers recruited 453 college-educated professionals to complete incentivized writing tasks in this study. Participants were randomly assigned to two groups:
The control group completed the tasks without ChatGPT
The experimental group completed the tasks with ChatGPT.
The results showed a clear increase in productivity for those workers using Generative AI/ChatGPT.
Our results show that ChatGPT substantially raised productivity: The average time taken decreased by 40% and output quality rose by 18%. Inequality between workers decreased, and concern and excitement about AI temporarily rose. Workers exposed to ChatGPT during the experiment were 2 times as likely to report using it in their real job 2 weeks after the experiment and 1.6 times as likely 2 months after the experiment.
Generative AI/ChatGPT increases the productivity of workers. Tasks take less time, and the job is done better. However, some workers receive more benefits from Generative AI/ChatGPT than others.
According to the authors, generative AI/ChatGPT was "especially helpful to those with poor writing and communication skills relative to their other skills."
Workers with poor writing and communication skills sped up and produced clear writing. One might also assume that this leads to less rework.
This outcome is a common theme in these studies: The bell curve of worker productivity moves up and narrows, with good writers receiving some benefits and poor writers benefiting more benefit from Generative AI/ChatGPT.
Read full study here.
“Generative Artificial Intelligence Enhances Creativity but Reduces the Diversity of Novel Content”
This study is from two researchers in the UK who looked at the type of output that workers were producing with AI. They concluded, "Access to GenAI ideas causes an increase in the writer's creativity with stories being evaluated as better written and more enjoyable, especially among less creative writers."
However, they found that while an individual using Generative AI/ChatGPT produces output that might be viewed as more creative, the content was less novel, meaning fewer new, original ideas. This outcome is what we might expect from an AI trained on a particular set of human-created information, which has been tasked with using that training set to inform probabilities on how to place words together meaningfully.
However, all Generative AI/ChatGPT systems are getting better rapidly, and techniques for generating novel content may also get better.
Read full study here.
“AI Assistance in Legal Analysis: An Empirical Study”
Another study from researchers at the University of Minnesota looked at Law students using generative AI to take law exams. The results were mixed, but in the cases where ChatGPT increased speed and quality of output, the researchers "found that GPT -4's impact depended heavily on the student's starting skill level; students at the bottom of the class saw huge performance gains with AI assistance"
Again we are seeing the trend of Generative AI having a dramatic impact on the productivity of poor/less-creative writers. The authors concluded:
This suggests that AI may have an equalizing effect on the legal profession, mitigating inequalities between elite and nonelite lawyers.
Read full study here.
“Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality”
The last paper that I want to highlight was released last week, although a pre-publication has been floating around for a couple of weeks. This study was conducted by a large multidisciplinary team and used multiple experimentation and data collection methods. This chart below shows a summary of the findings and here is how Ethan Mollick, a Wharton professor and one of the authors, summarized the work in his blog.
There is a ton of important and useful nuance in the paper but let me tell you the headline first: for 18 different tasks selected to be realistic samples of the kinds of work done at an elite consulting company, consultants using ChatGPT-4 outperformed those who did not, by a lot. On every dimension. Every way we measured performance. … Consultants using AI finished 12.2% more tasks on average, completed tasks 25.1% more quickly, and produced 40% higher quality results than those without.
Read full study here.
All of these studies point in the same direction. Generative AI/ChatGPT helps good writers a little and poor writers a lot.
What is your experience? Are you more productive with Generative AI/ChatGPT?
One question worth asking is if this trend will be maintained over time as more people adopt the technology and get used to having GAI in the workplace. Remember how Slack, teams, and other "unified" communications systems would help us become more productive? At the time, several studies showed how this system improved productivity. When used properly, these tools can be a productivity boost, but for too many, slack/MSteams has become just another time suck. Why? Well, It is just another tool we have to check, and its pings are a constant distraction since many people are inclined to use it WAY TOO MUCH. And use it for lots of non-work functions. Will Generative AI/ChatGPT fall into the same trap? Will employees prefer to use Generative AI to test new pickup lines, improve their tinder profiles, and generate dad jokes?
In the long term, technology adoption is a psychological and sociological phenomenon. These studies may have an underlying bias toward people and companies that are early adopters, the first to try new technologies. And they provided incentives to complete tasks. What incentives will be used by your organization to adopt AI?