Blog
In a head-to-head comparison, Narratize helped innovators communicate effectively and efficiently.
Alicia Surrao
July 2, 2024
We get asked a lot about how our platform compares to chatbots. Here’s how we like to describe it: unlike traditional prompting in chatbots, where users endure trial and error prompting, Narratize guides users through a simple Q&A process designed by expert writing consultants to pull insights from and reduce pressure on the user. The goal? Economize writing decision-making while improving the quality and accuracy of generated outputs. We call this ReversePrompting™.
But because co-authoring with Narratize does require hands-on writing time, people want to know more about how Narratize drives productivity differently. How does our approach to human-led AI lead to productivity gains?
Narratize R&D partnered with independent researchers to directly compare the writing productivity of innovators when using Narratize and ChatGPT. We’re still crunching the numbers, but here’s what we know right now: Narratize helped 84% of users create successful innovation pitches. Only 30% of ChatGPT users were able to do the same. And Narratize improved the quality of innovators’ writing as compared against their benchmark samples, while ChatGPT did not.
The primary takeaway: while innovators using ChatGPT did spend less time writing on average, they succeeded in completing the writing task at much lower rates.
Here’s what we know about writing: productivity is not just about speed, it's about the time it takes to get to quality. Because any time spent writing a C+ deliverable is time not well spent. We had a hunch that Narratize outperforms ChatGPT on productivity by helping writers get to great much, much faster. And we were right.
A full report of study outcomes, including our methodology, lit review, and quantitative outcomes, is coming soon. But because we are always fascinated by writers and writing, we wanted to share a few qualitative insights about how innovators are using generative AI today. Read on to learn:
Reports on AI tech tend to focus on speed and accuracy, but when it comes to writing, there’s so much more to effectiveness. Innovators in our study recognized that audience response, self-awareness, and positive affect are integral components to the effectiveness of writing. High speed without high impact isn’t efficient when it comes to writing. Likewise, achieving high accuracy without also achieving a genre and style that will increase audience engagement isn’t efficient. Writing productivity means knowing how to make the right writing decisions, for the right audience, fast. GenAI that doesn’t help writers economize their decision making can simply fail to deliver on productivity promises.
Here’s how a handful of innovators defined writing productivity:
“Effective writing includes receiving the specific response from readers and achieving the desired outcome in terms of persuading, informing, entertaining. [It] also [includes] seeing an increase in engagement in terms of likes, shares, or comments depending on Social Media platform.”
“The main way I can know if my writing is very effective is after writing. I have to read over what I've written. And then … I feel like there's a point of self awareness. You realize that you didn't know [or] use maybe enough vocabulary, [or] you could have done better with a particular language use, punctuation, and stuff.”
“How do you know that your writing is effective? Firstly, … my writing needs to affect me positively. You know, for it to also affect readers positively.”
We tracked how much time innovators spent in the four essential cognitive tasks associated with writing – Ideation, Generation, Rewriting, and Editing. The amount of time they spent in each task was comparable for both Narratize and ChatGPT – Narratize users spent slightly more time on task, on average. But when we asked participants what they were doing specifically in those moments, we found some fascinating differences: Narratize users spent more time thinking deeply about their research and innovation, while ChatGPT users spent more time thinking about how to prompt the chatbot.
Users described the ways that Narratize helps them to focus their mental energy on the parts of the writing process that required their knowledge and experience. The user experience in Narratize helped writers to clarify their messages even before they generated text. Unlike in a chatbot experience, where user-driven prompts presume that the writer knows what they want to say before they say it, Narratize recognizes that discoveries are made through dynamic interaction, and supports that deeper thinking.
One user stated that Narratize helped them focus on their reasoning, as opposed to trying to perform the genre of the deliverable perfectly:
“We were … trying to bring out the kind of work that will match what we are going to write. [With Narratize] you can reason and then try to set up with the Narratize from there to be able to produce some things.” Another innovator wrote that “[Co-authoring with Narratize] was much faster, convenient, and it also enabled me to think about my project in a deeper dimension because the questions that were prompted [in the Narratize platform] really made me question my project: “Is this really what I want to write?” And if I can't answer these few questions, it means that my project isn't that worthy. So, it really made me have an understanding of my project and also fine-tune [my explanations of] the important things.”
In contrast, no ChatGPT users reported a deeper engagement with their topic, research, or reasoning. And while they did report that the chatbot significantly sped up their writing process, they also shared that they spent time making important writing decisions about genre, format, and style, because the tool did not assist in those tasks:
“ChatGPT did not provide all the answers. I gave it a prompt but I immediately knew that something was wrong with the format. I had to make my own personal decisions around format, organization, and audience."
This innovator, despite spending more time writing than most in the group, ultimately did not complete a successful innovation pitch using ChatGPT.
Our sample showed that innovators felt confident in GenAI to support their writing work. Even those who struggled to use the platforms felt that the tools had positively augmented their writing skills (and we certainly saw the impact of AI familiarity on participants’ ability to complete the task, in both groups of innovators).
But here’s the wrinkle: while Narratize helped innovators who struggled on their manual, benchmark samples get up to speed (50% of Narratize users who failed the benchmark succeeded in the AI-assisted task), ChatGPT did not help any innovators perform better. Sixteen percent of this group was able to pass the benchmark task; the same sixteen percent succeeded in the AI-assisted task. ChatGPT did not help innovators who needed further writing assistance communicate effectively.
Even more interestingly, we heard just one innovator express distrust of the factual accuracy of the output in our discussions with the ChatGPT group. Rather, this group understood accuracy in terms of grammatical and mechanical errors – which of course, LLMs typically get right – as opposed to factual inaccuracies. Here’s how one user described their experience with ChatGPT:
"In terms of productivity, I think when using AI, your writing is mostly data-driven. So the things you're writing, the facts are there. You have a lot of facts to work with. You have more data. There's more resources in terms of prompts. You can just prompt and you get more resources on a particular topic that you're working on. "
Narratize users also expressed confidence in the tool and the output - and they noted that the platform actually increased their confidence in their ability to complete the writing task at hand. One user put it this way:
"I definitely think that [co-authoring with Narratize] was more efficient for me, especially in the latter two phases [of writing—revising and editing]. I spent some time ideating, and that was kind of similar to [when I don’t use AI tools]. And I would say I spent maybe more time in the second phase [generating content], especially filling out the forms. But I did feel more confident in that process, like that I had more elements of the pitch than maybe I did last week because I had that template to work from and then was pleasantly surprised with what the tool generated for me… like the pitch was more professional, more like, or it just, you know, and I could more sort of fine tune more quickly with what I came up with. So I thought it did a good job of that, better than I expected.”
Our study revealed that Narratize’s human-led AI approach, powered by ReversePrompting™, helps innovators communicate more effectively and efficiently than ChatGPT in a simulation of a writing workflow in an innovation context. By guiding users through a structured process to clarify their ideas before generating content, Narratize encouraged deeper, more innovative thinking, increased writers’ confidence in their abilities and ideas, and delivered more successful innovation pitches.
These are just a few of the preliminary insights our team has gleaned from our qualitative data set, but this study is teaching us so much more about how our users are experiencing our product and innovating with generative AI. Look out for a report detailing our mixed-methods study, to include quantitative time-savings outcomes and third-party quality assessments of participants’ writing samples. Narratize extends special thanks to all of our outstanding study participants, who shared their time, energy, frank feedback, and bold ideas to help our product grow. Here’s to their success.
Narratize is the generative AI co-author purpose-built for innovation. Learn more about how our product streamlines communication workflows for cross-functional teams doing innovation work by scheduling a demo with our team.
Sign up to learn how to accelerate time-to-market for your enterprise’s best, most brilliant ideas.
