Using ChatGPT for Text Analysis (& Something Even Better)
by
Bill Widmer
December 22, 2024
Manual text analysis is time-consuming and often overwhelming for research professionals.
Sifting through mountains of open-ended survey responses, lengthy PDFs, or customer reviews can feel like searching for a needle in a haystack.
Luckily, you can use ChatGPT for text analysis to make the process faster, easier, and more accurate.
However, while ChatGPT offers impressive capabilities, it has limitations that might leave professionals yearning for more precision, reliability, and control.
In this guide, we’ll explore how you can use ChatGPT for text analysis, the challenges it presents, and why specialized tools designed for research professionals might be a better fit.
Let’s get started!
ChatGPT Text Analysis: What is it?
Text analysis involves extracting meaningful information from text-based data. It's used in various fields, including:
Customer Feedback Analysis: By analyzing customer feedback and reviews, businesses can understand customer satisfaction and identify areas for improvement. This helps in enhancing products and services based on actual customer experiences.
Social Media Monitoring: Analyzing social media posts allows companies to understand public opinion and sentiment towards a brand or product. It helps in tracking brand reputation and identifying emerging trends.
Market Research: Text analysis is used to understand market trends, customer needs, and competitor activity. It provides valuable insights that inform strategic decisions and marketing campaigns.
Some common use cases for text analysis include identifying sentiment in customer reviews, extracting keywords from a product survey, summarizing lengthy documents, and uncovering themes in qualitative research data.
Traditional methods rely on manual analysis, which can be very time-consuming, whereas modern approaches leverage machine learning to enhance tools like ChatGPT for improved data analysis capabilities. In other words, AI makes it easier!
Types of Text Analysis
There are several types of text analysis, each serving a unique purpose:
Sentiment Analysis: This technique involves analyzing text data to determine the emotional tone or sentiment behind it, such as positive, negative, or neutral. It helps businesses understand customer emotions and reactions to products or services.
Language Translation: This involves translating text from one language to another, enabling organizations to communicate with customers and stakeholders across different languages and cultures. It is essential for global businesses aiming to reach a diverse audience.
Text Summarization: This technique condenses long pieces of text into shorter, more digestible versions, highlighting the key points and main ideas. It is particularly useful for quickly understanding large documents or articles.
Verbatim Coding:Verbatim coding is the process of adding “codes” to verbatim comments, or open-ended survey text data, to analyze it for common themes and actionable insights.
So how’s it actually done?
How to Perform ChatGPT Text Analysis Step-by-Step
You can use ChatGPT to do a lot of things, and text analysis is one of them. It uses natural language processing (NLP) through prompts. Prompts are the commands you type into ChatGPT, that guide it to assist you.
Here’s how to use ChatGPT for text analysis:
Access ChatGPT: Use the OpenAI platform or API.
Craft a Prompt: Tailor the prompt to your task. For example:some text
"What is the sentiment of this text?"
"Summarize this article in three sentences."
“Find the common themes of these reviews and code them.”
Refine for Better Outputs: Include examples or specify desired formats for clarity.
Export Results: Copy or download outputs for further analysis, if desired.
Let’s go through an example. Say you have a bunch of customer reviews for your business, and you want to analyze them to see what the sentiment of the reviews is and if there are any recurring themes.
First, you need to gather the data in a way that ChatGPT can understand; the easiest would be to upload all of the reviews into a spreadsheet and give that spreadsheet to ChatGPT.
Side note: You can download your Google Business Profile data (including reviews) using Google Takeout.
For this example, we’ll take this list of 10 reviews for a boot on Amazon:
We’ll then prompt ChatGPT with the following, being sure to attach the spreadsheet:
Please review the attached list of product reviews, and analyze it for common themes and trends. Make suggestions for what to change about the product based on your analysis.
Here are some of ChatGPT’s suggestions based on this analysis:
Not bad! These are solid suggestions. (Click here to see the full conversation.)
However, this was a tiny data set and a very easy prompt. Most text analyses will have thousands, if not tens of thousands of data points.
Plus, ChatGPT struggles in many cases. In this example, the user asked ChatGPT to analyze survey responses and count the number of times each character was mentioned.
GPT said there were “4” instances of the character “Mr. Freeze” — however, upon closer inspection, the character was mentioned 10 times. So it can be straight-up wrong.
And that’s not the only issue with using ChatGPT for text analysis…
ChatGPT Text Analysis Flaws & Limitations
While ChatGPT is a powerful tool, it has significant shortcomings for professional-grade text analysis:
Inability to Quantify: ChatGPT struggles with counting occurrences or providing numerical accuracy, often leading to unreliable results. If you ask ChatGPT, “How many times does the theme of 'customer satisfaction' appear in this dataset?”, it might respond with an estimate like, “It appears approximately 10 times” or worse, it responds with an exact number like “10 times”, but the number turns out to just be made up. This number is not derived from actual counting; it’s probabilistic and often incorrect, which could lead to inaccurate conclusions.
Hallucinations: ChatGPT can fabricate quotes or data that seem plausible but are not grounded in the original text. This makes it unsuitable for evidence-based research.
Data Privacy Concerns: OpenAI's privacy policy indicates that user inputs may be utilized to enhance and train their models, which could lead to the incorporation of sensitive data into future iterations. This practice has raised concerns about data privacy, as users' personal information might be inadvertently exposed or used without explicit consent.
Data Volume Limitations: Input size restrictions hinder the analysis of large datasets. If you try to input a 10,000-row CSV file of survey responses into ChatGPT, you’ll need to break it into smaller chunks due to token limits. This disrupts the workflow and increases the risk of inconsistent analysis across batches.
Inconsistent Results: Prompts can yield different outputs for the same dataset, reducing reliability. When you ask ChatGPT to identify themes in a dataset, it might deliver the result as a textual paragraph in one instance and a chart in another. If you have a script or SPSS code built to analyze the data, you’ll have to adjust it every time because the output format isn’t consistent.
Steep Learning Curve for Teams: Flexibility is a double-edged sword. Users must align on a consistent methodology, which requires training and oversight. It also requires a high level of tech savvyness and proficiency with ChatGPT.
So what should you do instead?
Why Blix is a Better Solution for Text Analysis
For professional analysts, Blix provides a purpose-built, reliable, and efficient alternative to ChatGPT. It is a tool made by researchers, for researchers.
Blix can understand context better than ChatGPT, making it more effective for tasks such as sentiment analysis and verbatim analysis.
Purpose-Built for Text Analysis
Designed specifically for tasks like verbatim analysis and categorization.
Handles large-scale datasets without input size restrictions.
Easy to Learn and Use
No prompting or complex configurations needed.
Intuitive workflows minimize the risk of human error.
Integrates easily with your research tools and stack
Enhanced Accuracy
No hallucinations.
Accurate quantification and reliable sentiment analysis.
Summaries and insights are grounded in actual data.
Built-In Features
Data Visualization Tools: Generate graphs, heatmaps, and other visuals.
Exportable Analytics: Easily share results in presentations or reports.
Translation: Analyze responses across multiple languages and get the report in your native language.
Privacy and Security
Your data is not used for model training.
Analysis is private, and you retain full ownership of your data.
One of the Best Market Research Tools Available
Use Blix’s AI-powered verbatim analysis software to gain actionable market research insights quickly & easily.
Situations requiring high precision and contextual understanding
Professional team that wants to deliver professional work
You want to work quickly and efficiently, at scale
One Redditor tested ChatGPT for text analysis and found Blix to be better, writing that they were “100% impressed by the results”.
Here’s what they found to be the issue with ChatGPT:
“After working with a test dataset that was previously manually coded for around 4 or 5 hours my conclusion is that CGPT 4 is just not ready for prime time. Despite the hype, for this fairly simple task it is not stable enough. Flashes of brilliance but way too frustrating. At one point the software says "you have hit your limit - come back in 3 hours". Things it could do one day could not be replicated with same prompts the next day. Error messages abound. It is slow.
You have to re-upload data, in a single CSV file which requires simple but added steps to get the data into the platform.
Multiple messages in the same comment presents a problem that it cannot easily solve consistently
I actually sought out help from a Data Analyst who just completed her Masters Degree just working with AI - to no real avail.
My conclusion it is not time efficient, maybe useful for a first pass on the data. It will be great once someone creates a fit for purpose tool with stability. Close but not there yet”
Or take the word of one Blix user, Yair Regev-Hass, CEO of Hillel - Beyond Orthodoxy, who tested ChatGPT vs Blix to analyze text and do some verbatim coding. Here’s a link to the conversation — he found that ChatGPT missed codes, hallucinated summaries of quotes, and overall just did a miserable job at verbatim analysis.
If you’re a professional seeking dependable tools, Blixoffers the precision and scalability that ChatGPT cannot. Whether you're analyzing customer feedback, conducting market research, or exploring social trends, Blix ensures accuracy, efficiency, and security.
