ChatGPT has been the greatest hit in technology in the last few years. Technology enthusiasts and researchers, including those associated with the education section, have been closely watching the ChatGPT developments unfold.
The earliest feedback from the experts and users shows evidence that the ChatGPT AI tool is an extremely smart technology that can generate human-style, fluent responses and is here to stay for a long time.
But how accurate and reliable is the information in the responses or answers ChatGPT provides? Can students rely on ChatGPT instead of conducting the necessary research independently?
ChatGPT Accuracy Test Results
At ResearchProspect, our team tested the ChatGPT tool comprehensively from several perspectives, including the language, coherence, fact-checking and accuracy of the information, and concluded that:
ChatGPT is an impressive AI tool that can certainly human-sounding responses based on your inputs. Using the ChapGPT tools is just like using another AI tool which requires creating a user account or connecting your existing social account with the platform to start asking questions and receiving responses from the tool.
However, we found that the information obtained from ChatGPT was sometimes factually incorrect, so using ChatGPT as a source of information, particularly for academic papers such as an essay, thesis, dissertation or report, can land you in hot waters. The accuracy of the output from ChatGPT cannot be trusted, so you should avoid using it for your academic assignments to protect your integrity.
What is ChatGPT Good at?
There are some things ChatGPT is good at. ChatGPT was fed an unbelievable amount of information; therefore, it has tonnes of knowledge about every topic or idea.
We found that ChatGPT can provide examples to back its answers.
Example:1 ChatGPT Provides Information About Systematic Review
Input by ResearchProspect: What is a systematic review?
A systematic review is a type of research study that aims to synthesise all available evidence on a particular research question in a structured and systematic way. It involves a rigorous and transparent process of searching for, selecting, appraising, and summarising relevant studies.
The goal of a systematic review is to provide an unbiased and comprehensive summary of the available evidence on a particular topic, with the aim of answering a specific research question or informing clinical practice.
Systematic reviews can be conducted in any field of research, and can include various types of studies, such as randomised controlled trials, observational studies, and qualitative research. They are often used in healthcare to inform clinical practice guidelines and policy decisions, but can also be used in other fields such as education, social sciences, and environmental sciences.
Systematic reviews are considered the highest level of evidence in the hierarchy of evidence-based practice because of the rigorous and transparent methods used to synthesise the available evidence.
Input by ResearchProspect: Can you provide a quick example of a systematic review?
Sure, here’s an example of a systematic review:
Title: The effectiveness of cognitive-behavioral therapy for anxiety disorders: A systematic review and meta-analysis
Objective: To evaluate the effectiveness of cognitive-behavioral therapy (CBT) in the treatment of anxiety disorders.
Methodology: The authors conducted a systematic review and meta-analysis of randomised controlled trials that compared CBT to other treatments or to no treatment for adults with anxiety disorders. They searched several databases for relevant studies, assessed the quality of the included studies, and performed a meta-analysis to estimate the overall effect size of CBT.
Results: The authors identified 42 studies that met their inclusion criteria, including a total of 2,644 participants. The meta-analysis revealed that CBT was significantly more effective than other treatments or no treatment in reducing symptoms of anxiety, with a moderate effect size (Cohen’s d = 0.64). The effect size was larger for studies that used individual rather than group therapy, and for studies that focused on specific anxiety disorders rather than multiple disorders.
Conclusion: The authors concluded that CBT is an effective treatment for anxiety disorders, with moderate to large effect sizes observed across a range of anxiety disorders and treatment modalities. They recommended that CBT be considered as a first-line treatment for anxiety disorders.
The inherited AI technology allows ChatGPT to provide answers according to the users’ needs. For example, we first asked ChatGPT for a general response, followed by a more specific request for a simpler explanation.
Example:1 Chat GP Explains the Theory of Gravity.
Input by ResearchProspect: Can you quickly explain the theory of gravity?
Sure, the theory of gravity is a scientific explanation of the force that attracts objects with mass towards each other. According to this theory, any two objects in the universe exert a gravitational force on each other, with the strength of the force depending on the masses of the objects and the distance between them.
The theory of gravity was first formulated by Sir Isaac Newton in the 17th century, and was later refined by Albert Einstein’s theory of general relativity in the 20th century. According to the theory of general relativity, gravity is not just a force between two objects, but a curvature of spacetime caused by the presence of mass and energy.
The theory of gravity has been extensively tested and has been shown to accurately predict the behavior of objects in our universe. It is the basis for our understanding of the motion of planets and stars, as well as the behavior of objects on Earth.
Input by ResearchProspect: This is too complex for me. Can you explain gravity in simpler terms?
Sure! Gravity is the force that pulls things towards each other. Everything in the universe that has mass or energy is affected by gravity. The bigger an object is, the more gravity it has. For example, the Earth has a lot of gravity, which is why we stay on the ground instead of floating off into space.
Gravity is what keeps the planets in our solar system orbiting around the sun. The sun’s gravity is so strong that it pulls the planets towards it, but the planets are moving so fast that they don’t fall into the sun.
Gravity is also what makes things fall to the ground when we drop them. When we drop something, it falls because the Earth’s gravity is pulling it towards the ground.
In summary, gravity is the force that pulls things towards each other, and it affects everything in the universe that has mass or energy.
Weaknesses & Limitations of ChatGPT
While there is no denying that ChatGPT is a powerful AI tool that can be a game changer in the world of technology, you must check it against other sources before blindly trusting the information it provides.
While testing, ResearchProspect also found that results from ChatGPT can go wrong when you ask a more specific question that could seem straightforward to a human.
Example: ChatGPT Limitations
ResearchProspect asks: How many countries start with the letter V?
ChatGPT answers: There are no countries in the world that start with the letter “V”.
As we can see, the ChatGPT tool suggests that no countries start with the letter “V”, even though ChatGPT still maintains its confident tone as if the information it provided was correct, which it clearly wasn’t.
Because we know that some countries start with the letter “V”, including Venezuela and Vietnam, it is vitally important to validate the answers ChatGPT offers to ensure they are credible. We recommend that students compare the answers against authentic sources to be confident of their authenticity.
Can the Tool Learn from Mistakes?
ChatGPT is being advertised as an AI tool capable of learning from its mistakes and autocorrecting its answers if the user points it to the mistakes it made. However, this has yet to be tested previously. At ResearchProspect, we tested whether or not ChatGPT can understand and implements feedback on its incorrect answers.
When we told ChatGPT its answer was incorrect, it acknowledged the mistakes and corrected its answer to include countries starting with the letter V.
Example:1 Chat GPT Learning From its Mistakes
ResearchProspect: But there are countries that start with the letter V.
ChatGPT answers: I apologise for my earlier response. You are correct, there are countries that start with the letter “V”. There are two countries in the world that start with the letter “V”:
1. Vanuatu – a small island nation located in the South Pacific Ocean.
2. Vatican City – a small city-state located within Rome, Italy, and is the headquarters of the Roman Catholic Church.
I hope this information is helpful!
We concluded that ChatGPT could not be relied upon because it can provide incorrect answers. However, we also found that ChatGPT can understand user feedback and incorporate it into the answers because it showed it could update its answers based on the feedback.
However, another education company tested ChatGPT’s understanding and learning capabilities through the following questions. It concluded that ChatGPT could also make mistakes in understanding feedback and continue giving the same output even if it recognised the mistakes.
Example:2 ChatGPT answers incorrectly
Company: Are there any singular nouns ending in a single “s”?
ChatGPT: Yes, there are singular nouns that end in just one “s”. Here are some examples:
Note that these nouns can still be made plural by adding “es” to the end, such as “glasses”, “grasses”, “basses”, etc.
Example:3 ChatGPT responds to a correction
Jack: Most of these examples end in a double “s.” ChatGPT: I apologise for the mistake. You are correct, most of the examples I gave end in a double “s.” Here are some examples of singular nouns that end in a single “s”:
These are just a few examples. There are many more singular nouns that end in a single “s.”
So, for students, we advise not using ChatGPT as the primary source of information. You must validate the answers obtained from ChatGPT to be certain that you are not feeding factually incorrect information to your readers.
ChatGPT has limitations; its answers could go wrong, and it may not understand the mistakes and learn from them; experts at ResearchProspect are confident that ChatGPT will continue to improve its learning capabilities and responses in the future through conversations with the users.
However, it will never be perfect because it is an AI tool. There will always be cases, even in future, where it could provide wrong answers and fail to learn from the user feedback.
Hire an Expert Writer
Orders completed by our expert writers are
- Formally drafted in an academic style
- Free Amendments and 100% Plagiarism Free – or your money back!
- 100% Confidential and Timely Delivery!
- Free anti-plagiarism report
- Appreciated by thousands of clients. Check client reviews
Why Does ChatGPT Make Mistakes?
The ChatGPT tool uses an AI language. It provides sophisticated, concise, clear and fluent responses to questions from the user. ChatGPT was exposed to a lot of text from millions of sources about all topics and issues during the training phase. It is not connected to any database where it could look for the requested information. Rather, it draws on patterns and themes it learnt during training.
One way to look at the ChatGPT limitations is to think about how ChatGPT answers are not based on facts but patterns learnt because the tool does not cite the information it generates. For example, when you ask the tool about research methodology, it does not look for the answer in a database but rather puts together the statements or texts it interacted with previously.
When you ask it an out-of-the-box question, you will notice ChatGPT responses will start to sound like AI.
ResearchProspect asked: “Is London the Capital City of Birmingham?”
ChatGPT responds: No, London is not the capital city of Birmingham. London is the capital city of England and the United Kingdom. Birmingham is a city in central England, but it is not the capital city of England or the United Kingdom. The capital of England and the UK is London, and it is where the UK’s government and monarchy are located.
The AI model doesn’t have the information; rather, it develops responses from the patterns it worked with. It tells you what it knows but does not know what is true or false. When the user asks an unusual question that it didn’t encounter previously, the answers will sometimes sound made up and self-contradicting.
ChatGPT is likely to provide correct answers to most of the questions, but when the users ask strange questions they do not know about, it starts to fabricate the answers. Moreover, our research found that ChatGPT was only trained on data up to 2021, which means some of the responses will be outdated or unreliable.
How to Use ChatGPT as a Student
ChatGPT has several advantages and limitations, but it is an extremely advanced and revolutionising AI technology that students can learn from and use in their studies. Here are some dos and donts about effectively using ChatGPT for assignments and academic projects.
Ask simple general knowledge questions
Use it to find ideas and issues for your research problem or questions.
Ask ChatGPT to critique your academic writing.
Use the tool as the primary source of information in your essay, assignment or dissertation.
Cite ChatGPT as a source in your paper.
Ask it to provide arguments for your essay or write the assignments.
Frequently Asked Questions