Cover Image generated by Dall-E in response to the prompt “Create a captivating cover image for an article titled ‘The Rise of Artificial Intelligence: Exploring the Future of Technology and Society.'” which was generated by GPT-4 on 5/24/23 in response to the prompt “Generate a prompt for Dall-E for a cover image for an article on ChatGPT.”
Ladies and gentlemen, put your hands together for the one, the only, the incomparable ChatGPT! That’s right, the language model extraordinaire is back and better than ever, ready to answer all your burning questions with a wit and charm that would make even the most seasoned comedian blush. So, buckle up and get ready to have some fun, because with ChatGPT around, you’re never more than a few keystrokes away from a good time!
Prompt: “Write a funny 250-word intro to a news article about ChatGPT.” Generated with GPT-4 on 3/24/23.
Artificial Intelligence (AI) algorithms have been available to the public for years through services like Siri, Photomath, and Speechify. In November of 2022, a new and revolutionary program joined the mix: ChatGPT.
ChatGPT enables users to generate full-length essays based on unique prompts, as well as customize the tone and writing level of written responses. Given this opening for widespread plagiarism, humanities departments across the country are debating the future of English courses and writing in education. In a piece titled “The End of the English Major,” The New Yorker observed the broader trend of declining enrollment in humanities majors and increased emphasis on STEM fields that has accompanied recent technological advancements. In direct response to the advent of ChatGPT, The Atlantic pondered “The Death of the College Essay,” and The New York Times published an op-ed titled, simply, “This Changes Everything.”
To assess these concerns and the current impact of ChatGPT on our community, we evaluated M-A teachers’ ability to distinguish student-written essays from AI-generated ones. We collected 27 sample prompts from different M-A English classes, generated essay responses to them using ChatGPT, and combined them with real M-A- student-written responses in packets of six to eight essays. We then surveyed 11 M-A English teachers to see how well they could distinguish between student and AI writing. The essays included analyses of poems and novels, paragraphs comparing Taylor Swift songs to lines from Shakespeare, narratives about embarrassing moments, argument essays, and more.
To prevent teachers from expecting one student-written and one chatbot-generated essay per prompt, which would not reflect their experiences reading actual assignments, we included two chatbot-generated essays for some of the prompts with two essays.
Teachers correctly identified 62% of the essays as written by either ChatGPT or students. They were 33% more likely to correctly identify student-written essays than those generated by ChatGPT. This indicates that teachers are unlikely to wrongly suspect a student of using ChatGPT, but may have more difficulty identifying ChatGPT-generated writing.
Teachers especially struggled to identify the chatbot’s writing when it was mixed in with essays written by upperclassmen. They had a 53% average accuracy rate on the junior and senior essays, compared to a 79% average accuracy rate on the freshman and sophomore essays. This could have been because the prompts we used for the upperclassmen essays covered topics like the death penalty and novels like The Great Gatsby, which are more widely discussed online and therefore easier for the chatbot to write about. It could also mean that, as students progress through the education system, their writing adheres to more writing conventions, and is thereby harder to distinguish from AI-generated writing. It’s also possible that the errors in less-developed writing help to distinguish it as human.
Teachers scored relatively high on creative and analytical writing prompts (75% and 73% respectively), but had a hard time correctly identifying the writers of argument essays (55%).
The essays used in data collection were all generated using a version of ChatGPT that ran on GPT3. Open AI has since updated this software to generate better answers and have announced that they will be replacing GPT3 with GPT4, which runs on more data.
The answers ChatGPT provides will only improve over time as programmers edit the software and add more data to its algorithm. While adding more data isn’t guaranteed to increase the quality of the answers, OpenAI CEO Sam Altman believes that the capabilities of AI will progress at an exponential rate. Additionally, developers are working on AI that can code for AI, which will further speed up this process.
ChatGPT-generated essays can be tricky to identify. However, there are a few common tip-offs that can help distinguish AI from human writing. For instance, essays with sentences that have little variation in sentence structure and length are likely to be AI-generated. AI-generated essays also tend to be more rigidly organized and have a monotonous, robotic tone.
GPTZero, an app that determines whether a text was written by a human or by an AI algorithm, is able to identify ChatGPT-written essays with an accuracy rate of over 85% through statistical analysis of these characteristics.
Identifying ChatGPT essays on actual assignments is admittedly different from identifying them on a survey. Teachers are unlikely to search for ChatGPT-written essays when they grade assignments as thoroughly as they would on a survey designed to test their ability to do so. At the same time, as one teacher pointed out, “In reality, teachers are familiar with each individual student’s writing style, so it is difficult to blindly identify whether these are written by students or AI if this is not our own students’ work.”
In an effort to curb cheating using the software, M-A has blocked ChatGPT for anyone using the school’s WiFi. However, students can still access ChatGPT at home, or even on campus if they connect their computers to phone hotspots.
Weeks after ChatGPT was first released, history teacher Anne Olson caught several students using ChatGPT to plagiarize essay revisions. Olson explained, “When I was confronted with essays that didn’t seem to flow the way that my students’ writing typically does, I was concerned. Plus, there were a few specific alarm bells in the writing such as oddly placed rhetorical questions throughout the paragraphs. I confronted the students about it and they admitted to having used ChatGPT.”
English teacher Kat Keigher also caught one of her students submitting a ChatGPT-written essay. “There were zero grammatical errors, and there was an answer, but it felt very generic. I think it was that sense of vague, generic, superficial observation that felt computer-generated,” she said.
Olson said, “I’m not anti-AI, but I’m anti-plagiarism. If you’re struggling to understand something, AI could be another way to learn, but taking work from ChatGPT and turning it in as your own is unacceptable.”
Open AI, ChatGPT’s developer, has done little to prevent cheating, even going so far as to say in a statement on Educator Considerations on their website, “We…caution taking punitive measures against students for using these technologies if proper expectations are not set ahead of time for what users are or are not allowed.”
While M-A’s Academic Integrity Policy does not specifically mention AI, it states that students can’t submit “a portion of a document copied from an author, or composed by another person, and present [it] as original work.”
It’s true that ChatGPT answers are harder to detect than answers found using a simple Google search because the platform combines answers from many different sources. However, ChatGPT plagiarism detectors such as GPTZero, Writer, ZeroGPT, and Writefull, which work by reverse-engineering the algorithms that ChatGPT runs on, are constantly improving. Turnitin, a more widely-used plagiarism detection site, is also currently developing technology to detect ChatGPT’s writing.
Despite issues surrounding plagiarism, ChatGPT has the potential to change the education system and all of our learning for the better. Perhaps in the future, teachers will integrate ChatGPT into their coursework. Khan Academy is currently developing an AI-powered learning tool aimed at making one-on-one tutoring more accessible, coaching students on how to improve their writing, and aiding teachers with lesson planning. Advancing AI could assist with tedious administrative practices, enable foreign language students to practice their conversation skills, or help translate dense, technical academic journals into simpler terms.
ChatGPT has already entered some M-A classrooms as an educational tool. Keigher had her students analyze ChatGPT-written personal statements in class and reported that many were surprised by how good its writing was. She said, “It was a nice reversal of a conversation about writing, because I think the shift in writing and writing education is now that you have to beat the robot. If you’re not writing better than the AI, then you’re not going to excel; you’re not going to succeed. You can use ChatGPT as a starting point, but if you can’t make it better, unique or unpredictable, then it’s not going to be useful.”
She continued, “I think for a lot of teachers, myself included, it means rethinking the sort of prompts we’re asking students to do and rethinking the format of giving the assignments. For example, starting everything in-person as opposed to at home. I think those are good first steps until students understand how to use ChatGPT as a resource rather than as an escape.”
However, incorporating AI into classrooms also brings potential for additional harm. If integrating ChatGPT into the classroom means teachers stop allowing essay revisions or assigning some writing altogether to avoid plagiarism, that hurts the education of all students, not just those who cheat.
As technology advances, the role of humanities departments will undoubtedly evolve. While many have raised concerns over the relevance of learning how to write with the development of systems capable of writing for humans, writing remains a crucial life skill.
English teacher Rachel Wan said, “Writing is a way to practice communicating ideas clearly. When you turn in an essay and get feedback, you learn how to be more clear. Writing is a reflection of the thoughts in our brains, and when we learn how to write more clearly, we learn how to think more clearly. When you work with other people, you need the skill of communicating thoughts and not having them be misunderstood.”
ChatGPT weighed in, “The future of writing and humanities education will undoubtedly be shaped by technology, but it’s important to remember that the human element of creativity and critical thinking will always be necessary to produce meaningful and insightful writing and to fully understand and appreciate the complexities of the human experience.”
