Maimonides grading student papers on a laptop (Image created using ChatGPT / DALL-E)
by Ian Jensen
What Can LLMs Do for Me?
If you’ve already read the first part of this guide, you might quite reasonably be asking “So what [legitimate] uses do these LLMs have?” It’s a good question and I’m not entirely sure there’s a clear answer at this time, except that LLMs are useful insofar as you need to produce fairly large amounts of grammatically correct text quickly. What the AI people keep telling us is that LLMs are able to e.g. A) give information clearly, B) help us generate ideas, C) help us revise our writing, D) summarize ideas and papers/articles, E) write emails, and so forth.
As we’ve already seen, I’m skeptical about A and B. Since the LLM has no sense of what it is generating it’s all too easy to get incorrect information from it. The same issue applies to helping us with ideas. Since the LLM has no ideas, since it doesn’t connect its tokenized version of language to anything other than its combinatorial options, whatever “ideas” it generates are only perceived by the reader and that reader might better spend her time looking at the actual material rather than querying chatbots.
Helping us revise may in fact prove useful, although at this time it’s difficult to retain or produce a writerly “voice” after you’ve run your text through the algorithm. To the extent that for work emails voice is immaterial, there is a use case here. At the same time, by the time you get what you want from the LLM (see below) you probably could’ve easily written the email yourself.
So we’re left with D, summary. I won’t get into details here, except to say that while any given summary of a text already in the AI’s dataset or one you’ve uploaded may turn out to be accurate, there is no guarantee. Because of the ways in which LLMs do summary, they can either try to extract and reword key ideas/phrases/sentences from the original text (your document or a Wikipedia entry) or they can process the entire document, compare it to the pattern and probability datasets the LLM has trained on and then attempt to condense and state the main points.
In either case, it all depends on how well the LLM is able to process the text it is provided, which means two things. First, the more straightforward that text is, formally speaking, the easier it will be for the LLM to accurately summarize it. Second, the LLM will do a better job of summary when it has been trained on many similar documents. Since much of the text that we rely on in the humanities is far from straightforward–one shudders to imagine ChatGPT processing English translations of German cultural theory or highly figurative, allusive writing like that of Homi K. Bhabha and Hortense Spillers–the summaries we’re going to get won’t be very satisfying or even accurate.
It’s worth noting here that paywalled databases of academic work–Project Muse, to take an example–have not been crawled, scraped, or linked to publicly available chatbots. Those databases will eventually be scraped and linked to LLMs. I see it going one of two ways. In the first scenario, tech companies pay large amounts of money to scrape academic databases for, say, AI-powered Google Scholar. Second, the databases license or buy LLM AI programs from the tech giants and then bake them into their databases. Either way, the amount of money to be made by middle-persons on our unpaid labor truly boggles the mind.
Predictions
Prognostication is a notoriously fickle business, but it is certainly interesting to think about where all of this is going. I do think that what we might call the routine writing of business and education –emails, memos, summaries–will largely be done by AI in the coming years. It is important to note that at this point, generative AI doesn’t threaten the humanities in terms of its usual benefits insight, research, argumentation, critical thought, or topicality in any way–other than incentivizing widespread academic dishonesty.
Unfortunately the humanistic skills are often not valued in our present economy or society, so I don’t have much hope for case that the humanities ought to be supported because they are bastions of “good” writing and thought. Students who are rightly focused on future employment (as many of them are) are generally going to struggle seeing the relevance of what the humanities and liberal arts offer them in their career searches. The truth of the matter is simply that most people don’t end up needing calculus or Latin directly, and I worry that even the very basics of the humanities writ large (rhetoric, rigor, argumentation) will be seen in a similar way. (NB: Clearly utility and value are not synonyms and I’m not suggesting that learning things without obvious “practical” value is pointless. But given the current economic situation, both in terms of cost of college and finding gainful employment after college, it’s a harder sell.) We can argue that the humanities “good for you” until we’re blue in the face, but outside pressures trump idealism most of the time—as they probably should: “The confusion over the purpose of the humanities has nothing to do with their relevance. The humanities are no more or less relevant now than they ever were. It is not the humanities that we have lost faith in, but the economic, political, and social order that they have been made to serve. Perhaps we demand a case for the humanities only because we cannot fathom having to make a case for anything else.”
In a more hopeful vein, it’s possible that publicly available untraceable LLM chatbots turn out to be to be a flash in the pan. One is reminded of the earliest years of sampling in music. The common view is that sample-heavy records like De La Soul’s 3 Feet High & Rising and the Beastie Boys/Dust Brothers’ Paul’s Boutique are essentially impossible to produce now (as legal, mass market works) for copyright reasons. The business of web crawling and scraping vs. copyright is a major one and it could all but kill off companies and products like OpenAI and ChatGPT. Given the immense investments every big tech company except Apple seem to be making in AI, this is probably a very unlikely outcome.
Postscript – What Can Teachers Do?
There have been lots of hot takes on this topic over the past year at the Chronicle and its ilk, as well as on Reddit, blogs, Twitter, and elsewhere. I’m not sure much of it is deeply useful–I’m skeptical of ideas like “flipping the classroom” and also of making my classes into what are essentially AI use trainings. I’m also not going to make students handwrite 4+ page essays in class, although I won’t be offering take-home exams either. There’s the idea that showing students how bad chatbot responses can be will be useful, but to me spending any more than half a class on that exercise seems excessive.
There are, as mentioned, free-ish AI detectors out there but they are said to be highly unreliable. GPT-Zero has gone so far as to create a Chrome browser extension that will record student work on Google Docs and send you a video of the composition of an essay. I hope never to have to mandate its use, but we’ll see how the semester goes.
Good luck out there and godspeed.