Over and over again, the story of AI is so far not the story of AI improving to the human standard so much as the human standard having been lowered over the course of the last decade or two to the point that what AI can produce is close enough.
Two things here - one is that I wonder how the other entrants from that area are feeling right now. And secondly, when I did my PhD, every single source I used for my exegesis had to be checked - by me - and no doubt my examiners also checked many of them. But the push to publish in academia is probably driving this - people trying to hang onto their jobs. Academia itself maybe needs to have a good look at what it expects of research output.
Yes, there are definitely lots of other pressures causing all of this. In general, the internet has made everyone feel like they have to pump out more and more content whether that's self-published authors doing 5-10 books a year or, well, a Substack newsletter putting out something every week. So, a tool that can produce quantity with minimal effort... it was going to get a lot of use.
I totally agree. Publication pressures are driving this. There is an extraordinary amount of over-publication in academia, which AI is now making infinitely worse. I'm a production editor for an interdisciplinary social science journal, and the volume of submissions we are receiving that are clearly generated by AI has gone up hugely in the past six months. Peer review - the key but flawed quality assurance mechanism for academic research - has been under strain for years. My prediction is that AI will completely break it. Of course, the response to that has been various publishers moving towards AI-first models of review. In fact, reviewers are already using AI to review papers, which authors themselves have quickly learned to game via various techniques: https://www.science.org/content/article/suspicious-phrases-peer-reviews-point-referees-gaming-system.
As the managing editor of an academic journal, I have been paying a lot of attention to these trends. I agree that the model of double-blind peer review is collapsing. Interestingly, submission volume for journals in my field (literary scholarship) is not up-- in fact, many journals do not receive enough unsolicited manuscripts. I can think of several reasons for that-- too much to fit in a reply here, but I'll write about it at some point.
How interesting that your submissions are down. My journal is a social science publication focusing on public health, which is probably relevant. I think problems around over-publication are much greater in journal-focused rather than book-focused fields (along with corresponding issues around authorship). I always feel like medicine sets the standards for other fields (e.g., in terms of research ethics, publishing ethics, publication expectations, etc.), but I think the humanities is somewhat more protected, because disciplinary norms push against these forces. These days, you can just about get away with not having a sole-authored monograph as a successful academic anthropologist (my field), especially if you have a lot of high-impact journal articles, but you can't if you're a literary scholar.
Disciplinary norms-- exactly! I have been thinking about this a lot recently because I know that submission volume for humanities journals in general is down, not just the one I work for. Humanities scholars are also among those most resistant to AI. I've been thinking and writing about this a lot (I have an article forthcoming that touches on some of this) and would love to continue the conversation with you!
I have a longstanding research interest in academic publishing, so am always happy to chat about it (often at length, always in full rant mode, and whether my interlocutor is interested in the topic or not). I've messaged you with my email address.
That all sounds like a horrible morass of AI and research that will be disregarded in the end. Which, for real, substantial and substantiated research is going to be disastrous, let alone academic careers. And yet this week I read an article that said Humanities graduates are going to be what companies look for because of their ability to think, see more than one side, all those "soft" skills that have been scoffed at for years.
"He’s launching an “investigation” into how he was so lazy."
Oh boy, I hope he finds what happened!
As for academic papers, as an academic librarian who is regularly looked down on by a certain kind of profs because, what is the point of us, with our citation formats and giving access to actual scholarship, etc - I might self combust. How can I go to classrooms explain the painstaking, rigorous research process to students when some profs are not even ashamed to make such comments in public? And when institutions keep pushing the use of "ethical" AI and discourage critical thoughts?
I guess I'm gonna have to shrug it off like a rain shaped bench-man like a wound that decides to close, or however the saying goes.
There have been so many journalists caught in so many ways over the years, fabricating sources and any number of other careerist crimes, and it has always seemed to be the same thing driving it — the desperate greed for acclaim and status. Think Stephen Glass as the ultimate exemplar.
Now LLMs have armed the mediocre and half-assed with a limitless power to SEEM to be what they are not. It magically puts within reach something the person could never have achieved on their own. Yet having to do it yourself and it being that hard to do were the reasons it was valuable in the first place and why anybody would’ve wanted it.
Every time somebody gets published and praised for doing something they didn’t actually do, it devalues everything many of us have devoted most of our lives to doing the old fashioned way.
I think the time has come for more witchhunting, not less
Maybe this is low-hanging fruit but the worst (by a hair) is the complaint that requiring authors to verify the accuracy of citations is onerous. Shouldn’t the baseline of any work of nonfiction be that it’s the truth? If you can’t prove that a statement is true, just don’t say it. As a former lawyer, I thought it hilarious when I first read of litigants rebuked for citing nonexistent cases, but citing lies (intentionally or not) continues. Which is crazy, if the truth has any value. It does, right?
Plainly not. Look how many publications do not see problems in their output unless and until readers call attention to it. It wasn’t New York Magazine editors, for example, who spotted alleged plagiarism in Ross Barkan’s work a couple weeks ago; it was readers. Or think back to a NY Times controversy during the first Trump term, when they published an editorial by a right wing Senator that called for military force against protesters. The op-ed page editor at the time acknowledged in a meeting he hadn’t bothered to read it until after it was published (https://www.politico.com/news/2020/06/07/nyt-opinion-bennet-resigns-cotton-op-ed-306317). It is a triumph of careerism that positions like these are held by people who don’t seem to find the work interesting enough to bother doing.
I did a quick search (on Google, so using AI) and apparently no jail time; only sanctions like fines and getting thrown off the case, and huge embarrassment, of course. I don't think you can do more unless the lawyers intended to lie, or acted recklessly. But that's off the top of my head. I suppose that with time, it will be assumed by the court that anyone using AI knows it may be hallucinating, and the latitude given to lawyers will evaporate; there'll be a presumption of recklessness if not intention to mislead the court. My 2 cents.
Did you ask it the right question? Cuz I got the opposite from AI: "Yes, lying by officers of the court—including police officers, prosecutors, and attorneys—is a jailable offense".
If AI says it's a jailable offense, that may be true; if it is, there are nuances: deliberate lie? lie by relying on a usually unimpeachable source? lie because of being too lazy to verify? Etc.
"Deliberate lie" is a redundancy. A lie can't be a lie unless deliberate. Being sloppy, lazy, or making an assumption is not a lie and yes, sanctions wouldn't include jail.
The Arvix situation seems *slightly* more complex, in that it appears there's groups of authors collaborating. Still... weird how their ire is directed at the enforcers of scholarship and not their peers who may be making stuff up. I thought we learned something from 'replication crisis' burning social science credibility, but I guess not.
Actually, 'collaboration' itself (I use the scare quotes because I would not call it genuine) is a symptom of the larger problem - which is the push towards massive over-publication in academia, most of which has been dealt with by growing levels of co-authorship. I don't have the stats to hand, but rates of co-authorship have massively increased over the past three decades in many fields, to the point where it's not uncommon in fields like public health for people to have co-authored hundreds of papers (again, I use the term authorship loosely) - many of which they will not have written a word of. Then we have fields like particle physics, where authors on papers can number in the hundreds. These practices were already straining models of authorship in academia, which I now think is a joke - especially the concept of the 'senior author'. Personally, I get a degree of schadenfreude out of seeing co-authors now trying to disclaim any responsibility for the content, when that is PRECISELY what authorship means. Who knows? Maybe one silver lining to all this is reining in the definition of academic authorship - one can hope.
Yeah, I was a little shocked at how many academics were saying they tend to add co-authors and work who they have little to no interactions with just to get more names on papers. (I'm a professor, but writing / creative writing doesn't exist in this space. We're publishing, but our own work under our own name.) This seems like LLMs just exposing an existing problem
Good point! I'm not in that world, but I understand publication and reference number equal prestige equal job security, so these are understandable inclinations. But academic integrity has to extend beyond the students if the institution wants to continue to be regarded as a source (let alone the source) for truth.
Agreed! There's a deep irony in everyone getting upset about students using AI to write their essays and then doing the same thing with their own papers. I'm currently on a grant selection panel for a national grant agency, and it's clear that AI has been used for virtually all the grant proposals (in fact, those applications where it hasn't been used stand out like the proverbial dog's bollocks). The speed at which it's been embedded in academia at every level is shocking.
It’s also important to remember that many academics are shit writers, and I say this as an academic and creative writer though I don’t like the binary of academic and creative.
What's striking about academics getting upset about a commonsensical policy from ArXiv is that it's exactly the same problem as students using AI to cheat: The incentive is to appear to have done good work, so you can get the legitimations you need for your career, rather than to actually do good work either because that's important to you or because you're actually going to be assessed rigorously on that. Broken incentive structures all the way around. Luckily, while that's not easy to fix, it's still a bit easier to address than trying to put the AI cat back in the bag
I think there's a bit of Emperor's New Clothes going on with a lot of the AI stuff. With the Shy Girl book I remember someone telling me they had read the self-published version and thought it was bad, but then it got praise and it was a Black writer and so they didn't want to say anything negative. Similarly in this case I bet one of the judges went "maybe in X region of the world such a metaphor makes sense!" or "Naturally people from X are super metaphoric" without pausing to think whether it was plain wonky. I'm pretty sure tokenizing and exotifying helped shaped both of these cases. We need to get a bit more diligent but also a bit more honest in our assessments.
I gotta laugh at the AI story quotes because I see that kind of shit everywhere, and honestly it makes me question myself as a writer, native speaker of English and avid reader. I keep thinking these metaphors are different in some way and maybe I'm not as clever as I thought I was. But then I wonder where this kind of language exists in prose -- is it borrowed from a genre? No, probably not, but again, probably so, if you think about it.
AI writing is here because people want to produce works of their own -- and this is my argument -- because selling books has become more important than writing them. Marketing, selling, metrics, rankings, followers -- it's about attention and self-esteem. I am worthy because I sold X number of books this month or have X number of subscribers. Let's don't pretend that's not real. I defy anyone telling me it's not. We've run off most of the accounts touting methods for getting more followers or how to make thousands selling stories, but that tainted atmosphere has fully manifested itself in this need to be seen -- seen through sales and followers -- in a quest for a book deal or some kind of recognition.
AI is in the book writing sphere because of this, which is why I keep saying I wish more writers would write about writing and stop trying to sell me books. Hey, my stories are out there, too, but if you never find and read them, it's cool. My joy is in writing them. I'm not in it for the cash or temporary fame.
Honestly, unless you’re making a business out of it, I don’t think read numbers ought to have much appeal. One deep and thoughtful comment is worth a million silent reads to me.
BUT! He's "A prolific poet and author, with books published and others forthcoming, he is particularly known for his love of poetry." Lol, how did this bio even get past someone?
It seems sadly (even tragically) inevitable that AI, which has no taste, will become a literary tastemaker. Those who learned to read, write, and appreciate good writing before AI will continue to do so. But kids who come up with AI parading as literature, AI writing their papers, and AI analyzing the literature they’re assigned to read will have a hard time differentiating between AI slop and human-authored writing. It feels bleak. I think schools must respond by having students read good books in class—aloud and together.
AI may have no taste, but writes pretty well when answering questions. The sentences are direct, short, with minimal use of adjectives, etc. very much like the writing books teach. That's better than most people are able to do, including, sadly, most writers.
Also, I have some incredible updates re: James D. Miller. A) He coauthored a 2016 pamphlet/debate transcript on global trade with fucking VOX DAY. B) He is in deep with the Longtermist/A.I. doomer crowd, and has ONLY written in those publications recently. His last conventionally peer reviewed article was published in *2011*. So his reference to "publishing 20 papers a year" is pure fantasy. C) He also alludes to graduate students, but Smith College doesn't have a PhD program in economics. https://x.com/davidzmorris/status/2057121156772020497
“misattributed and invented quotes" Quotes might be misattributed, given AI is a huge liar. After all, I caught it taking credit for some Elvis lyrics. But invented? No. AI can't invent anything. It can only data scrape and repeat.
Over and over again, the story of AI is so far not the story of AI improving to the human standard so much as the human standard having been lowered over the course of the last decade or two to the point that what AI can produce is close enough.
Two things here - one is that I wonder how the other entrants from that area are feeling right now. And secondly, when I did my PhD, every single source I used for my exegesis had to be checked - by me - and no doubt my examiners also checked many of them. But the push to publish in academia is probably driving this - people trying to hang onto their jobs. Academia itself maybe needs to have a good look at what it expects of research output.
Yes, there are definitely lots of other pressures causing all of this. In general, the internet has made everyone feel like they have to pump out more and more content whether that's self-published authors doing 5-10 books a year or, well, a Substack newsletter putting out something every week. So, a tool that can produce quantity with minimal effort... it was going to get a lot of use.
This is it right here.
I totally agree. Publication pressures are driving this. There is an extraordinary amount of over-publication in academia, which AI is now making infinitely worse. I'm a production editor for an interdisciplinary social science journal, and the volume of submissions we are receiving that are clearly generated by AI has gone up hugely in the past six months. Peer review - the key but flawed quality assurance mechanism for academic research - has been under strain for years. My prediction is that AI will completely break it. Of course, the response to that has been various publishers moving towards AI-first models of review. In fact, reviewers are already using AI to review papers, which authors themselves have quickly learned to game via various techniques: https://www.science.org/content/article/suspicious-phrases-peer-reviews-point-referees-gaming-system.
As the managing editor of an academic journal, I have been paying a lot of attention to these trends. I agree that the model of double-blind peer review is collapsing. Interestingly, submission volume for journals in my field (literary scholarship) is not up-- in fact, many journals do not receive enough unsolicited manuscripts. I can think of several reasons for that-- too much to fit in a reply here, but I'll write about it at some point.
How interesting that your submissions are down. My journal is a social science publication focusing on public health, which is probably relevant. I think problems around over-publication are much greater in journal-focused rather than book-focused fields (along with corresponding issues around authorship). I always feel like medicine sets the standards for other fields (e.g., in terms of research ethics, publishing ethics, publication expectations, etc.), but I think the humanities is somewhat more protected, because disciplinary norms push against these forces. These days, you can just about get away with not having a sole-authored monograph as a successful academic anthropologist (my field), especially if you have a lot of high-impact journal articles, but you can't if you're a literary scholar.
Disciplinary norms-- exactly! I have been thinking about this a lot recently because I know that submission volume for humanities journals in general is down, not just the one I work for. Humanities scholars are also among those most resistant to AI. I've been thinking and writing about this a lot (I have an article forthcoming that touches on some of this) and would love to continue the conversation with you!
I have a longstanding research interest in academic publishing, so am always happy to chat about it (often at length, always in full rant mode, and whether my interlocutor is interested in the topic or not). I've messaged you with my email address.
Wonderful, thank you! I am very interested (including for rant mode). I'll be in touch!
That all sounds like a horrible morass of AI and research that will be disregarded in the end. Which, for real, substantial and substantiated research is going to be disastrous, let alone academic careers. And yet this week I read an article that said Humanities graduates are going to be what companies look for because of their ability to think, see more than one side, all those "soft" skills that have been scoffed at for years.
Same! When I did my PhD I knew every single one of my citations. I’d read everything I’d cited. If I hadn’t my committee would have failed me.
"He’s launching an “investigation” into how he was so lazy."
Oh boy, I hope he finds what happened!
As for academic papers, as an academic librarian who is regularly looked down on by a certain kind of profs because, what is the point of us, with our citation formats and giving access to actual scholarship, etc - I might self combust. How can I go to classrooms explain the painstaking, rigorous research process to students when some profs are not even ashamed to make such comments in public? And when institutions keep pushing the use of "ethical" AI and discourage critical thoughts?
I guess I'm gonna have to shrug it off like a rain shaped bench-man like a wound that decides to close, or however the saying goes.
It’s so wild that one now has to explicitly spell out that the first reader of your work should be you, the writer.
If you can’t be bothered to put in the effort to even read your own work, why should anyone else read it?
Exactly this.
There have been so many journalists caught in so many ways over the years, fabricating sources and any number of other careerist crimes, and it has always seemed to be the same thing driving it — the desperate greed for acclaim and status. Think Stephen Glass as the ultimate exemplar.
Now LLMs have armed the mediocre and half-assed with a limitless power to SEEM to be what they are not. It magically puts within reach something the person could never have achieved on their own. Yet having to do it yourself and it being that hard to do were the reasons it was valuable in the first place and why anybody would’ve wanted it.
Every time somebody gets published and praised for doing something they didn’t actually do, it devalues everything many of us have devoted most of our lives to doing the old fashioned way.
I think the time has come for more witchhunting, not less
Maybe this is low-hanging fruit but the worst (by a hair) is the complaint that requiring authors to verify the accuracy of citations is onerous. Shouldn’t the baseline of any work of nonfiction be that it’s the truth? If you can’t prove that a statement is true, just don’t say it. As a former lawyer, I thought it hilarious when I first read of litigants rebuked for citing nonexistent cases, but citing lies (intentionally or not) continues. Which is crazy, if the truth has any value. It does, right?
that part confused me. as a nonfiction writer, i'm reading every source i use....are people not doing that???
You’re right, it would seem obvious…but no.
That is alarming
Plainly not. Look how many publications do not see problems in their output unless and until readers call attention to it. It wasn’t New York Magazine editors, for example, who spotted alleged plagiarism in Ross Barkan’s work a couple weeks ago; it was readers. Or think back to a NY Times controversy during the first Trump term, when they published an editorial by a right wing Senator that called for military force against protesters. The op-ed page editor at the time acknowledged in a meeting he hadn’t bothered to read it until after it was published (https://www.politico.com/news/2020/06/07/nyt-opinion-bennet-resigns-cotton-op-ed-306317). It is a triumph of careerism that positions like these are held by people who don’t seem to find the work interesting enough to bother doing.
Editors aren't fact checkers but I hear what you're saying.
My point isn’t that they are or ought to be; just that they ought to be interested enough in their own work to read it before the public does.
You’d think!
I wouldn’t think it now. Maybe 15 years ago.
Rebuked? I though lying by officers of the court is a jailable offence
I did a quick search (on Google, so using AI) and apparently no jail time; only sanctions like fines and getting thrown off the case, and huge embarrassment, of course. I don't think you can do more unless the lawyers intended to lie, or acted recklessly. But that's off the top of my head. I suppose that with time, it will be assumed by the court that anyone using AI knows it may be hallucinating, and the latitude given to lawyers will evaporate; there'll be a presumption of recklessness if not intention to mislead the court. My 2 cents.
Did you ask it the right question? Cuz I got the opposite from AI: "Yes, lying by officers of the court—including police officers, prosecutors, and attorneys—is a jailable offense".
If AI says it's a jailable offense, that may be true; if it is, there are nuances: deliberate lie? lie by relying on a usually unimpeachable source? lie because of being too lazy to verify? Etc.
"Deliberate lie" is a redundancy. A lie can't be a lie unless deliberate. Being sloppy, lazy, or making an assumption is not a lie and yes, sanctions wouldn't include jail.
The Arvix situation seems *slightly* more complex, in that it appears there's groups of authors collaborating. Still... weird how their ire is directed at the enforcers of scholarship and not their peers who may be making stuff up. I thought we learned something from 'replication crisis' burning social science credibility, but I guess not.
Actually, 'collaboration' itself (I use the scare quotes because I would not call it genuine) is a symptom of the larger problem - which is the push towards massive over-publication in academia, most of which has been dealt with by growing levels of co-authorship. I don't have the stats to hand, but rates of co-authorship have massively increased over the past three decades in many fields, to the point where it's not uncommon in fields like public health for people to have co-authored hundreds of papers (again, I use the term authorship loosely) - many of which they will not have written a word of. Then we have fields like particle physics, where authors on papers can number in the hundreds. These practices were already straining models of authorship in academia, which I now think is a joke - especially the concept of the 'senior author'. Personally, I get a degree of schadenfreude out of seeing co-authors now trying to disclaim any responsibility for the content, when that is PRECISELY what authorship means. Who knows? Maybe one silver lining to all this is reining in the definition of academic authorship - one can hope.
Yeah, I was a little shocked at how many academics were saying they tend to add co-authors and work who they have little to no interactions with just to get more names on papers. (I'm a professor, but writing / creative writing doesn't exist in this space. We're publishing, but our own work under our own name.) This seems like LLMs just exposing an existing problem
Exactly!
Good point! I'm not in that world, but I understand publication and reference number equal prestige equal job security, so these are understandable inclinations. But academic integrity has to extend beyond the students if the institution wants to continue to be regarded as a source (let alone the source) for truth.
Agreed! There's a deep irony in everyone getting upset about students using AI to write their essays and then doing the same thing with their own papers. I'm currently on a grant selection panel for a national grant agency, and it's clear that AI has been used for virtually all the grant proposals (in fact, those applications where it hasn't been used stand out like the proverbial dog's bollocks). The speed at which it's been embedded in academia at every level is shocking.
It’s also important to remember that many academics are shit writers, and I say this as an academic and creative writer though I don’t like the binary of academic and creative.
What's striking about academics getting upset about a commonsensical policy from ArXiv is that it's exactly the same problem as students using AI to cheat: The incentive is to appear to have done good work, so you can get the legitimations you need for your career, rather than to actually do good work either because that's important to you or because you're actually going to be assessed rigorously on that. Broken incentive structures all the way around. Luckily, while that's not easy to fix, it's still a bit easier to address than trying to put the AI cat back in the bag
I think there's a bit of Emperor's New Clothes going on with a lot of the AI stuff. With the Shy Girl book I remember someone telling me they had read the self-published version and thought it was bad, but then it got praise and it was a Black writer and so they didn't want to say anything negative. Similarly in this case I bet one of the judges went "maybe in X region of the world such a metaphor makes sense!" or "Naturally people from X are super metaphoric" without pausing to think whether it was plain wonky. I'm pretty sure tokenizing and exotifying helped shaped both of these cases. We need to get a bit more diligent but also a bit more honest in our assessments.
Yep, I would not be surprised if that was happening here.
I gotta laugh at the AI story quotes because I see that kind of shit everywhere, and honestly it makes me question myself as a writer, native speaker of English and avid reader. I keep thinking these metaphors are different in some way and maybe I'm not as clever as I thought I was. But then I wonder where this kind of language exists in prose -- is it borrowed from a genre? No, probably not, but again, probably so, if you think about it.
AI writing is here because people want to produce works of their own -- and this is my argument -- because selling books has become more important than writing them. Marketing, selling, metrics, rankings, followers -- it's about attention and self-esteem. I am worthy because I sold X number of books this month or have X number of subscribers. Let's don't pretend that's not real. I defy anyone telling me it's not. We've run off most of the accounts touting methods for getting more followers or how to make thousands selling stories, but that tainted atmosphere has fully manifested itself in this need to be seen -- seen through sales and followers -- in a quest for a book deal or some kind of recognition.
AI is in the book writing sphere because of this, which is why I keep saying I wish more writers would write about writing and stop trying to sell me books. Hey, my stories are out there, too, but if you never find and read them, it's cool. My joy is in writing them. I'm not in it for the cash or temporary fame.
Honestly, unless you’re making a business out of it, I don’t think read numbers ought to have much appeal. One deep and thoughtful comment is worth a million silent reads to me.
To follow up:
https://sjstone.substack.com/p/proof-purity-and-the-death-of-the
“who cares if sentences mean anything?” — basically an appraisal like this is why Trump is doing a second lap in the WH.
Fabulously disturbing and kookie piece.
“Thank you,” he said as a tiny bird exited his nostril and caused the earth to rattle like a dime in a Coke machine.
BUT! He's "A prolific poet and author, with books published and others forthcoming, he is particularly known for his love of poetry." Lol, how did this bio even get past someone?
It seems sadly (even tragically) inevitable that AI, which has no taste, will become a literary tastemaker. Those who learned to read, write, and appreciate good writing before AI will continue to do so. But kids who come up with AI parading as literature, AI writing their papers, and AI analyzing the literature they’re assigned to read will have a hard time differentiating between AI slop and human-authored writing. It feels bleak. I think schools must respond by having students read good books in class—aloud and together.
AI may have no taste, but writes pretty well when answering questions. The sentences are direct, short, with minimal use of adjectives, etc. very much like the writing books teach. That's better than most people are able to do, including, sadly, most writers.
Also, I have some incredible updates re: James D. Miller. A) He coauthored a 2016 pamphlet/debate transcript on global trade with fucking VOX DAY. B) He is in deep with the Longtermist/A.I. doomer crowd, and has ONLY written in those publications recently. His last conventionally peer reviewed article was published in *2011*. So his reference to "publishing 20 papers a year" is pure fantasy. C) He also alludes to graduate students, but Smith College doesn't have a PhD program in economics. https://x.com/davidzmorris/status/2057121156772020497
“misattributed and invented quotes" Quotes might be misattributed, given AI is a huge liar. After all, I caught it taking credit for some Elvis lyrics. But invented? No. AI can't invent anything. It can only data scrape and repeat.
That is such HORRID writing, and so AI. It won a prize? What the hell? Was the judge AI?