Before we get into the nitty-gritty of this article, we should define some terms. AI is the acronym for artificial intelligence. This is a catch-all term that is often a misnomer. True artificial general intelligence (AGI) is a long way away from where technology stands today—if it’s ever realized at all.
What Is “AI”?
What most people mean today are chatbots, searchbots, large language models (LLMs), image and video generators, sound (speech and music) generators, or some combination. You’ve almost certainly interacted with an “AI” already. Many companies now deploy chatbots for front-line customer service, tuned to their own products and services. They can “understand” and answer questions in common language, with varying degrees of success.
For this article, I’m focusing on the large language model, or LLM. The most recognized today is ChatGPT, and it makes headlines constantly—often because of confusion about what these models actually are.
An LLM’s ability to converse naturally and even adapt to a user’s tone can create the illusion of sentience, awareness, or expertise. But it’s none of those things. An LLM is software running on immense hardware, trained on vast amounts of publicly available text and images. All of that raw data gets digested into tokens, fragments of language that help the model predict context and generate a statistically plausible response.
Context is key. And context is also where things can go off the rails.
LLMs Don’t Live Up to the Hype
The hype machine has done these tools no favors. Some hail LLMs as the dawn of artificial general intelligence. Others issue dire warnings. In reality, they’re neither.
What LLMs aren’t:
- They’re not doctors, therapists, or lawyers.
- They’re not authors, playwrights, or journalists.
- They’re not substitutes for trained, educated professionals.
Yet they’re marketed like they could be. Companies have invested billions and need returns. That means overselling capabilities—and pushing these tools into the zeitgeist without much public literacy about how they actually work.
Where They Fail
When you use an LLM like ChatGPT, it takes your input prompt, runs statistical probabilities, and generates its best guess at a good response. That’s it. The better your prompt, the better the output.
But:
- They agree with anything. If two people debate opposite sides of an issue, ChatGPT will mirror both, because it’s designed to be agreeable.
- They forget fast. Every conversation has a “context window,” a finite number of tokens the model can track. Once you exceed it, earlier parts fall away. Long sessions can feel like talking to someone whose attention is slipping.
- They hallucinate. When an LLM makes something up and delivers it with confidence, that’s a hallucination. Lawyers have been sanctioned for filing legal briefs full of hallucinated case law. Academic journals are flooded with AI-generated submissions citing papers that don’t exist.
That’s why you should never confuse eloquence with reliability. An LLM doesn’t know anything—it just predicts what sounds like it knows something.
Where They Shine
So why use them at all? Because in the right lanes, they’re powerful.
- Organization: Great for cleaning up messy drafts, smoothing flow, and catching when you’ve repeated yourself or introduced something out of order.
- Editing: They won’t replace a line editor, but they can clear the underbrush—grammar slips, punctuation, clunky syntax—so your human editor can focus on substance.
- Outlining: LLMs are superb scaffolding assistants. They won’t generate your ideas, but they can suggest what’s missing, or provide new angles you hadn’t considered.
- Breaking Writer’s Block: This is where they shine brightest. Tell it your half-baked idea, and it will riff back. Think of it as a brainstorming partner with no ego. Writers talk about “plot bunnies”—those pesky story ideas that hop into your head uninvited. An LLM is a plot bunny on steroids. It never stops generating, and while most of its ideas won’t stick, the occasional gem is exactly what you need to move forward.
I once co-wrote a short story with ChatGPT called The Night the Lights Went Out on JD. It was patently insane, hilarious, and adorable—pure chaos energy. That’s the point: left to run wild, LLMs can produce manic, unpredictable brilliance. But the writer still has to choose what’s worth keeping.
When to Use Them
Timing matters. LLMs are most helpful when the work is either stuck or messy.
- Early stages: brainstorming, scaffolding outlines, and kicking over writer’s block.
- Mid-draft: reorganizing wandering paragraphs, smoothing transitions, tightening flow.
- Late stages: grammar, punctuation, and consistency cleanup before a human editor takes over.
In other words: use them when you need momentum or polish, not when you need substance.
Where They Belong (and Where They Don’t)
LLMs belong in the writing process as assistants, not as ghostwriters.
- They belong in creative work — riff sessions, character quirks, playful prose, “plot bunny on steroids” chaos.
- They belong in nonfiction as scaffolding tools — helping catch redundancies, reorganize chapters, and polish language.
- They do not belong in research-heavy fields like law, medicine, or academia, where accuracy and accountability matter more than fluency. And they absolutely don’t belong in therapy or journalism, where human judgment and empathy can’t be simulated.
Why They’re Useful
For all their faults, the “why” is straightforward:
- Speed: draft iterations in minutes, not days.
- Relief: they take the edge off the drudgery of grammar and syntax.
- Access: they democratize editorial support for people who don’t have the money or network to hire one.
- Creativity: the manic energy of an LLM riff session can shake a writer loose when nothing else works.
The caveat: they’re useful because you are steering them. Left to their own devices, they’ll happily veer into hallucination.
How to Use Them Wisely
- Treat their output as raw material, not gospel.
- Always fact-check — dates, quotes, citations, stats.
- Use prompts to guide tone and structure, but never outsource your voice.
- Think of the LLM as a very smart, very overeager intern: helpful, prolific, sometimes brilliant — but not someone you’d trust to ship the final draft without review.
The Bottom Line
LLMs aren’t muses, and they’re not authors. They’re assistants. Like any assistant, they’re only as valuable as the care, creativity, and judgment of the person using them.
Treat them as tools, and they can make you faster, sharper, and more organized. Treat them as replacements, and they’ll burn you with hallucinations and false confidence.
In the end, writing with machines is about remembering who holds the pen.
