Lessons Learned Writing a Book Collaboratively with LLMs
(Note: I'm not linking the resulting book. This post focuses solely on the process and practical lessons learned collaborating with LLMs on a large writing project.)
Hey HN, I recently finished a months-long project collaborating intensively with various LLMs (ChatGPT, Claude, Gemini) to write a book about using AI in management. The process became a meta-experiment, revealing practical workflows and pitfalls that felt worth sharing.
This post breaks down the workflow, quirks, and lessons learned.
Getting Started: Used ChatGPT as a sounding board for messy notes. One morning, stuck in traffic, tried voice dictation directly into the chat app. Expected chaos, got usable (if rambling) text. Lesson 1: Capture raw ideas immediately. Use voice/text to get sparks down, then refine. Key for overcoming the blank page.
My Workflow evolved organically: Conversational Brainstorming: "Talk" ideas through with the AI. Ask for analogies, counterarguments, structure. Treat it like an always-available (but weird) partner. Partnership Drafting: Let AI generate first passes when stuck ("Explain X simply for Y"), but treat as raw material needing heavy human editing/fact-checking. Or, write first, have AI polish. Often alternated. Iterative Refinement: The core loop. Paste draft > ask for specific feedback ("Is this logic clear?") -> integrate selectively > repeat. (Lesson 2: Vague prompts = vague results; give granular instructions. Often requires breaking down tasks: logic first, then style). Practice Safe Context Management: LLMs forget (context windows). (Lesson 3: You are the AI's external memory. Constantly re-paste context/style guides; use system prompts. Assume zero persistence across time). Read-Aloud Reviews: Use TTS or read drafts aloud. (Lesson 4: Ears catch awkwardness eyes miss. Crucial for natural flow).
The "AI A-Team": Different models have distinct strengths: ChatGPT: Creative "liberal arts" type; great for analogies/prose, but verbose/flattery-prone. Claude: Analytical "engineer"; excels at logic/accuracy/code, but maybe don't invite for drinks. Gemini: The "copyeditor"; good for large-context consistency. Can push back constructively. (Lessons 5 & 6: Use the right tool for the job; learn strengths via experimentation & use models to check each other. Feeding output between them often revealed flaws - Gemini calling out ChatGPT's tells was useful).
Stuff I Did Not Do Well:
Biggest hurdles:
AI Flattery is Real: Helpfulness optimization means praise for bad work. (Lesson 7: Prompt for critical feedback. 'Critique harshly'. Don't trust praise; human review vital). The "AI Voice" is Pervasive: Understand why it sounds robotic (training bias, RLHF). (Lesson 8: Combat AI-isms. Prompt specific tones; edit out filler/hedging/repetition/'delve'; kill em dashes unless formal). Verification Burden is HUGE: AI hallucinates/facts wrong. (Lesson 9: Assume nothing correct without verification. You are the fact-checker. Non-negotiable despite workload. Ground claims; be careful with nuance/lived experience). Perfectionism is a Trap: AI enables endless iteration. (Lesson 10: Set limits; trust judgment. Know 'good enough'. Don't let AI erode voice. Kill your darlings).
My Personal Role in This fiasco:
Deep AI collaboration elevates the human role to: Manager (goals/context), Arbitrator (evaluating conflicts), Integrator (synthesizing), Quality Control (verification/ethics), and Voice (infusing personality/nuance).
Conclusion: This wasn't push-button magic; it was intensive, iterative partnership needing constant human guidance, judgment, and effort. It accelerated things dramatically and sparked ideas, but final quality depended entirely on active human management.
Key takeaway: Embrace the mess. Capture fast. Iterate hard. Know your tools. Verify everything. Never abdicate your role as the human mind in charge. Would love to hear thoughts on others' experiences.
Thanks for posting this, it's a very interesting case study. Considering that the thing they seem to excel at is this type of writing, it's interesting that they still seem to be only ok at it if you're trying to produce a serious, genuinely useful output. This fits with my experience, though yours is much more extensive and thorough. In particular I fully concur with the voice/tone, and the need to verify everything (always the case anyway), and "Never abdicate your role as the human mind in charge" -- sometimes the suggestions it makes are just not that good.
Question is, do you think this process was faster using the various LLMs? Could two (or N) sufficiently motivated people produce the same thing in the same time? (and if so, what is N). I'm wondering if the caveats and limitations end up costing as much time as they save. Maybe you're 2x faster, if so that would be significant and good to know.
In the abstract, this is similar to my experience with AI produced code. Except for very simple, contained code, you ultimately, need to read and understand it well enough to make sure that it's doing all the things that you want and not producing bugs. I'm not sure this saves me much time.
Nice. I leverage the strengths of AI in a way that affirms the human element in the collaboration. AI as it exists in LLMs is a powerful source of potentially meaningful language but at this point LLMs don't have a consistent conscious mind that exists over time like humans do. So it's more like summoning a djinn to perform some task and then it disappears back into the ether. We of course can interweave these disparate tasks into a meaningful structure and it sounds like you have some good strategies for how to do this.
I have found that using an LLM to critique your writing is a helpful way of getting free generic but specific feedback. I find this route more interesting than the copy pasta AI voiced stuff. Suggesting that AI embodys a specific type of character such as a pirate can make the answers more interesting than just finding the median answer, add some flavor to the white bread.
One of the things I found helpful about getting out of the specific / formulaic feedback was asking the LLM to ask me questions. At one point I asked a fresh LLM to read the book and then ask me questions. It showed me where there were narrative gaps / confusing elements that a reader would run into, but didn't realy on the specific "answer" from the LLM itself.
I also had a bunch of personal stories interwoven in and it told me I was being "indulgent" which was harsh but ultimately accurate.
That's a great approach. I find LLMs work really well as Socratic sounding boards and can lead you as the writer to explore avenues you might have otherwise not even noticed.
Given that humans are 'wired for story', perhaps you should consider indulging. These could be what makes the books stand out after all.