Part 5 (Improved Revision)

Introduction

If you haven't seen the previous parts of this series, it might be worth going and reading those first.

Previously, I focused on adding a scrubbing step that would remove any editor notes from the final output, with mixed results.

Approach - Staged Chapter Generation

After conferring again with my Writing teacher at my university, we had an idea - to split up chapter generation into several stages, with a different LLM for each stage. In theory, it might result in better chapter outputs, where each LLM would only have to focus on a specific part of the generation process.

For example, it might go something like this for each chapter:

LLM1 -> Generate the initial chapter from the story outline
LLM2 -> Focus on adding some character development to the chapter
LLM3 -> Make sure the pacing is consistent in the chapter
LLM4 -> Fix any issues with the writing style, make sure the tone is right for the scene, etc.
LLM5 -> Add dialogue if it's not in there enough

In practice, that updated the block diagram to be as follows:

Results

This did yield a massive improvement in output quality - there was a very dramatic bump in both plot quality and writing style. For the most part, the LLM didn't really focus on generating a lot of detail for the sky and none on important scenes, the integrated revision and staged chapter generation seemed to fix that. There were still issues of course, it seemed that chapters were having issues flowing from one into the next, which was definitely hindering the output quality. An unfortunate side effect of this added complexity also meant that the LLMs would sometimes generate output that simply said "Good luck writing your chapter" or something along those lines.

During testing, I also found that it was best to disable chapter revision entirely, and instead only have that for the outline. I'm honestly not sure why that is, perhaps the chapter revision prompts weren't quite right but either way, the results were better with it disabled.

I did also manage to fix the above mentioned chapter flow issues somewhat by tweaking the way contexts were handled between chapters so that the LLMs always see the most important information at the bottom of the thread. There is of course still room for improvement here, as it will sometimes loose track of the chapters but that seems to be more caused by the LLMs writing "Good Luck!" as an entire chapter's worth of content, messing up subsequent chapters.

Conclusion

I think that this improvement definitely helped things overall, but there were still some noticeable issues with the output still. In the next attempt, I wanted to try adding some extra revision to fix the above mentioned "Good Luck!" issue, which was the next biggest problem. With that in mind, I focused on that next.