Constraints, Genre and Fiction in a Box
The most powerful tool a writer has in their box is the use of constraints, Genre, and placing limitations on themselves. Despite the way it might sound, limitation is the fertilizer of creativity.
Limitations...
...prevent the writer from going wild and over cluttering a story or setting. Imagine if limitations had been placed on the X-Files to keep the writers from crafting the often contradictory stories that made it to air.
...focus the writer so we have to develop and flesh out those things that we are allowed to have in the setting. This brings depth and clarity to the story.
...give the story a sense of reality. We are used to living with our limitations, and as the reader discovers the ones in our stories they come to understand what is and is not possible in the story.
...force creative implementations of the ideas we have allowed into the story. The only magic available in Avatar: The Last Airbender is element bending, so the writers integrated bending into every aspect of the setting and came up with creative uses of bending that are not obvious.
...help the writer plot the story. When writing Speculative Fiction, it is easy to get lost in the possible ways to accomplish every task in the tale. Limits clear the brush and make the way more apparent.
Limitations only work if they are carefully and deliberately chosen. If chosen carefully, they can even be a powerful way to find new stories to write.
Genre
Genre is the first limitation to pick. I know everyone says that, but no one explains why.
Your choice of Genre will immediately define you type of setting and the type of stories that can exist in that world. Do not pick an open Genre, drill down and find the one that fits what you are wanting to write in.
Take "Dune by Frank Herbert. There are many genres that books could have been written in, each would have changed the story immensely. It is the genre that makes that book what it is.
Speculative Fiction
Science Fiction
Soft Science Fiction
Space Opera
Galactic Empire
Planetary Romance
The entire setting and the majority of the plot is dictated by this choice of genre. Imagine how the story would have changed if instead of a Planetary Romance, he had chosen to do a Technofantasy, or if he had chosen to made the story Hard Science Fiction, limiting it to known physics. The entire story would change.
How do you find these subgenres? Wikipedia has a good number of them listed. I use the Encyclopedia of Science Fiction and the Encyclopedia of Fantasy, neither of which are still in print, but if you can get your hands on one, I would highly recommend them.
Take time and carefully pick yours before you move on.
Constraints
Once you have chosen your Genre, list the constraints that it gives you.
Now add your own. Do aliens exist? What about Ghosts? Answer the questions that make sense for the type of story you are wanting to write. Be deliberate and keep your list of exclusions near you as you write.
Fiction in a Box
After you have built your box, figure out how to exploit it. Like Neo in the Matrix, you have to learn that the rules can be bent but not broken.
What cultures would develop under those rules?
How do those rules effect the characters life, profession, and the tools they have at their disposal?
How do the rules change the technology the characters use?
What impossible thing do the characters long to do?
How can I use that impossible thing in the setting?
In Harry Potter, magic can do anything but bring back the dead. That impossibility effects Voldemort, Dumbledor, and Harry.
In Dune, it is impossible to see the future, truth, or to even travel through space without the spice. Those limits and the characters fears and hopes related to them drive the story.
After you have established all of your limitations, you need to come up with a story that would be unexpected in that setting.
Harry Potter is set in a traditional fantasy setting where they are struggling to defeat a Dark Lord, but the books themselves are written as mysteries rather than quests.
Dune tells the story of a coming messiah, but the story is about the psychology of a boy turned man who grapples with his visions of the future trying to stop them from happening.
"Brave Men Run - A Novel of the Sovereign Era" by Matthew Wayne Selznick is a Superhero story that follows the relationship between a boy and his family.
Liquid Sky is a story about a boy trying to defy fate, but it is told from the point of view of a coming of age story.
"Shine Like Thunder" is a dark space opera about characters trying to survive after they are marooned on a demon filled world, but it is told as a romance/mystery.
When you establish a convention, the reader will feel comfortable in the setting. When you tell the story in a unique way, you are able to surprise the reader without making them feel the story betrayed its premise.