Fortunately I've had a lot of practice genrefying when I used to work for Realms of Fantasy, the short fiction magazine. I was one of their editors and part of my job was helping authors to classify their stories so they market them to the right publications. And I really enjoy figuring out what genres can be applied to a book, believe it or not. Applying genres to a book is really similar to listing its subject headings.
It isn't always foolproof, but I ask myself the following questions when I'm trying to genrefy a book, and they usually do the trick for me:
In which section will this book likely be read MORE?
I can't speak for anyone else, but the main reason I'm genrefying my library is the promotion of pleasure reading. My students don't search (or research) fiction books. They don't have the time. If they're in here for a class and are using my catalog, almost invariably they are searching through my non-fiction, which will remain dewey. However, it just isn't their culture to use a catalog for pleasure reading. So, if I'm struggling between two (or five) genres that a book could be classified in, I would place it in the section that would most promote it. In our catalog the book will still have subject headings for ALL of the genres it technically meets, so if a student wants a list, I can provide one - or they can find it themselves. But the book would still circulate more than it did before.
How high up in the hierarchy is the genre?
When in doubt, place higher. If you want to get technical, ALL of fiction can be placed in one of two categories: realistic or speculative. EVERYTHING else falls in one of those two. A western is historical fiction that is also realistic fiction. Any realistic fiction, if you wait long enough, becomes historical fiction. Urban fiction is realistic fiction, unless it's set in the future of course. Horror, supernatural, paranormal, science fiction, and fantasy, all fall under the umbrella of speculative. Sometimes seeing the relationships helps me make the judgment call on where the book should go. In my next post, I'm going to make an infographic that visually lays out which genres go under which umbrellas. I think that will be fun to do and informative. (Yeah, I know, I'm weird).
If I take that genre OUT of the book, would the story be irreparably changed?
If the answer is YES, then the book is probably that genre. This question really works for science fiction. Generally, if you take the tech out of science fiction, the story just doesn't work anymore. That can help you decide where a book should end up. It doesn't always work, but it's a useful tool in my box of questions. You have to make sure you're looking at the plot though as opposed to the theme. Let's take a look at Hunger Games. Hunger Games is a dystopian science fiction post-apocalyptic romance story lol. I haven't finished making my decisions on my genre sections, but I would place probably it in science fiction. Post-apocalyptic is a sub-genre of science fiction. Dystopian is often science fiction as well. Romance spans all genres. Then again, if I had A LOT of post-apocalyptic fiction, it might warrant creating a section just for that, especially if I think that would get the book read more. But, in my opinion, if you got rid of the technology necessary for that war dome they fight in, and the tech they use to divide and monitor their society, it wouldn't be the same story at all.
In the end, I don't worry so much about what is the right category, as what will get the book in the most hands.