We are not the only site that identifies classics by statistically processing book lists. It’s a technique that’s catching on. However, we realized a problem with this system when creating version 4 of our list. All books can’t potentially be on all lists. If a fan poll was conducted in 1988, any title published after that date never had the chance of being on that poll. Or, books by male writers shouldn’t be on lists of all-time best books written by women writers.

The Left Hand of Darkness by Ursula K. Le Guin was on 43 of the 65 lists we found, the most for any book. In our final rankings, Dune by Frank Herbert and A Canticle for Leibowitz by Walter M. Miller Jr. came in higher than The Left Hand of Darkness, even though they were only on 39 and 37 total lists. What we decided as fair, was ranking the books by the percentage of lists they could have been on. The Herbert and Miller books were on 95% of the lists they could have been on, whereas the Le Guin book was only on 90% of the books it could have been on.

Where we show the books in order by rank, we show both the percentage, and the number of lists the book was on/could have been on.

This use of percentages also allowed us to use eight lists devoted entirely to women writers. We freely admit to wanting to see more women writers on version 4 of the Classics of Science Fiction, and we hope we have done it in a fair way.

Every time we make a new version of our list we hope to see more newer titles. We’ve discovered it takes time for a book to be remembered and time for it to be forgotten. If you look at the table Versions 1-4, you’ll see color coding. Blue are books new to version 4. Red are books that didn’t make it to version 4. There’s a good number of red titles, and a fair number of blue. That’s how it should be, but it’s still a shame how quickly books are forgotten.

Another area we had trouble mathematically was counting books belonging to trilogies or series. Sometimes a single book got on a list, and sometimes the name of the series. The most famous example is The Foundation Trilogy. We counted it as a single title. There have been omnibus editions of The Foundation Trilogy for decades, and we think many fans think of it as a whole. In other cases, like with Gene Wolfe, we kept both entries for The Shadow of the Torturer and The Book of the New Sun. We couldn’t decide if the voters loved just the first book in the series or the four-book series. To confuse the issue, the four books are now published in two volumes, with new titles for former pair of titles.

Using statistics to identify the best books is hard on anthologies and series books. Some writers, like Ray Bradbury, Isaac Asimov, and Clifford D. Simak, have collections of short stories that have retained their titles, and their fame. Probably because they became fix-up novels. Other writers, like Cordwainer Smith, James Tiptree Jr., Zenna Henderson, have had their stories often repackaged with new titles. However, overall readers don’t seem to remember collections or anthologies. And a series like the Vorkosigan Saga by Lois McMaster Bujold, have so many books that fans love, that votes for the best books have been divvied between them, weakening their chance to get on the list.

Popular prolific authors had the same problem. Fans and critics are divided over favorites. Because we used 65 lists, with a cutoff of being on 10 lists, any book on the Classics of Science Fiction list is well remembered. But if an author has a dozen well-remembered books, it lowers the odds of many of those titles being on ten or more of the 65 lists.

We’d love to find other methods for identifying well loved books. It would be fantastic to have sales figures for the last two centuries. I wish Goodreads would provide lists of books by rating, number of raters, and number of reviewers. For example, 4,442,480 Goodreads users have The Hunger Games in their library and rated it, with an average rating of 4.36. 149,935 readers took the time to write a review. It would be fantastic to see the Top 1,000 books as rated by Goodreads users.

We know fans studying our list will often fail to discover their favorite novel on our list. We did not pick these books. Books on the list come from a statistical system. We think it’s a valid system. We hope our system works to indicate which science fiction are remembered over time.

We believe classic novels are those stories which endure the test of time.