Deduplication of results

[DavidGarfinkle]

The prefix of all results are also returned. in another sense, results that are too similar to higher-ranking results are taking priority over novel results

decide on which ones we will omit, and provide options for including them

a quote from how google solves this problem
"In order to show you the most relevant results, we have omitted some entries very similar to the 80 already displayed.
If you like, you can repeat the search with the omitted results included.
"