Error when looking up high-numbered pages

[divergentdave]

I was alerted to an increase in 5xx error code by Google's search console. The offending URLs were all /reports.xml with high page arguments (over 300 or so). I reproduced the issue locally and got this output.

[query_phase_execution_exception] Result window is too large, from + size must be less than or equal to: [10000] but was [30000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting. :: {"path":"/oversight_read/reports/_search","query":{},"body":"{\"from\":29950,\"size\":50,\"query\":{\"bool\":{\"must\":{\"query_string\":{\"query\":\"*\",\"default_operator\":\"AND\",\"use_dis_max\":true,\"fields\":[\"text\",\"title\",\"summary\",\"pdf.title\",\"pdf.keywords\",\"doc.title\",\"docx.title\",\"docx.keywords\"]}}}},\"sort\":[{\"published_on\":\"desc\"}],\"highlight\":{\"encoder\":\"html\",\"pre_tags\":[\"<b>\"],\"post_tags\":[\"</b>\"],\"fields\":{\"text\":{},\"summary\":{}},\"order\":\"score\",\"fragment_size\":500},\"_source\":[\"report_id\",\"year\",\"inspector\",\"agency\",\"title\",\"agency_name\",\"url\",\"landing_url\",\"inspector_url\",\"published_on\",\"type\",\"file_type\",\"featured.author\",\"featured.author_link\",\"featured.description\",\"unreleased\",\"missing\"]}","statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"query_phase_execution_exception\",\"reason\":\"Result window is too large, from + size must be less than or equal to: [10000] but was [30000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"oversight-20170312\",\"node\":\"nMHMUiAjRLWeSvkbOso6Ag\",\"reason\":{\"type\":\"query_phase_execution_exception\",\"reason\":\"Result window is too large, from + size must be less than or equal to: [10000] but was [30000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.\"}}],\"caused_by\":{\"type\":\"query_phase_execution_exception\",\"reason\":\"Result window is too large, from + size must be less than or equal to: [10000] but was [30000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.\"}},\"status\":500}"}