Too small number of new cases?

[gbugmann]

Hello Tom,
I have made a bubble map animation in Python of the daily new cases based on your data in covid-19-cases-uk.csv. It works nicely (see one frame below), but the number of new cases, based on the difference of the totals for successive days is quite smaller than the numbers announced in the news. The total is also smaller. For instance on June 3:
the data show 275 new cases, but the news mention 1871.
the data show a total of 180K and new news around 280K.
Do you have an idea where the discrepancies come from? Are your data missing the pillar II for England? Could this account for the discrepancies?
(note that I did some beautifying of the data, replacing nan's with the last known value for the same area - see attached)
Sorry if that issue been raised before.
Thanks by the way for the brilliant work in collecting the data.

Case_history_with_analysis.zip

[gbugmann]

OK. I saw your comments on page http://tom-e-white.com/datavision/20-where-are-the-coronavirus-cases.html. These missing pillar II data would really give us a better picture of where cases keep popping up despite the lockdown...maybe the ONS has access?

[tomwhite]

Thanks @gbugmann. I don't think ONS do have the data, if indeed anyone does, see https://www.statisticsauthority.gov.uk/correspondence/sir-david-norgrove-response-to-matt-hancock-regarding-the-governments-covid-19-testing-data/

[gbugmann]

At least other people are asking too...

[gbugmann]

I checked today https://www.theguardian.com/world/2020/jun/05/coronavirus-uk-map-the-latest-deaths-and-confirmed-cases-near-you and summed up the local cases that included places in England, Scotland, Wales and NI. I obtained 180,805 whereas the national total on the same site is: 281,661. So, the Guardian (and the BBC too) miss 100K local cases....

[gbugmann]

Everyone is asking for Pillar II data....
https://www.theguardian.com/politics/2020/jun/23/no-10-not-sharing-covid-19-data-on-local-outbreaks-say-councils
https://www.bedfordindependent.co.uk/johnson-accused-of-dodging-pillar-2-data-query-at-prime-ministers-questions/