nis-py-api

Python API to the National Immunization Survey (NIS) data.

🚧 This tool is in alpha development. The API and data schema are not stable.

Getting started

• This a poetry-enabled project.
• Get a Socrata app token. Copy scripts/secrets_template.yaml to scripts/secrets.yaml and fill out the app_token.
• See scripts/demo.py for an example of how to cache and query the data:
  • nisapi.cache_all_datasets() to download, clean, and cache data
  • nisapi.get_nis() to get a lazy data frame pointing to that locally cached, clean data
  • nisapi.delete_cache() to clear the cache, if needed
• See scripts/demo_clean.py for an example of a script that you could run while iteratively developing the cleaning code in nisapi/clean/.
• See scripts/demo_cloud.py for a demo of how the data could be downloaded, cleaned, uploaded to Azure Blob Storage, and then downloaded from there. You will need to fill out the azure: keys in secrets.yaml.
• run streamlit run scripts/demo_streamlit.py to quickly query and visualize the data with a streamlit app.

Data dictionary

The data have these columns, in order, with these types:

column	type
vaccine	String
geographic_type	String
geographic_value	String
demographic_type	String
demographic_value	String
indicator_type	String
indicator_value	String
time_type	String
time_start	Date
time_end	Date
estimate	Float64
lci	Float64
uci	Float64

Note the paired use of "type" and "value" columns.

Rows that were suppressed in the raw data are dropped. This includes data with suppression flag "1", indicating small sample size, and data with flag ".", which may indicate that data were not collected.

vaccine

• One of "flu" or "covid"

geographic_type

• One of "nation", "region", "admin1", "substate"
• "Region" means HHS Region
• First-level administrative divisions include US states, territories, and the District of Columbia

geographic_value

• If geographic_type is "nation", then this is "nation"
• If "region", then a string of the form "Region 1"
• If "admin1", then the full name of the jurisdiction
• If "substate", no validation is currently applied

demographic_type

• There are multiple types, including "overall" and "age"
• Note that "overall" might refer only to certain age groups (e.g., 18+). See the dataset metadata for the relevant universe.

demographic_value

• If demographic_type is "overall", then this is "overall"
• If demographic_type is "age", then this is the age group, with the form "x-y years" or "x+ years"

indicator_type

• In newer data, this is always "4-level vaccination and intent"
• In historical COVID-19 data, there are a wide range of indicators

indicator_value

• The value of the indicator, e.g., "received a vaccination"

time_type

• One of "month" or "week"

time_start and time_end

Period of time associated with the observation. Note that "monthly" and "weekly" observations do not always align with calendar weeks or months, so we specify the two dates explicitly.

• Time start is always before time end

estimate

• Proportion (i.e., a number between 0 and 1) of the population (defined by geography and demography) that has the characteristic described by the indicator

lci and uci

• The lower and upper limits of the 95% confidence interval, measured in the same units as estimate
• Confidence interval always bracket the estimate

Contributing

Adding a new dataset

1. Find the dataset you want to clean on https://data.cdc.gov/.
2. Add the dataset to nisapi/datasets.yaml.
   • At a minimum, you must include the dataset ID.
   • It is helpful to also include URL, vaccine, date range, and universe (i.e., what "overall" demography means).
3. Create a dataset-specific module in nisapi/clean/. It should have a main function clean().
   • Start with a clean() function that does nothing and just returns the input data frame.
4. Add the import and elif statements for this dataset ID to clean_dataset() in nisapi/clean/__init__.py.
5. Run scripts/clean_demo.py. This should cache the raw dataset, run the cleaning function, and fail on validation.
6. Iteratively update the dataset-specific clean() function until validation passes.
   • Ideally, clean() should be a series of pipe functions.
   • If a cleaning step is specific to a single dataset, keep that in the dataset-specific submodule. If a step is shared between datasets, move it into helpers.py.
   • If multiple indicators are redundant, validate that redundancy in code, and then pick only one indicator. (E.g., ksfb-ug5d and sw5n-wg2p drop the up-to-date indicator in favor of the 4-level vaccination intent indicator.)
   • If you find some dataset-specific anomaly or validation problem, make a note of it in datasets.yaml.
7. Open a PR.
   • Include any validations if you needed to correct an anomaly.

Standard Notice

Anyone is encouraged to contribute to the repository by forking and submitting a pull request. (If you are new to GitHub, you might start with a basic tutorial.) By contributing to this project, you grant a world-wide, royalty-free, perpetual, irrevocable, non-exclusive, transferable license to all users under the terms of the Apache Software License v2 or later.

All comments, messages, pull requests, and other submissions received through CDC including this GitHub page may be subject to applicable federal law, including but not limited to the Federal Records Act, and may be archived. Learn more at http://www.cdc.gov/other/privacy.html.

Project Admin

• Scott Olesen ulp7@cdc.gov (CDC/CFA)

General Disclaimer

This repository was created for use by CDC programs to collaborate on public health related projects in support of the CDC mission. GitHub is not hosted by the CDC, but is a third party website used by CDC and its partners to share information and collaborate on software. CDC use of GitHub does not imply an endorsement of any one particular service, product, or enterprise.

Public Domain Standard Notice

This repository constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. This repository is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication. All contributions to this repository will be released under the CC0 dedication. By submitting a pull request you are agreeing to comply with this waiver of copyright interest.

License Standard Notice

This repository is licensed under ASL v2 or later.

Privacy Standard Notice

This repository contains only non-sensitive, publicly available data and information. All material and community participation is covered by the Disclaimer and Code of Conduct. For more information about CDC's privacy policy, please visit http://www.cdc.gov/other/privacy.html.

Records Management Standard Notice

This repository is not a source of government records but is a copy to increase collaboration and collaborative potential. All government records will be published through the CDC web site.