Evaluate certificates with zlint

[adamdecaf]

Similar to #187, #257, and #258, but to run zmap/zlint over the certificates.

This tool is pretty complete and has most of the BR checks included. (List: https://docs.google.com/spreadsheets/d/1ywp0op9mkTaggigpdF2YMTubepowJ50KQBhc_b00e-Y/edit#gid=0)

zlint looks to be gaining steam from (at least one) CA over their own tools. https://cabforum.org/pipermail/public/2018-April/013233.html Either way it's a really complete validator.

[adamdecaf]

The response from zlint is pretty easy to parse. I've included a full response:

$ zlint -pretty github.com.crt  | jq .
{
  "e_basic_constraints_not_critical": {
    "result": "NA"
  },
  "e_ca_common_name_missing": {
    "result": "NA"
  },
  "e_ca_country_name_invalid": {
    "result": "NA"
  },
  "e_ca_country_name_missing": {
    "result": "NA"
  },
  "e_ca_crl_sign_not_set": {
    "result": "NA"
  },
  "e_ca_is_ca": {
    "result": "NA"
  },
  "e_ca_key_cert_sign_not_set": {
    "result": "NA"
  },
  "e_ca_key_usage_missing": {
    "result": "NA"
  },
  "e_ca_key_usage_not_critical": {
    "result": "NA"
  },
  "e_ca_organization_name_missing": {
    "result": "NA"
  },
  "e_ca_subject_field_empty": {
    "result": "NA"
  },
  "e_cab_dv_conflicts_with_locality": {
    "result": "NA"
  },
  "e_cab_dv_conflicts_with_org": {
    "result": "NA"
  },
  "e_cab_dv_conflicts_with_postal": {
    "result": "NA"
  },
  "e_cab_dv_conflicts_with_province": {
    "result": "NA"
  },
  "e_cab_dv_conflicts_with_street": {
    "result": "NA"
  },
  "e_cab_iv_requires_personal_name": {
    "result": "NA"
  },
  "e_cab_ov_requires_org": {
    "result": "NA"
  },
  "e_cert_contains_unique_identifier": {
    "result": "pass"
  },
  "e_cert_extensions_version_not_3": {
    "result": "pass"
  },
  "e_cert_policy_iv_requires_country": {
    "result": "NA"
  },
  "e_cert_policy_iv_requires_province_or_locality": {
    "result": "NA"
  },
  "e_cert_policy_ov_requires_country": {
    "result": "NA"
  },
  "e_cert_policy_ov_requires_province_or_locality": {
    "result": "NA"
  },
  "e_cert_unique_identifier_version_not_2_or_3": {
    "result": "NA"
  },
  "e_distribution_point_incomplete": {
    "result": "pass"
  },
  "e_dnsname_bad_character_in_label": {
    "result": "pass"
  },
  "e_dnsname_contains_bare_iana_suffix": {
    "result": "pass"
  },
  "e_dnsname_empty_label": {
    "result": "pass"
  },
  "e_dnsname_hyphen_in_sld": {
    "result": "pass"
  },
  "e_dnsname_label_too_long": {
    "result": "pass"
  },
  "e_dnsname_left_label_wildcard_correct": {
    "result": "pass"
  },
  "e_dnsname_not_valid_tld": {
    "result": "pass"
  },
  "e_dnsname_underscore_in_sld": {
    "result": "pass"
  },
  "e_dnsname_wildcard_only_in_left_label": {
    "result": "pass"
  },
  "e_dsa_correct_order_in_subgroup": {
    "result": "NA"
  },
  "e_dsa_improper_modulus_or_divisor_size": {
    "result": "NA"
  },
  "e_dsa_params_missing": {
    "result": "NA"
  },
  "e_dsa_shorter_than_2048_bits": {
    "result": "NA"
  },
  "e_dsa_unique_correct_representation": {
    "result": "NA"
  },
  "e_ec_improper_curves": {
    "result": "NA"
  },
  "e_ev_business_category_missing": {
    "result": "pass"
  },
  "e_ev_country_name_missing": {
    "result": "pass"
  },
  "e_ev_locality_name_missing": {
    "result": "pass"
  },
  "e_ev_organization_name_missing": {
    "result": "pass"
  },
  "e_ev_serial_number_missing": {
    "result": "pass"
  },
  "e_ev_valid_time_too_long": {
    "result": "pass"
  },
  "e_ext_aia_marked_critical": {
    "result": "pass"
  },
  "e_ext_authority_key_identifier_critical": {
    "result": "pass"
  },
  "e_ext_authority_key_identifier_missing": {
    "result": "pass"
  },
  "e_ext_authority_key_identifier_no_key_identifier": {
    "result": "pass"
  },
  "e_ext_cert_policy_disallowed_any_policy_qualifier": {
    "result": "pass"
  },
  "e_ext_cert_policy_duplicate": {
    "result": "pass"
  },
  "e_ext_cert_policy_explicit_text_ia5_string": {
    "result": "NA"
  },
  "e_ext_cert_policy_explicit_text_too_long": {
    "result": "NA"
  },
  "e_ext_duplicate_extension": {
    "result": "pass"
  },
  "e_ext_freshest_crl_marked_critical": {
    "result": "NA"
  },
  "e_ext_ian_dns_not_ia5_string": {
    "result": "NA"
  },
  "e_ext_ian_empty_name": {
    "result": "NA"
  },
  "e_ext_ian_no_entries": {
    "result": "NA"
  },
  "e_ext_ian_rfc822_format_invalid": {
    "result": "NA"
  },
  "e_ext_ian_space_dns_name": {
    "result": "NA"
  },
  "e_ext_ian_uri_format_invalid": {
    "result": "NA"
  },
  "e_ext_ian_uri_host_not_fqdn_or_ip": {
    "result": "NA"
  },
  "e_ext_ian_uri_not_ia5": {
    "result": "NA"
  },
  "e_ext_ian_uri_relative": {
    "result": "NA"
  },
  "e_ext_key_usage_cert_sign_without_ca": {
    "result": "pass"
  },
  "e_ext_key_usage_without_bits": {
    "result": "pass"
  },
  "e_ext_name_constraints_not_critical": {
    "result": "NA"
  },
  "e_ext_name_constraints_not_in_ca": {
    "result": "NA"
  },
  "e_ext_policy_constraints_empty": {
    "result": "NA"
  },
  "e_ext_policy_constraints_not_critical": {
    "result": "NA"
  },
  "e_ext_policy_map_any_policy": {
    "result": "NA"
  },
  "e_ext_san_contains_reserved_ip": {
    "result": "pass"
  },
  "e_ext_san_directory_name_present": {
    "result": "pass"
  },
  "e_ext_san_dns_name_too_long": {
    "result": "pass"
  },
  "e_ext_san_dns_not_ia5_string": {
    "result": "pass"
  },
  "e_ext_san_edi_party_name_present": {
    "result": "pass"
  },
  "e_ext_san_empty_name": {
    "result": "pass"
  },
  "e_ext_san_missing": {
    "result": "pass"
  },
  "e_ext_san_no_entries": {
    "result": "pass"
  },
  "e_ext_san_not_critical_without_subject": {
    "result": "pass"
  },
  "e_ext_san_other_name_present": {
    "result": "pass"
  },
  "e_ext_san_registered_id_present": {
    "result": "pass"
  },
  "e_ext_san_rfc822_format_invalid": {
    "result": "pass"
  },
  "e_ext_san_rfc822_name_present": {
    "result": "pass"
  },
  "e_ext_san_space_dns_name": {
    "result": "pass"
  },
  "e_ext_san_uniform_resource_identifier_present": {
    "result": "pass"
  },
  "e_ext_san_uri_format_invalid": {
    "result": "pass"
  },
  "e_ext_san_uri_host_not_fqdn_or_ip": {
    "result": "pass"
  },
  "e_ext_san_uri_not_ia5": {
    "result": "pass"
  },
  "e_ext_san_uri_relative": {
    "result": "pass"
  },
  "e_ext_subject_directory_attr_critical": {
    "result": "NA"
  },
  "e_ext_subject_key_identifier_critical": {
    "result": "pass"
  },
  "e_ext_subject_key_identifier_missing_ca": {
    "result": "NA"
  },
  "e_generalized_time_does_not_include_seconds": {
    "result": "NA"
  },
  "e_generalized_time_includes_fraction_seconds": {
    "result": "NA"
  },
  "e_generalized_time_not_in_zulu": {
    "result": "NA"
  },
  "e_ian_bare_wildcard": {
    "result": "NA"
  },
  "e_ian_dns_name_includes_null_char": {
    "result": "NA"
  },
  "e_ian_dns_name_starts_with_period": {
    "result": "NA"
  },
  "e_ian_wildcard_not_first": {
    "result": "NA"
  },
  "e_inhibit_any_policy_not_critical": {
    "result": "NA"
  },
  "e_international_dns_name_not_nfkc": {
    "result": "pass"
  },
  "e_international_dns_name_not_unicode": {
    "result": "pass"
  },
  "e_invalid_certificate_version": {
    "result": "pass"
  },
  "e_issuer_dn_country_not_printable_string": {
    "result": "pass"
  },
  "e_issuer_field_empty": {
    "result": "pass"
  },
  "e_name_constraint_empty": {
    "result": "NA"
  },
  "e_name_constraint_maximum_not_absent": {
    "result": "NA"
  },
  "e_name_constraint_minimum_non_zero": {
    "result": "NA"
  },
  "e_old_root_ca_rsa_mod_less_than_2048_bits": {
    "result": "NA"
  },
  "e_old_sub_ca_rsa_mod_less_than_1024_bits": {
    "result": "NA"
  },
  "e_old_sub_cert_rsa_mod_less_than_1024_bits": {
    "result": "NA"
  },
  "e_path_len_constraint_improperly_included": {
    "result": "pass"
  },
  "e_path_len_constraint_zero_or_less": {
    "result": "pass"
  },
  "e_public_key_type_not_allowed": {
    "result": "pass"
  },
  "e_root_ca_extended_key_usage_present": {
    "result": "NA"
  },
  "e_root_ca_key_usage_must_be_critical": {
    "result": "NA"
  },
  "e_root_ca_key_usage_present": {
    "result": "NA"
  },
  "e_rsa_exp_negative": {
    "result": "pass"
  },
  "e_rsa_mod_less_than_2048_bits": {
    "result": "pass"
  },
  "e_rsa_no_public_key": {
    "result": "pass"
  },
  "e_rsa_public_exponent_not_odd": {
    "result": "pass"
  },
  "e_rsa_public_exponent_too_small": {
    "result": "pass"
  },
  "e_san_bare_wildcard": {
    "result": "pass"
  },
  "e_san_dns_name_includes_null_char": {
    "result": "pass"
  },
  "e_san_dns_name_starts_with_period": {
    "result": "pass"
  },
  "e_san_wildcard_not_first": {
    "result": "pass"
  },
  "e_serial_number_longer_than_20_octets": {
    "result": "pass"
  },
  "e_serial_number_not_positive": {
    "result": "pass"
  },
  "e_signature_algorithm_not_supported": {
    "result": "pass"
  },
  "e_sub_ca_aia_does_not_contain_ocsp_url": {
    "result": "NA"
  },
  "e_sub_ca_aia_marked_critical": {
    "result": "NA"
  },
  "e_sub_ca_aia_missing": {
    "result": "NA"
  },
  "e_sub_ca_certificate_policies_missing": {
    "result": "NA"
  },
  "e_sub_ca_crl_distribution_points_does_not_contain_url": {
    "result": "NA"
  },
  "e_sub_ca_crl_distribution_points_marked_critical": {
    "result": "NA"
  },
  "e_sub_ca_crl_distribution_points_missing": {
    "result": "NA"
  },
  "e_sub_cert_aia_does_not_contain_ocsp_url": {
    "result": "pass"
  },
  "e_sub_cert_aia_marked_critical": {
    "result": "pass"
  },
  "e_sub_cert_aia_missing": {
    "result": "pass"
  },
  "e_sub_cert_cert_policy_empty": {
    "result": "pass"
  },
  "e_sub_cert_certificate_policies_missing": {
    "result": "pass"
  },
  "e_sub_cert_country_name_must_appear": {
    "result": "NE"
  },
  "e_sub_cert_crl_distribution_points_does_not_contain_url": {
    "result": "pass"
  },
  "e_sub_cert_crl_distribution_points_marked_critical": {
    "result": "pass"
  },
  "e_sub_cert_eku_missing": {
    "result": "pass"
  },
  "e_sub_cert_eku_server_auth_client_auth_missing": {
    "result": "pass"
  },
  "e_sub_cert_given_name_surname_contains_correct_policy": {
    "result": "NA"
  },
  "e_sub_cert_key_usage_cert_sign_bit_set": {
    "result": "pass"
  },
  "e_sub_cert_key_usage_crl_sign_bit_set": {
    "result": "pass"
  },
  "e_sub_cert_locality_name_must_appear": {
    "result": "NE"
  },
  "e_sub_cert_locality_name_must_not_appear": {
    "result": "NE"
  },
  "e_sub_cert_not_is_ca": {
    "result": "pass"
  },
  "e_sub_cert_or_sub_ca_using_sha1": {
    "result": "pass"
  },
  "e_sub_cert_postal_code_must_not_appear": {
    "result": "NE"
  },
  "e_sub_cert_province_must_appear": {
    "result": "NE"
  },
  "e_sub_cert_province_must_not_appear": {
    "result": "NE"
  },
  "e_sub_cert_street_address_should_not_exist": {
    "result": "NE"
  },
  "e_sub_cert_valid_time_longer_than_39_months": {
    "result": "NE"
  },
  "e_sub_cert_valid_time_longer_than_825_days": {
    "result": "NE"
  },
  "e_subject_common_name_max_length": {
    "result": "pass"
  },
  "e_subject_common_name_not_from_san": {
    "result": "pass"
  },
  "e_subject_contains_noninformational_value": {
    "result": "pass"
  },
  "e_subject_contains_reserved_ip": {
    "result": "pass"
  },
  "e_subject_country_not_iso": {
    "result": "pass"
  },
  "e_subject_dn_country_not_printable_string": {
    "result": "pass"
  },
  "e_subject_dn_serial_number_max_length": {
    "result": "pass"
  },
  "e_subject_dn_serial_number_not_printable_string": {
    "result": "pass"
  },
  "e_subject_email_max_length": {
    "result": "pass"
  },
  "e_subject_empty_without_san": {
    "result": "pass"
  },
  "e_subject_given_name_max_length": {
    "result": "pass"
  },
  "e_subject_info_access_marked_critical": {
    "result": "NA"
  },
  "e_subject_locality_name_max_length": {
    "result": "pass"
  },
  "e_subject_not_dn": {
    "result": "pass"
  },
  "e_subject_organization_name_max_length": {
    "result": "pass"
  },
  "e_subject_organizational_unit_name_max_length": {
    "result": "pass"
  },
  "e_subject_postal_code_max_length": {
    "result": "pass"
  },
  "e_subject_state_name_max_length": {
    "result": "pass"
  },
  "e_subject_street_address_max_length": {
    "result": "pass"
  },
  "e_subject_surname_max_length": {
    "result": "pass"
  },
  "e_utc_time_does_not_include_seconds": {
    "result": "pass"
  },
  "e_utc_time_not_in_zulu": {
    "result": "pass"
  },
  "e_validity_time_not_positive": {
    "result": "pass"
  },
  "e_wrong_time_format_pre2050": {
    "result": "pass"
  },
  "n_ca_digital_signature_not_set": {
    "result": "NA"
  },
  "n_contains_redacted_dnsname": {
    "result": "pass"
  },
  "n_sub_ca_eku_missing": {
    "result": "NA"
  },
  "n_sub_ca_eku_not_technically_constrained": {
    "result": "NA"
  },
  "n_subject_common_name_included": {
    "result": "info"
  },
  "w_distribution_point_missing_ldap_or_uri": {
    "result": "pass"
  },
  "w_dnsname_underscore_in_trd": {
    "result": "pass"
  },
  "w_dnsname_wildcard_left_of_public_suffix": {
    "result": "pass"
  },
  "w_eku_critical_improperly": {
    "result": "pass"
  },
  "w_ext_aia_access_location_missing": {
    "result": "pass"
  },
  "w_ext_cert_policy_contains_noticeref": {
    "result": "pass"
  },
  "w_ext_cert_policy_explicit_text_includes_control": {
    "result": "NA"
  },
  "w_ext_cert_policy_explicit_text_not_nfc": {
    "result": "NA"
  },
  "w_ext_cert_policy_explicit_text_not_utf8": {
    "result": "NA"
  },
  "w_ext_crl_distribution_marked_critical": {
    "result": "pass"
  },
  "w_ext_ian_critical": {
    "result": "NA"
  },
  "w_ext_key_usage_not_critical": {
    "result": "pass"
  },
  "w_ext_policy_map_not_critical": {
    "result": "NA"
  },
  "w_ext_policy_map_not_in_cert_policy": {
    "result": "NA"
  },
  "w_ext_san_critical_with_subject_dn": {
    "result": "pass"
  },
  "w_ext_subject_key_identifier_missing_sub_cert": {
    "result": "pass"
  },
  "w_ian_iana_pub_suffix_empty": {
    "result": "NA"
  },
  "w_issuer_dn_leading_whitespace": {
    "result": "pass"
  },
  "w_issuer_dn_trailing_whitespace": {
    "result": "pass"
  },
  "w_multiple_issuer_rdn": {
    "result": "pass"
  },
  "w_multiple_subject_rdn": {
    "result": "pass"
  },
  "w_name_constraint_on_edi_party_name": {
    "result": "NA"
  },
  "w_name_constraint_on_registered_id": {
    "result": "NA"
  },
  "w_name_constraint_on_x400": {
    "result": "NA"
  },
  "w_root_ca_basic_constraints_path_len_constraint_field_present": {
    "result": "NA"
  },
  "w_root_ca_contains_cert_policy": {
    "result": "NA"
  },
  "w_rsa_mod_factors_smaller_than_752": {
    "result": "pass"
  },
  "w_rsa_mod_not_odd": {
    "result": "pass"
  },
  "w_rsa_public_exponent_not_in_range": {
    "result": "pass"
  },
  "w_san_iana_pub_suffix_empty": {
    "result": "pass"
  },
  "w_serial_number_low_entropy": {
    "result": "NE"
  },
  "w_sub_ca_aia_does_not_contain_issuing_ca_url": {
    "result": "NA"
  },
  "w_sub_ca_certificate_policies_marked_critical": {
    "result": "NA"
  },
  "w_sub_ca_eku_critical": {
    "result": "NA"
  },
  "w_sub_ca_name_constraints_not_critical": {
    "result": "NA"
  },
  "w_sub_cert_aia_does_not_contain_issuing_ca_url": {
    "result": "pass"
  },
  "w_sub_cert_certificate_policies_marked_critical": {
    "result": "pass"
  },
  "w_sub_cert_eku_extra_values": {
    "result": "pass"
  },
  "w_sub_cert_sha1_expiration_too_long": {
    "result": "NA"
  },
  "w_subject_dn_leading_whitespace": {
    "result": "pass"
  },
  "w_subject_dn_trailing_whitespace": {
    "result": "pass"
  }
}

Here's an example parsing the output. https://play.golang.org/p/-6gCyT3xSWu

Would we want to store the entire response? The result field is limited to just a few values. We could probably get away with only storing info, warn, error, and fatal.

Note: NE refers to checks which are date related, but aren't applicable during evaluation. See: https://github.com/zmap/zlint/blob/88032a5e59f98690016d7dd312a3620cecb3e2e0/lints/base.go#L76-L79

Values: https://github.com/zmap/zlint/blob/88032a5e59f98690016d7dd312a3620cecb3e2e0/lints/result.go#L54

cc @jvehent thoughts?

[jvehent]

I agree we should avoid storing passing tests results in the database. I think a list of failing tests (or info, warn, etc.) is really what we want.

Also, I'd rather not update the worker every time zlint adds a new check, which would be needed right now because they use check names as json keys (a bad practice, IMO). Could the worker be written in a way that parses the json as map[string]result where string is the test name? That would allow us to update to upstream zlint without having to modify the worker's go code.

[adamdecaf]

Could the worker be written in a way that parses the json as map[string]result where string is the test name?

I believe that's exactly what my example parser does. I was thinking we could delete from that map any NA, NE, or pass checks and save the resulting map in our db.

There's no way I want to encode specific zlint checks into tls-observatory. That's just going to be a nightmare to maintain.

I can it as technically possible to try and parse out BR sections, human readable descriptions, etc from that spreadsheet though. I'd probably save that for a second PR as it's probably messy to do and might require google auth.