IPv4_Coverage_99.65_percent ? #8

MaurUppi · 2024-01-05T11:49:58Z

Can you help me to understand what data source that can claim IPv4_Coverage_99.65_percent ?

Current IPv4 coverage: 99.65% (4.279.808.639 IPv4 in 464.168 networks) [0.20645 sec]
Current IPv6 coverage: 0.40% (1.364.509.245.393.659.395.027.678.354.934.333.440 IPv6 in 310.194 networks) [0.20645 sec]

rabuchaim · 2024-01-06T18:50:01Z

Sure..

All possible IPv4 on the internet is 4.294.967.296

And we have some reserved networks as below with the numbers os possible IPv4 on each network. These are invalid/reserved IPs, And it was included in the database for the simple reason of giving you an answer if you search for such IPs. But let's exclude them from the total calculation.

"0.0.0.0/8":            16777216
"10.0.0.0/8":           16777216
"100.64.0.0/10":         4194304
"127.0.0.0/8":          16777216
"169.254.0.0/16":          65536
"172.16.0.0/12":         1048576
"192.0.0.0/29":                8
"192.0.2.0/24":              256
"192.88.99.0/24":            256
"192.168.0.0/16":          65536
"198.18.0.0/15":          131072
"224.0.0.0/4":         268435456
"240.0.0.0/4":         268435456
"255.255.255.255/32":          1
   TOTAL RESERVED IPS: 592708105

let´s substract the total of reserved networks from all possible IPv4:

4.294.967.296 - 592.708.105 = 3.702.259.191 <<< ALL POSSIBLE VALID IPv4 ON THE INTERNET

If you sum all possible IPv4 of each network range included in Maxmind CSV files, and calculate the proportion, you will get 99.65% of coverage. If you don´t remove the reserved networks of the total of IPv4, you'll get the same percentage.

Maxmind's database does not have geolocation information for approximately 15 million IPs

You will understand better if you run this commands that shows all networks included and all ranges without geoinformation:

geoip2fast --coverage -v

# geoip2fast --coverage -v
GeoIP2Fast v1.2.2 is ready! geoip2fast.dat.gz loaded with 464.168 networks in 0.04187 seconds and using 25.63 MiB.

Use the parameter '-v' to see all networks included in your /usr/local/lib/python3.10/dist-packages/geoip2fast/geoip2fast.dat.gz file.

- Network: 0.0.0.0/8           IPs: 16777216   -- Reserved for self identification    0.000074573 sec
- Network: 1.0.0.0/24          IPs: 256        AU Australia                           0.000014341 sec
- Network: 1.0.1.0/24          IPs: 256        CN China                               0.000007466 sec
- Network: 1.0.2.0/23          IPs: 512        CN China                               0.000006263 sec
- Network: 1.0.4.0/22          IPs: 1024       AU Australia                           0.000005736 sec
- Network: 1.0.8.0/21          IPs: 2048       CN China                               0.000005349 sec
- Network: 1.0.16.0/20         IPs: 4096       JP Japan                               0.000006607 sec
- Network: 1.0.32.0/19         IPs: 8192       CN China                               0.000005168 sec
.
.
.
.
- Network: 223.255.244.0/22    IPs: 1024       IN India                               0.000007008 sec
- Network: 223.255.248.0/22    IPs: 1024       HK Hong Kong                           0.000015678 sec
- Network: 223.255.252.0/23    IPs: 512        CN China                               0.000010240 sec
- Network: 223.255.254.0/24    IPs: 256        SG Singapore                           0.000008366 sec
- Network: 223.255.255.0/24    IPs: 256        AU Australia                           0.000016277 sec
- Network: 224.0.0.0/4         IPs: 268435456  -- Reserved Multicast Networks         0.000015458 sec
- Network: 240.0.0.0/4         IPs: 268435456  -- Reserved for future use             0.000014673 sec
- Network: 255.255.255.255/32  IPs: 1          -- Reserved for broadcast              0.000009139 sec

Current IPv4 coverage:  99.65% (4.279.808.639 IPv4 in 464.168 networks) [16.17575 sec]
Current IPv6 coverage:   0.00% (0 IPv6 in 0 networks) [16.17575 sec]

and

geoip2fast --missing-ips -v

# geoip2fast --missing-ips
GeoIP2Fast v1.2.2 is ready! geoip2fast.dat.gz loaded with 464.168 networks in 0.06102 seconds and using 25.57 MiB.

Searching for missing IPs...

> From 1.0.164.22       to 1.0.164.22       > Network: 1.0.164.22/32       > Missing IPs: 1
> From 1.34.65.179      to 1.34.65.179      > Network: 1.34.65.179/32      > Missing IPs: 1
> From 1.46.23.235      to 1.46.23.235      > Network: 1.46.23.235/32      > Missing IPs: 1
> From 1.47.150.46      to 1.47.150.46      > Network: 1.47.150.46/32      > Missing IPs: 1
> From 2.12.211.171     to 2.12.211.171     > Network: 2.12.211.171/32     > Missing IPs: 1
> From 2.50.8.145       to 2.50.8.145       > Network: 2.50.8.145/32       > Missing IPs: 1
> From 3.143.99.33      to 3.143.99.33      > Network: 3.143.99.33/32      > Missing IPs: 1
> From 5.63.16.0        to 5.63.23.255      > Network: 5.63.16.0/21        > Missing IPs: 2048
.
.
.
> From 218.100.70.0     to 218.100.70.255   > Network: 218.100.70.0/24     > Missing IPs: 256
> From 218.100.79.0     to 218.100.79.255   > Network: 218.100.79.0/24     > Missing IPs: 256
> From 218.100.86.0     to 218.100.86.255   > Network: 218.100.86.0/24     > Missing IPs: 256
> From 220.152.116.0    to 220.152.119.255  > Network: 220.152.116.0/22    > Missing IPs: 1024
> From 220.158.148.0    to 220.158.151.255  > Network: 220.158.148.0/22    > Missing IPs: 1024
> From 222.164.170.156  to 222.164.170.156  > Network: 222.164.170.156/32  > Missing IPs: 1
> From 223.165.0.0      to 223.165.3.255    > Network: 223.165.0.0/22      > Missing IPs: 1024

>>> Valid IP addresses without geo information: 15.158.657 (0.35% of all IPv4 in 9.732 networks) [0.76021 sec]

If you have any questions, please let me know...

[...]s

MaurUppi · 2024-01-07T02:35:30Z

Thank you so much for the detailed reply!

Could you think about switching or adding support for the ipinfo.io/IP To Country database?

The reason is it's way better than MaxMind's based on my and some other's experience.
Well, I have to confess that I don't have the technical skills to check in detail why the file size has a huge delta. If you're interested, could you take a look?
GeoLite2-Country_20231222.mmdb 6.02 MB (6,320,117 bytes) vs. Free IP to Country.mmdb Jan 05, 2024, 31.7 MB (33,267,411 bytes)
it seems the database schema is diff than MaxMind?

rabuchaim · 2024-01-07T04:45:50Z

I´ll take a look on this.. I downloaded the free files list today and I will adapt geoip2dat.py to use this source.

And I think is possible to use the paid version also if you have a subscription.

I keep you informed on this issue.

MaurUppi · 2024-01-07T05:19:06Z

I´ll take a look at this.. I downloaded the free files list today and I will adapt geoip2dat.py to use this source.

And I think is possible to use the paid version also if you have a subscription.

I keep you informed on this issue.

Thx for your consideration!

If I may, I'd propose you think about adding location_latitude,location_longitude fields given you already include [ City and ASN databases with IPv4 and IPv6]
For the data source of City level w/ location info, I would recommend DB-IP.com dbip-city-lite, they provide both CSV and mmdb format.

type	database	File size
City level	recommedation DB-IP.com
MaxMind	GeoLite2-City.mmdb	66.3 MB (69,591,332 bytes)
DB-IP.com	dbip-city-lite-2023-12.mmdb	93.0 MB (97,576,399 bytes)

IP TO COUNTRY LITE
IP TO CITY LITE
IP TO ASN LITE

PS, seems too much to ask...

PPS, DB-IP.com did not provide a key for direct download, so I wrote a Downloader for DB-IP.com Databases

rabuchaim · 2024-01-07T06:19:57Z

I need to review my calcs and review all networks to check if they are overlaying one network over another, but at first, ipinfo.io have an excellent coverage with the free database

INCLUDING ALL RESERVED NETWORKS

>>> STARTING GeoIP2Dat v1.2.2 for ipinfo.io
- Checking directories... done! [0.000012 sec]
- Source info: IPInfo.io-CountryASN-IPv4IPv6-20240107
- IPv4 coverage: 4293280302 = 99.96%
- IPv6 coverage: 586040100630288400342571114829971456 = 0.17%
<<< EXITING GeoIP2Dat for ipinfo.io PROCESS [5.032327 seconds]

Not considering reserved networks, they have 99.95% of IPv4 coverage

I´ll also test the database from dbip (thanks for the links)

MaurUppi · 2024-01-07T07:53:44Z

Additional info, not quite sure if helpful.

tool : safing-mmdbmeld

dbip-city-lite-2024-01.mmdb

Total= 285,212,672 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
Network Mask Stats: Total= 27,087,005 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%

GeoLite2-City_20231215.mmdb

Total= 285,212,672 Country=0.18% Coords=0.18% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
Network Mask Stats: Total= 15,250,389 Country=99.96% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%

PS D:\Github\mmdbmeld> ./cmd/mmdbcheck/mmdbcheck all D:\Dev\GeoLocationData\dp-ip\dbip-city-lite-2024-01.mmdb
loading D:\Dev\GeoLocationData\dp-ip\dbip-city-lite-2024-01.mmdb with 96.44 MB

Running all checks:

Probing:
Probing 2000::/4 in /32 blocks
.:
................................................................................................................................................................................................................................................................:
::
analyzed with 0 lookup errors
Total= 285212672 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%

Network Mask Stats:
          Total=  27087005 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  2 Total=         2 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  3 Total=         1 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  4 Total=         2 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  5 Total=         1 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  6 Total=         1 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  7 Total=         2 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  8 Total=         6 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/  9 Total=         7 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 10 Total=         7 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 11 Total=         5 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 12 Total=         4 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 13 Total=         7 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 14 Total=         5 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 15 Total=         6 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 16 Total=        10 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 17 Total=        10 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 18 Total=        19 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 19 Total=        38 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 20 Total=        81 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 21 Total=       110 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 22 Total=       126 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 23 Total=       139 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 24 Total=       209 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 25 Total=       338 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 26 Total=      1422 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 27 Total=     26980 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 28 Total=     35080 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 29 Total=     48324 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 30 Total=     30790 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 31 Total=     40174 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 32 Total=     77830 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 33 Total=     23575 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 34 Total=     35546 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 35 Total=     60871 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 36 Total=    114885 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 37 Total=    179810 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 38 Total=    391069 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 39 Total=    777718 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 40 Total=   2602449 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 41 Total=    581637 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 42 Total=    631599 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 43 Total=    719460 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 44 Total=    624467 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 45 Total=    462859 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 46 Total=    450913 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 47 Total=    435983 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 48 Total=   1228605 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 49 Total=    121533 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 50 Total=    196949 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 51 Total=    224074 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 52 Total=    280757 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 53 Total=    154120 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 54 Total=    336919 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 55 Total=    645252 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 56 Total=   2383887 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 57 Total=    352584 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 58 Total=    394163 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 59 Total=    359860 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 60 Total=    331832 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 61 Total=    282436 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 62 Total=    179201 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 63 Total=     73378 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 64 Total=     67137 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 65 Total=       144 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 66 Total=       142 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 67 Total=       142 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 68 Total=       142 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 69 Total=       142 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 70 Total=       142 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 71 Total=       142 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 72 Total=       143 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 73 Total=       141 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 74 Total=       141 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 75 Total=       141 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 76 Total=       141 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 77 Total=       141 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 78 Total=       141 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 79 Total=       141 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 80 Total=       144 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 81 Total=       138 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 82 Total=       138 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 83 Total=       138 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 84 Total=       139 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 85 Total=       138 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 86 Total=       136 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 87 Total=       136 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 88 Total=       136 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 89 Total=       136 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 90 Total=       136 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 91 Total=       136 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 92 Total=       136 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 93 Total=       136 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 94 Total=       135 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 95 Total=       137 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 96 Total=       145 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 97 Total=       129 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 98 Total=       128 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 99 Total=       131 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/100 Total=       130 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/101 Total=       128 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/102 Total=       128 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/103 Total=       130 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/104 Total=       139 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/105 Total=       143 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/106 Total=       155 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/107 Total=       169 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/108 Total=       235 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/109 Total=       414 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/110 Total=      1031 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/111 Total=      2859 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/112 Total=     14800 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/113 Total=     13180 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/114 Total=     24907 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/115 Total=     53411 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/116 Total=    121230 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/117 Total=    270893 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/118 Total=    639909 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/119 Total=   1258788 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/120 Total=   4736858 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/121 Total=    687638 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/122 Total=    772350 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/123 Total=    704900 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/124 Total=    645447 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/125 Total=    556098 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/126 Total=    349795 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/127 Total=    138006 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/128 Total=    121016 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%

PS D:\Github\mmdbmeld> ./cmd/mmdbcheck/mmdbcheck all D:\Dev\GeoLocationData\MaxMind\GeoLite2-City_20231215\GeoLite2-City.mmdb
loading D:\Dev\GeoLocationData\MaxMind\GeoLite2-City_20231215\GeoLite2-City.mmdb with 69.59 MB

Running all checks:

Probing:
Probing 2000::/4 in /32 blocks
................................................................................................................................................................................................................................................................:
................................................................................................................................................................................................................................................................:

analyzed with 0 lookup errors
Total= 285212672 Country=0.18% Coords=0.18% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%

Network Mask Stats:
          Total=  15250389 Country=99.96% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 16 Total=         1 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 19 Total=         3 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 20 Total=        15 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 21 Total=        13 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 22 Total=        41 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 23 Total=        30 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 24 Total=        66 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 25 Total=        75 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 26 Total=       150 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 27 Total=       194 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 28 Total=       432 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 29 Total=     15312 Country=99.97% Coords=99.97% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 30 Total=      2222 Country=99.91% Coords=99.91% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 31 Total=      3637 Country=99.97% Coords=99.97% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 32 Total=     36013 Country=99.98% Coords=99.98% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 33 Total=     14097 Country=99.94% Coords=99.98% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 34 Total=     22452 Country=99.97% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 35 Total=     40041 Country=99.98% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 36 Total=     86530 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 37 Total=    153735 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 38 Total=    352360 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 39 Total=    524833 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 40 Total=    759231 Country=99.94% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 41 Total=    415775 Country=99.98% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 42 Total=    288573 Country=99.97% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 43 Total=    249389 Country=99.96% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 44 Total=    225410 Country=99.95% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 45 Total=    242478 Country=99.96% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 46 Total=    146115 Country=99.94% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 47 Total=    137771 Country=99.92% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 48 Total=    289756 Country=99.90% Coords=99.94% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 49 Total=     30419 Country=99.97% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 50 Total=     43547 Country=99.98% Coords=99.98% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 51 Total=     59958 Country=99.97% Coords=99.98% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 52 Total=    104729 Country=99.98% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 53 Total=    167983 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 54 Total=    348507 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 55 Total=    532826 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 56 Total=    761646 Country=99.94% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 57 Total=    409318 Country=99.98% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 58 Total=    282017 Country=99.97% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 59 Total=    248080 Country=99.97% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 60 Total=    218179 Country=99.95% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 61 Total=    213107 Country=99.96% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 62 Total=    158942 Country=99.95% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 63 Total=    149432 Country=99.93% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/ 64 Total=    501649 Country=99.95% Coords=99.97% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/104 Total=        10 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/105 Total=        18 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/106 Total=        34 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/107 Total=        86 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/108 Total=       218 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/109 Total=       506 Country=99.60% Coords=99.60% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/110 Total=      1408 Country=99.86% Coords=99.86% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/111 Total=      3820 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/112 Total=     12488 Country=99.94% Coords=99.94% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/113 Total=     17844 Country=99.94% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/114 Total=     32268 Country=99.98% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/115 Total=     64010 Country=99.98% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/116 Total=    150306 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/117 Total=    280942 Country=100.00% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/118 Total=    644228 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/119 Total=   1023708 Country=99.99% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/120 Total=   1476628 Country=99.94% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/121 Total=    782686 Country=99.98% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/122 Total=    528162 Country=99.97% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/123 Total=    454914 Country=99.96% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/124 Total=    386764 Country=99.95% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/125 Total=    366314 Country=99.96% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/126 Total=    231790 Country=99.94% Coords=100.00% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/127 Total=    199274 Country=99.90% Coords=99.99% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%
CIDR=/128 Total=    354874 Country=99.87% Coords=99.93% ASN=0.00% ASOrg=0.00% AC=0.00% SP=0.00% AP=0.00%

rabuchaim · 2024-01-08T01:42:42Z

Nice tool... from this tool I found a lot of repositories in geoip universe that I didn´t know rsrsrs

Let´s resume the coding for ipinfo and db-ip databases.. including latitude and longitude

MaurUppi · 2024-01-08T04:12:22Z

I’m not quite either is possible to merge those databases to a comprehensive one? Country + City + ASN Ricardo Abuchaim ***@***.***>于2024年1月8日周一09:42写道：

…

Nice tool... from this tool I found a lot of repositories in geoip universe that I didn´t know rsrsrs Let´s resume the coding for ipinfo and db-ip databases.. including latitude and longitude — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMADBUBB2FMBH27MGWBKY3YNNFJZAVCNFSM6AAAAABBOKAMNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBQGI4DIMBUHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

rabuchaim · 2024-01-08T07:07:07Z

Well.. I did some tests here with the ipinfo.io base only countries and IPv4 (to start).

They do not provide the network cidr, they provide a range with starting IP and ending IP... this range is not always a perfect network, sometimes the range is 100 IPs (for example).

For my method, CIDR is very important to know if the requested IP is inside the network returned by my search method. If don´t, I return a "not found in database" error. For the vast majority of people looking for a GeoIP library/database, the network CIDR is not important.

I have an application for mitigating abuse and attacks on websites and services that allows blocking by country, and when an IP from a blocked country appears, I block the entire network range of that country to save firewall rules. Sometimes internal network IPs appear in the logs and I have to know this without doing IFs or try except which is necessary in mmdb bases to avoid an exception. For me, I just need the country_code, the network cidr, speed and little memory usage. This is the reason to code GeoIP2Fast e return a json/dict so you don´t need to worry about exceptions. And for a single query on the command line, using an mmdb database is much faster.

But I can get around this by knowing how many IPs are in the range they provided. And I did 2 tests... one of them creating the networks to have the network suffix (/24, /12, etc.) and another test using the number of IPs in the ranges provided.

With the IPINFO IPv4 base, there are a total of 828,000 ranges... I had to use the ipaddress.summarize_address_range function to generate all possible subnets within the range they provide... look at this for example:

1.1.10.11,1.1.21.1,CN,China,AS,Asia
1.1.21.2,1.1.21.2,BR,Brazil,SA,South America
1.1.21.3,1.1.63.255,CN,China,AS,Asia

From the network 1.1.21.0/24, only the IP 1.1.21.2 is in Brazil.. the rest is in China. Including 1.1.21.0/32 and 1.1.21.1/32

Splitting the network 1.1.21.0/24 in possible subnets.. I have:
['1.1.21.0/31','1.1.21.3/32', '1.1.21.4/30', '1.1.21.8/29', '1.1.21.16/28', '1.1.21.32/27', ' 1.1.21.64/26', '1.1.21.128/25'] in China
and only ['1.1.21.2/32'] in Brazil.

As you can see, 2 lines, became 9 records... 8 for China and 1 for Brazil.

By doing this, the IPINFO database went from 828,000 to 1,523,000 records... and as a result, the IPv4 country database takes a while to load. But the search time in my method is the same for 50,000 records or 5,000,000 records... what changes is the database loading time and the memory usage.

Below is a comparison between the current version and the IPINFO database with and withou cidr.

	Maxmind.com with cidr	IPInfo.io no cidr	IPInfo.io with cidr
networks	465.000	828.000	1.520.000
file size	1.1Mb	2.2Mb	3.3 Mb
coverage	99.65%	99.96%	99.96%
ram footprint	25Mb	50Mb	82Mb
missing ips	15.158.657	1.688.116	1.688.116
load time	~0.04 sec	~0.08 sec	~0.18 sec

And some screeshots.. Note: I´m not using functools.lru_cache anymore.. this is useful when you have a function that tooks lot of time, but in the case of GeoIP2Fast, the binary searches tooks less than 0.00001 seconds.. and the time spent to check if the result is cached or not in functools.lru_cache takes longer than going through the function again. GeoIP2Fast is faster without functools and using the processor and operating system's own cache. The cached results below are without functools.

Without CIDR
With CIDR

I am very inclined to use this database in my application. I'm studying what one base doesn't have and the other has... for example, in class 23.0.0.0/8, maxmind is missing 3,701,508 IPs, and IPINFO is missing only 1. And I checked some ranges on Whois and they match the IPinfo.io database. And I'm seeing the possibility of merging these bases... But I will definitely create a new class to use the Ipinfo.io database and another for db-ip as well.

I'm going to do more tests now with ASN and IPv6.. I'd like the paid database with Cities, even if it's old, to do performance tests.. I'll try to find a sponsor lol

And I will do some tests with dbip database also and include the requested latitude and longitude.. the base will be bigger, but is possible. Let´s see what I can do this week. Split the database in 3 or more files with a main index is an option. Let´s think about this later.

rabuchaim · 2024-01-08T08:45:36Z

The db-ip CSV database is 100% of IPv4 as shown in mmdbcheck tool

Now I´m working to find the difference between maxmind CSV files and the MMDB file as shown in mmdbcheck tool... for exemple, I have a lot of networks gaps in 23.0.0.0/8 network in csv files, and the mmdbcheck tool shows 100%

the 3 databases deserve a detailed comparison.. I'll do that.

rabuchaim · 2024-01-08T09:26:12Z

This is why I don´t like to use mmdb files.. The network 23.0.0.0/8 is the one that intrigues me the most so I like to use it as an example. If you want to be happy, choose a geoip database and never check any other geoip database.

23.183.247.128/25,2635167,6252001,,0,0 	<< MAXMIND CSV FILES SAYS THAT IS United Kingdom

GeoIP2Fast returns the same, United Kingdom

IPINFO returns that it is in United Kingdom

Maxmind MMDB file returns IP ADDRESS NOT FOUND

# geoiplookup 23.183.247.129			
GeoIP Country Edition: IP Address not found

The whois database says that 23.183.247.129 is in UNITED STATES.

Look below, the network 23.185.175.0/24 is a gap in the Maxmind CSV files. So, GeoIP2Fast returns "not found in database".

23.185.128.0/24,6252001,6252001,,0,0
23.185.144.0/24,6252001,,,0,0
23.185.176.0/24,6252001,6252001,,0,0   
23.185.192.0/24,6252001,6252001,,0,0
23.185.208.0/24,6252001,6252001,,0,0
23.185.240.0/23,6252001,6252001,,0,0

But in Maxmind mmdb file, returned as United States

GeoIP Country Edition: US, United States

IPINFO says that is in United States..

{
   "ip": "23.185.175.2",
   "country_code": "US",
   "country_name": "United States",
   "city": "",
   "cidr": "",
   "hostname": "",
   "asn_name": "",
   "asn_cidr": "",
   "is_private": false,
   "elapsed_time": "0.000036999 sec"
}

And in the whois database returns "this network is not found" (in command line or who.is or whois.com websites)

whois 23.185.175.2
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
No match found for r + = 23.185.175.2.

whois 23.185.175.0/24
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
No match found for r + = 23.185.175.0/24.

MaurUppi · 2024-01-08T15:28:39Z

Wow~~~~~ What a detailed analysis you did in the past day!

You found something meaningful!

By doing this, the IPINFO database went from 828,000 to 1,523,000 records... and as a result, the IPv4 country database takes a while to load. But the search time in my method is the same for 50,000 records or 5,000,000 records... what changes is the database loading time and the memory usage.

Ok, that's the reason of ipinfo's database has a huge delta in terms of file size?

MaxMind GeoLite2-Country_20231222.mmdb 6.02 MB (6,320,117 bytes) vs. IPinfo Free IP to Country.mmdb Jan 05, 2024, 31.7 MB (33,267,411 bytes)

well, given IPinfo's database missing ips is ten times less than MaxMind's. I can understand it did eat more memory and increased loading time.

This is why I don´t like to use mmdb files.. The network 23.0.0.0/8 is the one that intrigues me the most so I like to use it as an example. If you want to be happy, choose a geoip database and never check any other geoip database.

Certainly, MMDB and CSV format have their cons and pros. Meanwhile, those two formats may not keep the same schema just like db-ip IP to Country Lite database

Description	CSV	MMDB
First IP address	x	x
Last IP address	x	x
Continent code		x
Continent name		x
Country ISO-3166-alpha2 code	x	x
Country EU membership		x
Country name		x

Given it seems you're quite happy to dig out a solution to merge those databases to be one if possible.
Am I making a happy trouble to you?

if you enjoy doing so, here is another one to take a look also.

IPLOCATION Lite

I did roughly take a look at two databases.

Both use IP range but not CIDRs.
What surprised me was that db-ip included county-level info.
meanwhile, I also wondering how both vendors can provide/know what IP range will it be. Maybe they are sourcing from 3rd party. I can only find City-level CIDR info from China's local community/lite ver.
Technically, IPLOCATION provided 31.224332, 121.468951 (six decimal places) are more precise than dbip's 31.1694, 121.408 (four decimal places), with a difference of approximately 10 meters. But given on City scale level, I don't think it required six decimal places for precision of geographical coordinates.
So, dp-ip remains in my favor.

database/IP range	file size/coutry-city	location
DB5.LITE IP-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE	545 MB (571,527,738 bytes)
"281471715262464","281471715266559",	"CN","China","Fujian","Zhangzhou",	"24.513271","117.655569"
dbip-city-lite-2024-01.mmdb	447 MB (469,554,567 bytes)
1.24.232.0,1.24.235.255,	AS,CN,Fujian,"Zhangzhou (Neimeng)",	24.6391,117.199

BTW, I'm working on China City level GeoLocation mapping, maybe I can share with you when you going to merge databases and include location info for the City.

Here is a sample, data sourcing a commercial map data service provider.

  {
    "area": "",
    "city": "Changchun",
    "country": "China",
    "lat": "43.816201670699584",
    "lng": "125.3239995997326",
    "province": "Jilin"
  },

rabuchaim · 2024-01-10T11:28:57Z

Hi there!!

i got some good results.. to receive the latitude and longitude data, the current file structure didn't have a good performance ( just a json saved with pickle )... it's still fast, but it consumed a lot of RAM, reaching 400mb and I want to do this... so I decided to code an idea and I think I'm on the right way... the trick is to read the file exactly at the offset where the data I want is, roughly speaking, a lot of small lists of integers saved sequentially in a file with a main index (maybe 2).

I lost one ZERO but the memory usage, which was 80Mb with the IPINFO database, dropped to 2Mb... And with a lookup of 0.001 second, functools helped a lot... but I'm testing other types of cache. It can be improved, it was my first test, and I think I can get one more ZERO without cache... And the file size increase a little bit, but I'm still evaluating what I can compress or not.

I´ll return soon...

MaurUppi · 2024-01-11T04:33:07Z

I lost one ZERO but the memory usage, which was 80Mb with the IPINFO database, dropped to 2Mb... And with a lookup of 0.001 second, functools helped a lot... but I'm testing other types of cache. It can be improved, it was my first test, and I think I can get one more ZERO without cache... And the file size increase a little bit, but I'm still evaluating what I can compress or not.

Wow, you got a huge improvement in terms of reducing memory consumption.

May I know if you have merged databases -OR- the first step is using the IPINFO to confirm the method to reduce mem consumption whether works?

I'm looking forward:

A merged database that enhanced IP coverage with improved accuracy of Country ( City with location could be a separate database if not suitable to be merged in a single one).
I would prefer you not only provide an app that can access the .DAT format but also an API for 3rd party app integration.

rabuchaim · 2024-01-15T07:15:12Z

Hi there!

And I tested the current version of maxminddb library and they have really good lookup time, around 0.00003... My last use of this library was 2 years ago and I didn´t like the results.. but I don´t want to create another dependency in my application. But they are really fast.

Well.. I got the result that I expected with ONLY COUNTRY database from dbip, ipinfo and maxmind csv files... but I didn´t like the lookup time using the dbip city+lat+long database...

The load time and ram usage are good for me, but the average lookup time is not good (for me). Is possible to reduce the ram usage to 3Mb, but the lookup time increase to 0.0001. My goal is 0.00003 (or less), less than 5Mb of RAM and load time of 0.01 (or less). This datafile has 55Mb.

The current source code is not "ellegant" to be released.. But I will... don´t worry.

And about the "API for 3rd party app integration", I´m coding an internal GeoIP API Server for a friend and I will release in a few weeks... the api source code and a docker container a nice and secure server with administration page, cache, health-check page, uses nginx for protection (optional)... and you can use with a GeoIP2FastSocketClient (really fast) or GeoIP2FastHTTPClient, both with fallback options and pure python,

I didn´t tested a merged version of ipinfo database, but I will... With this method the ram consumption is easy to control, I divide the lists in pieces (chunks), and create a main index for this and keep it in memory.. now, the ram usage depends only on the size of each chunk. Big chunks creates small indexes and use less memory, Small chunks, results in big indexes that uses more memory, but is faster. Will still working on this... I really need to reduce the ram usage for GeoIP2Fast with city databases.

MaurUppi · 2024-01-17T14:27:01Z

Hi there!

And I tested the current version of maxminddb library and they have really good lookup time, around 0.00003... My last use of this library was 2 years ago and I didn´t like the results.. but I don´t want to create another dependency in my application. But they are really fast.

I'm curious to ask, any idea why the MaxMind's db had the fastest lookup time?

Well.. I got the result that I expected with ONLY COUNTRY database from dbip, ipinfo and maxmind csv files... but I didn´t like the lookup time using the dbip city+lat+long database...

The load time and ram usage are good for me, but the average lookup time is not good (for me). Is possible to reduce the ram usage to 3Mb, but the lookup time increase to 0.0001. My goal is 0.00003 (or less), less than 5Mb of RAM and load time of 0.01 (or less). This datafile has 55Mb.

Looks like you had reduced mem consumption. I agree the lookup time isn't good compared to your goal. meanwhile, can I ask what reasons you set such a high target performance? is it because of massive concurrent inquiry?

And about the "API for 3rd party app integration", I´m coding an internal GeoIP API Server for a friend and I will release in a few weeks... the api source code and a docker container a nice and secure server with administration page, cache, health-check page, uses nginx for protection (optional)... and you can use with a GeoIP2FastSocketClient (really fast) or GeoIP2FastHTTPClient, both with fallback options and pure python,

Wow, looks amazing!!! looking forward to having fun with the GeoIP API Server.

I didn´t tested a merged version of ipinfo database, but I will... With this method the ram consumption is easy to control, I divide the lists in pieces (chunks), and create a main index for this and keep it in memory.. now, the ram usage depends only on the size of each chunk. Big chunks creates small indexes and use less memory, Small chunks, results in big indexes that uses more memory, but is faster. Will still working on this... I really need to reduce the ram usage for GeoIP2Fast with city databases.

It sounds like you're a very experienced code developer!!!! Can't wait to hear more updates about the database merging progress.

Not sure what I helped, but I think you may create the mmdb format database from CSV files. therefore, here is what is tried organized info for your reference.

City

target csv format for mmdb schema

start_ip,end_ip,continent_code,continent_name,country_code,country_name,state,city,latitude,longitude

ipapi.is ## Download IP to Geolocation Database

geolocationDatabaseIPv4.csv 124 MB (131,067,015 bytes)

ip_version,start_ip,end_ip,continent,country_code,country,state,city,zip,timezone,latitude,longitude,accuracy
4,159.253.45.0,159.253.45.255,AS,TR,Turkey,Denizli,Denizli,20890,Europe/Istanbul,37.738820042147495,29.09278392791748,2
4,46.163.50.208,46.163.50.215,EU,BA,Bosnia and Herzegovina,,Sarajevo,,Europe/Sarajevo,43.85506,18.39971,2

DB-IP

dbip-city-lite-2024-01.csv 447 MB (469,554,567 bytes)`

start_ip,end_ip,continent,country_code,state,city,latitude,longitude
0.0.0.0,0.255.255.255,ZZ,ZZ,,,0,0
1.0.0.0,1.0.0.255,OC,AU,Queensland,"South Brisbane",-27.4767,153.017
1.0.1.0,1.0.3.255,AS,CN,Fujian,Gaosha,26.4837,117.925

IP2LOCATION

IP2LOCATION-LITE-DB5.IPV6.CSV 545 MB (571,527,738 bytes)`

start_ip,end_ip,country_code,country,state,city,latitude,longitude
"281470698520576","281470698520831","US","United States of America","California","San Jose","37.339390","-121.894960"
"281470698520832","281470698521599","CN","China","Fujian","Fuzhou","26.061390","119.306110"
"281470698521600","281470698522623","AU","Australia","Victoria","Melbourne","-37.814007","144.963171"

MaxMind GeoLite2-City-CSV_20240105

GeoLite2-City-Blocks-IPv4.csv 198 MB (207,947,477 bytes)

network,geoname_id,registered_country_geoname_id,represented_country_geoname_id,is_anonymous_proxy,is_satellite_provider,postal_code,latitude,longitude,accuracy_radius
1.0.0.0/24,2077456,2077456,,0,0,,-33.4940,143.2104,1000
1.0.1.0/24,1814991,1814991,,0,0,,34.7732,113.7220,1000
1.0.2.0/23,1814991,1814991,,0,0,,34.7732,113.7220,1000
1.0.4.0/22,2147714,2077456,,0,0,2101,-33.6955,151.2466,1000

GeoLite2-City-Locations-en.csv 9.75 MB (10,228,292 bytes)

geoname_id,locale_code,continent_code,continent_name,country_iso_code,country_name,subdivision_1_iso_code,subdivision_1_name,subdivision_2_iso_code,subdivision_2_name,city_name,metro_code,time_zone,is_in_european_union
1392,en,AS,Asia,IR,Iran,02,Māzandarān,,,Shahr,,Asia/Tehran,0
5819,en,EU,Europe,CY,Cyprus,02,"Limassol District",,,Souni,,Asia/Nicosia,1
7240,en,AS,Asia,IR,Iran,28,"North Khorasan",,,Jahan,,Asia/Tehran,0
10293,en,AS,Asia,IR,Iran,26,"Qazvin Province",,,Kaman,,Asia/Tehran,0
10570,en,AS,Asia,IR,Iran,26,"Qazvin Province",,,Alvand,,Asia/Tehran,0

Country

target csv format for mmdb schema

start_ip,end_ip,country,country_name,continent,continent_name

DB-IP

dbip-country-lite-2024-01.csv 26.6 MB (27,941,303 bytes)

0.0.0.0,0.255.255.255,ZZ
1.0.0.0,1.0.0.255,AU
1.0.1.0,1.0.3.255,CN

IP2LOCATION

IP2LOCATION-LITE-DB1.CSV 10.8 MB (11,334,396 bytes)

"0","16777215","-","-"
"16777216","16777471","US","United States of America"
"16777472","16778239","CN","China"
"16778240","16779263","AU","Australia"

ipinfo

country.csv 101 MB (106,434,802 bytes)

start_ip,end_ip,country,country_name,continent,continent_name
1.0.0.0,1.0.0.254,US,United States,NA,North America
1.0.0.255,1.0.0.255,AU,Australia,OC,Oceania
1.0.1.0,1.0.3.255,CN,China,AS,Asia

MaxMind GeoLite2-Country_20231222

GeoLite2-Country-Blocks-IPv4.csv 16.2 MB (16,993,937 bytes)

network,geoname_id,registered_country_geoname_id,represented_country_geoname_id,is_anonymous_proxy,is_satellite_provider
1.0.0.0/24,2077456,2077456,,0,0
1.0.1.0/24,1814991,1814991,,0,0
1.0.2.0/23,1814991,1814991,,0,0
1.0.4.0/22,2077456,2077456,,0,0

GeoLite2-Country-Locations-en.csv 9.63 KB (9,866 bytes)

geoname_id,locale_code,continent_code,continent_name,country_iso_code,country_name,is_in_european_union
49518,en,AF,Africa,RW,Rwanda,0
51537,en,AF,Africa,SO,Somalia,0
69543,en,AS,Asia,YE,Yemen,0
99237,en,AS,Asia,IQ,Iraq,0

jkuj6 · 2024-02-21T11:24:32Z

@rabuchaim i just want to say thank you for your effort in providing users with a geoip.dat file alternative to maxmind.

I was about to use ipapi db and thought they were the most up to date. However their issue is they only provide csv files and to be honest i have no proof to see if they are accurate enough.
I tried to convert their csv files, but their format is very different so i created a script to get the format the same as mmdbctl tool uses from ipinfo, and from there i was only able to create a mmdb file. I was not able to create a .dat from that mmdb file, regardless of my efforts to do it with geoiplegacy tool.

Just a few things,

i assume the geoip2fast.dat.gz file you provide is compatible with the geoip nginx module, correct?
is your DB's you update more accurate than other providers like maxmind, ip-db and ipapi?
Is it at all possible to merge geoip2fast.dat.gz & geoip2fast-ipv6.dat.gz in one .dat file? or will that degrade performance?

jkuj6 · 2024-02-21T11:44:07Z

ps: i actually just tried the .dat file with the ngx_http_geoip_module and it doesnt seem to work or read the data. please let me know if im missing something

rabuchaim mentioned this issue Jun 5, 2024

Bad traceback #10

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IPv4_Coverage_99.65_percent ? #8

IPv4_Coverage_99.65_percent ? #8

MaurUppi commented Jan 5, 2024

rabuchaim commented Jan 6, 2024

MaurUppi commented Jan 7, 2024

rabuchaim commented Jan 7, 2024

MaurUppi commented Jan 7, 2024 •

edited

Loading

rabuchaim commented Jan 7, 2024

MaurUppi commented Jan 7, 2024

rabuchaim commented Jan 8, 2024

MaurUppi commented Jan 8, 2024 via email

rabuchaim commented Jan 8, 2024

rabuchaim commented Jan 8, 2024 •

edited

Loading

rabuchaim commented Jan 8, 2024

MaurUppi commented Jan 8, 2024 •

edited

Loading

rabuchaim commented Jan 10, 2024

MaurUppi commented Jan 11, 2024

rabuchaim commented Jan 15, 2024

MaurUppi commented Jan 17, 2024

jkuj6 commented Feb 21, 2024

jkuj6 commented Feb 21, 2024

IPv4_Coverage_99.65_percent ? #8

IPv4_Coverage_99.65_percent ? #8

Comments

MaurUppi commented Jan 5, 2024

rabuchaim commented Jan 6, 2024

MaurUppi commented Jan 7, 2024

rabuchaim commented Jan 7, 2024

MaurUppi commented Jan 7, 2024 • edited Loading

rabuchaim commented Jan 7, 2024

MaurUppi commented Jan 7, 2024

rabuchaim commented Jan 8, 2024

MaurUppi commented Jan 8, 2024 via email

rabuchaim commented Jan 8, 2024

rabuchaim commented Jan 8, 2024 • edited Loading

rabuchaim commented Jan 8, 2024

MaurUppi commented Jan 8, 2024 • edited Loading

rabuchaim commented Jan 10, 2024

MaurUppi commented Jan 11, 2024

rabuchaim commented Jan 15, 2024

MaurUppi commented Jan 17, 2024

City

target csv format for mmdb schema

ipapi.is ## Download IP to Geolocation Database

DB-IP

IP2LOCATION

MaxMind GeoLite2-City-CSV_20240105

Country

target csv format for mmdb schema

DB-IP

IP2LOCATION

ipinfo

MaxMind GeoLite2-Country_20231222

jkuj6 commented Feb 21, 2024

jkuj6 commented Feb 21, 2024

MaurUppi commented Jan 7, 2024 •

edited

Loading

rabuchaim commented Jan 8, 2024 •

edited

Loading

MaurUppi commented Jan 8, 2024 •

edited

Loading