GRIB2 Golang parser application and library

Parser and library for the grib2 file format for meteorological data representations.

For a full presentation of the library, see https://github.com/nilsmagnus/what-is-griblib

Usage

Install by typing

go get -u github.com/nilsmagnus/grib

Library Usage:

Have a look at 'main.go' for main usage:

gribfile, err := os.Open("somegrib2file.grib2")
if err != nil {
	log.Fatalf("Could not open test-file %v", err)
}
messages, err := griblib.ReadMessages(gribfile)

for _, message := range messages {
	// do your thing
}

Read first n messages in a grib-file:

gribfile, err := os.Open("somegrib2file.grib2")
if err != nil { log.Fatalf("Could not open test-file %v", err) }
messages, err := griblib.ReadNMessages(gribfile, 2)

for _, message := range messages {
    // do your thing with the n first messages
}

Application Usage:

$ grib -h 

Usage of grib:
 -category int
   	Filters on Category within discipline. -1 means all categories (default -1)
 -dataExport
   	Export data values. (default true)
 -discipline int
   	Filters on Discipline. -1 means all disciplines (default -1)
 -export int
   	Export format. Valid types are 0 (none) 1(print discipline names) 2(print categories) 3(json) 
 -file string
   	Grib filepath
 -latMax int
   	Maximum latitude multiplied with 100000. (default 36000000)
 -latMin int
   	Minimum latitude multiplied with 100000.
 -longMax int
   	Maximum longitude multiplied with 100000. (default 9000000)
 -longMin int
   	Minimum longitude multiplied with 100000. (default -9000000)
 -maxmsg int
   	Maximum number of messages to parse. Does not work in combination with filters. (default 2147483647)
 -operation string
   	Operation. Valid values: 'parse', 'reduce'. (default "parse")
 -reducefile string
   	Destination for reduced file. (default "reduced.grib2")

Examples:

Reduce input file to default output-file with discipline 0 (Meteorology):

grib -operation reduce -file testdata/reduced.grib2 -discipline 0

Filter on area on size of norway+sweden, output to json:

grib -file testdata/gfs.t00z.pgrb2.2p50.f003  -latMin 57000000 -latMax 71000000 -longMin 4400000 -longMax 32000000 -export 3

Filter on temperature only:

grib -file testdata/gfs.t00z.pgrb2.2p50.f003 -discipline 0 -category 0 

What works?

• basic binary parsing of GRIB2 GFS files from NOAA
• implemented only "Grid point data - complex packing and spatial differencing"
• Parsing Data3 type
• Parsing Data0 type (Thanks to Cyrille Meichel)

Development

Dependencies

grib itself has no dependencies and wants to stay that way to keep it simple. Therefore, there is no dep/glide and no dependency-configuration-files.

Build

Grib uses make to build. You probably need go-lint installed in order to build.

To build, simply type

$ make

to test:

$ make test

TODOs

• Support different types of grids, not only grid0
• Support different types of products, not only product0
• Tests for reduction
• Tests for reading all sections

Contributors

• "analogic", https://github.com/analogic
• Cyrille Meichel, https://github.com/landru29
• Nils Larsgård, https://github.com/nilsmagnus
• measuredweighed, https://github.com/measuredweighed

Help appreciated

Feel free to fork and submit pull requests or simply create issues for improvements :)

Performance

to run a performance-test go to griblib/gribtest and run make benchmark.

CPU

Sample output:

go version go1.11.2 linux/amd64
go test -bench=. -benchmem -memprofile memprofile.out -cpuprofile profile.out
category number 0,parameter number 0,surface type 1, surface value 0 max: 329.500000 min: 197.900000
goos: linux
goarch: amd64
pkg: github.com/nilsmagnus/grib/griblib/gribtest
BenchmarkReadMessages-4   	     100	  11681028 ns/op	 6587281 B/op	    9344 allocs/op
PASS
ok  	github.com/nilsmagnus/grib/griblib/gribtest	5.456s
go tool pprof -top profile.out
File: gribtest.test
Type: cpu
Time: Feb 9, 2019 at 6:44pm (CET)
Duration: 5.41s, Total samples = 5.34s (98.77%)
Showing nodes accounting for 5.12s, 95.88% of 5.34s total
Dropped 69 nodes (cum <= 0.03s)
      flat  flat%   sum%        cum   cum%
     1.58s 29.59% 29.59%      1.78s 33.33%  github.com/nilsmagnus/grib/griblib.(*BitReader).readBit
     1.07s 20.04% 49.63%      2.85s 53.37%  github.com/nilsmagnus/grib/griblib.(*BitReader).readUint
     0.53s  9.93% 59.55%      0.53s  9.93%  runtime.memclrNoHeapPointers
     0.36s  6.74% 66.29%      0.85s 15.92%  encoding/binary.(*decoder).value
     0.15s  2.81% 69.10%      2.78s 52.06%  github.com/nilsmagnus/grib/griblib.(*BitReader).readIntsBlock
     0.15s  2.81% 71.91%      0.15s  2.81%  github.com/nilsmagnus/grib/griblib.(*Data3).applySpacialDifferencing
     0.14s  2.62% 74.53%      0.26s  4.87%  reflect.Value.Index
     0.13s  2.43% 76.97%      0.20s  3.75%  bytes.(*Buffer).ReadByte
     0.11s  2.06% 79.03%      0.28s  5.24%  github.com/nilsmagnus/grib/griblib.(*Data2).scaleValues
     0.11s  2.06% 81.09%      0.19s  3.56%  reflect.Value.SetUint
     0.08s  1.50% 82.58%      0.08s  1.50%  github.com/nilsmagnus/grib/griblib.Data0.scaleFunc.func1
     0.08s  1.50% 84.08%      0.08s  1.50%  reflect.(*rtype).Kind (inline)
     0.07s  1.31% 85.39%      0.07s  1.31%  bytes.(*Buffer).empty (inline)
     0.06s  1.12% 86.52%      0.68s 12.73%  runtime.mallocgc
     0.06s  1.12% 87.64%      0.06s  1.12%  runtime.memmove
     0.05s  0.94% 88.58%      0.05s  0.94%  reflect.flag.mustBeAssignable
     0.05s  0.94% 89.51%      0.05s  0.94%  runtime.nextFreeFast (inline)
     0.04s  0.75% 90.26%      0.04s  0.75%  encoding/binary.(*decoder).uint8 (inline)
   

• As you can see from the sample output, it is the readBit and readUint that takes most of the time. If anyone know how to optimize these functions further, please let me know :)
• Performance can be better if you only want to read certain kinds of messages. The filter that is implemented will filter only after reading all the messages. (help appreciated :) )

Memory

go tool pprof -top memprofile.out
File: gribtest.test
Type: alloc_space
Time: Feb 12, 2019 at 8:19pm (CET)
Showing nodes accounting for 3.39GB, 99.31% of 3.41GB total
Dropped 43 nodes (cum <= 0.02GB)
      flat  flat%   sum%        cum   cum%
    1.52GB 44.71% 44.71%     2.05GB 60.09%  github.com/nilsmagnus/grib/griblib.(*Data2).extractData
    0.76GB 22.19% 66.90%     0.76GB 22.19%  github.com/nilsmagnus/grib/griblib.(*Data2).scaleValues
    0.48GB 14.09% 80.99%     0.48GB 14.09%  github.com/nilsmagnus/grib/griblib.(*BitReader).readIntsBlock
    0.26GB  7.68% 88.67%     0.34GB 10.01%  github.com/nilsmagnus/grib/griblib.(*Data2).extractBitGroupParameters
    0.08GB  2.38% 91.05%     0.08GB  2.38%  github.com/nilsmagnus/grib/griblib.(*BitReader).readUintsBlock
    0.06GB  1.90% 92.95%     3.34GB 97.93%  github.com/nilsmagnus/grib/griblib.readMessage
    0.06GB  1.76% 94.70%     0.06GB  1.77%  encoding/binary.Read
    0.06GB  1.70% 96.40%     3.40GB 99.64%  github.com/nilsmagnus/grib/griblib.ReadMessage
    0.06GB  1.62% 98.02%     0.06GB  1.62%  github.com/nilsmagnus/grib/griblib.makeBitReader
    0.04GB  1.29% 99.31%     0.04GB  1.29%  github.com/nilsmagnus/grib/griblib.(*bitGroupParameter).zeroGroup (inline)

• extractData is the most memory-hungry function. It could probably be more efficient, but is now optimized 80%.
• scaleValues could possibly be optimized, but is much better now than previous versions.

Grib Documentation

Grib specification:

https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/

Documentation from noaa.gov :

http://www.nco.ncep.noaa.gov/pmb/docs/on388/

Daily grib2 files from NOAA can be found at

http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/