lexi: add new database package #3033
Conversation
Add the Lexi DB package, which wraps badger DB to provide a simplified API, to capture the sometimes tedious mechanics that we've repeated in various places, and to add utilities for indexing data for quick retrieval of filtered data. This change is just the addition of the package. I will demonstrate it's use in tatanka separately, and also show how this can replace in our client/asset tx DB implementations.
| bLen := 1 + len(d.v) + wire.VarIntSerializeSize(uint64(len(d.v))) + wire.VarIntSerializeSize(uint64(len(d.indexes))) | ||
| for _, ib := range d.indexes { | ||
| bLen += len(ib) + wire.VarIntSerializeSize(uint64(len(ib))) | ||
| } | ||
| b := bytes.NewBuffer(make([]byte, 0, bLen)) |
There was a problem hiding this comment.
Oops. I forgot to doc this encoding stuff. It's pretty much like BuildyBytes, but I'm pre-allocating the buffer and allowing any blob size. And I'm leveraging wire for the var int stuff.
I guess its a lot of work just to pre-allocate the buffer though. derp.
There was a problem hiding this comment.
When reading the doc I noticed this:
dcrdex/dex/encode/passbytes.go
Line 24 in 9179024
"...satisfies the json.Unmarshaler interface..."
I think it is the json.Marshaller interface. I could be wrong.
There was a problem hiding this comment.
Tests all PASS on my machine.
| // concurrently. This bugs the hell out of me, because I though that if a | ||
| // database was ACID-compliant, this was impossible, but I guess not. Either | ||
| // way, the solution is to try again. | ||
| func (db *DB) Update(f func(txn *badger.Txn) error) (err error) { |
There was a problem hiding this comment.
Does this need to be exported?
| // lookup and iteration. | ||
| type Table struct { | ||
| *DB | ||
| name string |
There was a problem hiding this comment.
Looks like name isn't used anywhere.
There was a problem hiding this comment.
It's used in (*Table).AddIndex
| type setOpts struct { | ||
| replace bool | ||
| } | ||
|
|
||
| // SetOptions is an knob to control how items are inserted into the table with | ||
| // Set. | ||
| type SetOption func(opts *setOpts) | ||
|
|
||
| // WithReplace allows replacing pre-existing values when calling Set. | ||
| func WithReplace() SetOption { | ||
| return func(opts *setOpts) { | ||
| opts.replace = true | ||
| } | ||
| } |
There was a problem hiding this comment.
This stuff should be above UseDefaultSetOptions.
|
|
||
| // AddIndex adds an index to a Table. Once an Index is added, every datum | ||
| // Set in the Table will generate an entry in the Index too. | ||
| func (t *Table) AddIndex(name string, f func(k, v encoding.BinaryMarshaler) ([]byte, error)) (*Index, error) { |
There was a problem hiding this comment.
What if we want to add an index when the table already has some data? The table could be versioned, and if a new index is added that didn't exist before, then all the existing entries are added to the index, and the version is incremented. This could be added in the future though if it's actually needed.
There was a problem hiding this comment.
We're on the same page. There will be some upgrade logic at some point, but we don't need to do it here. We'll add functionality as we go.
| defer iter.Close() | ||
|
|
||
| if len(seek) == 0 { | ||
| iter.Rewind() |
There was a problem hiding this comment.
Do you need to Rewind if you haven't started iterating yet?
There was a problem hiding this comment.
That's how they do in the example. Maybe not necessary?
dex/lexi/index.go
Outdated
| iter.Seek(seek) | ||
| } | ||
|
|
||
| for ; iter.ValidForPrefix(prefix); iter.Next() { |
There was a problem hiding this comment.
Can just use iter.Valid() since you set the prefix in the opts.
dex/lexi/index.go
Outdated
| // ErrDeleteEntry can be returned from the function passed to Iterate to | ||
| // trigger deletion of the datum, all of its index entries, and its key-id | ||
| // entries. | ||
| ErrDeleteEntry = dex.ErrorKind("delete entry") |
There was a problem hiding this comment.
You probably know, but this isn't implemented.
There was a problem hiding this comment.
Oh right. Did (*Iter).Delete() instead.
dex/lexi/keyprefix.go
Outdated
|
|
||
| func lastKeyForPrefix(txn *badger.Txn, p keyPrefix) (k []byte) { | ||
| reverseIteratePrefix(txn, p[:], nil, func(iter *badger.Iterator) error { | ||
| k = iter.Item().Key()[prefixSize:] |
dex/lexi/index.go
Outdated
| if err != nil { | ||
| return | ||
| } | ||
| return k, item.Value(func(kB []byte) error { |
| // Iterate iterates the index, providing access to the index entry, datum, and | ||
| // datum key via the Iter. |
There was a problem hiding this comment.
Referencing the datum in these comments is confusing when looking at the godoc, as datum is not exported.
There was a problem hiding this comment.
Meh. Datum is also just a word for a single piece of data. Should I use "value" instead?
Add the Lexi DB package, which wraps badger DB to provide a simplified API, to capture the sometimes tedious mechanics that we've repeated in various places, and to add utilities for indexing data for quick retrieval of filtered data.
This change is just the addition of the package. I will demonstrate it's use in tatanka separately.
I'll add some more testing, but feel free to review as is.
See this diff for an example of how it would replace the tx DB for Ethereum.