Skip to content

Commit

Permalink
try mysql 8
Browse files Browse the repository at this point in the history
  • Loading branch information
janickr committed Jul 22, 2024
1 parent 13d054d commit 5ca9f29
Show file tree
Hide file tree
Showing 2 changed files with 93 additions and 5 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/service-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,4 @@ jobs:
with:
fetch-depth: 1
- name: mysql version
run: mysql --host 127.0.0.1 --port ${{ job.services.mysql.ports['3306'] }} -uroot -ppassword -e "SELECT @@VERSION"
run: mysql --host 127.0.0.1 --port ${{ job.services.mysql.ports['3306'] }} -uroot -ptest_db -e "SELECT @@VERSION"
96 changes: 92 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,101 @@

A set of mysql stored functions/procedures to read protobuf binary data

## Getting started

### Without `*.proto` files
This is similar to `protoc --decode_raw`.

Run the `myproto.sql` script on your MySQL DB. This script creates the stored functions and procedures necessary to decode protobuf.

#### Decode to textformat
For example to decode_raw the `0x1a03089601` binary data to textformat:
```mysql
select myproto_decode_to_textformat(0x1a03089601, null, null);
```
Returns:
```prototext
3: {
1: 150
}
```
#### Decode to JSON
```mysql
select myproto_decode_to_jsonformat(0x1a03089601, null, null);
```
Returns:
```json
{"3": {"1": 150}}
```

#### Limitations of decode_raw
Decode raw has limitations because protobuf binary data does not contain all info to properly decode the data.
- output will not contain field names, only field numbers
- packed repeated scalar values will be decoded as one binary string
- numbers will be decoded as unsigned integers

If you need proper decoding, then read on and learn how to use information in your `*.proto` files

### Using .proto files
The functions and stored procedures in `myproto.sql` are still needed: run the `myproto.sql` script in MySQL.

Let's say we have a `.proto` file like this:
```protobuf
package foo.bar;
message SubMessage {
optional int32 a = 1;
}
message ParentMessage {
optional SubMessage c = 3;
}
```
We need to compile these `*.proto` files in something MySQL can understand.

1) [Download and install](https://github.com/protocolbuffers/protobuf?tab=readme-ov-file#protobuf-compiler-installation) protoc
2) Install the myprotosql protoc plugin (you need python for this):
```bash
pip install myprotosql
```
3) Run the plugin using protoc:
`protoc --proto_path=<the-path-to-your-proto-files> --myprotosql_out=<the-output-path> --plugin=protoc-gen-myprotosql=<the-path-to-the-myprotosql-plugin> <the-path-to-your-proto-files>\*`
For example, on Windows if you used Virtualenv, with your proto files located in `.\proto` and your virtual env path is `.\venv`:
```bash
.\bin\protoc-27.2-win64\bin\protoc.exe --proto_path=proto --myprotosql_out=build --plugin=protoc-gen-myprotosql=.\venv\Scripts\protoc-gen-myprotosql.exe .\proto\*
```
This will generate a `myproto_descriptors.sql` file.
4) Run the `myproto_descriptors.sql` script in MySQL, this will create a `myproto_descriptors`
function which returns the necessary information to decode protobuf data that conforms to the `.proto` files.


#### Decode to textformat

For example to decode_raw the `0x1a03089601` binary data to textformat:
```mysql
select myproto_decode_to_textformat(0x1a03089601, 'foo.bar.ParentMessage', myproto_descriptors());
```
Returns:
```prototext
c: {
a: 150
}
```
#### Decode to JSON
```mysql
select myproto_decode_to_jsonformat(0x1a03089601, 'foo.bar.ParentMessage', myproto_descriptors());
```
Returns:
```json
{"c": {"a": 150}}
```
```
.\bin\protoc-27.2-win64\bin\protoc.exe --proto_path=proto --python_out=build --myprotosql_out=build --plugin=protoc-gen-myprotosql=.\venv\Scripts\protoc-gen-myprotosql.exe .\proto\*
```
## todo
- packed repeated fields
## Todo
- enums
- todos in code
- maps?
- bytes
- maps
- bytes/proper escaping

0 comments on commit 5ca9f29

Please sign in to comment.