Storing call_data in quotes and order_quotes tables #3124

mstrug · 2024-11-14T00:56:11Z

Description

Added storing of interactions received for /quote response sent by the solver, into new tables: quote_interactions and order_quote_interactions. Storing information if quote was validated in quotes and order_quotes tables.
Quote interactions are used for verification/auditing orders.

Changes

Added new column verified to quotes and order_quotes table.
Added new table: quote_interactions which stores solver returned interactions for particular quote. Data in this table is transient and is removed together with expired quote remove.
Added new table: order_quote_interactions which stores quote interactions permanently for particular order uid.
Added database migration script and updated readme file.

How to test

Existing tests, also added dedicated tests for this new functionality..

MartinquaXD

Given that we now want to store the data in the db the PR also needs a db migration file (database/sql directory) and an update in the db Readme for that.

crates/orderbook/src/database/orders.rs

github-actions · 2024-11-15T01:25:25Z

Reminder: Please update the DB Readme.

Caused by:

database/sql/V075__add_call_data_to_quotes_and_order_quotes.sql

squadgazzz · 2024-11-19T16:14:51Z

Reference discussion: link

Let's avoid sharing links to private discussions in open-source projects and instead share all the required information in the description. @mstrug, could you update the description explaining the reasoning and other considerations, if any?

database/sql/V075__add_call_data_to_quotes_and_order_quotes.sql

crates/database/src/orders.rs

crates/shared/src/trade_finding/external.rs

crates/shared/src/price_estimation/trade_verifier.rs

crates/autopilot/src/database/auction.rs

crates/database/src/orders.rs

mstrug · 2024-11-29T12:15:42Z

Do we really need 2 separate tables? Was it considering storing interactions in the same table as an array?

I was thinking about using only one table with interactions and store quote_id and order_uid for each interaction, but decided to use separate tables, as not all quote interactions are then stored in quote order interactions table. In my opinion it is better to have one table with transient data more frequently updated and the other one with persistent data.

database/sql/V075__add_call_data_to_quotes_and_order_quotes.sql

crates/database/src/quotes.rs

crates/database/src/orders.rs

crates/autopilot/src/database/auction.rs

crates/database/src/quotes.rs

MartinquaXD · 2024-12-02T07:02:25Z

crates/database/src/quotes.rs

+    SELECT q.*, i.index, i.target, i.value, i.call_data FROM quotes q
+    JOIN quote_interactions i ON quote_id = id
+    WHERE id = $1


Is it possible to aggregate the interactions directly in the quote row using SQL?

Yes, it can be done using Postgres aggregate function with custom sqlx::FromRow trait implementation, I've developed working solution which after some cleanups I will commit.

crates/orderbook/src/database/orders.rs

MartinquaXD · 2024-12-02T07:13:30Z

database/README.md

+Solver responding to quote request provides a list of interactions which need to be executed to fulfill the quote. When order is created
+these interactions are copied from quote. These interactions are stored persistently and can be used to audit auction winning order.


This should be treated like user facing documentation and should therefore use proper grammar.

database/sql/V075__add_call_data_to_quotes_and_order_quotes.sql

MartinquaXD · 2024-12-02T07:17:01Z

database/sql/V075__add_call_data_to_quotes_and_order_quotes.sql

+);
+
+-- Get a specific quote's interactions.
+CREATE INDEX quote_id_interactions ON quote_interactions USING HASH (quote_id);


Was there a particular reason to not use the default index type (BTree) here?

I decided to use hash because we don't need to query for ranges and hash map has little better performance on average (O(1) vs O(N log N)). Also we don't need to have unique index here.

crates/database/src/quotes.rs

squadgazzz · 2024-12-02T11:07:54Z

crates/shared/src/order_quoting.rs

+impl TryFrom<database::quotes::QuoteWithInteractions> for QuoteData {
+    type Error = anyhow::Error;
+
+    fn try_from(input: database::quotes::QuoteWithInteractions) -> Result<QuoteData> {


Either this:

Suggested change

fn try_from(input: database::quotes::QuoteWithInteractions) -> Result<QuoteData> {

fn try_from((quote, interaction): database::quotes::QuoteWithInteractions) -> Result<QuoteData> {

or use a regular QuoteWithInteractions struct with named arguments.

QuoteWithInteractions has been refactored to contain same fields as Quote struct + interactions vector.

MartinquaXD · 2024-12-03T10:34:34Z

Due to a recent change in the API we'd need to store a bunch more data for a quote. Specifically we'd also have to store pre-interactions and JIT orders returned in a quote.
I'm a bit worried that this will create even more tables. The pre-interactions could be stored in the same tables the PR already introduces (by adding another column which kind of interaction it is) but the JIT orders would theoretically need 2 more tables in the current approach. :/

Since we don't really need to introspect the quote data in the backend I'm considering whether it makes more sense to just add a single JSON column to the quotes and order_quotes tables that contain all the details: pre-interactions, regular interactions, JIT orders, clearing prices. WDYT?

mstrug · 2024-12-03T11:03:28Z

Due to a recent change in the API we'd need to store a bunch more data for a quote. Specifically we'd also have to store pre-interactions and JIT orders returned in a quote. I'm a bit worried that this will create even more tables. The pre-interactions could be stored in the same tables the PR already introduces (by adding another column which kind of interaction it is) but the JIT orders would theoretically need 2 more tables in the current approach. :/

Since we don't really need to introspect the quote data in the backend I'm considering whether it makes more sense to just add a single JSON column to the quotes and order_quotes tables that contain all the details: pre-interactions, regular interactions, JIT orders, clearing prices. WDYT?

So we will need to have one column of type json, with predefined schema. The only thing which can be not so optimal is updating data in that column. If we would need to add some data then we need to read json from the column and write updated json back. But if this is not going to be a frequent action, then there should be no issue with that.

MartinquaXD · 2024-12-03T16:25:37Z

I don't see a need to ever update the columns. The only thing that could happen is that in the future we want to store a different JSON format but in that case I'd say we should just leave the old values untouched.
If we want to return the data from an API we could simply not expect any format and return whatever we find in the DB. Overall this data is mostly for debugging purposes and not that critical.

Implementation

f46b82d

MartinquaXD reviewed Nov 14, 2024

View reviewed changes

crates/orderbook/src/database/orders.rs Outdated Show resolved Hide resolved

mstrug added 5 commits November 15, 2024 01:38

Added call_data to quotes table

d9eb189

Updated order_quotes table

2c0e92e

Added comments

006a873

Added db migration

7d2bb01

Fixed formatting

9947257

mstrug added 6 commits November 15, 2024 14:51

Fixed sql query

692d94e

Updated db querying for order with quote

6fb1ae7

Added test

a8f349f

Merge branch 'main' into feature/storing-call-data-in-quotes

eaf09d2

Fixed formatting

67c20ff

Updated tests

e9026a4

mstrug marked this pull request as ready for review November 15, 2024 22:28

mstrug requested a review from a team as a code owner November 15, 2024 22:28

squadgazzz reviewed Nov 19, 2024

View reviewed changes

database/sql/V075__add_call_data_to_quotes_and_order_quotes.sql Outdated Show resolved Hide resolved

m-lord-renkse reviewed Nov 19, 2024

View reviewed changes

database/sql/V075__add_call_data_to_quotes_and_order_quotes.sql Show resolved Hide resolved

m-lord-renkse reviewed Nov 19, 2024

View reviewed changes

crates/database/src/orders.rs Show resolved Hide resolved

m-lord-renkse reviewed Nov 19, 2024

View reviewed changes

crates/shared/src/trade_finding/external.rs Outdated Show resolved Hide resolved

squadgazzz reviewed Nov 19, 2024

View reviewed changes

crates/shared/src/price_estimation/trade_verifier.rs Outdated Show resolved Hide resolved

mstrug added 2 commits November 21, 2024 10:38

Added verified column to quotes table

6671767

Added quotes_interactions table

086e579

mstrug marked this pull request as draft November 23, 2024 00:20

mstrug added 5 commits November 23, 2024 01:29

Fixed tests compilation

bca89ac

Fixed formatting

7c6983c

Updated orders database tables

a847647

Fixed sql

1868aae

Update

507f6ae

squadgazzz reviewed Nov 29, 2024

View reviewed changes

crates/autopilot/src/database/auction.rs Outdated Show resolved Hide resolved

squadgazzz reviewed Nov 29, 2024

View reviewed changes

crates/database/src/orders.rs Outdated Show resolved Hide resolved

Added transaction for storing interactions

5239901

squadgazzz reviewed Nov 29, 2024

View reviewed changes

database/sql/V075__add_call_data_to_quotes_and_order_quotes.sql Outdated Show resolved Hide resolved

squadgazzz reviewed Nov 29, 2024

View reviewed changes

crates/database/src/quotes.rs Outdated Show resolved Hide resolved

squadgazzz reviewed Nov 29, 2024

View reviewed changes

crates/database/src/quotes.rs Outdated Show resolved Hide resolved

mstrug added 8 commits November 29, 2024 15:12

Updated removing of expired quote interactions

5aac1dc

Removed updated on conflict from insert order quote interaction

23b2182

Updated test

f4f040c

Updated db readme

8848b73

Ensuring repeated db inserts uses transaction

0afb9ad

Fixed tests

2f610ae

Optimized db queries using join

9979645

Added test

9a7f510

MartinquaXD reviewed Dec 2, 2024

View reviewed changes

Small refactorings

d8ff00b

squadgazzz reviewed Dec 2, 2024

View reviewed changes

crates/database/src/quotes.rs Outdated Show resolved Hide resolved

squadgazzz reviewed Dec 2, 2024

View reviewed changes

crates/database/src/quotes.rs Outdated Show resolved Hide resolved

squadgazzz reviewed Dec 2, 2024

View reviewed changes

crates/database/src/quotes.rs Outdated Show resolved Hide resolved

squadgazzz reviewed Dec 2, 2024

View reviewed changes

crates/database/src/quotes.rs Outdated Show resolved Hide resolved

squadgazzz reviewed Dec 2, 2024

View reviewed changes

mstrug added 5 commits December 2, 2024 12:25

Function rename

726376a

Using array_agg function for getting quote with interactions from db

1e23f0d

Removed new error variant, fixed comment in sql script

f5cbc3e

Merge branch 'main' into feature/storing-call-data-in-quotes

59e39a2

Updated table descriptions

ddaf4b2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storing call_data in quotes and order_quotes tables #3124

Storing call_data in quotes and order_quotes tables #3124

mstrug commented Nov 14, 2024 •

edited

Loading

MartinquaXD left a comment

github-actions bot commented Nov 15, 2024

squadgazzz commented Nov 19, 2024

mstrug commented Nov 29, 2024

MartinquaXD Dec 2, 2024

mstrug Dec 2, 2024

MartinquaXD Dec 2, 2024

mstrug Dec 3, 2024

MartinquaXD Dec 2, 2024

mstrug Dec 3, 2024

squadgazzz Dec 2, 2024

mstrug Dec 3, 2024

MartinquaXD commented Dec 3, 2024

mstrug commented Dec 3, 2024

MartinquaXD commented Dec 3, 2024

		Solver responding to quote request provides a list of interactions which need to be executed to fulfill the quote. When order is created
		these interactions are copied from quote. These interactions are stored persistently and can be used to audit auction winning order.

	fn try_from(input: database::quotes::QuoteWithInteractions) -> Result<QuoteData> {
	fn try_from((quote, interaction): database::quotes::QuoteWithInteractions) -> Result<QuoteData> {

Storing call_data in quotes and order_quotes tables #3124

Are you sure you want to change the base?

Storing call_data in quotes and order_quotes tables #3124

Conversation

mstrug commented Nov 14, 2024 • edited Loading

Description

Changes

How to test

MartinquaXD left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 15, 2024

squadgazzz commented Nov 19, 2024

mstrug commented Nov 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MartinquaXD commented Dec 3, 2024

mstrug commented Dec 3, 2024

MartinquaXD commented Dec 3, 2024

mstrug commented Nov 14, 2024 •

edited

Loading