You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.
I use the below code to save new and old data.
If the table exists - it appends to it. If it does not, it creates it and appends to it.
import pandas as pd
from questdb.ingress import Sender
df = pd.DataFrame({
'id': pd.Categorical(['toronto1', 'paris3']),
'temperature': [20.0, 21.0],
'humidity': [0.5, 0.6],
'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})
conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
sender.dataframe(df, table_name='sensors', at='timestamp')
This causes a few issue:
upon creation it always creates a partition by DAY
Feature request: add a parameter which is by default DAY but can be changed by the user to be MONTH, NONE or other partition frequency
It silently creates a table by default, even if one does not exist.
Feature request: add a parameter which is by default True (to create a table) but can be also set to False such that if a table does not exist, it will not create one.
This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.
The text was updated successfully, but these errors were encountered:
newskooler
changed the title
Add ability to partition table for new tables
Add ability for user to set create_table and table_partition when saving data to QDB
Jun 12, 2024
Hi,
I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.
I use the below code to save new and old data.
If the table exists - it appends to it. If it does not, it creates it and appends to it.
This causes a few issue:
DAY
Feature request: add a parameter which is by default
DAY
but can be changed by the user to beMONTH
,NONE
or other partition frequencyFeature request: add a parameter which is by default
True
(to create a table) but can be also set toFalse
such that if a table does not exist, it will not create one.This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.
The text was updated successfully, but these errors were encountered: