Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability for user to set create_table and table_partition when saving data to QDB #90

Open
newskooler opened this issue Jun 12, 2024 · 0 comments

Comments

@newskooler
Copy link

newskooler commented Jun 12, 2024

Hi,

I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.

I use the below code to save new and old data.
If the table exists - it appends to it. If it does not, it creates it and appends to it.

import pandas as pd
from questdb.ingress import Sender

df = pd.DataFrame({
    'id': pd.Categorical(['toronto1', 'paris3']),
    'temperature': [20.0, 21.0],
    'humidity': [0.5, 0.6],
    'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})

conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    sender.dataframe(df, table_name='sensors', at='timestamp')

This causes a few issue:

  1. upon creation it always creates a partition by DAY

Feature request: add a parameter which is by default DAY but can be changed by the user to be MONTH, NONE or other partition frequency

  1. It silently creates a table by default, even if one does not exist.
    Feature request: add a parameter which is by default True (to create a table) but can be also set to False such that if a table does not exist, it will not create one.

This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.

@newskooler newskooler changed the title Add ability to partition table for new tables Add ability for user to set create_table and table_partition when saving data to QDB Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant