Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(targets): Added a new prepare_table_columns method to the SQLConnector base class #2354

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 35 additions & 5 deletions singer_sdk/connectors/sql.py
Original file line number Diff line number Diff line change
Expand Up @@ -798,12 +798,42 @@ def prepare_table(
)
return

self.prepare_table_columns(
full_table_name=full_table_name,
schema=schema,
)

def prepare_table_columns(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should tests for this. tests/core/test_connector_sql.py might a good place for that.

self,
full_table_name: str,
schema: dict,
) -> None:
"""Adapt target table columns to provided schema if possible.

The advantage of encapsulating this in a separate method is that we reduce
the number of calls to the database by calling get_table_columns() only once
per table.

Args:
full_table_name: the target table name.
schema: the JSON Schema for the table.
"""
columns = self.get_table_columns(
full_table_name=full_table_name,
)
for property_name, property_def in schema["properties"].items():
self.prepare_column(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a deprecation warning when this function is called?

full_table_name,
property_name,
self.to_sql_type(property_def),
)
if property_name not in columns:
self._create_empty_column(
full_table_name=full_table_name,
column_name=property_name,
sql_type=self.to_sql_type(property_def),
)
else:
self._adapt_column_type(
full_table_name,
column_name=property_name,
sql_type=self.to_sql_type(property_def),
)

def prepare_column(
self,
Expand Down