Skip to content

Commit

Permalink
[SPARK-46532][CONNECT][PYTHON][FOLLOW-UP] Pass message parameters in …
Browse files Browse the repository at this point in the history
…Python Spark Connect client

### What changes were proposed in this pull request?

This PR is a followup of #44468 that addresses the additional metadata in Python Spark Connect client.

### Why are the changes needed?

For feature parity.

### Does this PR introduce _any_ user-facing change?

Yes, when `spark.sql.connect.enrichError.enabled` is disabled, users are sill able to get the message parameters.

### How was this patch tested?

Unittest was added.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #44528 from HyukjinKwon/SPARK-46532-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon committed Dec 29, 2023
1 parent f99b86a commit 2e918e8
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 0 deletions.
3 changes: 3 additions & 0 deletions python/pyspark/errors/exceptions/connect.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ def convert_exception(
if "errorClass" in info.metadata:
error_class = info.metadata["errorClass"]

if "messageParameters" in info.metadata:
message_parameters = json.loads(info.metadata["messageParameters"])

stacktrace: Optional[str] = None
if resp is not None and resp.HasField("root_error_idx"):
message = resp.errors[resp.root_error_idx].message
Expand Down
11 changes: 11 additions & 0 deletions python/pyspark/sql/tests/connect/test_connect_basic.py
Original file line number Diff line number Diff line change
Expand Up @@ -3486,6 +3486,17 @@ def test_can_create_multiple_sessions_to_different_remotes(self):
PySparkSession.builder.create()
self.assertIn("Create a new SparkSession is only supported with SparkConnect.", str(e))

def test_get_message_parameters_without_enriched_error(self):
with self.sql_conf({"spark.sql.connect.enrichError.enabled": False}):
exception = None
try:
self.spark.sql("""SELECT a""")
except AnalysisException as e:
exception = e

self.assertIsNotNone(exception)
self.assertEqual(exception.getMessageParameters(), {"objectName": "`a`"})


class SparkConnectSessionWithOptionsTest(unittest.TestCase):
def setUp(self) -> None:
Expand Down

0 comments on commit 2e918e8

Please sign in to comment.