ch 6 map -> create_map

[dpuddu86]

Spark-The-Definitive-Guide/code/Structured_APIs-Chapter_6_Working_with_Different_Types_of_Data.py

Line 313 in 38e8814

df.select(map(col("Description"), col("InvoiceNo")).alias("complex_map"))\

It seems SCALA version, same as #L319

[bllchmbrs]

Sorry, what's the issue? Could you open a pull request with the change?

[dpuddu86]

Hi,
lines 313 to 320 seem not working using pySpark.
The map command seems SCALA and not pyspark language, I think it should be replaced with create_map.
Even the book "Spark, The Definitive Guide" seems affected by the same error at pag 109.

I'm sorry but I'm not able to open a pull request. I'm just using the code as a reference for a tutorial on Databricks community available online. This is the first issue I report.

The following lines may help to understand.
L307(WORKING using create_map)
df.select(create_map(col("Description"), col("InvoiceNo")).alias("complex_map"))
.show(2)

L313 (NOT WORKING, here we have map instead of create_map of line 307)
df.select(map(col("Description"), col("InvoiceNo")).alias("complex_map"))
.selectExpr("complex_map['WHITE METAL LANTERN']").show(2)

L319 (NOT WORKING, here we have map instead of create_map of line 307)
df.select(map(col("Description"), col("InvoiceNo")).alias("complex_map"))
.selectExpr("explode(complex_map)").show(2)

[JoeKastner]

Just came across this as well - I'll submit a PR with the fix.