Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EvalStatement throws an AnalysisException when query includes non-existing field #47 #392

Open
kortemik opened this issue Oct 25, 2024 · 1 comment

Comments

@kortemik
Copy link
Member

kortemik commented Oct 25, 2024

ID <433>
Reporter <@q22u>

If a DPL query contains a non-existing field, spark throws an AnalysisException.

For example:
index=index_A | eval a = typeof(12) | eval b = typeof("string") | eval c = typeof(1==2) | eval d = typeof(badfield)

should return "Invalid" instead for ... | eval d = typeof(badfield)

results in

org.apache.spark.sql.AnalysisException: cannot resolve 'badfield' given input columns: [a, partition, host, index, _raw, _time, b, sourcetype, offset, c, source];;
'Project [_raw#28, _time#29, host#30, index#31, offset#32L, partition#33, source#34, sourcetype#35, a#87, b#108, c#131, 'typeOf('badfield) AS d#143]
+- Project [_raw#28, _time#29, host#30, index#31, offset#32L, partition#33, source#34, sourcetype#35, a#87, b#108, UDF((1 = 2)) AS c#131]
+- Project [_raw#28, _time#29, host#30, index#31, offset#32L, partition#33, source#34, sourcetype#35, a#87, UDF(string) AS b#108]
+- Project [_raw#28, _time#29, host#30, index#31, offset#32L, partition#33, source#34, sourcetype#35, UDF(12) AS a#87]
+- Filter index#31 LIKE index_A
+- Relation[_raw#28,_time#29,host#30,index#31,offset#32L,partition#3

should return "Invalid" instead for ... | eval d = typeof(badfield)

Developer Note:
Perhaps when visitEvalFieldType() is called, the existence of given field should be checked?

#95

@kortemik
Copy link
Member Author

Has a disabled test in EvalTest.java called "testEvalTypeofInvalid". Going to be looked at in #295

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant