modifying execute API to get column nullability state #686

minurajeeve · 2023-11-27T01:28:47Z

Wrangler will now have two types of columns : Nullable and Non-nullable. This PR handles all the code changes that are required to set the column as Nullable or Non-nullable (column Nullability state).

wrangler-proto/src/main/java/io/cdap/wrangler/proto/workspace/v2/DirectiveExecutionRequest.java

wrangler-proto/src/main/java/io/cdap/wrangler/proto/workspace/v2/Workspace.java

wrangler-service/src/main/java/io/cdap/wrangler/service/directive/AbstractDirectiveHandler.java

wrangler-service/src/main/java/io/cdap/wrangler/service/directive/RemoteExecutionTask.java

…ad of Workspace object

…s not required

albertshau

left general style/practice comments, but would like to understand why we want to introduce null specific handling when existing directives can already do mostly the same thing with more flexibility.

albertshau · 2024-01-04T17:55:59Z

wrangler-api/src/main/java/io/cdap/wrangler/api/NullHandlingException.java

@@ -0,0 +1,31 @@
+/*
+ * Copyright © 2016-2019 Cask Data, Inc.


albertshau · 2024-01-04T17:57:11Z

wrangler-api/src/main/java/io/cdap/wrangler/api/NullHandlingException.java

+/**
+ * A Null Handling specific exception used for communicating issues with Null Handling in a column.
+ */
+public class NullHandlingException extends Exception {


I don't see this being thrown or caught anywhere, how are we expecting to use it? Would be better to leave this out of the PR and include it in whatever PR actually uses it.

It was for future use, removed

albertshau · 2024-01-04T17:58:03Z

wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java

@@ -56,10 +59,13 @@ public final class RecipePipelineExecutor implements RecipePipeline<Row, Structu
  private final RecipeParser recipeParser;
  private final ExecutorContext context;
  private List<Directive> directives;
+  private HashMap<String, UserDefinedAction> nullabilityMap;


HashMap -> Map. In general, when declaring variables we should be using the interface and not the specific implementation.

this should also be final. Basically anything passed into the constructor should be final

albertshau · 2024-01-04T17:58:14Z

wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java


-  public RecipePipelineExecutor(RecipeParser recipeParser, @Nullable ExecutorContext context) {
+  public RecipePipelineExecutor(RecipeParser recipeParser, @Nullable ExecutorContext context,
+      HashMap<String, UserDefinedAction> nullabilityMap) {


HashMap -> Map

albertshau · 2024-01-04T17:59:04Z

wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java

    this.context = context;
    this.recipeParser = recipeParser;
+    this.nullabilityMap = nullabilityMap;


should do a defensive copy to make sure the map cannot change.

albertshau · 2024-01-04T18:10:34Z

wrangler-proto/src/main/java/io/cdap/wrangler/proto/workspace/v2/Workspace.java

+   */
+  public enum UserDefinedAction {
+    SKIP_ROW,
+    SEND_TO_ERROR_COLLECTOR,


ERROR_COLLECTOR is a pipeline specific concept, this can just be SEND_TO_ERROR

But by SEND_TO_ERROR_COLLECTOR, I actually mean the pipeline specific error collector hence the name.

albertshau · 2024-01-04T18:11:02Z

wrangler-proto/src/main/java/io/cdap/wrangler/proto/workspace/v2/Workspace.java

+   * UserDefinedAction enum.
+   */
+  public enum UserDefinedAction {
+    SKIP_ROW,


a better name would be FILTER

albertshau · 2024-01-04T18:12:37Z

wrangler-service/src/main/java/io/cdap/wrangler/service/directive/WorkspaceHandler.java

@@ -472,6 +474,12 @@ private DirectiveExecutionResponse execute(NamespaceSummary ns, HttpServiceReque

    WorkspaceDetail detail = wsStore.getWorkspaceDetail(workspaceId);
    UserDirectivesCollector userDirectivesCollector = new UserDirectivesCollector();
+    HashMap<String, UserDefinedAction> nullabilityMap = executionRequest.getNullabilityMap() == null ?


this logic should be in the getNullabilityMap() class

albertshau · 2024-01-04T18:12:50Z

wrangler-service/src/main/java/io/cdap/wrangler/service/directive/WorkspaceHandler.java

+    } catch (Exception e) {
+      throw new RuntimeException("Error in setting nullabilityMap of columns ", e);
+    }
+    }


indentation

albertshau · 2024-01-04T18:14:23Z

wrangler-service/src/main/java/io/cdap/wrangler/service/directive/WorkspaceHandler.java

+        new HashMap<>() : executionRequest.getNullabilityMap();
+    if (!nullabilityMap.isEmpty()) {
+      //change nullabilityMap in Workspace Object
+      changeNullability(nullabilityMap, workspaceId);


should update the workspace after executing the directives (this is already happening), not before. Otherwise the execution can fail and now there's a partially updated workspace.

We need the updated nullabilityMap before directives are executed.

minurajeeve force-pushed the null-handling-in-wrangler branch 4 times, most recently from f62d151 to 1dc23e5 Compare November 28, 2023 08:17

minurajeeve added the build Triggers unit test build label Nov 28, 2023

minurajeeve self-assigned this Nov 28, 2023

minurajeeve requested review from saimukkamala and harshdeeppruthi November 28, 2023 08:19

minurajeeve force-pushed the null-handling-in-wrangler branch from 1dc23e5 to f9fcbc1 Compare November 28, 2023 09:14

modifying execute API to get column nullability state

080216e

minurajeeve force-pushed the null-handling-in-wrangler branch from f9fcbc1 to 080216e Compare November 28, 2023 11:38

harshdeeppruthi reviewed Dec 1, 2023

View reviewed changes

minurajeeve added 2 commits December 1, 2023 15:07

modifying getter and setter names for NullabilityMap

d9cf5bc

setting nullabilityMap in workspace

6ae414e

minurajeeve requested review from tivv and sau42shri December 5, 2023 16:11

minurajeeve added 2 commits December 12, 2023 16:06

modifying executeDirectives definition to intake nullabilityMap inste…

b1fa542

…ad of Workspace object

removing NO_ACTION and NULLABLE options from UserDefinedAction as it'…

aa0e088

…s not required

minurajeeve requested a review from harshdeeppruthi December 12, 2023 10:45

albertshau reviewed Jan 4, 2024

View reviewed changes

minurajeeve requested a review from albertshau February 8, 2024 10:02

addressing review comments -1

e4a3a43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

modifying execute API to get column nullability state #686

modifying execute API to get column nullability state #686

minurajeeve commented Nov 27, 2023 •

edited

Loading

albertshau left a comment

albertshau Jan 4, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

albertshau Jan 4, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

albertshau Jan 4, 2024

minurajeeve Feb 8, 2024

modifying execute API to get column nullability state #686

Are you sure you want to change the base?

modifying execute API to get column nullability state #686

Conversation

minurajeeve commented Nov 27, 2023 • edited Loading

albertshau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

minurajeeve commented Nov 27, 2023 •

edited

Loading