dynamic parameters in XML extractor

[vitalz]

Need ability to use dynamic parameters in LinkMove XML (also CSV?) extractors. For instance, it is like that:

Parameter 'league' is passed from Java code:
<connectorId>${league}-game</connectorId>

Parameter 'season' is defined in XML extractor itself:

       <attribute>
		<type>java.lang.String</type>
		<source>${season}</source>
		<target>db:season</target>
	</attribute>

<parameters>
	<season>2016/2017</season>
</parameters>

[atomashpolskiy]

You should elaborate more on this. We definitely can't parameterize connectorId. We could potentially use parameters in the XPath expression, but it will require inventing our own syntax. Also, parameters are passed from the calling Java code, they can't be specified in the extractor XML in the way you've posted.

[atomashpolskiy]

I wonder why you need to have a parameterized connectorId? Do you have several documents that share the exact same format? We might think about multi-file connectors in such case

[vitalz]

@atomashpolskiy

• yes, we do - we have similar-formed documents existing on different uris
• java code changes mean release & deployment when LM config change does not require that

[vitalz]

@atomashpolskiy I believe multi-file connectors (or 'multi-connector') look a good idea. But on the other hand, job speed will be reducing because of many XML documents are being processed.

[atomashpolskiy]

java code changes mean release & deployment when LM config change does not require that

Well the params are not obliged to originate in Java code. They might come from an external source, like it happens with the job manager app, or from the configuration like in the newer BQ apps.

I believe multi-file connectors (or 'multi-connector') look a good idea. But on the other hand, job speed will be reducing because of many XML documents are being processed.

Sure, but the total running time obviously will not change, so I don't think it's a big problem. Jobs by definition are background and potentially long-running processes.

[atomashpolskiy]

@vitalz , I've added support for multiple connectors per #120
You may specify multiple connector IDs in the extractor config as follows:

<connectorIds>
    <connectorId>connector1-id</connectorId>
    <connectorId>connector2-id</connectorId>
    ...
<connectorIds>

Params and their values are going to be the same throughout the execution, regardless of the connector from which the data is being extracted, so I think you're still going to need several configs; but this should help reduce the sheer number of them by "grouping" connectors that have the same parameter values