Failure in Input Data Resolution

[arrabito]

Dear all,

Since about 1 week our regular jobs started failing during Input Data Resolution with the following error in the pilot output:

2023-08-17 06:28:21 UTC None/[19790527]JobWrapper WARN: 'InputData'
2023-08-17 06:28:21 UTC None/[19790527]JobWrapper WARN: Optimizer information could not be converted to a dictionary will call catalog directly
2023-08-17 06:28:22 UTC None/[19790527]JobWrapper INFO: Replica Lookup Time: 0.72 seconds 
2023-08-17 06:28:22 UTC None/[19790527]JobWrapper WARN: FileCatalog error ( 1604 : Failed to perform getReplicas from any catalog)
2023-08-17 06:28:22 UTC None/[19790527]JobWrapper INFO: Could not obtain replica information from Optimizer File Catalog information
2023-08-17 06:28:22 UTC None/[19790527]JobWrapper WARN: {'OK': False, 'Errno': 1604, 'Message': 'FileCatalog error ( 1604 : Failed to perform getReplicas from any catalog)', 'CallStack': ['  File "/home/execute/dir_2707373/DIRAC_DYU78Lpilot/job/Wrapper/Wrapper_19790527", line 256, in <module>\n    ret = execute(jobArgs)\n', '  File "/home/execute/dir_2707373/DIRAC_DYU78Lpilot/job/Wrapper/Wrapper_19790527", line 148, in execute\n    result = job.resolveInputData()\n', '  File "/home/execute/dir_2707373/DIRAC_DYU78Lpilot/diracos/lib/python3.11/site-packages/DIRAC/WorkloadManagementSystem/JobWrapper/JobWrapper.py", line 643, in resolveInputData\n    result = self.__checkFileCatalog(lfns, optReplicas)\n', '  File "/home/execute/dir_2707373/DIRAC_DYU78Lpilot/diracos/lib/python3.11/site-packages/DIRAC/WorkloadManagementSystem/JobWrapper/JobWrapper.py", line 709, in __checkFileCatalog\n    replicas = self.__getReplicaMetadata(lfns)\n', '  File "/home/execute/dir_2707373/DIRAC_DYU78Lpilot/diracos/lib/python3.11/site-packages/DIRAC/WorkloadManagementSystem/JobWrapper/JobWrapper.py", line 734, in __getReplicaMetadata\n    repsResult = self.dm.getReplicasForJobs(lfns)\n', '  File "/home/execute/dir_2707373/DIRAC_DYU78Lpilot/diracos/lib/python3.11/site-packages/DIRAC/DataManagementSystem/Client/DataManager.py", line 1702, in getReplicasForJobs\n    result = self.getReplicas(lfns, allStatus=allStatus, getUrl=getUrl)\n', '  File "/home/execute/dir_2707373/DIRAC_DYU78Lpilot/diracos/lib/python3.11/site-packages/DIRAC/DataManagementSystem/Client/DataManager.py", line 1667, in getReplicas\n    res = self.fileCatalog.getReplicas(lfnChunk, allStatus=allStatus)\n', '  File "/home/execute/dir_2707373/DIRAC_DYU78Lpilot/diracos/lib/python3.11/site-packages/DIRAC/Resources/Catalog/FileCatalog.py", line 315, in r_execute\n    return S_ERROR(DErrno.EFCERR, f"Failed to perform {self.call} from any catalog")\n']}

It's a little bit strange, since we haven't changed any setting.

We are using DIRAC 8.0.16.

Also rescheduling old successfull jobs result in the same failure.

Any idea?

Thank you,

Luisa

[chaen]

The error reporting is a bit crappy when there are multiple catalogs.

DIRAC/src/DIRAC/Resources/Catalog/FileCatalog.py

Lines 303 to 311 in 23b7407

	res = method(*parms, **kws)
	if res["OK"]:
	if "Successful" in res["Value"]:
	for key, item in res["Value"]["Successful"].items():
	successful.setdefault(key, item)
	failed.pop(key, None)
	for key, item in res["Value"]["Failed"].items():
	if key not in successful:
	failed[key] = item

You can add a if not res["OK"] condition to try

[arrabito]

The problem is that I cannot reproduce the error on my client. It just happens in jobs during Input Data Resolution, so I cannot easily modify these lines.

However I can temporarely use only 1 catalog to get a clearer message.

[arrabito]

Since I've noticed that times to times I've the error even during the WMS_Optimizer step, I tried to print some info on the server running the WMS_Optimizer and in that case I've got:

n3.9/site-packages/DIRAC/Core/DISET/private/Transports/BaseTransport.py", line 205, in receiveData\n return S_ERROR("Peer closed connection")\n'], 'rpcStub': [['DataManagement/FileCatalog', {'CatalogType': 'FileCatalog', 'AccessType': 'Read-Write', 'Status': 'Active', 'Master': 'True', 'CatalogURL': 'DataManagement/FileCatalog', 'MetaCatalog': 'True', 'timeout': 120, 'skipCACheck': False, 'keepAliveLapse': 150}], 'getReplicas', [{'/vo.cta.in2p3.fr/user/a/arrabito/testFile.txt': True}, False]]}

which seems to indicate a connection problem ?

However this error at the level of the WMS_Optimizer does not happen systematically. I've just tried to submit a second time an identical job and the WMS_Optimizer worked but not the job...

[arrabito]

Just to add that even after having left in the configuration only the DIRAC File Catalog I still get the same error...

[andresailer]

Any error messages in the DIRAC FileCatalog service?

[arrabito]

no error messages...
I'm not even sure if the service gets the request ...

[arrabito]

I cannot find for instance the timestamps related to the error...

[arrabito]

So, suspecting a connectivity problem, I've simply moved the DFC on another server and now jobs work well. It's relatively strange, since the new server is supposed to have the same network configuration as the first one. I will try to investigate with admins. Thanks for your help anyway.