Error: cannot set missing field for required type +PFloat64

[iris-garden]

Note

The following post was exported from discuss.hail.is, a forum for asking questions about Hail which has since been deprecated.

(2023-12-08T19:27:11.966Z) ldomenech said:

Hello,

I want to run a hail variant QC script but it gives me an error. To debug it, I just started checking if hail was importing the vcf file correctly, but it seems there’s something wrong there.

table = hl.import_vcf(
…: “/path/czi.vcf.gz”,
…: force_bgz=True, reference_genome=“GRCh38”,
…: )

table.write(“/path/data_table_hail_czi.mt”)

Java HotSpot™ 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.

Java HotSpot™ 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

CodeCache: size=131072Kb used=40083Kb max_used=43613Kb free=90988Kb

bounds [0x000000010641c000, 0x0000000108f0c000, 0x000000010e41c000]

total_blobs=13894 nmethods=12879 adapters=927

compilation: disabled (not enough contiguous free space left)

---------------------------------------------------------------------------/ 97]

FatalError Traceback (most recent call last)

Cell In[10], line 1

----> 1 table.write(“/path/data_table_hail_czi.mt”)

File ~/anaconda3/lib/python3.11/site-packages/decorator.py:232, in decorate..fun(*args, **kw)

230 if not kwsyntax:

231 args, kw = fix(args, kw, sig)

→ 232 return caller(func, *(extras + args), **kw)

File ~/anaconda3/lib/python3.11/site-packages/hail/typecheck/check.py:587, in _make_dec..wrapper(__original_func, *args, **kwargs)

584 @decorator

585 def wrapper(__original_func: Callable[…, T], *args, **kwargs) → T:

586 args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method)

→ 587 return original_func(*args, **kwargs)

File ~/anaconda3/lib/python3.11/site-packages/hail/matrixtable.py:2734, in MatrixTable.write(self, output, overwrite, stage_locally, _codec_spec, _partitions)

2731 _partitions_type = None

2733 writer = ir.MatrixNativeWriter(output, overwrite, stage_locally, _codec_spec, _partitions, _partitions_type)

→ 2734 Env.backend().execute(ir.MatrixWrite(self._mir, writer))

File ~/anaconda3/lib/python3.11/site-packages/hail/backend/backend.py:180, in Backend.execute(self, ir, timed)

178 result, timings = self._rpc(ActionTag.EXECUTE, payload)

179 except FatalError as e:

→ 180 raise e.maybe_user_error(ir) from None

181 if ir.typ == tvoid:

182 value = None

File ~/anaconda3/lib/python3.11/site-packages/hail/backend/backend.py:178, in Backend.execute(self, ir, timed)

176 payload = ExecutePayload(self._render_ir(ir), ‘{“name”:“StreamBufferSpec”}’, timed)

177 try:

→ 178 result, timings = self._rpc(ActionTag.EXECUTE, payload)

179 except FatalError as e:

180 raise e.maybe_user_error(ir) from None

File ~/anaconda3/lib/python3.11/site-packages/hail/backend/py4j_backend.py:213, in Py4JBackend._rpc(self, action, payload)

211 if resp.status_code >= 400:

212 error_json = orjson.loads(resp.content)

→ 213 raise fatal_error_from_java_error_triplet(error_json[‘short’], error_json[‘expanded’], error_json[‘error_id’])

214 return resp.content, resp.headers.get(‘X-Hail-Timings’, ‘’)

FatalError: HailException: cannot set missing field for required type +PFloat64

Java stack trace:

is.hail.utils.HailException: file:/path/czi.vcf.gz:offset 1952356320: error while parsing line

chr1 10146 rs375931351 AC A,* 2684.05 VQSRTrancheINDEL99.50to99.90 AC=17,3;AF=0.160,0.028;AN=106;AS_BaseQRankSum=-0.500,.;AS_FS=6.513,.;AS_FilterStatus=VQSRTrancheINDEL99.50to99.90,NA;AS_InbreedingCoeff=0.1305,0.2561;AS_MQ=41.52,0.00;AS_MQRankSum=1.500,.;AS_QD=13.56,3.55;AS_ReadPosRankSum=0.900,.;AS_SOR=1.021,.;AS_VQSLOD=-2.0431,NaN;AS_culprit=MQRankSum,NA;BaseQRankSum=-2.590e-01;DB;DP=2358;ExcessHet=0.1278;FS=10.838;InbreedingCoeff=0.1141;MLEAC=24,4;MLEAF=0.226,0.038;MQ=38.93;MQRankSum=1.66;NEGATIVE_TRAIN_SITE;QD=12.90;ReadPosRankSum=0.991;SOR=0.959 GT:AD:DP:GQ:PGT:PID:PL:PS 0/1:2,6,0:8:23:.:.:75,0,23,81,37,118 0/0:62,0,0:62:0:.:.:0,0,358,0,358,358 0/0:42,0,0:42:0:.:.:0,0,0,0,0,0 0/0:41,0,0:41:0:.:.:0,0,0,0,0,0 0/0:47,0,0:47:0:.:.:0,0,305,0,305,305 0/0:67,0,0:67:28:.:.:0,28,1550,28,1550,1550 0/0:84,0,0:84:90:.:.:0,90,1350,90,1350,1350 0/0:63,0,0:63:51:.:.:0,51,765,51,765,765 0/0:40,0,0:40:0:.:.:0,0,0,0,0,0 0/1:2,11,0:13:5:.:.:266,0,5,273,38,311 0/0:54,0,0:54:0:.:.:0,0,178,0,178,178 0/0:94,0,0:94:12:.:.:0,12,2114,12,2114,2114 0/0:40,0,0:40:0:.:.:0,0,0,0,0,0 0/0:55,0,0:55:0:.:.:0,0,180,0,180,180 0/0:69,0,0:69:60:.:.:0,60,900,60,900,900 0/0:35,0,0:35:0:.:.:0,0,114,0,114,114 0/0:49,0,0:49:0:.:.:0,0,325,0,325,325 0/0:61,0,0:61:0:.:.:0,0,100,0,100,100 0/0:55,0,0:55:36:.:.:0,36,540,36,540,540 0/0:41,0,0:41:0:.:.:0,0,290,0,290,290 0/0:58,0,0:58:31:.:.:0,31,1196,31,1196,1196 0/0:80,0,0:80:57:.:.:0,57,855,57,855,855 1/1:1,10,0:11:9:.:.:205,9,0,208,28,227 0/0:59,0,0:59:14:.:.:0,14,1212,14,1212,1212 0/0:39,0,0:39:0:.:.:0,0,0,0,0,0 0/0:65,0,0:65:13:.:.:0,13,1434,13,1434,1434 0/1:4,8,0:12:57:.:.:231,0,57,242,81,324 0/0:48,0,0:48:27:.:.:0,27,405,27,405,405 0/0:28,0,0:28:0:.:.:0,0,131,0,131,131 0/1:2,11,0:13:12:.:.:282,0,12,288,45,332 0/0:40,0,0:40:0:.:.:0,0,0,0,0,0 0/0:70,0,0:70:0:.:.:0,0,288,0,288,288 0/0:48,0,0:48:33:.:.:0,33,495,33,495,495 0/0:67,0,0:67:0:.:.:0,0,591,0,591,591 0/1:4,10,0:14:60:.:.:277,0,60,289,91,380 1/1:0,8,0:8:25:.:.:269,25,0,269,25,269 0/0:30,0,0:30:0:.:.:0,0,0,0,0,0 0/2:11,0,2:13:45:.:.:45,75,292,0,217,211 0/0:55,0,0:55:43:.:.:0,43,675,43,675,675 0/1:6,5,0:11:99:.:.:126,0,133,144,148,293 0/0:67,0,0:67:47:.:.:0,47,1398,47,1398,1398 0/0:49,0,0:49:0:.:.:0,0,463,0,463,463 0/1:5,6,0:11:83:.:.:138,0,83,152,100,252 0/1:4,13,0:17:64:.:.:201,0,64,213,94,307 1/1:0,9,0:9:21:.:.:143,21,0,143,21,143 0/0:58,0,0:58:0:.:.:0,0,306,0,306,300/1:7,5,0:12:99:.:.:106,0,133,127,148,274 0/1:3,5,0:8:44:.:.:136,0,44,145,59,204 0/0:66,0,0:66:0:.:.:0,0,496,0,496,496 0/0:42,0,0:42:33:.:.:0,33,495,33,495,495 2/2:1,0,8:9:17:.:.:213,215,220,17,22,0 0/0:53,0,0:53:65:.:.:0,65,1213,65,1213,1213 0|1:2,37,0:39:1:0|1:10146_AC_A:248,0,1,254,61,315:10146

at

is.hail.utils.HailException: cannot set missing field for required type +PFloat64

at

Hail version: 0.2.126-ee77707f4fab

Error summary: HailException: cannot set missing field for required type +PFloat64

Could you help me with this?

Many thanks!

Laura

(2024-01-03T18:10:26.948Z) ldomenech said:

Hi! I just wanted to check if you have any input for the error specified above. I’m getting the same type of error:

HailException: cannot set missing field for required type +PFloat64

Thanks!

(2024-01-03T18:39:21.950Z) danking said:

ldomenech Apologies, this is a terrible error message. You’re encountering [query] bad error message when user needs to use array_elements_required=False · Issue #13346 · hail-is/hail · GitHub . The quick fix is array_elements_required=False . The root cause is fields like AS_MQRankSum=1.500,. having missing elements. There’s an ambiguity in the VCF spec and Hail refuses to make a choice on your behalf.

This pull request improves the error message: [query] better error message when info array field has missing elements by danking · Pull Request #14105 · hail-is/hail · GitHub

Be forewarned, AS_MQRankSum=. is the ambiguous case. It’s not clear if that means a missing array or an array with one missing elements.