profiling and coverage will instrument your code to include calls to profile and coverage data collector logic.
for that purpose instrument will disable all inlining of included types except for abstracts. additionally it will switch off all null safety metadata on types and fields included for instrumentation.
for an example use of instrument run haxe build.hxml and play with different cli options (see commented out lines in build.hxml)
requires Haxe 4.1.x or higher
-lib instrument
--macro instrument.Instrumentation.coverage([include packages] | fileName, [include folders] | fileName, [exclude packages] | fileName)- include packages - takes an array of package and type names you want to collect coverage data on. names are matched against fully qualified type names ("
package name.type name") usingStringTools.startsWith - include folders - array of folders containing code for coverage data collection.
Compiler.includeis used to make sure your types get included - exclude packages - array of packages, types and fields to ignore, e.g. because they are for a different target. again using
StringTools.startsWithto match fully qualified type name ("package name.type name") and field names ("package name.type name.field name") against list.
instead of passing an array each parameter will also take a filename pointing to a file containing package, type, field or folder names, allowing you to shorten the macro call. files should contain one entry per line with a \n as line break. there is no difference in behaviour between passing an array or using files for configuration.
make sure you include your source folder(s), otherwise you will only see coverage stats of types touched during run.
coverage instrumentation runs independently of a test framework. so you can create a coverage report from a "normal" run of your code if you like.
confirmed to work with utest and munit.
coverage uses a resource named coverageTypeInfo to store type information it collects during compiletime so it's available for runtime coverage report generation.
-D coverage-console-summary-reporter - prints a summary of coverage stats to your console
=====================================
Coverage summary
=====================================
packages 6/7 (85.71%)
types 31/37 (83.78%)
fields 177/201 (88.05%)
branches 399/504 (79.16%)
expressions 1398/1451 (96.34%)
files 31/33 (93.93%)
lines 1472/1540 (95.58%)
=====================================
Overall: 91.55%
=====================================
overall coverage is (covered fields + covered branches + covered expressions) / (total fields + total branches + total expressions)
-D coverage-console-package-summary-reporter - prints a package summary of coverage stats to your console
=========================================================================================================================================================
| Files | Types | Fields | Branches | Expression | Lines | Overall
Package | Rate Num | Rate Num | Rate Num | Rate Num | Rate Num | Rate Num |
=========================================================================================================================================================
demo | 0% 0/2 | 0% 0/4 | 0% 0/18 | 0% 0/0 | 0% 0/0 | 0% 0/0 | 0%
instrument | 100% 3/3 | 100% 3/3 | 85.71% 6/7 | 80% 40/50 | 96.96% 64/66 | 94.8% 73/77 | 89.43%
instrument.coverage | 100% 9/9 | 100% 9/9 | 100% 34/34 | 100% 50/50 | 100% 235/235 | 100% 217/217 | 100%
instrument.coverage.reporter | 100% 8/8 | 80% 8/10 | 98.52% 67/68 | 85.78% 175/204 | 98.59% 700/710 | 97.78% 793/811 | 95.92%
instrument.profiler | 100% 1/1 | 100% 1/1 | 100% 7/7 | 58.92% 33/56 | 90.78% 128/141 | 88.96% 129/145 | 82.35%
instrument.profiler.reporter | 100% 7/7 | 100% 7/7 | 97.91% 47/48 | 66.93% 83/124 | 88.23% 195/221 | 86.15% 168/195 | 82.69%
instrument.profiler.summary | 100% 3/3 | 100% 3/3 | 84.21% 16/19 | 90% 18/20 | 97.43% 76/78 | 96.84% 92/95 | 94.01%
=========================================================================================================================================================
Total: | 93.93% 31/33 | 83.78% 31/37 | 88.05% 177/201 | 79.16% 399/504 | 96.34% 1398/1451 | 95.58% 1472/1540 | 91.55%
=========================================================================================================================================================
-D coverage-console-file-summary-reporter - prints a file by file summary of coverage stats to your console
======================================================================================================================================================================
| Types | Fields | Branches | Expression | Lines | Overall
FileName | Rate Num | Rate Num | Rate Num | Rate Num | Rate Num |
======================================================================================================================================================================
[src/instrument/]
FileBaseReporter.hx | 100% 1/1 | 100% 2/2 | 65% 13/20 | 100% 20/20 | 88.88% 8/9 | 83.33%
Instrumentation.hx | 100% 1/1 | 66.66% 2/3 | 70% 7/10 | 88.23% 15/17 | 84.21% 16/19 | 80%
InstrumentationType.hx | 100% 1/1 | 100% 2/2 | 100% 20/20 | 100% 29/29 | 100% 49/49 | 100%
coverage/BranchInfo.hx | 100% 1/1 | 100% 4/4 | 100% 2/2 | 100% 10/10 | 100% 8/8 | 100%
coverage/BranchesInfo.hx | 100% 1/1 | 100% 5/5 | 100% 4/4 | 100% 29/29 | 100% 30/30 | 100%
coverage/Coverage.hx | 100% 1/1 | 100% 3/3 | 0% 0/0 | 100% 15/15 | 100% 16/16 | 100%
coverage/ExpressionInfo.hx | 100% 1/1 | 100% 4/4 | 100% 2/2 | 100% 10/10 | 100% 8/8 | 100%
coverage/FieldInfo.hx | 100% 1/1 | 100% 5/5 | 100% 4/4 | 100% 40/40 | 100% 45/45 | 100%
coverage/FileInfo.hx | 100% 1/1 | 100% 3/3 | 100% 4/4 | 100% 27/27 | 100% 25/25 | 100%
coverage/LineCoverage.hx | 100% 1/1 | 100% 3/3 | 100% 16/16 | 100% 32/32 | 100% 21/21 | 100%
coverage/PackageInfo.hx | 100% 1/1 | 100% 3/3 | 100% 8/8 | 100% 35/35 | 100% 31/31 | 100%
coverage/TypeInfo.hx | 100% 1/1 | 100% 4/4 | 100% 10/10 | 100% 37/37 | 100% 33/33 | 100%
coverage/reporter/CodecovCoverageReporter.hx | 100% 1/1 | 100% 3/3 | 100% 12/12 | 100% 45/45 | 100% 47/47 | 100%
coverage/reporter/ConsoleCoverageFileSummaryReporter.hx | 100% 1/1 | 100% 13/13 | 76.82% 63/82 | 97.07% 166/171 | 94.32% 183/194 | 90.97%
coverage/reporter/ConsoleCoveragePackageSummaryReporter.hx | 100% 1/1 | 100% 9/9 | 80% 16/20 | 96.96% 64/66 | 97.56% 120/123 | 93.68%
coverage/reporter/ConsoleCoverageSummaryReporter.hx | 100% 1/1 | 100% 4/4 | 50% 2/4 | 95.45% 21/22 | 95.45% 21/22 | 90%
coverage/reporter/ConsoleMissingCoverageReporter.hx | 100% 1/1 | 100% 10/10 | 93.75% 30/32 | 100% 72/72 | 98.59% 70/71 | 98.24%
coverage/reporter/EMMACoverageReporter.hx | 50% 1/2 | 100% 10/10 | 100% 6/6 | 100% 81/81 | 100% 80/80 | 100%
coverage/reporter/JaCoCoXmlCoverageReporter.hx | 50% 1/2 | 100% 12/12 | 100% 16/16 | 100% 118/118 | 100% 127/127 | 100%
coverage/reporter/LcovCoverageReporter.hx | 100% 1/1 | 85.71% 6/7 | 93.75% 30/32 | 98.51% 133/135 | 98.63% 145/147 | 97.12%
profiler/Profiler.hx | 100% 1/1 | 100% 7/7 | 58.92% 33/56 | 90.78% 128/141 | 88.96% 129/145 | 82.35%
profiler/reporter/CPUProfileReporter.hx | 100% 1/1 | 100% 8/8 | 59.09% 26/44 | 87.14% 61/70 | 92.53% 62/67 | 77.86%
profiler/reporter/CSVSummaryReporter.hx | 100% 1/1 | 100% 7/7 | 62.5% 10/16 | 89.65% 26/29 | 81.81% 18/22 | 82.69%
profiler/reporter/ConsoleDetailReporter.hx | 100% 1/1 | 100% 6/6 | 0% 0/0 | 100% 6/6 | 100% 7/7 | 100%
profiler/reporter/ConsoleHierarchyReporter.hx | 100% 1/1 | 100% 7/7 | 77.77% 14/18 | 96.42% 27/28 | 96.77% 30/31 | 90.56%
profiler/reporter/ConsoleMissedExitReporter.hx | 100% 1/1 | 83.33% 5/6 | 50% 3/6 | 66.66% 18/27 | 53.84% 14/26 | 66.66%
profiler/reporter/ConsoleSummaryReporter.hx | 100% 1/1 | 100% 7/7 | 62.5% 10/16 | 89.65% 26/29 | 81.81% 18/22 | 82.69%
profiler/reporter/D3FlameHierarchyReporter.hx | 100% 1/1 | 100% 7/7 | 83.33% 20/24 | 96.87% 31/32 | 95% 19/20 | 92.06%
profiler/summary/FlatSummary.hx | 100% 1/1 | 80% 4/5 | 100% 4/4 | 100% 13/13 | 100% 18/18 | 95.45%
profiler/summary/HierarchicalSummary.hx | 100% 1/1 | 80% 4/5 | 0% 0/0 | 100% 19/19 | 100% 29/29 | 95.83%
profiler/summary/HierarchyCallData.hx | 100% 1/1 | 88.88% 8/9 | 87.5% 14/16 | 95.65% 44/46 | 93.75% 45/48 | 92.95%
[srcDemo/demo/]
Hello.hx | 0% 0/1 | 0% 0/1 | 0% 0/0 | 0% 0/0 | 0% 0/0 | 0%
MyTestApp.hx | 0% 0/3 | 0% 0/17 | 0% 0/0 | 0% 0/0 | 0% 0/0 | 0%
======================================================================================================================================================================
Total: | 83.78% 31/37 | 88.05% 177/201 | 79.16% 399/504 | 96.34% 1398/1451 | 95.58% 1472/1540 | 91.55%
======================================================================================================================================================================
-D coverage-console-missing-reporter - prints a line for every type, field, branch and expression with no coverage
======================
== missing coverage ==
======================
srcDemo/demo/Hello.hx:3: type Hello not covered
srcDemo/demo/Hello.hx:4: field main not covered
srcDemo/demo/Hello.hx:5: expression not covered
srcDemo/demo/MyTestApp.hx:35: branch not covered
srcDemo/demo/MyTestApp.hx:36: expression not covered
srcDemo/demo/MyTestApp.hx:37: branch not covered
srcDemo/demo/MyTestApp.hx:38: expression not covered
srcDemo/demo/MyTestApp.hx:39: branch not covered
srcDemo/demo/MyTestApp.hx:40: expression not covered
srcDemo/demo/MyTestApp.hx:53: branch not covered
srcDemo/demo/MyTestApp.hx:54: expression not covered
srcDemo/demo/MyTestApp.hx:59: branch not covered
srcDemo/demo/MyTestApp.hx:60: expression not covered
srcDemo/demo/MyTestApp.hx:65: branch not covered
srcDemo/demo/MyTestApp.hx:66: expression not covered
srcDemo/demo/MyTestApp.hx:68: branch not covered
srcDemo/demo/MyTestApp.hx:74: branch not covered
srcDemo/demo/MyTestApp.hx:131: branch not covered
srcDemo/demo/MyTestApp.hx:134: expression not covered
srcDemo/demo/MyTestApp.hx:138: branch not covered
srcDemo/demo/MyTestApp.hx:151: field download not covered
srcDemo/demo/MyTestApp.hx:152: expression not covered
srcDemo/demo/MyTestApp.hx:154: type Role not covered
writes coverage data using lcov format to a file. filename defaults to lcov.info in your workspace root. you can set a different name and folder. lcov reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
includes full coverage data down to lines and branches (visualisation of partially covered branches might depend on your tooling, some may show it, some won't).
-D coverage-lcov-reporter
-D coverage-lcov-reporter=lcov.infowrites coverage data using Codecov's Json coverage foramt. filename defaults to codecov.json in your workspace root. you can set a different name and folder. Codecov reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
includes line coverage for each file, partial branches show up as "1/2" or "3/4".
-D coverage-codecov-reporter
-D coverage-codecov-reporter=codecov.jsonwrites coverage data using emma xml format to a file. filename defaults to emma-coverage.xml in your workspace root. you can set a different name and folder. emma reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
only supports coverage down to method level (no branch and line coverage)
-D coverage-emma-reporter
-D coverage-emma-reporter=emma-coverage.xmlwrites coverage data using JaCoCo xml format to a file. filename defaults to jacoco-coverage.xml in your workspace root. you can set a different name and folder. JaCoCo reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
includes line coverage for each file.
-D coverage-jacocoxml-reporter or -D coverage-jacocoxml-reporter=jacoco-coverage.xml-lib instrument
--macro instrument.Instrumentation.profiling([include packages] | fileName, [include folders] | fileName, [exclude packages] | fileName)- include packages - takes an array of package and type names you want to collect coverage data on. names are matched against fully qualified type names ("
package name.type name") usingStringTools.startsWith - include folders - array of folders containing code for coverage data collection.
Compiler.includeis used to make sure your types get included - exclude packages - array of packages, types and fields to ignore, e.g. because they are for a different target. again using
StringTools.startsWithto match fully qualified type name ("package name.type name") and field names ("package name.type name.field name") against list.
instead of passing an array each parameter will also take a filename pointing to a file containing package, type, field or folder names, allowing you to shorten the macro call. files should contain one entry per line with a \n as line break. there is no difference in behaviour between passing an array or using files for configuration.
profiling was written with multithreading in mind, but's currently untested
-D profiler-console-detail-reporter - prints entry and exit of every function as it happens including call id, location type and function name and duration of call
>>> [2] srcDemo/demo/Hello.hx:4: Hello.main
srcDemo/demo/Hello.hx:5: Hello Haxe
<<< [2] srcDemo/demo/Hello.hx:4: Hello.main 0.080108642578125ms
-D profiler-console-summary-reporter - prints a summary listing all locations, fields, number of invocations and duration spent inside (includes duration of calls to other functions)
==================
== Call Summary ==
==================
srcDemo/demo/MyTestApp.hx:143: MyTestApp.main 1 3.10015678405761719ms
srcDemo/demo/MyTestApp.hx:6: MyTestApp.new 1 1.54900550842285156ms
srcDemo/demo/MyTestApp.hx:117: MyTestApp.noBody4 1 0.174999237060546875ms
srcDemo/demo/MyTestApp.hx:148: MyTestApp_Fields_.moduleBody 7 0.14591217041015625ms
srcDemo/demo/MyTestApp.hx:71: MyTestApp.opBoolOr 3 0.145196914672851562ms
srcDemo/demo/MyTestApp.hx:89: MyTestApp.initTranslations 1 0.134944915771484375ms
srcDemo/demo/MyTestApp.hx:102: MyTestApp.noBody3 2 0.129222869873046875ms
srcDemo/demo/MyTestApp.hx:19: MyTestApp.<anon-1> 2 0.116825103759765625ms
srcDemo/demo/MyTestApp.hx:82: MyTestApp.sortColsFunc 1 0.0898838043212890625ms
srcDemo/demo/MyTestApp.hx:57: MyTestApp.switchVal 2 0.0722408294677734375ms
srcDemo/demo/MyTestApp.hx:111: MyTestApp.noBody2 3 0.051021575927734375ms
srcDemo/demo/MyTestApp.hx:122: MyTestApp.getInt 2 0.0469684600830078125ms
srcDemo/demo/MyTestApp.hx:99: MyTestApp.noBody 1 0.04482269287109375ms
srcDemo/demo/MyTestApp.hx:114: MyTestApp.noBody5 3 0.03910064697265625ms
srcDemo/demo/MyTestApp.hx:130: MyTestApp.noCover 1 0.0278949737548828125ms
srcDemo/demo/MyTestApp.hx:137: MyTestApp.whileLoop 1 0.0209808349609375ms
srcDemo/demo/MyTestApp.hx:43: MyTestApp.<anon-2> 1 0.0159740447998046875ms
srcDemo/demo/MyTestApp.hx:83: MyTestApp.<anon-3> 1 0.0140666961669921875ms
srcDemo/demo/MyTestApp.hx:4: MyTestApp.<anon-arrow-0> 1 0.011920928955078125ms
-D profiler-console-missing-reporter - prints a list of all calls that didn't exit properly. (might indicate an issue in instrument library)
-D profiler-console-hierarchy-reporter - prints a hierachical summary of calls showing which function was called from what function, again listing all locations, fields, number opf invocations and duration
====================
== Call Hierarchy ==
====================
+ <root> 9.17577743530273438ms
---+ thread-0 9.17577743530273438ms
------+ srcDemo/demo/MyTestApp.hx:141: MyTestApp.main 1 4.44388389587402344ms
---------+ srcDemo/demo/MyTestApp.hx:6: MyTestApp.new 1 4.13799285888671875ms
------------+ srcDemo/demo/MyTestApp.hx:97: MyTestApp.noBody 1 0.030994415283203125ms
------------+ srcDemo/demo/MyTestApp.hx:109: MyTestApp.noBody2 1 0.0269412994384765625ms
------------+ srcDemo/demo/MyTestApp.hx:100: MyTestApp.noBody3 2 0.159025192260742188ms
---------------+ srcDemo/demo/MyTestApp.hx:146: MyTestApp_Fields_.moduleBody 2 0.0522136688232421875ms
------------+ srcDemo/demo/MyTestApp.hx:112: MyTestApp.noBody5 1 0.02193450927734375ms
------------+ srcDemo/demo/MyTestApp.hx:146: MyTestApp_Fields_.moduleBody 5 1.22761726379394531ms
------------+ srcDemo/demo/MyTestApp.hx:4: MyTestApp.<anon-arrow-0> 1 0.02288818359375ms
------------+ srcDemo/demo/MyTestApp.hx:115: MyTestApp.noBody4 1 0.294923782348632812ms
---------------+ srcDemo/demo/MyTestApp.hx:19: MyTestApp.<anon-1> 2 0.210046768188476562ms
------------------+ srcDemo/demo/MyTestApp.hx:112: MyTestApp.noBody5 2 0.03719329833984375ms
------------------+ srcDemo/demo/MyTestApp.hx:109: MyTestApp.noBody2 2 0.0393390655517578125ms
------------+ srcDemo/demo/MyTestApp.hx:87: MyTestApp.initTranslations 1 0.142812728881835938ms
------------+ srcDemo/demo/MyTestApp.hx:128: MyTestApp.noCover 1 0.031948089599609375ms
------------+ srcDemo/demo/MyTestApp.hx:120: MyTestApp.getInt 2 0.0612735748291015625ms
------------+ srcDemo/demo/MyTestApp.hx:135: MyTestApp.whileLoop 1 0.0259876251220703125ms
------------+ srcDemo/demo/MyTestApp.hx:41: MyTestApp.<anon-2> 1 0.0240802764892578125ms
------------+ srcDemo/demo/MyTestApp.hx:80: MyTestApp.sortColsFunc 1 0.072956085205078125ms
---------------+ srcDemo/demo/MyTestApp.hx:81: MyTestApp.<anon-3> 1 0.019073486328125ms
------------+ srcDemo/demo/MyTestApp.hx:55: MyTestApp.switchVal 2 0.118017196655273438ms
------------+ srcDemo/demo/MyTestApp.hx:69: MyTestApp.opBoolOr 3 0.70667266845703125ms
writes profiling data to a CSV file using thread;invocations;total time in ms;class;function;location columns. filename defaults to summary.csv in your workspace root. you can set a different name and folder. csv reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
-D profiler-csv-reporter
-D profiler-csv-reporter=summary.xmlwrites a json file compatible with d3-flame-graph javascript library.f ilename defaults to flame.json in your workspace root. you can set a different name and folder. d3 reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
-D profiler-d3-reporter
-D profiler-d3-reporter=flame.jsonwork in progress - supposed to write a file in .cpuprofile format to be used in e.g. vscode-js-profile-flame VSCode extension.
-D profiler-cpuprofile-reporter
-D profiler-cpuprofile-reporter=profiler.cpuprofileinstrument tries to find your main function and all calls to Sys.exit to set up automatic detection of program exit. however if you exit your program by any other means or if your main function is excluded from instrumentation, then you might have to add a call to instrument's endProfiler or endCoverage functions:
#if instrument
instrument.coverage.Coverage.endCoverage(); // when measuring coverage
instrument.profiler.Profiler.endProfiler(); // when profiling
#endyou can invoke end instrumentation functions multiple times, results should include all data since start of application up to your end instrumentation call. file based reporters will overwrite their files with latest results.
you can exclude types / fields from instrumentation by adding metadata:
@ignoreInstrumentor@:ignoreInstrumentwill ignore a type / field from all instrumentation@ignoreCoverageor@:ignoreCoveragewill ignore a type / field from coverage instrumentation@ignoreProfileror@:ignoreProfilerwill ignore a type / field from profiler instrumentation
you can use -D instrument-quiet to suppress instrumentation output during compilation
adding -D debug-instrumentation will make instrument print out each field's expression before and after instrumentation. so you can see what code instrument library adds to each of your fields.