[PATCH v4] test: ml_perf: add ML performance test #2138
Conversation
odpbuild
changed the title
| } | ||
| } | ||
|
|
||
| optind = 1; /* reset 'extern optind' from the getopt lib */ |
There was a problem hiding this comment.
Some ODP testers has this and some don't, does this have any actual effect?
There was a problem hiding this comment.
Only if getopt*() is called again, I guess. So at least in this case, no effect. Left it there anyway. Maybe we should clean these up at some point.
| }; | ||
|
|
||
| typedef struct { | ||
| int mode, num_threads, rounds, latency, warmup; |
There was a problem hiding this comment.
These as well as num_batch could be unsigned integers as them having a negative value does not make much sense.
There was a problem hiding this comment.
It doesn't do any harm, either. The negative range is not used now, but it could be used at some point for some special values, etc. I didn't change these now.
There was a problem hiding this comment.
Currently, user can pass e.g. negative round and warmup values and tester happily continues, giving weird results. There should at least be some sanity checks after parsing the command line.
There was a problem hiding this comment.
As a typical user of this tool, I don't need such checks. As a developer, I think if we want to do user input sanitation (and I don't), then we should probably implement it in a library and do it consistently in all of our testing tools, etc.
| if (out_size_final != glb->ref_file_size) { | ||
| ODPH_ERR("Output size mismatch: %" PRIu64 | ||
| " differs from reference file size %" PRIu64 "\n", | ||
| out_size_final, glb->ref_file_size); | ||
| ret = -1; | ||
| goto error; | ||
| } | ||
|
|
||
| if (memcmp(glb->ref_file_data, output_final, out_size_final)) { | ||
| ODPH_ERR("Output differs from reference\n"); | ||
| ret = -1; | ||
| goto error; | ||
| } |
There was a problem hiding this comment.
I don't think there's need to fail the thread due to these, this could simply be made visible when printing results at the end.
There was a problem hiding this comment.
At least I would definitely want the program to exit with an error if the output doesn't match the reference because in that case something went horribly wrong. Also in CI we want an error exit in that case.
| mdl = odp_ml_model_create(glb->opt.model_name, &model_param); | ||
| if (mdl == ODP_ML_MODEL_INVALID) { | ||
| ODPH_ERR("odp_ml_model_create() failed\n"); | ||
| ret = -1; | ||
| goto error; | ||
| } | ||
|
|
||
| if (odp_ml_model_destroy(mdl)) { | ||
| ODPH_ERR("odp_ml_model_destroy() failed\n"); | ||
| ret = -1; | ||
| goto error; | ||
| } |
There was a problem hiding this comment.
For stuff that's being tested, it's probably better to record their failures as some error counter increases instead of failing the thread. Same thing for loading and unloading in test_ml_load().
There was a problem hiding this comment.
Again, better to just quit. We're measuring speed, not e.g. the probability of success.
There was a problem hiding this comment.
Well, in a performance testing setting, speed and probability of success might be related. Anyways, up to you.
JereLeppanen
force-pushed
the
dev/test-ml-perf
branch
from
46968bd
to
411a1b5
Compare
odpbuild
changed the title
|
v2: All comments processed. |
JereLeppanen
force-pushed
the
dev/test-ml-perf
branch
from
411a1b5
to
d6c372c
Compare
odpbuild
changed the title
JereLeppanen
force-pushed
the
dev/test-ml-perf
branch
from
d6c372c
to
7737811
Compare
odpbuild
changed the title
Add machine learning (ML) performance test application. The application measures nanoseconds used for various ML operations, such as creating and desroying a model, loading and unloading a model, and inferencing. Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Add ML performance test run for linux-gen. This runs some basic performance tests using the conv.onnx model. Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
No description provided.