Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a more useful profile guided optimization test script #1299

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

svaarala
Copy link
Owner

@svaarala svaarala commented Jan 16, 2017

The test script should exercise usual code paths for all built-ins so that the performance profiling data accurately reflects the relative frequency of most practical code paths. Current PGO test set in the Makefile is a dummy placeholder.

  • Come up with a test set as a separate test file or as sequence of individual tests
  • The full tests/ecmascript test run could be useful (execute with runtest.py), but might be too heavy for practical Makefiles
  • Add Octane to profiling set
  • Document that duktape.c can be profile optimized with a dummy main (e.g. "duk") and the profile then used in an application build
  • Consider bundling the PGO test file into the distributable along with a PGO example

@svaarala
Copy link
Owner Author

svaarala commented Jan 16, 2017

Here's some results with the current quite dummy profile test set:

  • duk.O2.master is from make duk.O2from master before hot/cold attributes
  • duk.O2.hotcold is master + hot/cold attributes merge
  • duk.O2.pgo is from make duk-pgo.O2 which is the same as make duk.O2 but with PGO
  • duk.O2.perfpgo is make duk-perf.O2 with PGO

Most operations are faster in the PGO version, for example test-fib.js improves from 6.4 seconds to 3.8 seconds, test-call-basic-1.js improves from 8.0 seconds to 5.8 seconds. The -O2 PGO binary is also 24kB smaller.

But tests for functionality not covered by the profile set can be slower, for example TextEncoder/TextDecoder and Unicode handling is a bit slower. Those would be fixed by improving the profile test set. Ideally the distributable would provide a single JS file to execute for getting a reasonable PGO baseline.

duk.O2.perfpgoduk.O2.pgoduk.O2.hotcoldduk.O2.master
test-add-fastint.js0.430.470.660.67
test-add-float.js0.420.470.680.63
test-add-nan-fastint.js0.600.650.780.74
test-add-nan.js0.410.510.660.67
test-arith-add.js1.711.922.582.73
test-arith-div.js6.366.677.317.33
test-arith-mod.js6.356.556.696.71
test-arith-mul.js2.993.143.983.62
test-arith-sub.js1.782.042.782.69
test-array-append.js0.490.490.540.52
test-array-cons-list.js0.730.770.880.83
test-array-foreach.js2.152.232.672.73
test-array-literal.js1.531.541.631.59
test-array-pop.js1.601.702.242.21
test-array-push.js1.821.932.442.39
test-array-read-lenloop.js1.441.521.912.06
test-array-read.js1.641.801.801.78
test-array-sort.js3.433.513.283.42
test-array-write-length.js1.181.401.661.61
test-array-write.js1.562.511.781.73
test-assign-add.js3.253.613.703.70
test-assign-addto-nan.js0.971.021.351.23
test-assign-addto.js3.293.633.713.74
test-assign-boolean.js4.714.714.704.71
test-assign-const-int.js2.112.392.492.44
test-assign-const-int2.js4.734.945.844.65
test-assign-const.js3.653.984.013.85
test-assign-literal.js3.303.763.753.78
test-assign-proplhs-reg.js2.292.472.552.73
test-assign-proprhs.js1.881.902.682.81
test-assign-reg.js2.422.682.812.80
test-base64-decode-whitespace.js1.881.891.871.86
test-base64-decode.js1.631.621.511.52
test-base64-encode.js1.921.891.711.70
test-bitwise-ops.js1.461.561.831.95
test-break-fast.js1.151.321.401.43
test-break-slow.js6.697.257.757.75
test-buffer-nodejs-read.js2.242.352.362.34
test-buffer-nodejs-write.js3.643.733.093.06
test-buffer-object-read.js2.232.362.372.32
test-buffer-object-write.js3.623.733.103.07
test-buffer-plain-read.js1.791.851.931.91
test-buffer-plain-write.js1.701.751.881.84
test-call-basic-1.js5.275.788.087.99
test-call-basic-2.js5.265.788.027.99
test-call-basic-3.js6.957.7610.0110.07
test-call-basic-4.js14.1115.4218.2018.43
test-call-native.js10.7411.7113.9713.47
test-call-prop.js2.732.904.354.32
test-call-reg-new.js5.265.516.476.51
test-call-reg.js2.002.163.042.98
test-call-var.js4.104.387.227.17
test-compile-mandel-nofrac.js10.1410.4110.3910.49
test-compile-mandel.js13.1113.5513.5213.48
test-compile-short.js5.856.166.526.59
test-compile-string-ascii.js6.506.496.686.70
test-continue-fast.js1.521.831.801.78
test-continue-slow.js7.197.337.807.85
test-empty-loop-slowpath.js1.121.201.691.66
test-empty-loop.js1.141.341.721.61
test-enum-basic.js3.093.313.493.54
test-equals-fastint.js0.490.510.580.52
test-equals-nonfastint.js0.600.670.610.61
test-error-create.js1.481.521.972.01
test-fib.js3.743.846.526.41
test-global-lookup.js4.284.407.377.81
test-hello-world.js0.000.000.000.00
test-hex-decode.js3.603.613.603.61
test-hex-encode.js2.953.012.822.84
test-jc-serialize-indented.js6.816.793.303.29
test-jc-serialize.js2.272.272.182.16
test-json-parse-hex.js4.094.103.183.22
test-json-parse-integer.js2.772.702.892.78
test-json-parse-number.js5.125.215.045.04
test-json-parse-string.js5.715.695.595.36
test-json-serialize-fastpath-loop.js6.666.733.503.46
test-json-serialize-forceslow.js9.699.889.099.17
test-json-serialize-hex.js1.531.531.491.45
test-json-serialize-indented-deep100.js1.921.951.691.69
test-json-serialize-indented-deep25.js7.107.023.163.21
test-json-serialize-indented-deep500.js1.191.201.071.09
test-json-serialize-indented.js12.1512.115.555.55
test-json-serialize-jsonrpc-message.js2.012.042.001.95
test-json-serialize-nofrac.js0.600.640.590.58
test-json-serialize-plainbuf.js2.452.462.462.50
test-json-serialize-slowpath-loop.js4.314.464.194.16
test-json-serialize.js6.866.926.066.07
test-json-string-bench.js2.512.513.313.35
test-json-string-stringify.js5.315.325.065.10
test-jx-serialize-bufobj-forceslow.js4.004.084.394.35
test-jx-serialize-bufobj.js1.541.581.581.59
test-jx-serialize-indented.js6.846.833.283.32
test-jx-serialize.js2.272.352.142.16
test-mandel.js2.022.242.962.84
test-object-garbage-2.js4.634.654.934.94
test-object-garbage.js4.024.094.394.66
test-object-literal.js2.932.943.133.11
test-prop-read-1024.js2.232.303.063.17
test-prop-read-16.js2.222.333.063.14
test-prop-read-256.js2.252.313.083.13
test-prop-read-32.js2.292.363.013.15
test-prop-read-4.js2.202.243.213.34
test-prop-read-48.js2.232.313.043.14
test-prop-read-64.js2.262.303.053.18
test-prop-read-8.js2.242.303.113.19
test-prop-read.js2.232.333.283.42
test-prop-write-1024.js2.583.183.002.95
test-prop-write-16.js2.642.962.912.95
test-prop-write-256.js2.612.902.952.92
test-prop-write-32.js2.622.952.922.99
test-prop-write-4.js2.582.852.983.07
test-prop-write-48.js2.602.802.972.89
test-prop-write-64.js2.592.882.982.89
test-prop-write-8.js2.592.912.952.94
test-prop-write.js2.583.002.983.02
test-random.js1.451.632.312.26
test-reg-readwrite-object.js3.053.213.113.16
test-reg-readwrite-plain.js1.742.022.022.04
test-regexp-case-insensitive.js0.0029.5624.8625.69
test-regexp-compile.js2.112.112.102.10
test-regexp-execute.js1.371.401.451.44
test-regexp-string-parse.js10.7710.635.875.87
test-strict-equals-fastint.js0.470.510.620.54
test-strict-equals-nonfastint.js0.550.570.630.60
test-string-array-concat.js4.904.995.545.43
test-string-arridx.js1.481.621.671.69
test-string-charlen-ascii.js1.111.131.101.14
test-string-charlen-nonascii.js2.502.532.622.66
test-string-compare.js1.851.932.082.00
test-string-decodeuri.js4.013.893.743.50
test-string-encodeuri.js4.374.394.014.12
test-string-garbage.js3.874.094.064.08
test-string-intern-grow-short.js3.343.453.903.88
test-string-intern-grow-short2.js3.293.373.813.79
test-string-intern-grow.js4.584.624.694.73
test-string-intern-grow2.js1.291.341.411.46
test-string-intern-match-short.js1.671.681.972.00
test-string-intern-match.js0.190.180.220.22
test-string-intern-miss-short.js1.891.972.262.27
test-string-intern-miss.js1.051.021.141.20
test-string-number-list.js0.610.650.810.79
test-string-plain-concat.js0.860.850.880.84
test-string-uppercase.js2.763.622.192.22
test-textdecoder-ascii.js2.812.522.312.48
test-textdecoder-nonascii.js3.443.403.033.01
test-textencoder-ascii.js5.004.484.294.34
test-textencoder-nonascii.js13.7713.329.9910.77
test-try-catch-nothrow.js2.352.462.542.56
test-try-catch-throw.js34.7934.7638.3038.09
test-try-finally-nothrow.js2.943.213.263.29
test-try-finally-throw.js43.8946.3448.0247.20

@svaarala svaarala modified the milestones: v2.2.0, v2.1.0 Mar 29, 2017
@svaarala svaarala force-pushed the profile-optimization-training-set branch from 22245b2 to 9f62e76 Compare April 8, 2017 12:13
@svaarala svaarala force-pushed the profile-optimization-training-set branch from 9f62e76 to 0e6a808 Compare July 23, 2017 01:55
@svaarala
Copy link
Owner Author

Octane is now included by the Makefile.

@svaarala svaarala removed this from the v2.2.0 milestone Aug 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant