Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download and use Javy plugin when building JS functions #4831

Merged
merged 9 commits into from
Nov 22, 2024

Conversation

jeffcharles
Copy link
Contributor

@jeffcharles jeffcharles commented Nov 11, 2024

WHY are these changes introduced?

Javy is being updated to require a Javy plugin, which is a WebAssembly module exposing a specific API, when building dynamically linked modules. Not specifying a plugin when using javy build -C dynamic will cause Javy to exit with an error saying a plugin is required. So, the Shopify CLI, will need to be updated to download and use a Javy plugin. This change will use the default Javy plugin.

This change WILL NOT WORK until a new version of Javy is released that supports the -C plugin=... argument for dynamically linked modules. I would like to get an advance review of this change to ensure there isn't a long wait for when the Shopify CLI will be able to use future versions of Javy before I publish a new release of Javy that will include this breaking change.

WHAT is this pull request doing?

  1. Updates the CLI to download the default Javy plugin. This includes making DownloadableBinary an interface and creating two implementation classes. One is the existing DownloadableBinary renamed to Executable and the other JavyPlugin. This is because the logic for determining the path and download URL are different. Also renamed installBinary to downloadBinary.
  2. Updates the call to javy build to use the plugin that was downloaded by specifying an additional argument of -C plugin=<path_to_plugin>.

How to test your changes?

pnpm clean followed by pnpm shopify app function build --path <path_to_js_function_extension>. It'll complain about Javy returning an exit code of 1 and the underlying message from Javy will be that plugins are not currently supported for dynamically linked modules.

Measuring impact

How do we know this change was effective? Please choose one:

  • n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix
  • Existing analytics will cater for this addition
  • PR includes analytics changes to measure impact

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes

Copy link
Contributor

github-actions bot commented Nov 11, 2024

Coverage report

St.
Category Percentage Covered / Total
🟡 Statements
74.56% (+0.05% 🔼)
8554/11472
🟡 Branches
70.36% (+0.07% 🔼)
4173/5931
🟡 Functions
74.1% (+0.04% 🔼)
2246/3031
🟡 Lines
75.11% (+0.05% 🔼)
8090/10771

Test suite run success

1949 tests passing in 886 suites.

Report generated by 🧪jest coverage report action from be45523

@andrewhassan andrewhassan requested review from a team, jacobsteves, saga-dasgupta and lopert and removed request for a team and saga-dasgupta November 11, 2024 21:16
Copy link
Contributor

@lopert lopert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do I get to the expected underlying error from javy?
When tophatting, I get error exit code 2 🤔

 ~/src/github.com/Shopify/cli  jc.javy-plugin *22 ?1  pnpm shopify app function build --path /Users/lopert/code/superset/superset-alex-local/extensions/unstable-ccv-vars-1                    1 ✘  22.2.0 

> @0.0.0 shopify /Users/lopert/src/github.com/Shopify/cli
> nx build cli && node packages/cli/bin/dev.js "app" "function" "build" "--path" "/Users/lopert/code/superset/superset-alex-local/extensions/unstable-ccv-vars-1"

...more cli output

Building GraphQL types ...

── external error ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Error coming from `/Users/lopert/src/github.com/Shopify/cli/packages/app/dist/cli/services/bin/javy build -C dynamic -C
plugin=/Users/lopert/src/github.com/Shopify/cli/packages/app/dist/cli/services/bin/javy_quickjs_provider_v3.wasm -C
wit=/private/var/folders/57/c41xs80j07vdldqk6k1ntxxm0000gn/T/d6830e15470cd535466f1ccfc4c02bb7/javy-world.wit -C wit-world=shopify-function -o
/Users/lopert/code/superset/superset-alex-local/extensions/unstable-ccv-vars-1/dist/function.wasm dist/function.js`

Command failed with exit code 2: /Users/lopert/src/github.com/Shopify/cli/packages/app/dist/cli/services/bin/javy build -C dynamic -C
plugin=/Users/lopert/src/github.com/Shopify/cli/packages/app/dist/cli/services/bin/javy_quickjs_provider_v3.wasm -C
wit=/private/var/folders/57/c41xs80j07vdldqk6k1ntxxm0000gn/T/d6830e15470cd535466f1ccfc4c02bb7/javy-world.wit -C wit-world=shopify-function -o
/Users/lopert/code/superset/superset-alex-local/extensions/unstable-ccv-vars-1/dist/function.wasm dist/function.js

describe('javy-plugin', () => {
test('properties are set correctly', () => {
expect(javyPlugin.name).toBe('javy_quickjs_provider_v3')
expect(javyPlugin.version).match(/^v\d.\d.\d$/)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should part of the version identifier here match the expected version? Or rather, some kind of "minimum version"

Copy link
Contributor Author

@jeffcharles jeffcharles Nov 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There isn't really a concept of a minimum version for this. The version should generally be whatever the latest release of Javy is until we switch to the Shopify Functions plugin. But it shouldn't be set to the latest version if there are breaking changes in the latest version until the Shopify CLI code has been updated to not break when using that version. So essentially, the version should be whatever is set in the binaries.ts file.

I opted to go with a check on the format of the version rather than the version itself since the format of a version should be constant over time whereas the version should change frequently. If you have to update the test every time the version changes, it's arguably more likely the test won't catch an incorrect change since it'll just be updated to whatever is in binaries.ts.

@jeffcharles
Copy link
Contributor Author

@lopert can you comment out the stderr: 'inherit' on line 192 in build.ts in the exec command under runJavy and try running it again and see what gets emitted?

Also I'm not sure why the contents of the stderr stream are not being displayed when it's set to inherit or why it was set to inherit. But that's separate from this PR.

@isaacroldan
Copy link
Contributor

/snapit

Copy link
Contributor

🫰✨ Thanks @isaacroldan! Your snapshot has been published to npm.

Test the snapshot by intalling your package globally:

pnpm i -g @shopify/cli@0.0.0-snapshot-20241115163229

After installing, validate the version by running just shopify in your terminal
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

@jeffcharles
Copy link
Contributor Author

@lopert it looks like the cause was I had a more recent version of Javy cached in my local dev env for the CLI package so exit code and error message emitted to stderr was different. I'll have to rebase this PR on top of the changes in main anyway so that'll cause the slightly newer Javy to be used if you run pnpm clean or manually clean the binary cache.

@jeffcharles
Copy link
Contributor Author

@lopert the rebased PR has an exit code of 1. You will want to run pnpm clean before tophatting (I've updated the tophatting instructions).

@jeffcharles
Copy link
Contributor Author

/snapit

Copy link
Contributor

🫰✨ Thanks @jeffcharles! Your snapshot has been published to npm.

Test the snapshot by intalling your package globally:

pnpm i -g @shopify/cli@0.0.0-snapshot-20241115171206

After installing, validate the version by running just shopify in your terminal
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

Copy link
Contributor

@lopert lopert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got the right exit code 1 this 🎩

@jeffcharles jeffcharles marked this pull request as ready for review November 20, 2024 16:03
@jeffcharles jeffcharles requested review from a team as code owners November 20, 2024 16:03
Copy link
Contributor

@lopert lopert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎩 'd with the latest again and build successfully

Copy link
Member

@jacobsteves jacobsteves left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

Copy link
Contributor

@shauns shauns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving from app-inner-loop. I left some small naming/comment nits. I think the prior reviewers were best placed to review the purpose and the main thrust behind this change as its so specific -- I've looked at this only really from CLI stewarding.

packages/app/src/cli/services/function/binaries.ts Outdated Show resolved Hide resolved
@jeffcharles
Copy link
Contributor Author

jeffcharles commented Nov 22, 2024

Latest errors seem to be:

cli-kit: ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
cli-kit:  FAIL  src/public/node/base-command.test.ts > applying environments > does not apply a environment when none is specified
cli-kit:  Test Files  1 failed | 98 passed (99)
cli-kit: AssertionError: expected '╭─ info ─────────────────────────────…' to deeply equal ''
cli-kit:       Tests  1 failed | 777 passed (778)
cli-kit: - Expected
cli-kit:    Start at  15:03:08
cli-kit: + Received
cli-kit:    Duration  65.13s (transform 1.91s, setup 290ms, collect 82.05s, tests 21.86s, environment 18ms, prepare 10.34s)
cli-kit: + ╭─ info ───────────────────────────────────────────────────────────────────────╮
cli-kit: + │                                                                              │
cli-kit: + │  Release notes for 3.70.0                                                    │
cli-kit: + │                                                                              │
cli-kit: + │  Release highlights:                                                         │
cli-kit: + │                                                                              │
cli-kit: + │    - Ruby is no longer required                                              │
cli-kit: + │    - The `theme dev` command is more stable                                  │

cli-kit: + │    - A new notification system (expect more of these!)                       │
> nx run cli:test
cli-kit: + │    - Numerous bug fixes                                                      │

cli-kit: + │                                                                              │
cli-kit: + │                                                                              │
cli-kit: + │                                                                              │
cli-kit: + │  Read the complete release notes [1]                                         │
cli-kit: + │                                                                              │
cli-kit: + ╰──────────────────────────────────────────────────────────────────────────────╯
cli-kit: + [1] https://github.com/Shopify/cli/releases/tag/3.70.0
cli-kit: +
cli-kit:  ❯ src/public/node/base-command.test.ts:158:31
cli-kit:     156|       someStringWithDefault: 'default stringy',
cli-kit:     157|     })
cli-kit:     158|     expect(outputMock.info()).toEqual('')
cli-kit:        |                               ^
cli-kit:     159|   })
cli-kit:     160| 
cli-kit:  ❯ src/public/node/base-command.test.ts:130:9
cli-kit:  ❯ runTask ../../node_modules/.pnpm/tempy@3.0.0/node_modules/tempy/index.js:18:10
cli-kit:  ❯ src/public/node/base-command.test.ts:128:7
cli-kit: ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

which does not seem to be related to my change

Copy link
Contributor

@shauns shauns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending it turning green

@jeffcharles jeffcharles added this pull request to the merge queue Nov 22, 2024
Merged via the queue into main with commit d6b1598 Nov 22, 2024
27 checks passed
@jeffcharles jeffcharles deleted the jc.javy-plugin branch November 22, 2024 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants