Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MBT upgrade build parameter ignore attribute #1050

Open
wants to merge 41 commits into
base: master
Choose a base branch
from

Conversation

young-yang03
Copy link
Contributor

@young-yang03 young-yang03 commented May 18, 2023

Description

MBT upgrade build parameter ignore attribute to support full Glob Pattern

https://wiki.one.int.sap/wiki/pages/viewpage.action?spaceKey=LCNCBJ&title=MBT+Support+Package+Specified+Content+into+MTA+archive

Checklist

  • Code compiles correctly
  • Relevant tests were added (unit / contract / integration)
  • Relevant logs were added
  • Formatting and linting run locally successfully
  • All tests pass
  • UA review
  • Design is documented
  • Extended the README / documentation, if necessary
  • Open source is approved

Young Yang added 2 commits May 15, 2023 17:26
	modified:   internal/archive/mta_location.go
	new file:   micromatch/micromatch-wrapper.js
	new file:   micromatch/package-lock.json
	new file:   micromatch/package.json
	new file:   micromatch/run_test.sh
	new file:   micromatch/test-micromatch.js
	new file:   micromatch/test_micromatch_for_mta.go
	new file:   micromatch/test_micromatch_wrapper_bin.go
	new file:   micromatch/test_micromatch_wrapper_js.go
Young Yang added 26 commits May 18, 2023 16:28
	modified:   internal/archive/fsops.go
	modified:   micromatch/micromatch-wrapper.js
	modified:   micromatch/test_micromatch_wrapper_bin.go
	modified:   micromatch/test_micromatch_wrapper_js.go
	modified:   micromatch/test_micromatch_wrapper_bin.go
	modified:   micromatch/test_micromatch_wrapper_js.go
	modified:   internal/archive/fsops.go
	modified:   internal/archive/fsops_test.go
	modified:   internal/artifacts/module_arch.go
	modified:   internal/artifacts/module_arch_test.go
	modified:   internal/artifacts/mtar.go
	modified:   internal/commands/commands.go
	modified:   internal/platform/model.go
	modified:   micromatch/micromatch-wrapper.js
	modified:   micromatch/run_test.sh
	modified:   micromatch/test-micromatch.js
	deleted:    micromatch/test_micromatch_for_mta.go
	modified:   micromatch/test_micromatch_wrapper_js.go
	modified:   internal/archive/fsops_test.go
	modified:   micromatch/micromatch-wrapper.js
	modified:   micromatch/test-micromatch.js
	modified:   micromatch/test_micromatch_wrapper_js.go
	modified:   micromatch/test-micromatch.js
	modified:   internal/archive/fsops_test.go
	new file:   internal/archive/testdata/testignorewithsymlink/another_content/symlink_to_test4.txt
	new file:   internal/archive/testdata/testignorewithsymlink/another_content/test3.txt
	new file:   internal/archive/testdata/testignorewithsymlink/clean-symlink-env.bat
	new file:   internal/archive/testdata/testignorewithsymlink/content/symlink_dir_to_another_content
	new file:   internal/archive/testdata/testignorewithsymlink/content/test.txt
	new file:   internal/archive/testdata/testignorewithsymlink/content/test_dir/test1.txt
	new file:   internal/archive/testdata/testignorewithsymlink/dir_with_recursive_symlink/subdir/symlink_dir_recursion_to_parent_dir
	new file:   internal/archive/testdata/testignorewithsymlink/dir_with_recursive_symlink/subdir/symlink_dir_to_sibling
	new file:   internal/archive/testdata/testignorewithsymlink/dir_with_recursive_symlink/subdir2/symlink_dir_to_sibling
	new file:   internal/archive/testdata/testignorewithsymlink/moduleNew/package.json
	new file:   internal/archive/testdata/testignorewithsymlink/moduleNew/symlink_dir_to_content
	new file:   internal/archive/testdata/testignorewithsymlink/prepare-symlink-env.bat
	new file:   internal/archive/testdata/testignorewithsymlink/symlink_dir_to_moduleNew
	new file:   internal/archive/testdata/testignorewithsymlink/symlink_dir_to_symlink_dir_broken
	new file:   internal/archive/testdata/testignorewithsymlink/symlink_to_symlink_broken
	new file:   internal/archive/testdata/testignorewithsymlink/test4.txt
	modified:   internal/artifacts/module_arch.go
	modified:   internal/artifacts/mtar.go
	modified:   internal/archive/fsops.go
	modified:   internal/archive/fsops_test.go
	new file:   internal/archive/testdata/testproject/ui5app/webapp/view/.invisibleView
	new file:   cmd/testdata/mtaignore/mta-ignore-node_modules-subfolders.yaml
	new file:   cmd/testdata/mtaignore/node-js/gulpfile.js
	new file:   cmd/testdata/mtaignore/node-js/package-lock.json
	new file:   cmd/testdata/mtaignore/node-js/package.json
	new file:   cmd/testdata/mtaignore/node-js/server.js
	modified:   cmd/build_parameter_test.go
	modified:   cmd/cmd_test.go
	modified:   cmd/init.go
	modified:   internal/archive/fsops.go
	deleted:    internal/archive/testdata/testignorewithsymlink/another_content/symlink_to_test4.txt
	deleted:    internal/archive/testdata/testignorewithsymlink/content/symlink_dir_to_another_content
	deleted:    internal/archive/testdata/testignorewithsymlink/dir_with_recursive_symlink/subdir/symlink_dir_recursion_to_parent_dir
	deleted:    internal/archive/testdata/testignorewithsymlink/dir_with_recursive_symlink/subdir/symlink_dir_to_sibling
	deleted:    internal/archive/testdata/testignorewithsymlink/dir_with_recursive_symlink/subdir2/symlink_dir_to_sibling
	deleted:    internal/archive/testdata/testignorewithsymlink/moduleNew/symlink_dir_to_content
	deleted:    internal/archive/testdata/testignorewithsymlink/symlink_dir_to_moduleNew
	deleted:    internal/archive/testdata/testignorewithsymlink/symlink_dir_to_symlink_dir_broken
	deleted:    internal/archive/testdata/testignorewithsymlink/symlink_to_symlink_broken
	modified:   package-lock.json
	modified:   internal/archive/fsops.go
	deleted:    internal/artifacts/testdata/mtahtml5/.mtahtml5_mta_build_tmp/xs-security.json
	new file:   micromatch/build-install-test.sh
	new file:   micromatch/build-install.sh
	modified:   micromatch/micromatch-wrapper.js
	new file:   micromatch/node-js/gulpfile.js
	new file:   micromatch/node-js/package.json
	new file:   micromatch/node-js/server.js
	deleted:    micromatch/run_test.sh
	new file:   micromatch/test.sh
	modified:   micromatch/test_micromatch_wrapper_js.go
	modified:   .goreleaser.yml
	modified:   Dockerfile_mbtci_template
	modified:   cmd/init_test.go
	modified:   cmd/testdata/mta/mta.yaml
	modified:   integration/cloud_mta_build_tool_test.go
	modified:   internal/archive/fsops_test.go
	new file:   micromatch/Darwin/micromatch-wrapper
	new file:   micromatch/Linux/micromatch-wrapper
	new file:   micromatch/Windows/micromatch-wrapper.exe
	modified:   micromatch/build-install.sh
	modified:   package.json
	modified:   index.js
	modified:   micromatch/Darwin/micromatch-wrapper
	modified:   micromatch/Linux/micromatch-wrapper
	modified:   micromatch/Windows/micromatch-wrapper.exe
	modified:   package-lock.json
	modified:   cmd/build_parameter_test.go
	modified:   cmd/init_test.go
	modified:   micromatch/Darwin/micromatch-wrapper
	modified:   micromatch/Linux/micromatch-wrapper
	modified:   micromatch/Windows/micromatch-wrapper.exe
	modified:   cmd/init_test.go
	modified:   cmd/build_parameter_test.go
	modified:   cmd/init_test.go
	modified:   Dockerfile_mbtci_template
	modified:   cmd/init_test.go
	modified:   Makefile
	modified:   test/goss/goss_template.yaml
	modified:   configs/builder_type_cfg.yaml

- [Download](https://github.com/SAP/cloud-mta-build-tool/releases) the latest source code.

- Install [pkg](https://github.com/vercel/pkg/) on your operating system
Copy link

@ShiraManas ShiraManas Jul 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Install pkg on your operating system:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

npm install -g pkg
```

- Build micromatch wrapper under `micromatch` subfolder of source code

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build the micromatch wrapper under the micromatch subfolder of the source code:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

```
- Add the binary file to your `~/bin` path according to your operating system:

* In Darwin, copy binary file `micromatch-wrapper` to the `~/usr/local/bin/` folder, for example: `cp micromatch-wrapper-macos /usr/local/bin/micromatch-wrapper`
Copy link

@ShiraManas ShiraManas Jul 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Darwin, copy the micromatch-wrapper binary file to the ~/usr/local/bin/ folder.
Here is a sample command: cp micromatch-wrapper-macos /usr/local/bin/micromatch-wrapper

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK


* In Darwin, copy binary file `micromatch-wrapper` to the `~/usr/local/bin/` folder, for example: `cp micromatch-wrapper-macos /usr/local/bin/micromatch-wrapper`

* In Linux, copy binary file `micromatch-wrapper` to the `~/usr/local/bin/` folder, for example: `cp micromatch-wrapper-linux /usr/local/bin/micromatch-wrapper`
Copy link

@ShiraManas ShiraManas Jul 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Linux, copy the micromatch-wrapper binary file to the ~/usr/local/bin/ folder.
Here is a sample command: cp micromatch-wrapper-linux /usr/local/bin/micromatch-wrapper

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK


**Install using npm**

Run the command below, it will install mbt and micromatch-wrapper together
Copy link

@ShiraManas ShiraManas Jul 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run the command below to install MBT and the micromatch wrapper together:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

@@ -41,3 +41,24 @@ If you want to use the old `mvn -B package` command instead of the `maven` build
```

The same approach can be implemented if the `maven` builder is used in the global build step.

## v1.2.25
The build-parameters ignore attribute will be upgraded to support [Full Glob Pattern](https://en.wikipedia.org/wiki/Glob_(programming)).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 'build-parameters ignore' attribute will be upgraded to support glob patterns.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

## v1.2.25
The build-parameters ignore attribute will be upgraded to support [Full Glob Pattern](https://en.wikipedia.org/wiki/Glob_(programming)).

Wildcards, such as `*`,`!`,`?` can be used to match characters. You can package specified content by using negation pattern `!`
Copy link

@ShiraManas ShiraManas Jul 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wildcards, such as *,!,? can be used to match characters. You can package specific content by using the ! negation pattern.
(Is there a mistake? How can it be that ! is used to match and to negate?
Also, can we say "select specific content" in the second sentence above instead of "package specified content"? Is this correct?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By using negation pattern ! in the build-parameters ignore attribute, because not ignore equal to include, you can select specified content into the package.

build-result: myfolder
ignore: ["node_modules/**", "!node_modules/mtainclude"]

# In this example, all files and subfolders of node_modules will not be packaged in to MTA archive

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this example, all files and subfolders of 'node_modules' will not be packaged into the MTA archive, except for the 'mtainclude' subfolder of 'node_modules'.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

Copy link

@ShiraManas ShiraManas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi. I've added some suggestions here.

Young Yang added 2 commits July 5, 2023 21:42
	modified:   internal/artifacts/module_arch.go
	modified:   internal/artifacts/mtar.go
	modified:   docs/docs/download.md
	modified:   docs/docs/micromatch-wrapper.md
	modified:   docs/docs/whatsnew.md
@young-yang03
Copy link
Contributor Author

@ShiraManas I recommit the documents, pls review it, thanks

@@ -204,6 +204,23 @@ For example:
# In this example, the following will not be packaged in to MTA archive:
# all files with the"txt" extension and the "mtaignore" folder within the "myfolder" directory.

```
From MBT 1.2.25 version, the `build-parameters ignore` attribute will be upgraded to support [full glob pattern](https://en.wikipedia.org/wiki/Glob_(programming)). Wildcards, such as `*`,`!`,`?` can be used to match characters.
Copy link

@ShiraManas ShiraManas Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-From the MBT 1.2.25 version,
-"full glob patterns" instead of "full glob pattern"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK Done

@@ -32,3 +32,6 @@ Run the command below.
npm install -g mbt
```

From MBT 1.2.25 version, the `build-parameters ignore` attribute will be upgraded to support [full glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)). You must install the micromatch wrapper to use the glob patterns.

You can install micromatch-wrapp by instruction [Here](https://github.com/SAP/cloud-mta-build-tool/tree/master/docs/docs/micromatch-wrapper.md)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow the instructions to install the micromatch wrapper.

@@ -0,0 +1,52 @@
#### Overview
[Micromatch](https://github.com/micromatch/micromatch) support various matching features, such as glob patterns and advanced globbing

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just add the missing period at the end of the sentence:
Micromatch support various matching features, such as glob patterns and advanced globbing.

#### Overview
[Micromatch](https://github.com/micromatch/micromatch) support various matching features, such as glob patterns and advanced globbing

Micromatch is a Node.js application, while MBT is a Go application. The micromatch wrapper is a package that enables MBT to use the functionalities provided by micromatch. From MBT 1.2.25 version, we create [micromath wrapper](https://github.com/SAP/cloud-mta-build-tool/tree/master/micromatch) to package it.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the MBT 1.2.25 version, we provide the micromatch wrapper to package it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, Done

```
From MBT 1.2.25 version, the `build-parameters ignore` attribute will be upgraded to support [full glob pattern](https://en.wikipedia.org/wiki/Glob_(programming)). Wildcards, such as `*`,`!`,`?` can be used to match characters.

By using negation pattern `!` in the `build-parameters ignore` attribute, because `not ignore` equal to `include`, it means you can select specified content into the package.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By using the ! negation pattern in the build-parameters ignore attribute, you can select specified content into the package (because not ignore equals include).

@@ -32,3 +32,6 @@ Run the command below.
npm install -g mbt
```

From MBT 1.2.25 version, the `build-parameters ignore` attribute will be upgraded to support [full glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)). You must install the micromatch wrapper to use the glob patterns.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the MBT 1.2.25 version,

@@ -41,3 +41,23 @@ If you want to use the old `mvn -B package` command instead of the `maven` build
```

The same approach can be implemented if the `maven` builder is used in the global build step.

## v1.2.25
The `build-parameters ignore` attribute will be upgraded to support [full glob pattern](https://en.wikipedia.org/wiki/Glob_(programming)). Wildcards, such as `*`,`!`,`?` can be used to match characters.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build-parameters ignore attribute will be upgraded to support full glob patterns.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, done

## v1.2.25
The `build-parameters ignore` attribute will be upgraded to support [full glob pattern](https://en.wikipedia.org/wiki/Glob_(programming)). Wildcards, such as `*`,`!`,`?` can be used to match characters.

By using negation pattern `!` in the `build-parameters ignore` attribute, because `not ignore` equal to `include`, it means you can select specified content into the package.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By using the ! negation pattern in the build-parameters ignore attribute, you can select specified content into the package (because not ignore equal to include).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, my recommit just a little different with your suggestion.

To be consistent with the front, equal to -> equals

Copy link

@ShiraManas ShiraManas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks much better! Added my last few comments here.

Young Yang added 5 commits July 13, 2023 10:42
	modified:   cmd/testdata/mta/mta.yaml
	new file:   cmd/testdata/mtaignore/mta-ignore-all-invisible-dir-and-files.yaml
	modified:   cmd/testdata/mtaignore/mta-ignore-all-node_modules.yaml
	modified:   cmd/testdata/mtaignore/mta-ignore-node_modules-subfolders.yaml
	new file:   cmd/testdata/mtaignore/node-js/.invisible-file
	new file:   cmd/testdata/mtaignore/node-js/.invisibledir-1/.invisible-file-1
	new file:   cmd/testdata/mtaignore/node-js/.invisibledir-2/visible-file-2
	modified:   internal/artifacts/module_arch.go
	modified:   internal/artifacts/mtar.go
	modified:   micromatch/test-micromatch.js
…les.yaml

	modified:   cmd/testdata/mtaignore/mta-ignore-all-node_modules.yaml
	modified:   cmd/testdata/mtaignore/mta-ignore-node_modules-subfolders.yaml
	renamed:    cmd/testdata/mtaignore/node-js/.invisibledir-1/.invisible-file-1 -> cmd/testdata/mtaignore/node-js/.invisible-dir/.invisible-file
	renamed:    cmd/testdata/mtaignore/node-js/.invisibledir-2/visible-file-2 -> cmd/testdata/mtaignore/node-js/.invisible-dir/.invisible-sub-dir/.invisible-sub-file
	new file:   cmd/testdata/mtaignore/node-js/.invisible-dir/.invisible-sub-dir/visible-sub-file
	new file:   cmd/testdata/mtaignore/node-js/.invisible-dir/visible-file
	new file:   cmd/testdata/mtaignore/node-js/.invisible-dir/visible-sub-dir/.invisible-sub-file
	new file:   cmd/testdata/mtaignore/node-js/.invisible-dir/visible-sub-dir/visible-sub-file
	modified:   internal/artifacts/module_arch.go
	modified:   internal/artifacts/mtar.go
	modified:   docs/docs/download.md
	modified:   docs/docs/micromatch-wrapper.md
	modified:   docs/docs/whatsnew.md
	modified:   docs/docs/download.md
	modified:   docs/docs/whatsnew.md
@young-yang03
Copy link
Contributor Author

@ShiraManas, these days I added a new bool type attribute build-parameters ignore-use-full-glob-pattern to avoid regression. So there are some document update, and I recommit the PR and the new added docs are in commit: f3c1f60

@@ -209,6 +209,7 @@ From the MBT 1.2.25 version, the `build-parameters ignore` attribute will be upg

By using the negation pattern `!` in the `build-parameters ignore` attribute, you can select specified content into the package (because `not ignore` equals `include`).

Notice, to avoid regression, a new bool type attribute `build-parameters ignore-use-full-glob-pattern` (default value is `false`) is added. Only when it is true, `build-parameters ignore` will support [full glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)), otherwise MBT package behavior and result remain unchanged.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid regression, a new bool type attribute, build-parameters ignore-use-full-glob-pattern (the default value is false), has been added. Only when it is set to true, build-parameters ignore will support full glob patterns. Otherwise the MBT package behavior and result remain unchanged.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, Thanks

@@ -47,6 +47,8 @@ The `build-parameters ignore` attribute will be upgraded to support [full glob p

By using the `!` negation pattern in the `build-parameters ignore` attribute, you can select specified content into the package (because `not ignore` equals `include`).

Notice, to avoid regression, a new bool type attribute `build-parameters ignore-use-full-glob-pattern` (default value is `false`) is added. Only when it is true, `build-parameters ignore` will support [full glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)), otherwise MBT package behavior and result remain unchanged.
Copy link

@ShiraManas ShiraManas Jul 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid regression, a new bool type attribute, build-parameters ignore-use-full-glob-pattern (the default value is false), has been added. Only when it is set to true, build-parameters ignore will support full glob patterns. Otherwise the MBT package behavior and result remain unchanged.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, Thanks

	modified:   docs/docs/whatsnew.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants