Distribution of API Developer Portal #3343

piyumaldk · 2024-11-06T11:57:38Z

Description

The task is to determine the most suitable approach for distributing the API Developer Portal, focusing on efficiency and minimising platform dependencies. This involves exploring, implementing, and testing various methods to finalise the best distribution strategy.

Affected Component

None

Version

No response

Related Issues

No response

Suggested Labels

No response

piyumaldk · 2024-11-06T12:24:20Z

Update [2024-11-06]

Participated meeting Dev Portal Plan @10.30AM to 11.00AM
- Decided to assign me to $Subject task in the meeting
Had a discussion after the meeting with UI/UX APIM Lead to discuss the objective of the task
Setup repo: https://github.com/wso2/api-developer-portal
- Noticed below key facts while doing the setup
- Postgres is required to run the product (Follow the pgAdmin 4 flow)
- Later java version is required to run the product (Was't able to run with java11, had to switch java17)
- devportal core: https://github.com/wso2/api-developer-portal-core is imported to devportal as a node module which is a decent practice
In the morning discussion, a concern of user having all the modules (Including core related modules) when user is installing core node module was raised.
- Since devportal only had core as the dependency, decided to try with packaging (pkg) to get rid of dependencies.
When packaging after required changes (starting points of executable file as app.js, multi-tenant.js, and single-tenant.js, we can get executable files but concern is they are not os independent. Due to this, generated linux, mac, and windows files.
- Compared the executable files (~40mb) with dev portal node module size (~500mb). Considering the sizes, packaging was promising.

Studied the devportal code and found out

startup.sh is running when running the project.
In the script, depending on the situation, multi and single tenant related js is executing.

if [ "$tenant" == "single" ] || [ "$tenant" == "dev" ]; then
    npm run build-css --watch & node ../node_modules/devportal-webapp/src/single-tenant.js
else
   node ../node_modules/devportal-webapp/src/multi-tenant.js
fi

Change the script to run generated executable files

# Check the OS
OS=$(uname -s)

# Path to the bundled executables
BIN_DIR="./devportal-webapp"

# Set executable file based on OS
case "$OS" in
 "Darwin")
   EXECUTABLE="$BIN_DIR/devportal-webapp-macos"
   ;;
 "Linux")
   EXECUTABLE="$BIN_DIR/devportal-webapp-linux"
   ;;
 CYGWIN*|MINGW*)
   EXECUTABLE="$BIN_DIR/devportal-webapp-win.exe"
   ;;
 *)
   echo "Unsupported OS: $OS"
   exit 1
   ;;
esac

# Run the correct tenant mode with CSS watch
if [ "$tenant" == "single" ] || [ "$tenant" == "dev" ]; then
 npm run build-css --watch & exec "$EXECUTABLE" single-tenant
else
 exec "$EXECUTABLE" multi-tenant
fi

While testing, notice that watcher is requiring below modules, hence wasn't able to get rid of the whole node modules.
- chokidar
- fs-extra
Even after installing both, product fails while finding express
- In theory, packing should include required node modules, it seems it has missed express. Found out that this can happen because pkg sometimes fails to recognize dynamically required modules.

TODO: Try adding express related modules manually to packaging and test

piyumaldk · 2024-11-07T07:38:06Z

Update [2024-11-07]

Change the pkg settings like below to include all the required files and modules

"bin": {
    "devportal-webapp": "src/app.js",  
    "multi-tenant": "src/multi-tenant.js",
    "single-tenant": "src/single-tenant.js"
  },
  "pkg": {
  "scripts": [
    "**/*.js"
  ],
  "assets": [
    "src/**/*",       
    "package.json",  
    "watcher.js"      
  ],
  "targets": [
    "node16-linux-x64",
    "node16-macos-x64",
    "node16-win-x64"
  ],
  "outputPath": "dist"
  }

This solved the above issue but executable files will become around ~200mb. Since we have 3 files for 3 operating systems, size wise, npm nodules will be lighter (Due to the duplication of executable files)
When running the devportal with executable files, faced an issue.
- When investigating the issue, noticed that in core module, we have used relative paths (for example: '../../../../src/')
- This ideally tells the code to refer outside the node modules and in the devportal
- This has been referred in multiple places which is a problem (With the current core, we cannot move to pkg packaging because it will manipulate the file structure which will eventually results ENOENT
Tried npm pack flow and pack the tar file locally.
- Even though this method removes the need of getting npm modules from cloud, found out that this process needs a npm i to work properly.
- This is due to the pack avoids node modules

   "dependencies": {
    "devportal-webapp": "file:./artifacts/devportal-webapp-1.0.59.tgz",
    "chokidar": "^4.0.1",
    "fs-extra": "^11.2.0",
    "path": "^0.12.7"
  }

However since above process looked promising, tried a way to pack node modules as well.
- Tried [1] in order to pack node modules as well.
- When trying out, noticed that even if this correctly pack the module and dependencies, to work properly, npm install is a must where this pack's modules will be installed, hence this option is also not sufficient for the requirement.
Had a discussion with a team member (owner of devportal code) and discussed the issue faced when using pkg method.
- After the discussion, decided to get the path as absolute and use inbuilt process path of node to do so. [2]
- Changed the core code (9 files which has relative path) and tried the new module with packing. No errors were found.
- 4 relative paths were ignored due to the wrong path. (This needs to be further discussed)
Faced an issue when using pkg packaging with changed paths. Found below in the debugging process.
- Found out that path is still wrong due to the combination of dir path. Fix that by removing dir paths from core code and checked.
- Execution path was not from the script but from the mac exe file.
Faced another issue when using pkg packaging with changed paths with removed dir paths. Found below in the debugging process.
- Path is starting from executable file and not from script itself which is resulting the wrong path.
- Added '/../../..' to each path to navigate back to root location and tried again.
- Initial issues were not there but faced another issue which is mentioned below.

> devportal-developer-webapp@1.0.0 devportal
> sh startup.sh single

Starting Backend service...
Starting Devportal application...

> devportal-developer-webapp@1.0.0 build-css
> node watcher.js

pkg/prelude/bootstrap.js:657
    const error = new Error(
                  ^

Error: File '/**/api-developer-portal-core/src/utils/Users/piyumal/Desktop/forked-apim/api-developer-portal/src/pages/home/page.hbs' was not included into executable at compilation stage. Please recompile adding it as asset or script.
    at error_ENOENT (pkg/prelude/bootstrap.js:657:19)
    at readFileFromSnapshot (pkg/prelude/bootstrap.js:1046:29)
    at Object.readFileSync (pkg/prelude/bootstrap.js:1094:18)
    at renderTemplate (/snapshot/api-developer-portal-core/src/utils/util.js:83:33)
    at loadOrgContentFromFile (/snapshot/api-developer-portal-core/src/controllers/orgContentController.js:36:12)
    at loadOrganizationContent (/snapshot/api-developer-portal-core/src/controllers/orgContentController.js:16:22)
    at Layer.handle [as handle_request] (/snapshot/api-developer-portal-core/node_modules/express/lib/router/layer.js:95:5)
    at next (/snapshot/api-developer-portal-core/node_modules/express/lib/router/route.js:149:13)
    at Route.dispatch (/snapshot/api-developer-portal-core/node_modules/express/lib/router/route.js:119:3)
    at Layer.handle [as handle_request] (/snapshot/api-developer-portal-core/node_modules/express/lib/router/layer.js:95:5) {
  errno: -2,
  code: 'ENOENT',
  path: '/snapshot/api-developer-portal-core/src/utils/Users/piyumal/Desktop/forked-apim/api-developer-portal/src/pages/home/page.hbs',
  pkg: true
}

TODO: Debug the new issue

[1] https://www.npmjs.com/package/npm-pack-all
[2] https://www.geeksforgeeks.org/node-js-process-execpath-property/

piyumaldk · 2024-11-08T12:05:01Z

Update [2024-11-08]

Debugged above issue.
- Found out that this is because of some paths try to access the inside of node module.
- Since changing all the paths, this looks troublesome so focused on new methods.
Found out about below methods.
- Nexe
- Webpack with Node File System (fs) Plugin
- Bytenode
- Node.js Modules with Docker
From above methods, tried Bytenode. It was more capable of simple or single files so gave up on that approach.
Docker was initially considered but gave up due to the the fact that user have to depend on Docker.
Compared Nexe and Webpack and it was said that Webpack is the better solution for the situation on multiple places.
Tried Webpack apporach including node modules (As the requirement is to get rid of node modules in dev portal)
Below are the changes made for core to work with Webpack

webpack.config.js in root

const path = require('path');
const CopyWebpackPlugin = require('copy-webpack-plugin');

module.exports = {
  mode: 'production',
  stats: {
    errorDetails: true
  },
  entry: {
    // app: './src/app.js',              
    // multiTenant: './src/multi-tenant.js', 
    singleTenant: './src/single-tenant.js' 
  },

  output: {
    path: path.resolve(__dirname, 'dist'), 
    filename: '[name].bundle.js', 
  },

  target: 'node',

  module: {
    rules: [
      {
        test: /\.js$/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: ['@babel/preset-env'],
          },
        },
      },
    ],
  },

  plugins: [
    new CopyWebpackPlugin({
      patterns: [
        { from: './src/pages/tryout', to: './src/pages/tryout' },
      ],
    }),
  ],

  resolve: {
    extensions: ['.js', '.json'],
    alias: {
      '@src': path.resolve(__dirname, 'src'),
    },
  },
};

As you can see, node modules were not excluded.
./src/pages/tryout contains non js files so they were copied manually with CopyWebpackPlugin
Add three entries, and commented out 2 as I only need single tenant for testing.
Change the mode to production

package.json in root

"scripts": {
    "design-mode": "npm run build-css --watch & nodemon src/dev-server.js",
    "devportal": "npm run build-css --watch & nodemon src/app.js",
    "multi-tenant": "npm run build-css --watch & nodemon src/multi-tenant.js",
    "build-css": "node watcher.js",
    "build": "webpack --config webpack.config.js"
  },
"devDependencies": {
    "@babel/core": "^7.26.0",
    "@babel/preset-env": "^7.26.0",
    "babel-loader": "^9.2.1",
    "copy-webpack-plugin": "^12.0.2",
    "webpack": "^5.96.1",
    "webpack-cli": "^5.1.4"
  }

Webpacck to work, added above dev dependencies.
Added build script to build the webpack
Change the devportal startup script as below

if [ "$tenant" == "single" ] || [ "$tenant" == "dev" ]; then
    node ../artifacts/devportal-webapp/singleTenant.bundle.js
else
    node ./artifcats/devportal-webapp/multiTenant.bundle.js
fi

Added the built webpack and resources to the correct path (devportal-webapp) and tested
- Later removed the watcher to test without node modules (As watcher is using few node modules)
- Build was broken on multiple places
Debugged and changed the core code accordingly and tested again.
- Made changes
In src/utils/util
var filePrefix = '../../src/';

In src/single-tenant
var filePrefix = '../../src/';
copyStyelSheet('../../../src/');
app.use('/styles', express.static(path.join(__dirname, '../../src/' + '/styles')));
const folderToDelete = path.join(__dirname, '../../src/' + '/styles');

Note: Other changes are not required as only these files are used with current devportal
- Portal started fine
- Tried adding an API and it also worked fine

TODO: There is a warning in the server like below.

Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.

This needs to be further investigated

piyumaldk · 2024-11-11T08:32:16Z

Update [2024-11-11]

Debug the mentioned warning

Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.

Warning is happening because core webpack is using MemoryStore, which isn't recommended in production because it can cause memory leaks and won't work well in a multi-process environment.
Bug further and found out that, in core, express-session is being used and config like below

app.use(session({
    secret: secret,
    resave: false,
    saveUninitialized: true,
    cookie: { secure: false } // Set to true if using HTTPS
}));

According to sources, we need to add a store to get rid of the issue like store: new RedisStore({ client: redisClient })
Otherwise it will use MemoryStore
However, according to some people in stackoverflow, using a separate store for simple tasks in an overkill. [1]
There are another suggestions like changing the implementation to handle proper clean up or using cookie-session instead of express-session. This needs to be fixed before going to production.
Created a project to track devportal issues [2]
Study path changes needed for the webpack to work.
- All relative path changes should change to '../..' because bundle file is inside 2 directories of parent project's root.
- This is because even though there are a complex file structure in core, all js code will be bundled into one when doing the webpack build.
Scheduled a call to discuss the progress and to get the approval to the approach.
- @1.30 [Discussion] Devportal distribution
- Got to know that a key participant is busy so had to cancel the call

TODO: Once the approval is there, should work on the Jenkins build

[1] https://stackoverflow.com/questions/10760620/using-memorystore-in-production
[2] https://github.com/orgs/wso2/projects/115/views/1

piyumaldk · 2024-11-12T09:43:06Z

Update [2024-11-12]

Did some study on Jenkins build
Got to know that it's way easier to handle the expected behaviour using GitHub actions
Wrote below GitHub action for webpack build (Build was successful, however there were some errors in releasing)

name: Build and Release on PR Merge

on:
  pull_request:
    types: [closed]
    branches: [main] # Adjust if you want a different branch

jobs:
  build:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4 # Update to the latest version

      - name: Set up Node.js
        uses: actions/setup-node@v4 # Update to the latest version
        with:
          node-version: '20' # Use Node.js 20

      - name: Install dependencies
        run: npm install

      - name: Build project
        run: npm run build

      - name: Upload dist folder as an artifact
        uses: actions/upload-artifact@v4 # Update to the latest version
        with:
          name: dist-output
          path: dist/

  release:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    needs: build

    steps:
      - name: Download dist folder artifact
        uses: actions/download-artifact@v4 # Update to the latest version
        with:
          name: dist-output

      - name: Create release directory
        run: mkdir release && cp -R dist/* release/

      - name: Create GitHub release
        uses: ncipollo/release-action@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          tag: v1.${{ github.event.pull_request.number }} # Custom tag format using PR number
          name: Release for PR #${{ github.event.pull_request.number }}
          body: "Automated release generated on PR merge."
          draft: false
          prerelease: false
          artifacts: "release/*"

Final WebPack config is like below

const path = require('path');
const CopyWebpackPlugin = require('copy-webpack-plugin');
const TerserPlugin = require('terser-webpack-plugin');

module.exports = {
  mode: 'production',
  stats: {
    errorDetails: true,
  },
  entry: {
    // app: './src/app.js',
    // multiTenant: './src/multi-tenant.js',
    singleTenant: './src/single-tenant.js',
  },

  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: '[name].bundle.js',
  },

  target: 'node',

  module: {
    rules: [
      {
        test: /\.js$/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: ['@babel/preset-env'],
          },
        },
      },
    ],
  },

  plugins: [
    new CopyWebpackPlugin({
      patterns: [
        { from: './src/pages/tryout', to: './src/pages/tryout' },
      ],
    }),
  ],

  resolve: {
    extensions: ['.js', '.json'],
    alias: {
      '@src': path.resolve(__dirname, 'src'),
    },
  },

  optimization: {
    minimize: true,
    minimizer: [
      new TerserPlugin({
        terserOptions: {
          format: {
            comments: false, 
          },
        },
        extractComments: false, 
      }),
    ],
  },
};

Participated the PR review call for devportal
- In there, it was suggested that to ignore webpack as it needs node to run and try a method in exe.
Again start working on pkg (As mentioned in the issue, I have worked on this approach for some time so I did not have to start from the beginning)
Found out that when using execute path [1], it's like below which will also include the exe file.
- exe path - /Users/piyumal/Desktop/forked-apim/api-developer-portal/artifacts/devportal-webapp
- Because of that, change the _dirname with execute path like below

- app.use('/images', express.static(path.join(__dirname, filePrefix + 'images')));
+ app.use('/images', express.static(path.join(process.execPath, filePrefix + 'images')));

All filePrefix were changed to back twice accordingly.
Tested the MacOs exe file. Worked fine.
- Below is the pkg setup in package.json.

"bin": {
    "single-tenant": "src/single-tenant.js"
  },
  "pkg": {
    "scripts": [
      "**/*.js"
    ],
    "assets": [
      "src/pages/*"
    ],
    "targets": [
      "node18-macos-x64",
      "node18-linux-x64",
      "node18-win-x64"
    ],
    "outputPath": "dist"
  },
  "devDependencies": {
    "pkg": "^5.8.1"
  }

Change build script to run pkg .
Change the earlier written GitHub action to work (Tested for multiple times in forked repo's master branch_
- Below is the working GitHub action (Path: .github/workflows/build-and-release.yml)

name: Build and Release on PR Merge

on:
  pull_request:
    types: [closed]
    branches: [master] # Only trigger for PRs targeting the master branch

permissions:
  contents: write  # Ensures the workflow can create releases

jobs:
  build:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code from master branch
        uses: actions/checkout@v4
        with:
          ref: master  # Explicitly checkout master branch

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install

      - name: Build project
        run: npm run build

      - name: Upload dist folder as an artifact
        uses: actions/upload-artifact@v4
        with:
          name: dist-output
          path: dist/

  release:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    needs: build

    steps:
      - name: Checkout master branch
        uses: actions/checkout@v4
        with:
          ref: master  # Ensure we're on master for tagging

      - name: Download dist folder artifact
        uses: actions/download-artifact@v4
        with:
          name: dist-output
          path: ./dist-output  # Download artifact to a local folder

      - name: Create tag for release
        run: |
          git config --global user.name "github-actions[bot]"
          git config --global user.email "github-actions[bot]@users.noreply.github.com"
          git tag v1.${{ github.event.pull_request.number }}
          git push origin v1.${{ github.event.pull_request.number }}

      - name: Debug downloaded artifact contents
        run: ls -R ./dist-output  # Lists the contents for debugging purposes

      - name: Create GitHub release
        uses: ncipollo/release-action@v1
        with:
          tag: v1.${{ github.event.pull_request.number }}  # Use the manually created tag
          name: Release for PR #${{ github.event.pull_request.number }}
          body: "Automated release generated on PR merge."
          draft: false
          prerelease: false
          artifacts: "./dist-output/*"  # Include all files in the dist-output folder as release assets

Below is the output as expected
Tested the macos file in devportal. Script failed due to permission issue.

TODO: Check on the permission issue

[1] https://www.geeksforgeeks.org/node-js-process-execpath-property/

piyumaldk · 2024-11-13T12:05:29Z

Update [2024-11-13]

Permission issue is happening because macOS does not trust downloading executable files and it removes the excutable permission. Changing the permission worked.
Had a meeting about the progress. Asked to change the workflow not to trigger when PR is merged.
- Changed the workflow to work when a tag release is done. Tested and worked successfully.
- Below is the new workflow

name: Build and Release on Tag Push

on:
  push:
    tags:
      - 'v*'  

permissions:
  contents: write  

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install

      - name: Build project
        run: npm run build

      - name: Upload dist folder as an artifact
        uses: actions/upload-artifact@v4
        with:
          name: dist-output
          path: dist/

  release:
    runs-on: ubuntu-latest
    needs: build

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Download dist folder artifact
        uses: actions/download-artifact@v4
        with:
          name: dist-output
          path: ./dist-output  

      - name: Zip each file in dist-output
        run: |
          mkdir -p zipped-output
          for file in ./dist-output/*; do
            filename=$(basename "$file")
            zip -r "zipped-output/$filename.zip" "$file"
          done

      - name: Debug zipped artifact contents
        run: ls -R ./zipped-output  

      - name: Create or Update GitHub Release
        uses: ncipollo/release-action@v1
        with:
          tag: ${{ github.ref_name }}  
          name: Release ${{ github.ref_name }}
          body: "Automated release generated on tag push."
          draft: false
          prerelease: false
          artifacts: "./zipped-output/*"  
          allowUpdates: true

New core changes were received, so started working on building executable file on new core.
- Came up with UUID module not found error. Installing that fixed the error.
- Currently fails to load the portal after all the required changes.

TODO: Debug and find out the reason of failing the portal

piyumaldk · 2024-11-14T12:41:35Z

Update [2024-11-14]

Fixed all the bugs and failures and sent the PR
- Capability to release, build executable files, and run in node api-developer-portal-core#41
Automated both released
- One to package executables
  Example release done: https://github.com/piyumaldk/api-developer-portal-core/releases/tag/v0.0.4
- One to refer other repo assets and pack the needed files
  Example release done: https://github.com/piyumaldk/api-developer-portal/releases/tag/v0.0.9

piyumaldk added Type/Task Component/DevPortal labels Nov 6, 2024

piyumaldk self-assigned this Nov 6, 2024

github-actions bot added the Missing/Component label Nov 6, 2024

piyumaldk mentioned this issue Nov 6, 2024

Distribution of API Developer Portal wso2/api-developer-portal-core#34

Closed

piyumaldk removed the Missing/Component label Nov 6, 2024

piyumaldk closed this as completed Nov 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distribution of API Developer Portal #3343

Distribution of API Developer Portal #3343

piyumaldk commented Nov 6, 2024

piyumaldk commented Nov 6, 2024 •

edited

Loading

piyumaldk commented Nov 7, 2024 •

edited

Loading

piyumaldk commented Nov 8, 2024

piyumaldk commented Nov 11, 2024 •

edited

Loading

piyumaldk commented Nov 12, 2024 •

edited

Loading

piyumaldk commented Nov 13, 2024

piyumaldk commented Nov 14, 2024

Distribution of API Developer Portal #3343

Distribution of API Developer Portal #3343

Comments

piyumaldk commented Nov 6, 2024

Description

Affected Component

Version

Related Issues

Suggested Labels

piyumaldk commented Nov 6, 2024 • edited Loading

Update [2024-11-06]

piyumaldk commented Nov 7, 2024 • edited Loading

Update [2024-11-07]

piyumaldk commented Nov 8, 2024

Update [2024-11-08]

piyumaldk commented Nov 11, 2024 • edited Loading

Update [2024-11-11]

piyumaldk commented Nov 12, 2024 • edited Loading

Update [2024-11-12]

piyumaldk commented Nov 13, 2024

Update [2024-11-13]

piyumaldk commented Nov 14, 2024

Update [2024-11-14]

piyumaldk commented Nov 6, 2024 •

edited

Loading

piyumaldk commented Nov 7, 2024 •

edited

Loading

piyumaldk commented Nov 11, 2024 •

edited

Loading

piyumaldk commented Nov 12, 2024 •

edited

Loading