Merge pull request #526 from highcharts/enhancement/readme-and-change…

…log-update

enhancement/readme-and-changelog-update

.env.sample

@@ -75,7 +75,7 @@ OTHER_HARD_RESET_PAGE = false
 
 # DEBUG CONFIG
 DEBUG_ENABLE = false
-DEBUG_HEADLESS = false
+DEBUG_HEADLESS = true
 DEBUG_DEVTOOLS = false
 DEBUG_LISTEN_TO_CONSOLE = false
 DEBUG_DUMPIO = false

CHANGELOG.md

@@ -1,62 +1,85 @@
 # 4.0.0
 
+_New Features:_
+
+- Implemented debug mode, including new environment variables, a config section, 'console' event listener, and npm script for debugging the headful Puppeteer browser.
+- Added the `HIGHCHARTS_CACHE_PATH` option available through `.env` to set a custom directory for the fetched files.
+- Added a moving average indicator for the exporting success rate ratio.
+
 _Enhancements:_
 
 - Improved server-related error handling by introducing new centralized error middlewares.
 - Improved overall error handling by adding a main try-catch block to correctly capture and log errors occurring throughout the code.
 - Introduced two new types of custom errors: `ExportError` for functionality-related errors and `HttpError` for server-related errors.
 - Introduced a new error logging mechanism with stack tracing using new function called `logWithStack`.
-- Added unit tests for certain parts of the code.
+- Expanded some error logs with request IDs.
+- Set headless mode to 'shell' for better performance, utilizing an older yet more efficient headless instance.
+- Set the `defaultViewport` to null and optimized code to trigger `setViewport` only once, reducing performance impact during export.
+- Removed unnecessary initial page on browser launch using `waitForInitialPage` and the `--no-startup-window` Chrome flag.
+- Revised Chromium flags sent to the browser, now located in the args array within the config file.
+- Optimized code by reducing evaluate function calls to enhance performance and minimize jumping between NodeJS and browser processes.
+- Optimized and moved chart creation initialization scripts from the HTML template to a separate module named `highcharts.js`.
+- Optimized the `clearPage` function to ensure content cleaning is only performed once, during resource release.
+- Introduced `hardResetPage` option for resetting the page's content (including Highcharts scripts) each time the page is released to the pool (defaulting to `false`).
+- Optimized creating and acquiring pages from the pool.
+- Optimized adding and releasing additional JS and CSS resources.
+- Made corrections for gracefully shutting down resources, including running servers, ongoing intervals, browser instance, created pages, and workers pool.
+- Updated `createImage` and `createPDF` functions with faster execution options including `optimizeForSpeed` and `quality`.
+- Set `waitUntil` to 'domcontentloaded' for `setContent` and `goto` functions to improve performance.
+- Replaced browser's deprecated `isConnected()` with the `onnected` property.
+- Added information on all available pool resources.
+- Numerous minor improvements for performance and stability.
 - Changed the `customCode` section of options to `customLogic` in order to avoid confusion with the existing `customCode` property within.
-- Changed the names of environment variables for a better representation of their roles (refer to all envs in the README's `Environment Variables` section).
-- Added parsing of envs based on `zod` package.
-- Added a new section to the server configuration options, `proxy`, along with corresponding environment variables.
-- Added a moving average indicator for the exporting success rate ratio.
-- Added new environment variables (`HIGHCHARTS_ADMIN_TOKEN`, `SERVER_BENCHMARKING`, and `OTHER_NODE_ENV`) to the `.env.sample` file, along with their descriptions in the README.
-- Added the `HIGHCHARTS_CACHE_PATH` option available through `.env` to set a custom directory for the fetched files.
-- Added several new functions to the `highcharts-export-server` module, including `logWithStack`, `setLogLevel`, `enableFileLogging`, `manualConfig`, `printLogo`, and `printUsage`.
-- Added a new `initLogging` function where the `setLogLevel` and `enableFileLogging` logic are consolidated into one place.
-- Added a new utility function, `isObjectEmpty`.
-- Added a new logging level (`5`) for benchmarking logs.
-- Added legacy names of options to the `defaultConfig` and `mapToNewConfig` function in order to support the old, PhantomJS-based structure of options.
-- Reordered the `error` and `info` arguments in the callback of the `startExport` function.
+- Renamed the environment variables for a better representation of their roles (refer to all envs in the README's `Environment Variables` section).
 - Renamed the `HIGHCHARTS_MODULES` and `HIGHCHARTS_INDICATORS` environment variables respectively to `HIGHCHARTS_MODULE_SCRIPTS` and `HIGHCHARTS_INDICATOR_SCRIPTS`.
 - Renamed the `scripts` property of the config options to `customScripts`.
 - Renamed the `initPool` function to `initExport` in the main module.
 - Renamed the `init` function to `initPool` in the pool module.
-- Moved the `listenToProcessExits` from the `pool` to the `other` section of the options.
 - Renamed the environment variable `POOL_LISTEN_TO_PROCESS_EXITS` to `OTHER_LISTEN_TO_PROCESS_EXITS`.
+- Moved the `listenToProcessExits` from the `pool` to the `other` section of the options.
+- Replaced the temporary benchmark module with a simpler server benchmark for evaluating export time.
+- Removed unnecessary separate `body-parser` package (already implemented in Express v4.16+).
+- Added parsing of envs based on `zod` package.
+- Added unit tests for certain parts of the code.
 - Added the `shutdownCleanUp` function for resource release (ending intervals, closing servers, destroying the pool and browser) on shutdown. It will be called in the process exit handlers.
+- Added new environment variables (`HIGHCHARTS_ADMIN_TOKEN`, `SERVER_BENCHMARKING`, and `OTHER_NODE_ENV`) to the `.env.sample` file, along with their descriptions in the README.
+- Added a new section to the server configuration options, `proxy`, along with corresponding environment variables.
+- Added several new functions to the `highcharts-export-server` module, including `initPool`, `logWithStack`, `setLogLevel`, `enableFileLogging`, `manualConfig`, `printLogo`, and `printUsage`.
+- Added a new `initLogging` function where the `setLogLevel` and `enableFileLogging` logic are consolidated into one place.
+- Added a new utility function, `isObjectEmpty`.
+- Added a new logging level (`5`) for benchmarking logs.
+- Added legacy names of options to the `defaultConfig` and `mapToNewConfig` function in order to support the old, PhantomJS-based structure of options.
 - Added a new process event handler for the `SIGHUP` signal.
-- Updated the `killPool` function.
+- Added `mapChart` and `ganttChart` constructors in the exporting UI (#503).
+- Reordered the `error` and `info` arguments in the callback of the `startExport` function.
 - Updates were made to the `config.js` file.
-- Replaced the temporary benchmark module with a simpler server benchmark for evaluating export time.
+- Updated the `killPool` function.
 - The `uncaughtException` handler now kills the pool, browser, and terminates the process with exit code 1, when enabled.
 - The browser instance should be correctly closed now when an error occurs during pool creation.
 - Corrected error handling and response sending in the `/change_hc_version.js` route.
 - Corrected the `handleResources` function.
 - Corrected samples, test scenarios, and test runners.
-- Removed unnecessary separate `body-parser` package (already implemented in Express v4.16+).
-- Bumped versions of most packages, with an update for the deprecated `Puppeteer` v21.1.1.
-- Added `mapChart` and `ganttChart` constructors in the exporting UI (#503).
+- Bumped versions of most packages, with an updating deprecated `Puppeteer` from `v21.1.1` to latest.
 - Added missing Highcharts modules to stay up-to-date with the latest updates.
 - Added missing JSDoc descriptions.
 - Revamped all log messages, error messages, prompt messages, and info for improved clarity of information.
-- README has been revised and corrected by incorporating additional information, improving descriptions, adding missing details, including new API information, and expanding with new sections such as `Available Endpoints`, `Examples`, and a `Note about Deprecated Options`.
+- README has been revised and corrected by incorporating additional information, improving descriptions, adding missing details, including new API information, and expanding with new sections such as `Debugging`, `Available Endpoints`, `Examples`, and a `Note about Deprecated Options`.
 - Updated Wiki pages with a new `Samples` section.
 
 _Fixes:_
 
 - Fixed `multer` related error: 'Field value too long'.
 - Fixed the SSL handshake error (#307).
 - Fixed missing background color transparency (#492).
+- Fixed missing `foreignObject` elements issue.
 - Fixed type compatibility issues in the `pairArgumentValue` function, arising from CLI string arguments.
 - Fixed the 'httpsProxyAgent is not a constructor' issue with the `https-proxy-agent` module.
 - Fixed the issue of being unable to run both HTTP and HTTPS servers simultaneously.
 - Fixed the issue with the `multiselect` type of values in prompt functionality triggered by the `--createConfig` option.
 - Fixed the error handling in the `postWork` function which resulted in doubled errors.
 - Fixed the deprecated description of the pool from the `generic-pool` to `tarn` notation, triggered by the `getPoolInfo` and `getPoolInfoJSON` functions.
 - Fixed the issue of not gracefully terminating the process when an error occurs and a pool or browser already exists.
+- Fixed the 'Could not clear the content of the page... - Target closed' error.
 - Made minor corrections to ESLint and Prettier configuration.
 - Other minor stability, linting and text corrections have been implemented.
 

README.md

@@ -2,9 +2,9 @@
 
 Convert Highcharts.JS charts into static image files.
 
-## Upgrade Notes for v3.0
+## Upgrade Notes
 
-In most cases, v3 should serve as a drop-in replacement for v2. However, due to changes in the browser backend, various tweaks related to process handling (e.g., worker counts, and so on) may now have different effects than before.
+In most cases, v4 should serve as a drop-in replacement for v2 and v3. However, due to changes in the browser backend, various tweaks related to process handling (e.g., worker counts, and so on) may now have different effects than before.
 
 Significant changes have been made to the API for using the server as a Node.js module. While a compatibility layer has been created to address this, it is recommended to transition to the new API described below. It is worth noting that the compatibility layer may be deprecated at some point in the future.
 
@@ -14,6 +14,8 @@ Additionally, with the v3 release, we transitioned from HTTP to HTTPS for export
 
 ## Changelog
 
+**The v4 introduces numerous breaking changes. For further details, please refer to the changelog document provided below.**
+
 The full change log for all versions can be viewed [here](CHANGELOG.md).
 
 # What & Why
@@ -806,6 +808,40 @@ If you need Google Fonts in your custom installation, they can be had here: http
 
 Download them, and follow the above instructions for your OS.
 
+# Debug Mode
+
+Version 4.0.0 introduced a new mode that allows debugging the Puppeteer browser instance. This is particularly useful when setting up a custom server. It helps to delve into the implementation, observe how things work, and analyze and resolve potential problems.
+
+## Launching
+
+Setting the `--enableDebug` to `true` passes all debug options to the `puppeteer.launch()` function on startup. Together with the `--headless` option set to `false`, it launches the browser in a headful state providing a full version of the browser with a graphical user interface (GUI). While this serves as the minimal configuration to simply display the browser, Puppeteer offers additional options. Here is the full list:
+
+- `--enableDebug`: Enables passing debug options to the `puppeteer.launch()`.
+- `--headless`: Sets the browser's state.
+- `--devtools`: Allows turning on the DevTools automatically upon launching the browser.
+- `--listenToConsole`: Allows listening to messages from the browser's console.
+- `--dumpio`: Redirects the browser's process `stdout` and `stderr` to `process.stdout` and `process.stderr` respectively.
+- `--slowMo`: Delays Puppeteer operations by a specified amount of milliseconds.
+- `--debuggingPort`: Specifies a debugging port for a browser.
+
+## Debugging
+
+There are two main ways to debug code:
+
+- By adding a `debugger` statement within any client-side code (e.g., inside a `page.evaluate` callback). With the `--devtools` option set to `true`, the code execution will stop automatically.
+
+- By running the export server with the `--inspect-brk=<PORT>` flag, and adding a `debugger` statement within any server-side code. Subsequently, navigate to `chrome://inspect/`, input the server's IP address and port (e.g., `localhost:9229`) in the Configure section. Clicking 'inspect' initiates debugging of the server-side code.
+
+The `npm run start:debug` script from the `package.json` allows debugging code using both methods simultaneously. In this setup, client-side code is accessible from the devTools of a specific Puppeteer browser's page, while server-side code can be debugged from the devTools of `chrome://inspect/`.
+
+For more details, refer to the [Puppeteer debugging guide](https://pptr.dev/guides/debugging).
+
+## Additional Notes
+
+- Ensure to set the `--headless` to `false` when the `--devtools` is set to `true`. Otherwise, there's a possibility that while DevTools may be recognized as enabled, the browser won't be displayed. Moreover, if a `debugger` is caught within the browser, it might lead to the entire debugging process getting stuck. In such scenarios, you can set the IP address and port (using the value of the `--debuggingPort` option) the same way as described in the section for debugging server-side code. This allows you to access DevTools and resume code execution.
+
+- When using the `--listenToConsole` and `--dumpio` options, be aware that the server's console may become 'polluted' with messages from the browser. If you prefer to avoid this, simply set both options to false.
+
 # Performance Notice
 
 In cases of batch exports, using the HTTP server is faster than the CLI. This is due to the overhead of starting Puppeteer for each job when using the CLI.

package.json

@@ -62,7 +62,7 @@
     "jsdom": "^24.0.0",
     "multer": "^1.4.5-lts.1",
     "prompts": "^2.4.2",
-    "puppeteer": "^22.8.2",
+    "puppeteer": "^22.9.0",
     "tarn": "^3.0.2",
     "uuid": "^9.0.1",
     "zod": "^3.23.8"