v1.8.0: July 2021 Release

Release v1.8.0

New Features

• Marketplace related update

  • Please see Marketplace for more details

• Alert manager

  • Send alert to users when job status changed #5337

• Webportal

  • Support UX of Job Priority #5417

• Others

  • Customizable Autoscaler #5412
  • Add custom ssl port support #5386
  • Clean up repo. Remove obsolete code #5489

v1.7.0: April 2021 Release

Release v1.7.0

New Features

• Marketplace related update

  • Please see Marketplace for more details

• New job submission page

  • Please refer to new submission tutorial for how to use new submission page.
  • New submission page replaces Advanced with More info and places it under each section to improve user experience.
  • In new submission page, the sidebar can be shrank to give the main area more visual space.
  • The new submission page moves the yaml editor into a single page, which allows user to focus on setting config or editing yaml protocol.
  • The new submission page improves the responsive design in small and medium resolution.

  Know Issue: Tensorboard tool is not implemented in the new submission page yet. If you need to use it, please use the old version.

• Alert system enhancement

  • Add alert & auto-fix for GPU perf issue #5342 #5383
  • Refine kill-low-efficiency-job-alert email templates #5384
  • Add alert for API server cert expired #5334

• Support sort by completionTime for get job list API #5347

• Deployment

  • Support add/remove nodes via paictl.py #5321 #5167. (Warning config.yaml need to be added for this feature. refer: https://github.com/microsoft/pai/blob/master/docs/manual/cluster-admin/how-to-add-and-remove-nodes.md#pull--modify-cluster-settings)

Bug fixes:

• Webportal package build issue #5378

v1.6.0: Mar 2021 Release

Release v1.6.0

Upgrade Guide

Before upgrade, we recommend you to check this issue first.

New Features

• Job protocol update: Add prerequisites #5145

• Marketplace related update

  • Please see Marketplace for more details.
  • Warning Marketplace v1.6.0 has some breaking change, upgrade from previous should follow the Marketplace upgrade guide

• Introduce an optional docker cache in cluster #5290

• A regular GPU utilization report can be set up for admins #5281, #5294, #5324, #5331

  • #5324 introduces a schema change for pai-bearer-token in the alert-manager section. The old configuration still works but is deprecated. If you have configured pai-bearer-token of alert-manager, please refer to #5331 to modify the previous configuration.

• Users can save frequently-used SSH publish keys on the profile page #5223

• Improve log experience #5271 #5272

• Reduce ansible logs when deploy #5305

Bug Fixes:

• Database controller: Tolerant to wrong framework spec #5284
• Database controller: Remove sensitive fields in db #5289
• Database controller: Fix memory leak #5309
• Set correct launchTime in rest-server #5307
• Database may use unmounted host path #5343

v1.5.0: Jan 2021 Release

Release v1.5.0

New Features

• Improve Web Portal Experience

  • Fix Home page overlap issue #5213 #5180
  • Add filter, search box and export csv button in task detail list #5175

• Create a new page for yaml editor #5172

• Marketplace related update

  • Please see microsoft/openpaimarketplace#152 for more details

• Support different types of computing hardware #5138

• Deployment process refinement

  • master.csv + worker.csv -> layout.yaml
  • move config.yaml, layout.yaml under quick-start folder, remove all the argument parse logic
  • Add support for cpu-only worker installation
  • Add support for heterogeneous workers
  • Unify version requirements: pai version, pai image tag
  • Set default value in config files
  • Generate hiveD config with layout.yaml #5179
  • Check layout before installing k8s #5184 #5181
  • Config folder structure arrangement
  • Refine installation logs
  • Add skip service list argument #5193

• Log manager

  • Change get logs api return code #5125

v1.4.1: Dec 2020 Release

Release v1.4.1

Bug Fixes

• Marketplace
  • Fix initializing blob data issue (#5189)
• Log Collection
  • Fix getting wrong log for retried task & frontend crash issue (#5190)

v1.4.0: Dec 2020 Release

Release v1.4.0

New Features

• multi-cluster (#4929)
  • Support job transfer (#5082, #5088)
• Autoscaler
  • Update docs for Cluster Autoscaler on AKS Engine (#5057)
• Log Collection (#4992)
  • Rest API
  • Webportal
• Https configuration document (#5076, #5078)
• Marketplace (microsoft/openpaimarketplace#73)
  • Data
    • Move NFS to Azure Blob as backend
    • Upload Job output to Azure Blob
    • Download data from azure blob to local
    • Use Azure storage SDK for privacy
    • Refactor data use logic after change storage to blob
    • Update project development doc and manual
  • Service Deployment
    • Start Local Rest Server
    • Deployed Rest Server in PAI
    • Start database and save items into it
    • Register in PAI pylon (#5066)
    • Add azure storage to service configuration (#5104)
• Web Portal
  • Fix stop job button issue #5079
• Admin Experience
  • Prometheus alert rules update (#5021)
  • Refine deployment process (#5077, #5085)

Bug Fixes

• Fix updateUserGroupList API issue (#5121)
• Fix hived config issue caused by k8s coreDNS deployment (#5071)

v1.3.0: Nov 2020 Release

Release v1.3.0

New Features

• Marketplace
  • New templates in marketplace (microsoft/openpaimarketplace#60)
• HiveD Scheduler
  • Support cluster autoscale with HiveD scheduler on AKS (#4868)
  • Support dynamic sku types for different vc on webportal (#4900)
• Advanced job debug mode
  • Add per task retry history (microsoft/frameworkcontroller#62, #4958, #4966)
  • Expose Kubernetes events (#4939, #4975)
• GPU monitoring and utilization
  • Support job tagging (#4924)
  • Stop low GPU utilization job with alert-manager (#4940)
  • Cordon node with GPU ECC Errors (#4942)
• Documentation
  • Fix document according to DRI tickets (#4828)
  • Add distributed examples (#4821)
• Webportal
  • Add help info for items on webportal (#4950)

Known Issues

• Job stop button no feedback after click successfully (#5023)
• Alert handler stop-job notice not clear to end user (#5021)
• DB framework / Rest-Server job inconsistency (#5027)

v1.2.1: Oct 2020 Release

Release v1.2.1

Bug Fixes:

• Fix config generate bug #4970
• Fix database controller dependency #4978

v1.2.0: Sep 2020 Release

Release v1.2.0

New Features

• Database
  • New RestServer Arch: RestServer -> DB -> ApiServer (#4651)
• Webportal
  • Job list paging in server side (#4651)
  • Change job createdTime to submissionTime (#4761)
  • VC && Group experience for Admin (AAD Mode) (#4800)
  • Support SKU count and SKU type in job submission page (#4796)
  • Upgrade api/v1 code to api/v2 (#4704)
  • Show "More Diagnostics" in job detail page (#4670)
• Marketplace
  • Unified PAI protocol
  • Merge examples
• Others
  • Quick start for AKS-Engine

Improvements

• HiveD improvement (#4868)
• Robustness improvement (#4694)

Bug Fixes

• Fix logrotate issue (#4792)
• Fix runtime image check issue
• Fix error message for SKU (#4602)

Known Issues

• Old framework retry history cannot be shown after upgrading to v1.2.0
• Upgrade Guide

v1.1.1: July 2020 Release

Release v1.1.1

Bug Fixes:

• Fix SDK request timeout #4756