Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Windows CI #1028

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
c3cd476
initial windows build
RUIJIEZHONG66166 Aug 16, 2024
9df224e
add windows build in pull.yml
RUIJIEZHONG66166 Aug 16, 2024
b4669fd
set ut input as not required
RUIJIEZHONG66166 Aug 16, 2024
4170cee
remove unsupported cmd
RUIJIEZHONG66166 Aug 16, 2024
5c1ab12
add for conda activate
RUIJIEZHONG66166 Aug 16, 2024
1c42704
add call for all conda cmd
RUIJIEZHONG66166 Aug 19, 2024
8877db0
update rm to rmdir
RUIJIEZHONG66166 Aug 20, 2024
9505441
change pytorch check
RUIJIEZHONG66166 Aug 20, 2024
5ab9b63
update copy torch-xpu-ops
RUIJIEZHONG66166 Aug 20, 2024
66d6e50
add dependancy
RUIJIEZHONG66166 Aug 20, 2024
7c37368
add USE_XPU=1
RUIJIEZHONG66166 Aug 20, 2024
5215d4a
use USE_KINETO=0
RUIJIEZHONG66166 Aug 22, 2024
691f66c
update install_oneapi_bundle.bat
RUIJIEZHONG66166 Aug 22, 2024
db51cef
update VS2022INSTALLDIR
RUIJIEZHONG66166 Aug 22, 2024
bccc6a0
add upload windows binary
RUIJIEZHONG66166 Aug 23, 2024
b7834b7
update some name
RUIJIEZHONG66166 Aug 23, 2024
76fa0fd
update upload path
RUIJIEZHONG66166 Aug 23, 2024
1ecea60
add build log upload
RUIJIEZHONG66166 Aug 23, 2024
ead02a3
add build log upload
RUIJIEZHONG66166 Aug 23, 2024
b30feeb
update pti install
RUIJIEZHONG66166 Aug 27, 2024
bf5158b
update bundle install check and call
RUIJIEZHONG66166 Aug 27, 2024
82d6a59
add uninstall print
RUIJIEZHONG66166 Aug 27, 2024
06c4a56
add VS2022INSTALLDIR set
RUIJIEZHONG66166 Aug 28, 2024
628ab2c
update msvc call
RUIJIEZHONG66166 Aug 28, 2024
f00afcc
update msvc call
RUIJIEZHONG66166 Aug 28, 2024
d47060d
update '' to for msvc
RUIJIEZHONG66166 Aug 28, 2024
8b76c05
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Sep 6, 2024
ade7744
rebase windows build
RUIJIEZHONG66166 Sep 6, 2024
fc7f59d
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Sep 18, 2024
082f888
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Sep 29, 2024
4a0d432
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Oct 21, 2024
cc6dd56
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Oct 30, 2024
49afae7
update windows build
RUIJIEZHONG66166 Oct 30, 2024
c8e0dee
remove useless env
RUIJIEZHONG66166 Oct 30, 2024
c4bcbe3
modify the conda activate
RUIJIEZHONG66166 Oct 30, 2024
25b99f4
update runner name
RUIJIEZHONG66166 Oct 30, 2024
f6c29b6
add long path fix
RUIJIEZHONG66166 Oct 30, 2024
14df2fe
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Oct 30, 2024
69e2786
update windows runner
RUIJIEZHONG66166 Oct 30, 2024
cd75bdf
fix pip install issue
RUIJIEZHONG66166 Oct 30, 2024
ebd49a3
update pip install packages
RUIJIEZHONG66166 Oct 30, 2024
5148f50
try unuse path fix
RUIJIEZHONG66166 Oct 30, 2024
317a45e
fix conda install issue
RUIJIEZHONG66166 Oct 31, 2024
e9ab309
Merge branch 'main' into ruijie/add_windows_ci
xytintel Nov 1, 2024
158d050
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Nov 6, 2024
1997473
change runner name
RUIJIEZHONG66166 Nov 6, 2024
5af30c5
right runner label
RUIJIEZHONG66166 Nov 6, 2024
cee5641
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Nov 13, 2024
b36cbf1
set conda env path
RUIJIEZHONG66166 Nov 13, 2024
db1d5bb
set shell = cmd
RUIJIEZHONG66166 Nov 13, 2024
7fd8a58
update conda env path
RUIJIEZHONG66166 Nov 13, 2024
dd6e561
update conda env path
RUIJIEZHONG66166 Nov 13, 2024
dbdda8a
remove unuseless path set
RUIJIEZHONG66166 Nov 13, 2024
d45e420
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Nov 19, 2024
1d5c98b
Merge branch 'main' into ruijie/add_windows_ci
RUIJIEZHONG66166 Nov 28, 2024
7360a22
update miniforge path
RUIJIEZHONG66166 Nov 28, 2024
ab8e69e
set max_job and add extend ut test
RUIJIEZHONG66166 Nov 28, 2024
71ae54c
set MAX_JOBS=4
RUIJIEZHONG66166 Nov 29, 2024
1ae2f90
set MAX_JOBS=2
RUIJIEZHONG66166 Nov 29, 2024
8557ee2
set MAX_JOBS=1
RUIJIEZHONG66166 Dec 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 114 additions & 0 deletions .github/scripts/install_xpu.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
@echo on
REM Description: Install Intel Support Packages on Windows
REM BKM reference: https://www.intel.com/content/www/us/en/developer/articles/tool/pytorch-prerequisites-for-intel-gpus.html

set XPU_INSTALL_MODE=%~1
if "%XPU_INSTALL_MODE%"=="" goto xpu_bundle_install_start
if "%XPU_INSTALL_MODE%"=="bundle" goto xpu_bundle_install_start
if "%XPU_INSTALL_MODE%"=="driver" goto xpu_driver_install_start
if "%XPU_INSTALL_MODE%"=="all" goto xpu_driver_install_start

:arg_error

echo Illegal XPU installation mode. The value can be "bundle"/"driver"/"all"
echo If keep the value as space, will use default "bundle" mode
exit /b 1

:xpu_driver_install_start
:: TODO Need more testing for driver installation
set XPU_DRIVER_LINK=https://downloadmirror.intel.com/830975/gfx_win_101.5972.exe
curl -o xpu_driver.exe --retry 3 --retry-all-errors -k %XPU_DRIVER_LINK%
echo "XPU Driver installing..."
start /wait "Intel XPU Driver Installer" "xpu_driver.exe"
if errorlevel 1 exit /b 1
del xpu_driver.exe
if "%XPU_INSTALL_MODE%"=="driver" goto xpu_install_end

:xpu_bundle_install_start

set XPU_BUNDLE_PARENT_DIR=C:\Program Files (x86)\Intel\oneAPI
set XPU_BUNDLE_URL=https://registrationcenter-download.intel.com/akdlm/IRC_NAS/9d1a91e2-e8b8-40a5-8c7f-5db768a6a60c/w_intel-for-pytorch-gpu-dev_p_0.5.3.37_offline.exe
set XPU_BUNDLE_PRODUCT_NAME=intel.oneapi.win.intel-for-pytorch-gpu-dev.product
set XPU_BUNDLE_VERSION=0.5.3+31
set XPU_BUNDLE_INSTALLED=0
set XPU_BUNDLE_UNINSTALL=0
set XPU_EXTRA_URL=https://registrationcenter-download.intel.com/akdlm/IRC_NAS/9d1a91e2-e8b8-40a5-8c7f-5db768a6a60c/w_intel-pti-dev_p_0.9.0.37_offline.exe
set XPU_EXTRA_PRODUCT_NAME=intel.oneapi.win.intel-pti-dev.product
set XPU_EXTRA_VERSION=0.9.0+36
set XPU_EXTRA_INSTALLED=0
set XPU_EXTRA_UNINSTALL=0

if not [%XPU_VERSION%]==[] if [%XPU_VERSION%]==[2025.0] (
set XPU_BUNDLE_URL=https://registrationcenter-download.intel.com/akdlm/IRC_NAS/efc86abd-cb77-452e-a03f-a741895b8ece/intel-deep-learning-essentials-2025.0.0.336_offline.exe
set XPU_BUNDLE_PRODUCT_NAME=intel.oneapi.win.deep-learning-essentials.product
set XPU_BUNDLE_VERSION=2025.0.0+335
set XPU_BUNDLE_INSTALLED=0
set XPU_BUNDLE_UNINSTALL=0
set XPU_EXTRA_URL=NULL
set XPU_EXTRA_PRODUCT_NAME=intel.oneapi.win.compiler.product
set XPU_EXTRA_VERSION=2025.0.1+1226
set XPU_EXTRA_INSTALLED=0
set XPU_EXTRA_UNINSTALL=0
)

:: Check if XPU bundle is target version or already installed
if exist "%XPU_BUNDLE_PARENT_DIR%\Installer\installer.exe" goto xpu_bundle_ver_check
goto xpu_bundle_install

:xpu_bundle_ver_check

"%XPU_BUNDLE_PARENT_DIR%\Installer\installer.exe" --list-products > xpu_bundle_installed_ver.log

for /f "tokens=1,2" %%a in (xpu_bundle_installed_ver.log) do (
if "%%a"=="%XPU_BUNDLE_PRODUCT_NAME%" (
echo %%a Installed Version: %%b
set XPU_BUNDLE_INSTALLED=1
if not "%XPU_BUNDLE_VERSION%"=="%%b" (
start /wait "Installer Title" "%XPU_BUNDLE_PARENT_DIR%\Installer\installer.exe" --action=remove --eula=accept --silent --product-id %%a --product-ver %%b --log-dir uninstall_bundle
set XPU_BUNDLE_UNINSTALL=1
)
)
if "%%a"=="%XPU_EXTRA_PRODUCT_NAME%" (
echo %%a Installed Version: %%b
set XPU_EXTRA_INSTALLED=1
if not "%XPU_EXTRA_VERSION%"=="%%b" (
start /wait "Installer Title" "%XPU_BUNDLE_PARENT_DIR%\Installer\installer.exe" --action=remove --eula=accept --silent --product-id %%a --product-ver %%b --log-dir uninstall_bundle
set XPU_EXTRA_UNINSTALL=1
)
)
if not "%%b" == "Version" if not [%%b]==[] if not "%%a"=="%XPU_BUNDLE_PRODUCT_NAME%" if not "%%a"=="%XPU_EXTRA_PRODUCT_NAME%" (
echo "Uninstalling...."
start /wait "Installer Title" "%XPU_BUNDLE_PARENT_DIR%\Installer\installer.exe" --action=remove --eula=accept --silent --product-id %%a --product-ver %%b --log-dir uninstall_bundle
)
)
if errorlevel 1 exit /b 1
if exist xpu_bundle_installed_ver.log del xpu_bundle_installed_ver.log
if exist uninstall_bundle rmdir /s /q uninstall_bundle
if "%XPU_BUNDLE_INSTALLED%"=="0" goto xpu_bundle_install
if "%XPU_BUNDLE_UNINSTALL%"=="1" goto xpu_bundle_install

:xpu_extra_check

if "%XPU_EXTRA_URL%"=="NULL" goto xpu_install_end
if "%XPU_EXTRA_INSTALLED%"=="0" goto xpu_extra_install
if "%XPU_EXTRA_UNINSTALL%"=="1" goto xpu_extra_install
goto xpu_install_end

:xpu_bundle_install

curl -o xpu_bundle.exe --retry 3 --retry-all-errors -k %XPU_BUNDLE_URL%
echo "XPU Bundle installing..."
start /wait "Intel Pytorch Bundle Installer" "xpu_bundle.exe" --action=install --eula=accept --silent --log-dir install_bundle
if errorlevel 1 exit /b 1
del xpu_bundle.exe
goto xpu_extra_check

:xpu_extra_install

curl -o xpu_extra.exe --retry 3 --retry-all-errors -k %XPU_EXTRA_URL%
echo "Intel XPU EXTRA installing..."
start /wait "Intel XPU EXTRA Installer" "xpu_extra.exe" --action=install --eula=accept --silent --log-dir install_bundle
if errorlevel 1 exit /b 1
del xpu_extra.exe

:xpu_install_end
8 changes: 8 additions & 0 deletions .github/workflows/pull.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,14 @@ jobs:
abi: 0
ut: op_extended
runner: linux.idc.xpu

preci-windows-build:
# Don't run on forked repos and draft PRs
if: ${{ (github.repository_owner == 'intel') && (github.event.pull_request.draft == false) }}
uses: ./.github/workflows/windows.yml
with:
ut: op_extended
runner: Windows_CI

Inductor-XPU-E2E-CI-Tests:
runs-on: pvc_e2e
Expand Down
160 changes: 160 additions & 0 deletions .github/workflows/windows.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
name: Windows Test

on:
workflow_call:
inputs:
pytorch:
required: false
type: string
default: 'main'
description: Pytorch branch/commit
keep_torch_xpu_ops:
required: false
type: string
default: 'false'
description: Keep torch-xpu-ops pin. `true` means use pined commit
ut:
required: true
type: string
default: ''
description: UT scope. `op_regression,op_regression_dev1,op_extended,op_ut,torch_xpu` Delimiter is comma
abi:
required: false
type: string
default: 1
description: ABI version. Default abi as 1.
python:
required: false
type: string
default: '3.10'
description: Python version
runner:
required: true
type: string
default: 'linux.idc.xpu'
description: Runner label

permissions: read-all

env:
USE_XPU: 1

jobs:
Torch-XPU-Windows-Build:
runs-on: ${{ inputs.runner }}
timeout-minutes: 900
steps:
- name: Checkout torch-xpu-ops
uses: actions/checkout@v4
# - name: Install oneAPI bundle and pti
# shell: cmd
# run: |
# call .github\scripts\install_xpu.bat
# if errorlevel 1 exit /b 1
- name: Prepare Stock Pytorch
shell: cmd
run: |
echo "C:\ProgramData\miniforge3\Scripts" >> "$GITHUB_PATH"
echo "C:\ProgramData\miniforge3\Library\bin" >> "$GITHUB_PATH"
call "C:\ProgramData\miniforge3\Scripts\activate.bat"
call conda clean -ay
call conda remove --all -y -n windows_ci
call conda create -n windows_ci python=${{ inputs.python }} cmake ninja -y
call conda activate windows_ci
cd ..
if exist "pytorch" (
rmdir /s /q pytorch
)
git clone https://github.com/pytorch/pytorch pytorch
cd pytorch && git checkout ${{ inputs.pytorch }}
pip install pyyaml requests
call conda install -y libuv
git config --system core.longpaths true
git status
git show -s
git submodule sync && git submodule update --init --recursive
if ${{ inputs.keep_torch_xpu_ops }} == 'true' (
echo "Don't replace torch-xpu-ops!"
) else (
echo "Replace torch-xpu-ops!"
cd third_party
if exist "torch-xpu-ops" (
rmdir /s /q torch-xpu-ops
)
cd ..
Xcopy ..\torch-xpu-ops third_party\torch-xpu-ops /E/H/Y/F/I
)
- name: Build Pytorch XPU
shell: cmd
run: |
call "C:\ProgramData\miniforge3\Scripts\activate.bat"
call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
call "C:\Program Files (x86)\Intel\oneAPI\pytorch-gpu-dev-0.5\oneapi-vars.bat"
call "C:\Program Files (x86)\Intel\oneAPI\ocloc\2024.2\env\vars.bat"
call conda activate windows_ci
cd ../pytorch
pip install -r requirements.txt
pip install cmake setuptools==72.1.0 clang-format
set USE_KINETO=0
set CMAKE_SHARED_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_MODULE_LINKER_FLAGS=/FORCE:MULTIPLE
set CMAKE_EXE_LINKER_FLAGS=/FORCE:MULTIPLE
if ${{ inputs.abi }} == '0' (
set _GLIBCXX_USE_CXX11_ABI=0
echo "ABI=0"
) else (
set _GLIBCXX_USE_CXX11_ABI=1
echo "ABI=1"
)
python setup.py clean
set MAX_JOBS=1
python setup.py bdist_wheel > build_torch_wheel_log.log
for %%f in (${{ github.workspace }}\..\pytorch\dist\torch-*.whl) do (
set "TORCH_WHL=%%f"
echo "INFO - torch whl found"
)
if defined TORCH_WHL (
pip install %TORCH_WHL%
echo "INFO - torch whl installed"
) else (
echo "INFO - No torch whl found."
exit /b 1
)
pip install --force-reinstall dist/*.whl
pip install -r .ci/docker/requirements-ci.txt
- name: Torch Config
shell: cmd
run: |
call "C:\ProgramData\miniforge3\Scripts\activate.bat"
call conda activate windows_ci
call "C:\Program Files (x86)\Intel\oneAPI\pytorch-gpu-dev-0.5\oneapi-vars.bat"
call "C:\Program Files (x86)\Intel\oneAPI\ocloc\2024.2\env\vars.bat"
python -c "import torch; print(torch.__config__.show())"
python -c "import torch; print(torch.__config__.parallel_info())"
python -c "import torch; print(torch.__config__.torch.xpu.device_count())"

- name: Upload Windows build log
if: ${{ ! cancelled() }}
uses: actions/upload-artifact@v4
with:
name: Torch-XPU-Windows-Log-${{ github.event.pull_request.number || github.sha }}
path: 'C:\runner\actions-runner\_work\torch-xpu-ops\pytorch\build_torch_wheel_log.log'

- name: Upload Windows binary
if: ${{ ! cancelled() }}
uses: actions/upload-artifact@v4
with:
name: Torch-XPU-Windows-Binary-${{ github.event.pull_request.number || github.sha }}
path: 'C:\runner\actions-runner\_work\torch-xpu-ops\pytorch\dist'

- name: Run XPU OP Extended UT
if: contains(inputs.ut, 'op_extended') || github.event_name == 'schedule'
shell: cmd
run: |
call "C:\ProgramData\miniforge3\Scripts\activate.bat"
call conda activate windows_ci
call "C:\Program Files (x86)\Intel\oneAPI\pytorch-gpu-dev-0.5\oneapi-vars.bat"
call "C:\Program Files (x86)\Intel\oneAPI\ocloc\2024.2\env\vars.bat"
set PYTORCH_TEST_WITH_SLOW=1
cd ../pytorch/third_party/torch-xpu-ops/test/xpu/extended/
timeout 10000 python run_test_with_skip.py
Loading