Skip to content

windows_support.md

Latest commit

 

History

History
478 lines (372 loc) · 30.6 KB

windows_support.md

File metadata and controls

478 lines (372 loc) · 30.6 KB

Windows Support

TheRock aims to support as many subprojects as possible on "native" Windows (as opposed to WSL 1 or WSL 2) using standard build tools like MSVC.

Warning

While Windows source builds of TheRock (including PyTorch!) are working for some expert developers, this support is relatively new and is not yet mature. There are several known issues and active development areas on the Windows platform support bringup. If you encounter any issues not yet represented there, please file an issue.

Supported subprojects

ROCm is composed of many subprojects, some of which are supported on Windows:

This table tracks current support status for each subproject in TheRock on Windows. Some subprojects may need extra patches to build within TheRock (on mainline, in open source, using MSVC, etc.).

Note

Many ROCm components are now developed inside super-repositories such as rocm-libraries and rocm-systems, while others remain standalone repositories. The table below lists the canonical upstream location for each component.

Component group Component Upstream Supported Notes
base aux-overlay therock ✅
base rocm-cmake standalone ✅
base rocm-core rocm-systems ✅
base rocm_smi_lib rocm-systems ❌ Unsupported
base rocprofiler-register rocm-systems ❌ Unsupported
base rocm-half standalone ✅
compiler amd-llvm llvm-project ✅ Limited runtimes
compiler amd-comgr llvm-project ✅
compiler hipcc llvm-project ✅
compiler hipify standalone ✅
core amdsmi rocm-systems ❌ Unsupported
core ROCR-Runtime rocm-systems ❌ Unsupported
core rocminfo rocm-systems ❌ Unsupported
core hipInfo (hip-tests) rocm-systems ✅
core clr rocm-systems 🟡 Needs a folder with prebuilt static libraries
debug-tools amd-dbgapi rocm-systems ❌ Unsupported
debug-tools rocr-debug-agent rocm-systems ❌ Unsupported
debug-tools rocgdb standalone ❌ Unsupported
profiler aqlprofile rocm-systems ❌ Unsupported
profiler rocprofiler-sdk rocm-systems ❌ Unsupported
profiler rocprofiler-compute rocm-systems ❌ Unsupported
profiler rocprofiler-systems rocm-systems ❌ Unsupported
comm-libs rccl rocm-systems ❌ Unsupported
media-libs rocDecode rocm-systems ❌ Linux only (requires VA-API / Mesa)
media-libs rocJPEG rocm-systems ❌ Linux only (requires VA-API / Mesa)
math-libs rocRAND rocm-libraries ✅
math-libs hipRAND rocm-libraries ✅
math-libs rocPRIM rocm-libraries ✅
math-libs hipCUB rocm-libraries ✅
math-libs rocThrust rocm-libraries ✅
math-libs rocFFT rocm-libraries ✅
math-libs hipFFT rocm-libraries ✅
math-libs (support) mxDataGenerator rocm-libraries ❌ Unsupported
math-libs (BLAS) hipBLAS-common rocm-libraries ✅
math-libs (BLAS) rocRoller rocm-libraries ❌ Unsupported
math-libs (BLAS) hipBLASLt rocm-libraries ✅
math-libs (BLAS) rocBLAS rocm-libraries ✅
math-libs (BLAS) rocSPARSE rocm-libraries ✅
math-libs (BLAS) hipSPARSE rocm-libraries ✅
math-libs (BLAS) hipSPARSELt rocm-libraries ❌ Unsupported
math-libs (BLAS) rocSOLVER rocm-libraries ✅
math-libs (BLAS) hipSOLVER rocm-libraries ✅
math-libs (BLAS) hipBLAS rocm-libraries ✅
math-libs rocWMMA rocm-libraries ✅
math-libs libhipcxx standalone ✅
ml-libs Composable Kernel rocm-libraries ✅
ml-libs MIOpen rocm-libraries ✅
ml-libs hipDNN rocm-libraries ✅
ml-libs MIOpen Provider rocm-libraries ✅
ml-libs MIOpen Legacy Plugin rocm-libraries ✅
ml-libs hipBLASLt Provider rocm-libraries ✅

Building TheRock from source

These instructions mostly mirror the instructions in the root README.md, with some extra Windows-specific callouts.

Validating your environment

Before diving into the full setup, you can run the environment validation script to check that all prerequisites are met:

.\build_tools\validate_windows_install.ps1

The script checks RAM, disk space, long path support, symlink capability, MSVC, CMake, Ninja, Git, Python, DVC, Strawberry Perl/gfortran, ccache, and git configuration. It is safe to re-run at any time.

Prerequisites

Set up your system

Install tools

Tip

These tools are available via package managers like winget on Windows:

winget install --id Microsoft.VisualStudio.2022.BuildTools --source winget --override "--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add
Microsoft.VisualStudio.Component.VC.CMake.Project --add Microsoft.VisualStudio.Component.VC.ATL --add
Microsoft.VisualStudio.Component.Windows11SDK.22621"
winget install --id Git.Git -e --source winget --custom "/o:PathOption=CmdTools"
winget install cmake -v 3.31.0
winget install ninja-build.ninja ccache python strawberryperl bloodrock.pkg-config-lite
winget install --id Iterative.DVC --silent --accept-source-agreements

If you prefer to install tools manually, you will need:

Warning

Prefer to install Python for the current user only and to a path without spaces like C:\Users\<username>\AppData\Local\Programs\Python\Python312.

Important tool settings

Important

Git should be configured with support for symlinks and long paths:

git config --global core.symlinks true
git config --global core.longpaths true

Important

After installing MSVC, use its 64 bit tools in your build environment.

  • If you use the command line, see Use the Microsoft C++ toolset from the command line. Typically this means either Start > x64 Native Tools Command Prompt for VS 2022 or running vcvars64.bat in an existing shell.
  • If you build from an editor like VSCode, CMake can discover the compiler among other "kits".
  • Our windows-base or windows-release presets in CMakePresets.json set the CMake compiler and linker for you.
  • You can also tell CMake to use MSVC's tools explicitly with -DCMAKE_C_COMPILER=cl.exe -DCMAKE_CXX_COMPILER=cl.exe -DCMAKE_LINKER=link.exe.

Set the locale

If the build system is a non-English system. Make sure to switch to utf-8.

chcp 65001

Clone and fetch sources

# Clone the repository
git clone https://github.com/ROCm/TheRock.git
cd TheRock

# Init python virtual environment and install python dependencies
python -m venv .venv
.venv\Scripts\Activate.bat
pip install -r requirements.txt

# Download submodules and apply patches
python ./build_tools/fetch_sources.py

Build configuration

Unsupported subprojects like RCCL are automatically disabled on Windows. See the instructions in the root README for other options you may want to set.

cmake -B build -GNinja . -DTHEROCK_AMDGPU_FAMILIES=gfx110X-all

# If iterating and wishing to cache, add these:
#  -DCMAKE_C_COMPILER_LAUNCHER=ccache \
#  -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
#  -DCMAKE_MSVC_DEBUG_INFORMATION_FORMAT=Embedded \

Tip

ccache does not support MSVC's /Zi flag which may be set by default when a project (e.g. LLVM) opts in to policy CMP0141. Setting -DCMAKE_MSVC_DEBUG_INFORMATION_FORMAT=Embedded instructs CMake to compile with /Z7 or equivalent, which is supported by ccache.

Tip

Ensure that MSVC is used by looking for lines like these in the logs:

-- The C compiler identification is MSVC 19.42.34436.0
-- The CXX compiler identification is MSVC 19.42.34436.0

If you see some other compiler there, refer to the MSVC setup instructions up in Important tool settings.

CMake build usage

cmake --build build --target therock-dist
cmake --build build --target therock-archives

This will start building using MSVC. Once the amd-llvm subproject is built, subprojects like the ROCm math libraries will be compiled using clang.exe and other tools from the amd-llvm toolchain.

When the builds complete, you should have a build of ROCm / the HIP SDK in build/dist/rocm/ and artifacts in build/artifacts. See the Build Artifacts guide for more information about the build outputs.

Building ROCm Python wheels

To build Python wheels, you will need an "artifacts" directory, either from a source build of therock-archives (see above) or by running the fetch_artifacts.py script to download artifacts from a CI run.

Once you have an artifacts directory, you can run the build_python_packages.py script.

Building PyTorch

PyTorch builds require Python wheels, either by building from source (see above) or by downloading from one of TheRock's release indices. See the instructions at external-builds/pytorch.

Run tests

Test builds can be enabled with -DBUILD_TESTING=ON (the default).

Some subproject tests have been validated on Windows, like rocPRIM:

ctest --test-dir build/math-libs/rocPRIM/dist/bin/rocprim --output-on-failure

Build troubleshooting and known issues

No such file or directory: (long path to a source file)

If the build looks for a source file with a long path that does not exist, ensure that you have long path support enabled in git (see Important tool settings), then re-run

python ./build_tools/fetch_sources.py

Once the git operations have run with the new setting, confirm that the missing files now exist.

error: directive requires gfx90a+

Errors like this indicate that the value of -DTHEROCK_AMDGPU_FAMILIES= or -DTHEROCK_AMDGPU_TARGETS= is currently unsupported by one or more libraries.

lld-link: m.lib does not exist

Since msvc 14+, m.lib has become an implicit dependency and should not be linked (it does not exist). TheRock does not link it, but cmake could decide on its own to try and link it because it found a libm.a in the path, coming from leftover installs on the build machine, like a w64devkit or msys2 install.

lld-link: error: duplicate symbol

Several developers have reported link errors in rocBLAS and rocSPARSE like

[rocSPARSE] lld-link: error: duplicate symbol: __hip_cuid_2f7b343d50a9613
[rocSPARSE] >>> defined at library/CMakeFiles/rocsparse.dir/src/level3/csrmm/row_split/csrmm_device_row_split_256_16_8_3_4.cpp.obj
[rocSPARSE] >>> defined at library/CMakeFiles/rocsparse.dir/src/level3/csrmm/row_split/csrmm_device_row_split_256_16_8_7_4.cpp.obj

[rocBLAS] lld-link: error: duplicate symbol: rocblas_stbsv_batched
[rocBLAS] >>> defined at library/src/CMakeFiles/rocblas.dir/blas2/rocblas_tbsv_batched.cpp.obj
[rocBLAS] >>> defined at library/src/CMakeFiles/rocblas.dir/handle.cpp.obj
[rocBLAS]
[rocBLAS] lld-link: error: duplicate symbol: rocblas_dtbsv_batched
[rocBLAS] >>> defined at library/src/CMakeFiles/rocblas.dir/blas2/rocblas_tbsv_batched.cpp.obj
[rocBLAS] >>> defined at library/src/CMakeFiles/rocblas.dir/handle.cpp.obj
[rocBLAS]
[rocBLAS] lld-link: error: duplicate symbol: __hip_cuid_a201499d9a86b1da

These have been worked around by disabling ccache.

pyYAML cannot be found by cmake

It is recommended to build TheRock using the .venv python3 virtual environment. The build steps explain that you must do a pip install -r requirements.txt that installs PyYAML in the venv so maybe this step was not done.

The problem can also be that you have another python install available in your path and that cmake chose to use it. Make sure to check which python points to the python in your .venv

gfortran cannot be found by cmake

If you have installed strawberry perl as recommended but gfortran cannot be found by cmake, you can create a FC environment variable pointing to where gfortran.exe is located.

Other notes

Platform support lessons, tips, and gotchas

Filesystem paths

  • Windows paths may have spaces in them, so properly quote paths
  • Windows paths may use \ instead of / as a component delimiter
    • The \ character is often an escape character in raw strings, so properly escape paths if passing them between systems
    • In Python, pathlib has good support for performing transformations on paths like joining or splitting components in a cross-platform way, but some outgoing uses still need to use .resolve(), .as_posix(), and other functions to resolve symlinks, convert slashes, etc.
    • In C++, <filesystem> has good support for performing transformations on paths like joining or splitting components in a cross-platform way. See ROCm/rocm-libraries#948 for example.
    • In CMake, take care when loading environment variables into strings and treating them as paths. Use https://cmake.org/cmake/help/latest/command/file.html#path-conversion and https://cmake.org/cmake/help/latest/command/cmake_path.html instead of working with strings directly, like so: 
      -set(HIP_PATH $ENV{HIP_PATH})
+file(TO_CMAKE_PATH "$ENV{HIP_PATH}" HIP_PATH)`
  • Windows paths (and commands) can have max length limitations, so avoid long file names
  • Windows paths do not allow some characters (https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file)

Other platform differences

Building CLR from partial sources

We are working on enabling flexible open source builds of https://github.com/ROCm/clr (notably for amdhip64_7.dll) on Windows. Historically this has been a closed source component due to the dependency on Platform Abstraction Library (PAL) and providing a fully open source build will take more time. As an incremental step towards a fully open source build, we are using an interop folder containing header files and static library .lib files for PAL and related components.

An incremental rollout is planned:

  1. The interop folder must be manually copied into place in the source tree. This will allow AMD developers to iterate on integration into TheRock while we work on making this folder or more source files available.
  2. The interop folder will be available publicly (currently at https://github.com/ROCm/rocm-systems/tree/develop/shared/amdgpu-windows-interop).
  3. (We are here today) The interop folder will be included automatically from a git repository using dvc.
  4. A more permanent open source strategy for building the CLR (the HIP runtime) from source on Windows will eventually be available.

If configured correctly, outputs like build/core/clr/dist/bin/amdhip64_6.dll should be generated by the build.

If the interop folder is not available, sub-project support is limited and features should be turned off:

-DTHEROCK_ENABLE_CORE=OFF \
-DTHEROCK_ENABLE_MATH_LIBS=OFF \
-DTHEROCK_ENABLE_ML_LIBS=OFF \