🐛 Describe the bug

The latest Intel GPU Driver introduces breaking changes that may lead to torch.compile failure with the error message of Triton Error [ZE]: 0x78000011 on Windows.

For instance, one might fail in the following cases:

• Due to the renaming of imf function. It means that Triton can't find the former libdevice name, so it will raise a load failure. For example: Change __imf_powif -> __imf_pownf and __imf_powi -> __imf_pown intel/intel-xpu-backend-for-triton#3936
• There are errors when there is select_scatter, and it might fail on interacting with the slice op. For example: [Pytorch Upstream][Release 2.7][Regression] Triton Error [ZE]: 0x78000011 after update to latest public driver. intel/intel-xpu-backend-for-triton#3916

Currently, the Intel GPU driver is working on the process to fix the failure. During the fixing stage, it can be mitigated by rolling back to the prior version. Please see the final Solution section for details.

Error logs

Here is an example of the error message.

Traceback (most recent call last):
"C:\Users\user\AppData\Local\Temp\torchinductor_user\7n\c7nlelwrngcnizvcufhtqm7gds4sbaacn7qzxwal23rzxxt34efu.py", line 49, in <module>
    triton_poi_fused_0 = async_compile.triton('triton_poi_fused_0', '''
...
  File "F:\miniforge\envs\nightly\lib\site-packages\triton\compiler\compiler.py", line 422, in _init_handles
    self.module, self.function, self.n_regs, self.n_spills = driver.active.utils.load_binary(
  File "F:\miniforge\envs\nightly\lib\site-packages\triton\backends\intel\driver.py", line 208, in load_binary
    return self.shared_library.load_binary(args)
RuntimeError: Triton Error [ZE]: 0x78000011

Solution

A temporary solution is to roll back to the previous version of the driver.

Affected Driver

Only on Windows did we detect such an issue. The 32.0.101.6734 and 32.0.101.6737 are affected.

Rollback method

One could download the previous driver from Intel® Arc™ & Iris® Xe Graphics - Windows* page. One could use the previous version, like 32.0.101.6647 for example.

Please click on the "perform a clean installation" checkbox when installing.

Also, note that if you failed with other issues of Triton on XPU Windows, it might be because of the setting of LEVEL_ZERO_V1_SDK_PATH. The old driver will need to set this manually. Please refer to Windows.md for details.

Normally, it requires two additional steps:

1. Download level-zero-win-sdk-..*.zip from https://github.com/oneapi-src/level-zero/releases and extract it somewhere, like C:'\level_zero.
2. Set an environment variable. For example, if you are using PowerShell, use $env:LEVEL_ZERO_V1_SDK_PATH = "C:\level_zero".

Then you could run normally.

cc @gujinghui @EikanWang @fengyuan14 @guangyey