VAAPI Encoder (2nd branch/attempt) #7467

sgothel · 2025-11-29T17:59:37Z

This is the 2nd clean VAAPI branch tackling issue 1083 #1083
and superseding pull-request #3039

supported: h264, hevc, av1, untested: vp8 and vp9

h264, hevc works on AMD >= NAVI10
av1 works on AMD > NAVI10(?)
Bitrate either VBR (variable) or CQP (constant)
Using Mesa 25 / libva 2.22 (Debian 13),
vaapi supports writing required sequence- and global-packet.
Therefor, no more patch is required to write into a MKV container.

On Debian 12, Mesa 22, these packets are not written
and hence no MKV container can be used.

+++

Tested on:
Test environment:
- GNU/Linux, Debian 13
- vainfo: VA-API version: 1.22 (libva 2.22.0)
- vainfo: Driver version: Mesa Gallium driver 25.0.7-2 for AMD Radeon RX 5700 XT (radeonsi, navi10, LLVM 19.1.7, DRM 3.61, 6.12.57+deb13-amd64)

libhb/handbrake/common.h

libhb/vaapi_common.c

bradleysepos · 2025-11-29T19:36:42Z

Thank you for this, looks pretty clean. Looking forward to again setting up Linux on my AMD machine to test (not a blocker).

contrib/ffmpeg/A23-avformat-matroskaenc-tolerate-no-global-header.patch

libhb/encavcodecaudio.c

libhb/encavcodec.c

libhb/handbrake/common.h

libhb/hb.c

…al_verbosity_level >= 3

sgothel · 2025-11-30T18:04:55Z

I have force pushed this branch, addressed some comments.

rathann · 2025-11-30T20:00:46Z

This is the 2nd clean VAAPI branch tackling issue 1083 #1083 and superseding pull-request #3039

Thanks for keeping at it!

h264 works on AMD >= NAVI10, unsupported/untested: h265, vp8 and vp9

I've got a NAVI22, which supports h264 and h265 encoding. I'll try to test this soon.

* Bitrate either VBR (variable) or CQP (constant)
This approach works w/o patched Mesa gallium va frontend, instead it uses a ffmpeg/avformat/matrokaenc patch allowing h264 streams w/o sequence nor global header. See https://trac.ffmpeg.org/ticket/10667.

FWIW, the current code in mesa looks like this:

         case VAConfigAttribEncPackedHeaders:
            value = VA_ENC_PACKED_HEADER_NONE;
            if ((u_reduce_video_profile(ProfileToPipe(profile)) == PIPE_VIDEO_FORMAT_MPEG4_AVC))
               value |= ENC_PACKED_HEADERS_H264;
            else if ((u_reduce_video_profile(ProfileToPipe(profile)) == PIPE_VIDEO_FORMAT_HEVC))
               value |= ENC_PACKED_HEADERS_HEVC;
            else if (u_reduce_video_profile(ProfileToPipe(profile)) == PIPE_VIDEO_FORMAT_AV1)
               value |= ENC_PACKED_HEADERS_AV1;

and the ENC_PACKED_HEADERS_* constants referenced above do include VA_ENC_PACKED_HEADER_SEQUENCE | VA_ENC_PACKED_HEADER_SLICE | VA_ENC_PACKED_HEADER_MISC.

To me, this means mesa doesn't need to be patched anymore, right?

sgothel · 2025-12-01T06:11:11Z

Thanks for keeping at it!
.. and thank you for keep looking at it too.

I've got a NAVI22, which supports h264 and h265 encoding. I'll try to test this soon.
Great. Thx.

FWIW, the current code in mesa looks like this:

         case VAConfigAttribEncPackedHeaders:
            value = VA_ENC_PACKED_HEADER_NONE;
            if ((u_reduce_video_profile(ProfileToPipe(profile)) == PIPE_VIDEO_FORMAT_MPEG4_AVC))
               value |= ENC_PACKED_HEADERS_H264;
            else if ((u_reduce_video_profile(ProfileToPipe(profile)) == PIPE_VIDEO_FORMAT_HEVC))
               value |= ENC_PACKED_HEADERS_HEVC;
            else if (u_reduce_video_profile(ProfileToPipe(profile)) == PIPE_VIDEO_FORMAT_AV1)
               value |= ENC_PACKED_HEADERS_AV1;

and the ENC_PACKED_HEADERS_* constants referenced above do include VA_ENC_PACKED_HEADER_SEQUENCE | VA_ENC_PACKED_HEADER_SLICE | VA_ENC_PACKED_HEADER_MISC.

To me, this means mesa doesn't need to be patched anymore, right?

Yes, looks like!

Mesa commit https://gitlab.freedesktop.org/mesa/mesa/-/commit/fc4abbe27d063337a420d147cf8c9fa492789f71,
not sure which release that is (from July 2024).

So we may need to unblind vaapi in ffmpeg public API, allowing to query these values?
At least w/ my (old) Debian 12.12 version of my test machine this change didn't sooth ffmpeg yet,
still claiming no packet headers produced. Will have a check here.

sgothel · 2025-12-04T09:53:52Z

Force pushed w/ added av1_vaapi support (untested due to lack of testing hardware, like hevc_vaapi)

Mastergatto · 2025-12-06T13:25:09Z

Hi, I own a Radeon RX 9070 XT so I'm able to test this PR with H.264, HEVC and AV1.

First of all, here is what I get with vainfo on Linux:

Trying display: wayland
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Mesa Gallium driver 25.2.7-arch1.1 for AMD Radeon RX 9070 XT (radeonsi, gfx1201, LLVM 21.1.5, DRM 3.64, 6.17.9-arch1-1)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointEncSlice
      VAProfileNone                   :	VAEntrypointVideoProc

As you can see, VAAPI works on my system and my GPU can encode with H.264, HEVC and AV1, but not with VP9.

The command line used for this test is: HandBrakeCLI -i 'video.ogg' -o '<codec>.mkv' -e <encoder parameter> where video.ogg is a sample video with a length of 8 seconds, is codified with Ogg/Theora and has a framerate of 25 fps. File size is 2,9 MB. Here's what I have found by trying to transcode it to H.264, HEVC and AV1.

H.264:
Despite it saying the encode has failed, it has converted the video although the result is so-so: the new video is effectively a slideshow going at 8 fps, and its size is reduced to 773 kB which doesn't make much sense.
https://pastebin.com/3KV5W2aY

HEVC:
The conversion is successful, the result is really good. The video size is reduced to 1,3 MB.
https://pastebin.com/BrgeStK1

AV1:
The conversion has been a failure. And to nitpick regarding the parameter name for flag -e (for Handbrake's CLI), for transcoding with AV1 through VAAPI should be 'vaapi_av1' not 'vaapi_avi1'.
https://pastebin.com/1y3St0gV

As a control test, I have also tried to transcode with ffmpeg, using VAAPI, and it has worked flawlessly for each codec.

sgothel · 2025-12-06T20:03:32Z

Thank you!

for transcoding with AV1 through VAAPI should be 'vaapi_av1' not 'vaapi_avi1'
fixed typo

Hi, I own a Radeon RX 9070 XT so I'm able to test this PR with H.264, HEVC and AV1.
Great! I just figured I also have at least HEVC encoding available, just not AV1.

First of all, here is what I get with vainfo on Linux:
Trying display: wayland
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Mesa Gallium driver 25.2.7-arch1.1 for AMD Radeon RX 9070 XT (radeonsi, gfx1201, LLVM 21.1.5, DRM 3.64, 6.17.9-arch1-1)
vainfo: Supported profile and entrypoints
As you can see, VAAPI works on my system and my GPU can encode with H.264, HEVC and AV1, but not with VP9.

Yup!

The command line used for this test is: HandBrakeCLI -i 'video.ogg' -o '<codec>.mkv' -e <encoder parameter> where video.ogg is a sample video with a length of 8 seconds, is codified with Ogg/Theora and has a framerate of 25 fps. File size is 2,9 MB. Here's what I have found by trying to transcode it to H.264, HEVC and AV1.

H.264: Despite it saying the encode has failed, it has converted the video although the result is so-so: the new video is effectively a slideshow going at 8 fps, and its size is reduced to 773 kB which doesn't make much sense. https://pastebin.com/3KV5W2aY

I could reproduce this here also with ffmpeg transcoding, stuttering/jumping frames from theora,
but not from like AV1 or H264.

My test script:

LOGL="-loglevel 40"

ffmpeg -y ${LOGL} -ss 05:00 -vaapi_device /dev/dri/renderD128 -i m0.mkv -t 00:20 -vf format=nv12,hwupload -c:v h264_vaapi -c:a copy m0_h264_vaapi.mp4 2>&1 | tee m0_h264_vaapi.log
ffmpeg -y ${LOGL} -ss 05:00 -vaapi_device /dev/dri/renderD128 -i m0.mkv -t 00:20 -vf format=nv12,hwupload -c:v hevc_vaapi -c:a copy m0_hevc_vaapi.mp4 2>&1 | tee m0_hevc_vaapi.log
ffmpeg -y ${LOGL} -ss 05:00 -vaapi_device /dev/dri/renderD128 -i m0.mkv -t 00:20 -vf format=nv12,hwupload -c:v av1_vaapi -c:a copy m0_av1_vaapi.mp4 2>&1 | tee m0_av1_vaapi.log
ffmpeg -y ${LOGL} -ss 05:00 -i m0.mkv -t 00:20 -c:v libtheora -c:a opus -strict -2 m0_theora.ogg 2>&1 | tee m0_theora.log
ffmpeg -y ${LOGL} -ss 05:00 -i m0.mkv -t 00:20 -an -c:v libtheora m0_theora.aonly.ogg 2>&1 | tee m0_theora.aonly.log

IN_AV=m0_theora.ogg
IN_V=m0_theora.aonly.ogg

ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN_AV} -vf format=nv12,hwupload -c:v h264_vaapi -c:a copy m0_theora_h264_vaapi.av.mp4 2>&1 | tee m0_theora_h264_vaapi.av.log
ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN_V} -vf format=nv12,hwupload -c:v h264_vaapi m0_theora_h264_vaapi.v.mp4 2>&1 | tee m0_theora_h264_vaapi.v.log

The last two m0_theora_h264_vaapi.av.mp4 and m0_theora_h264_vaapi.v.mp4 show the bug.
Same w/ hevc/h265 vaapi from theora.

Notable, I use the same ffmpeg 8.0.1 version (manual build) as our current Handbrake build
commit 894da5ca7d742e4429ffb2af534fcda0103ef593 (HEAD -> b_8_0_1, tag: n8.0.1)

ffmpeg version n8.0.1 Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14+deb12u1)
  configuration: --enable-gpl --enable-shared --enable-runtime-cpudetect --prefix=/opt-linux-x86_64/ffmpeg-8.0.1 --enable-libmp3lame --enable-libtwolame --enable-libx264 --enable-libx265 --enable-libsvtav1 --enable-libvpx --enable-libtheora --enable-encoder=h264_nvenc --enable-encoder=hevc_nvenc --enable-encoder=h264_vaapi --enable-encoder=hevc_vaapi --enable-encoder=av1_vaapi --enable-encoder=vp8_vaapi --enable-encoder=vp9_vaapi
  libavutil      60.  8.100 / 60.  8.100
  libavcodec     62. 11.100 / 62. 11.100
  libavformat    62.  3.100 / 62.  3.100
  libavdevice    62.  1.100 / 62.  1.100
  libavfilter    11.  4.100 / 11.  4.100
  libswscale      9.  1.100 /  9.  1.100
  libswresample   6.  1.100 /  6.  1.100

This is potentially an ffmpeg issue ...

HEVC: The conversion is successful, the result is really good. The video size is reduced to 1,3 MB. https://pastebin.com/BrgeStK1

AV1: The conversion has been a failure. And to nitpick regarding the parameter name for flag -e (for Handbrake's CLI), for transcoding with AV1 through VAAPI should be 'vaapi_av1' not 'vaapi_avi1'. https://pastebin.com/1y3St0gV

As a control test, I have also tried to transcode with ffmpeg, using VAAPI, and it has worked flawlessly for each codec.

The HEVC + AV1 detection is also fixed, I was using h264 only parameter in hb_avcodec_test_encoder.
Thank you for pointing this out.

Will force push soon after some additional tests.

Mastergatto · 2025-12-06T20:06:08Z

With the proposed changes above, encode with av1 through VAAPI now works and with good results, though it still complains about missing profile:

[av1_vaapi @ 0x5621d0bfed80] No usable encoding profile found.
[20:40:58] hb_avcodec_test_encoder_available encoder=0x40018082: err -3

And a warning:
[20:40:58] encavcodecInit: Unknown avcodec option q

Also, while with CLI every codec works, but in the GUI only H.264 (vaapi) is available.

Mastergatto · 2025-12-06T20:16:51Z

This is potentially an ffmpeg issue ...

Not sure about the bug being in ffmpeg. I use the same version, 8.0.1.

In my case transcoding directly with ffmpeg was flawless with each codec, it was done with a command line like this, e.g. for H.264: ffmpeg -vaapi_device /dev/dri/renderD128 -i video.ogg -vf 'format=nv12,hwupload' -c:v h264_vaapi h264.mkv

sgothel · 2025-12-06T21:51:35Z

@Mastergatto OK pushed revision w/ our noted changes

Below my test scripts.

Tested w/ below scripts and a proper theora test file (my bad),
no stuttering etc.

Couldn't test AV1, due to hardware - but h264 + hevc/h265 works OK w/ vaapi here.

If you test w/ HB, please ensure the --verbose=4 commandline,
which will enable ffmpeg verbose logging of the vaapi core.

+++

cat ffmpeg_test.sh

#!/bin/sh

LOGL="-loglevel 40"
IN=m1_h264.mp4

ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN} -vf format=nv12,hwupload -c:v h264_vaapi -c:a copy m1_h264_vaapi.mp4 2>&1 | tee m1_h264_vaapi.log
ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN} -vf format=nv12,hwupload -c:v hevc_vaapi -c:a copy m1_hevc_vaapi.mp4 2>&1 | tee m1_hevc_vaapi.log
ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN} -vf format=nv12,hwupload -c:v av1_vaapi -c:a copy m1_av1_vaapi.mp4 2>&1 | tee m1_av1_vaapi.log

ffmpeg -y -i ${IN} -c:v libtheora -r 25 -qscale:v 10 -c:a opus -strict -2 m1_theora.ogg 2>&1 | tee m1_theora.log
ffmpeg -y -i ${IN} -an -c:v libtheora -r 25 -qscale:v 10 -strict -2 m1_theora.v.ogg 2>&1 | tee m1_theora.v.log

IN_TAV=m1_theora.ogg
IN_TV=m1_theora.v.ogg

ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN_TAV} -vf format=nv12,hwupload -c:v h264_vaapi -c:a copy m1_theora_h264_vaapi.av.mp4 2>&1 | tee m1_theora_h264_vaapi.av.log
ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN_TV} -vf format=nv12,hwupload -c:v h264_vaapi m1_theora_h264_vaapi.v.mp4 2>&1 | tee m1_theora_h264_vaapi.v.log

ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN_TAV} -vf format=nv12,hwupload -c:v hevc_vaapi -c:a copy m1_theora_h265_vaapi.av.mp4 2>&1 | tee m1_theora_h265_vaapi.av.log
ffmpeg -y ${LOGL} -vaapi_device /dev/dri/renderD128 -i ${IN_TV} -vf format=nv12,hwupload -c:v hevc_vaapi m1_theora_h265_vaapi.v.mp4 2>&1 | tee m1_theora_h265_vaapi.v.log

cat hb_test.sh

#!/bin/sh

IN=m1_h264.mp4
IN_TAV=m1_theora.ogg
IN_TV=m1_theora.v.ogg

HB=/opt-linux-x86_64/HandBreak/bin/HandBrakeCLI
HB_OPTS="--verbose=4 --no-dvdnav"

${HB} ${HB_OPTS} -i ${IN} -o 'hb_x264_to_h264.mkv' -e vaapi_h264 2>&1 | tee hb_x264_to_h264.log
${HB} ${HB_OPTS} -i ${IN} -o 'hb_x264_to_h265.mkv' -e vaapi_hevc 2>&1 | tee hb_x264_to_h265.log

${HB} ${HB_OPTS} -i ${IN_TAV} -o 'hb_theora_to_h264.av.mkv' -e vaapi_h264 2>&1 | tee hb_theora_to_h264.av.log
${HB} ${HB_OPTS} -i ${IN_TAV} -o 'hb_theora_to_h265.av.mkv' -e vaapi_hevc 2>&1 | tee hb_theora_to_h265.av.log

${HB} ${HB_OPTS} -i ${IN_TV} -o 'hb_theora_to_h264.v.mkv' -e vaapi_h264 2>&1 | tee hb_theora_to_h264.v.log
${HB} ${HB_OPTS} -i ${IN_TV} -o 'hb_theora_to_h265.v.mkv' -e vaapi_hevc 2>&1 | tee hb_theora_to_h265.v.log

sgothel · 2025-12-06T21:56:55Z

With the proposed changes above, encode with av1 through VAAPI now works and with good results, though it still complains about missing profile:
[av1_vaapi @ 0x5621d0bfed80] No usable encoding profile found.
[20:40:58] hb_avcodec_test_encoder_available encoder=0x40018082: err -3
And a warning: [20:40:58] encavcodecInit: Unknown avcodec option q

Also, while with CLI every codec works, but in the GUI only H.264 (vaapi) is available.

I revised the hb_avcodec_test_encoder and added av1_vaapi encoder in the ffmpeg module w/ last forced push,
should be visible in UI and working.

Mastergatto · 2025-12-06T23:04:37Z

H.264, still not good:
https://pastebin.com/M9PbtSH5
I have tried to convert other videos (Theora, AV1 and HEVC) to H.264, still the same result. I have also tried with a much longer video, Handbrake only converted the first 30 seconds and skipped the rest.

For what's worth, I have also tested AV1 - OK:
https://pastebin.com/npZ1gvjS

And HEVC - OK:
https://pastebin.com/akN9UGYY

For GUI, now all three VAAPI encoders are available in the menu, and they produce same result as the CLI.

Thank you for your hard work!

sgothel · 2025-12-06T23:25:27Z

available

H.264, still not good: https://pastebin.com/M9PbtSH5 I have tried to convert other videos (Theora, AV1 and HEVC) to H.264, still the same result. I have also tried with a much longer video, Handbrake only converted the first 30 seconds and skipped the rest.

Hmm, weird .. as it works for me :|

What I see is this in your log

[matroska @ 0x7f6445e1ed00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 120 >= 40
ERROR: avformatMux: track 0, av_interleaved_write_frame failed with error 'Invalid argument'
[matroska @ 0x7f6445e1ed00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 120 >= 80

So this may need some more elaboration, could be due to
our different Mesa/vaapi implementation or hardware.

+++

Notable, your new Mesa/vaapi produces the packed header (our dire debate above), good.
(see last line - shows the packet header extradata size by vaapi)

[23:35:11] encavcodec: avcodec_open options: rc_mode=CQP,qp=22,b_depth=2
[h264_vaapi @ 0x7f6445cf82c0] Input surface format is nv12.
[h264_vaapi @ 0x7f6445cf82c0] Using VAAPI profile VAProfileH264High (7).
[h264_vaapi @ 0x7f6445cf82c0] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[h264_vaapi @ 0x7f6445cf82c0] Using VAAPI render target format YUV420 (0x1).
[h264_vaapi @ 0x7f6445cf82c0] RC mode: CQP.
[h264_vaapi @ 0x7f6445cf82c0] Block Level bitrate control: OFF.
[h264_vaapi @ 0x7f6445cf82c0] RC quality: 22.
[h264_vaapi @ 0x7f6445cf82c0] RC framerate: 25/1 (25.00 fps).
[h264_vaapi @ 0x7f6445cf82c0] Driver does not report any additional prediction constraints.
[h264_vaapi @ 0x7f6445cf82c0] Using intra, P- and B-frames (supported references: 1 / 1).
[h264_vaapi @ 0x7f6445cf82c0] All wanted packed headers available (wanted 0xd, found 0x1f).
[23:35:11] Writing Metadata to output file...
[matroska @ 0x7f6445e1ed00] mkv_assemble: h264: extradata_size 46

For what's worth, I have also tested AV1 - OK: https://pastebin.com/npZ1gvjS

And HEVC - OK: https://pastebin.com/akN9UGYY

For GUI, now all three VAAPI encoders are available in the menu, and they produce same result as the CLI.

Great.

Thank you for your hard work!

Thank you!

sgothel · 2025-12-08T07:30:21Z

force pushed changing inline function hb_vcodec_is_vaapi to regular function hb_video_encoder_is_vaapi, as required to build w/ debugging options --debug=std --optimize=none.

sgothel · 2025-12-11T03:47:21Z

Forced pushed w/o sub-optimal ffmpeg patch:

Using Mesa 25 / libva 2.22 (Debian 13),
vaapi supports writing required sequence- and global-packet.
Therefor, no more patch is required to write into a MKV container.

On Debian 12, Mesa 22, these packets are not written
and hence no MKV container can be used.

Edit:

test machine vainfo: Driver version: Mesa Gallium driver 25.0.7-2 for AMD Radeon RX 5700 XT (radeonsi, navi10, LLVM 19.1.7, DRM 3.61, 6.12.57+deb13-amd64)
on same machine w/ same test file: fps ~177 -> ~350 w/ vaapi h264 or h265
h264 conversion is not corrupted (see earlier discussion w/ @Mastergatto VAAPI Encoder (2nd branch/attempt) #7467 (comment))

galad87

Please use the same coding style as the rest of libhb:

blah *foo;
if (foo)
{
printf("hello");
}

libhb/hb.c

libhb/vaapi_common.c

libhb/encavcodec.c

supported: h264, hevc, av1, untested: vp8 and vp9 - h264, hevc works on AMD >= NAVI10 - av1 works on AMD > NAVI10(?) - Bitrate either VBR (variable) or CQP (constant) - Using Mesa 25 / libva 2.22 (Debian 13), vaapi supports writing required sequence- and global-packet. Therefor, no more patch is required to write into a MKV container. On Debian 12, Mesa 22, these packets are not written and hence no MKV container can be used. +++ Test environment: - GNU/Linux, Debian 13 - vainfo: VA-API version: 1.22 (libva 2.22.0) - vainfo: Driver version: Mesa Gallium driver 25.0.7-2 for AMD Radeon RX 5700 XT (radeonsi, navi10, LLVM 19.1.7, DRM 3.61, 6.12.57+deb13-amd64) Codev h264_vaapi, avcodec_open options: rc_mode=CQP,qp=24,b_depth=2,profile=100,level=40 (profile high, level 4.0) HD1080 source ~45min with 212-330fps PSNR and SSIM difference to x264 below 1%, having x264 using profile high, level 4.0, preset fast, rc 23. Both using constant-quality w/ variable bitrate. ++++ If FEATURE vaapi is enable (autodetected), the following libraries are linked: X11 va va-drm va-x11. - hb.c: - Adding global static 'vaapi_device_ctx0' default device initialized via hb_avcodec_init() and released via hb_avcodec_free(). The 'vaapi_device_ctx0' is used in vaa_common hb_avcodec_vaapi_set_hwframe_ctx(..). - hb_avcodec_vaapi_set_hwframe_ctx() attaches the new AVHWFramesContext to the AVCodecContext at initializing the encoder. This is the final sink for the vaapi hw-encoder and Encode(..) performs the frame conversion into it. - hb_avcodec_test_encoder(..) adds VAAPI branch, needs refinement - encavcodec.c - adding VAAPI branches, enabling available, needs refinement. Currently profile and level are set to the user preferences, additionally 'b_depth=2' is being passed. - added hb_avcodec_test_encoder*(..) functions, currently only used to check vaapi-*codec*-availability. We used this codefragment regularly in an earlier HB version. +++ Discussion: VAAPI Workflow: - It uses hardware frames on the target device, which are being transported from the software device. The frame transport implicitly converts the pix_fmt. - AVCodecContext's pix_fmt uses AV_PIX_FMT_VAAPI - AVHWFramesContext: format uses AV_PIX_FMT_VAAPI and sw_format AV_PIX_FMT_NV12, the latter hinting on the actual hw-frame's target format. - AV_PIX_FMT_NV12 uses interleaved UV data, where AV_PIX_FMT_YUV420P uses seperated planes. Both use a separated Y plane upfront. Therefor both formats are not picture compatible, memory requirements are same. - Encode(..) allocates a hw_frame and the source frame is being transported to the hw-frame target using the pic_fmt conversion to NV12. Finally the hw_frame is being sent 'avcodec_send_frame' and the code-path re-aligns with non VAAPI. Further fixes to do: Validate whether 'b_depth=2' (b-frames) works for all vaapi implementations, add custom extra-video-encoder field.

sgothel · 2025-12-19T21:16:48Z

Changes:

See VAAPI Encoder (2nd branch/attempt) #7467 (comment)
Also moved
- hb_vaapi_init into hb_common_global_hw_init
- hb_vaapi_free into new hb_common_global_hw_close

Selection of vaapi device has not been addressed here yet,
perhaps a detail for a followup patch?
This could involve a potential UI change to select and exceeds my time right now.
Edit: Alternatively, we could use an environment variable to optionally select a device path.

HexStan · 2025-12-21T08:43:33Z

Hi, thanks for your hard work! I tried converting a few videos to HEVC, but they all glitched out. My GPU is the Vega 8 integrated into the R7-5800H. It features VCN 2.2, which is higher than the VCN 2.0 found in Navi 10, so I believe it should fully meet the hardware requirements.

https://pastebin.com/bqE7jZ7m

https://pastebin.com/4tmAj46U

720p x265-vaapi qp29 opus mp4_20251221_163129 080

AVC works fine.

sgothel · 2025-12-21T18:21:30Z

Hi, thanks for your hard work! I tried converting a few videos to HEVC, but they all glitched out. My GPU is the Vega 8 integrated into the R7-5800H. It features VCN 2.2, which is higher than the VCN 2.0 found in Navi 10, so I believe it should fully meet the hardware requirements.

https://pastebin.com/bqE7jZ7m

https://pastebin.com/4tmAj46U

Please

run w/ environment variable VERBOSE=99 or -loglevel 4 (or higher), showing us ffmpeg verbose + debug messages
show your vainfo output.
also see VAAPI Encoder (2nd branch/attempt) #7467 (comment) (*) where one user also showed similar issue with a higher version Mesa driver and AMD GPU.
I suspect some issues w/ ffmpeg, mesa and/or our encoding configuration
- try uncommenting libhb/encavcodec.c line 1023 av_dict_set(&av_opts, "b_depth", "2", 0); and report

AVC works fine.

Setup 1 (mine)

- GNU/Linux, Debian 13
- vainfo: VA-API version: 1.22 (libva 2.22.0)
- vainfo: Driver version: Mesa Gallium driver 25.0.7-2 for AMD Radeon RX 5700 XT (radeonsi, navi10, LLVM 19.1.7, DRM 3.61, 6.12.57+deb13-amd64)

- H264: OK
- HEVC: OK
- AV1: not available

Setup 2(*)

VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Mesa Gallium driver 25.2.7-arch1.1 for AMD Radeon RX 9070 XT (radeonsi, gfx1201, LLVM 21.1.5, DRM 3.64, 6.17.9-arch1-1)

- H264: buggy
- HEVC: OK
- AV1: OK

HexStan · 2025-12-21T20:12:40Z

Please

run w/ environment variable VERBOSE=99 or -loglevel 4 (or higher), showing us ffmpeg verbose + debug messages

show your vainfo output.

also see VAAPI Encoder (2nd branch/attempt) #7467 (comment) (*) where one user also showed similar issue with a higher version Mesa driver and AMD GPU.

I suspect some issues w/ ffmpeg, mesa and/or our encoding configuration

try uncommenting libhb/encavcodec.c line 1023 av_dict_set(&av_opts, "b_depth", "2", 0); and report

Here is my vainfo output:

libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_20
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.20 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 25.0.7-0ubuntu0.24.04.2 for AMD Radeon Graphics (radeonsi, renoir, ACO, DRM 3.57, 6.8.0-90-generic)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

Regarding libhb/encavcodec.c, I checked the source code and noticed that the line av_dict_set(&av_opts, "b_depth", "2", 0); was actually not commented out in the original code.

Here are the logs running with the original code (where b_depth is active):
https://pastebin.com/aHVm2XFQ

I tried commenting out av_dict_set(&av_opts, "b_depth", "2", 0); to test, but the output video is still glitchy. Visually, the glitches appear exactly the same as before, and the bitrate is nearly identical. Here are the logs for this test:
https://pastebin.com/90FF3yRV

On the other hand, encoding to AVC and HEVC directly using the ffmpeg CLI works perfectly fine. Here are the commands and logs for that:
https://pastebin.com/WMvfviA3

sgothel · 2025-12-21T21:47:44Z

Regarding libhb/encavcodec.c, I checked the source code and noticed that the line av_dict_set(&av_opts, "b_depth", "2", 0); was actually not commented out in the original code.

My bad, yes - comment out, my cold makes me a bit fuzzy :)
(So no change here, good to know - thx)

Thank you for testing.

+++

Tricky .. You and the other tester use a more up-to-date GPU than me.
Your Ubuntu version is almost on par w/ mine, while the other one is more up-to-date w/ arch.

As you both wrote, the build in vanilla ffmpeg 6.1.1-3ubuntu5 produces flawless conversion,
our HB setup w/ 8.0.1 (edited) does not.

coding error on my behalf only impacting newer cards (sync?)
- the Encode block is pretty straight forward though?
ffmpeg version + GPU version + GPU firmware issue?

HexStan · 2025-12-21T22:42:00Z

My bad, yes - comment out, my cold makes me a bit fuzzy :) (So no change here, good to know - thx)

Thank you for testing.

+++

Tricky .. You and the other tester use a more up-to-date GPU than me. Your Ubuntu version is almost on par w/ mine, while the other one is more up-to-date w/ arch.

As you both wrote, the build in vanilla ffmpeg 6.1.1-3ubuntu5 produces flawless conversion, our HB setup w/ 8.0.1 (edited) does not.

coding error on my behalf only impacting newer cards (sync?)

the Encode block is pretty straight forward though?

ffmpeg version + GPU version + GPU firmware issue?

Get well soon! I actually just recovered from a cold myself, so I know the feeling.

I just tried using a static build of FFmpeg 8.0.1 to see if I could reproduce the issue outside of HandBrake. However, I encountered a symbol resolution error:
implib-gen: libva.so.2: failed to resolve symbol 'vaMapBuffer2' via dlsym: /lib/x86_64-linux-gnu/libva.so.2: undefined symbol: vaMapBuffer2

Instead of the visual glitches I saw earlier, this resulted in a 0kb output mp4 file.

This might be unrelated to your current investigation (likely just a library version mismatch on my system), but I'm sharing the logs just in case they are useful to you:

https://pastebin.com/ZCVx0imf

HexStan · 2025-12-24T12:03:13Z

Tricky .. You and the other tester use a more up-to-date GPU than me. Your Ubuntu version is almost on par w/ mine, while the other one is more up-to-date w/ arch.

As you both wrote, the build in vanilla ffmpeg 6.1.1-3ubuntu5 produces flawless conversion, our HB setup w/ 8.0.1 (edited) does not.

coding error on my behalf only impacting newer cards (sync?)

the Encode block is pretty straight forward though?

ffmpeg version + GPU version + GPU firmware issue?

Uh oh, new finding: outputting to .mkv seems to work fine.

sgothel · 2025-12-25T23:14:53Z

Below is a summary of our above tests, including @HexStan mkv note #7467 (comment)
HBIssues-va_api.md

Perhaps we can merge this PR as-is and create a followup
to address the remaining issues w/ below test machines
regarding h264 and mp4 container?

Test Machine Setup

Machine 1 (author)
- Mesa Gallium driver 25.0.7-2
  - LLVM 19.1.7
  - DRM 3.61
- Navi10
  - AMD Radeon RX 5700 XT
- Linux kernel 6.12.57+deb13-amd64
Machine 2 (mastergatto)
- Mesa Gallium driver 25.2.7-arch1.1
  - LLVM 21.1.5
  - DRM 3.64
- gfx1201
  - AMD Radeon RX 9070 XT
- Linux kernel 6.17.9-arch1-1
- Works w/ ffmpeg 8.0.1 cli
Machine 3 (HexStan)
- Mesa Gallium driver 25.0.7-0ubuntu0.24.04.2
  - ACO
  - DRM 3.57
- Renoir
  - AMD Radeon Graphics R7-5800H (Vega 8)
- Linux kernel 6.8.0-90-generic)
- Works w/ ffmpeg 6.0.1 cli

Test Results

X: corrupted encoding output
?: untested

Machine	h264 mp4	h264 mkv	h265 mp4	h265 mkv	av1 mp4	av1 mkv
1	ok	ok	ok	ok	n/a	n/a
2	?	X	?	ok	?	ok
3	?	?	X	ok	X	ok

sgothel mentioned this pull request Nov 29, 2025

Merge branch hw-encoder-vaapi #3039

Closed

bradleysepos added the Enhancement label Nov 29, 2025

sr55 requested a review from galad87 November 29, 2025 19:06

sr55 added this to the 1.11.0 milestone Nov 29, 2025

sr55 reviewed Nov 29, 2025

View reviewed changes

libhb/handbrake/common.h Show resolved Hide resolved

libhb/vaapi_common.c Outdated Show resolved Hide resolved

bradleysepos self-requested a review November 29, 2025 19:34

galad87 requested changes Nov 30, 2025

View reviewed changes

encavcodecInit: Enable libav verbosity and show codec options if glob…

116c13e

…al_verbosity_level >= 3

sgothel force-pushed the hw-encoder-vaapi2 branch from 7eaee89 to 1a5379c Compare November 30, 2025 17:53

sgothel force-pushed the hw-encoder-vaapi2 branch 2 times, most recently from 15ca91a to 646822a Compare December 1, 2025 06:02

sgothel force-pushed the hw-encoder-vaapi2 branch 2 times, most recently from ddab085 to 6f6728d Compare December 4, 2025 09:51

sgothel force-pushed the hw-encoder-vaapi2 branch from 6f6728d to 0b77490 Compare December 6, 2025 21:45

sgothel force-pushed the hw-encoder-vaapi2 branch from 0b77490 to baac9a8 Compare December 8, 2025 07:28

mujahidaminu01-dot mentioned this pull request Dec 9, 2025

Yes, I also don't know whether this is filled correctly using this hack attaching a blob. #7483

Closed

sgothel force-pushed the hw-encoder-vaapi2 branch from baac9a8 to f560738 Compare December 11, 2025 03:08

sgothel requested a review from galad87 December 11, 2025 03:49

galad87 reviewed Dec 11, 2025

View reviewed changes

libhb/hb.c Outdated Show resolved Hide resolved

libhb/vaapi_common.c Show resolved Hide resolved

libhb/encavcodec.c Show resolved Hide resolved

sgothel force-pushed the hw-encoder-vaapi2 branch from f560738 to e0d003d Compare December 19, 2025 21:08

HexStan mentioned this pull request Dec 20, 2025

[Feature request] Support AMD VCE encoder which is supported by official HandBrake jlesage/docker-handbrake#441

Open

VAAPI Encoder (2nd branch/attempt) #7467

Are you sure you want to change the base?

VAAPI Encoder (2nd branch/attempt) #7467

Conversation

sgothel commented Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bradleysepos commented Nov 29, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sgothel commented Nov 30, 2025

Uh oh!

rathann commented Nov 30, 2025

Uh oh!

sgothel commented Dec 1, 2025

Uh oh!

sgothel commented Dec 4, 2025

Uh oh!

Mastergatto commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sgothel commented Dec 6, 2025

Uh oh!

Mastergatto commented Dec 6, 2025

Uh oh!

Mastergatto commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sgothel commented Dec 6, 2025

Uh oh!

sgothel commented Dec 6, 2025

Uh oh!

Mastergatto commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sgothel commented Dec 6, 2025

Uh oh!

sgothel commented Dec 8, 2025

Uh oh!

sgothel commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

galad87 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sgothel commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HexStan commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sgothel commented Dec 21, 2025

Uh oh!

HexStan commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sgothel commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HexStan commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HexStan commented Dec 24, 2025

Uh oh!

sgothel commented Dec 25, 2025

Test Machine Setup

Test Results

Uh oh!

Reviewers

sgothel commented Nov 29, 2025 •

edited

Loading

Mastergatto commented Dec 6, 2025 •

edited

Loading

Mastergatto commented Dec 6, 2025 •

edited

Loading

Mastergatto commented Dec 6, 2025 •

edited

Loading

sgothel commented Dec 11, 2025 •

edited

Loading

sgothel commented Dec 19, 2025 •

edited

Loading

HexStan commented Dec 21, 2025 •

edited

Loading

HexStan commented Dec 21, 2025 •

edited

Loading

sgothel commented Dec 21, 2025 •

edited

Loading

HexStan commented Dec 21, 2025 •

edited

Loading