Hardware acceleration

[Former user]

(see also: http://forum.serviio.org/viewtopic.php?f=7&t=21577)

Dear Developer(s),

It took a while, but I have a hardware accelerated ffmpeg intallation using the h264_qsv codec (intel!), which enables me to transcode on my GPU, which is confirmed working.

Obviously I like to add this to the ..../config/application-profiles.xml (specific: Ios profile, see below) but on a restart I receive this error:

2016-02-19 04:55:16,682 ERROR [XmlUtils] XML Profiles.xsd didn't pass validation, reason: cvc-enumeration-valid: Value 'h264_qsv' is not facet-valid with respect to enumeration '[mpeg2video, wmv2, flv, h264]'. It must be a value from the enumeration. 2016-02-19 04:55:16,682 ERROR [MediaServer] An unexpected error occured. Ending the application. Message: org.serviio.profile.ProfilesDefinitionException: Profiles XML file is not valid (according to the schema). Check the log. java.lang.RuntimeException: org.serviio.profile.ProfilesDefinitionException: Profiles XML file is not valid (according to the schema). Check the log. at org.serviio.profile.ProfileManager.parseProfilesFromFile(ProfileManager.java:183) at org.serviio.profile.ProfileManager.loadProfiles(ProfileManager.java:157) at org.serviio.MediaServer.main(MediaServer.java:127)

Where can I tell serviio that it 'knows' about this codec?

---

xml file:

<Profile id="ios" name="iOS" extendsProfileId="1" alwaysEnableTranscoding="true" selectable="false"> <!-- http://developer.apple.com/library/ios/ ... tions.html --> <Transcoding> <!-- only play mp4/h264_qsv(main3 and less)/aac natively --> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true" forceVTranscoding="true"> <Matches container="" vCodec="h264_qsv" profile="high" levelGreaterThan="4.1"/> </Video> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true"> <Matches container="3gp" /> <Matches container="asf" /> <Matches container="avi" /> <Matches container="flv" /> <Matches container="matroska" /> <Matches container="mpeg" /> <Matches container="mpegts" /> <Matches container="mpegvideo" /> <Matches container="ogg" /> <Matches container="rm" /> <Matches container="wtv" /> <Matches container="mp4" vCodec="mpeg4" /> <Matches container="mp4" vCodec="msmpeg4" /> <Matches container="mp4" aCodec="ac3" /> </Video> <Audio targetContainer="mp3"> <Matches container="flac" /> <Matches container="ogg" /> <Matches container="lpcm" /> <Matches container="adts" /> <Matches container="asf" /> <Matches container="wavpack" /> <Matches container="mpc" /> <Matches container="ape" /> </Audio> </Transcoding> <OnlineTranscoding> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true"> <Matches container="" /> </Video> </OnlineTranscoding> <GenericTranscoding> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true" /> </GenericTranscoding> <AlternativeQualities> <Quality type="MEDIUM"> <Transcoding> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true" maxVBitrate="1000" maxHeight="480"> <Matches container="" /> </Video> </Transcoding> <OnlineTranscoding> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true" maxVBitrate="1000" maxHeight="480"> <Matches container="" /> </Video> </OnlineTranscoding> <GenericTranscoding> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true" maxVBitrate="1000" maxHeight="480"/> </GenericTranscoding> </Quality> <Quality type="LOW"> <Transcoding> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true" aSamplerate="22050" maxVBitrate="500" maxHeight="240"> <Matches container="" /> </Video> </Transcoding> <OnlineTranscoding> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true" aSamplerate="22050" maxVBitrate="500" maxHeight="240"> <Matches container="" /> </Video> </OnlineTranscoding> <GenericTranscoding> <Video targetContainer="applehttp" targetVCodec="h264_qsv" targetACodec="aac" forceStereo="true" aSamplerate="22050" maxVBitrate="500" maxHeight="240"/> </GenericTranscoding> </Quality> </AlternativeQualities> <Subtitles> <HardSubs> <TextBased supported="true" /> </HardSubs> </Subtitles> <ThumbnailsResolution>HD</ThumbnailsResolution> <AutomaticImageRotation>true</AutomaticImageRotation> </Profile>

[patters]

Background

The first Intel hardware acceleration implementation was present in its Media Processor system on-a-chip (SoC) designs. These were Intel CPUs which were originally aimed at set-top-box appliances or Bluray playes - the Intel CE3100 (Canmore), CE4100 (Sodaville), CE5300 (Berryville, later Evansport). These products all used evolutions of a media architecture Intel named "Streaming Media Driver". Very little public information about this exists except parts of the Intel SDK which Synology are obligated to publish under the terms of the GPL. This Media SDK includes a version of FFmpeg which Intel have forked to add hardware support for these chipsets - and this hardware acceleration is known as Intel SMD. It would appear that Intel have employed some of the same conventions from Intel SMD to the recent Intel QSV (QuickSync Video) that has been added to FFmpeg.

Zeranoe's builds of FFmpeg include Intel QSV support.

Hardware codec support is more complicated than it first appears...

Encoding

Command "ffmpeg -encoders" can be used by Serviio to probe the binary for hardware encoding support. If the encoders "h264_qsv", "hevc_qsv", or "mpeg2_qsv" are detected then Serviio should prefer these above "x264", "x265", "mpeg2video" respectively. If these encoders are detected, then perhaps a Hardware Transcoding option could be advertised in the Delivery tab of Serviio Console if the CPU cannot be probed automatically for support. This would provide the user with some fine control.

I have not yet experimented with the actual encoder parameters though, so they are likely to need amending in this case. It's possible that they diverge from the typical parameters of libx264, though in practice the Intel SMD implementation seems to be a hacked libx264. Quality settings will not need to be as low (no very fast profile for instance). I'm not sure how Intel QSV handles race conditions on the QSV encoder. The Intel SMD version writes out a file containing a pid if the hardware encoder is already in use. This would need to be tested for QSV since Serviio can have multiple clients requesting transcoded files, and needs to gracefully fall back to software encoding if possible.

Decoding

Command "ffmpeg -decoders" can be used by Serviio to probe the binary for hardware decoding support. If the decoders "h264_qsv", "hevc_qsv", "mpeg2_qsv", or "vc1_qsv" are detected then Serviio should prefer these above "x264", "x265", "mpeg2video", and "vc1" respectively, assuming Hardware Transcoding is enabled in Serviio Console.

In the case of the hardware transcoding Synology package the actual logic in my custom FFmpeg wrapper is actually quite tricky, since you want to avoid needlessly occupying the hardware decoder under the following scenarios:

1. gathering media metadata,

2. remuxing video,

3. generating thumbnails,

4. rendering subtitles into the video stream (With Intel SMD "pipeline mode" is not supported when using inline video filters - not sure if using Intel QSV to decode directly to Intel QSV encode uses a similar "pipeline mode" - needs testing).

5. or gathering online stream metadata.

The Intel Evansport CPU can HW decode 4 separate streams so I don't bother checking for contention, but this will need investigating for Intel QSV. Can Serviio fall back to software decode if the QSV hardware is busy with another job? For reference, the logic of the Synology hardware encoding wrapper is listed in the collapsed section titled "ffmpeg-wrapper-i686evansport.sh"

I'm also not yet sure how the Intel QSV enabled versions of FFmpeg automatically select the HW decoders. With Intel SMD, FFmpeg uses an additional parameter which must precede all other parameters "-prefer_smd". This tells it to prefer its "h264_smd", "mpeg2video_smd", "mpeg4_smd", "vc1_smd", and "wmv3_smd" decoders without Serviio/FFmpeg/wrapper having to do additional probing. There does not seem to be an equivalent to this for Intel QSV.

Sadly, Intel QSV is only supported on Windows and Linux builds of FFmpeg. Mac OS X is excluded. I wonder if this will change.

Further consideration

The above notes are my thoughts on Intel QSV support specifically. From what I can understand, Intel QSV only offloads certain aspects of the video codecs (mo-comp, iDCT etc.). However, the newer VA-API support added to FFmpeg 2.8 allowed it to leverage the proper hardware fixed function encoders/decoders present in the CPU's integrated GPU. So for Intel Braswell and Skylake CPUs that allows full HEVC decoding and encoding in hardware. Serviio should certainly aim to integrate FFmpeg's support for libVA (Video Acceleration library) since this suddenly opens up realtime transcoding of 4K content to ordinary PCs. More info on FFmpeg's site here.

One pain point on the Synology hardware transcoding version of Serviio is that I have to fork profiles.xml. I do this to make h264 the preferred transcoding codec for all renderers that natively support H.264 playback since it's actually lower computational cost to deliver than mpeg2video. The pain is that I have to manually merge in all the profiles.xml changes between versions of Serviio. Are you able to make Serviio set this transcoding codec preference automatically with a separate option in the Delivery tab in Serviio Console perhaps? Pseudo code would be: does renderer support H.264, and is transcoding set to prefer H.264 over MPEG2, if so use h264 codec. I propose that this transcoding codec preference toggle (h264/mpeg2video) would be greyed out unless Hardware Transcoding is enabled (can only be enabled if FFmpeg supports it).

The way forward with this feature request might be to add Intel QSV as a first step for version 1.7, then add libVA in the next version? I suppose it depends on your target release date for 1.7.

[Petr Nejedly]

@patters It looks like the FFmpeg support actually needs a linked dlls (on Windows) that are part of the Intel SDK. I didn't find a way to compile this statically into a single FFmpeg executable - which is a problem for users.

So this ticket could deal with enabling the codecs via Serviio, but not actually including it in the provided FFmpeg, so that only people with the proper environment could run it.

[patters]

The Zeranoe build is a single EXE. Does it expect an external DLL as well, or has he managed to link it statically? I haven't yet tested it.

[Petr Nejedly]

I tried it and it doesn't work. Will try with the SDK installed later. But got another confirmation as well that is the case - you need the SDK instaled.

[Robert Nagtegaal]

Example command for screen grabbing, as said its not what you want but most help full I hope ;)

nohup /usr/local/bin/ffmpeg -y -s 1920x1080 -f x11grab -draw_mouse 0 -i :0.0 -c:v h264_qsv -map 0:v -pix_fmt nv12 -r 25 -profile:v high -level 4.2 -b:v 1000k -maxrate 1500k -hls_flags delete_segments -g 50 -reset_timestamps 1 -preset fast -segment_list_size 4 -f segment -segment_time 2 -segment_format mpeg_ts -segment_list /var/www/html/stream/0/index.m3u8 -segment_list_type m3u8 -segment_list_flags live /var/www/html/stream/0/segment_%05d.ts

[Robert Nagtegaal]

To remove any doubt!

I you want hardware accelerated transcoding/encoding/decoding making usage of the QSV codec (ffmpeg ... -c:v h264_qsv ...) you need:

• 0 Supported hardware (see intel documentation upon h264_qsv supported hardware)
• 1 A kernel compiled WITH THE INTEL KERNEL PATCHES
• 2 ffmpeg build against mediaserverstudioessentials (ver >= 2015r6, e.g 2016)
• 3 preferable mplayer (or mpv) compiled agains intel vaapi patches

IF AND ONLY IF 0, 1 and 2 IS TRUE you're good to go, capice?

[Petr Nejedly]

Support for an external ffmpeg with QSV included (--enable-libmfx) is now implemented. It's done only for h264 encoding. To enable this, you need

• the proer FFmpeg and
• set -Dserviio.useQSVAcceleration system variable in the startup script of the server

[Robert Nagtegaal]

Hi Petr,

I will test it asap and let you know the results

Kind regards and many thanks!

Robert Nagtegaal

[Petr Nejedly]

I will need to send you a other build.