A bump of StackOverflow post about Media Foundation design flaw related to video encoding.
Set attributes via ICodecAPI for a H.264 IMFSinkWriter Encoder
I am trying to tweak the attributes of the H.264 encoder created via ActivateObject() by retrieving the ICodecAPI interface to it. Although I do not get errors, my settings are not taken into account. […]
Media Foundation’s Sink Writer is a simplified API with a encoder configuration question slipped away. The fundamental problem here is that you don’t own the encoder MFT and you are accessing it over the writer’s head, then the behavior of encoders around changing settings after everything is set up depends on implementation, which is in encoder’s case a vendor specific implementation and might vary across hardware.
Your more reliable option is to manage encoding MFT directly and supply Sink Writer with already encoded video.
Your potential trick to make things work with less of effort is to retrieve IMFTransform of the encoder as well and clear and then set back the input/output media types after you finished with ICodecAPI update. Nudging the media types, you suggest that encoder re-configures the internals and it would do this already having your fine tunings. Note that this, generally speaking, might have side issues.
The ‘trick’ seems to work for some of the ICodecAPI parameters (e.g. CODECAPI_AVEncCommonQualityVsSpeed) and only for Microsoft’s h.264 encoder. No effect on CODECAPI_AVEncH264CABACEnable. The doc indeed seems to be specifically for Microsoft’s encoder and not be a generic API. I’m using the QuickSync and NVidia codecs, do you know if those are configurable via the ICodecAPI assuming I create the MFT myself?
Vendor provided encoders fall under Certified Hardware Encoder requirements, so they must support ICodecAPI values mentioned in the MSDN article. Important is that it is not defined what the order of configuration calls is. If you are managing encoder yourself you would do ICodecAPI setup before setting up media types. In Sink Writer scenario it already configured the media types, then you jump in with your fine tuning. Hence, my trick suggestion includes the part of resetting existing media types. Because this trick is sensitive to implementation details I would suggest to get current media types, then clear them on the MFT, do ICodecAPI thing and get the types back. I assume that this should work in greater number of scenarios, not just MS encoder. Yet it still remains an unreliable hack.
IMO Nvidia’s encoder implementation is terrible (worst across vendors), Intel’s is better but it still has its own issues. Again IMO the MFTs are only provided to meet minimal certification requirements for hardware video encoding and for this reason their implementation is not well aligned. Various software packages prefer to implement video encoding via vendor SDKs rather than Media Foundation Transform interface. In one of the projects I used to also skip the idea of leveraging MFTs for encoding, and implemented my own MFTs on top of vendor SDKs.
Would the class factory approach in this post work with the
IMFSinkWriter? This would avoid writing too much code…
I suppose that yes, this should work even though I feel that it’s not a pleasant work to patch it that way. Also you might need to take into account support of HW encoders because Sink Writer also tends to use hardware assisted encoding in some cases, including scenario where it’s given a DXGI device.
Another sort of a hack, which is similar but maybe a bit less intrusive (although in its implementation you would have to have a better understanding of internals) is to redefine vendor specific encoder CLSIDs within Sink Writer initialization scope. There are just three encoders (AMD, Intel, Nvidia; okay there is fourth from Shanghai Zhaoxin Semiconductor but it is not really popular) and their CLSIDs are known. If you CoRegisterClassObject in a smart way, you could hook MFT instantiation letting Media Foundation to decide which encoder to choose. It is just another idea though, so it might depend what is the best to do on other factors.
See also: