Microsoft AAC Encoder’s MF_E_TRANSFORM_NEED_MORE_INPUT after MFT_OUTPUT_STATUS_SAMPLE_READY

Media Foundation AAC Encoder is a pure MFT, as opposed to legacy DSP’s which are made dual DMO/MFT interfaced and presumably have higher chances for smaller artifacts.

The transform is synchronous and is supposed to be simpler inside: fully passive and drives by input/output calls.

Nevertheles, advertising MFT_OUTPUT_STATUS_SAMPLE_READY via IMFTransform::GetOutputStatus call, if might falsely indicate availability of data. Subsequent ProcessOutput call returns MF_E_TRANSFORM_NEED_MORE_INPUT… Documented behavior:

If the method returns the MFT_OUTPUT_STATUS_SAMPLE_READY flag, it means you can generate one or more output samples by calling IMFTransform::ProcessOutput.

MFTs are not required to implement this method. If the method returns E_NOTIMPL, you must call ProcessOutput to determine whether the transform has output data.

The method is optional, but it is implemented on this particular MFT. Also, this MFT is one of the stock transforms that are documented for public use. Microsoft could apparently have done a better job implementing it cleanly.

See also:

IMFAttributes::CopyAllItems freeze on copying to self

An attempt to copy Media Foundation attribute collection to itself results in a deadlock. Well, it’s not a good idea and a practical one to do a nonsense like this, but the implementation should be resistant to such use either, esp. avoiding the unexpected deadlock.

#include "stdafx.h"
#include <mfapi.h>

#pragma comment(lib, "mfplat.lib")
#pragma comment(lib, "mfuuid.lib")

int main()
{
    _ATLTRY
    {
        ATLENSURE_SUCCEEDED(MFStartup(MF_VERSION));
        CComPtr<IMFAttributes> pAttributes;
        ATLENSURE_SUCCEEDED(MFCreateAttributes(&pAttributes, 1));
        ATLENSURE_SUCCEEDED(pAttributes->SetGUID(MF_MT_SUBTYPE, GUID_NULL));
        ATLENSURE_SUCCEEDED(pAttributes->CopyAllItems(pAttributes)); // <<--- Freeze
    }
    _ATLCATCHALL()
    {
    }
    return 0;
}

The freeze takes place around SRW locks, that is presumably the implementation attempts to lock the data for reading and then once again for writing immediately afterwards.

    ntdll.dll!_NtWaitForAlertByThreadId@8()    Unknown
    ntdll.dll!RtlAcquireSRWLockExclusive()  Unknown
    mfplat.dll!CMFAttributesImpl<struct IMFAttributes,class CMFSRWLock>::DeleteAllItems(void)   Unknown
    mfplat.dll!CMFAttributesImpl<struct IMFAttributes,class CMFSRWLock>::_CloneAllAttributes(struct IMFAttributes *)    Unknown
    mfplat.dll!CMFAttributesImpl<struct IMFAttributes,class CMFSRWLock>::CopyAllItems(struct IMFAttributes *)   Unknown
>   MfSample01.exe!main() Line 15   C++

Media Foundation’s MFT_MESSAGE_SET_D3D_MANAGER with Frame Rate Converter DSP

It might look weird why would someone try Direct3D mode with a DSP, which is not supposed to be Direct3D aware, but still. I am omitting the part why I even got to such scenario. The documentation says a few things about MFT_MESSAGE_SET_D3D_MANAGER:

  • This message applies only to video transforms. The client should not send this message unless the MFT returns TRUE for the MF_SA_D3D_AWARE attribute (MF_SA_D3D11_AWARE for Direct3D 11).
  • Do not send this message to an MFT with multiple outputs.
  • An MFT should support this message only if the MFT uses DirectX Video Acceleration for video processing or decoding.
  • If an MFT supports this message, it should also implement the IMFTransform::GetAttributes method and return the value TRUE…
  • If an MFT does not support this message, it should return E_NOTIMPL from ProcessMessage. This is an exception to the general rule that an MFT can return S_OK from any message it ignores.

Frame Rate Converter DSP is a hybrid DMO/MFT, which in turn basically means that its “legacy” DMO upgraded to MFT using specialized wrapper. It is not supposed to be Direct3D aware, not documented as such.

However it could presumably normalize frame rate of Direct3D aware samples by dropping/duplicating samples respectively. It could easily be Direct3D aware since it does not need, in its simplest implementation, to change the data. It is easy to see that the MFT satisfies the other conditions: it is single output video transform.

The MFT correctly and expectedly does not advertise itself as Direct3D aware. It does not have transform attributes.

However, it fails to comply with documented behavior on returning E_NOTIMPL in MFT_MESSAGE_SET_D3D_MANAGER message. The message is defined to be an exception, however DSP implementation seems to be ignoring that. The wrapper could possibly be created even before the exception was introduced in first place.

The DSP does not make an exception, returns success code as if it does handle the message, and does not act as documented.

Bug in Media Foundation MPEG-4 File Source related to timestamping video frames of a fragmented MP4 file

Some recent update in Media Foundation platform introduced a new bug related to fragmented MP4 files and H.264 video. The bug shows up consistently with file versions:

  • mfplat.dll – 10.0.14393.351 (rs1_release_inmarket.161014-1755)    15-Oct-16 05:48
  • mfmp4srcsnk.dll – 10.0.14393.351 (rs1_release_inmarket.161014-1755)    15-Oct-16 05:45

The nature of the problem is that MPEG-4 File Source is incorrectly time stamping the data: frame time stamps are incorrect, they seems to be getting wrong durations and increments, then quickly jumps into future… and on playback this leads to unobvious playback freezes. As Media Foundation is used by Windows Media Player, Windows 10 Movies & TV Player, the bug is present there as well.

The original report is on MSDN Forums.

Presumably it is possible to roll certain Windows Update package back, or alternatively one has to wait for Microsoft to fix the problem and deliver a new update deploying the fix.

DirectShowSpy: REGDB_E_CLASSNOTREG with IMMDevice::Activate

A DirectShow developer complained on sudden failure of Core Audio IMMDevice::Activate call supposed to instantiate a DirectShow filter for a given device.

The problem appeared to be related to installed DirectShowSpy and its interference with the API calls. The symptom was of the following kind: when Activate was called for different types of objects, the calls all succeeded except interoperation with DirectShow (activation for IBaseFilter), e.g. EnumerateAudioDevices output:

    IAudioClient            0x00000000
    IAudioEndpointVolume    0x00000000
    IAudioMeterInformation  0x00000000
    IAudioSessionManager    0x00000000
    IAudioSessionManager2   0x00000000
    IBaseFilter             REGDB_E_CLASSNOTREG
    IDeviceTopology         0x00000000
    IMFTrustedOutput        0x00000000

When Core Audio is requested to do DirectShow activation, the API creates and instance of System Device Enumerator, which is forwarded the activation call to. DirectShowSpy intercepts these calls, however what it did not do was support for unknown COM interfaces, and support for undocumented IMMDeviceActivator interface which is used internally by the APIs to forward the activation call.

So, System Device Enumerator implements documented ICreateDevEnum and then it also implements undocumented internal IMMDeviceActivator. The entire sequence call is as follows:

// Top level code:

CComPtr<IMMDevice> pDevice = ...; // Audio endpoint interface
pDevice->Activate(..., __uuidof(IBaseFilter), ...)

// API:

STDMETHOD(Activate)(...)
{
    // ...
    if(requested is IBaseFilter)
    {
        CComPtr<IMMDeviceActivator> pDeviceActivator;
        pDeviceActivator.CoCreateInstace(CLSID_SystemDeviceEnum);
        return pDeviceActivator->Activate(pDevice, ...)
    }

DirectShowSpy’s failure to provide IMMDeviceActivator resulted in symptom in question and is fixed with version 1.0.0.2106 and on. The failure code is not so much descriptive, but of course the APIs did not expect external hook and failure is not actually a supposed possible behavior there.

System Device Enumerator matches the known devices to the provided Core Audio device and creates an instance of respective filter – this is how APIs work together. DirectShowSpy prints these calls out to its log.

roatlbase.h(1582): TraceModuleVersion: "D:\...\DirectShowSpy-Win32.dll" version is 1.0.0.2107, running in "D:\...\EnumerateAudioDevices-Win32.exe" at 0x63210000
dllmain.h(36): CDirectShowSpyModule::CDirectShowSpyModule: this 0x633963A4
SystemDeviceEnumeratorSpy.h(669): CSystemDeviceEnumeratorSpyT<...>::CSystemDeviceEnumeratorSpyT: this 0x02F1DA68
SystemDeviceEnumeratorSpy.h(681): CSystemDeviceEnumeratorSpyT<...>::FinalConstruct: pszPath "D:\...\EnumerateAudioDevices-Win32.exe", this 0x02F1DA68, m_dwRef 1
SystemDeviceEnumeratorSpy.h(49): CSystemDeviceEnumeratorSpyT<...>::InternalQueryInterface: 0x02F1DA68, Interface {3B0D0EA4-D0A9-4B0E-935B-09516746FAC0}, Result 0x00000000
SystemDeviceEnumeratorSpy.h(49): CSystemDeviceEnumeratorSpyT<...>::InternalQueryInterface: 0x02F1DA68, Interface {3B0D0EA4-D0A9-4B0E-935B-09516746FAC0}, Result 0x00000000
SystemDeviceEnumeratorSpy.h(808): CSystemDeviceEnumeratorSpyT<...>::Activate: this 0x02F1DA68, InterfaceIdentifier {56A86895-0AD4-11CE-B03A-0020AF0BA770}, pMmDevice 0x0054E7F8
SystemDeviceEnumeratorSpy.h(815): CSystemDeviceEnumeratorSpyT<...>::Activate: nActivateResult 0x00000000 
SystemDeviceEnumeratorSpy.h(673): CSystemDeviceEnumeratorSpyT<...>::~CSystemDeviceEnumeratorSpyT: this 0x02F1DA68

Download links

Calling convention violator broke streaming loop pretty far away

A really nasty problem coming from MainConcept AVC/H.264 SDK Encoder was destroying media streaming pipeline. SDK is somewhat old (9.7.9.5738) and the problem might be already fixed, or might be not. The problem is a good example of how a small bug could become a big pain.

The problem was coming up in 64-bit Release builds only. Win32 build? OK. Debug build where you can step things through? No problem.

The bug materialized in GDCL MP4 Demultiplexer filter streaming (Demultiplexer filter in the pipeline below) generating media samples with incorrect time stamps.

Pipeline

Initial start and stop time are okay, and further go as _I64_MIN (incorrect).

Clipbrd3

The problem appears to be SSE optimization and x64 calling convention related. This explains why it’s only 64-bit Release build suffering from the issue. MS compiler decided to use XMM7 register for dRate variable in this code fragment:

REFERENCE_TIME tStart, tStop;
double dRate;
m_pParser->GetSeekingParams(&tStart, &tStop, &dRate);

[...]

for(; ; )
{
    [...]

    tSampleStart = REFERENCE_TIME(tSampleStart / dRate);
    tSampleEnd = REFERENCE_TIME(tSampleEnd / dRate);

dRate is the only floating point thing here and it’s clear why the compiler optimized the variable into register: no other floating point activity around.

However sample delivery goes pretty deep into other functions and modules reaching MainConcept H.264 encoder. One of its functions is violating x64 calling convention and does not preserve XMM6+ register values. OOPS! Everything is about working right, but after media sample delivery dRate value is destroyed and further media samples receive incorrect time stamps.

It is not really a problem of MP4 demultiplexer, of course, however media sample delivery might involve a long delivery chain where any violator would break streaming loop. In the same time, it is not really a big expense to de-optimize the floating point math in the demultiplexer for those a few time stamp adjustment operations. A volatile specifier breaks compiler optimization and makes the loop resistant to SSE2 register violators:

// HOTFIX: Volatile specifier is not really necessary here but it fixes a nasty problem with MainConcept AVC SDK violating x64 calling convention;
//         MS compiler might choose to keep dRate in XMM6 register and the value would be destroyed by the violating call leading to incorrect 
//         further streaming (wrong time stamps)
volatile DOUBLE dRate;
m_pParser->GetSeekingParams(&tStart, &tStop, (DOUBLE*) &dRate);

This makes H.264 this build of encoding SDK unstable and the problem is hopefully already fixed. The SDK indeed gave other troubles on specific architectures leading to undefined behavior.

Windows 10 AVI Splitter bug

There were a few reports that in Windows 10 it is unable to play AVI files, which played fine in earlier versions of Windows, AVI files specifically.

OK, the problem does exist. More to say, the problem exist in Windows component that implements AVI Splitter DirectShow filter. One of the reporters mentioned he had a problem with a DV AVI flie. I build one and it indeed showed the problem:

AVI Splitter bug in GraphStudioNext

Playback stops at the same frame every time the filter graph is run. The error is 0x8004020D VFW_E_BUFFER_OVERFLOW “The buffer is not big enough” coming from AVI Splitter’s worker thread. The buffers on the memory allocators look appropriate, so the bug looks related to AVI Splitter implementation details, CBaseMSRWorker class that reads from file and delivers frames downstream.

AVI Splitter bug call stack

The problem exists in 32 and 64 bit versions, but not in Media Foundation. With certain luck Microsoft will fix the problem on their side.