Although I personally have no experience with tricky DirectShow filters that decide on possibility of connection not only looking at media type and other capabilities, there has been a number of cases mentioned that certain video decoders will only connect to renderers or video renderers in order to avoid interception of decoded data.
It was recently mentioned that one of such filters checked Misc Flags obtained through IAMFilterMiscFlags interface to make sure it is connected to renderer downstream. Before we proceed, let us make it clear that it is silly and can only protect from beginner, a complete newbie. At the very least it should have checked peer filter’s CLSID and compare against white list of stock video renderer: Video Renderer, Video Mixing Rendeder 7, Video Mixing Rendeder 9, Enhanced Video Renderer. A custom video renderer or even just a transformation filter with a renderer flag would be able to receive decoded data and make it available for any further processing.
What I am going to do is to take Sample Grabber Filter as a base and make a new filter from it which will only connect to renderer (it will check the flags as described above). Then another filter, which will also use Sample Grabber Filter base, will accurately fool the first one and connect to output of the first filter and will render video to complete the graph.
Both new filters will use COM aggregation to reuse existing implementation of Sample Grabber Filter, refer to DirectShow Fitler through COM Aggregation post for details. For a brief explanation, new filter aggregates another COM object and exposes all its interfaces except main IUnknown (as required by aggregation), IPersist/IMediaFilter/IBaseFilter in order to override CLSID and flag check, and additionally the second filter will implement its own IAMFilterMiscFlags interface natively. The filter classes register themselves using Filter Mapper‘s IFilterMapper2::RegisterFilter as a part of regular COM registration.
The first filter class CImpudence is going to check the downstream connection. Because we are basing on an initialized sample grabber, the filter will accept any connection and pass media sample further downstream without any modification. It makes no principal difference where to check the connection at, but for simplicity and code brevity, since we are already implementing IBaseFilter, we will do the check inside IBaseFilter::Pause call and in case of unsatisfactory connection on the output the filter would return an error code, let it be VFW_E_CERTIFICATION_FAILURE to make sure we see the message in GraphEdit coming right from our code. So the method is:
STDMETHOD(Pause)() throw()
{
ATLTRACE2(atlTraceCOM, 4, _T("...\n"));
_ATLTRY
{
FILTER_STATE State;
ATLENSURE_SUCCEEDED(GetState(0, &State));
if(State == State_Stopped)
{
CComPtr<IPin> pOutputPin = _FilterGraphHelper::GetFilterPin(this, PINDIR_OUTPUT);
ATLENSURE_THROW(pOutputPin, E_NOINTERFACE);
CComPtr<IPin> pOutputPeerPin = _FilterGraphHelper::GetPeerPin(pOutputPin);
ATLENSURE_THROW(pOutputPeerPin, E_NOINTERFACE);
CComPtr<IBaseFilter> pPeerBaseFilter = _FilterGraphHelper::GetPinFilter(pOutputPeerPin);
ATLASSERT(pPeerBaseFilter);
{
CComQIPtr<IAMFilterMiscFlags> pAmFilterMiscFlags = pPeerBaseFilter;
ATLENSURE_THROW(pAmFilterMiscFlags, VFW_E_CERTIFICATION_FAILURE);
const ULONG nFlags = pAmFilterMiscFlags->GetMiscFlags();
ATLTRACE2(atlTraceGeneral, 2, _T("nFlags 0x%x\n"), nFlags);
ATLENSURE_THROW(nFlags & AM_FILTER_MISC_FLAGS_IS_RENDERER, VFW_E_CERTIFICATION_FAILURE);
}
}
}
_ATLCATCH(Exception)
{
_ATLTRY
{
CComQIPtr<IMediaEventSink> pMediaEventSink = GetFilterGraph(this);
if(pMediaEventSink)
ATLVERIFY(SUCCEEDED(pMediaEventSink->Notify(EC_ERRORABORT, (HRESULT) Exception, 0)));
}
_ATLCATCHALL()
{
// TODO: Trace exception
}
return Exception;
}
return m_pInnerBaseFilter->Pause();
}
Briefly, if the fitler is stopped and is going to be paused we are checking connected pin’s filter on our filter output, query its IAMFilterMiscFlags, if it’s absent we are not going to work, if the flags have no AM_FILTER_MISC_FLAGS_IS_RENDERER we are not going to work and return the error code. Additionally, error code is sent through IMediaEventSink::Notify so that the graph could stop.
Expectedly, if the filter is connected to something which is not a renderer, e.g. Color Space Converter Filter, it fails to run:
The checking filter, which is an imitation of the real life decoder requiring connections to renderer only, is ready and I am getting down to the second part, the non-renderer filter which will still connect and receive the media samples.
The goal is to show AM_FILTER_MISC_FLAGS_IS_RENDERER flag to upstream filter only and show proper zero flags to the rest of the graph. This approach ensures proper graph operation and will also do the thing if we implemented CLSID comparison (as mentioned in the top of the post) showing wrong CLSID to the upstream filter and correct CLSID to the graph to allow load/save and other IPersistStream-based operations.
The second filter CFix is also using Sample Grabber Filter as a base and embeds it through COM aggregation. What is additionally required to the embedded implementation here is to add IAMFilterMiscFlags interface and implement GetMiscFlags method to return the flags.
Inside the method call it is unable to tell who exactly is calling the interface. Let us consider sufficient to know what is the image file name of the DLL from where the call has been received. Provided that it is known in advance what is the DLL that host implementation of the filter to fool, this information is sufficient for decision.
In order to identify the caller I am using debug helper StackWalk64 API and initialize STACKFRAME64 argument to be able to step out of the current method. The structure is initialized using current CPU register values and is passed to the debug API to step out of the function.
STACKFRAME64 StackFrame;
ZeroMemory(&StackFrame, sizeof StackFrame);
StackFrame.AddrPC.Mode = AddrModeFlat;
StackFrame.AddrFrame.Mode = AddrModeFlat;
StackFrame.AddrStack.Mode = AddrModeFlat;
__asm
{
call __1
__1:
pop dword ptr [StackFrame.AddrPC.Offset]
mov dword ptr [StackFrame.AddrFrame.Offset], ebp
mov dword ptr [StackFrame.AddrStack.Offset], esp
}
A StackWalk64 call that follows initializes return address from the function which already belongs to the caller code. Using information from EnumerateLoadedModules64 API it is possible to locate the caller DLL name and decide on the value to return.
const CLoadedModule& LoadedModule = LoadedModuleList.GetAt(Position);
ATLTRACE2(atlTraceGeneral, 4, _T("LoadedModule.m_sName %s, .m_nBase 0x%08x, .m_nSize 0x%08x\n"), LoadedModule.m_sName, (ULONG) LoadedModule.m_nBase, LoadedModule.m_nSize);
if(LoadedModule.m_sName.CompareNoCase(_T("RendererOnlySample.dll")) == 0)
return (ULONG) AM_FILTER_MISC_FLAGS_IS_RENDERER;
This appears to be sufficient to connect CFix to the output of CImpudence filter and get the graph operational and receive media samples.
It is worth mentioning that nevertheless debug API functions are used, the functionality is easily implementable using a tiny insertion of assembler code and standard Windows API.
Reference source code (will require additional headers to compile): RendererOnlySample.01.zip (note that Release build binary is included)