Unfortunately, AMAP_3D_TARGET appears to be useless. Internally, surface allocation comes up with the following capabilities for the surface: DDSCAPS_VIDEOMEMORY | DDSCAPS_LOCALVIDMEM | DDSCAPS_OFFSCREENPLAIN | DDSCAPS_3DDEVICE. Sadly, DirectDraw responds (might respond?) with E_NOTIMPL.

While one can advance one step further by removing DDSCAPS_VIDEOMEMORY | DDSCAPS_LOCALVIDMEM, another problem is hit that you are no longer able to create a surface with is both FourCC enabled and in the same time has 3D rendering capabilities. Remove your YUV four character code (such as NV12, YV12, YUY2 you would normally have as codec’s output) or fail miserably with DDERR_INVALIDPIXELFORMAT.

Back to the original problem, the AMAP_3D_TARGET flag failing to work makes it impossible to allocate 3D-enabled DirectDraw surface with DirectShow Video Mixing Renderer. While it might sound as deprecated technology, it is yet a supported Windows SDK component, the most efficient video rendering component, the least hardware sensitive video rendering component, and – the most immediately important – the only way an SDK I am working with can deliver its video overlay.

So, there is no easy way to request a 3D enabled surface through a custom allocator-presenter, with a hook in the IVMRSurfaceAllocator::AllocateSurface updating the allocation flags. It does not make much sense to replace the whole allocator-presenter either: while an earlier DirectShow/Windows SDK provides a decent base for custom allocator-presenter, it is going to eventually hit the same problems mentioned above. And you cannot blit from 3D surface into a plain offscreen surface either without getting E_FAIL.

What appears to still be possible is allocating an addition 3D enabled surface, such as within IVMRImagePresenter::StartPresenting and using it as a replacement surface within IVMRImagePresenter::PresentImage. Having a custom allocator-presenter receive video frame, you can blit the picture into 3D-enabled surface, do your thing and pass the updated VMRPRESENTATIONINFO to default allocator-presenter so that it presents the additional update surface, not the original one.

The question is being asked from time to time. Everyone knows what is full screen video. Video renderers implement full screen capability since long ago through their IVideoWindow::put_FullScreenMode property, and even Filter Graph Manager exposes its own IVideoWindow interface to forward calls to filter’s implementation of IVideoWindow interface.

However, for Video Mixing Renderers, version 7 and 9, the preferred and recommended mode is windowless, where no IVideoWindow interface is available.

Note The IVMRWindowlessControl or IVMRWindowlessControl9 interface is now preferred over IVideoWindow. For more information, see Using Windowless Mode.

So in order to implement full screen mode it takes the application to attach Video Mixing Renderer filter to a private frameless window, to its entire client area and expand the window to entire monitor area.

The sample project FullScreenWindowlessVmrSample01 is illustrating this technique.

Main application dialog owns a special window to host renderer, implemented by CRendererDialog class.

class CRendererDialog :
    public CDialogImpl<CRendererDialog>
{
public:

    enum { IDD = IDD_RENDERER };

BEGIN_MSG_MAP_EX(CRendererDialog)
    MSG_WM_INITDIALOG(OnInitDialog)
    MSG_WM_DESTROY(OnDestroy)
    MSG_WM_ERASEBKGND(OnEraseBkgnd)
    MSG_WM_PAINT(OnPaint)
    MSG_WM_DISPLAYCHANGE(OnDisplayChange)
    MSG_WM_KEYDOWN(OnKeyDown)
    COMMAND_ID_HANDLER_EX(IDCANCEL, OnCancel)
END_MSG_MAP()

private:
    CMainDialog& m_Owner;
//...

The dialog template does not have a caption, it has no frame and has its window rectangle equal to client area. As a result, spanning the Video Mixing Renderer to the entire client area and proper positioning of the dialog window is showing video on the entire monitor.

ATLVERIFY(m_RendererDialog.Create(m_hWnd));
ATLVERIFY(m_RendererDialog.MoveWindow(MonitorData.m_Position));
m_RendererDialog.ShowWindow(SW_SHOWNORMAL);
// ...
ATLENSURE_SUCCEEDED(pVmrWindowlessControl->SetVideoClippingWindow(m_RendererDialog));
ATLENSURE_SUCCEEDED(pVmrWindowlessControl->SetVideoPosition(NULL, m_RendererDialog.GetVideoPosition()));

Visual C++ .NET 2008 source code is available from SVN, release binary included. The project is using ATL and WTL libraries.

This is a begged MFC code for multiple windowless video renderers. MFC project, two independent video renderers hosted by the same parent window (actually through owned controls but this makes no major difference), VMR9 in windowless mode.

There are no WM_PAINT/WM_ERASEBKGND handlers, IVMRWindowlessControl9::RepaintVideo calls and other basically required code, instead a minimalistic snippet to make video rendered the requested way.

A Visual C++ .NET 2008 source code is available from SVN, release binary included.

Although I personally have no experience with tricky DirectShow filters that decide on possibility of connection not only looking at media type and other capabilities, there has been a number of cases mentioned that certain video decoders will only connect to renderers or video renderers in order to avoid interception of decoded data.

It was recently mentioned that one of such filters checked Misc Flags obtained through IAMFilterMiscFlags interface to make sure it is connected to renderer downstream. Before we proceed, let us make it clear that it is silly and can only protect from beginner, a complete newbie. At the very least it should have checked peer filter’s CLSID and compare against white list of stock video renderer: Video Renderer, Video Mixing Rendeder 7, Video Mixing Rendeder 9, Enhanced Video Renderer. A custom video renderer or even just a transformation filter with a renderer flag would be able to receive decoded data and make it available for any further processing.

What I am going to do is to take Sample Grabber Filter as a base and make a new filter from it which will only connect to renderer (it will check the flags as described above). Then another filter, which will also use Sample Grabber Filter base, will accurately fool the first one and connect to output of the first filter and will render video to complete the graph.

Both new filters will use COM aggregation to reuse existing implementation of Sample Grabber Filter, refer to DirectShow Fitler through COM Aggregation post for details. For a brief explanation, new filter aggregates another COM object and exposes all its interfaces except main IUnknown (as required by aggregation), IPersist/IMediaFilter/IBaseFilter in order to override CLSID and flag check, and additionally the second filter will implement its own IAMFilterMiscFlags interface natively. The filter classes register themselves using Filter Mapper‘s IFilterMapper2::RegisterFilter as a part of regular COM registration.

The first filter class CImpudence is going to check the downstream connection. Because we are basing on an initialized sample grabber, the filter will accept any connection and pass media sample further downstream without any modification. It makes no principal difference where to check the connection at, but for simplicity and code brevity, since we are already implementing IBaseFilter, we will do the check inside IBaseFilter::Pause call and in case of unsatisfactory connection on the output the filter would return an error code, let it be VFW_E_CERTIFICATION_FAILURE to make sure we see the message in GraphEdit coming right from our code. So the method is:

STDMETHOD(Pause)() throw()
{
	ATLTRACE2(atlTraceCOM, 4, _T("...\n"));
	_ATLTRY
	{
		FILTER_STATE State;
		ATLENSURE_SUCCEEDED(GetState(0, &State));
		if(State == State_Stopped)
		{
			CComPtr<IPin> pOutputPin = _FilterGraphHelper::GetFilterPin(this, PINDIR_OUTPUT);
			ATLENSURE_THROW(pOutputPin, E_NOINTERFACE);
			CComPtr<IPin> pOutputPeerPin = _FilterGraphHelper::GetPeerPin(pOutputPin);
			ATLENSURE_THROW(pOutputPeerPin, E_NOINTERFACE);
			CComPtr<IBaseFilter> pPeerBaseFilter = _FilterGraphHelper::GetPinFilter(pOutputPeerPin);
			ATLASSERT(pPeerBaseFilter);
			{
				CComQIPtr<IAMFilterMiscFlags> pAmFilterMiscFlags = pPeerBaseFilter;
				ATLENSURE_THROW(pAmFilterMiscFlags, VFW_E_CERTIFICATION_FAILURE);
				const ULONG nFlags = pAmFilterMiscFlags->GetMiscFlags();
				ATLTRACE2(atlTraceGeneral, 2, _T("nFlags 0x%x\n"), nFlags);
				ATLENSURE_THROW(nFlags & AM_FILTER_MISC_FLAGS_IS_RENDERER, VFW_E_CERTIFICATION_FAILURE);
			}
		}
	}
	_ATLCATCH(Exception)
	{
		_ATLTRY
		{
			CComQIPtr<IMediaEventSink> pMediaEventSink = GetFilterGraph(this);
			if(pMediaEventSink)
				ATLVERIFY(SUCCEEDED(pMediaEventSink->Notify(EC_ERRORABORT, (HRESULT) Exception, 0)));
		}
		_ATLCATCHALL()
		{
			// TODO: Trace exception
		}
		return Exception;
	}
	return m_pInnerBaseFilter->Pause();
}

Briefly, if the fitler is stopped and is going to be paused we are checking connected pin’s filter on our filter output, query its IAMFilterMiscFlags, if it’s absent we are not going to work, if the flags have no AM_FILTER_MISC_FLAGS_IS_RENDERER we are not going to work and return the error code. Additionally, error code is sent through IMediaEventSink::Notify so that the graph could stop.

Expectedly, if the filter is connected to something which is not a renderer, e.g. Color Space Converter Filter, it fails to run:

The checking filter, which is an imitation of the real life decoder requiring connections to renderer only, is ready and I am getting down to the second part, the non-renderer filter which will still connect and receive the media samples.

The goal is to show AM_FILTER_MISC_FLAGS_IS_RENDERER flag to upstream filter only and show proper zero flags to the rest of the graph. This approach ensures proper graph operation and will also do the thing if we implemented CLSID comparison (as mentioned in the top of the post) showing wrong CLSID to the upstream filter and correct CLSID to the graph to allow load/save and other IPersistStream-based operations.

The second filter CFix is also using Sample Grabber Filter as a base and embeds it through COM aggregation. What is additionally required to the embedded implementation here is to add IAMFilterMiscFlags interface and implement GetMiscFlags method to return the flags.

Inside the method call it is unable to tell who exactly is calling the interface. Let us consider sufficient to know what is the image file name of the DLL from where the call has been received. Provided that it is known in advance what is the DLL that host implementation of the filter to fool, this information is sufficient for decision.

In order to identify the caller I am using debug helper StackWalk64 API and initialize STACKFRAME64 argument to be able to step out of the current method. The structure is initialized using current CPU register values and is passed to the debug API to step out of the function.

	STACKFRAME64 StackFrame;
	ZeroMemory(&StackFrame, sizeof StackFrame);
	StackFrame.AddrPC.Mode = AddrModeFlat;
	StackFrame.AddrFrame.Mode = AddrModeFlat;
	StackFrame.AddrStack.Mode = AddrModeFlat;
	__asm
	{
		call __1
__1:
		pop dword ptr [StackFrame.AddrPC.Offset]
		mov dword ptr [StackFrame.AddrFrame.Offset], ebp
		mov dword ptr [StackFrame.AddrStack.Offset], esp
	}

A StackWalk64 call that follows initializes return address from the function which already belongs to the caller code. Using information from EnumerateLoadedModules64 API it is possible to locate the caller DLL name and decide on the value to return.

const CLoadedModule& LoadedModule = LoadedModuleList.GetAt(Position);
ATLTRACE2(atlTraceGeneral, 4, _T("LoadedModule.m_sName %s, .m_nBase 0x%08x, .m_nSize 0x%08x\n"), LoadedModule.m_sName, (ULONG) LoadedModule.m_nBase, LoadedModule.m_nSize);
if(LoadedModule.m_sName.CompareNoCase(_T("RendererOnlySample.dll")) == 0)
	return (ULONG) AM_FILTER_MISC_FLAGS_IS_RENDERER;

This appears to be sufficient to connect CFix to the output of CImpudence filter and get the graph operational and receive media samples.

It is worth mentioning that nevertheless debug API functions are used, the functionality is easily implementable using a tiny insertion of assembler code and standard Windows API.

Reference source code (will require additional headers to compile): RendererOnlySample.01.zip (note that Release build binary is included)