Screen recording using Desktop Duplication API and hardware H.264 encoder

The application takes advantage of three powerful Windows APIs at a time:

MediaFoundationDesktopRecorder initializes a desktop duplication session and sends obtained desktop images to H.264 video encoder producing a standard MP4 recording. Optionally, it can add an audio track capturing data from one of the standard inputs.

The best performance is achieved when used with hardware H.264 encoder: not only the performance of hardware encoder is better, but additionally desktop images are transferred to the encoder efficiently, without being copied through system memory. With respective hardware, recording is pretty efficient.

There are certain limitations: duplication API is Windows 8+, encoder availability depends on hardware and OS versions. The application let API pick encoder automatically and in worth case scenario falls back to software encoder, which is typically a performance hit.

MediaFoundationDesktopRecorder UI

When started, the application prints initial information, esp. regarding availability of devices, and appends as actions and events take place.

The application uses configuration file with the same name and location as the application, and .INI extension. Changes to the configuration file take effect when the application is restarted.

The application registers Win+F5, Win+F8 hotkeys globally to start/stop recording when the application is in background (that is, when user interacts with another application).

The application generates .MP4 files in the directory of its own location. There will be a video track, and optionally one additional audio track – depending on settings. Video is taken from one of the monitors, and audio – from one of the available standard audio input devices.

The application also generates log files at one the locations:

  • C:\ProgramData\MediaFoundationDesktopRecorder.log
  • C:\Users\$(UserName)\AppData\Local\MediaFoundationDesktopRecorder.log (in case the first path above is inaccessible, esp. due to insufficient permissions)

Configuration

The configuration .INI file might contain a few settings that set up and alter the behavoir of the application:

[Input]
;Video Adapter Description=NVIDIA GeForce GTX 750
Video Output Device Name=\\.\DISPLAY2
;Audio Friendly Name=Stereo Mix (Realtek High Definition Audio)

When started, the application enumerates (“found video…”, “found audio…”) available video and audio inputs. These discoveries are compared against configuration file settings in order to identify monitor for recording, and possibly audio input device.

Default behavior is to take first available monitor, which happens when settings do not instruct otherwise. By default, no audio is recorded. Audio is recorded and added to resulting file if input device is provided explicitly.

The application also prints which devices are taken for further recording (“using adapter…”).

[Format]
;Video Frame Rate=30000
;Video Frame Rate Denominator=1001
Video Bitrate=4096000
Video Texture Pool Capacity=24
Video Throttle=70
Audio Bitrate=192000

Default behavior is to identify monitor’s refresh rate and produce output file with video at the same frame rate. Video Frame Rate and Video Frame Rate Denominator settings offer an override to target file frame rate. With the former value only, it is the frame rate. With both values they define a ratio, e.g. values of 30000 and 1001 result in 29.97 fps file.

Frame rate reduction is a good way to reduce encoding complexity and overall graphics subsystem load.

Bitrate values define respective bitrates for the encoded content.

Details

As recording goes, the application grabs new desktop snapshots and sends them to encoder. There are no specific expectations about frame rate stability and reduction in case of overload of graphics subsystem. When the complexity is excessive, it is expected that some frames might be lost without breaking the entire playability of the output file.

The application provides additional information when it creates a file, for example:

Using Direct3D 11 at feature level D3D_FEATURE_LEVEL_11_0
Using Desktop Duplication mode: Resolution 1680 x 1050, Refresh Rate 59954/1000, Format DXGI_FORMAT_B8G8R8A8_UNORM
Using path “D:\Projects\...\Output\20160707-070707.mp4”
Using video transform Direct3D 11 Aware, Category MFT_CATEGORY_VIDEO_PROCESSOR, Input MFVideoFormat_ARGB32, Output MFVideoFormat_NV12
Using video transform NVIDIA H.264 Encoder MFT, Direct3D 11 Aware, Category MFT_CATEGORY_VIDEO_ENCODER, Input MFVideoFormat_NV12, Output MFVideoFormat_H264
Started writing…
PPP frames written (QQQ frame timeouts, RRR early frame skips, SSS late frame skips)
Stopped writing
Output file size is TTT bytes

When started the application might experience a condition when certain hardware resource is no longer available, e.g. the desktop itself is locked by user. The application will close the file, and attempt to automatically restart recording into new file. The attempts keep going until user explicitly stops recording.

The application does NOT do the following (among things it could):

  • the application is limited to record from one monitor only; to record from two at a time it is possible to start several instances however the produced result will not be synchronized
  • the application does not provide options to record single window image, to cut a section of monitor image or to scale image down
  • the application does not offer choices for video encoders (e.g. there are two or more hardware H.264 encoders), it will always use encoder picked by the system
  • the application only offers bitrate setting for video encoding
  • the application does not provide flexibility in audio encoding settings, it also expects that audio device is available throughout the entire recording session (esp. is not unplugged as recording goes)

References (Informational)

Download links

Build Incrementing for Visual Studio C++ Projects

Over long time I used an automatic build incrementer add-in for Visual Studio and C++ projects, which proved to be helpful. Having increments in file information, the binaries were easy to identify. It was easy to find a matching symbol information etc. Long story short, a tool like this has been a must.

The add-in has problems or downsides though. It kept patching the .RC source and touched it when no other changes existed in the build, touching source code forced rebuilds on its own and reloaded resource-related files opened in Visual Studio editors. I was annoying even though more or less acceptable.

Visual Studio 2015 Community Edition does not support add-ins because of 2015 or because it’s Community Edition. Either way it was time to update the incrementer ot make things nicer overall.

This time I preferred to change things a bit. No longer source code patching: the incrementer can be attached as a post-build event and patch VERSIONINFO resource on the built binary. This requires that current build number is kept somewhere but not in the .RC text, so I am using an additional .INI file. The good thing is that this file can still be included in version control system and the version history can be tracked relatively easily. No longer source code modification which makes code base dirty and forces another rebuild.

Command line syntax:

C:\>IncrementBuild-Win32
Syntax: IncrementBuild-Win32.exe argument [argument...]

Arguments:
  help - displays syntax
  configuration <path> - path to .INI file holding configuration information (mandatory)
  binary <path> - path to binary to be patched with file version update (mandatory)
  string <name> <value> - add, update or remove specific version information string (optional; multiple arguments possible)
  dump - print version information data block dump before and after update

Additional feature is that incrementer can attach additional version strings (see example below – it adds build configuration as a version information string).

Setting up is easy. First, the project should have a version information resource, so that the binary has data to patch in first place.

Then, there should be an .INI file which tracks version numbers. The binary will be build with .RC numbers and then incrementer will apply the least significant number from the .INI file incrementing it along the way.

[General]

[VersionInformation]
;Language=133 ;MAKELANGID(LANG_ENGLISH, SUBLANG_ENGLISH_US)
;Version String Format=%d.%d.%d.%d
Current Build Number=4

Next thing, project post-build event needs a command for patching:

Post-Build Event in VS for C++ Project

"$(AlaxInfo_Common)\..\Utilities\IncrementBuild\_Bin\IncrementBuild-$(PlatformName).exe" configuration "$(ProjectDir)Module.ini" binary "$(TargetPath)" string "ConfigurationName" "$(ConfigurationName)" 

The command takes Module.ini from the projects directory for configuration file, patches build output and also attaches build configuration as an additional version information string.

Build output looks like this:

—— Rebuild All started: Project: EnumerateTransforms, Configuration: Release Win32 ——
stdafx.cpp
Application.cpp
Generating code
Finished generating code
EnumerateTransforms.vcxproj -> D:\Projects…_Bin\Win32\Release\EnumerateTransforms.exe
EnumerateTransforms.vcxproj -> D:\Projects…_Bin\Win32\Release\EnumerateTransforms.pdb (Full PDB)
Configuration Path: D:\Projects…\Module.ini
Binary Path: D:\Projects…_Bin\Win32\Release\EnumerateTransforms.exe
Incrementing build number, product version 1.0.0.1, file version 1.0.0.4
Applying version information string, name “ConfigurationName”, value “Release”

Presumably, it is not necessary to use same bitness tool for a binary, since version information patching API should be able to patch resources of mismatching build, but I normally use a matching tool anyway, why not?

Download links

DirectShowSpy: Automated Media Sample Traces, Load Trace from File

Some time ago DirectShowSpy received a capability to record media sample traces (in specific points of the pipeline). DirectShowSpy UI visualized the chronology of the streaming which facilitated detection of various problems, especially out of order media samples, lost or late delivered samples, thread racing conditions.

A developer can easily integrate his filters with DirectShowSpy media sample tracing, and example of this is a fork of MP4 filters where this powerful option is integrated.

This time DirectShowSpy is improved to offer automated traces. Typical use scenario is automatic creation of media sample trace file so that it could be logged/attached to specific export process and be available for further review. More specifically, an application could run a transcoding session and generate a detailed media sample trace alogn with the output in order to troubleshoot specific issues (dropped frame etc).

DirectShowSpy’s IFilterGraphHelper (implemented by FilterGraphHelper class) object received new additional methods:

interface IFilterGraphHelper : IDispatch
{
    [...]
    [id(7)] HRESULT ResetMediaSampleTrace([in] VARIANT vProcessIdentifier);
    [id(8)] HRESULT LockMediaSampleTrace([out, retval] IUnknown** ppLockUnknown);
    [id(9)] HRESULT GetMediaSampleTrace([in] VARIANT vProcessIdentifier, [out, retval] BSTR* psText);

ResetMediaSampleTrace clears currently recorded in-memory trace. The method is similar to pressing Backspace key in media sample trace UI. Optional argument allows cleanup for specific process (not implemented, everything is reset).

LockMediaSampleTrace creates a COM object which internally references newly created traces so that they don’t get freed with the respective filter graph release. Traces are owned by respective filter graphs and even though are accessible across process boundaries they are disposed with the release of graphs-creators. Unless anyone else accesses and references the traces, in which case they are available for review even is originating graph is already gone. This is the behavior of media sample trace UI, which is automatically referencing new traces as long as UI is visible. Developer can review traces for graphs already terminated (even crashed!). LockMediaSampleTrace method returns an interface pointer which owns an internals thread monitoring and referencing new traces. Once the interface pointer is releases, all referenced traces are released as well.

GetMediaSampleTrace obtains a copy of media sample trace for all or for specific process, in multi-line TSV format similar to text created by user interactively when data is copied to clipboard or saved into file with the help of UI.

All together, a controlling application can create a trace automatically using the following code:

#import "libid:B9EC374B-834B-4DA9-BFB5-C1872CE736FF" raw_interfaces_only // AlaxInfoDirectShowSpy

// ...

CComPtr<AlaxInfoDirectShowSpy::IFilterGraphHelper> pFilterGraphHelper;
pFilterGraphHelper.CoCreateInstance(__uuidof(AlaxInfoDirectShowSpy::FilterGraphHelper));
ULONGLONG nAlaxInfoDirectShowSpyFileVersion = 0;
CComPtr<IUnknown> pMediaSampleTraceLock;
if(pFilterGraphHelper)
{
    const CComQIPtr<AlaxInfoDirectShowSpy::IModuleVersionInformation> pModuleVersionInformation = pFilterGraphHelper;
    if(pModuleVersionInformation)
        _V(pModuleVersionInformation->get_FileVersion((LONGLONG*) &nAlaxInfoDirectShowSpyFileVersion));
    if(nAlaxInfoDirectShowSpyFileVersion >= _VersionInfoHelper::MakeVersion(1, 0, 0, 2060)) // Available in 1.0.0.2060+
    {
        _V(pFilterGraphHelper->ResetMediaSampleTrace(CComVariant((LONG) GetCurrentProcessId())));
        _V(pFilterGraphHelper->LockMediaSampleTrace(&pMediaSampleTraceLock));
    }
}

// DirectShow filter graph runs go here

if(pFilterGraphHelper && nAlaxInfoDirectShowSpyFileVersion >= _VersionInfoHelper::MakeVersion(1, 0, 0, 2060))
    _ATLTRY
    {
        CComBSTR sText;
        __C(pFilterGraphHelper->GetMediaSampleTrace(CComVariant((LONG) GetCurrentProcessId()), &sText));
        if(sText && *sText)
        {
            CPath sPath = Combine(m_sOutputDirectory, _T("MediaSampleTrace.tsv"));
            CAtlFile File;
            __C(File.Create(sPath, GENERIC_WRITE, FILE_SHARE_READ, CREATE_ALWAYS));
            // SUGG: UTF-8 BOM
            CStringA sTextA = Utf8StringFromString(CStringW(sText));
            __C(File.Write(sTextA, (DWORD) sTextA.GetLength()));
        }
    }
    _ATLCATCHALL()
    {
        _Z_EXCEPTION();
    }

Additionally, interactive UI for reviewing media sampla traces (DoMediaSampleTracePropertySheetModal DLL export, see respective BAT file) reviewed a new command to load external TSV file (such as generated with the help of code above!) for review:

DirectShowSpy Media Sample Trace

Once data loaded from file, disabled Refresh link indicates use of non-live data. Context menu command offers option to switch back to live data.

Download links

Utility Clearance: Matrox DSX SDK Capabilities

The tool gathers and formats capture capabilities of hardware accessible via Matrox DSX SDK:

Matrox DSX SDK is a feature-rich development toolkit that enables Matrox’s industry-leading DSX hardware capabilities. OEMs use this hardware and software to build complete solutions that ensure 24/7 on-air reliability in a broadcast environment. Matrox DSX SDK customers have access to unlimited premium support from experienced applications engineers, personalized training courses, design review programs, and feature-customization to meet their specific needs.

Specifically, I used it a lot (X.AVCio, MXO2 series and other) to identify inputs, their properties, genlock, auto-detection of video signal.

(more…)

DirectShowFileMediaSamples Update: Command Line Mode

It appears that the tool was never mentioned before (just mentioned in general software list). The application takes a media file on the input and applies respective DirectShow demultiplexer to list individual media samples.

DirectShowFileMediaSamples UI

  • for MP4 files the application attempts to use GDCL MPEG-4 Demultiplexer first
  • it is possible to filter a specific track/stream
  • ability to copy data to clipboard or save to file
  • drag and drop a file to get it processed

Now the tool has command line mode too:

DirectShowFileMediaSamples-Win32.exe input-path [/no-video] [/no-audio] [output-path]

  • /no-video – excludes video tracks
  • /no-audio – excludes audio tracks

Default output path is input path with extension renamed to .TSV. If DirectShowSpy is installed, the file also contains filter graph information used (esp. media types).

For example,

D:\>DirectShowFileMediaSamples-Win32.exe “F:\Media\Ленинград — Экспонат.mp4”

Typical command line use: troubleshooting export/transcoding sessions where on completion you need a textual information about the export to make sure time accuracy of individual samples: start, stop times, gaps etc.

Interactively one can also achieve the same goal using GraphStudioNext‘s built-in Analyzer Filter.

Download links

Enumeration of DirectShow Capture Capabilities (Video and Audio)

The tool appears to be unmentioned here, and this is to fix the problem.

DirectShowCaptureCapabilities application enumerates video and audio capture devices and lists their typical DirectShow properties, specifically:

  • Moniker names (including USB identification)
  • Pins and property pages
  • Supported interfaces
  • Formats and capabilities available through IAMStreamConfig interface
  • Video and audio devices

DirectShowCaptureCapabilities Screenshot

The utility allows to save output and post it over Internet, what some users already did and here is capabilities of some hardware.

Some of the files might be useful to provide sample data for AM_MEDIA_TYPE structures for typical YUY2, UYVY, MJPG, H264 formats.

Download:

(more…)

ConnectivityWatchdog: Inaccessible TCP based Service Notification via Email

This is dual “yet another…” and utility clearance series. Once again an attempt to easily automate network availability check for a service.

The application is doing periodic TCP connectivity check and sends an email to notify on inaccessible service. If you got systems running at 2+ sites, they can be set up to check one another and inform timely on network issues.

ConnectivityWatchdog Screenshot

Configuration is provided through ConnectivityWatchdog.ini file to be put into the directory of the binary. Sample file is as follows:

[General]

[Host]
$(SomeUniqueIdentifer)=$(FriendlyName), tcp, $(TcpHostOrIpAddress), $(TcpPort)
$(SomeUniqueIdentifer)=$(FriendlyName), tcp, $(TcpHostOrIpAddress), $(TcpPort)

[Probe.TCP]
Connection Time=45 ; Timeout for TCP connection attempt, seconds
Success Idleness Time=60 ; Next attempt in case of successful connection, seconds
Failure Idleness Time=10 ; Next attempt in case of failed connection, seconds
Notification Failure Count=3, 33, 333 ; Numbers of failure attempts which trigger email notification 

[Notification]
Delay=450 ; Delay in notification in order to combine messages into single email, seconds 

[Notification.SMTP]
Host=smtp.gmail.com
Sender=$(UserName)@gmail.com
To=$(UserName)@gmail.com
TLS=1
Authentication Methods=login
Authentication Name=$(UserName)@gmail.com
Authentication Password=$(Password)

Once the application notices TCP connectivity issues, it updates UI status and as configured (see “Notification Failure Count” value above) it will trigger a notification email:

The following are the updates on $(Host) system:

 * Remote host $(FriendlyName) connection SUCCEEDED after 8 failures [2013-12-12 02:15:06 - 4 minutes ago]
 * Remote host $(FriendlyName) connection FAILED 3 times in a row (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond - 0x8007274c) [2013-12-12 02:12:21 - 7 minutes ago]

--
  System: $(Host)
  Time: 2013-12-12 02:19:51
  Agent: Alax.Info ConnectivityWatchdog Application/1.0.0.58

All $(…) values above are placeholders.

The application has a convenient option to enable automatic startup with user logon.

ConnectivityWatchdog Current User Startup

Download links