Yesterday I received a complaint from a Most Important Customer about a weird problem on a few of their digital video recording servers. The problem was that a new version of software, client part, would out of a sudden auto-close without any notice. The problem was quite persistent and basically of not a difficult kind because such fatal problems are automatically logged with a descriptive enough log to identify the cause in an extremely effective way. However the log files appeared to be empty. Another unexpected part was that a few systems showed the issue while the other did not and software worked in a quite expected and stable way there.
After a couple of phone calls and setting up remote access using TightVNC and port forwarding I could see this myself and it was obvious that there is an exception, such as memory access vioaltion, which takes place however on a thread created outside of my code and runnning without a proper guarding section to at the very least catch the exception and log its call stack. The visual symptoms led to supposition that it is something DirectShow related and chances are high that it is either a hardware acceleration issue, or a software compatibility issue. Luckily, hardware acceleration assumption is easy to eliminate by switching off, letting it run without and seeing if there is any difference. There was not.
Then it appeared that while these system were close to clean installation state, there was another DVR software installed also, to be used for tasks not covered by our software. And my first guess was if it installed certain DriectShow filter of the crappy kind, which could interfere with our award winning software, the best of the best. A quick test of renaming “C:\Program Files\Another-DVR” into a different name making all contained binaries unopenable by path showed that nothing changed essentially. So it was GraphEdit, which was downloaded there and a list of installed filters was insepected visually.
There were a few filters that was present in the system but which were obviously installed additionally to the system, as a part of Another-DVR product or in a different way. There was a few filters by I. and then my look stumbled on something which I immediately thought of “That’s It!”. Well, taking into consideration time spent on the problem it was rather “That’s Fucking It!”.
A rename of the hosting file pvmjpg21.dll in %WINDIR%\system32 immediately fixed the problem! Oo-omm-m “PICVideo MJPEG Decompressor” by Pegasus Imaging Corporation, crap! It was one of those moments sung of by Fort Minor:
This is twenty percent crap, eighty percent lame
And a hundred percent reason to remember the name!
However, let us what was going on there. A quick look at the registration information makes it clear the video decoding bastard registered itself under MERIT_PREFERRED (0x00800000) merit and generic MEDIATYPE_Video/MEDIASUBTYPE_NULL media type. Why on earth a properly made codec would claim to be able to decode any video as prefered software? It obviously adds overhead to the entire DirectShow subworld on that system by poking its pin into anything that moves without a proper luck to actually connect.
However, it would be just an overhead, maybe even unnoticable, if the bastard was a thread safe library. But it was unfortunately not. Since my software starts showing multiple video feeds simultaneously, this piece of junk was simultaneously instantiated in multiple threads and just crashed the entire process.
P.S. Bonus: “pvmjpg21.dll crash” on Google