While encoding video, Xvid Video Encoder provides an optional status window displaying information on encoding progress.

Since streaming typically takes place in a worker thread, with possibly no windows at all, and no message pump, the window is to be created on GUI thread and the encoder needs to synchronize progress updates with the window.

Such synchronization is a typical point where a deadlock can occur. Carelessly a window message may be sent in a blocking way so that worker thread is waiting while the message is processed on the window thread. If this sending happens at the moment of GUI thread trying to stop worker thread, a deadlock takes place. This problem often comes up when message sending takes place indirectly, e.g. being wrapped by an API call, such as SetWindowText, or as it can be seen below IsDlgButtonChecked.

In order to avoid deadlocks one should never SendMessage from a worker thread. Instead there should be a PostMessage with possibly a wait function which waits for either synchronization event (set by window) or worker thread termination request:

{
  CLock Lock(m_CriticalSection);
  m_sText = sText;
  m_SynchronizationEvent.Reset();
}
PostMessage(...);
HANDLE phObjects[] = { ThreadTerminationEvent, m_SynchronizationEvent };
const DWORD nWaitResult =  WaitForMultipleObjects(..., phObjects, ...);
if(nWaitResult == WAIT_OBJECT_0 + 1 ) // m_SynchronizationEvent
{
  CLock Lock(m_CriticalSection);
  // ...
}

Back to Xvid encoder: trying to abort encoding process, the application deadlocks. Thread checking shows worker thread state:

That is, a described deadlock in action.

While it is already made this way letting deadlock occur, is there any workaround to avoid locking? Yes, there is. Stopping encoding, GUI thread should keep pumping messages while waiting for worker thread termination. Before closing worker thread handle, one needs to signal the thread to terminate and wait using thread handle with MsgWaitForMultipleObjects, so that messages possibly sent while terminating the thread are dispatched to target windows.