We had this problem on our Designer machine (unresponsive Debug Monitor) and I put in a ticket, a helpful Ross tech support dude TeamViewered into our machine and fixed it. Here's what he appears to have done on the command line (ie. Command Prompt app). Obviously adjust for the correct path where your xpDbgMon.exe file actually is on your system, if different, for example it may be in C:\Program Files\XPressionStudio. You may also need to have admin rights on your login to perform this, I'm not sure. As mentioned above, make sure Preferences>Advanced>Enable debug monitor for scripting is checked first.
Hope this helps someone because not having debugMessage() available makes it really hard to debug anything. As a backup, you can use the engine.writeToLog() function, which will show up in Tools>Debug>Live Log, but there's a lot of noise in that channel, where Debug Monitor just gives you what you send to it.
cd C:\Program Files\XPressionDesigner
xpDbgMon.exe /regserver
Debug Monitor now works, and continued to work through a system restart.
#XPression