I’m not having any issues on Win 10 64-bit.
Make sure you keep the rxtxSerial.dll file in the same folder as Breakout Server64.exe.
If that is not the problem then try using the Windows Event Viewer application to see if you can see any logs from Breakout Server when you attempt to open the exe file.