Macs can RDP to Windows Server, but my PC can't?

February 4th at 12:00am

An interesting problem surfaced earlier this year that prevented our systems administrators from using RDP to connect to a windows server if they use a windows laptop. Paradoxically, SysAdmins who run Macs were not affected. It took a little time to track this down and now that I've been through the troubleshooting process I know how to fix it and can see how we wound up in this situation in the first place


References


Symptoms

When you try to connect to a Windows Server 2008 R2 system (other versions may be impacted as well) using the RDP client from any version of windows (I tried 10, 8.1, 7 and XP) you get the following error message:

    "An Internal Error Has Occurred"

When you use Wireshark to examine the connection, you see this:


Broken out the traffic exchange looks like this:

  • Client - SYN
  • Server - SYN, ACK
  • Client - ACK
  • Client - RDP
  • Server - ACK
  • Server - RST, ACK
You can see that the Server resets the connection suddenly and without explanation.

An RDP session initiated from a Mac works and looks like this:


Getting to the bottom of this

This issue appears to be tied to disabling TLS 1.0 on the Windows Server. After TLS is disabled, RDP with NLA ceases to work. The SysAdmin in charge of the system had unchecked the "Allow connections only from computers running Remote Desktop with Network Level Authentication (recommended)" checkbox to allow RDP to work again on his Mac.

The big question I had was why RDP was working from a Mac and not from any of our PCs. The answer lies in the way the Microsoft RDP client for Mac works: It tries to connect to the server using reasonable security settings by setting requestedProtocols to TLS, CredSSP and Early User Authorization Result PDU Supported. When that fails (the mac gets the hard-reset from the server), it tries again and this time it sets requestedProtocols to 0x00000000. Once the Mac RDP client has passed in all zeros for its requestedProtocols, the server responds back with a valid RDP session that can be used.

So, mystery solved: If the Mac encounters an RDP server that does not support TLS, it falls-back to 'RDP Security' and 'dont negotiate' by default. Windows RDP clients do not fall-back to an insecure mode and instead give you a cryptic error message which is meaningless.


Solution

When looking for solutions to this problem I found that I could either re-enable TLS 1.0 -or- enable some obscure flags in a saved .RDP session file to force the windows RDP client to connect. Re-enabling TLS 1.0 did not make sense in our case, so we we created a special saved-RDP session that could be used to connect to the server.

Steps to create a .RDP file that can connect to a Windows Server system that has TLS 1.0 disabled:

  • Open the MS RDP Client (The UWP Client may not work)
  • Configure your RDP session
  • Save the session as a .RDP file
    • Open the .RDP file in a text editor and add these lines to the end: 
      negotiate security layer:i:0
authentication level:i:0
enablecredsspsupport:i:0
prompt for credentials:i:0
  • Double-click the .RDP file and you will be able to connect to the server

I wound up creating a serverfault question about this before identifying the solution. As tends to happen to me, I tend to find the answer to my question relatively shortly after asking a question on a StackExchange site.