NetScaler Gateway: Internal Server Error 43531

After upgrading a HA Pair of NetScalers from 10.1 to the latest 10.5 Build (10.5-52.11 to be exact) I stumbled upon a critical Error when trying to access the Receiver for Web Site.

After authenticating successfully I would be presented with an Http/1.1 Internal Server Error 43531

After some Troubleshooting I could narrow it down to a specific Setting in my Session Policies for the NetScaler Gateway. If you used my previous Blogpost on how to configure the NetScaler Gateway for Storefront Remote Access you might also run into the same Problem.

The following Setting was working with the NetScaler 10.1 Builds:

After upgrading to the 10.5 NetScaler Build you need to edit your "Receiver for Web" Session Policy and move the Storefront URL from the "Home Page" Field under Client Experience to the "Web Interface Address" Field under Published Applications. You also need to enable "ICA Proxy" and set it to ON. Below are two Screenshots with the corrected Session Policy Settings:

I'm not sure if this is intended or just a Bug in the Code. Feel free to let me know in the Comments if you encountered the same Problem when upgrading from 10.1 to 10.5.

Enable Forward Secrecy and Secure Ciphers on a NetScaler MPX

See my new Blogpost for an updated Configuration with full ECDHE Cipher Support

I recently "hardened" our public facing NetScaler vServers (mainly our Access Gateway) and thought I'd share the Steps I took to accomplish that.

I wanted to enable/use Perfect Forward Secrecy (PFS) on our Access Gateway vServer and only use strong and secure Ciphers (no more RC4 with TLS 1.x).

  • Create a new Cipher Group with secure Ciphers

So let's create a new Cipher Group on the NetScaler. You can do this unter the "Traffic Management" -> "SSL" -> "Cipher Group" Node in the GUI.

 Name your new Cipher Group and add the following Cipher Suites:

  • TLS1-DHE-RSA-AES-128-CBC-SHA (provides Forward Secrecy (PFS) for some Browsers)
  • TLS1-DHE-RSA-AES-256-CBC-SHA (provides Forward Secrecy (PFS) for some Browsers)
  • TLS1-AES-128-CBC-SHA
  • TLS1-AES-256-CBC-SHA
  • SSL3-RC4-SHA (remove if you don't need SSL3 Compatibility for WinXP and only want to use TLS)
  • SSL3-DES-CBC3-SHA (use this Cipher to enable IE8 on WinXP to access your vServer)

Your newly created Cipher Group should look like this:

  • Create a Diffie Hellman Key (DH) for PFS

The first step is to create a Diffie Hellman Key under the Traffic Management -> SSL Node.

I created a 1024-bit DH Key because a larger DH Key will decrease the SSL Performance of the NetScaler and I'm not sure if the SSL Accelerator Card in the MPX 5500 supports DH Keys larger than 1024 (maybe a Citrite can enlighten us in the Comments). It's currently a compromise between Security and Performance.

Update: Since NetScaler Firmware 10.5 DH Keys with 2048 Bits are supported to be offloaded to the SSL Accelerator Cards in MPX Appliances. So use a 2048 bit DH Key instead of a 1024 bit Key if possible.

  • Enable the new Cipher Suite and DH Key on your vServers

The last step is to enable the new Cipher Suite and the DH Key on the vServers you want to provide PFS. Let's open up a vServer (your Access Gateway vServer for example) and configure it:

First let's go into the Ciphers Menu. Clear out everything under "Configured Cipher Groups" and "Configured Ciphers" and only add your newly created Cipher Group (called THEN-Default in my example). It should look like this:

Now let's pop into the SSL Parameters Menu from your vServer. Make sure to tick the "Enable DH Param" Box and select the newly created DH Key File. Also make sure to use a fairly high number under the "Refresh Count", I choose a value of 1000. When finished it should look something like this:

Now save your running config and go ahead and test your public vServer with the SSL Labs Test Suite. When configured correctly you should see the following results from the SSL Labs Test:

The green FS next to a Browser/OS Combination means PFS can and will be used when connecting to your vServer, just the way we like it. 

Adding Two-Factor Authentication to your NetScaler Gateway

We recently added Two-Factor Authentication to our NetScaler Gateway (formerly known as Access Gateway) and that meant I had to make a few Configuration Changes to the Session and Authentication Policies on our NetScalers to get the Authentication working from Mobile Devices and the native Citrix Receiver.

The Configuration of the NetScaler Gateway and the required Policies (without Two-Factor Authentication) in this Post are all based upon my previous Blogpost so make sure to read that first if you are starting from scratch.

In the first Step we have to edit and/or create our existing LDAP and RADIUS Authentication Policies on the NetScaler. In my case I already had two LDAP Authentication Policies configured, one for each of the two Domain Controllers we are authenticating against, so go ahead and open those up.

They should look something like this with an ns_true as Expression.

So go ahead and remove the ns_true Expression. Now change it from Match Any Expression to Advanced Free-Form in the Dropdown Menu so you can easily Copy and Paste the following Expression into the Field.

Expression: REQ.HTTP.HEADER User-Agent NOTCONTAINS CitrixReceiver

Okay, so we edited our existing LDAP Authentication Policy. Now create a new LDAP Authentication and append something like _mobile in the Name to distinguish it from our existing LDAP Policy and use the following Expression:

Expression: REQ.HTTP.HEADER User-Agent CONTAINS CitrixReceiver 

For every AD Controller you want to authenticate against you should now have 2 LDAP Policies. One is for normal Devices (named auth_then_ldaps_servername01 in my case) and one is for Mobile Devices (named auth_then_ldaps_servername01_mobile) in my case. When finished it should look similar as in the following Screenshot:

Next Step is to create the RADIUS Authentication Policies we need. For every RADIUS Server you want to authenticate against we will have to create 2 RADIUS Authentication Policies. In my example I will only authenticate against a single RADIUS Server. 

I'm not covering the creation of the "Authentication Server" with the Shared Secret and the IP and Port of the Radius Server as this depends on the Product you will be using.

Create/Edit the RADIUS Authentication Policy for the normal Devices and use the following Expression:

Expression: REQ.HTTP.HEADER User-Agent NOTCONTAINS CitrixReceiver

Let's create another RADIUS Authentication Policy for Mobile Devices with the following Expression:

Expression: REQ.HTTP.HEADER User-Agent CONTAINS CitrixReceiver 

You should end up with something like this for your RADIUS Authentication Policies:

Now we have to bind those edited and newly created Authentication Policies to the NetScaler Gateway (Access Gateway) vServer. First let's bind the Primary Authentication Policies in the right order and make sure to give the RADIUS Mobile Authentication Policy the lowest Priority (this means it's the first to be hit and evaluated - the NetScaler through the Policies from the lowest to the highest Number).

The Order should be (lowest to highest priority):

  • RADIUS Mobile Authentication Policy
  • LDAP Authentication Policy (AD Controller #1)
  • LDAP Authentication Policy (AD Controller #2)

You can refer to the following Screenshot for the correct Order:

We have to repeat the same Steps for the Secondary Authentication Policies but this time in another Priority Order. Bind the following Policies in this Order (lowest to highest):

  • LDAP Mobile Authentication Policy (AD Controller #1)
  • LDAP Mobile Authentication Policy (AD Controller #2)
  • RADIUS Authentication Policy

Every Authentication Policy (6 in my Example (4 LDAP and 2 Radius)) should only be bound once under the Primary and Secondary Authentication Policies.

Last step is to edit your existing Session Policy for the native Receiver (this Policy is used by the Mobile Receivers on Android and iOS and also by the native Receiver for Windows). So head over to the Session Policies bound to your NetScaler Gateway and edit the pol_ReceiverNative Policy.

Change the Credential Index to Secondary. This is needed because the Mobile Receivers and the Native Receiver for Windows sends the Token Code (RADIUS) before he sends the Username/Password for LDAP.

If something is confusing or you have question regarding this Article I would appreciate Feedback in the Comments or via Twitter.

Troubleshooting Tip: If you want to see which Authentication and Session Policies are hit/used when connecting and authenticating via the NetScaler Gateway you can use the following Commands from this Citrix Support Article. Especially the following command is very useful:

nsconmsg –d current –g pol_hits

Implement HSTS/STS on your NetScaler Gateway

Today another small Blogpost on making your NetScaler Gateway / Access Gateway Enviroment (or any other Website published through a NetScaler) a little more secure by implementing HTTP Strict Transport Security (HSTS or STS). This should take you no longer than 15 Minutes to implement.

Let's get started. First we have to create a Rewrite Action and a Rewrite Policy. We start with the Rewrite Action:

Configure the Rewrite Action with the following Parameters as shown in the Screenshot.

  • Name: Insert_STS_Header (feel free to name it whatever you want to)
  • Type: INSERT_HTTP_HEADER
  • Header Name: Strict-Transport-Security
  • String: "max-age=157680000"

The max-age Value is given in seconds.

For all the CLI Fans here are the needed Snippets:

The next Step is to create the Rewrite Policy.

Create the Rewrite Policy with the following Parameters (see the following Screenshot)

  • Name: Enforce_STS
  • Action: select the previously created Rewrite Action
  • Expression: TRUE

Last Configuration Step is to bind it to your Netscaler Gateway / Access Gateway vServer under Policies -> Rewrite (Response) and make sure to save your running Config.

To make sure you configure everything correct you can use the SSL Scanner from Qualys Labs. You should see something similar to this:

Feel free to read my previous Blogpost on how to disable TLS/SSL Client Renegotiations on NetScaler for more SSL/TLS Security.

Disable TLS/SSL Client Renegotiation on NetScaler

Just a quick Reminder for securing your Citrix Enviroment: If you are running a NetScaler Gateway for your Remote Access to your Citrix XenApp/XenDesktop Enviroment (or pretty much anything else) make sure to disable secure and nonsecure TLS/SSL renegotiations initiated by a client to avoid potential Denial of Service Attacks.

Unfortunately the default setting (as of Netscaler Release 10.1) is set to an unsecure setting of allowing TLS/SSL Renegotiations.

You can set the appropriate Options via the NetScaler GUI under the "Advanced SSL Settings".

Make sure to select "FRONTEND_CLIENT" "ALL" in the Dropdown menu for Deny SSL Renegotiation.

Update: Since NetScaler 10.5 (for fresh Installations) the new Default is to deny all SSL Renegotiations. So the new recommendation is to set "Deny SSL Renegotian" to ALL instead of FRONTEND_CLIENT if you have upgraded your NetScalers from <10.5.

Thanks to Boris Groenhout for letting me know via the Comments.

Afterwards make sure everything is set appropriately by using the SSL Scanner from Qualys Labs under: https://www.ssllabs.com/ssltest. If you have configured everything correct (regarding TLS/SSL Renegotiation) you should see the following in the Report generated by the SSL Scanner:

You can also use the SSL Scanner to check if your SSL Certificate Chaining is working correctly and if you are using secure Cipher Suites on your NetScaler Gateway.

Updated 03.02.2016 - Added Notice regarding new Defaults for fresh NetScaler 10.5 Installations.