Public Key Pinning (HPKP) with NetScaler

I recently decided to implement HTTP Public Key Pinning (HPKP) for some of our external facing Services and our NetScaler Gateway/Access Gateway.

Before we go on I recommend reading through the Wikipedia Article and this Blogpost from Tim Taubert to get a basic understanding of how HPKP works. Tim can also explain it a lot better than I can.

In this Guide I'm assuming you already have some kind of Load balancing vServer or NetScaler Gateway set up and running (including an existing SSL Certificate). So let's start:

In my Example I choose to pin the used Certificate itself and the Intermediate Certificate from RapidSSL where the Certificate itself was issued from. Please make sure you know the consequences when choosing the Certificate Hashes you are going to pin.

First off we need to generate the SHA256 Hashes for the Certificates we want to "pin". I choose to do it via the OpenSSL Interface in the NetScaler GUI but you could also choose to do the same via the NetScaler Shell CLI.

Just copy and paste the following Command into the Command Window and let OpenSSL do its magic. Copy or write down the created Hash as we will need it later on.

x509 -in /nsconfig/ssl/name-of-your-servercertificate.cer -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

Rinse and repeat for the second Certificate (the Backup Hash) you want to pin. In my case this is the issuing Intermediate Certificate from RapidSSL.

x509 -in /nsconfig/ssl/name-of-your-intermediate.cer -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

Next up we need to create the Rewrite Action and the Rewrite Policy to insert the needed HPKP HTTP Header into the HTTP Responses for our vServers and/or NetScaler Gateways.

First lets create a Rewrite Action. I named mine "insert_HPKP_header":

Name: insert_HPKP_header
Type: INSERT_HTTP_HEADER
Header Name: Public-Key-Pins
Expression: "pin-sha256=\"yourcerthashgoeshere\"; pin-sha256=\"yourbackuphashgoeshere\"; max-age=60; includeSubDomains"

Important: For the max-age I recommend starting with a low Value like 60 seconds during the Implementation Phase because if you somehow fuck up your Hashes you are only locked out for 60 Seconds. After successfully testing your HPKP Headers you should then ramp it up to something like 5184000 seconds.

Next Step is to create the needed Rewrite Policy itself. Mine is called enforce_HPKP and is just using TRUE as Expression.

Now its time to bind the newly created Rewrite Policy onto the vServer and/or the NetScaler Gateway Server. If you have multiple Rewrite Actions with different Priorities bound to the vServer (like in my case) make sure to set the "Goto Expression" Option to NEXT or otherwise only the first Rewrite Action will be applied. 

Last but not least you should check if the Public-Key-Pins Header is added successfully to your HTTP Responses. An easy Way is using the SSLLabs.com Scanner and check the Public Key Pinning Test. If everything is working you can now ramp up the max-age Value to 5184000.

If you don't like the NetScaler GUI you can also use the following CLI Commands to implement it:

add rewrite action insert_HPKP_header insert_http_header Public-Key-Pins q{"pin-sha256=\"yourcerthashgoeshere\"; pin-sha256=\"yourbackuphashgoeshere\"; max-age=60; includeSubDomains"} 
add rewrite policy enforce_HPKP TRUE insert_HPKP_header
bind vpn vserver nameofyourvserver -policy enforce_HPKP -priority 100 -gotoPriorityExpression END -type RESPONSE

Update: Implement HPKP Reporting

HPKP includes a Reporting Functionality for the Clients (Browsers) to send a Report in case of an Error. There is a great and free Service called report-uri.io from Scott Helme you can use to avoid having to set up your own Report Server.

After registering at report-uri.io you are given a unique Report URL you have to add to the HTTP HPKP Header as an additional Parameter. In this Example we would use the Public-Key-Pins-Report-Only Header (instead of the Public-Key-Pins Header) without enforcing HPKP itself. This would be a good first Step to see if your calculated Certificate Hashes in the Header are correct without blocking your Site in case of a wrong Hash.

Name: insert_HPKP_header_reportonly
Type: INSERT_HTTP_HEADER
Header Name: Public-Key-Pins-Report-Only
Expression: "pin-sha256=\"yourcerthashgoeshere\"; pin-sha256=\"yourbackuphashgoeshere\"; max-age=60; includeSubDomains; report-uri=\"https://report-uri.io/report/667d4f42bdc27b357863fefdd41574a4\""

The CLI Command for creating the would be

add rewrite action insert_HPKP_header_reportonly insert_http_header Public-Key-Pins-Report-Only q{"pin-sha256=\"yourcerthashgoeshere\"; pin-sha256=\"yourbackuphashgoeshere\"; max-age=2592000; includeSubDomains; report-uri=\"https://report-uri.io/report/667d4f42bdc27b357863fefdd41574a4\""}
add rewrite policy enable_HPKP_Reporting TRUE insert_HPKP_header_reportonly
bind vpn vserver nameofyourvserver -policy enable_HPKP_Reporting -priority 100 -gotoPriorityExpression END -type RESPONSE 

 As always Feedback and Comments are greatly appreciated.

Perfect Forward Secrecy and NetScaler MPX Revisited

My last Blogpost regarding Perfect Forward Secrecy on NetScaler got a lot of Comments and in the meantime Citrix released a new NetScaler Firmware Versions (10.5-53.9.nc) which now enables us to use ECDHE Ciphers even on "low end" NetScaler MPX Models like the MPX 5500.

With the new Firmware it's now possible to enable PFS for all modern Clients/Browser and receive an A+ in the SSLLabs Test.

Okay, let's start:

First step is to create a new Cipher Group where we will bind all the new Ciphers we want and need.

Unfortunately changing the Order via the NetScaler GUI is not very user-friendly (Hey Citrix, please enable us to easily change the Cipher Order) so I resorted to the Command Line. Below are the commands to create a new Cipher Group called "THEN-High" and bind the Ciphers in a preferred Order (I'm open for Discussions on the Order and Ciphers if you disagree or got a better Order):

Important: Make sure to order the Ciphers in the preferred order (thanks again to David Chivers for pointing this out) or you may not get Forward Secrecy to work on some Clients/Browser.

Cipher List for MPX/SDX and VPX (starting with Build 11.0-65.31) Appliances:

Legacy Cipher List for MPX/SDX and VPX (starting with Build 11.0-65.31) Appliances (includes Support for WinXP, Java and Android 2.x):

Cipher List for VPX starting with Build 10.5-57.7 up to 11.0-64.34:

Legacy Cipher List for VPX Builds starting with Build 10.5-57.7 up to 11.0-64.34 (includes Support for WinXP, Java and Android 2.x):

Since NetScaler 11 Citrix now seems to enforce compatible Ciphers on the VPX Platform.

All SHA2 and GCM Cipher Suites are not (yet?) supported on the VPX. Since NetScaler Build 11.0-65.31 the VPX now supports all AES-GCM/SHA2 Ciphers. Hooray!

And thats what it looks like in the GUI:

Don't forget to create a Diffie Hellmann Key (2048bit Keysize recommended) and configure it under the SSL Parameters on the vServer (or use the new SSL Profiles Feature). You can find a detailed How-To in my old Blogpost here.

The next Step is to bind the newly created Cipher Group to the vServer. When I did this via the GUI my Cipher Order wasn't honored (Bug? Anyone else?) so again I needed to use the Command Line:

bind ssl vserver <vServerName> -cipherName THEN-High

(Update: With the 10.5-57.7 NetScaler Build this Bug has been fixed by Citrix! You can now use the GUI to bind the Cipher Group.)

Don't forget to bind ECC Curves on the vServer or you won't be able to use the newer ECDHE Ciphers. I recommend binding the Curves P_256, P_384 and P_521.

bind ssl vserver <vServerName> -eccCurveName P_256 P_384 P_521

Let's take a look in the GUI (note the correct Cipher Order):

After configuring everything correctly you should check it using the SSLLabs Test. Here is my Result with the latest Firmware and the described Settings:

(Update: With the 10.5-57.7 NetScaler Build you can now get the A+ Rating in the SSLLabs Test because Citrix added TLS_FALLBACK_SCSV Support)

As always feel free to leave a Comment.

Changelog:

#1 - 19.05.2015 - Cipher Reordering Bug has been fixed by Citrix with the 10.5-57.7 NetScaler Build
#2 - 19.05.2015 - Note regarding TLS_FALLBACK_SCSV Support in the 10.5-57.7 NetScaler Build
#3 - 22.05.2015 - Added specific Cipher List for NetScaler VPX with the 10.5-57.7 NetScaler Build
#4 - 21.06.2015 - Added Note regarding binding ECC Curves.
#5 - 30.06.2015 - Updated MPX/SDX Cipher List with some new Ciphers for broader Client Support.
#6 - 01.07.2015 - Changed VPX Cipher Suite for NetScaler 11 regarding the support of SHA2 and GCM Cipher Suites
#7 - 10.09.2015 - Added MPX/SDX Cipher List without DHE Ciphers (Logjam)
#8 - 02.03.2016 - Added 2 new Ciphers for the VPX Platform Cipher Group
#9 - 04.03.2016 - moved Code to GitHub Gists for Versioning and Change Tracking
#10 - 22.03.2016 - Since NetScaler 11.0-65.31 the VPX now supports all AES-GCM/SHA2 Ciphers. Hooray!

NetScaler Gateway: Internal Server Error 43531

After upgrading a HA Pair of NetScalers from 10.1 to the latest 10.5 Build (10.5-52.11 to be exact) I stumbled upon a critical Error when trying to access the Receiver for Web Site.

After authenticating successfully I would be presented with an Http/1.1 Internal Server Error 43531

After some Troubleshooting I could narrow it down to a specific Setting in my Session Policies for the NetScaler Gateway. If you used my previous Blogpost on how to configure the NetScaler Gateway for Storefront Remote Access you might also run into the same Problem.

The following Setting was working with the NetScaler 10.1 Builds:

After upgrading to the 10.5 NetScaler Build you need to edit your "Receiver for Web" Session Policy and move the Storefront URL from the "Home Page" Field under Client Experience to the "Web Interface Address" Field under Published Applications. You also need to enable "ICA Proxy" and set it to ON. Below are two Screenshots with the corrected Session Policy Settings:

I'm not sure if this is intended or just a Bug in the Code. Feel free to let me know in the Comments if you encountered the same Problem when upgrading from 10.1 to 10.5.

Enable Forward Secrecy and Secure Ciphers on a NetScaler MPX

See my new Blogpost for an updated Configuration with full ECDHE Cipher Support

I recently "hardened" our public facing NetScaler vServers (mainly our Access Gateway) and thought I'd share the Steps I took to accomplish that.

I wanted to enable/use Perfect Forward Secrecy (PFS) on our Access Gateway vServer and only use strong and secure Ciphers (no more RC4 with TLS 1.x).

  • Create a new Cipher Group with secure Ciphers

So let's create a new Cipher Group on the NetScaler. You can do this unter the "Traffic Management" -> "SSL" -> "Cipher Group" Node in the GUI.

 Name your new Cipher Group and add the following Cipher Suites:

  • TLS1-DHE-RSA-AES-128-CBC-SHA (provides Forward Secrecy (PFS) for some Browsers)
  • TLS1-DHE-RSA-AES-256-CBC-SHA (provides Forward Secrecy (PFS) for some Browsers)
  • TLS1-AES-128-CBC-SHA
  • TLS1-AES-256-CBC-SHA
  • SSL3-RC4-SHA (remove if you don't need SSL3 Compatibility for WinXP and only want to use TLS)
  • SSL3-DES-CBC3-SHA (use this Cipher to enable IE8 on WinXP to access your vServer)

Your newly created Cipher Group should look like this:

  • Create a Diffie Hellman Key (DH) for PFS

The first step is to create a Diffie Hellman Key under the Traffic Management -> SSL Node.

I created a 1024-bit DH Key because a larger DH Key will decrease the SSL Performance of the NetScaler and I'm not sure if the SSL Accelerator Card in the MPX 5500 supports DH Keys larger than 1024 (maybe a Citrite can enlighten us in the Comments). It's currently a compromise between Security and Performance.

Update: Since NetScaler Firmware 10.5 DH Keys with 2048 Bits are supported to be offloaded to the SSL Accelerator Cards in MPX Appliances. So use a 2048 bit DH Key instead of a 1024 bit Key if possible.

  • Enable the new Cipher Suite and DH Key on your vServers

The last step is to enable the new Cipher Suite and the DH Key on the vServers you want to provide PFS. Let's open up a vServer (your Access Gateway vServer for example) and configure it:

First let's go into the Ciphers Menu. Clear out everything under "Configured Cipher Groups" and "Configured Ciphers" and only add your newly created Cipher Group (called THEN-Default in my example). It should look like this:

Now let's pop into the SSL Parameters Menu from your vServer. Make sure to tick the "Enable DH Param" Box and select the newly created DH Key File. Also make sure to use a fairly high number under the "Refresh Count", I choose a value of 1000. When finished it should look something like this:

Now save your running config and go ahead and test your public vServer with the SSL Labs Test Suite. When configured correctly you should see the following results from the SSL Labs Test:

The green FS next to a Browser/OS Combination means PFS can and will be used when connecting to your vServer, just the way we like it. 

Adding Two-Factor Authentication to your NetScaler Gateway

We recently added Two-Factor Authentication to our NetScaler Gateway (formerly known as Access Gateway) and that meant I had to make a few Configuration Changes to the Session and Authentication Policies on our NetScalers to get the Authentication working from Mobile Devices and the native Citrix Receiver.

The Configuration of the NetScaler Gateway and the required Policies (without Two-Factor Authentication) in this Post are all based upon my previous Blogpost so make sure to read that first if you are starting from scratch.

In the first Step we have to edit and/or create our existing LDAP and RADIUS Authentication Policies on the NetScaler. In my case I already had two LDAP Authentication Policies configured, one for each of the two Domain Controllers we are authenticating against, so go ahead and open those up.

They should look something like this with an ns_true as Expression.

So go ahead and remove the ns_true Expression. Now change it from Match Any Expression to Advanced Free-Form in the Dropdown Menu so you can easily Copy and Paste the following Expression into the Field.

Expression: REQ.HTTP.HEADER User-Agent NOTCONTAINS CitrixReceiver

Okay, so we edited our existing LDAP Authentication Policy. Now create a new LDAP Authentication and append something like _mobile in the Name to distinguish it from our existing LDAP Policy and use the following Expression:

Expression: REQ.HTTP.HEADER User-Agent CONTAINS CitrixReceiver 

For every AD Controller you want to authenticate against you should now have 2 LDAP Policies. One is for normal Devices (named auth_then_ldaps_servername01 in my case) and one is for Mobile Devices (named auth_then_ldaps_servername01_mobile) in my case. When finished it should look similar as in the following Screenshot:

Next Step is to create the RADIUS Authentication Policies we need. For every RADIUS Server you want to authenticate against we will have to create 2 RADIUS Authentication Policies. In my example I will only authenticate against a single RADIUS Server. 

I'm not covering the creation of the "Authentication Server" with the Shared Secret and the IP and Port of the Radius Server as this depends on the Product you will be using.

Create/Edit the RADIUS Authentication Policy for the normal Devices and use the following Expression:

Expression: REQ.HTTP.HEADER User-Agent NOTCONTAINS CitrixReceiver

Let's create another RADIUS Authentication Policy for Mobile Devices with the following Expression:

Expression: REQ.HTTP.HEADER User-Agent CONTAINS CitrixReceiver 

You should end up with something like this for your RADIUS Authentication Policies:

Now we have to bind those edited and newly created Authentication Policies to the NetScaler Gateway (Access Gateway) vServer. First let's bind the Primary Authentication Policies in the right order and make sure to give the RADIUS Mobile Authentication Policy the lowest Priority (this means it's the first to be hit and evaluated - the NetScaler through the Policies from the lowest to the highest Number).

The Order should be (lowest to highest priority):

  • RADIUS Mobile Authentication Policy
  • LDAP Authentication Policy (AD Controller #1)
  • LDAP Authentication Policy (AD Controller #2)

You can refer to the following Screenshot for the correct Order:

We have to repeat the same Steps for the Secondary Authentication Policies but this time in another Priority Order. Bind the following Policies in this Order (lowest to highest):

  • LDAP Mobile Authentication Policy (AD Controller #1)
  • LDAP Mobile Authentication Policy (AD Controller #2)
  • RADIUS Authentication Policy

Every Authentication Policy (6 in my Example (4 LDAP and 2 Radius)) should only be bound once under the Primary and Secondary Authentication Policies.

Last step is to edit your existing Session Policy for the native Receiver (this Policy is used by the Mobile Receivers on Android and iOS and also by the native Receiver for Windows). So head over to the Session Policies bound to your NetScaler Gateway and edit the pol_ReceiverNative Policy.

Change the Credential Index to Secondary. This is needed because the Mobile Receivers and the Native Receiver for Windows sends the Token Code (RADIUS) before he sends the Username/Password for LDAP.

If something is confusing or you have question regarding this Article I would appreciate Feedback in the Comments or via Twitter.

Troubleshooting Tip: If you want to see which Authentication and Session Policies are hit/used when connecting and authenticating via the NetScaler Gateway you can use the following Commands from this Citrix Support Article. Especially the following command is very useful:

nsconmsg –d current –g pol_hits