Public Key Pinning (HPKP) with NetScaler

I recently decided to implement HTTP Public Key Pinning (HPKP) for some of our external facing Services and our NetScaler Gateway/Access Gateway.

Before we go on I recommend reading through the Wikipedia Article and this Blogpost from Tim Taubert to get a basic understanding of how HPKP works. Tim can also explain it a lot better than I can.

In this Guide I'm assuming you already have some kind of Load balancing vServer or NetScaler Gateway set up and running (including an existing SSL Certificate). So let's start:

In my Example I choose to pin the used Certificate itself and the Intermediate Certificate from RapidSSL where the Certificate itself was issued from. Please make sure you know the consequences when choosing the Certificate Hashes you are going to pin.

First off we need to generate the SHA256 Hashes for the Certificates we want to "pin". I choose to do it via the OpenSSL Interface in the NetScaler GUI but you could also choose to do the same via the NetScaler Shell CLI.

Just copy and paste the following Command into the Command Window and let OpenSSL do its magic. Copy or write down the created Hash as we will need it later on.

x509 -in /nsconfig/ssl/name-of-your-servercertificate.cer -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

Rinse and repeat for the second Certificate (the Backup Hash) you want to pin. In my case this is the issuing Intermediate Certificate from RapidSSL.

x509 -in /nsconfig/ssl/name-of-your-intermediate.cer -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

Next up we need to create the Rewrite Action and the Rewrite Policy to insert the needed HPKP HTTP Header into the HTTP Responses for our vServers and/or NetScaler Gateways.

First lets create a Rewrite Action. I named mine "insert_HPKP_header":

Name: insert_HPKP_header
Type: INSERT_HTTP_HEADER
Header Name: Public-Key-Pins
Expression: "pin-sha256=\"yourcerthashgoeshere\"; pin-sha256=\"yourbackuphashgoeshere\"; max-age=60; includeSubDomains"

Important: For the max-age I recommend starting with a low Value like 60 seconds during the Implementation Phase because if you somehow fuck up your Hashes you are only locked out for 60 Seconds. After successfully testing your HPKP Headers you should then ramp it up to something like 5184000 seconds.

Next Step is to create the needed Rewrite Policy itself. Mine is called enforce_HPKP and is just using TRUE as Expression.

Now its time to bind the newly created Rewrite Policy onto the vServer and/or the NetScaler Gateway Server. If you have multiple Rewrite Actions with different Priorities bound to the vServer (like in my case) make sure to set the "Goto Expression" Option to NEXT or otherwise only the first Rewrite Action will be applied. 

Last but not least you should check if the Public-Key-Pins Header is added successfully to your HTTP Responses. An easy Way is using the SSLLabs.com Scanner and check the Public Key Pinning Test. If everything is working you can now ramp up the max-age Value to 5184000.

If you don't like the NetScaler GUI you can also use the following CLI Commands to implement it:

add rewrite action insert_HPKP_header insert_http_header Public-Key-Pins q{"pin-sha256=\"yourcerthashgoeshere\"; pin-sha256=\"yourbackuphashgoeshere\"; max-age=60; includeSubDomains"} 
add rewrite policy enforce_HPKP TRUE insert_HPKP_header
bind vpn vserver nameofyourvserver -policy enforce_HPKP -priority 100 -gotoPriorityExpression END -type RESPONSE

Update: Implement HPKP Reporting

HPKP includes a Reporting Functionality for the Clients (Browsers) to send a Report in case of an Error. There is a great and free Service called report-uri.io from Scott Helme you can use to avoid having to set up your own Report Server.

After registering at report-uri.io you are given a unique Report URL you have to add to the HTTP HPKP Header as an additional Parameter. In this Example we would use the Public-Key-Pins-Report-Only Header (instead of the Public-Key-Pins Header) without enforcing HPKP itself. This would be a good first Step to see if your calculated Certificate Hashes in the Header are correct without blocking your Site in case of a wrong Hash.

Name: insert_HPKP_header_reportonly
Type: INSERT_HTTP_HEADER
Header Name: Public-Key-Pins-Report-Only
Expression: "pin-sha256=\"yourcerthashgoeshere\"; pin-sha256=\"yourbackuphashgoeshere\"; max-age=60; includeSubDomains; report-uri=\"https://report-uri.io/report/667d4f42bdc27b357863fefdd41574a4\""

The CLI Command for creating the would be

add rewrite action insert_HPKP_header_reportonly insert_http_header Public-Key-Pins-Report-Only q{"pin-sha256=\"yourcerthashgoeshere\"; pin-sha256=\"yourbackuphashgoeshere\"; max-age=2592000; includeSubDomains; report-uri=\"https://report-uri.io/report/667d4f42bdc27b357863fefdd41574a4\""}
add rewrite policy enable_HPKP_Reporting TRUE insert_HPKP_header_reportonly
bind vpn vserver nameofyourvserver -policy enable_HPKP_Reporting -priority 100 -gotoPriorityExpression END -type RESPONSE 

 As always Feedback and Comments are greatly appreciated.

Perfect Forward Secrecy and NetScaler MPX Revisited

My last Blogpost regarding Perfect Forward Secrecy on NetScaler got a lot of Comments and in the meantime Citrix released a new NetScaler Firmware Versions (10.5-53.9.nc) which now enables us to use ECDHE Ciphers even on "low end" NetScaler MPX Models like the MPX 5500.

With the new Firmware it's now possible to enable PFS for all modern Clients/Browser and receive an A+ in the SSLLabs Test.

Okay, let's start:

First step is to create a new Cipher Group where we will bind all the new Ciphers we want and need.

Unfortunately changing the Order via the NetScaler GUI is not very user-friendly (Hey Citrix, please enable us to easily change the Cipher Order) so I resorted to the Command Line. Below are the commands to create a new Cipher Group called "THEN-High" and bind the Ciphers in a preferred Order (I'm open for Discussions on the Order and Ciphers if you disagree or got a better Order):

Important: Make sure to order the Ciphers in the preferred order (thanks again to David Chivers for pointing this out) or you may not get Forward Secrecy to work on some Clients/Browser.

Cipher List for MPX/SDX and VPX (starting with Build 11.0-65.31) Appliances:

Legacy Cipher List for MPX/SDX and VPX (starting with Build 11.0-65.31) Appliances (includes Support for WinXP, Java and Android 2.x):

Cipher List for VPX starting with Build 10.5-57.7 up to 11.0-64.34:

Legacy Cipher List for VPX Builds starting with Build 10.5-57.7 up to 11.0-64.34 (includes Support for WinXP, Java and Android 2.x):

Since NetScaler 11 Citrix now seems to enforce compatible Ciphers on the VPX Platform.

All SHA2 and GCM Cipher Suites are not (yet?) supported on the VPX. Since NetScaler Build 11.0-65.31 the VPX now supports all AES-GCM/SHA2 Ciphers. Hooray!

And thats what it looks like in the GUI:

Don't forget to create a Diffie Hellmann Key (2048bit Keysize recommended) and configure it under the SSL Parameters on the vServer (or use the new SSL Profiles Feature). You can find a detailed How-To in my old Blogpost here.

The next Step is to bind the newly created Cipher Group to the vServer. When I did this via the GUI my Cipher Order wasn't honored (Bug? Anyone else?) so again I needed to use the Command Line:

bind ssl vserver <vServerName> -cipherName THEN-High

(Update: With the 10.5-57.7 NetScaler Build this Bug has been fixed by Citrix! You can now use the GUI to bind the Cipher Group.)

Don't forget to bind ECC Curves on the vServer or you won't be able to use the newer ECDHE Ciphers. I recommend binding the Curves P_256, P_384 and P_521.

bind ssl vserver <vServerName> -eccCurveName P_256 P_384 P_521

Let's take a look in the GUI (note the correct Cipher Order):

After configuring everything correctly you should check it using the SSLLabs Test. Here is my Result with the latest Firmware and the described Settings:

(Update: With the 10.5-57.7 NetScaler Build you can now get the A+ Rating in the SSLLabs Test because Citrix added TLS_FALLBACK_SCSV Support)

As always feel free to leave a Comment.

Changelog:

#1 - 19.05.2015 - Cipher Reordering Bug has been fixed by Citrix with the 10.5-57.7 NetScaler Build
#2 - 19.05.2015 - Note regarding TLS_FALLBACK_SCSV Support in the 10.5-57.7 NetScaler Build
#3 - 22.05.2015 - Added specific Cipher List for NetScaler VPX with the 10.5-57.7 NetScaler Build
#4 - 21.06.2015 - Added Note regarding binding ECC Curves.
#5 - 30.06.2015 - Updated MPX/SDX Cipher List with some new Ciphers for broader Client Support.
#6 - 01.07.2015 - Changed VPX Cipher Suite for NetScaler 11 regarding the support of SHA2 and GCM Cipher Suites
#7 - 10.09.2015 - Added MPX/SDX Cipher List without DHE Ciphers (Logjam)
#8 - 02.03.2016 - Added 2 new Ciphers for the VPX Platform Cipher Group
#9 - 04.03.2016 - moved Code to GitHub Gists for Versioning and Change Tracking
#10 - 22.03.2016 - Since NetScaler 11.0-65.31 the VPX now supports all AES-GCM/SHA2 Ciphers. Hooray!

Enable Forward Secrecy and Secure Ciphers on a NetScaler MPX

See my new Blogpost for an updated Configuration with full ECDHE Cipher Support

I recently "hardened" our public facing NetScaler vServers (mainly our Access Gateway) and thought I'd share the Steps I took to accomplish that.

I wanted to enable/use Perfect Forward Secrecy (PFS) on our Access Gateway vServer and only use strong and secure Ciphers (no more RC4 with TLS 1.x).

  • Create a new Cipher Group with secure Ciphers

So let's create a new Cipher Group on the NetScaler. You can do this unter the "Traffic Management" -> "SSL" -> "Cipher Group" Node in the GUI.

 Name your new Cipher Group and add the following Cipher Suites:

  • TLS1-DHE-RSA-AES-128-CBC-SHA (provides Forward Secrecy (PFS) for some Browsers)
  • TLS1-DHE-RSA-AES-256-CBC-SHA (provides Forward Secrecy (PFS) for some Browsers)
  • TLS1-AES-128-CBC-SHA
  • TLS1-AES-256-CBC-SHA
  • SSL3-RC4-SHA (remove if you don't need SSL3 Compatibility for WinXP and only want to use TLS)
  • SSL3-DES-CBC3-SHA (use this Cipher to enable IE8 on WinXP to access your vServer)

Your newly created Cipher Group should look like this:

  • Create a Diffie Hellman Key (DH) for PFS

The first step is to create a Diffie Hellman Key under the Traffic Management -> SSL Node.

I created a 1024-bit DH Key because a larger DH Key will decrease the SSL Performance of the NetScaler and I'm not sure if the SSL Accelerator Card in the MPX 5500 supports DH Keys larger than 1024 (maybe a Citrite can enlighten us in the Comments). It's currently a compromise between Security and Performance.

Update: Since NetScaler Firmware 10.5 DH Keys with 2048 Bits are supported to be offloaded to the SSL Accelerator Cards in MPX Appliances. So use a 2048 bit DH Key instead of a 1024 bit Key if possible.

  • Enable the new Cipher Suite and DH Key on your vServers

The last step is to enable the new Cipher Suite and the DH Key on the vServers you want to provide PFS. Let's open up a vServer (your Access Gateway vServer for example) and configure it:

First let's go into the Ciphers Menu. Clear out everything under "Configured Cipher Groups" and "Configured Ciphers" and only add your newly created Cipher Group (called THEN-Default in my example). It should look like this:

Now let's pop into the SSL Parameters Menu from your vServer. Make sure to tick the "Enable DH Param" Box and select the newly created DH Key File. Also make sure to use a fairly high number under the "Refresh Count", I choose a value of 1000. When finished it should look something like this:

Now save your running config and go ahead and test your public vServer with the SSL Labs Test Suite. When configured correctly you should see the following results from the SSL Labs Test:

The green FS next to a Browser/OS Combination means PFS can and will be used when connecting to your vServer, just the way we like it. 

Implement HSTS/STS on your NetScaler Gateway

Today another small Blogpost on making your NetScaler Gateway / Access Gateway Enviroment (or any other Website published through a NetScaler) a little more secure by implementing HTTP Strict Transport Security (HSTS or STS). This should take you no longer than 15 Minutes to implement.

Let's get started. First we have to create a Rewrite Action and a Rewrite Policy. We start with the Rewrite Action:

Configure the Rewrite Action with the following Parameters as shown in the Screenshot.

  • Name: Insert_STS_Header (feel free to name it whatever you want to)
  • Type: INSERT_HTTP_HEADER
  • Header Name: Strict-Transport-Security
  • String: "max-age=157680000"

The max-age Value is given in seconds.

For all the CLI Fans here are the needed Snippets:

The next Step is to create the Rewrite Policy.

Create the Rewrite Policy with the following Parameters (see the following Screenshot)

  • Name: Enforce_STS
  • Action: select the previously created Rewrite Action
  • Expression: TRUE

Last Configuration Step is to bind it to your Netscaler Gateway / Access Gateway vServer under Policies -> Rewrite (Response) and make sure to save your running Config.

To make sure you configure everything correct you can use the SSL Scanner from Qualys Labs. You should see something similar to this:

Feel free to read my previous Blogpost on how to disable TLS/SSL Client Renegotiations on NetScaler for more SSL/TLS Security.

Disable TLS/SSL Client Renegotiation on NetScaler

Just a quick Reminder for securing your Citrix Enviroment: If you are running a NetScaler Gateway for your Remote Access to your Citrix XenApp/XenDesktop Enviroment (or pretty much anything else) make sure to disable secure and nonsecure TLS/SSL renegotiations initiated by a client to avoid potential Denial of Service Attacks.

Unfortunately the default setting (as of Netscaler Release 10.1) is set to an unsecure setting of allowing TLS/SSL Renegotiations.

You can set the appropriate Options via the NetScaler GUI under the "Advanced SSL Settings".

Make sure to select "FRONTEND_CLIENT" "ALL" in the Dropdown menu for Deny SSL Renegotiation.

Update: Since NetScaler 10.5 (for fresh Installations) the new Default is to deny all SSL Renegotiations. So the new recommendation is to set "Deny SSL Renegotian" to ALL instead of FRONTEND_CLIENT if you have upgraded your NetScalers from <10.5.

Thanks to Boris Groenhout for letting me know via the Comments.

Afterwards make sure everything is set appropriately by using the SSL Scanner from Qualys Labs under: https://www.ssllabs.com/ssltest. If you have configured everything correct (regarding TLS/SSL Renegotiation) you should see the following in the Report generated by the SSL Scanner:

You can also use the SSL Scanner to check if your SSL Certificate Chaining is working correctly and if you are using secure Cipher Suites on your NetScaler Gateway.

Updated 03.02.2016 - Added Notice regarding new Defaults for fresh NetScaler 10.5 Installations.