VMware Certificate Authority – My Favorite New Feature of vSphere 6.0 – Part 1 – The History

This is a repost of my VMware Consulting Blog!

https://blogs.vmware.com/consulting/2015/08/vmware-certificate-authority-my-favorite-new-feature-of-vsphere-6-0-part-1-the-history.html

------

Anyone who knows me knows I have a particular passion (however misplaced that may be…) for the VMware Certificate Story. This multi-part blog series will discuss a bit of background on the certificate story, what you need to know about the architectural design of it in an environment and some new features you may not know about.

Let me start off by saying this passion started off several years ago when I was in Global Support Services, and I realized that too few people had an understanding about certificates. I am not even talking certificates in the context of VMware, but in general. This was compounded when we released vSphere 5.1 due to the fact that strict certificate checking was enabled.

A Bit of History

Although certificates were used for securing communication prior to vSphere 5.1, they were self-signed and there was no verification performed to ensure the certificate was valid. Therefore, for example, the certificate could be expired, or be used for multiple services at the same time (such as for the vCenter Server service and the vCenter Inventory service). This is obviously not a good practice, but it nevertheless was allowed.

When vCenter Single Sign-On was released with vSphere 5.1, it enforced strict certificate checking. This included not only the certificate uniqueness, but such information as the validity period of the certificates as well. Therefore, if any of the components were not using a unique and valid certificate, they would not be accepted when registering the different services as solutions in Single Sign-On. This would turn out to be a pretty large issue as upgrades would fail with very little detail as to why.

That being said, if all services in vCenter 5.1 and 5.5 have their certificate replaced, seven unique certificates are required.

• vCenter Single Sign-On
• vCenter Inventory Service
• vCenter Server
• vSphere Web Client
• vSphere Web Client Log Browser
• vCenter Update Manager
• vCenter Orchestrator

The process to change the certificates is not straightforward and caused a significant amount of trouble amongst customers and global support services alike. This is when we raised it as a concern internally and helped to get a short-, medium- and long-term plan in place in time to make it easier to replace certificates when required. The plan was as follows:

• Short term – We ensured the KB articles relating to certificate replacement were accurate and easy to follow.
• Medium term – We helped in the development of the SSL Certificate Automation Tool, which dramatically reduced the number of steps and made it fairly easy to replace the certificates. 
• Long term – We forced focus on the issue so a solution could be built into the product.

Prior to moving from VMware Support to Professional Services Engineering we had released the tool and the larger plan was in place. The following are two blog posts I did for the tool:

http://blogs.vmware.com/kb/2013/04/introducing-the-vcenter-certificate-automation-tool-1-0.html

http://blogs.vmware.com/kb/2013/05/ssl-certificate-automation-tool-version-1-0-1.html 

With vSphere 6.0 the larger long-term solution is finally coming to fruition with the introduction of the VMware Certificate Authority. It solves many of the problems that were seen.

Introduction to the VMware Certificate Authority

With vSphere 6.0, the base product installs an internal certificate authority (CA) called the VMware Certificate Authority. This is a part of the Platform Services Controller installation and has changed the architecture significantly for the better. No longer are the default certificates self-signed, rather, they are issued and signed by the VMware Certificate Authority.

This works in one of two ways:

• VMware Certificate Authority acts as the root certificate authority. This is the default configuration and allows for an out-of-the-box configuration that is fully signed. All the clients need to do is to trust the root certificate and the communication is fully trusted.
• VMware Certificate Authority acts as an Intermediate CA, integrating into an existing CA infrastructure in the environment. This allows for certificates to be issued that are already trusted throughout the environment.

In each of these two modes, it acts in the same way granting certificates to not only the solutions connected to the management infrastructure, but to ESXi hosts as well. This occurs when the solution/host is added to vCenter server. By default, communication is secure and trusted, and therefore, everything on the management network that was previously difficult to secure is trusted.

Introduction to the VMware Endpoint Certificate Store 

In addition to the certificate authority itself, vSphere 6 certificates are now managed and stored in a “wallet.” This wallet is called the VMware Endpoint Certificate Store (VECS). The benefit here is certificates and private keys are no longer stored on disk in various locations. They are centrally managed in VECS on every vSphere node. This allows for a greatly simplified configuration for the environment because you no longer need to update trusts when the certificates are replaced because it is done automatically by VECS.

The VECS is installed on all platform services controller installation, including both embedded and external configurations.

The following different stores for certificates are used:

• The Machine Certificates store contains the Machine SSL Certificate and private key, which is used for the Reverse Proxy, discussed next.
• The Root CA Certificates store contains trusted root certificates and revocation lists, from any VMware Certificate Authority in the environment, or the third-party certificate authority being used. Solutions use this store to verify certificates.
• The Solution User Certificates store contains the certificates and private keys for any solutions such as vCenter and the vSphere Web Client.

A single location for all certificates is a welcome change to the previous versions. 

The Reverse Proxy – (Machine SSL Certificate)

Finally, before we get into the recommended architectures, the Reverse Proxy is the last thing I want to introduce. The change here addresses one of the biggest problems that was seen in previous versions of vCenter, which is that there are so many different services installed that require SSL communication. To be honest, the real challenge here is not the number of services rather, trying to get signed certificates all of them from the SSL administrator for the same host.

To combat this, solution users were consolidated with vCenter 6.0 to four vpxd, vpxd-extension, vsphere-webclient and machine. This is to say that where possible many of the various listening ports on the vCenter Server have been replaced with a single Reverse Web Proxy for communication. The Reverse Web Proxy uses the newly created Machine SSL Certificate to secure communication. 

Therefore, all communication to the different services is routed to the appropriate service based on the type of request through the reverse proxy. This can be seen in the figure below. 

It is still possible to change the certificate of the solution users behind it, however, these are only used internally and do not necessarily need to be changed. More on this in the next part of this series.

With all of this background detail out of the way, I think it is a good place to pause. Part 2 of this article will discuss the architecture decision points and some configuration details. On another note, I am actually going to be off to VMworld very soon and will be manning the VMware booth, as well as speaking at several sessions! It is unlikely I will get part 2 done before then, but if you have any questions look for me (downstairs at the Solution Exchange, in the VMware Booth, at the Technology Advisor station), and we can talk more on this and other topics with the other members of my team!