Bye Bye to free Blog. Say hello to my new website

Dear friends and Blog Readers,

I have started my new website

All new posts will now be published to my new website. This blog will be running up for a few days until I am confirmed that migration of all posts/links/media items were smooth and nothing is broken in my new website.

If you love to read my blogs and are following this, then a humble request to all of you is to follow my new website so that you are notified whenever I post new contents.

If you come across anything broken in my new site then feel free to let me know about that so that I can fix it.

Thanks for all your support as always.

Yours Sincerely

Manish Jha aka Alex Hunt


Posted in Vmware | Leave a comment

Learning Apache Cassandra-Part-4-Adding Node To Cassandra Cluster

In last post of this series we learnt how to install cassandra on Rhel 6. In this post we will look into additional configuration parameter that is needed to configure in order to facilitate other nodes to join the cassandra cluster.

If you have missed earlier posts of this series then you can read them from below links:

1: Introduction to Cassandra

2: Understanding Cassandra Read/Write Mechanism

3: Installing Cassandra on RHEL6

At the moment we have one node cassandra cluster. Before going ahead and installing cassandra on other nodes, we will first perform following configuration changes in cassandra.yaml file.

Navigate to cassandra.yaml file which is located in cassandra_install_dir/conf folder. Open the file in editor of your choice and look for following options:

  • Listen_address: Address where gossip will be listening to. This address can’t be localhost or, because the rest of nodes will try to connect to this address.
  • RPC_address: This is the address where thrift will be listening. We must put a existing IP address (it may be localhost, if we want to), or if we want to listen through all of them. This is the address to which client applications interact with cassandra DB.
  • Seeds: Seed nodes are the nodes which will provide cluster info to the new nodes which are bootstrapped and are ready to join the cluster. Seed nodes become a reference for any new nodes to join cluster in trustable way.

This settings we need to configure in cassandra.yaml file on each node which we want to put into the cluster.

Note: Be sure to install the same version of Cassandra as is installed on the other nodes in the cluster.

Procedure to add new nodes in cassandra cluster:

1: Install Cassandra on the new nodes, but do not start Cassandra.

2: Set the following properties in the cassandra.yaml and, depending on the snitch, the or configuration files:

  • auto_bootstrap – This property is not listed in the default cassandra.yaml configuration file, but it might have been added and set to false by other operations. If it is not defined in cassandra.yaml, Cassandra uses true as a default value. For this operation, search for this property in the cassandra.yaml file. If it is present, set it to true or delete it..
  • cluster_name – The name of the cluster the new node is joining. Ensure that cluster name is same for all nodes which will be part of cluster.
  • listen_address – Can usually be left blank. Otherwise, use IP address or hostname that other Cassandra nodes use to connect to the new node.
  • endpoint_snitch – The snitch Cassandra uses for locating nodes and routing requests. In my lab I am using simple snitch which is present as default in cassandra.yaml file and so I did not change or edit this.
  • num_tokens – The number of vnodes to assign to the node. If the hardware capabilities vary among the nodes in your cluster, you can assign a proportional number of vnodes to the larger machines.
  • seeds – Determines which nodes the new node contacts to learn about the cluster and establish the gossip process. Make sure that the -seeds list includes the address of at least one node in the existing cluster.

Post installing cassandra on your second node and making the configuration change as stated above, go ahead and start cassandra service on second node and do a watch on nodetool status on cassandra node 1 and you will see the new node joining the cluster.

Nodetool Status Output

Every 2.0s: /opt/apache-cassandra/bin/nodetool status Sun Jan 8 23:31:44 2017

Datacenter: datacenter1
|/ State=Normal/Leaving/Joining/Moving
— Address               Load           Tokens Owns (effective) Host ID Rack
UN 214.99 KiB  256      100.0%                    14ba62c6-59e4-404b-a6a6-30c9503ef3a4       rack1
UN 103.47 KiB  256       100.0%                    3b19bc83-f483-4a60-82e4-109c90c49a14      rack1

we have to repeat the same steps for each node which we want to place in our cluster.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 1 Comment

Learning Apache Cassandra-Part-3-Installing Cassandra on RHEL6

In last 2 posts of this series we learnt about Cassandra architecture and understood the cassandra Read/Write process.

If you have missed earlier posts of this series then you can read them from below links:

1:Introduction to cassandra

2: Understanding Cassandra Read/Write Mechanism

In this post we will learn about installing cassandra on Redhat Linux.

Before jumping into lab and start with installation, I want to touch down on few concepts first which will help in understanding installation process.


Bootstrapping is the process in which a newly-joining node gets the required data from the neighbors in the ring, so it can join the ring with the required data. Typically, a bootstrapping node joins the ring without any state or token and understands the ring structure after starting the gossip with the seed nodes; the second step is to choose a token to bootstrap.

During the bootstrap, the bootstrapping node will receive writes for the range that it will be responsible for after the bootstrap is completed. This additional write is done to ensure that the node doesn’t miss any new data during the bootstrap from the point when we requested the streaming to the point at which the node comes online.

Seed Nodes

The seed node designation has no purpose other than bootstrapping the gossip process for new nodes joining the cluster. Seed nodes are not a single point of failure, nor do they have any other special purpose in cluster operations beyond the bootstrapping of nodes.

In cluster formation, nodes see each other and “join”. They do not join just any node which respects the protocol, however. This would be risky: old partitioned replicas, different clusters, even malicious nodes, so on. So a cluster is defined by some initial nodes which are available at clear addresses and they become a reference for that cluster for any new nodes to join in trustable way. The seed nodes can go away after some time, the cluster will keep on.

Installation Prerequisites

In my lab I am setting up a 4 node cassandra cluster on RHEL 6 and performed following steps on each node.

1: Set a static IP on all 4 nodes and make sure all 4 nodes are reachable to each other via hostname/IP.

2: Set hostname in /etc/sysconfig/network file.

3: Update /etc/hosts file

Enter information of your cassandra nodes in /etc/hosts file.                 cassdb01             #SEED                 cassdb02            #Worker                 cassdb03             #Worker                cassdb04             #Worker

4: Open Firewall Ports

If you are using iptables, then the ports you need to open for Cassandra are 7000 and 9160. For each port you need to open, you can use the iptables command similar to this:

# iptables -A INPUT -p tcp –dport 7000 -j ACCEPT

5: Create cassandra user with sudo permissions.

You can use below script which will create a user on server with sudo permissions.

# wget

# chmod 777

# sh -s cassandra

After running the above script, make sure cassandra user/group is created on server

[root@cassdb01 ~]# cat /etc/passwd | grep cassandra

[root@cassdb01 ~]# cat /etc/group | grep cassandra

6: Install Java on server

Make sure you install oracle java (jdk or jre) version 7 or greater and JAVA_HOME set. You can install java via rpm based installer or using tar file.

In my lab, I installed java using rpm jdk-8u111-linux-x64.rpm.

Note: If you have openjdk installed on your system then please remove it before installing oracle java.

Note: Cassandra 3.0 and later require Java 8u40 or later

Verify that JAVA_HOME is set correctly and you are getting an output for java -version command

[root@cassdb01 ~]# cat .bash_profile | grep JAVA_HOME

[root@cassdb01 ~]# java -version
java version “1.8.0_111”
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

7: Install/Configure cassandra

Latest cassandra version can be downloaded from cassandra Home Page

Download and extract apache-cassandra tar.gz file in a directory of your choice. I used /opt as destination directory.

[root@cassdb01 opt]# tar -zxvf apache-cassandra-3.9-bin.tar.gz

[root@cassdb01 opt]# ln -s /opt/apache-cassandra-3.9 /opt/apache-cassandra

[root@cassdb01 opt]# chown cassandra:cassandra -R /opt/apache-cassandra

[root@cassdb01 opt]# chown cassandra:cassandra -R /opt/apache-cassandra-3.9

Next is to create necessary directories (for cassandra to store data)  and assign permissions on those directories.

[root@cassdb01 ~]# mkdir /var/lib/cassandra/data

[root@cassdb01 ~]# mkdir /var/log/cassandra

[root@cassdb01 ~]# mkdir /var/lib/cassandra/commitlog

[root@cassdb01 ~]# chown -R cassandra:cassandra /var/lib/cassandra/data

[root@cassdb01 ~]# chown -R cassandra:cassandra /var/log/cassandra/

[root@cassdb01 ~]# chown -R cassandra:cassandra /var/lib/cassandra/commitlog

Now we can start the cassandra service by using below command

[root@cassdb01 lib]# $CASSANDRA_HOME/bin/cassandra -f -R

On server startup, you will see below messages on command prompt which suggests that cassandra have been started without issues.

INFO 11:31:15 Starting listening for CQL clients on localhost/ (unencrypted)...
INFO 11:31:15 Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO 11:31:24 Scheduling approximate time-check task with a precision of 10 milliseconds
INFO 11:31:25 Created default superuser role 'cassandra'

If you want to start cassandra as a service, you can use this script from github.

Change value of following variable as per your environment


save the file in /etc/init.d directory. I saved my file with name cassandra.

Execute below command to add cassandra as service

[root@cassdb01 init.d]# chmod +x /etc/init.d/cassandra

[root@cassdb01 init.d]# chkconfig –add cassandra

[root@cassdb01 init.d]# chkconfig cassandra on

Start cassandra service and verify it started properly by checking the system.log file

[root@cassdb01 init.d]# service cassandra status
Cassandra is running.

last line of my system.log reads as

INFO  12:45:50 Node localhost/ state jump to NORMAL

Nodetool reporting node as UP and Normal

[root@cassdb01 init.d]# $CASSANDRA_HOME/bin/nodetool status
Datacenter: datacenter1
|/ State=Normal/Leaving/Joining/Moving
— Address Load Tokens Owns (effective) Host ID Rack
UN 168.62 KiB 256 100.0% 14ba62c6-59e4-404b-a6a6-30c9503ef3a4 rack1

With this installation of cassandra is completed.

In next post of this series we will make a few configuration changes and will configure other nodes in order to facilitate them to join the cassandra cluster effectively. Stay Tuned!!!

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 2 Comments

Learning Apache Cassandra-Part-2:Understanding Cassandra Read/Write Mechanism

In first post of this series we discussed about what is cassandra, what are the benefits of using cassandra. We also discussed a little bit about from where cassandra came and finally we looked at the architecture of cassandra and discussed some important terms like snitch, gossip, data replication, partitioner etc.

In this post we will see how cassandra Read/Write mechanism works.

Let’s discuss about some key components of cassandra first before discussing about Read/Write. Important components of cassandra can be summarized as below:

1: Node – This is the most basic component of cassandra and it is the place where data is stored.

2: Data Center – In simplest term a datacenter is nothing but a collection of nodes. A datacenter can be a physical datacenter or virtual datacenter.

3: Cluster – Collection of many data centers is termed as cluster.

4: Commit Log – Every write operation is written to Commit Log. Commit log is used for crash recovery. After all its data has been flushed to SSTables, it can be archived, deleted, or recycled.

5: Mem-table – A mem-table is a memory-resident data structure. Data is written in  commit log and mem-table simultaneously. Data stored in mem-tables are temporary and it is flushed to SSTables when mem-tables reaches configured threshold.

6: SSTable – This is the disk file where data is flushed when Mem-table reaches a certain threshold.

7: Bloom filter These are nothing but quick, nondeterministic, algorithms for testing whether an element is a member of a set. It is a special kind of cache. Bloom filters are accessed after every query.

8: Cassandra Keyspace Keyspace is similar to a schema in the RDBMS world. A keyspace is a container for all your application data.

9: CQL Table – A collection of ordered columns fetched by table row. A table consists of columns and has a primary key.

Cassandra Write Operation

Since cassandra is a distributed database technology where data are spread across nodes of the cluster and there is no master-slave relationship between the nodes, it is important to understand how data is stored (written) within the database.

Cassandra processes data at several stages on the write path, starting with the immediate logging of a write and ending in with a write of data to disk:

  • Logging data in the commit log
  • Writing data to the memtable
  • Flushing data from the memtable
  • Storing data on disk in SSTables

When a new piece of data is written, it is written at 2 places i.e Mem-tables and in the commit.log on disk ( for data durability). The commit log receives every write made to a Cassandra node, and these durable writes survive permanently even if power fails on a node.

Mem-tables are nothing but a write-back cache of data partition. Writes in Mem-tables are stored in a sorted manner and when Mem-table reaches the threshold, data is flushed to SSTables.

Flushing data from the Mem-Table

Data from Mem-table is flushed to SSTables in the same order as they were stored in Mem-Table. Data is flushed in following 2 conditions:

  • When the memtable content exceeds the configurable threshold
  • The commit.log space exceeds the commitlog_total_space_in_mb

If any of the condition reaches, cassandra places the memtables in a queue that is flushed to disk. The size of queue can be configured by using options memtable_heap_space_in_mb or memtable_offheap_space_in_mb in cassandra.yaml configuration file. Mem-Table can be manually flushed using command nodetool flush

Data in the commit log is purged after its corresponding data in the memtable is flushed to an SSTable on disk.

Storing data on disk in SSTables 

Memtables and SSTables are maintained per table. The commit log is shared among tables. Memtable is flushed to an immutable structure called and SSTable (Sorted String Table).

Every SSTable creates three files on disk

  1. Data (Data.db) – The SSTable data
  2. Primary Index (Index.db) – Index of the row keys with pointers to their positions in the data file
  3. Bloom filter (Filter.db) – A structure stored in memory that checks if row data exists in the memtable before accessing SSTables on disk

Over a period of time a number of SSTables are created. This results in the need to read multiple SSTables to satisfy a read request.  Compaction is the process of combining SSTables so that related data can be found in a single SSTable. This helps with making reads much faster.

The commit log is used for playback purposes in case data from the memtable is lost due to node failure. For example the node has a power outage or someone accidently shut it down before the memtable could get flushed.



Read Operation

Cassandra processes data at several stages on the read path to discover where the data is stored, starting with the data in the memtable and finishing with SSTables:

  • Check the memtable
  • Check row cache, if enabled
  • Checks Bloom filter
  • Checks partition key cache, if enabled
  • Goes directly to the compression offset map if a partition key is found in the partition key cache, or checks the partition summary if not.
  • If the partition summary is checked, then the partition index is accessed

  • Locates the data on disk using the compression offset map
  • Fetches the data from the SSTable on disk


There are three types of read requests that a coordinator sends to replicas.

  1. Direct request
  2. Digest request
  3. Read repair request

The coordinator sends direct request to one of the replicas. After that, the coordinator sends the digest request to the number of replicas specified by the consistency level and checks whether the returned data is an updated data.

After that, the coordinator sends digest request to all the remaining replicas. If any node gives out of date value, a background read repair request will update that data. This process is called read repair mechanism.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 3 Comments

Learning Apache Cassandra-Part-1:Introduction

Cassandra was very new to me when I joined the vCloud Air operations team back in 2015. Over last 1.5 years I have got a bit of understanding about cassandra now and it provoked me to learn this wonderful database technology.

To start my journey with Cassandra, I am following training course from Infinite-Skills and youtube videos along with documentation available from datastax.

As my journey will progress, I will keep writing and sharing my experience of learning about cassandra.

This is very first post of learning cassandra series where I will try to touch down few basic things about cassandra.

What is cassandra?

cassandra is a NoSQL database technology. NoSQL is referred as not only SQL and it means an alternative to traditional relational database technologies like MySQL, Oracle, MSSQL. Apache cassandra is a distributed database. With cassandra database do not lives only on one server but is spread across multiple servers. This allows database to grow almost infinitely as it is no longer dependent on specifications of one server.

It is a big data technology which provides massive scalability.

History of Cassandra

Cassandra project was developed by Avinash Lakshman (the author of Amazon Dynamo) and Prashant Malik at Facebook, to solve their Inbox-search problem. In combination with Google BigTable and Amazon DynamoDB, they developed Cassandra.

Cassandra code was published in July 2008 as free software, under the Apache V2 license. Since then Cassandra is developed and maintained by the Apache foundation for the community version and by DataStax for the enterprise and commercial versions.


Why use cassandra?

The top features which makes cassandra so powerful can be summarized as below:

1: Fault Tolerant/High Availability

Cassandra is deployed as a clustered system  which contains many node. All the nodes replicate data to each other for fault-tolerance. Even if few nodes from a cassandra cluster goes down, data is not lost. Data Replication across multiple data centers is supported. Also the failed nodes can be replaced with no downtime.

2: Decentralized

In a cassandra cluster, each node has same functionality. There is no master-slave concept with cassandra and thus there are no single points of failure. Every node in the cluster is identical. Cassandra cluster is not impacted by network bottlenecks.

Cassandra is designed to have peer-to-peer symmetric nodes, instead of master-slave nodes, to ensure there can never be a single point of failure. Cassandra automatically partitions data across all the nodes in the database cluster ( this setting is controlled by the administrator who determine what data will be replicated and how many copies of the data will be created)

3: Scalable

Cassandra is highly scalable database technology and a simplest cassandra cluster can contains as few as 2-3 nodes and also can grow upto 1000 nodes.

As per Apache cassandra Home Page , Some of the largest production deployments include Apple’s, with over 75,000 nodes storing over 10 PB of data, Netflix (2,500 nodes, 420 TB, over 1 trillion requests per day), Chinese search engine Easou (270 nodes, 300 TB, over 800 million requests per day), and eBay (over 100 nodes, 250 TB).

4: Durable

Since cassandra supports replication of data across datacenters, it is suitable for those applications that can’t afford to lose data, even when an entire data center goes down.

5: Elastic

Cassandra cluster is very elastic as new nodes can be added to cluster without any downtime or interruption to applications and thus increasing Read and write throughput increase linearly.

6: Distributed Writes

Within cassandra cluster, data can be read from and write to anywhere in the cluster at any time.

7: Flexible Schema:

With Cassandra,  you don’t have to decide what fields you need in your records ahead of time. You can add and remove arbitrary fields on the fly.

Cassandra Architecture

The architecture of Cassandra greatly contributes to its being a database that scales and performs with continuous availability. As we discussed earlier that cassandra do not uses legacy  RDBMS master-slave topology and instead it is a masterless “ring” distributed architecture that is elegant, and easy to setup and maintain.

Not having to distinguish between a Master and a Slave node allows you to add any number of machines to any cluster in any datacenter, without having to worry about what type of machine you need at the moment. Every server accepts requests from any client. Every server is identical.


Cassandra’s built-for-scale architecture means that it is capable of handling large amounts of data and thousands of concurrent users/operations per second, across multiple data centers. Cluster capacity can be increased on the fly without any configuration change.


Since cassandra is a masterless technology, question comes in mind how the nodes in cassandra cluster knows about each other. Answer to this question is ‘Snitch’

Using snitch, nodes updates each other about which node is in which datacenter or which node is in which rack within a datacenter. Snitch is how the nodes in a cluster know about the topology of cluster.

There are various options for configuring snitch in a cluster including:

  • Dynamic Snitching
  • Simple Snitch
  • Rack Inferring Snitch
  • PropertyFileSnitch
  • Gossipingpropertyfilesnitch
  • EC2Snitch
  • EC2 MultiRegion Snitch

You must configure a snitch when you create a cluster. All snitches use a dynamic snitch layer, which monitors performance and chooses the best replica for reading.

The default Simple Snitch do not recognize datacenter or rack information. Simple snitch can be only used with cluster that are within one datacenter.

The GossipingPropertyFileSnitch is recommended for production. It defines a node’s datacenter and rack and uses gossip for propagating this information to other nodes.

To learn more about snitch please read this Article from Edureka.


Gossip is how node in cluster talks with each other. Every one second, each node communicate with up to 3 other nodes, exchanging information about itself and all the other nodes that it has information about.

The nodes exchange information about themselves and about the other nodes that they have gossiped about, so all nodes quickly learn about all other nodes in the cluster. A gossip message has a version associated with it, so that during a gossip exchange, older information is overwritten with the most current state for a particular node.


Gossip is the internal communication method for nodes in a cluster to talk to each other.

For external communication such as from an app to a cassandra database, CQL (cassandra query language) or Thrift is used.

On a lighter node, this sounds to me like Women’s gossip as they always talk about how other women is looking or which dress she worn yesterday, with whom she is hanging out etc 😀

Data distribution and replication

Data distribution is done through consistent hashing, to strive for even distribution of data across the nodes in a cluster. Rather than all of the rows of a table existing in only one node, the rows are distributed across the nodes in the cluster, in an attempt to evenly spread out the load of the table’s data.

To distribute rows across the node, a partitioner is used. A partitioner is a hash function that derives a token from the primary key of a row. The partitioner uses the token value to determine which nodes in the cluster receive the replicas of that row. The default partitioner in Cassandra is Murmur3.

To understand this consider below example of a home alarm system


Murmur3 takes the value in the first column or can take value from more than one column to generate a unique number between –263  – 263.

Let’s suppose Murmur3 generated below hashed values for data stored in first column.


Each node in the cluster has an endpoint value assigned to it. Each node is responsible then for values leading up to it including its value.


Now the question comes where do these endpoint values comes from?

There are couple of ways to do it:

1: Calculate the token range Manually

You can use below formula to generate the token range value. Below formula was used to generate token range for a 4 node cassandra cluster.

python -c 'print [str(((2**64 / number_of_tokens) * i) - 2**63) for i in range(number_of_tokens)]'

If you have more than 4 nodes, replace 4 with number of nodes in your cluster.  For example, to generate tokens for 6 nodes:

python -c 'print [str(((2**64 / 6) * i) - 2**63) for i in range(6)]'

The command displays the token for each node:

[ '-9223372036854775808', '-6148914691236517206', '-3074457345618258604', 
  '-2' , '3074457345618258600', '6148914691236517202' ]

2: Murmur3 Calculator

Go to this website and navigate to Cassandra Token Calculator and select Murmur3Partitioner and in front of Number of nodes, enter value equal to number of nodes in your cluster and click on calculate Tokens to generate the endpoint values which will be assigned to each node.


Now these token values will be assigned to each one of the node.

The hashed value which was calculated earlier will be matched against the token value and then decision will be taken that on which node a particular row data will be stored.

Replication factor

Whenever a database is defined, a replication factor is defined along with. The replication factor specifies how many instances of the data there will be within a given database.

We can set replication factor as ‘1’ but it is always recommended to use a value greater than one so that in case a node goes down, there is at least one replica of the data and data is not lost with down node.

If in a 4 node cluster, replication factor is defined as 2, then there will be 2 nodes which will store same data. If replication factor is set to 4 then it means each node of cluster will have replica of the data which is stored on any node.

For e.g, if the replication factor for a database is 3, the -7456322128261119046 data will be written to 3 nodes and even if one node goes down, we will not lose data.


Replication Strategy

A replication strategy determines which nodes to place replicas on.

Two replication strategies are available:

  • SimpleStrategy: Use only for a single datacenter and one rack. If you ever intend more than one datacenter, use the NetworkTopologyStrategy. SimpleStrategy places the first replica on a node determined by the partitioner. Additional replicas are placed on the next nodes clockwise in the ring without considering topology.
  • NetworkTopologyStrategy: Highly recommended for most deployments because it is much easier to expand to multiple datacenters when required by future expansion. NetworkTopologyStrategy places replicas in the same datacenter by walking the ring clockwise until reaching the first node in another rack. It attempts to place replicas on distinct racks because nodes in the same rack often fail at the same time due to power, cooling, or network issues.

That’s it for this post.In next posts of this series we will learn how to install and configure casandra and we will also touchdown of administering and troubleshooting part.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 3 Comments

Find Snapshot Creation Date of a vCloud Director VM

Last month while working on a customer ticket, I came across a request from customer where he wanted to know snapshot creation date for one of his VM as he can not find this detail from vCD UI.

To confirm this, I logged into vCD and navigated to my test lab to see what are the information available.

On navigating through vCD I found that vCD only tells that whether or not snapshot exists for a vApp/VM.

You can see in below screenshot in top right corner that there is no option for selection snapshot creation date.


On drilling down to VM level also, I did not found any option for checking snapshot creation date.


After banging my head for 20 minutes or so, I decided to use API calls as many times I have found some info which is not visible in vCD GUI. And the trick worked for me. I was able to pull the snapshot creation date via API calls.

The steps for doing so is summarized as below:

For sake of security, I have replaced the original token code with ‘xxxxx’

1: Generate session key/Auth token

# curl -sik -H “Accept:application/*+xml;version=5.6” -u “admin@system” -X POST | grep auth
Enter host password for user ‘admin@system’:

The above query will return auth token which you can use in next queries

x-vcloud-authorization: XXXXXXXXXXXXXXXXXXX

2: Get Otg UUID

Next is to grab the Org UUID by executing below command:

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: XXXXXXXXXXXXXXXXXXX” -X GET | grep bdd75fd4-a319-47d5-b4f2-77aad691488f

By executing above query, you will get href of your org.

<Org href=”” name=”bdd75fd4-a319-47d5-b4f2-77aad691488f” type=”application/”/>

3: Get Org VDC UUID

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: XXXXXXXXXXXXXXXXXXX” -X GET | grep vdc

The above query will return you a list of Org vDC’s (in case if you have more than one VDC)

<Link rel=”down” href=”” name=”Manish-Lab-DontDel” type=”application/vnd.vmware.vcloud.vdc+xml”/>

<Link rel=”down” href=”; name=”VCHS” type=”application/vnd.vmware.vcloud.vdc+xml”/>

<Link rel=”down” href=”; name=”vSphere-6.5-Lab-Manish” type=”application/vnd.vmware.vcloud.vdc+xml”/>

My VM exists in VDC ‘Manish-Lab-DontDel’ so I used the corresponding VDC href in my next query

4: Find VM UUID

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: XXXXXXXXXXXXXXXXXXX” -X GET | grep SQLSRV

<ResourceEntity href=”” name=”SQLSRV” type=”application/vnd.vmware.vcloud.vApp+xml”/>

5: Find Snapshot details

Now to get snapshot info, append /snapshotSection at the end of the url which you got in previous query

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: XXXXXXXXXXXXXXXXXXX” -X GET

You will get an XML output which contains the snapshot creation date



Posted in vCloud Director, Vmware | 1 Comment

Exploring vSphere 6.5 API-Part 3: Esxi Host

In last 2 post of this series, we learn about digging out info about datacenter,cluster and virtual machines.

In this post we will learn about API options available for Esxi hosts.

If you have missed earlier posts of this series, you can read them from here:

1: Exploring vSphere 6.5 API-Datacenter & Cluster

2: Exploring vSphere 6.5 API-Virtual Machines

Let’s get started with fetching info about esxi hosts.

1: List all hosts present in a vCenter

Following query will list all esxi hosts that are present across all cluster/datacenter which are there in a vcenter.

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 7c639fb1e988b4c59dd77adef4c5d06c’ -X GET https://vcentersrv01.alex.local/rest/vcenter/host







2: List all host in a specific datacenter

You can limit search of esxi host to specific datacenter by using filter.datacenters option

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 7c639fb1e988b4c59dd77adef4c5d06c’ -X GET

3: List all hosts in a specific cluster

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 7c639fb1e988b4c59dd77adef4c5d06c’ -X GET https://vcentersrv01.alex.local/rest/vcenter/host?filter.clusters=domain-c7


"value": [
"host": "host-35",
"name": "esxi04.alex.local",
"connection_state": "CONNECTED",
"power_state": "POWERED_ON"

4: Info about a particular host

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 7c639fb1e988b4c59dd77adef4c5d06c’ -X GET https://vcentersrv01.alex.local/rest/vcenter/host?filter.hosts=host-33


"value": [
"host": "host-33",
"name": "esxi03.alex.local",
"connection_state": "CONNECTED",
"power_state": "POWERED_ON"

5: List all standalone host

If you have hosts that are not part of any cluster and are added as standalone host, you can list them by using filter.standalone=true

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 7c639fb1e988b4c59dd77adef4c5d06c’ -X GET https://vcentersrv01.alex.local/rest/vcenter/host?filter.standalone=true

6: List all hosts that are connected

The below query will filter out those hosts which are currently in disconnected state in vCenter

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 7c639fb1e988b4c59dd77adef4c5d06c’ -X GET


7: Disconnect a host from the vCenter server.

POST https://vcentersrv01.alex.local/rest/vcenter/host/{host}/disconnect


# curl -sik -X GET -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 7c639fb1e988b4c59dd77adef4c5d06c’ -X POST https://vcentersrv01.alex.local/rest/vcenter/host/host-28/disconnect

8: Connect to the host previously added to the vCenter server.

POST https://vcentersrv01.alex.local/rest/vcenter/host/{host}/connect

9: Remove a standalone host from the vCenter Server.

DELETE https://vcentersrv01.alex.local/rest/vcenter/host/{host}

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | Leave a comment

Storage Migrate a vCloud Director VM using Rest API

If you are using vCloud Director in your environment and if you have ever tried doing a Storage vMotion of a VM from vSphere directly, you will notice a warning saying that its not recommended to modify the entity since its managed by VCD.


This is because of the the fact that with VCD, the management layer lies with itself and not with vSphere. All changes to the entities should be made via vCD and not vSphere.

Although doing a storage migration will not break anything but as a best practice we should avoid that.

So what to do if one of your backend LUN is full and you need to evacuate that by migrating some vm’s to another datastore which have enough free space. The answer is by using “Rest API”.

About the VMware vCloud API

The VMware vCloud API provides support for developers who are building interactive clients of VMware vCloud Director using a RESTful application development style. vCloud API clients communicate with servers over HTTP, exchanging representations of vCloud objects. These representations take the form of XML elements.

You use HTTP GET requests to retrieve the current representation of an object, HTTP POST and PUT requests to create or modify an object, and HTTP DELETE requests to delete an object.

To know more about vCD Rest API please read vCloud API Programming Guide.

Which tools to use?

There are various tools available on internet which can be used to interact with vCD Rest API. Most common tools include:

1: Postman Rest Client (chrome extension) which can be added to chrome from webstore.

2: Rest Client for firefox. It can be downloaded from here

3: Using curl which can be downloaded from here

Since I am fan of command line, in my lab I have installed curl on one of my linux box.

Once you have downloaded and installed curl, you can follow below steps to storage migrate your VM.

Note: I am using API calls against my environment which is hosted in vCloud Air


1: Generate session key 

The first thing to use API is to generate an Auth token which then can be passed to the later commands. You need to authenticate against VCD first to get a session token.

# curl -sik -H “Accept:application/*+xml;version=5.6” -u “Administrator@system:Administrator password” -X POST https://vCloud Director FQDN]/api/sessions


curl -sik -H "Accept:application/*+xml;version=5.6" -u "admin@system" -X POST | grep auth

Enter host password for user 'admin@system':

x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7edf

Once you receive auth token, we have to change the header in above command to:

“x-vcloud-authorization: Auth Token”

2: Get Otg UUID

Next is to grab the Org UUID by executing below command:

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7edf” -X GET | grep bdd75fd4-a319-47d5-b4f2-77aad691488f

The above command will give you href to org URL

<Org href="" name="bdd75fd4-a319-47d5-b4f2-77aad691488f" type="application/"/>

3: Get Org VDC UUID

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7edf” -X GET | grep vdc


<Link rel=”down” href=”; name=”Manish-Lab-DontDel” type=”application/vnd.vmware.vcloud.vdc+xml”/>


4: Get list of vApps

curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7edf” -X GET | grep vapp

The above command will give you full list of all the vApps which you have created in vCD

<ResourceEntity href="" name="Esxi" type="application/vnd.vmware.vcloud.vAppTemplate+xml"/>

<ResourceEntity href="" name="Openfiler" type="application/vnd.vmware.vcloud.vApp+xml"/>

<ResourceEntity href="" name="CASRV01-VApp" type="application/vnd.vmware.vcloud.vApp+xml"/>

<ResourceEntity href="" name="Esxi-2" type="application/vnd.vmware.vcloud.vApp+xml"/>

<ResourceEntity href="" name="vCenterSrv" type="application/vnd.vmware.vcloud.vApp+xml"/>

<ResourceEntity href="" name="Esxi-1" type="application/vnd.vmware.vcloud.vApp+xml"/>

<ResourceEntity href="" name="Alex-DC-VApp" type="application/vnd.vmware.vcloud.vApp+xml"/>


5: Get VM UUID

Since now we have href to the vApps, we can now get UUID of the VM which we are planning to move to other datastore.

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7edf” -X GET

6: Get list of datastores

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7edf” -X GET

This query will return a lot of info about the datastores like provisioned storage, used space, free space available on datastore, datastore Mo-ref and href to datastores.


<DatastoreRecord datastoreType=”VMFS5″ isDeleted=”false” isEnabled=”true” moref=”datastore-108014″ name=”z_d10p15s3vnx0lu66_tStd_inuse” numberOfProviderVdcs=”1″ provisionedStorageMB=”13059087″ requestedStorageMB=”6521317″ storageMB=”10485504″ storageUsedMB=”7595588″ vc=”; vcName=”vc-fqdn” href=”“/>

<DatastoreRecord datastoreType=”VMFS5″ isDeleted=”false” isEnabled=”true” moref=”datastore-19022″ name=”z_d10p15s3vnx0lu57_tSSDAcc_inuse” numberOfProviderVdcs=”1″ provisionedStorageMB=”7991302″ requestedStorageMB=”8479229″ storageMB=”10485504″ storageUsedMB=”2016348″ vc=”; vcName=”vc-fqdn” href=”“/>

<DatastoreRecord datastoreType=”VMFS5″ isDeleted=”false” isEnabled=”true” moref=”datastore-17670″ name=”z_d10p15s3vnx0lu51_tStd_inuse” numberOfProviderVdcs=”1″ provisionedStorageMB=”12538920″ requestedStorageMB=”10196156″ storageMB=”10485504″ storageUsedMB=”6006352″ vc=”; vcName=”vc-fqdn” href=”“/>

<DatastoreRecord datastoreType=”VMFS5″ isDeleted=”false” isEnabled=”true” moref=”datastore-17671″ name=”z_d10p15s3vnx0lu56_tSSDAcc_inuse” numberOfProviderVdcs=”1″ provisionedStorageMB=”8810546″ requestedStorageMB=”8773278″ storageMB=”10485504″ storageUsedMB=”7094757″ vc=”; vcName=”vc-fqdn” href=”“/>

<DatastoreRecord datastoreType=”VMFS5″ isDeleted=”false” isEnabled=”true” moref=”datastore-19203″ name=”z_d10p15s3vnx0lu58_tSSDAcc_inuse” numberOfProviderVdcs=”1″ provisionedStorageMB=”6293195″ requestedStorageMB=”6070760″ storageMB=”10485504″ storageUsedMB=”2509893″ vc=”; vcName=”vc-fqdn” href=”“/>

<DatastoreRecord datastoreType=”VMFS5″ isDeleted=”false” isEnabled=”true” moref=”datastore-66433″ name=”z_d10p15s3vnx0lu65_tStd_inuse” numberOfProviderVdcs=”1″ provisionedStorageMB=”7326909″ requestedStorageMB=”4607043″ storageMB=”10485504″ storageUsedMB=”3819394″ vc=”; vcName=”vc-fqdn” href=”“/>


7: Find out datastore name where your VM is residing

To do this, dump the VM settings into an xml file and grep for datastore.

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7af0c” -X GET > vminfo.xml

# cat vminfo.xml | grep datastore




You can match the Mo-Ref returned with output of query 6 and you will know the name of the datastore where your VM is presently residing.

8: Prepare for migration

Now before we start migration, we need to create an input file which contains the destination datastore UUID where we want to send the VM. We will supply the input file alongwith API query to perform the storage migration.

Note: In output of query 6, we got a bunch of datastore alongwith details of free space on each datastore. You can select that datastore which has  max available free space.

Info about the necessary parameters can be obtained from API Schema guide

I created a file named ‘dest-ds.xml’ with below content:

<?xml version=”1.0″ encoding=”UTF-8″?>
<RelocateParams xmlns=”″&gt;
<Datastore href=”; />

Make sure formatting is correct in the xml file else you will get error. My XML file looks like


If you have trouble with xml formatting, you can go to Online XML formatter and paste your text in XML Formatter area and click on Format XML


You will see the formatted xml as output


9: Perform the migration of VM

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7af0c” -H “Content-Type:application/vnd.vmware.vcloud.relocateVmParams+xml” -d @dest-ds.txt -X POST 

The above will generate a task id:

HTTP/1.1 202 Accepted
Date: Sat, 24 Dec 2016 13:39:22 GMT
Strict-Transport-Security: max-age=15768000; includeSubDomains
X-Frame-Options: SAMEORIGIN
x-vcloud-authorization: XXXXXXX

If you have access to backend vSphere, you can see the migration task there



In case if you dont have access to backend, you can monitor the migration process by using below query

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: d18d26b54cb346c8a9a097ab24d7af0c” -X GET | grep Progress


if the progress reads 100, it means migration have been completed.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂


Posted in vCloud Director | Leave a comment

Exploring vSphere 6.5 API-Part 2: Virtual Machines

In last post of this series we looked into some basic Rest API’s to fetch info about datacenter and cluster.

In this post we will explore API options for virtual machines. Out of all the components like host, cluster etc, max number of available API options are for virtual machines.

Let’s start with figuring out available options:

To start exploring the different API options available for virtual machine, you can use below query:

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 3762fda1e69484ce746e93d9332f6ada’ -X GET https://vcentersrv01.alex.local/rest/com/vmware/vapi/rest/navigation/resource/id:VirtualMachine

1: List all VM’s in all datacenter

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 3762fda1e69484ce746e93d9332f6ada’ -X GET https://vcentersrv01.alex.local/rest/vcenter/vm

Output of above query will give you VM ID, cpu/memory stats, VM name and their power status.


2: List Powered off VM’s

You can list all powered off VM’s in datacenter using the filter power_states in above query.

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 3762fda1e69484ce746e93d9332f6ada’ -X GET

3: List all VM in a specific datacenter

If you have more than one datacenter/vcenter, then you can use filter.datacenters to list VM’s belonging to that datacenter. Remember that we have to supply datacenter ID and not the name in below query:

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 3762fda1e69484ce746e93d9332f6ada’ -X GET

4: List all VM in a cluster

Below query will list details of all VM’s that are in a clusters. Again we have to use a filter for cluster and supply the cluster id.

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 3762fda1e69484ce746e93d9332f6ada’ -X GET

5: List all VM on a particular host

You can list all VM’s that resides on a particular host by using the filter for hosts.

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 3762fda1e69484ce746e93d9332f6ada’ -X GET


You can grab id of your esxi hosts using below query

# curl -sik -X GET -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 68398bd2a6aa39a9aa81fe0bc65ab3d1’ https://vcentersrv01.alex.local/rest/vcenter/host





6: Delete a particular vm

Below query will remove a particular VM from your environment. For doing this you should know the VM ID of the VM which you want to delete. You can use query#1 of this post to grab VM ID

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 3762fda1e69484ce746e93d9332f6ada’ -X DELETE

You can verify VM deletion in Web Client Task and Events.

7: Info about particular vm

Below query will return a lot of info for a VM like number of disks and their type, info about CD ROM and whether or not its connected. Also CPU/Memory stats with hot add enabled or not etc.

curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 3762fda1e69484ce746e93d9332f6ada’ -X GET

 "cdroms": [
 "value": {
 "start_connected": false,
 "backing": {
 "auto_detect": false,
 "device_access_type": "EMULATION",
 "type": "HOST_DEVICE",
 "host_device": "CD/DVD drive 0"
 "allow_guest_control": true,
 "label": "CD/DVD drive 1",
 "ide": {
 "primary": true,
 "master": true
 "state": "NOT_CONNECTED",
 "type": "IDE"
 "key": "3000"

"memory": {
 "hot_add_increment_size_MiB": 128,
 "size_MiB": 10240,
 "hot_add_enabled": true,
 "hot_add_limit_MiB": 163840

"disks": [
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 9
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_8.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 9",
 "type": "SCSI",
 "capacity": 1073741824
 "key": "2009"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 8
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_7.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 8",
 "type": "SCSI",
 "capacity": 10737418240
 "key": "2008"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 6
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_6.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 7",
 "type": "SCSI",
 "capacity": 16106127360
 "key": "2006"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 5
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_5.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 6",
 "type": "SCSI",
 "capacity": 10737418240
 "key": "2005"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 4
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_4.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 5",
 "type": "SCSI",
 "capacity": 10737418240
 "key": "2004"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 3
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_3.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 4",
 "type": "SCSI",
 "capacity": 26843545600
 "key": "2003"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 2
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_2.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 3",
 "type": "SCSI",
 "capacity": 26843545600
 "key": "2002"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 1
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_1.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 2",
 "type": "SCSI",
 "capacity": 1918894080
 "key": "2001"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 12
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_11.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 12",
 "type": "SCSI",
 "capacity": 107374182400
 "key": "2012"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 0
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 1",
 "type": "SCSI",
 "capacity": 12884901888
 "key": "2000"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 11
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_10.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 11",
 "type": "SCSI",
 "capacity": 10737418240
 "key": "2011"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 10
 "backing": {
 "vmdk_file": "[iSCSI-1] vcentersrv02.alex.local/vcentersrv02.alex.local_9.vmdk",
 "type": "VMDK_FILE"
 "label": "Hard disk 10",
 "type": "SCSI",
 "capacity": 10737418240
 "key": "2010"
 "parallel_ports": [],
 "sata_adapters": [],
 "cpu": {
 "hot_remove_enabled": true,
 "count": 2,
 "hot_add_enabled": true,
 "cores_per_socket": 1
 "scsi_adapters": [
 "value": {
 "scsi": {
 "bus": 1,
 "unit": 7
 "pci_slot_number": 32,
 "label": "SCSI controller 1",
 "type": "LSILOGIC",
 "sharing": "NONE"
 "key": "1001"
 "value": {
 "scsi": {
 "bus": 0,
 "unit": 7
 "pci_slot_number": 16,
 "label": "SCSI controller 0",
 "type": "LSILOGIC",
 "sharing": "NONE"
 "key": "1000"
 "power_state": "POWERED_ON",
 "floppies": [],
 "name": "vcentersrv02.alex.local",
 "nics": [
 "value": {
 "start_connected": true,
 "pci_slot_number": 160,
 "backing": {
 "connection_cookie": 1788152499,
 "distributed_switch_uuid": "50 14 df fd 3e 62 45 e5-86 bb 10 3f d1 e3 88 c5",
 "distributed_port": "4",
 "network": "dvportgroup-21"
 "mac_address": "00:50:56:94:51:90",
 "mac_type": "ASSIGNED",
 "allow_guest_control": true,
 "wake_on_lan_enabled": true,
 "label": "Network adapter 1",
 "state": "CONNECTED",
 "type": "VMXNET3",
 "upt_compatibility_enabled": true
 "key": "4000"
 "boot": {
 "delay": 0,
 "retry_delay": 10000,
 "enter_setup_mode": false,
 "type": "BIOS",
 "retry": false
 "serial_ports": [],
 "guest_OS": "OTHER_3X_LINUX_64",
 "boot_devices": [],
 "hardware": {
 "upgrade_policy": "NEVER",
 "upgrade_status": "NONE",
 "version": "VMX_10"

8: Power state of a VM

You can query power state of a virtual machine using below query

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 50c05b3bd6d2b2493e59b44bb00ea421’ -X GET https://vcentersrv01.alex.local/rest/vcenter/vm/vm-57/power

HTTP/1.1 200 OK
Date: Sun, 18 Dec 2016 15:55:54 GMT
Content-Type: application/json
Transfer-Encoding: chunked




9: Reset a powered-on VM

You can restart a powered on VM using a post call against the VM id as shown below:

# curl -sik  -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 50c05b3bd6d2b2493e59b44bb00ea421’ -X POST https://vcentersrv01.alex.local/rest/vcenter/vm/vm-57/power/reset

The list is long and its not possible for me to test each and every query in lab and paste the output here. So I have compiled a list of various options that you can test in your environment.


Resets a powered-on virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/power/reset

Powers off a powered-on or suspended virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/power/stop

Powers on a powered-off or suspended virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/power/start

Suspends a powered-on virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/power/suspend

Returns the power state information of a virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/power

Returns the boot-related settings of a virtual machine

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/boot

Returns an ordered list of boot devices for the virtual machine. If the list is empty, the virtual machine uses a default boot sequence.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/boot/device

Updates the boot-related settings of a virtual machine.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/boot

Sets the virtual devices that will be used to boot the virtual machine.

The virtual machine will check the devices in order, attempting to boot from each, until the virtual machine boots successfully. If the list is empty, the virtual machine will use a default boot sequence.

There should be no more than one instance of Device.Entry for a given device type except ETHERNET in the list.

PUT https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/boot/device

Returns information about a virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}

Deletes a virtual machine

DELETE https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}

Returns the virtual hardware settings of a virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware

Updates the virtual hardware settings of a virtual machine.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware

Upgrades the virtual machine to a newer virtual hardware version.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/action/upgrade

Returns the memory-related settings of a virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/memory

Updates the memory-related settings of a virtual machine.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/memory

Returns the CPU-related settings of a virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cpu

Updates the CPU-related settings of a virtual machine.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cpu


Returns information about the Ethernet adapters belonging to the virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/ethernet

Returns information about a Ethernet adapter.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/ethernet/{nic}

Adds a Ethernet adapter to the virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/ethernet

Removes a Ethernet adapter from the virtual machine.

DELETE https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/ethernet/{nic}

Updates the configuration of a Ethernet adapter.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/ethernet/{nic}

Connects a Ethernet adapter of a powered-on virtual machine.

For a powered-off virtual machine, the Ethernet.update operation may be used to configure the virtual Ethernet adapter to start in the connected state when the virtual machine is powered on

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/ethernet/{nic}/connect

Disconnects a Ethernet adapter of a powered-on virtual machine.

The virtual device is still present and its backing configuration is unchanged, but from the perspective of the guest operating system, the Ethernet adapter is not connected to its backing resource.

For a powered-off virtual machine, the Ethernet.update operation may be used to configure the virtual Ethernet adapter to start in the disconnected state when the virtual machine is powered on.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/ethernet/{nic}/disconnect

Virtual disk
Returns information about the virtual disks belonging to the virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/disk

Updates the configuration of a virtual disk.

An update operation can be used to detach the existing VMDK file and attach another VMDK file to the virtual machine.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/disk/{disk}

Returns information about a virtual disk.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/disk/{disk}

Adds a virtual disk to the virtual machine.

While adding the virtual disk, a new VMDK file may be created or an existing VMDK file may be used to back the virtual disk.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/disk

Removes a virtual disk from the virtual machine.

This operation does not destroy the VMDK file that backs the virtual disk. It only detaches the VMDK file from the virtual machine. Once detached, the VMDK file will not be destroyed when the virtual machine to which it was associated is deleted.

DELETE https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/disk/{disk}

SATA Adapters

Returns information about a virtual SATA adapter.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/adapter/sata/{adapter}

Returns information about SATA adapters belonging to the virtual machine.


Adds a virtual SATA adapter to the virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/adapter/sata

Removes a virtual SATA adapter from the virtual machine.

DELETE https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/adapter/sata/{adapter}

SCSI Adapters

Returns information about a virtual SCSI adapter.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/adapter/scsi/{adapter}

Returns information about the SCSI adapters belonging to the virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/adapter/scsi

Adds a virtual SCSI adapter to the virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/adapter/scsi

Removes a virtual SCSI adapter from the virtual machine.

DELETE https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/adapter/scsi/{adapter}

Updates the configuration of a virtual SCSI adapter.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/adapter/scsi/{adapter}


Returns information about a virtual CD-ROM device.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cdrom/{cdrom}

Returns information about the virtual CD-ROM devices belonging to the VM.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cdrom

Adds a virtual CD-ROM device to the virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cdrom

Removes a virtual CD-ROM device from the virtual machine.

DELETE https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cdrom/{cdrom}

Updates the configuration of a virtual CD-ROM device.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cdrom/{cdrom}

Connects a virtual CD-ROM device of a powered-on virtual machine.

Connecting the virtual device makes the backing accessible from the perspective of the guest operating system.

For a powered-off virtual machine, the Cdrom.update operation may be used to configure the virtual CD-ROM device to start in the connected state when the virtual machine is powered on.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cdrom/{cdrom}/connect

Disconnects a virtual CD-ROM device of a powered-on virtual machine.

The virtual device is still present and its backing configuration is unchanged, but from the perspective of the guest operating system, the CD-ROM device is not connected to its backing resource.

For a powered-off virtual machine, the Cdrom.update operation may be used to configure the virtual CD-ROM device to start in the disconnected state when the virtual machine is powered on.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/cdrom/{cdrom}/disconnect

Unmounts a previously mounted CD-ROM using an ISO image as a backing.

POST https://vcentersrv01.alex.local/rest/com/vmware/vcenter/iso/image/id:{vm}?~action=unmount

Serial Ports
Returns information about a serial port.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/serial/{port}

Returns information about the serial ports belonging to the virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/serial

Adds a virtual serial port to the virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/serial

Removes serial port from the virtual machine.

DELETE https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/serial/{port}

Updates the configuration of a serial port.

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/serial/{port}

Connects a virtual serial port of a powered-on virtual machine to its backing.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/serial/{port}/connect

Disconnects a virtual serial port of a powered-on virtual machine from its backing.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/serial/{port}/disconnect

Parallel Ports
Returns information about a virtual parallel port.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/parallel/{port}

Returns information about the parallel ports belonging to the virtual machine.

GET https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/parallel

Adds a virtual parallel port to the virtual machine.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/parallel

Removes parallel port from the virtual machine.

DELETE https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/parallel/{port}

Updates the configuration of a virtual parallel port

PATCH https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/parallel/{port}

Connects a parallel port of a powered-on VM.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/parallel/{port}/connect

Disconnects a virtual parallel port of a powered-on virtual machine from its backing.

POST https://vcentersrv01.alex.local/rest/vcenter/vm/{vm}/hardware/parallel/{port}/disconnect

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in vSphere 6 | 1 Comment

Exploring vSphere 6.5 API-Part 1: Datacenter & Cluster

I am using API for quite a bit now in our prod environment which is based on vCloud Director and many times API’s had proved a handy way to troubleshoot issues where GUI was not providing a way to proceed.

Inspired by vCD API’s, i decided to test that in my vSphere 6.5 lab and in this post I will try to demonstrate few queries which can be helpful in fetching info in your infrastructure.

In my lab I am exploring REST API’s using a linux tool called curl.

1: You can browse list of API’s by browsing https://vc-fqdn/ and clicking on “Browse vSphere Rest API’s”



2: To start with you can use below query to see what are the different options available

# curl -sik -H ‘Accept:application/json’ -u “vc-user” -X GET https://vc-fqdn/rest/

You will see below URL’s in output:



3: You can list the available components which can be explored via REST API by using below query

# curl -sik -H ‘Accept:application/json’ -u “vcadmin@alex” -X GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/component

Available Options

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.vcenter.ovf

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=applmgmt

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.cis

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.vcenter.inventory

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.vcenter

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.vapi.vcenter

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.cis.tagging

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.content

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=vmon_vapi_provider

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.vcenter.iso

GET https://vcentersrv03.alex.local/rest/com/vmware/vapi/rest/navigation/service?component_id=com.vmware.vapi

Let’s start with exploring available options for Datacenter and Clusters

List all datacenter in a vCenter

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: cf2cc0c3b07bd3a01224f06aa00fea59’ -X GET https://vcentersrv01.alex.local/rest/vcenter/datacenter

Output of above query will contain name of all datacenter that has been created in a vCenter



Details about particular datacenter

You can fetch details like host_folder, network_folder etc for a specific datacenter by using below query. You can use those details in doing scripting for automated deployment of stuffs.

Note: Do not use datacenter name in below query. Use id which you obtained in above step.

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: cf2cc0c3b07bd3a01224f06aa00fea59’ -X GET https://vcentersrv01.alex.local/rest/vcenter/datacenter/datacenter-2






Delete an empty datacenter

You can delete an empty datacenter using below query

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: cf2cc0c3b07bd3a01224f06aa00fea59’ -X DELETE https://vcentersrv01.alex.local/rest/vcenter/datacenter/datacenter-58

If you get Http 200 OK it means query have been executed successfully. You can also verify successful deletion of datacenter from vCenter Web Client.

HTTP/1.1 200 OK
Date: Thu, 15 Dec 2016 18:16:17 GMT
Content-Length: 0

Force removal of datacenter (when datacenter is not empty)

If a datacenter is not empty, you can force delete the contents and datacenter by specifying force=true in the API query

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: cf2cc0c3b07bd3a01224f06aa00fea59′ -X DELETE https://vcentersrv01.alex.local/rest/vcenter/datacenter/datacenter-68?force=true&#8217;


Show all cluster in a vCenter

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 4ca9afd3971255748599bc9e47c4f4a1’ -X GET https://vcentersrv01.alex.local/rest/vcenter/cluster

Output of above query will list all clusters present in a vCenter. Also you will get cluster id and details about cluster properties like HA/DRS etc.


List particular Cluster

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 4ca9afd3971255748599bc9e47c4f4a1’ -X GET https://vcentersrv01.alex.local/rest/vcenter/cluster?filter.names=Cybertron

Output will tell you the resource_pool id which is associated with cluster ‘cybertron’


List all cluster in a given Datacenter

If you have multiple virtual datacenters in vcenter and you want to fetch details of all clusters which are in a given datacenter, you can do so by applying a filter with datacenter id in the query.

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 4ca9afd3971255748599bc9e47c4f4a1’ -X GET https://vcentersrv01.alex.local/rest/vcenter/cluster?filter.datacenters=datacenter-2

Details of a particular cluster 

Details of a particular cluster can be fetched using below query.

Note that after /cluster I have given cluster id and not the cluster name. If you specify cluster name here, you will get error that ‘resource do not exist’

# curl -sik -H ‘Accept: application/json’ -H ‘vmware-api-session-id: 4ca9afd3971255748599bc9e47c4f4a1’ -X GET


Thats it for this post. This a very basic overview of using API calls. There is a lot to explore and I will try to dive a bit deep in future posts of this series.

Posted in Vmware | 2 Comments

vCloud Air: Access Your Linux Server Over SSH From Outside

This week while working in my lab, I came across a situation where I wanted to run few commands on my linux server which is running in vCloud Air. To access my lab from outside I have configured a Windows jump server and from there I access my lab components (using SSH or RDP to other server). At times it is annoying to switch back and forth between your home desktop and the RDP session.

I went ahead and configured my main Linux server to access it over SSH directly from my home computer (without logging into my windows jump server).

This process is not difficult, but as a habit I tend to document the new things which I do in my lab, and so I am writing a blog post on the same.

Prerequisite: The only prerequisite for this process is that you should have a Public IP purchased and assigned on your edge gateway.

1: To start with the process, login to and navigate to your virtual datacenter.


2: Within your VDC, navigate to Gateways tab and double click on your edge gateway.


3: Add a DNAT rule for access to your server from outside by clicking on Add button.


4: Under External IP, select the Public IP which via which you want to access your server.

You can leave the source and destination port to Any-Any at the moment (we will see later how to modify this rule for a specific port)

Under Translated IP, provide the local IP of your linux server which you want to access.


5: Hit finish button to add the NAT rule to your Edge gateway.


6: It will take a few seconds to apply this config on your edge.


7: After few seconds, you can verify that the newly added rule appears under NAT rule list.


8: Next is to add a firewall rule for the outside access. You can add the rule by navigating to ‘Firewall Rules’ tab and clicking on Add button.


9: You can name this rule as per your choice. Also if you wish you can chose to log the traffic details (a good idea for auditing purpose).

You can select source as ‘ANY’ (if you want to access this server from anywhere) or you can limit the source to an IP or range of IP’s (in case if you want this server to be accessed only from your on-prem production environment and not from home)

Under Destination, select Specific CIDR/IP option and provide the Public IP in the box.

Unfortunately this window do not give you ability to limit this rule to a specific port. But no need to worry about it as we will see in next steps how to achieve that.

Hit Next when you are done with filling the required entries.



10: Hit ‘Finish’ to complete the firewall rule creation process.


11: Now its time to modify the NAT/firewall rules (in order to limit them to a specific port to harden the security)

To do so navigate back to Gateways tab and click on ‘Manage in vCloud Director’


12: From the VCD page, select your edge and right click on it and chose ‘Edge Gateway Services’. A new window will pop-up here (unless you have subscribed to Advance Networking Services where you will be taken to hybridity page)


13: Under NAT, select the DNAT rule created by you and hit edit.


14: You can leave original port to ANY. Change the translated port to 22 (or any other, in case you have changed  default ssh port on your server). Hit OK


15: Now from the firewall rule list, select the newly created rule and click edit button.


16: Again you can leave source port to ANY and change the destination port to your SSH port. Make sure you have select allow as Action for this rule. Hit OK.


17: Thats it. You can access your server directly from your source computer now.


I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in vCloud Air | Leave a comment

Learning vSphere 6.5-Part-10-VCHA failover testing

Last 2 post of this series were revolving around the high availability feature for vCenter that is introduced in vSphere 6.5 and we discussed the VCHA architecture and also learnt how to configure VCHA.

In this post we will be testing the HA feature and will see what happens when the Active Node of VCHA cluster goes down.

If you have missed earlier post of this series, you can read them from below links:

1: Installing and Configuring Esxi

2: VCSA Overview

3: vCenter Server and PSC Deployment Types

4: System Requirements for Installing vCenter Server

5: Installing vCenter Server on Windows

6: Deploying vCSA with embedded PSC

7: Deploying External PSC for vCSA

8: Understanding vCenter Server High Availability (VCHA)

9: Configuring vCenter Server High Availability (VCHA)

Lets jump into lab and test this awesome feature.

We will be testing failover via 2 method:

  • Automated failover (let system do the magic)
  • Manual failover (user will intentionally bring down active node of VCHA cluster)

Automated Failover Testing

1: To test the failover, login to vCenter web client and navigate to Configuration > vCenter HA and before performing a failover look at the Active/Passive node info and note which IP is active at the moment.

To start failover, hit the Initiate Failover button at top right corner.


2: System will ask you that if you want to initiate the failover process and if you want to start failover immediately. If you check this box, then the recent DB changes from Active to Passive will not be replicated to Passive node.

I think it should be best to let database commit the running transactions to the passive node and because of that I chose not to check mark the immediate failover option.



3: Once failover is initiated and if you try access IP/FQDN of active node you will see below screen


4: Once the failover is completed, you will notice that the previous passive node now had become active. Compare this screenshot with screenshot shown in step 1 and you will notice the difference in IP for the active node.


5: If you click on HA monitoring tab, system will report you that all nodes are up and running and overall health status of VCHA cluster is good and application state/ DB replication etc are all in place and working fine.

So what happens in automated failover testing is that the Active node is forced to fail by system so that the Passive node will become Active and the active node will become passive once it is recovered from failed state (recovery is done by system itself)


2: Manual Failover Testing

A: To perform a manual failover testing, lets power off the Active node intentionally.


B: After few seconds of powering off the Active node, if you try to access the vCSA IP you will see message about “Failover in Progress”


c: Once the failover is completed and vSphere web client allow you to login again, you will observe that  health status of VCHA cluster is deteriorated and now you have a new active node and the previous active node had become passive and is currently down (because we have not powered on the node yet)


D: Also if you go to Monitoring tab for VCHA, you will see that system is reporting that VCHA cluster has lost one of its node and the DB replication between Active node and Passive node is not happening.  Also you will notice Application state out of sync.


At this point I hope you have understood the difference between the Automated failover test and Manual failover test and what happens when during failover.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | Leave a comment

Learning vSphere 6.5-Part-9-Configuring vCenter Server High Availability (VCHA)

In last post of this series we discussed about the High Availability feature for vCenter 6.5 and saw the architecture of VCHA and how it works. In this post we will go ahead and actually configures that in lab.

If you have missed earlier post of this series, you can read them from below links:

1: Installing and Configuring Esxi

2: VCSA Overview

3: vCenter Server and PSC Deployment Types

4: System Requirements for Installing vCenter Server

5: Installing vCenter Server on Windows

6: Deploying vCSA with embedded PSC

7: Deploying External PSC for vCSA

8: Understanding vCenter Server High Availability (VCHA)

Before jumping into lab and start configuring VCHA, its important to understand the deployment options and prerequisites  for VCHA first.

vCenter HA Deployment Options

You can set up your vCenter HA environment with an embedded PSC or with an external PSC. If you decide to use an external Platform Services Controller, you can place it behind a load balancer for protection in case of Platform Services Controller failure.

To know more about deployment option please refer  VMware documentation for vSphere 6.5

vCenter High Availability (VCHA) Prerequisites

  • An instance of the vCenter Server Appliance deployed.
  • Separate datastores to deploy all three nodes. (If possible)
  • At the least three ESXi hosts to deploy the three nodes.
  • A public IP Address that will be used to connect to the active node. (provided during the installation of the appliance).
  • Three private IP addresses (1 IP per node) for replication and internal communication called the internal cluster network.
  • Separate port group for vCenter HA network. When you configure vCenter server 6.5 HA, second vNIC  will get added (eth1) during the configuration.This vCenter HA network is used for internal communication between the vCenter HA nodes.

Network Configuration

Before proceeding with deployment ensures networking has been setup correctly. VMware recommends to have a separate network configured for VCHA traffic. To set the foundation for the vCenter HA network, you add a port group to each ESXi host, and add a new virtual NIC (eth1) to the vCenter Server Appliance that later becomes the Active node.


Requirements for vCenter HA network

  • The vCenter HA network IP addresses for the Active, Passive, and Witness nodes must be static and map to FQDNs.
  • The vCenter HA network must be on a different subnet than the management network. The three nodes can be on the same subnet or on different subnets.
  • Network latency between the Active, Passive, and Witness nodes must be less than 10 milliseconds.

Lets jump into lab and see the actual deployment process.

1: To configure vCenter HA login to the vCSA appliance and select vCenter from top and navigate to Configuration > vCenter HA and click on configure button.


2: In select configuration option you get 2 choices i.e go with Basic or Advanced option. Lets discuss a little bit about these before moving forward.

Basic Configuration Workflow

Basic configuration automatically clones the Active node. You must meet one of the following requirements to perform Basic configuration.

  • Either the vCenter Server Appliance that will become the Active node is managing its own ESXi host and its own virtual machine. This configuration is sometimes called a self-managed vCenter Server.
  • Or the vCenter Server Appliance is managed by another vCenter Server (management vCenter Server) and both vCenter Server instances are in the same vCenter Single Sign-On domain. That means they both use an external Platform Services Controller and both are running vSphere 6.5.

If you meet the requirements the Basic workflow is as follows.

1: Deploy the first vCenter Server Appliance, which will become the Active node.

2: Adds  second network (port group) for vCenter HA traffic on each ESXi host.

3: Sart the vCenter HA configuration, selects Basic and supplies the IP addresses, the target ESXi host or cluster, and the datastore for each clone.

4: The system clones the Active node and creates a Passive node with precisely the same settings, including the same host name.

5: The system clones the Active node again and creates a more light-weight Witness node.

6: The system sets up the vCenter HA network on which the three nodes communicate, for example, by exchanging heartbeats and other information.

Advanced Configuration Workflow

If you environment do not meet criteria for  Basic option or you want more control over your deployment, you can perform Advanced configuration.

With this option, you are responsible for cloning the Active node yourself as part of vCenter HA setup. If you select this option and remove the vCenter HA configuration later, you are responsible for deleting the nodes that you created.

For the Advanced option, the workflow is as follows.

1: Deploy the first vCenter Server Appliance, which will become the Active node.

2: Add a second network (port group) for vCenter HA traffic on each ESXi host.

3: Add a second network adapter (NIC) to the Active node.

4: Login to the vCenter Server Appliance (Active node) with the vSphere web client and start the vCenter HA configuration, selects Advanced, and supplies IP address and subnet information for the Passive and Witness nodes. Optionally youcan override the failover management IP addresses.

5: Login to the management vCenter Server and creates two clones of the vCenter Server Appliance (Active node).

7: Return to the configuration wizard on the vCenter Server Appliance and completes the configuration process.

8: The system sets up the vCenter HA network on which the three nodes exchange heartbeats and replication information. The vCenter Server Appliance is  now protected by vCenter HA.

In my lab I chose to go with Advanced option.


3: Provide the IP for the Passive Node and Witness node.

Note: This IP should be in different subnet than your management network to which first NIC of vCSA is connected.


4: On the clone VM page do not hit finish.

Keep this page opened until additional steps is completed.


5: Return to the management vCenter server and chose the vCSA appliance and create a clone of this VM.


6: Chose where you want to deploy the cloned VM and hit Next.


7: Select the cluster/host where you want to deploy the clone. You can chose individual Esxi host for deployment or leave the selection to cluster if you have a DRS enabled cluster.


8: Select destination datastore for cloned VM. Its recommended to keep Active, passive and Witness nodes on different datastores.


9: On select clone options page, check mark ‘customize the OS’ and ‘Power on VM’ option.


10: On customize guest os page, click the + button to create a new guest os customization specification template.


11: Provide a name for the template and hit Next.


12: Under computer name, give the same hostname which you had set on vCSA that is going to be active Node. Also provide the domain name same as Active vCSA node.


13: Ensure the timezone is consistent with the Active node.


14: Under configure Network option, select “Manually select custom settings’. Select NIC 1 and click on pencil button to edit the settings.


15: Select IPv4 and chose 3rd option and enter the IP information for the passive node.

Provide the same IP address here which you have set on management NIC (eth0) of the Active vCSA node. 

If you give any other IP address here, the VCHA deployment is going to fail.


16: For NIC 2 of the passive node, enter the same IP address which you had given in step 3.

Do not enter any gateway IP for NIC 2


17: Hit Next when you have configured both NIC cards for passive node.


18: Under DNS settings, enter Primary DNS server and in search path provide your domain name and hit Next.


19: On ready to complete page, review your settings and hit Finish.


20: Now under customize guest os page you can see the newly created customization specification template. Chose that and hit Next.

The cloned VM will take the values (which you provided in step 12-19) from this template.


21: Under customize vApp properties, review the values that is going to be applied on the cloned VM and hit next.


22: On ready to complete page, have a good look on the values again because this is best chance to modify any values that you might have supplied wrong. Once you hit finish and realize that you have missed something, then again creating a clone is the only option.

Hit finish post reviewing your settings.


23: Repeat Steps 5-22 for witness node.

24: Once both Passive and Witness node have been deployed and are up and running, go back to Active vCSA where you have to finish the HA configuration which you paused in step 4.


25: Now you will see the message on vCenter HA page that “vCenter HA is being configured”


26: Under Tasks you can monitor the status of configuration. You will see a new HA cluster being configured which comprises of your Active Node, Passive Node and Witness node.



27: Once the configuration is completed, You can see the message that vCenter HA is enabled and also can see the status of Active, passive and Witness node here.


28: Clicking on vCenter HA monitoring, will take you to the monitoring tab where you can see more details about VCHA like replication status etc. You can also monitor tasks and events or if there are any issues with VCHA config from here.


I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 1 Comment

Learning vSphere 6.5-Part-8-Understanding vCenter Server High Availability (VCHA)

In last post of this series learned how to External PSC for vCSA. In this post of vSphere 6.5 series, we will look into one of the best feature which is included in v 6.5 i.e high availability for vCenter Server.

If you have missed earlier post of this series, you can read them from below links:

1: Installing and Configuring Esxi

2: VCSA Overview

3: vCenter Server and PSC Deployment Types

4: System Requirements for Installing vCenter Server

5: Installing vCenter Server on Windows

6: Deploying vCSA with embedded PSC

7: Deploying External PSC for vCSA

Customers were demanding a lot for high availability solution for vCenter Server for quite some time and VMware listened to customers and made this happen in version 6.5 of vSphere.

What is vCenter High Availability (VCHA)?

vCenter High Availability (vCenter HA) protects vCenter Server Appliance against host and hardware failures. The active-passive architecture of the solution can also help you reduce downtime significantly when you patch vCenter Server Appliance.

Note: vCenter HA is only available with vCSA and not windows based vCenter Server.

VCHA Architecture

High Availability for vCSA is achieved by creating a three-node cluster which comprises of one Active node, one Passive node, and one Witness node.

Active node is nothing but the original vCSA appliance that you deploy in your environment post deployment of external PSC. This is the primary node where you connect using the vSphere Web-Client.

The Passive Node is a clone of Active node and since this is a clone it is identical to the Active node in every aspect i.e same CPU, Memory, and disk.

The Third node is the Witness node, a lightweight node that is there to form the Cluster Quorum. Witness node is responsible to verify the heartbeat interval in order to ensure failover occurs as a result of VM unresponsiveness or a host crash.


Active & passive node share a common public ip. During the failover, passive node would activate its nic which is assigned with same public ip taking over the complete charge whenever active node fails. Rest of the time NIC status is down in order to avoid ip conflict.

All the three nodes communicate via an internal IP also called as cluster ip. These three nodes would share a same subnet in order to communicate internally.

A minimum of two nodes must be online. You cannot suffer the loss of more than one node in the VCHA cluster and expect to continue to have service.

When the active node suffers failure, the passive node becomes active and thus help minimizing downtime for the vCenter server. Below diagram illustrates the process of failover.


Graphic Thanks to Féidhlim O’Leary

Note: The Witness is not capable of ever taking over the role of active and becoming a usable vCenter Server instance.

How VCHA works?

When VCHA is configured and HA cluster is formed, replication starts between the active node and the passive node. The embedded database i.e vPostgres captures the state of the appliance and info about the config.

There are two different types of replication that takes place in VCHA cluster:

File-level replication: This basically consists of all the configuration and log related data being replicated which is achieved using the mechanism known as “rsync” keeping the data upto date.

vPostgres DB replication: This is done to replicate the VCDB database and the VUM database. Database replication is needed to ensure during the failover we do not lose any changes that are being actively performed on vCenter DB.

If the active node has a failure, then the current Passive node takes on the role of the Active node. The Public IP address is switched from the Active to the Passive node and the clients now connect to the now active node, once the vCenter Services are started.


Graphic Thanks to Féidhlim O’Leary

When a failover occurs, the state of replication is reversed as the former active node which is now passive (because of failure) comes back online and re-joins the VCHA Cluster and becomes active again.

Note: Replication always flows from the current active to the current passive.


Graphic Thanks to Féidhlim O’Leary

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 4 Comments

Learning vSphere 6.5-Part-7-Deploying External PSC for vCSA

This is part 7 of the vSphere 6.5 learning series and in last part of this series we learned how to install vCSA with embedded PSC and touched on some basic settings which is needed to be done post deployment.

In this post we will learn how to deploy external PSC for vCSA. The goal here is to deploy external PSC first and then install vCSA which will use the external PSC.

If you have missed earlier post of this series, you can read them from below links:

1: Installing and Configuring Esxi

2: VCSA Overview

3: vCenter Server and PSC Deployment Types

4: System Requirements for Installing vCenter Server

5: Installing vCenter Server on Windows

6: Deploying vCSA with embedded PSC

In last post we discussed and saw how easy VMware has made deployment process of vCSA and the process has changed a bit. The steps needed to deploy external PSC is no different then what we learned in last post.

1: Post downloading the vCSA iso and mounting it in DVD drive, we need to navigate to  ‘vcsa-ui-installer\win32’ and click on installer exe to launch the deployment wizard.


2: The deployment wizard will tell you that installation is a 2 step process.


3: Accept EULA and hit Next.


4: Select Platform Services Controller and hit Next.


5: Select esxi host or vCenter as deployment target for PSC and provide credentials to connect to this target and hit Next.


6: Accept the SSL certificate presented to you.


7: Select folder in which you want to deploy the appliance.


8: Chose cluster/host where PSC will be deployed and hit next.


9: Chose a hostname for PSC and enter the root credentials.


10: Select destination datastore for the deployment.


11: Enter networking information for the appliance and hit Next.


12: On ready to complete page, review your settings and hit and finish to start the deployment.


13: Let the wizard do the rest for you and complete first phase of installation.



14: Hit continue post phase 1 of deployment is completed.


15: Hit Next to start configuration phase of the installation.


16: Enter NTP settings for PSC appliance and enable SSH for remote access via CLI.


17: If you have any existing SSO domain, you can make PSC part of that domain or can create a new one if don’t have any existing domain.


18: If you want to take part in customer Experience Improvement Program check mark the Join CEIP button and hit Next.


19: On final screen review your settings and hit finish to start second phase of deployment.


20: Wait for deployment to finish.



21: Login to PSC by accessing https://psc_fqdn/psc and login with user@sso-domain which you created just a while back.


22: Next is to make PSC part of your organization domain.

Under Appliance Settings click on manage tab and click on Join button to add your domain info.


23: Populate your domain info and hit OK. PSC will be rebooted for changes to take effect.


24: You can also manage additional settings related to PSC appliance by clicking on ‘View Platform Services Appliance’


25:  Login to PSC VAMI page using root credentials.


26: You can monitor the overall health of PSC fro m this page and can also configure additional settings related to Network/ NTP syslog etc.


27: Go back to PSC page and click on Configuration > Identity Source to add identity source to your PSC by clicking on ‘+’ button.


28: If you have windows based AD select AD (integrated Windows Authentication) and enter your domain information and hit OK.


There is nothing more than this to configure on PSC appliance at the moment.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 3 Comments

Learning vSphere 6.5-Part-6-Deploying vCSA with embedded PSC

In last post of this series we learnt how to install vCenter with embedded PSC on windows server.

In this post we will learn about deploying and configuring vCSA with embedded PSC.

If you have missed earlier post of this series, you can read them from below links:

1: Installing and Configuring Esxi

2: VCSA Overview

3: vCenter Server and PSC Deployment Types

4: System Requirements for Installing vCenter Server

5: Installing vCenter Server on Windows

In vSphere 6.5 VMware has made the installation of vCenter appliance a lot easier than before. Earlier, deployment of vCSA needed client integration plugin to be installed on your system and the deployment was done through a browser. Client integration plugin had multiple compatibility issues. Now with vSphere 6.5 client integration plugin is no longer required. The deployment is going to be via an ISO which would have an installation wizard that can be executed on Windows, MAC or Linux.

The vCenter Server Appliance consists of a 2-stage deployment.
1. Deploying vCSA 6.5
2. Configuring vCSA 6.5

Since vSphere 6.5 is now GA, vCSA can be downloaded from here. Download the vCSA iso on any of the machine which can communicate with your existing virtual infrastructure and navigate to directory ‘vcsa-ui-installer\win32’ and click on installer exe to launch the deployment wizard.

1: Click on install button to start the wizard.


2: The deployment wizard will tell you that there are two steps involved in the installation. The first step will deploy a vCenter Server appliance and the second step will be configuring this deployed appliance.


3: Accept EULA and hit Next.


4: In the deployment type choose the one which suits your need. If you want to know more about available deployment type and requirements, you can read this article which I posted earlier.


5: Next is to choose the target for deployment. You can either choose Esxi host as target or any existing vCenter server. In my lab I already have a windows based vCenter server and I choose that as target.

Note: If you select Esxi host as target, provide username as root and password of Esxi host. With vCenter you can chose a windows account that have access to vCenter server.


6: Accept the thumbprint of the SSL certificate presented.


7: Select the folder where you want to deploy vCSA and hit next.


8: Chose cluster/host where vCSA will be deployed and hit next.


9: Provide a name for the vCenter appliance VM and set the root password.


10: Based upon your environment size, select the sizing of the vCenter appliance.


11: Select the destination datastore where appliance will be deployed and hit next. You can chose to deploy appliance in a thin provisioned mode by checking the box “Enable Thin Disk mode’


12: Configure the networking of vCenter appliance.  Make sure to use a valid IP which can be resolved both forward / reverse prior to this to prevent any failures during installation.


13: On ready to complete page, review your settings and hit and finish to start the deployment.


14: Let the wizard complete first phase of installation.




14: If you have chose existing vCenter as target during deployment, you can watch the deployment progress there.


16: Hit continue to begin second phase of deployment once phase 1 of deployment is completed.


17: Hit Next to start with phase 2 of deployment i.e configuring vCSA.


18: The first option that appears in phase 2 is to configure NTP  settings for the appliance and enable SSH access.


19: Next we have to define the SSO domain name, the SSO password and the Site name for the appliance.

20: If you would like to enable Client Experience Improvement Program, you can do so, else you can skip and proceed to completion.


21:  On Ready to Complete page, review your settings and hit finish to start the phase 2 of deployment.


At this point the deployment wizard gives you one more chance to review your settings by clicking cancel. Hitting OK will trigger the appliance configuration as per your selections made above.


22: You can sit back and relax now and let the wizard do the magic. It will take 5-7 minutes typically for configuration to finish.



23: Post completion of deployment, hit close. You can manage the appliance now by opening URL https://vcsa_fqdn:443/vsphere-client/


24: Login to appliance using user@sso_domain_name and password


25: On logging in, you will be presented with a warning message that your vCenter server needs to be licensed. You can go ahead and assign the license immediately or can dismiss the message if your are planning to use vCenter in evaluation mode or will apply license later.


26: The next thing you wanna configure is to make vCSA part of your company’s domain so that domain users (vsphere admins) can access vCenter using their own credentials. You can do so by clicking on “System Configuration” within home page.


27:  Select Nodes and select your vCSA and click on Manage tab. Under settings page,select Active Directory. Click on Join button to supply your domain information.


28: Define your domain name and a user using which vCSA will be attached to domain. The username is preferred to be a service account. Wizard will warn you that a reboot is required post adding vCSA to domain for changes to take effect.


29: To reboot, right click on the vCenter and select reboot.


30: If you wish you can provide a reason for why you are rebooting the appliance.


31: Post reboot of appliance, you can verify that its now joined to the domain.


32: If at this point you are hoping that you can now login to vCSA using your Active directory account, you are wrong. We have not yet finished the configuration.

Next step in this is to define the identity source for your appliance. From home page go to Administration > Configuration. Click on green ‘+’ button to add an identity source.


33: Select the identity source type. I have my domain running as AD in windows.


34: Provide the domain name here and chose use machine account and hit next.


35: Hit finish to complete adding identity source.


36: Now under Global permissions hit + button to provide access to your domain users on this vCSA instance.


37: Click on Add button to start adding the user/users and assign appropriate role to the user.


And that’s it. Now we can login to vCSA using the ad account which were added in previous step.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 4 Comments

Learning NSX-Part-11-Replacing NSX default SSL Certficates with CA Signed Certificates

I am a big advocate of not using the default SSL certs on any VMware products and I prefer using Signed certs from my CA server on my lab components. I have my CA server running in Windows Server 2012.

Earlier in my lab I had replaced the vSphere (Esxi + vCenter) SSL certs and if you want to know how to do it, you can read them from below links:

1: Replacing Esxi SSL Certificates

2: Replacing vCenter Server SSL Certs

If you are like me and new to replacing SSL certs and looking for how to setup a CA server, you can read it from Here for a step by step installation/configuration of CA server.

1: Introduction to VMware NSX

2: Installing and Configuring NSX Manager

3: Deploying NSX Controllers

4: Preparing Esxi Hosts and Cluster

5: Configure VXLAN on the ESXi Hosts

6: Logical Switching

7: Distributed Logical Router Tidbits

8: Installing Distributed Logical Router

9: NSX Edge Services Gateway

10: Upgrade NSX Manager From 6.2 to 6.2.4

All right lets dive into lab and look into how to replace the default SSL certs of VMware NSX.

Before starting with certs replacement, we need to create a customized certificate template in CA server prior to submitting the Certificate Signing Request (CSR) created on the NSX Manager.

You can follow VMware KB-2112009 to learn how to create certificate template. I have already created mine when I replaced SSL certs in my vSphere 6 infrastructure.

1: Login to NSX manager with the admin user. At the top you can see the Red triangle sign which indicates that certificate is not a signed cert.


2: Go to Manage > SSL Certificates and click on Generate CSR


3: Populate the information needed to generate CSR and hit OK


4: Now you download the CSR by clicking on ‘Download CSR’ button.


Note: The download does not give you a .csr file but instead gives you a file with the type “File.”

5: Open the downloaded file in a notepad. It should look like below. Copy entire contents from this file including –Begin Certificate Request— and —End Certificate Request— line


6: Login to your CA server by browsing https://FQDN_or_IP/certsrv and click on “Request a certificate”


7: Click on advanced certificate request.


8: Paste the content which you copied in step 5 under Saved Request and select the correct Certificate Template and hit Submit.


9: Select Base 64 encoded and click on Download certificate chain


10: Right click on downloaded chain file and select open, then drill down to certificates directory and in right hand side you will see your cert.

Right click on the cert which corresponds to NSX and click on Export.


11: Select Base-64 encoded X.509 in Certificate Export Wizard and hit Next.


13: Save the file as .cer file on your computer.



14: Now we need to download CA Server root certificate. From the CA Server webpage click on home on right hand side upper corner and click on “Download a CA certificate” option.


15: Select Base 64 option and click on “Download CA certificate


16: At this point you should have 2 .cer files


We need to join these file to create a single .cer file. You can use below command to do that

copy nsxmgr.cer+Root64.cer  new.cer

17: Once you have the new.cer file, log back into the NSX Manager Web Interface and select Import and provide your new.cer file.


Browse to the location where you have stored your new.cer file and hit Import buttonnsx-ssl18

You should now see your new certificate and root certificate as show below. Also the NSX Web UI will display a message that appliance needs a reboot for new certs to take effect.


18: Reboot the NSX appliance. Once NSX manager is UP again, log back into the UI and you should see the certificate is now showing in green color. If its shows red eve after appliance reboot just refresh the page and you should immediately see red changing to green.



And that’s it. We have successfully replaced the default self-signed certificate of NSX manager with a signed certificate obtained from our Certificate Authority.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in NSX | Leave a comment

Learning NSX-Part-10-Upgrade NSX Manager From 6.2 to 6.2.4

This week I was trying my hands on upgrading NSX to version 6.2.4 which was released earlier this year in August.

I had no experience earlier with upgrading NSX manager and associated components, so I spent a lot of time in reading blogs and watching videos on how to perform the upgrade.

Before starting with upgrade process please consult the NSX 6.2.4 Release Notes and also follow VMware KB-2144295 which explains recommended minimum versions for VMware NSX for vSphere, ESXi, vCenter Server and Guest Introspection Driver (GID).

To keep track of latest versions you can use vTracker or the Latest Versions page hosted on

You can follow the below upgrade matrix to find out the upgrade path available in your environment.


While going through the blogs on how to start upgrading NSX to 6.2.4, I came across screenshot of a cool table which Anthony has posted on his Blog.

This table gives you important info on how and when to upgrade from previous versions of NSX and also it gives a great overview of the pathways required to get to 6.2.4.


Below screenshot provides a high level overview of the upgrade procedure:


Pre Upgrade Considerations:

  1. Consult NSX release notes.
  2. Download NSX upgrade bundle and check MD5.
  3. Take backup of the NSX infrastructure. Snapshot the NSX manager VM prior to upgrade, take backup of configuration from Web-Client.

Additionally you can:

  • Take backup of vCenter Server database.
  • Snapshot vCenter Server and vCenter Server database.

To start with the upgrade process, go to the NSX download page and download the upgrade file ‘NSX for vSphere 6.2.4 Upgrade Bundle’


Verify the download upgrade bundle file name ended with tar.gz Some browser may remove the gz extension, if the file look like:


Change it to:


Lets start with upgrade process.

*******************NSX Manager Upgrade**************************

1: Login to NSX manager admin console and click on Upgrade button.


2: From the Manage page, click on upgrade button.


3: Click “Browse” and open the download upgrade file and click Continue:


4: System will take some time to upload the file.


Note: NSX Manager will reboot during upgrade process, traffic flow of existing VM workloads will not affected during this step.

5: Post upload of file you will be presented with below page. Choose if you want to enable SSH and participate in Customer Experience Improvement Program .

Click Upgrade to proceed.


6: Wait for upgrade to complete.


7: Post upgrade is complete you can verify the NSX manager version.


8: Make sure all services are running on NSX manager before proceeding with Next steps of upgrade.


*************************NSX Controller Upgrade*************************

Next part of the upgrade process is to upgrade the NSX controllers.

1: To update NSX controller cluster, login to vSphere Web Client and go to Networking&Security > Installation and Click on Upgrade Available button.


2: Click on “YES” to proceed with Controllers upgrade.


3: Wait for upgrade to complete. In my case it took more than 20 minutes to complete the upgrade process.


4: The controller nodes will be rebooted during the upgrade process. Patience is key here.


5: Post upgrade process is complete, you can verify the version.


****************************Upgrade Compute Cluster***********************

Next task in upgrade process is to upgrade NSX VIB’s on host cluster. During the upgrade cluster task, Esxi hosts requires reboot. reboot of host will not have any effect on the data planes as DRS is there to move VM around when host reboot will take place.

Important Note before proceeding with host cluster update: If you running with only 2 or 3 hosts cluster (probably in lab environment) and if you have created DRS anti-affinity rules to separate the NSX controller nodes, it will prevent DRS from evacuating VM’s off this host and thus halt host reboot process (overall upgrade will be halted).

Please disable anti-affinity rule for control cluster nodes or DLR/Edge VM’s (if you have created any).

1: To start upgrading the compute cluster, navigate to Networking & Security > Installation > Host Preparation tab and click on Upgrade available.


2: Click on ‘Yes’ to start with compute cluster upgrade.


3: On expanding the cluster you can see upgrade is in progress. In vSphere Client uninstalls and install tasks will be visible , basically process uninstalls old VIB’s and install new VIB’s.

Because old VIB’s are uninstalled, Esxi hosts needs a reboot to complete the uninstall of old VIB’s.


4: If you see Installation Status as ‘Not Ready’, click on red error and hit resolve all.


5: In vSphere Client you will see that DRS has starting migrating the VM’s, putting host in MM and reboot the host.


6: Post reboot of Esxi hosts you can see the host update is completed


***********************Upgrade DLR and ESG’s****************************

1: To update DLR/ESG’s version, navigate to Networking & Security > NSX Edges and select the DLR or ESG and right click on it and chose Upgrade version.


2: Click on ‘Yes’ to start upgrading edge version.

You get a warning that there will be service disruption. Please refer NSX documents carefully to understand the impact of the upgrade.


3: During the upgrade process new ESG VM is deployed alongside the existing one, when the new ESG is ready, old ESG vnic are disconnected and new ESG vnic’s connected.

You can see all these tasks in vCenter Inventory.


4:  Post upgrade verify that DLR/ESG status is deployed and at correct version.


I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂


Posted in NSX | 1 Comment

vExpert 2017 Applications are Now Open

Good news for fellow vExperts. Application for vExpert 2017 program is now open. Deadline to apply for the 2017 vExpert is 16th December 2016. The vExpert 2017 announcement will be on February 8th 2017. If you are a current vExpert then  you need to apply through the fastrack application only.

For those who are new to VMware technology and don’t know much about vExpert program, please read below about this

The VMware vExpert program is VMware’s global evangelism and advocacy program. The program is designed to put VMware’s marketing resources towards your advocacy efforts. Promotion of your articles, exposure at our global events, co-op advertising, traffic analysis, and early access to beta programs and VMware’s roadmap.

Basically its an Award or Honor given by VMware for your contribution towards VMware community. VMware believes in idea that don’t learn alone and help people in learning and growing with you.

Contributions towards VMware communities includes blogging about vSphere products, Publishing white paper’s. books etc.Then people who are part of VMUG group (group created for collaboration of people working on same platform) who hosts events, webinars etc to share valuable information with people.

Advantage of being a vExpert?

You may ask this question that what’s the fuss about vExpert and is there any advantage which vExpert gets?

The answer is yes. Being a vExpert brings certain advantages like access to private betas, free licenses, early access briefings, exclusive events, free access to VMworld conference materials online, exclusive vExpert parties at VMworld and other opportunities to interact with VMware product teams. They also get access to a private community and networking opportunities.

In which path you have to file your application?

Well if you are a new guy who is applying for the vExpert application for the first time, and you start filling up the application form and got stuck on wonder which path you have to file your application, then please read below to more on different paths available to achieve vExpert.

Evangelist Path
The Evangelist Path includes book authors, bloggers, tool builders, public speakers, VMTN contributors, and other IT professionals who share their knowledge and passion with others with the leverage of a personal public platform to reach many people. Employees of VMware can also apply via the Evangelist path. A VMware employee reference is recommended if your activities weren’t all in public or were in a language other than English.

Customer Path
The Customer Path is for leaders from VMware customer organizations. They have been internal champions in their organizations, or worked with VMware to build success stories, act as customer references, given public interviews, spoken at conferences, or were VMUG leaders. A VMware employee reference is recommended if your activities weren’t all in public.

Partner Network Path
The VPN Path is for employees of our partner companies who lead with passion and by example, who are committed to continuous learning through accreditation and certifications and to making their technical knowledge and expertise available to many. This can take shape of event participation, video, IP generation, as well as public speaking engagements. A VMware employee reference is required for VPN Path candidates.

How to apply for vExpert 2017?

Current vExperts (all vExperts including VCDX) apply thought the fastrack application at Here

New applicants please use the full application form at Here

Hurry up guys and file your application 😉

Posted in Vmware | Leave a comment

Learning vSphere 6.5-Part-5-Installing vCenter Server on Windows

Till now in this series we have discussed a lot of theory about vCenter Server/VCSA, deployment type etc and also we looked into system requirements for vCenter Server and PSC installation. Now its lab time where we will see the installation/configuration of the components.

If you have missed earlier post of this series, you can read them from below links:

1: Installing and Configuring Esxi

2: VCSA Overview

3: vCenter Server and PSC Deployment Types

4: System Requirements for Installing vCenter Server

In my lab I am going to use an external SQL server database. My database version is SQL 2014.

1: Preparing vCenter Server Databases for Install

You can use following script to install the database and configure login,permission etc. This script will also configure the database schema for you


/ *** Create a SQL Server Database and User for vCenter Server *** /

use [master]
(NAME = 'vcenter65',
FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\vcenter65.mdf',
SIZE = 10MB,
(NAME = 'vcdb_log',
FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\Log\vcenter65.ldf',
SIZE = 1000KB,
COLLATE SQL_Latin1_General_CP1_CI_AS
use VCDB
CREATE USER [vpxuser] for LOGIN [vpxuser]
use MSDB
CREATE USER [vpxuser] for LOGIN [vpxuser]

/ *** Create Database Roles and the VMW Schema and set Database Permissions ***/


if not exists (SELECT name FROM sysusers WHERE issqlrole=1 AND name = 'VC_ADMIN_ROLE')


if not exists (SELECT name FROM sysusers WHERE issqlrole=1 AND name = 'VC_USER_ROLE')
sp_addrolemember VC_USER_ROLE , [vpxuser]
sp_addrolemember VC_ADMIN_ROLE , [vpxuser]
use MSDB
if not exists (SELECT name FROM sysusers WHERE issqlrole=1 AND name = 'VC_ADMIN_ROLE')
GRANT SELECT on msdb.dbo.syscategories to VC_ADMIN_ROLE
GRANT SELECT on msdb.dbo.sysjobsteps to VC_ADMIN_ROLE
GRANT SELECT ON msdb.dbo.sysjobs to VC_ADMIN_ROLE
GRANT SELECT ON msdb.dbo.sysjobs_view to VC_ADMIN_ROLE
GRANT EXECUTE ON msdb.dbo.sp_delete_job TO VC_ADMIN_ROLE
GRANT EXECUTE ON msdb.dbo.sp_add_jobstep TO VC_ADMIN_ROLE
GRANT EXECUTE ON msdb.dbo.sp_update_job TO VC_ADMIN_ROLE
GRANT EXECUTE ON msdb.dbo.sp_add_jobserver TO VC_ADMIN_ROLE
GRANT EXECUTE ON msdb.dbo.sp_add_jobschedule TO VC_ADMIN_ROLE
GRANT EXECUTE ON msdb.dbo.sp_add_category TO VC_ADMIN_ROLE
sp_addrolemember VC_ADMIN_ROLE , [vpxuser]
use master
grant VIEW SERVER STATE to [vpxuser]

/ *** Create a vCenter Server User by Using the dbo Schema and db_owner Database Role *** /

use VCDB
sp_addrolemember @rolename = 'db_owner', @membername = 'vpxuser'
use MSDB
sp_addrolemember @rolename = 'db_owner', @membername = 'vpxuser'

2: Create 64 bit DSN for ODBC Connection

Instructions for doing so can be read from here

Install vCenter Server

1) Mount the vSphere 6.0  installation media. Execute the “autorun.exe

The installation wizard appears. In the left pane select “vCenter Server for Windows”  and click on Install.


2: Press Next to continue


3: Accept the licence agreement and hit“Next”


4) Under Select Deployment type choose “Embedded Platform Services Controller” and press “Next”


5) Verify the “System Name” is a valid FQDN and press “Next”.

Note that system name can't be changed post installation.


6) On “vCenter Single Sign-ON Configuration” page enter vCenter Single Sign-On domain (default is vsphere.local), “Password” for administrator user  and “Site Name”and press next.


7) In the “vCenter Service Account” window, select “specify a user service account” and supply the username in format Domain\username and the password for service account and press “Next”.


Note: If you are planning to run your vCenter Server using a dedicated service account then make sure it has been granted “Log on as a service” rights and added to local Administrator group of same machine where you are installing vcenter Server.

8) On “Database Settings” page , select “Use an external database” and enter the DSN name, DB username and password and hit Next to continue.


9) On  “Configure Ports” page, verify the default ports and press “Next”.


10) In the “Destination Folder” window, change the destination folder if necessary and select “Next”.


11) On ready to Install page review your settings and if everything is OK hit Install button.


12) Let the installer run and install the components



13) Once the installer has finished installation, Click on Launch vSphere Web-Client and then click on Finish button to complete installation.


14: Clicking on launch vSphere Web Client will bring the login page of vCenter Web-Client. Login with administrator@vsphere.local user


Verify that login to Web-Client is successful.


In next post of this series we will look into basic configuration of vcenter server.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 5 Comments

Virtual Machine Reset By HA-VMware Tools Heartbeat failure

Today while investigating one production issue, I came across an incident where a virtual machine restart was performed by HA.

Navigating through task and events for the VM in vCenter, I was seeing message similar to

This virtual machine reset by HA. Reason: VMware Tools heartbeat failure. A screenshot is saved at /vmfs/volumes/4c4850e4-0dcce710-28d9-00215-a5d36b8/XYZ/xyz-0.png

On investigating further, I came to find out that VMware tools running on VM was out of date.


I checked for HA settings related to VM monitoring on the cluster  and did not see any aggressive value set there which might have caused the vm restart. The VM monitoring settings were set at 30 secs for failure interval which means that if for continuous 30 seconds VM is unable to send any heartbeat to HA, a restart of VM will be performed.

Keep in mind that this is not related to host failure where all VM running on failed host is restarted on remaining nodes of the cluster. In case of single VM failure, HA will restart the VM on same host.


I started my investigation with checking hostd.log file of the Esxi host where VM was running and observed following messages which hints on VM restart

2016-11-06T06:00:52.281Z info hostd[50A40B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/521a8820-77938204-1461-002590c50caa/XYZ (4c4850e4-0dcce710-28d9-00215-a5d36b8)/XYZ (4c4850e4-0dcce710-28d9-00215-a5d36b8).vmx] State Transition (VM_STATE_ON -&gt; VM_STATE_CREATE_SCREENSHOT)

2016-11-06T06:00:52.353Z info hostd[5114CB70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/521a8820-77938204-1461-002590c50caa/XYZ (4c4850e4-0dcce710-28d9-00215-a5d36b8)/XYZ (4c4850e4-0dcce710-28d9-00215-a5d36b8).vmx] State Transition (VM_STATE_CREATE_SCREENSHOT -&gt; VM_STATE_ON)

2016-11-05T00:26:30.020Z info hostd[50F44B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/521a8820-77938204-1461-002590c50caa/XYZ (4c4850e4-0dcce710-28d9-00215-a5d36b8)/XYZ (4c4850e4-0dcce710-28d9-00215-a5d36b8).vmx] Send config update invoked

2016-11-05T00:26:30.049Z info hostd[502C6B70] [Originator@6876 sub=Snmpsvc] VmConfigListener: vm state change received, queueing reload request

2016-11-05T01:58:44.821Z info hostd[50507B70] [Originator@6876 sub=Snmpsvc] VmConfigListener: vm state change received, queueing reload request

2016-11-05T01:58:49.097Z info hostd[50244B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/521a8820-77938204-1461-002590c50caa/XYZ (4c4850e4-0dcce710-28d9-00215-a5d36b8)/XYZ (4c4850e4-0dcce710-28d9-00215-a5d36b8).vmx] Upgrade is required for virtual machine, version: 10

Next was to check vmware.log for VM XYZ. I observed following log entries for VM reset

2016-11-01T18:13:53.321Z| vmx| I120: Vix: [43327 vmxCommands.c:685]: VMAutomation_Reset. Trying hard reset
2016-11-01T18:13:53.321Z| vmx| W110:
2016-11-01T18:13:53.321Z| vmx| W110+
2016-11-01T18:13:53.321Z| vmx| W110+ VMXRequestReset
2016-11-01T18:13:53.321Z| vmx| I120: Vigor_Reset: Attaching to reset.
2016-11-01T18:13:53.321Z| vmx| I120: Stopping VCPU threads...

On digging vmware.log more, I found the messages regarding VMware tools version out of date

2016-10-27T04:25:34.389Z| vmx| I120: VMXVmdb_SetToolsVersionStatus: status value set to 'oldTools', 'supportedOld', install possible

2016-10-27T04:25:37.656Z| vcpu-3| I120: ToolsSetVersionWork did nothing; new tools version (9344) matches old Tools version

2016-10-27T04:25:37.832Z| Worker#1| I120: ToolsVersion: Status is supported old because this is status of monolithic version, and vsock_win2k3 and 25 more components are missing from the guest.

2016-10-27T04:25:37.832Z| Worker#1| I120: ToolsVersionGetStatusWorkerThread: Tools status 2 derived from environment

2016-10-27T04:25:37.832Z| vmx| I120: ToolsUpdateManifestInfoWorkerThreadDone: Compared tools manifest from host and from the guest. Status = 2.

2016-10-27T04:25:37.832Z| vmx| I120: ToolsUpdateManifestInfoWorkerThreadDone: Updating the manifest info.

2016-10-27T04:25:37.832Z| vmx| I120: VMXVmdb_SetToolsVersionStatus: status value set to 'oldTools', 'supportedOld', install possible

2016-10-27T04:25:37.832Z| vmx| I120: TOOLS installed legacy version 9344, available legacy version 9541

2016-10-27T04:25:37.832Z| vmx| I120: TOOLS manifest update status is 2

2016-10-27T04:25:37.832Z| vmx| I120: TOOLS can be autoupgraded.

2016-10-27T04:25:37.832Z| vmx| I120: TOOLS VM tools upgrade policy 1

2016-10-27T04:25:37.832Z| vmx| I120: TOOLS need not be autoupgraded according to upgrade policy.

2016-10-27T04:25:37.832Z| vmx| I120: TOOLS Resumed after power-on autoupgrade check.

2016-10-27T04:25:37.832Z| vmx| I120: TOOLS Setting autoupgrade-checked TRUE.

Also found VMware tools heartbeat timing out which eventually forced HA to reset the VM

2016-10-27T04:33:10.583Z| vcpu-0| I120: Tools: Tools heartbeat timeout.

2016-10-27T04:34:26.229Z| vcpu-3| I120: GuestMsg: Channel 0, Protocol error, state: 0

2016-10-27T04:34:26.230Z| vcpu-3| I120: GuestMsg: Cannot close channel 0: it is not opened

2016-10-27T04:34:28.597Z| vcpu-0| I120: Guest: toolbox: Version: build-1280544

2016-10-27T04:34:28.597Z| vcpu-0| W110: GuestRpc: application toolbox, changing channel 65535 -&gt; 0

2016-10-27T04:34:28.597Z| vcpu-0| I120: GuestRpc: Channel 0, guest application toolbox.

2016-10-27T04:34:29.924Z| vcpu-1| I120: TOOLS autoupgrade protocol version 2

From the above logs it was evident that vmware tools were unable to send heartbeat to HA and which caused HA to perform a VM reset.

VMware KB-1027734 was very helpful to me in carrying out my investigation.

Conclusion: From the above I can conclude that Vmware tools update is needed on the VM so that HA should not try to reset VM again.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 1 Comment

Learning vSphere 6.5-Part-4-System Requirements for Installing vCenter Server

In last post of this series we discussed about various deployment models available for deploying vCenter Server and PSC. In this post we will look into minimum system requirements  that must be met for installing windows based vcenter Server.

If you have missed earlier post of this series, you can read them from below links:

1: Installing and Configuring Esxi

2: VCSA Overview

3: vCenter Server and PSC Deployment Types

vCenter Server for Windows Requirements

To install vCenter Server on a windows, following hardware and software requirements must be met:

1: NTP ready environment : Clocks must be synchronized on the virtual machines where vCenter Server and PSC will be deployed. I previously wrote a blog on configuring Domain controller as NTP and you can read the article from here

2: Environment should be DNS Ready: Make sure you can successfully resolve hostnames of the servers. Also ensure Name to IP and IP to name resolution is happening fine for all the servers.

3: Dedicated Service Account for vCenter: If your vCenter Server service is running in a user account other than the Local System account, verify that the user account in which the vCenter Server service is running has the following permissions:

  • Member of the Administrators group
  • Log on as a service

4: Verify that the system on which you are installing vCenter Server is not an Active Directory domain controller.

5: Static IP for vCenter: Make sure you are using static IP for your vcenter Server.

Important Note:

If the system that you use for your vCenter Server installation belongs to a workgroup rather than a domain, not all functionality is available to vCenter Server. If assigned to a workgroup, the vCenter Server system is not able to discover all domains and systems available on the network when using some features.

6: Your host machine must be connected to a domain if you want to add Active Directory identity sources after the installation.

7: Verify that the LOCAL SERVICE account has read permission on the folder in which vCenter Server is installed and on the HKLM registry.

8: Verify that the connection between the virtual machine or physical server and the domain controller is working.

Pre-Install Checks for vCenter Server and PSC on Windows

When vCenter Server and Platform Services Controller are installed on Windows, the installer does a pre-check, for example, to verify that enough space is available on the virtual machine or physical server where you are installing or upgrading vCenter Server, and verifies that the external database, if any, can be successfully accessed etc.

The pre-install checker typically performs following  checks:

  • Windows version
  • Minimum processor requirements
  • Minimum memory requirements
  • Minimum disk space requirements
  • Permissions on the selected install and data directory
  • Internal and external port availability
  • External database version
  • External database connectivity
  • Administrator privileges on the Windows machine
  • Any credentials that you enter

Hardware Requirements for vCenter Server and PSC on Windows

Below list summarizes the hardware requirements for vCenter and PSC


Storage Requirements for vCenter Server and PSC on Windows


Software Requirements for vCenter Server and PSC on Windows

1: Verify that your operating system supports vCenter Server. For a list of supported OS please see VMware KB-2091273.

2: vCenter Server requires a 64-bit operating system, and the 64-bit system DSN is required for vCenter Server to connect to the external database.

Database Requirements for vCenter Server on Windows

For environments with up to 20 hosts and 200 virtual machines, you can use the bundled PostgreSQL database. A larger installation requires a supported external database for the size of the environment.

Following external databases are supported:

  • SQL 2008 R2, SQL 2012 and SQL 2014
  • Oracle 11g and 12c

In next post of this series we will look into vCenter Server Installation on Windows.

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in Vmware | 7 Comments

Learning vSphere 6.5-Part-3-vCenter Server and PSC Deployment Types

In last post of this series we had an overview of VCSA and touched a little bit on what are the new features that are included in VCSA v6.5.

In this post we will discuss about available deployment models for vcenter Server and VCSA.

If you have missed earlier post of this series, you can read them from here:

1: Installing and Configuring Esxi

2: VCSA Overview

Windows based vCenter Server or VCSA can be deployed either with an embedded PSC or an external PSC. You can also deploy a PSC as an appliance or install it on Windows. Depending on your infrastructure needs, it can be mix and match of both.

Before proceeding with installation of vCenter Server Appliance or vCenter Server for Windows, you must determine the deployment model that is suitable for your environment. The deployment types can be categorised in mainly 3 categories:


vCenter Server with an Embedded Platform Services Controller

This is a standalone deployment type that has its own vCenter Single Sign-On domain with a single site. vCenter Server with an embedded PSC is suitable for small environments. Other vCenter Server or PSC instances could not be joined to this vCenter Single Sign-On domain.

vCenter Server with an embedded PSC offers following advantages:

  1. The connection between vCenter Server and the Platform Services Controller is not over the network, and vCenter Server is not prone to outages caused by connectivity and name resolution issues between vCenter Server and the Platform Services Controller.
  2. If you install vCenter Server on Windows virtual machines or physical servers, you need fewer Windows licenses.
  3. You manage fewer virtual machines or physical servers.

Installing vCenter Server with an embedded Platform Services Controller is a recommended solution for small-scale environments.

You can configure the vCenter Server Appliance with an embedded PSC in vCenter High Availability configuration.

Note: A vcenter Server deployed with an embedded PSC can be reconfigured later to change the deployment type to vCenter Server with an external Platform Services Controller. I will cover this part later.

vCenter Server with an External Platform Services Controller

When you deploy or install a Platform Services Controller instance, you can create a vCenter Single Sign-On domain or join an existing vCenter Single Sign-On domain. Joined PSC instances replicate their infrastructure data, such as authentication and licensing information, and can span multiple vCenter Single Sign-On sites.

Multiple vCenter Servers can be registered to one external PSC instance. The vCenter Server instances assume the vCenter Single Sign-On site of the Platform Services Controller instance with which they are registered. All vCenter Server instances that are registered with one common or different joined Platform Services Controller instances which are connected in Enhanced Linked Mode.

Below image shows 2 vCenter Server registered to a common External PSC


Graphic Thanks to

Mixed Operating Systems Environment

In a mixed OS environment you can have:

  • A vCenter Server instance installed on Windows can be registered with either a PSC installed on Windows or a PSC appliance.
  • A vCenter Server Appliance can be registered with either a PSC installed on Windows or a PSC appliance.
  • Both vCenter Server and the vCenter Server Appliance can be registered with the same Platform Services Controller.

Below image shows mixed OS environment With an External PSC on Windows


Example of a mixed OS environment with an external PSC Appliance


Thats it for this post. Now we have an idea about available deployment models for vCenter Server/PSC deployment, we will try to explore them in coming posts of this series.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable :)

Posted in Vmware | 8 Comments

Learning vSphere 6.5-Part-2-VCSA Overview

In first post of this series Installing and Configuring Esxi we looked into Installation of Esxi 6.5 and as I stated in that post installation method has not changed at all. In this post we will look into overview of VCSA and will discuss about VCSA components.

Staring with vSphere 6.0, vCenter Server Appliance (vCSA) offers same scalability as provided by windows based vCenter Server installation. Earlier there was a limitation with vCSA that it cannot support huge infrastructure where you have a lot of Esxi hosts and virtual machines running. Same holds true for v6.5 as well.

Earlier in v6.0 there was a limitation that if you are planning to use VUM in your infrastructure, you have to install it on a windows server.

Lot of customers had provided the feedback to VMware to include VUM with linux based VCSA and VMware listened to them and now in v6.5 VUM can be used alongside VCSA on same node. This I guess was a bit of challenge for VMware and Engineers have done a commendable job to make this happen. We all should be highly thankful to the team for making this a reality.

The vCenter Server Appliance is a preconfigured Linux-based virtual machine that is optimized for running vCenter Server and the associated services. VCSA is available in ovf format and thus it reduces the deployment time of vCenter Server and also provides a low-cost alternative to the Windows-based vCenter Server installation.

The vCenter Server Appliance contains the following software packaged inside:

  • Project Photon OS® 1.0
  • The Platform Services Controller group of infrastructure services
  • The vCenter Server group of services
  • PostgreSQL

vCenter Server Components and Services

vCenter Server provides a centralized platform for managing, operating and resource provisioning for virtual machines and Esxi hosts.

vCenter server can be either installed with embedded Platform Service Controller or can use external PSC.

The following components are included in the vCenter Server and vCenter Server Appliance installations:

Platform Services Controller:  It contains vCenter Single Sign-On, License service, Lookup Service, and VMware Certificate Authority.

vCenter Server: contains vCenter Server, vSphere Web Client, vSphere Auto Deploy, and vSphere ESXi Dump Collector. vCenter Server for Windows also contains the VMware vSphere Syslog Collector. The vCenter Server Appliance also contains the VMware vSphere Update Manager Extension service.

A note on VCSA 6.5

VCSA 6.5 is deployed with hardware version 10, which supports 64 virtual CPUs per virtual machine in ESXi. It uses the embedded PostgreSQL database that has the scalability of up to 2,000 hosts and 35,000 virtual machines.

In v6.5, the VCSA and Platform Services Controller appliance support file-based backup and restore.

VCSA High Availability

Starting with vSphere 6.5, the vCenter Server Appliance supports high availability.This solution consists of an Active, Passive, and Witness nodes. The Passive and Witness nodes are clones from the original (Active) vCenter Server.

There are two supported workflows: Basic & Advanced.

The Basic workflow will clone the nodes, and create rules for placement. DRS and Storage DRS are not requirements but provide automated placement when available. The Basic workflow is flexible, allowing the choice of node placement within a cluster or across clusters within a vCenter Server.

Placing nodes in different clusters, datacenters, or vCenter Servers requires the Advanced workflow. This is a manual process, including cloning of the nodes and placement.

Thats it for this post. In next few posts of this series, we will dive more into vCenter Server and learn about the deployment types and also will see how to install Windows based vcenter  server and VCSA.

Disclaimer: Info about VCSA High availability has been taken from Emad Younis blog VCSA 6.5 What’s New

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂


Posted in Vmware | 9 Comments

Learning vSphere 6.5-Part-1-Installing and Configuring Esxi

We all know that vSphere 6.5 was announced by VMware 2 weeks back in VMworld Barcelona. With the release of 6.5 version of vSphere, VMware has done some fantastic work with HA and DRS enhancements. Several folks wrote blogs on the new features and enhancements that are included in this new release.

If you have not read about the new features/enhancements in vSphere 6.5, you can read it from here. Brian has done fantastic job by summing up all that you want to know about vSphere 6.5.

After reading a bit about vSphere 6.5, I decided to deploy and test it in my lab.

Note: The GA and beta version has not been released yet by VMware.

Let’s begin with part 1 of this series where we will see installation/configuration of Esxi host.

vSphere is a sophisticated product with multiple components to install and setup. Typically installing vSphere includes the following tasks:



We are starting with deploying Esxi host. The installation process has not changed and its pretty simple as always.

1: Boot your server from the Esxi iso file and select the installaer option from the menu.


2: Let the components load


3: Press Enter to continue installation


4: Press F11 to Accept EULA and continue with installation.


5: Installer will scan for available devices where Esxi can be installed. be patient to let installer discover all devices.


6: Once the devices have been discovered, select the appropriate one where you want to install and hit Enter to continue.


7: Select the keyboard layout and hit Enter to continue.


8: Enter root password and hit enter to continue.


9: Press F11 to begin with installation.


10: Installation is not gonna take more than 5-7 minutes, so sit back and relax. In the meanwhile installation is in progress, you can get a cup of coffee for you or can listen to your favourite music or do anything that pleases you 😉


11: Reboot the server to complete the installation


12: Let the server go through the boot process and load all components.


13: Once Esxi is loaded completely, hit F2 to start configuring it.


14: The very first thing we need is to configure an IP address on which this esxi host can be accessed. From the system customization menu select “Configure Management Network” and hit enter.


15: On the Configure Management Network menu, select IPv4 Configuration and hit Enter


16: Select the 3rd option in the menu and set IP/Netmask and gateway address for this esxi host. Hit Enter to save the IPv4 configuration.


17: If you are not using any Ipv6, you can choose to disable it. Note that it will require a host reboot for change to take effect. Select Disable IPv6 and hit enter to disable it.


18: From the Configure Management Network menu, select DNS Configuration and hit Enter. Enter the DNS server IP and set a hostname for the esxi host.


19: Provide the DNS suffixes for name resolution and hit Enter.


20: From the Configure Management Network menu, press ESC to exit the menu. It will prompt to a host reboot (since we have chose to disable IPv6). Also it will apply all the network configuration which we have done in earlier steps.


21: Post reboot of host, you can chose to test the management network to verify host is good to go.


Thats all for this post. In next post of this series we will look into how to install vcenter 6.5. Stay tuned !!!!

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable :)

Posted in Vmware | 16 Comments

vCloud Air Data Protection:Part-3: Restore a Deleted VM From Backup

In last post of this series Virtual Machine Backup we had a look into how to configure Data Protection Service on a Virtual Datacenter and how to take backup of a virtual machine.

In this post we will learn how to recover a VM from the backup.

To learn more about DPS I would recommend reading first 2 posts of this series from below links

1: Introduction to vCloud Air Data Protection

2: Taking Virtual Machine Backup using DPS

Let’s jump into lab now and see how restore works within DPS.

1: Login to vCloud Air and Navigate to Data Protection tab and select any VM on which you have previously taken backup.

The DPS dashboard will show how many restore points are available for this VM and how much disk space it is consuming on data protection storage.


2: Clicking on the restore point will bring up a new window which will list the details of restore points available.

Selecting anyone of the restore points will highlights additional options for recovery i.e Either Restore as New vApp (Out of place restore) or Restore vApp In-place (In-place Restore). We have discussed about these 2 modes in the Introduction post of this series.


3: Let’s delete our original virtual machine now by navigating to Virtual machines tab and selecting delete option by clicking on the arrow button available on extreme right corner.


4: Wait for the portal confirmation about delete VM job.


5: To restore the deleted VM, navigate to Deleted vApps option within Data Protection page.

You will see the list of all those VM’s which have been deleted from your VDC. Since this is my lab environment so I have purposefully deleted only one VM for purpose of demonstrations.


6: To restore the VM, click on the available restore point and it will bring a new window with list of the restore points.

Notice that In-place restore option is not available anymore on the restore points because we have deleted the original VM. This option is available when Original VM is intact and you want to restore the backup in the same VM instead of creating a new one.

Click on Restore as New vApp to start the restoration process.


7: You will be asked for in which VDC you want to restore the VM (in case you have more than one VDC protected by DPS). Also you have to provide a name for the restored vApp.

For restore DPS will ask you whether you want to restore entire vApp of individual virtual machines from the vApp. In my lab I have only one VM in my vApp so I selected Entire vApp option.

Hit restore button to start Restore job.


8: Portal will display a message regarding restore job has been kicked off.


9: Once the restore job is completed, vCloud Air portal is kind enough to let you know about that.


10: If you now navigate to Virtual machines tab again, you can see the restored VM listed there. This VM will be in powered-off state.

Also note that on the restored vApp, DPS will be disabled because DPS do not know this is the same vApp which was getting backed up before and was deleted.

You can power-on the VM from here and verify things.


I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable :)

Posted in vCloud Air | Leave a comment

vCloud Air Data Protection:Part-2: Virtual Machine Backup

In last post of this series Introduction to DPS we had a look on what DPS is and how it works. We also discussed the benefits which a customer can get via DPS for his critical workloads running in vCloud Air.

In this post we will look into how to configure Org vDC’s for backups, edit backup policies and how to take backup of a VM.

Prerequisites for DPS:

1: You should have a Dedicated/Virtual Private Cloud Instance running in vCloud Air

2: You should have purchased DPS subscription.

Lets jump into lab now.

1: Login to vCloud Air portal and navigate to Data protection tab. This tab gives you information about how many vDC’s are being currently protected via DPS, how many vApps are configured to be backed up, how much storage backups are consuming and number of VM’s which have been deleted (intentionally/accidentally)


2: Select the VDC on which you want to enable DPS and from Actions tab select Enable Data Protection


A new window will pop-up. Select Enable Data Protection.


3: Post enabling DPS, you will get a confirmation message that DPS have been successfully enabled on the selected VDC.


4: Next is to define the backup policy for your workloads. From Actions tab, choose Edit Policy to define the backup policy for your VDC.


5: On the Edit Policy window, choose the retention period for backups, define the backup window (this is the time when backup is triggered) and optionally define an email address for any notifications regarding backup (failed or successful backup). Hit save to save the policy.


6: Once you have defined the backup policy, its time to take backup.

Note that when you are not using self-service option in DPS, all vApps that belongs to a VDC are automatically configured to be backed up and the backup policy is enforced on all vApps.

I am demonstrating adhoc backup in this post.

From the VDC, select the vApp on which you want to run adhoc backup and click on small arrow button in corner and it will bring a drop-down menu. Choose Run backup now.


A new window will pop-up. Hit Run backup Now button


7: The portal will display a message that backup has been initiated on the vApp.


8: Once the backup is completed, portal will notify you of the completion status.

You can see now restore point reporting as 1. Also you can see amount of storage used.


Thats it. We have successfully executed backup on a VM and verified backup is completed.

isn’t is pretty easy?

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable :)

Posted in vCloud Air | 2 Comments

vCloud Air Data Protection:Part-1:Introduction

Everybody knows how important is to backup your critical workloads in your infrastructure. The need to protect workloads is the same whether it’s onsite or in the cloud.vCloud Air Data Protection services ensures business continuity for the workloads which are running in your vCloud Air environment.

So what is vCloud Air Data Protection exactly?  

VMware vCloud® Air Data Protection offers secure, policy-based backup and recovery in the cloud for virtual machines hosted exclusively on vCloud Air. It is an agent-less, self-service, policy-driven backup and recovery service with works on the principle of image-level backups.

As compared to traditional file-based backup and recovery solutions, vCloud Air Data Protection uses image-level backups to ensure all operating system, file system and application data encapsulated within a virtual machine are captured as a snapshot image before being committed to backup media.

The self-service backup features give customers a confidence to run their critical workloads in vCloud Air without worrying about data loss.

The below diagram illustrates high level overview of how DPS works in vCloud Air


Graphic Thanks to

Benefits of vCloud Air Data Protection?

Following are the benefits which an end user will get with DPS is vCloud Air:

  • Backup policy affinity controls per Virtual DataCenter (VDC) or per vApp
  • Daily (24-hour) Recovery Point Objective (RPO) guarantee
  • Virtual machine (image-level) Restore Granularity Objective (RGO)
  • Custom backup window scheduling
  • Configurable data retention
  • On-demand backups
  • Intelligent consumption tracking and activity reports

How vCloud Air DPS Works?

Data Protection uses snapshots to take the image level backup of Virtual Machines. When a backup is initiated, a snapshot of that VM is taken.  Post completion of the snapshot, it is then stored on the Data Protection storage (which is different than the storage which customers get when they purchase compute bits from vCloud Air) for the duration of the retention period defined in the policy.

The first backup is always a full backup and post that DPS leverages Change Block Tracking (CBT) and only backups the delta changes.Once the retention period is expired, the very first backup of the VM is deleted.

Below image summarizes the high level workflow of DPS


Graphic Thanks to

Data Protection Policies

Using the Data Protection Service, you can set data protection policies for your virtual data centers and vApps to meet your protection needs. You can apply data protection policies in the following ways:

1: To a virtual data center (Enabled mode): When you enable data protection for a virtual data center, all the virtual machines added to that virtual data center are protected automatically.

2: To a vApp directly (Self Service mode): When you switch to Self Service mode, the Data Protection Service does not back up all virtual machines in that data center automatically. You have the option to choose which virtual machines to back up and to apply individual policies to those virtual machines.

Note: To enable data protection for a virtual data center, you must be logged into vCloud Air as a virtual infrastructure administrator. If you are logged in as an end user, you can set backup policies only for the vApps you own.

Why Should Customer choose vCloud Air Data Protection Service?

DPS is different from the traditional backup and recovery techniques. Below are the 2 main features which makes DPS so popular:

1: Non-intrusive Protection for any Operating System and Application

By leveraging the Storage APIs at the vSphere layer in vCloud Air, DPS is able to use a combination of inline I/O quiescing and snapshot techniques at the virtualization layer to capture a crash-consistent image replica of any vApp and its virtual machine members.

These point-in-time replicas are made possible in Data Protection without having to install custom software agents, which often result in resource contention and potential performance degradation during an active backup window.

2: Multilayer Restore Workflows for Advanced Recovery

Data Protection helps streamline data recovery operations by offering dual self-service restore modes that can each address variable use cases:

  • In-place restore : Recovering from a backup replaces the protected virtual machine
  • Out-of-place restore: Recovered VM from backup is deployed as a separate VM.

Whether recovering a single virtual machine that has been accidentally deleted or an entire vApp with multiple virtual machine members assigned, each restore mode ensures that data is safely recovered with minimal disruption to ongoing operations.

Storage Considerations for DPS:

  • Backups on standard storage are limited by 200 IOPs per virtual machine disk (VMDK) and a 23 hour timeout. This allows ~300GB backups per VMDK for a backup if there is very minimal load on the disks. This should be considered when planning backups for vApps and individual virtual machines.
  • Recommended sizes for backups when using SSD accelerated storage are no greater than 2TB per VMDK with a total vApp size under 10 terabytes.
  • If the virtual machines are on standard storage, VMWare recommends that the VMDKs be no larger than 300 gigabytes each with a total vApp size under 2 terabytes.

For full list of DPS Best Practices please see VMware KB-2119537

How to Buy DPS?

A subscription to Data Protection can be purchased through MyVMware®. After the subscription is activated, a customer can begin using Data Protection through the vCloud Air console.

Thats it for this post. In next post of this series we will look into how to schedule a backup on a vApp/VM using DPS and how to restore it from backup.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable :)


Posted in vCloud Air | 3 Comments

Troubleshooting Edge Gateway High Availability

Yesterday I was working on Edge Services Gateway in my Lab and deployed the edge gateway in HA mode. Soon after the deployment when I checked the HA status from vCenter, it reported status as Down


To counter any UI bug which might be reporting HA status as down (as this was a brand new deployment), I decided to check the HA status by logging onto edge vm’s directly.

On checking for the HA status on the VM, below message was displayed

Highavalibity healthcheck server is stopped



I did a search on google for this message and didn’t get much results. Then I checked the Admin guide for NSX and came to know the fact that you should have at least one vNIC configured as High availability traffic flows on one of the internal interface.

By design the edge High Availability Service will only kick in once the first Internal vNIC has been added and configured. If you have enabled HA after doing the initial interface configurations you won’t have this issue as during the HA setup you are asked which vNIC to choose. If you enable HA without a vNIC configured the service won’t kick in until that vNIC is in play.

On checking my HA configuration I found that I have not configured any internal interface on my edge gateway.


I went ahead and finally configured vNIC1 interface


As soon as the internal interface was configured, High availability was established on the edge gateway.


On doing a check on edge VM’s, confirmed that one of em is active and other is standby



I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable :)

Posted in NSX | Leave a comment

Learning NSX-Part-9-Edge Services Gateway

In last 2 post of the series we discussed about Distributed Logical Router. Moving forward in NSX learning series, we will look into what is Edge Service Gateway and will discuss on when to use edge gateway. We will look into deploying ESG and configuring it and then finally some touch down points on monitoring Edge gateways.

If you have missed earlier posts of this series, you can read them from below links:

1: Introduction to VMware NSX

2: Installing and Configuring NSX Manager

3: Deploying NSX Controllers

4: Preparing Esxi Hosts and Cluster

5: Configure VXLAN on the ESXi Hosts

6: Logical Switching

7: Distributed Logical Router Tidbits

8: Installing Distributed Logical Router

In VMware NSX for vSphere there are two different types of NSX routers which can be deployed in virtual network infrastructure.

  1. The NSX Edge Services Router (ESR)
  2. The NSX Distributed Logical Router (DLR)

Both the ESR and DLR can run dynamic routing protocols, or not. Both of them support Static/default routes as well.

Although we can use Logical route to route between 2 or more logical networks, and physical networks, the recommended way is to use an Edge Services router. The ESR has a number of features such as Firewall, DHCP Services, NAT, load balancing and VPN services.

Difference between ESR and DLR?

ESR can be think of a router contained within a VM. Both the control plane and data plane resides in the VM, but ESR is more than just a router. It provides supports for L4-L7 services like FW, LB, NAT, VPN. ESR can establish routing protocol sessions with other routers and all of the traffic flows through this VM.

In DLR the data plane is distributed in kernel modules at each vSphere host, while the control plane exists in a VM (LRC).  The control plane VM in turn relies on the NSX controller cluster to push routing updates to the kernel modules.

When to use ESR?

In last post of the series we discussed that with evolution of virtualization/cloud computing, the amount of East-West traffic in datacenter have grown significantly. We also discussed about the hair-pinning of traffic and how DLR can be used to reduce the East-West traffic and get rid of hair-pinning issue.

As compared to DLR, ESR is unique in its own way. It’s more than a just router.  Using ESR we can leverage features such as firewall, load balancer, and VPN etc and because of this, it works well as the device handling the North-South traffic at the perimeter of your virtual network.

Since ESR is deployed as a VM, we can have virtually unlimited ESR’s running in parallel, each establishing the secure perimeter for their own.

Lets jump into the lab now and see the installation/configuration of ESG.

The edge gateway is deployed in the same way as a distributed router instance. To deploy an ESG, navigate to Networking & Security > NSX Edge. 

1: Click on the  (plus) button to open the deployment wizard.


2: Select Edge Services Gateway and enter the name of your Edge gateway . You can also configure a hostname.

If you want edge gateway to be highly available, check mark the “Enable High Availability” option. Hit Next to continue.


3: Provide the password for the admin user (atleast 12 characters long)

If you want to manage Edge Gateway over ssh then enable ssh access. Enabling auto rule generation, which will create necessary rules automatically when edge services such as VPN and load balancing are enabled:


4: Click Next and select the Appliance Size. There are a number of options for the appliance size. I have chosen ‘Compact’ in my lab environment. Click on (plus)  to select the Cluster/Resource Pool/Datastore/ Host etc where the edge will be deployed.

More on what should be an ideal choice for edge size, please read this article




5:  Click on (plus)  to Configure edge gateway’s interfaces.


6: Select the uplink interface and select the port-group to which this uplink will connect to. Assign an IP Address and subnet mask for the interface.

Note that it could be either a Public IP (in case of Uplink connected to a public network) either a private (in case of a Firewall inter-vlan or internal network)

Hit OK once you have entered all the information.


Click on next to continue the deploy wizard.

7:  For Default gateway settings, select the vNIC where your gateway will be connected to and enter the Gateway IP. Hit Next to continue.


8: On the next screen you can, optionally, select to configure the Firewall default policy:


9: On Ready to Complete page review your settings and hit finish to complete ESG deployment wizard.


10: In vCenter Recent task pane you can see edge vm deployment has been kicked off.


11: Under NSX Edges you can see your newly created edge gateway listed there.


12: Under VM and Templates view in vCenter, you can verify that 2 edge VM’s have been deployed (as we deployed edge in HA mode). You can select the edge VM to verify that VM is now up and running. Additionally you can check how much RAM/CPU has been assigned to the edge VM.


16: Double click on the edge gateway from Network & Security > NSX Edges view to open up the edge gateway configuration options.

Click on Manage > Settings > Configuration to add additional configuration items like defining a syslog server where edge gateway can forward the logs, DNS Settings on edge gateway etc.

Also verify the HA status of edge VM from this page.


17: By navigating to Monitor > Statistics tab, you can grab the Interface throughput stats and concurrent connection data for your edge gateway.


18: Finally if you click Actions button, it will bring a drop down menu which provides additional options available on edge gateway like Force Sync (it reboots the edge gateway backing VM’s), Edge re-deployment etc (this deletes the existing edge VM’s from vCenter a new VM is deployed).

You can also change the edge gateway admin user credentials by selecting Change CLI Credentials, also you change logging level etc from the dropdown menu.


I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable :)

Posted in NSX | 1 Comment