Bye Bye to free Blog. Say hello to my new website

Dear friends and Blog Readers,

I have started my new website http://www.vstellar.com.

All new posts will now be published to my new website. This blog will be running up for a few days until I am confirmed that migration of all posts/links/media items were smooth and nothing is broken in my new website.

If you love to read my blogs and are following this, then a humble request to all of you is to follow my new website so that you are notified whenever I post new contents.

If you come across anything broken in my new site then feel free to let me know about that so that I can fix it.

Thanks for all your support as always.

Yours Sincerely

Manish Jha aka Alex Hunt

 

Posted in Vmware | Leave a comment

Setup SSL Certificate For vSphere Lab-Part-3-Creating SSL Web Certificates Template for VMware

In this post we are going to cover the SSL Web Certificate creation for VMware. If you have missed earlier posts of this series I would recommend reading them first from below links:

1: Installing and Configuring CA Server

2: Creating Certificate Templates

Lets begin with creating SSL Web certificates for VMware.

1: Launch the Certificate Authority MMC and navigate to Certificate Templates folder. Right click the folder and select Manage.

ssl-38

2: From the displayed list of templates, select Web Server template and right click on it and select Duplicate Template.

ssl-39

3: Select the Server 2008 Enterprise option. You can also choose Server 2003 option if you are looking for backward compatibility. Hit OK.

ssl-40

4: Provide a new name for this template. Modify the validity period and renewal period if you want longer period of time for this option and don’t want to go with default time period. Hit Apply OK.

ssl-41

5: Navigate to Extensions tab and select the Key Usage Extension and click on Edit button.

ssl-42

6: Select the “Signature is proof of origin and “allow encryption of user data” check box. Hit OK.

ssl-43

7: Now under Extensions tab select “Application Policies” extension and hit edit button.

ssl-44

8: Add Client Authentication to the list of policies.Hit OK.

ssl-39-2

9: Now right click on Certificate Template folder and choose New> Certificate Template to Issue.

ssl-45

10: Select the Lab-SSL certificate template which we just created. Hit OK.

ssl-46

So now we have finished creating SSL Web Certificate template for VMware infrastructure. In next post of this series we will see how to request and add the SSL certificates to our infrastructure. Stay Tuned!!!

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in SSL Certficates | 6 Comments

Setup SSL Certificates For vSphere Lab-Part-2-Creating Certificate templates

In our last post Setup SSL Certificate Authority For vSphere Lab we saw how to add CA Server Role to a windows server 2008 machine. In this post we will see how to generate certificates.

1: Launch Certificate Authority console from Administrative Tools.

ssl-20

2: Right Click on Certificate Template and click Manage.

ssl-21

3: Select the Windows Authentication Template and right click on it and select Duplicate Template.

ssl-22

4: Select Windows server 2008 Enterprise and hit OK.

ssl-23

5: Give the new certificate template a name. Also we need to change some of the properties of the new template.

I have changed the validity period to 5 years and selected Publish certificate in AD and Do not automatically reenroll option.

ssl-24

6: Go to Security tab and  change the “Domain Computers” permissions to read and autoenroll the certificate.

ssl-25

7: Go to Extensions Tab and change the Application Policies to include both Client and Server Authentication.

Select Application Policies and click on Edit.

ssl-26

Click on Add button to see list of policy available

ssl-28

From the Add Application Policy list select “Server Authentication” and click OK.

ssl-29

Once Server Authentication policy is added hit OK.

ssl-30

8: Under Subject Name tab, add the UPN checkbox and hit Apply OK.

ssl-31

9: Now again go back to the Certificate Authority MMC.  Right click on the Certificate Template Folder and choose New–> Certificate Template to Issue.

ssl-32

10: Select the certificate template that we have just created and hit OK.

ssl-33

Creating Group Policy

Now to enable computers to automatically grab the certificates which we created and install them as trusted certificates we have to create a group policy.

If you remember during certificate Template creation we have selected  “Autoenroll”. That doesn’t do anything until we configure a GPO to tell the computers to look for these certs.

11: To create a new group policy, go to Run and type “gpedit.msc“. Navigate to Windows Settings > Security Policies > Public Key Policies and select Certificate Services Client-Auto Enrollment and right click to open properties.

ssl-34

12: Under Configuration Model select “Enabled” and select the options Renew expired certificates and update certificates that use certificate template. Click on Apply OK.

ssl-35

13:Now select “Certificate Services Client-Certificate Enrollment Policy” and right click to open properties. Under Configuration Model select Enabled and Checkmark the box in front of Active Directory Enrollment. Hit Apply OK.

ssl-36

Now we have created certificates and selected the appropriate policies. In our next post we will see how to generate signed certificates for use in our vSphere Infrastructure.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in SSL Certficates | 1 Comment

Setup SSL Certificates For vSphere Lab-Part-1-Configuring CA Server

This week I was looking for setting up CA Server for generating SSL certificates which can be used in my vSphere Home Lab. Using Self-Signed certificates usually work in a lab environment, but its good to know how to work with signed certificates as in production environment organizations don’t use self-signed certificates and rely on SSL certificates bought from 3rd party like Thawte or Verisign.

Having your own CA is useful for testing SSL and other services that require certificates without the need to purchase certificates from a third party.  However, these certificates will not be automatically trusted by computers external to your AD domain, so there are some limitations.

In this post I am going to share the steps needed to configure a Windows 2008 R2 Server as Certificate Authority.

Prerequisites

  • Active Directory Domain already setup and configured
  • Server 2008 installed and joined to domain

Lets begin with configuring Server 2008 as CA server.

1: Launch Server Manager and click on Add Roles. From the list of roles available select “Active Directory Certificate Service” and hit Next.

ssl-1

2: Hit Next on Introduction to AD CS page.

ssl-2

3: Under Role Services select “Certification Authority” and hit Next.

ssl-3

4: Select “Enterprise” as setup type for your CA server and hit Next.

For SSL deep dive I would recommend reading this Article by Derek Seamen.

ssl-4

5: Under Specify CA type select “Root CA” and hit Next.

ssl-5

6: This is a new CA without existing keys so select Create an new private key and hit Next.

ssl-6

7: Keep the default CSP, hashing method, and key length and hit Next.

ssl-7

8: Keep the default CA name and hit Next.

ssl-8

9: Keep the default validity period of 5 years and hit Next.

ssl-9

10: Dont change the default database location for certs unless you have specific requirements. Hit Next.

ssl-10

11: Click on Install button on Confirm Installation Selections page.

ssl-11

12: Wait for installation to finish.

ssl-12

Installing Certification Authority Web Enrollment service

The Web Enrollment service is very useful while making requests for certificates from computers that are not members of AD domain.

Once “Certificate Authority” role is installed completely, you can add Certification Authority Web Enrollment service to it from server manager page.

13: Click on Add Role Services.

ssl-13

14: Under Role Services select “Certification Authority Web Enrollment” and hit Next.`

ssl-14

15: Click on Add Required Role Services button to add the IIS services.

ssl-15

16: On IIS page hit Next.

ssl-16

17: Keep the default selection and hit Next. If you have specific requirements you can add additional options by selecting the appropriate components check boxes.

ssl-17

18: Hit Next to start installing the services and components.

ssl-18

19: Hit Close once the components are installed.

ssl-19

With this installation of CA Server role has finished. In our Next post we will see how to configure and use signed certificates.

Additional References:

1: Install an Enterprise Certificate Authority in Windows 2008 R2

2: Create a Windows Enterprise CA and issue certificates for vRA and other VMware Products with examples

3: Install Certification Authority in Windows Server 2008 R2

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Posted in SSL Certficates | 5 Comments

VMware Fling- View Pool Manager

Today I was going through the VMware Flings website and came across one cool fling which I think can be very handy for VM Admins working on VDI.

View Pool Manager is a Fling that allows VMware View administrators to easily manage users across desktop pools and security groups.

Maintaining desktop pools in large VDI environment is one of the biggest challenges for a VMware View administrators. Commonly each desktop pool has Active Directory security groups with the number of users a given desktop pool will support.

As employees leave/join the organization, security groups require maintenance to ensure the number of users in each group correctly match the amount of desktops available. VMware View Pool Manager allows VMware View administrators to easily manage users across desktop pools and security groups for large deployments.

This tool allow administrators to bind an Active Directory connection and specify:

  • How many users there should exist per group (based on VMware View pool size)
  • Floating or Persistent deployment
  • Source Security Groups (all users to be distributed to pools)
  • Destination Security Groups (All groups the users may be assigned to)

This tool help VMware View administrators to save hours of work adding, removing and maintaining user’s, desktop pools and security groups while ensuring each security group does not hold more users than the amount of desktops pool assigned to a given VMware View desktop pool.

view-pool-manager

System Requirements

The View Pool Manager requires .NET Framework 4.0. The .NET 3.5 SP1 installation might require Internet connectivity to download more files. Verify that you are a member of the Administrators group on the domain.

The View Pool Manager fling was build by Andre Leibovici. You can follow him on twitter @andreleibovici

You can download this tool from Here

Posted in Vmware, VMware Flings | Leave a comment

vSphere 5 Memory Management and Monitoring diagram

The VMware vSphere 5 Memory Management and Monitoring diagram provides VM Admins a comprehensive look into the ESXi memory management mechanisms and reclamation methods.

dia

Graphic Thanks to VMware.com

You can download the above diagram from Here

Posted in Vmware | Leave a comment

vCenter Server 6.0 Availability Guide

vCenter Server is instead the most critical part of any virtual infrastructure environment. High availability of vCenter server has become very crucial as there are many features which doesn’t work or partially work when your vCenter server is not available. When vCenter server is not designed for availability it can lead to a single point of failure.

There are many changes (related to vCenter Server and its components) done with the release of vCenter Server 6 and thus careful consideration has to be made in the design of its architecture.

Multiple solutions can be used for achieving high availability for vCenter Server. Many of these options can be combined to provide different levels of availability. vSphere HA, FT, vCenter Watchdog services and in guest clustering solutions can be combined depending on customer requirements for availability.

vCenter server 6.0 availability guide can be downloaded from VMware Techresources Library

Some of the topics covered in this book are:

  • Availability Considerations for vCenter Server 6.0.
  • vCenter Server High-Availability Solutions and Configurations.
  • High Availability for the Platform Services Controller.
  • Recovery Options.

So guys if you are looking for deploying vCenter Server 6.0 or already deployed in your environment, then this is a must see document.

Posted in vCenter Server, Vmware | Leave a comment

Whats New in vSphere 6.0- vNUMA Enhancements

Before going into this post about vNUMA lets recall what NUMA is how it works. NUMA can be explained as follows:

Non-Uniform Memory Access (NUMA) is a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.

In modern day physical server with two or more sockets (physical cpu) memory is distributed in a way that one slot (generally 8) of memory is local to one cpu and other slot of memory is local to the other cpu. A socket (cpu), its local memory and the bus connecting the two components is called a NUMA node. Both sockets are connected to the other sockets’ and thus allowing remote access of memory.

numa1

It is not mandatory that an additional socket in a system means it is NUMA supported. Two or more sockets can be connected to memory with no distinction between local and remote. This type of architecture where one or more sockets connected to the same RAM is called UMA (uniform memory access) system. Check your server specifications for whether or not your server supports NUMA.

For more information on NUMA I would recommend reading this wonderful blog by Frannk

vNUMA was introduced in vSphere version 5.0 to improve the performance of the CPU scheduling by exposing the underlying NUMA architecture of physical server to the VM. vNUMA is automatically enabled on a VM if it has more than 8 vCPU. You can also explicitly enable vNUMA on your VM even if you have less than 8 vCPU.

vNUMA is designed for modern OS’s that are NUMA aware and can make intelligent page management decisions based on locality. Prior to vSphere 6.0, vNUMA was not aware of how memory allocation works when a memory is hot added to a VM.

With the release of vSphere 6.0, there are also improvement in NUMA in terms of memory. Memory hot-add is now vNUMA aware. To explain this lets have a look on below example:

Note: This post was originally posted on http://plain-virt.blogspot.in/ by Wee Kiong Tan.

Let’s start with what happen in prior with vSphere 6 when a VM is hot-added with memory. We will take an example where a VM is configured with 3 GB of memory.

numa2

Graphics Thanks to http://plain-virt.blogspot.in/

As visible in above picture the VM memory is uniformly distributed on their physical counterpart i.e. the physical cpu present in the Esxi host because vNUMA is enabled on the VM.

Now suppose this VM have Hot-Add feature enabled and additional 3 GB of memory is hot added to this VM. Memory will now be allocated by placing to the first NUMA node follow by the next if enough memory is not available to schedule the threads on previous nodes.

numa3

Graphics Thanks to http://plain-virt.blogspot.in/

As you can see that after adding the additional memory, the memory allocation on physical component is no longer uniform.

In vSphere 6.0 VMware has addressed this issue and made memory Hot-Add more friendly to NUMA.

numa4

Graphics Thanks to http://plain-virt.blogspot.in/

You can see in the above picture now the memory distribution is even across the physical NUMA nodes.

Posted in Memory_management, Vmware | Leave a comment

VMware Project Fargo- A Technique for Rapid Cloning of Running VMS

Today I was going through a blog and found one very interesting blogpost which I am going to share with you.

At VMworld 2014 VMware announced a very interesting new project called Project Fargo. This is another name for the Project VMFork.

What is this Project Fargo ?

Fargo/VMFork is a platform currently in technology preview which enables a rapid cloning of running VMs.

It lets you quickly spin up linked clones from a VM that’s already running instead of using disk images. Rather than having a master image stored on a virtual hard disk , the base desktop is an actual running VM. When you need to give a user that same desktop, you can just clone that VM instead of booting up a whole other one. The aim of Fargo is to provides a fast, scalable differential clone of a running VM.

fargo1

The key benefits of using this method is that it is instantaneous and can be done from a running VM,  so a new VM spawned would typically take less than 1 second and is in the same running state from where it is cloned. Also since only changed blocks are written in new VM so this solution will take up dramatically less disk space.

While there are many potential use cases for Fargo, it was presented with virtual desktop in mind where providing an instant clone of running non-persistent desktop would avoid the boot storms to the storage subsystem. The forking process would be highly useful in scaling up a set of virtual machines quickly to meet increasing traffic on a website or in some other compute-intensive, high -demand scenario.

Kit Colbert, CTO of end user computing at VMware, said from the VMworld stage that Project Fargo will enable production of virtual desktops 30 times faster than current rates.

Here are the excerpts from the session presented by Kim:

While we have solved the real-time application delivery problem, we still need to address the issue of desktop delivery.  The reality is that today the delivery of virtual desktops is time consuming.  From cloning to powering on and OS boot to customization, the end-to-end process can take many minutes before the virtual desktop is ready to accept user logins.  This problem compounds itself when trying to deliver hundreds or thousands of desktops at one time.  We need a better solution.

The cloned VM is identical in every way to the original and initially shares all memory and disk with it as well.  Project Fargo is very cool for two reasons: first, it gets you a new running VM in under a second.  Second, it’s a very lightweight VM because it shares all memory and disk with the original.  (To be clear, both the memory and disk are “copy on write” so if new VMs modify bits of their memory or disk, a separate copy is made for that VM.  We thus preserve security and isolation between VMs.)

The Project Fargo feature will become likely become available in vSphere 6.0 (scheduled for 2015). It will be also possible to use Fargo features using a vCenter API. Early benchmarks show that 120 sessions can be handled on a 2 socket/8 core host with 30% less CPU consumption than same number of RDSH session.

When the VM has been provisioned applications are assigned to this VDI workstation using the solution CloudVolumes, VMware claims a 30 x faster VDI provisioning. The combination of Project Fargo + CloudVolumes is internally called Project Meteor.

Project Meteor is focused on delivering these desktops to any device with an HTML5-based browser. View Composer is being replaced by Project Fargo to increase provisioning upto 30x.

Duncan Epping has written an excellent blog on Project Fargo.

Rob Beekmans has also written a blog on Fargo and Meteor which is worth reading.

Posted in Vmware | Leave a comment

Does disabling Inter-VM TPS affect your environment

Before starting with this post I would like to recall what is TPS and what it does in a virtualized environment.

Transparent page sharing is a method by which redundant copies of pages are eliminated. This helps to free memory that a virtual machine would otherwise be using. Because of the way TPS works with hardware-assisted memory virtualization systems like Intel EPT Hardware Assist and AMD RVI Hardware Assist, esxtop may show zero or few shared pages in these systems. Page sharing will show up in esxtop only when host memory is overcommitted.

The below diagram will show you an overview of what TPS is doing

TPS

Earlier TPS was enabled by default to save memory space by sharing the same memory block between the VM’s running the similar type of applications. In VDI environment where most of the VM are running with same OS and contains similar applications, TPS was saving a hell lot of space and making the memory management technique more efficient.

But sometimes back a research brought the darker side of using the TPS technique where it is demonstrated that TPS is a serious security threat. The research indicated that TPS can be abused to gain unauthorized access to data under certain highly controlled conditions.

Even though VMware believes information being disclosed in real world conditions is unrealistic, out of an abundance of caution upcoming ESXi Update releases will no longer enable TPS between Virtual Machines by default.

Starting with update releases in December, 2014, default setting for TPS will be Disabled and will continued to be disabled for all future versions of vSphere.

VMware has mentioned and addressed this issue in their KB Article 2080735. I would recommend to have a look into that. Also the KB article 2091682 “Additional Transparent Page Sharing management capabilities in ESXi 5.5, 5.1, and 5.0 patches in Q4, 2014” has explained how to overcome this issue for existing version of vSphere.

Now the question is “How disabling Inter-VM TPS impact your environment?”

If your environment is not using the concept of “Large Pages” and using the default 4kb page size then TPS will be handy to save some memory blocks for you especially when the environment consists of VM’s with same type of OS and running similar application for e.g. VDI.

But if your environment which consists of modern processors with MMU support (Intel EPT/AMD RVI Hardware Assist) then your ESXi host is leveraging the use of Large Pages (2MB) and by default there won’t be huge impact as TPS is not effective there (because the chances of 2 pages to be exactly similar will be very less). Also TPS is not called until host comes under memory pressure and pages are broken down to 4KB.

This is explained very well in VMware KB 1021095

Additionally modern operating systems like Windows 2008/2012 or Linux are leveraging security feature called Address Space Layout Randomization (ASLR), which is preventing TPS to be effective, especially in ZERO’ing when large pages are used.

TPS will be very advantageous in following 2 scenarios:

  • If you are having VDI deployments especially with floating (not dedicated) desktops into your environment and when the Esxi hosts are under memory pressure.
  • Also if you have older operating systems prior to Windows Server 2008, then your operating systems will be using small pages 4KB by default and hence they will be benefited from TPS a lot.
  • Also if you have disabled  the use of the large pages by using advance value Mem.AllocGuestLargePage 0, then in this case you should be benefiting from TPS even in case of “modern” operating systems.

If you want to check how much you are utilizing TPS then check performance tab in your vSphere client (on cluster or host level, but can be done on VM level too). In memory view look for SHARED and ZERO metrics.

SHARED is all memory which is saved with TPS and ZERO is memory with zeroed blocks collapsed into one. If you subtract ZERO from SHARED you will get an actual estimate of your savings from deduplication in general.

Note: Unfortunately there is now way to check how much savings you have from Inter-VM TPS

What to do if you dont want to wait for patches for disabling TPS?

Although VMware believes that the reported possible information disclosure in TPS can only be abused in very limited configuration scenarios, VMware advices customers who are sufficiently concerned about this possibility to proactively disable TPS on their ESXi hosts. Customers do not have to wait for the either the Patch or the Update releases to do this.

For environments using ESXi 5.x, perform the following steps:

  1. Login to ESXi or vCenter Server using the vSphere Client.
  2. Select the relevant ESXi host.
  3. In the Configuration tab, click Advanced Settings under the software section.
  4. In the Advanced Settings window, click Mem.
  5. Look for Mem.ShareScanGHz and set the value to 0.
  6. Click OK.

Perform one of the following to make the TPS changes effective immediately:

  • Migrate all the virtual machines to other host in cluster and back to original host.
  • Shutdown and power-on the virtual machines.

What if you want to continue using TPS after the Patch/Update?

A couple of new Advanced Configuration options are introduced by the Patch and is explained in KB2091682

  • Mem.ShareForceSalting: This is a host-level configuration option. This is what disables/enables TPS on an ESXi host. If this is set to “0”, it means that TPS is STILL enabled on the host. If set to “1”, it means that TPS has been disabled on the Host, and salting is required in order for TPS to work on any VM located on that host.
  • sched.mem.pshare.salt: This value enables customers to selectively enable page sharing between/among specific VMs. When ShareForceSalting is set to “1” on an ESXi host, the only way for two or more VMs to share a page is for both their salt and the content of the page to be same. The salt is the value specified by customers for this per-VM Advanced Configuration option. This value must be identical on all the VMs that you intend to enable page sharing for.
  1. Log in to ESXi or vCenter with the VI-Client.
  2. Select the ESXi relevant host.
  3. In the Configuration tab, click Advanced Settings under the software section.
  4. In the Advanced Settings window, click Mem.
  5. Look for ‘Mem.ShareForceSalting’ and set the value to 1.
  6. Click OK.
  7. Power off the VM, which you want to set salt value.
  8. Right click on VM, click on Edit settings.
  9. Select options menu, click on General under Advanced section
  10. Click on Configuration Parameter
  11. Click on Add Row, new row will be added.
  12. On left hand side add text ‘sched.mem.pshare.salt’ and on the right hand side specify the unique string.
  13. Power on the VM so that salting can take effect.
  14. Repeat steps 7 to 13 to set the salt value for individual VMs.
  15. Same salting values can be specified to achieve the page sharing across VMs.

IF ShareForceSalting is set to “1” and the sched.mem.pshare.salt is not set on a VM, the VM’s vc.uuid will be substituted for the salt value instead. Because the vc.uuid is unique to a VM, that VM will only be able to share page with itself – effectively, no sharing for this VM.

For those who wants to dive more deeper into this I would recommend reading this  whitepaper: https://eprint.iacr.org/2014/435.pdf

Posted in Memory_management, Vmware | 3 Comments

Understanding Advanced Snapshot Management

Deleting virtual machine snapshots without wasting disk space

Before using snapshots on your VM, analyzing free disk space on the VMFS volume is very important.  As a best practice or thumb rule you should have least 20% of the virtual machine’s total disk size as free disk space before using snapshots. But this amount can vary depending upon the type of server or how long you will keep the snapshots or if you are planning to use multiple snapshots.

If you are planning to use snapshots on servers like database servers or file servers the amount of free space that should be present on underlying datastore or VMFS volume will change drastically as comparison to using snapshots on servers like web servers or say DNS server because the amount of data written on disks in case of file or database server is much more than any other type of servers.

More importantly if you are planning to include the memory state of the VM’s with snapshots, you’ll also need to allow for extra disk space equal to amount of RAM assigned to the VM.

VM’s with only one snapshot requires no extra disk space when deleting, or committing the snapshots. An extra helper delta file also is created at the time of deleting the snapshots. This helper delta file contains any changes that are made to the VM’s disk while the snapshot is deleted. The size of the helper delta file varies and it’s based on how long the snapshot takes to delete. But in general this file is small in size as most snapshots are deleted in less than an hour.

The amount of extra disk space that is required while deleting multiple snapshot depends on the vSphere version in use because of the way they are merged into the original disk file. The process for deleting multiple snapshots has changed across vSphere versions.

In vSphere 4.0 versions and VMware Infrastructure 3, if a VM has 3 active snapshots and delete operation is performed then the following process occurs:

Snapshot 3 is copied to Snapshot 2, which is then copied to Snapshot 1. Next, Snapshot 1 is copied to the original disk file, and the helper snapshot is copied to the original disk file, as outlined below.

deletesnaphots1

Graphic Thanks to searchvmware.techtarget.com

This process requires extra disk space because each snapshot grows as the previous snapshot is added to it. If there isn’t sufficient free disk space on the data store, the snapshots cannot be committed.

In later vSphere 4.0 versions and vSphere 4.1, each snapshot is merged directly into the original disk, instead of merging with the previous snapshot. The figure below explains what happens when a VM has 3 snapshots active and you deleted them.

deletesnaphots2

Graphic Thanks to searchvmware.techtraget.com

Because each snapshot is directly merged into the original one at a time, no extra disk space is needed, except for the helper file.

Eric Siebert has mentioned one very good word of caution regarding snapshot operation on searchvmware.techtarget.com which is as follows:

Don’t run a Windows disk defragmentation while the VM has a snapshot running. Defragment operations change many disk blocks and can cause very rapid growth of snapshot files

How long does it take to delete a snapshot?

When deleting snapshots through the vSphere Client, the task status bar can be misleading. Generally, the task status jumps to 95% complete fairly quickly, but you’ll notice it will stay at 95% without changing until the entire commit process is completed. vCenter Server has a default 15-minute timeout for all tasks, which can be increased. Thus, even though your files are still committing, vCenter Server will report that the operation has timed out.

One simple method for finding out when a task completes is to look at the VM’s directory using the Datastore Browser in the vSphere Client. When the delta files disappear you know that the snapshot deletion has completed.

There is also command-line method available in ESXi that you can use to monitor the status of snapshot deletions. It is explained in this VMware KB article

Snapshots that have been active for a very long time becomes extremely large in size and can take a very long time to commit when deleted. The amount of time the snapshot takes to commit varies depending on the VM’s activity level; it will commit faster if it is powered off. The amount of activity your host’s disk subsystem is engaging also affects the time the snapshot takes to commit.

A 100 GB snapshot can take hours to merge into the original disk, which can affect VM and host performance. For this reason you should limit the length of time you keep snapshots and delete them as soon as you no longer need them.

Effect of Snapshots and metadata locks on host performance

Snapshots have a negative impact on the performance of your host and virtual machines in several ways.

When the snapshot is taken for the first time activities on the VM activity are paused briefly. Even you will experience a few ping timeouts on your VM when snapshot creation is in progress. Also, creating a snapshot causes metadata updates, which can cause SCSI reservations conflicts that briefly lock your LUN. As a result, the LUN will be available exclusively to a single host for a brief period of time.

When a VM has an active snapshot, the performance of the VM is degraded because of the fact that the host writes to delta files differently and less efficiently than it does to standard VMDK files.

Also, as the delta file grows by each 16MB increment it will cause another metadata lock. This can affect your VMs and hosts. How big an impact on performance this will have varies based on how busy your VM and hosts are.

deletesnapshot3

Deleting/committing a snapshot also creates a metadata lock. In addition, the snapshot you are deleting can create greatly reduced performance on its VM while the delta files are being committed; this will be more noticeable if the VM is very busy. To avoid this problem, it’s better to delete large/numerous snapshots during off-peak hours when the Esxi host is less busy.

Snapshot Best Practices

There are certain things which should be kept in mind while using snapshots. These are discussed as below:

Never expand a disk file with a snapshot running

You should never expand a virtual disk while snapshots are active. You can expand disks using the vmkfstools –X command or the vSphere Client.

In VI3, if you expand a disk using the VI Client, it reports that the task completes successfully. But it won’t actually expand the disk file. And if the virtual disk is expanded with vmkfstools command while a snapshot is active, the VM will no longer start, and you will receive an error:

” Cannot open the disk “.vmdk” or one of the snapshot disks it depends on. Reason: The parent virtual disk has been modified since the child was created”

In later version of vSphere, it is not possible to expand a VM’s virtual disk while a snapshot is running. Also vmkfstools command fails with an error:

” Failed to extend the disk. Failed to lock the file”

The option to resize the disk of VM (select VM disk in edit settings) is grayed out in vSphere Client when a snapshot is running . But once the snapshot is deleted, you can resize the virtual disk.

If a VM has a RDM disk attached, the disk size is managed by the physical storage system and not by vSphere. As a result, you can increase the disk size of an RDM disk while snapshots are active.

Caution: But this action can corrupt the RDM disk, so always ensure that you delete snapshots before increasing the size of an RDM disk.

Excluding virtual disks from using snapshots

If a VM has more than one disk then it is possible to exclude a disk from being included in a snapshot. For this you have to edit the VM’s settings and change the disk mode to Independent (make sure you select Persistent). The independent setting provides you the means to control how each disk functions independently, there is no difference to the disk file or structure. Once a disk is Independent it will not be included in any snapshots.

Note: You will not be able to include memory snapshots on a VM that has independent disks. This is done to protect the independent disk in case you revert back to a previous snapshot with a memory state that may have an application running which was writing to the independent disk. Since the independent disk is not reverted when the other disks are it could potential corrupt data on it.

For VMs that have RDM disks, if the RDM was configured in physical compatibility mode, it will not be included in any VM snapshots. But if the RDM was configured in virtual compatibility mode, it will be included in snapshots.

Posted in Snapshots, Storage, Vmware | 6 Comments

How Snapshots work in VMware

What is a Snapshot

We have heard this term time and again during our journey with working on VMware and many of us are aware of what it is. Still I am including the formal definition of snapshot as below:

Disk Snapshot in VMware refers to a copy of the virtual machine disk file at a certain point in time. It preserves the disk file system and system memory of your virtual machine by enabling you to revert to the snapshot in case something goes wrong.

Snapshots are very helpful in the cases when you are planning to upgrade or patch your mission critical applications and servers.

When a snapshot is taken on a VM then a new disk file by the name *–delta.vmdk is created in the same folder where VM disks are residing (Unless and until you are not using WorkingDir parameter)

All the write operations are freeze on the original vmdk disk present in your VM and the new write operations will be performed on the newly created delta disk. The original vmdk file becomes read only. When a snapshot is taken on a VM certain new files are created in the VM directory. Let’s have a look on those and understand the purpose of those files.

Files associated with snapshot

*–delta.vmdk file: This is the differential file which created snapshot is taken on a VM. It is also known as the redo-log file. A delta file will be created for each snapshot that you create for a VM. An extra delta helper file will also be created to hold any disk changes when a snapshot is being deleted or reverted. These files are automatically deleted when the snapshot is deleted or reverted in snapshot manager.

*.vmsd file: This file is used to store metadata and information about snapshots. This file is in text format and will contain information such as the snapshot display name, unique identifier (UID), disk file name, etc. The size of this file is 0 byte initially until you create first snapshot of a VM. From that point it will populate the file and continue to update it whenever new snapshots are taken.

*.vmsn file: This is the snapshot state file, which stores the exact running state of a virtual machine at the time you take that snapshot. This file will either be small or large depending on if you select to preserve the VM’s memory as part of the snapshot. If you do choose to preserve the VM’s memory, then this file will be a few megabytes larger than the maximum RAM memory allocated to the VM.

A .vmsn file will be created for each snapshot taken on the VM and are automatically deleted when the snapshot is removed.

If you want to look deeper into how snapshot works actually I would recommend reading this wonderful blog post:

http://www.cubicrace.com/2012/02/how-do-virtual-machine-snapshots-work.html

Understanding Snapshot rate of growth and disk space utilization

Many of times I have seen people asking questions (in different group/communities) related to what will be the size of the snapshot delta disk when snapshot is taken and how delta disk grows. By looking into the discussion I have seen sometimes very misleading information given by the users. So let’s understand how delta disk behaves actually.

The size of a snapshot file can never exceed the size of the original disk file. Any time a disk block is changed it is written in the delta file and is updated as changes are made. If every single disk block is changed in your VM after a snapshot is taken, then your snapshot would be of the same size as your original disk file.

There’s some additional overhead disk space that contains information used to manage the snapshots. The maximum overhead disk space varies; it’s based on the Virtual Machine Files System (VMFS) block size. Please refer to the below table:

Block size  Maximum VMDK size  Maximum overhead
 1 MB  256 GB  2 GB
 2 MB  512 GB  4 GB
 4 MB  1024 GB  8 GB
 8 MB  2048 GB  16 GB

The required overhead disk space can cause snapshot creation to fail if a VM’s virtual disk is close the maximum VMDK size for a VMFS volume. To understand this consider the below scenario:

A VM’s virtual disk  size is 512 GB on a VMFS volume with a 2 MB block size.In this case the maximum snapshot size would be 516 GB (512 GB + 4 GB), which would exceed the 512 GB maximum VMDK size for the VMFS volume and cause the snapshot creation to fail.

So as a best practice if you are planning to use snapshots, you should always create VMs with a virtual disk size that’s smaller than the maximum VMDK size by the amount of the maximum overhead (e.g., 512 GB – 4GB = 508 GB).

Note: Reference of the above wonderful example is taken from Eric Siebert post on searchvmware.techtarget.com

Initially the size of Snapshot files is small (16 MB), but will grow as writes are made to the VM’s disk files. Snapshots grow in 16 MB increments to help reduce SCSI reservation conflicts. When requests are made to change a block on the original disk, it is instead changed in the delta file. If the previously changed disk block in a delta file is changed again, it will not increase the size of the delta file because it simply updates the existing block in the delta file.

The rate of growth of a snapshot will be determined by how much disk write activity occurs on your server. Servers that have disk-write intensive applications, such as SQL and Exchange, will see snapshot files grow rapidly. On the other hand, servers with mostly static content and fewer disk writes, such as Web and application servers, will grow at a much slower rate.

When we create multiple snapshots, new delta files are created and the previous delta files become read-only. With multiple snapshots, each delta file can potentially grow as large as the original disk file.

Posted in Snapshots, Storage, Vmware | Leave a comment

AirWatch HOL Released!

Good news for those people who wants to try their hands learning vCloud AirWatch.

VMware has released the AirWatch Hands-On-Labs yesterday.  These labs can be accessed by going to AirWatch HOL.

The AirWatch labs are designed to give you an introduction to AirWatch and Mobile Device Management (MDM). You can find more information about AirWatch by visiting their website at https://www.air-watch.com.

The AirWatch HOL is broken up into three modules that can be done together or independently.  Module 1 can be skipped by those who are currently using AirWatch and want to see how easy it is to integrate it with your corporate infrastructure then you can go to Module 2 “Advanced MDM Using the AirWatch Cloud Connector” directly.

Module 3 is designed to introduce you to the single device/multiple user concept which has many uses cases including schools, hospitals and more.

airwatch-hol

More information about this can be found on VMware Blogs

Posted in Vmware | Leave a comment

vCloud Air Disaster Recovery Product Walkthrough

VMware vCloud® Air™ is a secure, dedicated hybrid cloud platform built on the VMware vSphere® foundation. It supports existing workloads and third-party applications, as well as new application development.

VMware has made available the product walkthrough of vCloud Air Disaster Recovery.

Following topics are covered in this video series:

1: DRaaS End to End Overview

2: Configure vSphere Replication for vCloud Air DRaaS

3: Connect vSphere Replication to vCloud Air

4: Configure a VM for Replication

5: Test Failover

6: Planned Failover

7: Failback

8: Advanced Concepts

Posted in Vmware | Leave a comment

Managing Auto Deploy using GUI

Auto deploy is used for PXE booting/installation of Esxi over the network. Unfortunately Auto Deploy can be configured only through command line (PowerCLI) and most of the VMware Admins are not very comfortable with command line initially.

To help solve this problem there is a fling available from VMware called Auto Deploy GUI which can help you in configuring/deploying Esxi servers easily.

The Auto Deploy GUI is a vSphere plug-in for the VMware vSphere Auto Deploy component. The GUI plug-in allows a user to easily manage the setup and deployment requirements in a stateless environment managed by Auto Deploy. Some of the features provided through the GUI include the ability to add/remove Depots, list/create/modify Image Profiles, list VIB details, create/modify rules to map hosts to Image Profiles, check compliance of hosts against these rules and remediate hosts.

autodeploy

Instructions for using Auto Deploy GUI

  1. Download the Auto Deploy GUI Plugin zip file, extract it and double-click the AutodeployGUI5xTP_build number.exe file.
  2. Follow the prompts in the wizard to complete the installation.

For detailed instructions on how to use Auto Deploy GUI  please download the below guide:

VMwareAutoDeployGUIPracticalGuide.pdf

The scope of this document is to demonstrate how to configure and use the Auto Deploy
GUI to manage stateless ESXi environments.

This fling can be downloaded from Here

Posted in Vmware, VMware Flings | Leave a comment

VM Resource and Availability Service

Today I was checking the VMware flings site and found one very cool fling which I think is worth sharing as it can help you analysing your environment and help you designing in such a way that resources are highly available to you in case of any disaster.

This Fling enables you to perform a what-if analysis for host failures on your infrastructure. You can simulate failure of one or more hosts from a cluster (in vSphere) and identify how many:

  • VMs would be safely restarted on different hosts
  • VMs would fail to be restarted on different hosts
  • VMs would experience performance degradation after restarted on a different host

With this information, you can better plan the placement and configuration of your infrastructure to reduce downtime of your VMs/Services in case of host failures.

vRasScreenshot

Instructions for using this fling

  1. Open hasimulator.vmware.com to access the web service.
  2. Click on “Simulate Now” to accept the EULA terms, upload the DRM dump file and start the simulation process.
  3. Click on the help icon (at the top right corner) for a detailed description on how to use this service.

This fling can be downloaded from Here

Posted in Vmware, VMware Flings | Leave a comment

Certificate Manager for vCenter Server Appliance 5.5

This Fling is a GUI application to replace digital certificates on the vCenter Server Appliance.  Easily deploy by selecting the components that need digital certificates replaced. This Fling works with vCenter Server Appliance 5.5.

The GUI wizard-based tool helps you by:

  • Replacing certificates for vCenter Server, Inventory Service, Log Browzer, and Auto Deploy
  • Providing Single-Sign On (SSO) that uses the same certificate as the vCenter Server certificate
  • Collecting backups of previously deployed certificates and associated files
  • Providing tool level logging

This fling can be downloaded from Here

tool_ui

System Requirements

  • vCenter Server Appliance 5.5
  • Windows OS with Java SE Runtime Environment 8

Instructions for Installing the fling

  • Make sure sshd service is running on vCSA
  • Run this command on vCSA to get the lookup URL: cat /etc/vmware-sso/ls_url.txt
  • On a Windows machine, make sure the  JRE_HOME system variable is set to the Java SE Runtime Environment 8
  • Do NOT place the certificate chain file, key and pfx of various component, on the same directory in file system

Previously deployed certificate files are copied with the old_rui prefix. Logs are available in the user home directory.

Steps:

  1. Launch the certmgr.jar
  2. Select the components whose certificates need to be replaced
  3. Provide the credentials of root user in vCSA
  4. Provide the credentials for vCenter Single-Sign On. In IP/Hostname field, use FQDN if lookup service url is using FQDN or vice-versa. User Name should be the system tenant (Administrator@vsphere.local) only
  5. Drag and drop certificate chain file, and the key and pfx from file system to their respective text fields
  6. Click on the Start button and wait until the Run Complete message appears
  7. Click on Logs to determine whether certificate replacement was success/fail
Posted in Vmware, VMware Flings | Leave a comment

Exporting Horizon View Events Database

The VMware View Events Database is used to record all the events that happen in a View environment. The database contains all the information but at times it can be difficult to extract those information for troubleshooting or analysis.

This is a fling from VMware called “Horizon View Events Database Export” which allows administrators to easily apply very detailed filtering to the data and export it to a .CSV file.

You can filter on time range, event severity, event source, session type usernames and event types. The application allows for extremely granular export of data. The exported columns can also be customized and the application will export data from both the live and the historical tables in the View Events Database.

This utility can be downloaded from VMware site at

https://labs.vmware.com/flings/horizon-view-events-database-export-utility

Posted in Vmware, VMware Flings | Leave a comment

VMware Fault Definitions

Today I was going through Vmware Resource Management Guide and found the below information and thought of sharing it across. The below information points to the faults which can occur in an environment and the reason why this fault has happened.

Having knowledge of common faults that can occur in an environment can help us to understand the issue and starting the troubleshooting.

Virtual Machine is Pinned

This fault occurs when DRS cannot move a virtual machine because DRS is disabled on it. That is, the virtual machine is “pinned” on its registered host.

Virtual Machine not Compatible with any Host

This fault occurs when DRS cannot find a host that can run the virtual machine.

This might occur, for example, if no host can satisfy the virtual machine’s CPU or memory resource needs or if no host currently has network or storage access needed by the virtual machine.

VM/VM DRS Rule Violated when Moving to another Host

This fault occurs when more than one virtual machines running on the same host and share affinity rules with each other cannot be moved to another host.

This might occur because not all the virtual machines can vMotion off the current host. For example, one of the virtual machines in the group is DRS-disabled.

Host Incompatible with Virtual Machine

This fault occurs when DRS considers migrating a virtual machine to a host, but finds that the host is incompatible with the given virtual machine.

This might occur because the target host does not have access to the network or storage connection needed by the virtual machine. Another reason this fault occurs is if the target host has a CPU that differs sufficiently from the current host so that using vMotion amongst the hosts is not supported.

To avoid this, create clusters such that all hosts are configured consistently and vMotion is compatible amongst the hosts.

Another reason the host is incompatible with the virtual machine is that there is a required VM/Host DRS rule in place that instructs DRS to never place this virtual machine on this host.

Host has Virtual Machine that Violates VM/VM DRS Rules

This fault occurs when the virtual machine, when powered on or moved by starting vMotion, would violate a VM/VM DRS rule.

The virtual machine can still be manually powered on or moved with vMotion, but vCenter Server cannot automatically do so.

Host has Insufficient Capacity for Virtual Machine

This fault occurs when the host does not have enough CPU or memory capacity for running the virtual machine.

Host in Incorrect State

This fault occurs when the host is entering maintenance or standby state when needed for DRS action to occur.

To address this fault, cancel the request for the host to enter standby or maintenance mode.

Host has Insufficient Number of Physical CPUs for Virtual Machine

This fault occurs when the host hardware does not enough physical CPUs (hyperthreads) to support the number of virtual CPUs in the virtual machine.

Host has Insufficient Capacity for Each Virtual Machine CPU

This fault occurs when the host does not have enough CPU capacity for running the virtual machine.

The Virtual Machine is in vMotion

This fault occurs when DRS cannot move a virtual machine because it is already in vMotion state.

No Active Host in Cluster

This fault occurs when the cluster in which the virtual machine is being moved does not contain any hosts that are connected and in a non-maintenance state.

This can occur, for example, if all the hosts are disconnected or in maintenance mode.

Insufficient Resources

This fault occurs when an attempted operation conflicts with a resource configuration policy.

This fault may occur, for example, if a power-on operation reserves more memory than is allocated to a resource pool.

Retry the operation after adjusting the resources to allow more memory.

Insufficient Resources to Satisfy Configured Failover Level for HA

This fault occurs when the HA configuration of CPU or memory resources reserved for failover is violated or cannot be met by the DRS operation under consideration.

This fault is reported when:

  • The host is requested to enter maintenance or standby mode.
  • The Virtual machine violates failover when it attempts to power on.

No Compatible Hard Affinity Host

No host is available for the virtual machine that satisfies its mandatory VM/Host DRS affinity or anti-affinity rules.

No Compatible Soft Affinity Host

No host is available for the virtual machine that satisfied its preferred VM/Host DRS affinity or anti-affinity rules.

Soft Rule Violation Correction Disallowed

DRS migration threshold is set at mandatory-only.

This does not allow the generation of DRS actions to correct non-mandatory VM/Host DRS affinity rules.

Posted in Vmware | Leave a comment

VMware launched VCP 6.0 training & certification

Finally VMware has announced the most awaited training and certification for VMware vSphere 6.0.

vSphere 6 Beta is out in market a while ago and many VMware admins have already dirtied their hands by playing around it. Now its time to get an official training and get certified on the same.

The updated certifications not only provides increased flexibility and career growth opportunities but also help in increasing credibility with employers, colleagues and clients. These revisions in certification path give you more options when choosing certification levels and technology areas.

VMware Certification Roadmap

Cert_Roadmap_2015Q1_v5_final_WEB

VMware vSphere 6: Install, Configure, Manage

This course is known as VMware’s one of the most popular certification course features hands-on intensive training that focuses on installing, configuring and managing VMware vSphere 6 that include VMware ESXi™ 6 and VMware vCenter Server™ 6.

All the essential information related to vSphere 6.0 certification is available at MyLearn Page

VMware-vSphere-Install-Configure-Manage

There are 3 distinct ways to attain a vSphere 6.0 training/certification and you can choose the one among the 3 that best suits your need.

So hurry up and register yourself for a web based training or instructor led classroom training.

Posted in Vmware | 1 Comment

VCP6-DCV Certification Now Available

Good news for all VMware Admins. Most awaited VCP6-DCV certification has been announced by VMware today.

This certification that validates your ability to deploy, configure, administer and scale a vSphere 6 virtualized data center, including administering and troubleshooting virtualization technologies.

Check out the requirements for new candidates and migration paths for current VCP on  MYlearn Website, where you can also find the complete objectives for each exam related to this certification:

Posted in Vmware | Leave a comment

PowerOn guest o.s in VMware Workstation using command

If you are running a lot of guest o.s in VMware Workstation and tired of going across tabs to power on these VM’s then there is a simpler way to do this. We can achieve this by using an application called vmrun which is part of VMware Workstation.

You can find this application in the directory where your VMware Workstation is installed.

For E.g: G:\Program Files (x86)\VMware\VMware Workstation

vmrun

How to use this utility

Go To CMD and browse to the path where vmrun.exe is placed and use the below command:

G:\Program Files (x86)\VMware\VMware Workstation>vmrun.exe start “C:\Lab VM’s\DC\DC.vmx”

Note you have to give path to your guest os vmx file in order to start it

vmrun2

You can even create a batch file if you want to power on your guest o.s with windows startup.

You can download the latest version of VMware workstation from Here

Hope this is informational to you. Share it on social media if you find it useful. Happy learning !!!

Posted in Vmware | Leave a comment

Understanding svMotion operation on a VM with snapshots

Prior to vSphere 5.o it was not possible to storage vMotion a VM that is running with snapshots. With 5.0 VMware has enhanced svMotion and included this feature as lot of customers were demanding for that.

But there is one caveat in this feature which many of us don’t know due to lack of documentation on this. I found one very useful article on VMware Blog and thought of sharing with you.

workingDir parameter

The location of a virtual machine’s snapshot redo log file is defined by the virtual machine .vmx file setting workingDir. By default, the workingDir property is the same as the virtual machine’s home directory. What this means is your snapshot is stored in the same directory where your VM base disks are residing.

if you don’t have enough space on datastore where your VM disk is currently placed and you want to take snapshot on that VM, then by use of workingDir parameter you can store snapshot  to another location/datastore.

This is described in detail in KB article 1002929.

Change in workingDir Parameter property in vSphere 5.0

In vSphere 5.0, the workingDir parameter behaviour has been changed and it is no longer used as a pointer to a location for storing snapshot delta disks. The workingDir setting will now only be used to determine the location for the snapshot data (.vmsn) file. The delta disks are now always stored in the same home folder as the base disk.

Now the question is why Vmware has changed the functionality of this setting in vSphere 5.0? The answer is for sake of Storage DRS. This change means that snapshot delta disks now share the same performance, availability and storage consumption characteristics as parent disks, making Storage DRS work more accurately.

It should also be noted that if you do a Storage vMotion of a VM with snapshots and the VM has the workingDir parameter set, the workingDir setting will be removed from the .vmx & the .vmsn snapshot data file will be moved to the home folder of the VM on the destination datastore. However you will get a warning in the migration wizard about this as shown below.

snapshot

Can this new behaviour be changed?

Yes this behaviour can be overridden but there is a catch.

If you really want to keep your snapshots on a different datastore to the base disk in vSphere 5.0, there is a new parameter which you must set along with the workingDir parameter. The new parameter,snapshot.redoNotWithParent = “TRUE”, must be placed in the VM’s .vmx file. This means that the workingDir parameter setting will now be used as the location for snapshot redo delta disks, as well as the .vmsn.

However the same caveat described above applies in this case – the workingDir setting will be overridden by a Storage vMotion operation, and all snapshot deltas will get migrated to the same folder as the VM’s base disk on the destination. Also, the workingDir setting will be removed from the .vmx file after the migration.

Therefore, if you use the snapshot.redoNotWithParent = “TRUE” parameter, you should refrain from doing Storage vMotion operations. The following warning will appear when a migration is attempted on a VM that has workingDir set in the .vmx file:

snapshot

In fact, this warning also appears if you have just the workingDir setting in the .vmx file as mentioned earlier.

Conclusion

Storage vMotion and Storage DRS both work fine with Virtual Machines that have snapshots, but we should refrain from doing Storage vMotion operations with VMs that have the workingDir & snapshot.redoNotWithParent parameters set in their .vmx in vSphere 5.0.

Share this post on social media if it is informational to you. Happy Learning !!!!

Posted in Snapshots, svMotion, Vmware | Leave a comment

Troubleshooting Esxi host Disconnection from vCenter issue

Last week I was facing a serious issue in my home lab where my esxi host is getting disconnected from my vCenter Server randomly. Whenever I am doing any configuration changes like enabling ssh or creating a new vSwitch the host got disconnected immediately. I was damn frustrated and was looking for a solution because it was very hard for me to work. So I started troubleshooting by going through my vCenter log files and found following:

2015-03-16T23:06:11.270+05:30 [06304 info ‘vpxdvpxdMoHost’ opID=BADE9DBF-0000007B-b1] [HostMo] host connection state changed to [CONNECTED] for host-35
2015-03-16T23:06:11.273+05:30 [06304 info ‘vpxdvpxdMoHost’ opID=BADE9DBF-0000007B-b1] [HostMo::SetComputeCompatibilityDirty] Marked host-35 as dirty.

2015-03-16T23:02:09.628+05:30 [04380 info ‘vpxdvpxdHostCnx’ opID=SWI-7e4a49e9] [VpxdHostCnx] No heartbeats received from host 5294adb1-584a-2f13-8987-7b52ed31c84b within 120665000 microseconds

2015-03-16T23:02:09.628+05:30 [09928 info ‘vpxdvpxdInvtHostCnx’] [VpxdInvtHost] Got lost connection callback for host-35
2015-03-16T23:02:09.629+05:30 [05548 info ‘commonvpxLro’] [VpxLRO] — BEGIN task-internal-46 — host-35 — VpxdInvtHostSyncHostLRO.Synchronize —
2015-03-16T23:02:09.629+05:30 [05548 warning ‘vpxdvpxdInvtHostCnx’] [VpxdInvtHostSyncHostLRO] Connection not alive for host host-35
2015-03-16T23:02:09.629+05:30 [05548 info ‘vpxdvpxdInvtHostCnx’] [VpxdInvtHost::FixNotRespondingHost] Attempting to fix not responding host host-35
2015-03-16T23:02:10.052+05:30 [05548 info ‘vpxdvpxdHostAccess’] Got VpxaCnxInfo over SOAP version vpxapi.version.version9 for host megatron.alex.local

2015-03-16T23:06:32.368+05:30 [07760 warning ‘Default’] Failed to connect socket; <io_obj p:0x000000000a6fa038, h:3300, <TCP ‘0.0.0.0:0’>, <TCP ‘[::1]:32010’>>, e: system:10061(No connection could be made because the target machine actively refused it)
2015-03-16T23:06:33.369+05:30 [07760 warning ‘Proxy Req 00047’] Connection to localhost:32010 failed with error class Vmacore::SystemException(No connection could be made because the target machine actively refused it).

So I guess something wrong was happening related to heartbeat exchange between my host and vCenter server. I started my troubleshooting by following below steps:

1: Checked whether Esxi is able to reach my vCenter server or not by pinging and doing a telnet from Esxi host to vCenter Server on port 902

Note: Telnet command wont work in Esxi so you have to use “nc -z” command

Res-1

Res-2

So as you can see I  was able to reach my vCenter from my Esxi host successfully.

2: Next I checked whether or not my Esxi host is listening on port 902 (heartbeat port)

Res-3

The above command verified yes my host is listening on port 902

4: I added the host disconnection timeout string in Advance Setttings of vCenter and increased the value to 120

Res-7

I verified once again that value has been added.

Res-8

4: Next I check my vCenter Server for “Managed IP Setting”. Sometimes if the vCenter IP is not listed then also you can face this issue.

Res-5

In my case I manually entered IP under Run Time Settings as shown in above image.

5: I checked the same settings on my Esxi host.

Res-4

So from above image it is pretty clear that my Esxi host is configured to managed by correct vCenter server.

6: Next I checked for Heartbeat Port Value on my Esxi host by running the command:

# grep -i serverport /etc/vmware/vpxa/vpxa.cfg

The output which I got was something strange as my Esxi host was using port 922 for heartbeats exchange instead of using default port 902.

According to VMware KB Article 2040630

This issue is caused by dropped, blocked, or lost heartbeat packets between the vCenter Server and the ESXi/ESX host. If there is an incorrect configuration of the vCenter Server managed IP address, the host receives the heartbeat from vCenter Server but cannot return it.

It is important to remember that the default heartbeat port is UDP 902, and these packets must be sent between vCenter Server and the ESXi/ESX host for the host to stay connected and remain in the vCenter Server inventory.

Res-9

I changed the port to 902 by editing the vpxa.cfg file and removed and added back my Esxi host to vCenter Server and hoped that my issue is now resolved. But surprisingly I was still getting the disconnection problem. Once again I connected my Esxi host using ssh and checked vpxa.cfg file and found the port has been again changed to 922. This was strange.

On digging more I found that this is happening because of heartbeat port specified as 922 in the registry key of vCenter server. I got this clue from one of the issue 2437489 posted in VMware Community group.

The full registry key is :

HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter

Res-10

As you can see in above image the heartbeat port is 922 which is causing all the troubles. I changed it to 902 and restarted my vCenter Server and bingo my issue is resolved.

Hit Like and share on social media if above information is helpful to you. Happy Learning!!!

Posted in Vmware | 10 Comments

RVTools v3.7 Released

Good new for the folks who are big fan of RV Tools and use this tool quite a lot in their production environment.

RVTools is a windows based application designed to use for VMware admins and it lists information about VMs, CPU, Memory, Disks, Partitions, Network, Floppy drives, CD drives, Snapshots, VMware tools, Resource pools. It also displays stats of Clusters, ESX hosts, HBAs, Nics, Switches, Ports, Distributed Switches, Distributed Ports, Service consoles, VM Kernels, Datastores, Multipath info and health checks.

RVTools can be used to disconnect the cd-rom/floppy drives from the virtual machines and even update the VMware Tools installed inside each virtual machine to the latest version.

rvtool

The new features/enhancements that are included in v 3.7 are :

  • VI SDK reference changed from 5.0 to 5.5
  • Extended the timeout value from 10 to 20 minutes for really big environments
  • New field VM Folder on vCPU, vMemory, vDisk, vPartition, vNetwork, vFloppy, vCD, vSnapshot and vTools tabpages
  • On vDisk tabpage new Storage IO Allocation Information
  • On vHost tabpage new fields: service tag (serial #) and OEM specific string
  • On vNic tabpage new field: Name of (distributed) virtual switch
  • On vMultipath tabpage added multipath info for path 5, 6, 7 and 8
  • On vHealth tabpage new health check: Multipath operational state
  • On vHealth tabpage new health check: Virtual machine consolidation needed check
  • On vInfo tabpage new fields: boot options, firmware and Scheduled Hardware Upgrade Info
  • On statusbar last refresh date time stamp
  • On vhealth tabpage: Search datastore errors are now visible as health messages
  • You can now export the csv files separately from the command line interface (just like the xls export)
  • You can now set a auto refresh data interval in the preferences dialog box
  • All datetime columns are now formatted as yyyy/mm/dd hh:mm:ss
  • The export dir / filenames now have a formated datetime stamp yyyy-mm-dd_hh:mm:ss
  • Bug fix: on dvPort tabpage not all networks are displayed
  • Overall improved debug information

This tools is developed by Rob de Veij and can be downloaded from Rob’s Homepage

A million thanks to Rob for developing such a good tool and making the life of all VMware admins easy.

Posted in Vmware | 1 Comment

Migrating Windows VC with MSSQL to VCSA with Postgres DB

Earlier it was not possible to migrate windows based vCenter Server installation with MS-SQL database to vCenter Server Appliance with embedded Postgres DB. But now it is possible to achieve this using a fling called VCS to VCSA Converter. 

Here is a diagram to visualize this process.

vcs-to-vcsa-converter-appliance-diagram

This fling is available on VMware Flings.

Summary

The VCS to VCVA Converter Appliance allows customers to migrate from Windows vCenter Server with an External Microsoft SQL Server Database to the vCenter Server Appliance with an embedded vPostgres database. The Fling migrates the vCenter database, roles, permissions, privileges, certificates and inventory service. The target appliance will run at the same IP address as the source vCenter.

vcs to vcsa

System Requirements

  • vCenter Server running on Windows – vSphere 5.5 or greater
  • The Windows vCenter Server and the vCenter Server Appliance should be running the same version (e.g. vCenter Server Windows 5.5u1 to VCSA 5.5u1)
  • The vCenter Server Appliance should be deployed with at least the same number of CPUs and at least the same amount of memory as the Windows vCenter Server host
  • vCenter Components (Inventory Service, vSphere Web Client and VMware Single Sign On) must be running on the same host as the vCenter Server
  • External Microsoft SQL Server 2008R2 or later for the vCenter Database (VCDB)
  • vSphere Web Client Plugins connected registered with an Active Directory user
  • VMware Single Sign On User/Groups are currently not migrated (require re-registration)
  • Migration Appliance must be able to communicate with the Windows vCenter Server Database and its database as well as the new vCenter Server Appliance. The following ports are used for this communication and should be open on the vCenter Windows server and on the VCSA:
  • Ports: 22 (ssh), 443 (https), 445 (SMB)

Limitations:

  • Microsoft SQL Server and vCenter Server must be on separate hosts
  • Microsoft SQL Express Database is not supported in version 0.9
  • VMware Single Sign On Users and Groups are not migrated in version 0.9
  • Windows Local Users and Groups are not migrated in version 0.9
  • vCenter Alarm action scripts are not migrated in version 0.9
  • The migration will require some downtime for the vCenter Server
  • Linked Mode configuration is not migrated. Multiple vCenters must be migrated separately
  • Any VMware or 3rd party vSphere Web Client plug-ins (e.g. VUM, NSX) that are running on the same host as the vCenter Server will not be migrated

You can watch the below video to walkthrough the process

Posted in Vmware | Leave a comment

Explore and play with vSphere 6.0

VMware-vSphere6

vSphere 6.0 GA has been released by VMware some days back. After reading what’s new in vSphere 6.0 I am all excited to give it a shot like all other VM admins. So here are the download links and related informations to start with.

Release Notes:

Before diving into downloading and start installing I encourage to look for the release notes. The link to the release notes is vSphere 6.0 Release Notes

What’s New in vSphere 6.0

Several posts have been written by our fellow VM admins on this topic and I too wrote a brief post on this. However you can access the official documents from below link:

http://www.vmware.com/files/pdf/vsphere/VMware-vSphere-Platform-Whats-New.pdf

Upgrade Considerations:

If you are planning for upgrading to vSphere 6.0 from previous versions instead of new installation then I would strongly recommend to go through the upgrade recommendations by VMware.

vSphere 6.0 Upgrade Information

Now we are good to jump into download section. Click on vSphere 6.0 Download Links and login to My VMware with your registered user account and request for new trial to start downloading.

Training Videos and Documentation Center

For links to Training videos and Documentation Center you can read my post vSphere 6.0 Training Videos and Docs

Hit like and share this info on social media if the above information is helpful for you. Happy Learning !!!!!

Posted in Vmware | Leave a comment

vSphere 6.0- Training Videos and Documents

vSphere 6.0 GA is out now in market and I know we VM admins are very excited to get our hands dirty into implementation and learning.

So here I am sharing some useful links which help us all in getting started with learning vSphere 6.0

Video Links:

Documents Links:

https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-6-pubs.html

Posted in Vmware | 3 Comments

Tweaking vCenter Server for home lab

Today I was looking for some tricks to tweak vCenter server 5.5 for my home lab as I have only 8 GB RAM available and my lab was damn slow. Below is my lab configuration

vCenter Server v5.5 (with embedded MS SQL Server) – Installed on Server 2008 R2 SP1 OS (Physical)

2 Esxi Server v5.5 – 2.5 GB RAM each (Virtual)

Domain Controller + DNS (server 2003) – 512 MB RAM (Virtual)

Openfiler – 768 MB RAM (Virtual)

After powering on all the VM’s which are running inside vmware workstation (installed on my server 2008) my LAB has become too slow and memory utilization was shooting upto 7.8 GB.

It was nearly impossible to work anything from GUI (VI Client + Web Client) as my Esxi hosts were getting disconnected from VC time and again whenever I was doing any configuration (addition or modification) due to the SQL timeout and I was relying on just command line.

This is when I decided to search some tweaking methods for my Lab and found some good stuff which I am sharing in this post.

I was doing some analysis in Task Manager and open Resource Monitor to find out what piece of my lab is eating more RAM and found that the main culprit was the “VMWare vCenter Inventory Service” and “VMware Web Client service”. The Web Client service had the 1.5GB Java process and the Inventory service java process was consuming around 800 MB of RAM.

1

Initially I thought of shutting down Inventory service as I have only a few components in my LAB and it will not affect much as it is only used to store the client INVENTORY data in a memory cache and helps in searching across vCenter faster.

I cannot shutdown the Web Client service as all the new vSphere 5.5 features are available with Web Client only and the thick VI Client is of very less use.

So I got some tweaks to fix this issue. Here are the fixes:

If you want the vCenter Inventory Service, and the Web Client running, there is a way to trim down the memory that JAVA uses.  You can tweak config files (wrapper.conf) on the vCenter Server. Here is how to reduce the memory requirements.

Inventory Service Tweaking

Edit C:\Program Files\VMware\Infrastructure\Inventory Service\conf\wrapper.conf

Note: Before editing save a backup copy of WRAPPER.CONF

Open WRAPPER.CONF with a text editor and look for the section:

# Maximum Java Heap Size (in MB)

Modify “wrapper.java.maxmemory”

Mine was set to 3072, I changed it to 800 MB.

Save the file.

Open Task Manager and then open Resource Monitor and look for the JAVA process using the most RAM.

Shutdown / Restart the “VMware vCenter Inventory Service” and observe the memory difference.

You can also tweak the other services the same way.

Web Client Tweaking

Wrapper File Location

C:\ProgramFiles\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf\wrapper.conf

SSO Tweaking

Configuration file location:

C:\ProgramData\VMware\CIS\runtime\VMwareSTS\conf\wrapper.conf

Search the below string and set the appropriate heap size for you SSO java process

wrapper.java.additional.9=”-Xmxheap_sizeM”

Virtual Center Management Web Services Tweaking

Wrapper File Location:

C:\Program Files\VMware\Infrastructure\tomcat\conf\wrapper.conf

Log Browser  

Edit: C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\wrapper.conf

If the wrapper.java.maxmemory value isn’t there you can add it.

Note: You could shut down the “VMware VirtualCenter Management Webservices” also; however, this will cause an IE type error message in the vSphere Client if you try to look at OVERVIEW on the PERFORMANCE tab.

Windows Server 2008 Tweaking

Apart from tweaking vCenter Server components I also shutdown some of the unnecessary services running on my Server 2008 machine. Here is the complete list.

  • Clipbook service
    This service is a relic of NT3.x. Used to support Clipbook Viewer which allows remote viewing of the clipbook. Default for workstation is manual. Ensure it is set to manual or disabled.
  • Computer Browser
    The browser service is used to maintain the list of PCs you see in Network Neighborhood. This is normally a server function. A home user can set this to Manual.
  • Distributed Transaction Coordinator
    W2K/XP service. Coordinates transactions that are distributed across two or more databases, message queues, file systems, or other transaction-protected resource managers. A home user can set this to Manual.
  • Fax Service
    W2K/XP service. Set to Manual if you don’t need fax services.
  • Internet Connection Sharing
    W2K/XP service. If you are want to share an Internet connection for your home network, then set this to Automatic. If not, leave this set to Manual.
  • IPSEC Policy Agent
    W2K/XP service. Manages IP security policy and starts the ISAKMP/Oakley (IKE) and the IP security driver. If not, leave this set to Manual.
  • Network DDE
    Supports network transport of DDE (Dynamic Data Exchange) connections. Such connectivity is mostly a relic from the NT 3.x days.
  • NT LM Security Support Provider
    Provides security to remote procedure call (RPC) programs that use transports other than named pipes. A home user can set this to Manual
  • Plug and Play
    Give something like PnP functionality but unless you are using unimodem modems, don’t bother.
  • Remote Registry Service
    W2K/XP service. Allows remote registry manipulation. A home user can set this to Manual.
  • Runas service
    W2K/XP service. Enables starting processes under alternate credentials. A home user can set this to Manual.
  • Server service
    you can disable the server service unless you are sharing files on your hard drive or your printer. If you have a DSL or cable modem, stop this service. Hackers will get nowhere if you do.
  • Spooler
    Print Spooler service in W2K/XP. Spooler in NT. Loads files to memory for later printing. If you don’t have a printer, you can set it to manual.
  • TCP/IP NetBIOS Helper
    Provides support for name resolution via a lookup of the LMHosts file. If you are not using LMHOSTS name resolution, you can set it to Manual.
  • Telephony Service
    Provides Telephony API (TAPI) support for programs that control telephony devices and IP based voice connections on the local computer and, through the LAN, on servers that are also running the service. Normally set to Manual on workstations. Leave it on Manual.

The last tweaking which I had done is to increase the page file size in my windows server 2008 box by setting it to larger value than what my system was providing by default.

I hope this post is informational to you. Hit like and share it. Be socialable.

Posted in Vmware | 2 Comments

vExpert Freebies

Every year vExperts are chosen all round the world for their contribution towards the VMware community and for their efforts VMware offers the vExperts access to community (private forum, access to beta products etc.)

Besides the public recognition of being a vExpert, they get additional benefits like free subscriptions, NFR licenses and so on from 3rd party vendors. However the information given below is available on the vExpert Community Page but still I am sharing this so that if anyone has missed it, they can checkout the benefits for year 2015 from here.

Vendor vExpert freebies

Vendor (Software/Goodie) Category? Link
Pluralsight (Free subscription for one year) Learning Free training for MVPs and vExperts
Login VSI Software license VIP Program – Login VSI
Starwind Software license Free NFR License V8
Veeam Software license FREE NFR LICENCE FOR Veeam Management Suite for VMware and Hyper-V
Datacore Software license http://pages.datacore.com/Free_NFR_Software.html
Solarwinds Software license Virtualization Manager NFR
Symantec Software license Free Backup Exec 2014 V-Ray Edition with Support
Royal TS/X Software license Royal TS/X NFR License
Proximal Data Software license vExperts Give Us a Try
Devolutions Remote Desktop Manager Software license Send an email to mtrottier@devolutions.net along with a link to your VMTN profile.
HP Storage (StoreVirtual VSA) Software license Send an email to hpstorageguy@hp.com – 3 year NFR license. (Have offered this for 2+ years)
Darren Woolard (vExpert.me URL Shortener and a vExpert Sticker) Goodies URL Shorten Tool: http://vexpert.me/vexpertme and vExpert Stickerhttp://vexpert.me/wordcloud
Liquidware Labs (25 licenses VDI Essentials + 1 Flex-io server license – perpetual license) Software License Send an email from company address along with date certified vExpert tosales@liquidwarelabs.com
VSS Labs (vCert Mgr) Software license More Info No responses to requests apparently.
ControlUp Software License Request license
Naviko Backup Software License VMware Backup Free NFR License | NAKIVO Backup &amp;amp; Replication | NAKIVO
Opvizor predictive Analysis – Enterprise. Software License Request License, http://try.opvizor.com/vexpert/ or e-mail Atif Siddiquiasiddiqui@opvizor.com / 312-363-7506  say Roger Lund sent you.
Nexenta Software License Free One-Year NexentaConnect for VSAN License
KEMP Technologies Software License FREE KEMP LoadMaster NFR License – VMware Certified Professional and vExpert – LoadMaster NFR License
Posted in Vmware | Leave a comment