Two Weeks in 2017

Two weeks have passed like a flash, and I could sense things are flooding be information or for that matter work. I want to blog about Nutanix. But no time so far. I finished two posts on Nutanix Block Service here and here. I have yet to complete 3rd part. I’m hoping to finish it before next week. As I’m on Nutanix, I’m extremely glad to be considered as Nutanix Technology Champion 2017


In this post, I’m covering what has happened for last two weeks. I must regret what’s going on is missing

Microsoft Azure Exams Promotion is here.  Please go and enroll. At least I have enrolled.

Learn and build Azure skills
Learn and build Azure skills

Are you still thinking about advancing your skills on Public Cloud? Read what this VCDX is planning.

Good Read White paper to build an efficient Availability strategy for your remote and branch offices by Veeam.

In Brief,

  1. What are the characteristic of modern data center and how Veeam is supporting those changes

  2. Integration with the Public Cloud. I came across this restore directly feature (I think 9.5 exclusive feature) to Azure

  3. The importance of 3-2-1 principle.  With Veeam 3-2-1 can be achieved using

  4. Replicate VMs to another location

  5. Moving backup jobs to remote location using Backup Copy Job

  6. Moving Backup job to Public cloud using Veeam Cloud Connect

  7. Finally, tape  out the backup


Veeam agent for Microsoft Windows is supported for Windows Server as Window Desktops


self-service restore of file server and choice of file types restore

Lot of people fail to understand file servers are having low rate of change on the underlying disk as most read rather write operations

Azure Backup protects against ransomware

The article explains how critical it is to ensure backup are inaccessible to a malicious attacker. Similar post done by Veeam has been shared by me here. Azure backup has introduced new features to provide enhanced protection for the backup data. Further, you can read FBI Recommendation on backing up data to offsite/cloud here.

If a hacker does delete backup data, Azure Backup will store the deleted backup data for up to 14 days after deletion.

Know your design – My approach for VCDX defense prep

This is the core skill any Architecture must hold. In the know, your design post, Paul (VCDX) does a great job. I  appreciate for putting these thoughts in the post. I’m also the big fan of this VCAP-DCD design which I used during my exam preparation.

What’s New with vSphere 6.5

Test-drive the full capabilities of vSphere 6.5 by running the new hands-on lab!

New Walkthroughs for vCenter High Availability

There is no better way to learn vSphere and its new features. Walk through focuses on the use case and new features and configuration you must be aware of.


AOS 5.0: Performance improvements in Nutanix AHV

Kees explains the improvements in AHV. I’m seriously impressed with AHV and where it is heading. The only passionate company can take KVM to the next level. So far I have not heard any major features on other KVM version.  But with KVM the biggest improvement (To Me) is about writing a driver to leverage Change Block Tracking (CBT).  CBT feature was crucial as it was stopping from all Vendors to provide image level backup for KVM.

In my personal experience, I have noticed latency decreased by 50% just by updating AOS version which AOS 4.5 to AOS 4.7

Setting Up and Using Acropolis File Services (AFS) on Nutanix AOS 5.0

AFS is the killing feature, and its impact will very soon be seen in the industry. In the traditional world, you have to depend on EMC/NetApp and other vendors for NAS functionality for just for VDI which again adds the huge cost to it. I can’t forget the Antivirus required just for NAS filer. With this release, even it is supported on ESXi.

Below Dennis walks us through AFS setup

Recovering Data and VM Using Vembu

So far I have covered how to take backup of Files, Virtual Machines, and multiple methods to protect the data. Today I will demonstrate restoration process to restore Virtual machines. Recovering data in Vembu is simplified. It is few steps I had to perform.

Restore Option

I noticed there are several options to restore which is seen below.

Restoration Options in Vembu
Restoration Options in Vembu

Continue reading Recovering Data and VM Using Vembu

Data Protection Approach -HealthCare

There is a growing trend in using Electronic Medical Records in all advanced hospitals. EMR has provided tremendous advantages in keep track of patient health records. As technology has advanced so is the threat to exploit this highly sensitive data has grown exponentially. Data breaches in Healthcare industry has seen steep growth. To protect highly sensitive data traditional approach is not sufficient. Defense in depth approach is essential to protect against data breaches, as each layer offer unique protection mechanism

Data Protection Mechanism

Standard approach to Data Protection are

  1. Backup of the data
  2. Access control on the data
  3. Governance of the data
  4. Protection of the backup data by encrypting the backup copy
  5. Encryption of data while it is as rest to protection against drive theft

Each of the approaches has the direct impact on protection level of the data. All EMR data must be backed up and encrypted. Encryption provides the extra layer of security against physical theft. Apart from backing up the data, it must be sent off site or tape out. An offsite copy of data protects against any tampering of data, a practice usually observed by Ransomware attacks.

Access control on the data

Security controls on who can access the data, when she can access the data and why needs to access the data must be reviewed against all the highly sensitive data. Access to data must be logged for Auditing purpose. Any abnormal access of the data even by unauthorized personnel must be alerted. Examples of access control could be restricting access to files and folders to authorized personnel during particular time only. Complex password policies with password histories must be enabled. Single sign-on or biometric authentication solution can be a solid safeguard against unauthorized usage of the system. While data is accessed over networking, it must be guaranteed that data passes using encrypted channel e.g. SSL

Prevention Against Data Breaches

Data breaches could occur via end points. Antivirus and URL filtering is the must, but it is not enough in today world. The majority of attacks occur via the web browser in the form of Malware and Malvertising Servers and application must be patched periodically. Any vulnerability on the system provides an easy method for exploiting it and infecting the servers. Mail hygiene is another critical element which can guard IT system and filter all email attachment reaching in your inbox Admin privileges can create a disaster if Exploiting Kit finds a vulnerability in an application and implant malicious code. Limited privileges will safeguard against these attacks spreading across the systems and restrict the damage it could cause with admin privileges. Group policies could be used to restrict device access, restriction on USB access, disabling serial ports, camera on laptop must be a part of the security framework.


I hope you had a chance to review Acropolis Block Services Introduction covered in Part-01 of this series. If not, please consider it as it is the foundation of Part-02 i.e. this post. In this post, I will explain pre-requisites, Terminology, and some considerations to keep in mind.


  1. Ports 3260 and 3205 are opened between iSCSI Initiator and iSCSI Target
  2. IP Address for iSCSI Target. By the way, Acropolis Block Services (ABS) refers this IP Address as External Data services IP Address
  3. AOS is 4.7 & Above
  4. iSCSI Initiator Name is also known as IQN (iSCSI Qualified Name)
  5. Volume Group

What is iSCSI Initiator and iSCSI Target

While iSCSI initiator is the client sitting on Operating system where iSCSI target is on the storage side waiting to be discovered and to respond to queries of iSCSI initiator.

Basics of iSCSI (initiator & Target)
Basics of iSCSI (initiator & Target)

What is volume group?

The Volume Group (VG) as the name denotes the group of Virtual Disks (vDisks). VG are created on containers. vDisk can also be referred as LUN (old school).

Few Considerations in Acropolis Block Services

  • vDisk is owned by Single Controller Virtual Machine (CVM). As a result, vDisks are always reached from this CVM which is referred as preferred CVM and therefore preferred Path to reach VG/vDisks.  To put it in another way, each iSCSI Target has preferred CVM. Preferred CVM is automatically selected based on Load Balancing Algorithm. Although in some special cases you can select CVM of your choice.
  • Starting with AOS 4.7, iSCSI initiator no longer initiates a direct connection with CVM. Instead, iSCSI initiator discovers VG using External Data Services IP. External Data Services IP Address is configured at the cluster level. Hint: Cluster design considerations.
  • External Data services IP act as Discovery Portal. External Data services IP Address is responsible for Path management and load balancing. MPIO of native OS is not required.
  • Login redirection occurs on per Target basis. Target can be multiple VGs or Single VG with multiple vDisks also referred as Virtual Targets
  • Total 32 Virtual targets can be configured per VG. In other words, if VG contains 32 or more vDisks, then an iSCSI client will see only 32 vDisks.
  • CVMs CPU utilization will be decided not only by the number of vDisks accessed but also by the number of VMs accessing it. Subsequently, there might be a situation where one CVM will consume more CPU than others. However, there is no sizing/design consideration required Since ABS and ADS are tightly integrated, and there is 85% threshold configured by default on CPU utilization of CVM. Let me reiterate ADS is set by default.
  • By default vDisks are thinly provisioned.(Design Consideration: Right-Sizing for storage)
  • Online expansion of vDisks. (Forget Design Consideration for Right-Sizing for Storage)
  • Name of the VG target starts with VG name and ends with Virtual Target number. Virtual Target Number starts with “0.”
CVM Failure Scenario

Whenever CVM fails/unavailable there is Zero impact to storage connectivity between iSCSI Initiator (VMs/Physical Server connectivity) and iSCSI Target (VG). For instance, there is an interruption of only 15-20 seconds which is fairly within the usual disk timeout sustained by various Operating Systems. Let me explain this using some simple picture.

imageiSCSI initiator sends a discovery request to External Data Services IP Address. External Data Services IP Address responds with discovery Target (VG1).

Acropolis Block Services
Acropolis Block Services

image iSCSI initiators send login request (using CHAP) credential to access VG1. External Data Services IP Address redirects Login request to CVM01.

imageCVM responds back to VM1  with login success. Here on all request to access storage goes via CVM01 till CVM01 either fails or ADS intervenes whichever occurs first.

When CVM Fails

imageCVM01 goes down. TCP session is lost. Since CVM01 is unreachable disk timeout errors will be observed inside the Guest OS until new iSCSI session is established (less 20 seconds). Login request is sent to External Data Services IP Address

imageThis time External Data Services IP Address redirects the login request to CVM02

Acropolis Block Services Failure Scenario
Acropolis Block Services CVM Failure Scenario

imageCVM02 acknowledge the request and responds with Target VG1 upon successful login.

Path failure (1 to 3 tasks) are executed in less than 20 seconds which is well within 60 seconds.

If VMs are sharing vDisks, then both the VMs are directed to preferred CVM.Automatic failback is configured e.g. If CVM goes down and comes back, the path will fail back to preferred CVM. When iSCSI Target is shared between VMs especially for configuring WSFC, both the nodes of the cluster are redirected to Preferred CVM.


External data services IP address must be on the same subnet as CVM IP address to avoid any delays in path failover. To clarify no routing should occur between iSCSI Initiator and iSCSI Target. I think it is better illustrated in figures above.

Since iSCSI Initiator and iSCSI Target can establish only single iSCSI connection per target (as there is single External Data Services IP Address), Nutanix strongly recommends configuring NIC teaming (Bonding) especially for iSCSI initiators on the Physical server.

Receive Side Scaling (RSS) recommendations are no different than previously recommended by VMware and Microsoft. I will state here for the sake of completing the post

  1. VMware ESXi – VMXNET3 driver must be installed on VM to leverage RSS.
  2. Hyper-V enable VMQ to take advantage of RSS

Likewise, Jumbo frames recommendation remains unchanged. If you wish to enable Jumbo Frames enable it end to end right from VM – CVM – Virtual Switch – Physical Switch – Physical Server

Lastly, at least configure one-way CHAP at a minimum.

Acropolis Dynamic Scheduler [ADS]–AOS 5.0

In this post, My focus is on features released as part of AOS 5.0 more specifically on AHV. It must be remembered that AHV had released two years ago since then it is quickly gaining the feature parity with other hypervisors.  Although AHV are on KVM yet it stands apart from other KVM version when it comes ease, support, Reliability, and performance. The Focus of AHV it to make it real for Enterprise cloud. I believe Nutanix is changing the game by completely rephrasing private cloud term into Enterprise cloud by providing AWS kind of ease, flexibility, speed and performance inside your Datacenter. In the light of upcoming AOS 5.0 release, AHV has been supplemented with various features, Acropolis Dynamic Scheduler (ADS) is one among them. It can be argued ADS feature is similar to vSphere DRS, my stand on it is little different. Regardless of whether they are same feature or function Nutanix focus is on solving contention issue instead of load balancing. The vision for ADS is simple “Resources are fully consumed without compromising end user performance.” Being a Pioneer in HCI, Nutanix is at the tremendous advantage in providing QoS service to VMs as they can measure contention at Compute and Storage level without depending on 3rd party vendor/tools/injecting drivers.

Highlights of Acropolis Dynamic Scheduler

  1. ADS enabled by default. It makes total sense to me as Nutanix cluster need minimum three nodes to function. I cannot image a use case where you might have to disable ADC permanently.
  2. Initial placement i.e. VM is powered ON on the host which has less CPU and Storage Hotspots. It was there since early days.
  3. ADS keeps checking for hotspots every 15 minutes.
  4. The following Data is collected every 10 minutes for Historical utilization which forms the basis to make an intelligent placement decision. This data is referred as RunTime Metrics
    1. CPU Utilization of Host,
    2. CPU Utilization of VM
    3. CPU Utilization of Stargate
    4. CPU Utilization of vDisk threads

The data collected every 10 minutes (stats) are maintained by Arithmos in Prism and stored in NoSQL Cassandra.


85% CPU Utilization threshold is configured per CVM (Stargate) and Node level. If this limit is exceeded on either of the fronts, then VMs on that particular node will be live migrated to other hosts. And in scenarios where Storage bottleneck is observed, Acropolis Block Services (ABS) will be migrated to different hosts. I understand this is a very simple explanation.

Affinity Rules

By and Large Affinity rules are required If you would like to separate VMs on the different host or colocate the VMs on the same host. These separation and co-location are needed to meet the licensing requirement or to make sure High availability is maintained at application layer even if the host goes down. On the one hand, there are some applications which perform at highest efficiency if they are on the same host (Affinity Rule), on the contrary, there are some applications which are deployed in redundant form to protect against the Virtualization host failure (Anti-Affinity Rule). Two types of Affinity rules are available.

  1. VM-Host Affinity Rule (Must Rule)
  2. VM-VM Anti-Affinity Rule (Should Rule)

VM-Host Affinity Rule is deployed to contain VM to specific host or group of hosts. VM-Host restriction is achieved using Must rule. Must rule is not violated under any circumstances or put it another way, “Must Rule” will always be respected. VM-VM Anti-Affinity Rule is different from VM-Host as it is just taken as only should rule. The VM-VM rule is deployed to restrict VMs on the different group of hosts. The should rule therefore is only “Best effort” rule. If should rule cannot be maintained Alert will be generated. Alerts are stored in Alert DB

Below is the high-level overview of how Acropolis Dynamic scheduler work. The figure below depicts the Anomalies which are scanned every 15 minutes, in the event any of anomalies are detected, the scheduler gets into action with appropriate remediation path shown below.

Acropolis Dynamic Scheduler High Level Overview
Acropolis Dynamic Scheduler High-Level Overview

In summary, ADS is the best value add to AHV which is aimed at fully utilizing resource without compromising end user performance. ADS is enabled by default and there no configuration parameters required to tune the ADS. In my opinion, ADS is deployment is transparent (invisible) At the same, there is more extensive and detailed blog posts are published by Andre Leibovici at  on AOS 5.0.

One more thing, you don’t need worry about Enhanced vMotion Compatibility (EVC), it is just taken care.


Nutanix Acropolis Block Services – PART01

First and Foremost, Acropolis Block Services (ABS) is not new, in fact, it was always there in the form of iSCSI based block storage presented for WSFC since AOS 4.5 release. As a matter of fact, the newly released Acropolis Block Services (ABS) has more manageability and better performance. As a result, it is recommended by Nutanix to use Acropolis Block Service going forward. If you are using MPIO right now, it won’t affect your current configuration post upgrade to AOS4.7, but Nutanix recommends you start planning the conversion of Existing Volume groups and redirect iSCSI initiators to use ABS.

While Acropolis Block Services will perform a role of iSCSI Target, the iSCSI initiator will be either a Non-Nutanix cluster, VMs on Nutanix cluster or outside the cluster. Finally, it is a Block Storage carved out of volume groups which act as iSCSI Target.

The most noteworthy point is that iSCSI initiator discovers iSCSI Target using single IP address published by ABS. As a result, client management is simplified, and above all, it obviates load balancing of iSCSI sessions. iSCSI throughput becomes now a responsibility of Acropolis Block Services.

Few things to note about Acropolis Block Services

  • To clarify ABS was not designed to present LUNs to ESXi subsequently unsupported
  • Not supported on NX-2000,2050 and NX-3000
  • unsupported on Synchronous replication or metro availability

Use cases of Acropolis Block Services

  • Oracle database continues to be unsupported all hypervisor except Oracle’s hypervisor. Notwithstanding this fact, Acropolis Block Services decides to address this constraint by keeping Oracle database on bare metal while Application and web tier stay to run on hypervisors as VMs.
Acropolis Block Services
Acropolis Block Services

You ordinarily do not use physical servers for test and development rather use VM for efficient use of resources. Now with ABS, Test and development cycle can be optimized by using Nutanix Cloning feature. As a first step, clone volume groups presented to Oracle database (on the physical server), post cloning you can present the cloned volume group to Test and Development VMs without impacting production. Thus Developers can take a live copy of databases in parallel, develop applications based on active database copy without having to review the compatibility issues.

In the real world what happens is that your storage is the end of life and earlier just happen to buy a brand new servers. Although you make a smart decision of buying Nutanix to replace of EOS storage array at the same, you can just ignore brand new servers.  Specifically for such situation, ABS can be used to present the Volume groups to these physical servers. Later, over a period (generally three years) you can move these applications as is to Nutanix. You simply have to rebuilt to OS, install Application after which you re-represent the volume groups which you earlier presented to the physical servers to the new built VMs.

Supported Operating System on Acropolis Block Services
Operating System Supported
Windows 2012 R2
Windows 2008 R2
Oracle Linux 6.7
Oracle Linux 7.2

Nutanix has went further and tested the performance and scalability of Oracle MS SQL database and MS Exchange on acropolis block services

What is Happening and has happened -11Dec

Last week I started a new blog series on weekly update around UAE/World. Though the title was ok I felt most appropriate title would be what has happened and what will happen. While it means more or less same yet I want to make the title more simple. “Happening” is the one which is scheduled to happen in an upcoming week and “has happened” is the one I missed to include in this post. And, let me point out I’m not biased against any technology if you wish to cover any of events you’re more than welcome. Continue reading What is Happening and has happened -11Dec

Why a HomeLab need a Shared Storage and How to choose one

Do you have a home lab to practice virtualization technologies? Then  you’re going to need centralized storage to make the full use of all the available features. The most optimal way to enable this is to use a storage system which can virtualize itself. Several Virtual Storage Appliances (VSA) are available and it can be tricky to decide  which is the best to use.  Neil Anderson ( ) has published a list of all the available VSA’s including their system requirements and links to setup guides for each one. There’s also recommendations on which VSA is most suitable for you given your needs.

Migrate from any hypervisor to any hypervisor using vembu

We can use Vembu to migrate VMs to any hypervisor. Such tools are available in the market, but what is unique about Vembu it offers this feature as a part of the backup suite. I can backup VM in VMDK (VMWare) format and restore it to hyper-v format. This Feature can of great use if you are using different hypervisor in DR than in production. Let me describe the use case.

Use Case: Migrate from VMware to Hyper-V, VMware to KVM, Hyper-V to VMware, Hyper-V to KVM, KVM to Hyper-V, KVM to VMware  (Any to Any Hypervisor) Continue reading Migrate from any hypervisor to any hypervisor using vembu

Distrupting Datacenter