Adventures in Babysitting - Hypervisors -- A Desktop Virtualization Journey

ADVENTURES IN VIRTUALIZATION TECHNOLOGY 

January 8th, 2022

STEP1 - What is this Virtualization stuff all about? 2013/2014

We were sitting down in Florida in 2013/2014 and when we arrived we were blessed with an HP AMD Server which was at the time about 5-10 years old. But it was designed as a server and had an advanced BIOS which supported AMD's MP platform. It was running windows 7 ultimate which is an incredible operating system. It had a 500GB drive, which was sufficient to repartition, and play with a bit, but other than we were bored we had no reason to touch the machine. It ran MS-OFFICE and played on the internet, and the BIOS actually had VMX disabled, or at least the AMD version of VT architecture disabled.

We wanted to look at server 2012 so we shrunk the C, partition, dropped in new volume, and installed Windows Server 2012 on the new partition. And since Linux was the craze, we decided to see what it looked like, so we installed VMWARE Desktop, which applies these strange kernel patches we had never seen before, and we get an error that virtualization was not enabled which would restrict its capabilities. So we rebooted, hit F10, and enabled virtualization and the error went away. Soon without delay we had installed three different versions of linux (about 20G each), and were experimenting with Windows 10, everything worked. As HPE is a great company. And VMWare is a great company. And it just so happened that every we choose was mainstream class A stuff across the board. And the AMD server was much (and I do mean much) faster than Intel equivalents running the same OS. We couldnt believe it.

We had an AMAZING EXPERIENCE and within a year, we understood the market, we thought. By August of 2014(now 8 years ago), we had a working platform... just one problem, we only had two cores so we knew we couldnt scale. Additionally HyperV on this same platform was cool for testing Android Phones we didnt own, but it barely even worked in 2013. It was horrible even as the drivers HyperV were using were mostly proprietary tools which only worked with Windows VM's. (Yes we tried HyperV on Server 2012) However fortunately we installed HyperV after VMWARE, and we knew right away VMWARE was better and didnt break Server 2012, like HyperV did. Back then you couldnt install HyperV on a domain controller, etc. While VMWARE didnt care.

What would we say... well VMWARE and HPE were production ready for virtualization in 2013. Had we had a quad core server, we might have gotten 4 VM server running simultaneously, but we didnt.... and new they were going for on the order of 5K dollars or more. Furthermore veterans of HPE will tell you, its usually not the system board and memory which kills you... its the array controller and the drives. A Fully loaded HPE Server with an array controller and 1TB of disk in 2015 and 4 cores was about 20K dollars. There must be a cheaper way?

We were in the latest steps of our LAB, and we created a third partition for SUSE Linux as we had heard IBM was porting it to the Z, and could run 10K cores on it now. SUSE Linux has a ZEN Hypervisor you can install as Type2 solution, and it works pretty well, but not really. KVM has a horrible provisioning system in comparison to VMWARE and we stopped nearly right away. But we did look at VMWARE for Linux Desktop, and installed that right away with a similar beautiful result.

Summary

After 18 months we ended our process with the Result HPE + AMD + VMWARE was a winner both at the Linux Level, and the Windows platform. We became so confident we even installed VMWARE desktop on Windows 7 Ultimate (our working platform), and then our laptop. Conclusions... its nice but is it practical? Its not good enough for massive scale computing, datacenters, or even LAN rooms but sure is nice for the IT Administrator to get used to different versions of Linux,  or even start developing apps across domain.

BUT....

We did some research, and understood that VMWARE was focusing on a new server platform ESXi, and that Intel was working on Multicore platforms which would scale well beyond 4 cores.... but in 2014 to be honest they werent practical due to expense or a solution seeking a problem. See 1RU servers dropped to 2K dollars, and rolling out 10 of them is still 20K dollars...why buy one and kill yourself operationally spending the same money... why nearly destroying your processes, and hardware segmentation.

We have worked for many fortune 500 customers, and our opinion thumbs down on trying to scale this technology and beat our existing operating performance and pricing. Furthermore, Dell convinced us of the same by dropping FiberChannel connectors into our 1RU farms, and simply using all our space for disk drives instead of compute fabrics. They also bought brocade to convince us they were right.

We ended the year with a nice science experiment, but it was still not an Enterprise Recommendation we could live with.

STEP2 - SEARHCING FOR QUAD CORE ARCHITECTURES (2014-2018) & CLOUD COMPUTING SANDBOXES

In about January of 2015, after hitting a wall with our experiment, we realized we could run two OS's simultaneously, but only in a desktop model. We couldnt replace a LAN room with a consolidated server without a quad core machine.... we received a Mac Mini in the mail from a friend. It was a quad core with no CDROM... and those which have used MAC's before know that without a CDROM its very hard to do anything with the platform. But thats a long story in itself. A Blessing received as a gift was now a curse as it started to take down our previous complaints... we easily could spin up 4 Unix platforms using the same technology, MACOS + VMWARE fusion. We also tested Parallels, VMWARES competitor with a nearly identical result.

HPE Quad Cores even on Ebay were 3K dollars and we decided the gift we received in the mail was the best we could do. (even still).

So we installed VMWARE Fusion, the MACOS version of VMWARE and it worked beautifully, and had all these new cool features which made our head spin. But VMWARE creates all these weird interface drivers/NICS which are proprietary, and even to do things like DHCP you have to read books, and teach yourself how to hack these very intricate configuration files....much like unix. We ended up getting lost in this technology for almost two more years using MACOS.

By 2018 we had hit the wall with the MacMini only because it didnt support enough memory to run 4 copies of Windows Server(about what is needed to run a BU LAN). See to run a business you need two web servers one internal, and one external, and two domain controllers. In practice Windows DNS is not sufficient to operate on the internet, so really you need at least one linux platform running Bind which is 5.

Now we are good engineers, and we can in fact get 5 machines to run on a quad core with 16GB of RAM but its very hard to do(more than 4 years of work), and at an investment of about 500K dollars in labor. To the average company it still was thousands of times more expensive than simply buying 1RU servers and desktops and even more so to stick with Windows7, as we dont think most of the features of Windows Server were around replacing Cisco RAS technologies for their existing customers. See Microsoft is going to war against Cisco in nearly ever market if you havent noticed.

We ended the year 2017 saying... yes this technology works... its the future.... but we STILL DO NOT RECOMMEND IT TO OUR CUSTOMERS.......

At the same time we bought our first Google Cloud services, after experimenting briefly with Azure which we didnt like as much. We also bought a cloud cPanel service from Web.com which also used a different type of Virtualization which we liked much better.

STEP3 - 2018 - THE QUANTUM LEAP - PUBLIC WEB SERVICES & LARGER MP SERVERS

In 2008/2009 Apple started building high end workstations with 8 cores mainly because their customers complained about running the very intensive MAC GUI on anything less than 4 cores and 8GB of RAM for Sahara and above OS lines. In fact the MacPro's in the 3,1 line have 8 3GHZ cores and were over 20K dollars when first launched. The MacPro was designed to run Windows as a dual boot option as people didnt really like the MAC desktop, and they were open on the Intel architecture to doing windows two. So Apple started to compete with Dell and HPE as just a WINTEL integrator that could do MACOS as well for Mobile.

In 2018 we bought a MACPRO 3,1 which was less than 100.00 on Ebay. People started dumping them because Google, Microsoft, and Amazon started offering cloud services which were a fraction of the cost of Integrating Stuff on MAC Hardware. The problem with the 3,1 is its nearly 100lbs and it costs over 150.00 to shop it across the country. But we knew... that that 3,1 platform was still a 20K Intel top of the line server which even today competes with equivalent Oracle, and HPE top of the line platforms. You dont have to run MACOS on a server so I decided I had no risk.....but I was wrong.

Why? Well the Ancient MACOS bioses believe it or not wouldnt even read Windows EFI install media. You actually had to install MACOS clean, install Bootcamp as a desktop OS and get it working, before upgrading it to Windows Server. And to do all of that, takes an additional 2 years of training, believe me. Maybe 4.

We installed El Capitan clean from an Image we bought on ebay(which we were greatful to buy) on USB stick, reinstalled, installed bootcamp, and Windows Ultimate just like we had in Florida, and used the exact same process to add Windows Server.... just one problem... on MACOS the bios is so different it wasnt as good as the AMD we had on a smaller scale....we did in fact get it working but it took more than 1 year to get it patched and functional, and took a MAC public computing lab to make it work.

However... getting to our target 5 VM's was easy to do in resources... but we started to crash the core Operating system, and doing backups wasnt very easy. We kept our Google Accounts, Added Amazon, and tested a new company focused on Joomla Operations, called Siteground(A Great company by the way).

We have literally tested every version of MACOS on the MacPro up to Catalina extensively. We love El Capitan, but it has a tragic flaw. It doesnt get security updates anymore. High Sahara is very stable, and modern even on laptops.

WE ENDED 2019 WITH A RECOMMENDATION WHICH WAS WORKABLE FOR ENTERPRISES IN MP COMPUTING... A POTENTIAL REPLACEMENT FOR THE MAINFRAME? BUT HOW CAN WE BUILD AN ENTERPRISE ARCHITECTURE AROUND IT, AND CAN WE COMPETE AGAINST GOOGLE AND AMAZON?

But here's the problem, most enterprises wont buy one of anything, only two, and if one fails, you have to move all of the VM's from two production platforms to one... and thats hard to do so you then get to three or four...and they are 25Kx4 in cost or 100K to do it. Plus 1 Man Year of labor or 200K dollars (Or roughly 1/2 of the cost of just 4 years earlier in TCO).... Then an incredible thing happened just as we were about to go out the door with an offering.....

Google dropped its Virtual Machine cost to just $4.00 a month for a VM with 2GB of RAM and 1 core which can burst to 150% of commit, and thats about 1/100th the costs over a lifecycle.

STEP4 - HAVE PUBLIC CLOUD SERVICES SECURITY IMPROVED ENOUGH?

Ok Google...How Do I secure your service.... Can I do it.... Not without IPSEC/IKE... which is two more years of labor. But MS Azure has native connectors which link to Win7/Win10 workstations which replace Cisco AnyConnect VPN technology. Here's the problem... Google charges 50-100/month for windows and their initial loads ran terribly.

We said okay lets spin up a Windows Server on Amazon and its awesome, unbelievable... so wanted to know why...they are only 30/month for a 10G enabled server. Google we hate you/love/ you okay now we are confused. Amazon whats your secret.... they installed LARGE HYPERV platforms with lots of RAM....

Uh-oh Redmond has religion and they intend to take this away from everyone too. Even Apple.

Our Recommendation Changed:

IBM Z for Big Platforms Running SUSE with VMWARE.

MACPRO with HyperV for LAN Deployments.

SUMMARY

VirtualBox, and VMWare have nearly flawless Desktop Virtualization platforms which are sufficient to run a LAN room with 5 VM's. Furthermore, VMWare has mirroring tools which make mirroring to standby, or clustered platforms easier. VirtualBox is a poor mans solution but used in conjunction with traditional LAN Administrators is much better than a 1996 era deployment.

We tested in our labs:

1) 6 Hypervisors - ZEN on SUSE LINUX, VMWARE Fusion, VMWARE Desktop for Linux, VMWARE Desktop for Windows, Parallels for MACOS, and HyperV

2) 6 Flavors of Linux - Suse 11.3+, Ubuntu, Debian, Centos7, RHEL, Mint Linux.

3) 3 Windows Server Platforms (2012, 2016, 2019)

4) Windows 10 Releases 1-6+, Windows 8.1, Windows 10

5) 7 MACOS Versions - Lion, Mountain Lion, Sahara, High Sahara, El Capitan, Mojave, Catalina

6) 3 Full Cloud Computing Platforms - Azure, Google, Amazon

7)  LG Cloud Citrix Thin Cloud Virtual Desktops

8) More than 6 DBMS Platforms - MySQl, Postgres SQL, SQL Server 2012,2016,2019, Oracle 19c+, Oracle 21c.

9) Novell, NT4.0, Windows XP Legacy Technology on VirtualBox

10) Ivanti Enpdoint Manager, Sharepoint Manager.

11) HPE AMD MP Architecture, HPE INTEL MP Architecture, Every Major Apple HW Platform.

12) More than 100 Applications found in Installatron.

MORE THAN 8 YEARS OF CONTINOUS LABRATORY DEVELOPMENT TO GET TO WHERE WE ARE TODAY AND MORE THAN 30K PROFESSIONAL HOURS OF LABOR.

FUTURE CONVERSATIONS?

1) What if you do private networking to a Public Compute platform and dont cross the Internet?

2) What if IBM runs HyperV on a Z Fabric Natively.

3) Whats Oracle doing with Grid which makes us stop and take notice of them?

4) What is Hybrid Compute?