Nested VMWare ESXi

Virtual datacenter

My physical ESXi box, a lowly AMD FX 8320

One of the more cooler features of VMWare 5 is that it is capable of running itself.  While it

may seem silly at first, this is a great way to build a virtual environment where you can test scripts for provisioning (and destroying) stuff, along with testing API level calls without having to worry about screwing up production, begging for a ‘lab’ environment, or even better snapshotting the whole thing so you can revert whatever it is you are doing at a block level.  In short virtual datacenters really rock, especially for people like me who like to play in a really destructive manner.

So the first thing is that for this test I’m going to use a ‘freebie’ ESXi that I’ve been using for quite some time.  Naturally this should work for version 6, but since the stuff that I’m testing is all in 5.0 and 5.5 (the majority being 5.5) I wan to build a 5.5 environment.  In my insane scenario I have deployment access to a 2003 R2 x64 server with .net 2.0, and the majority of the environment I care about is VMWare ESX 5.5 along with vCenter. 5.5 on Windows 2008 servers.  So trying to mirror this a bit, that means that I’m going to skip the new fangled appliance, I don’t know if it really matters for what I want, but for the sake of trying to keep things the same I’m going to match what I can.

Creating the VMWare ESXi VM

I setup a generic ‘Other 2.6.x Linux (64-bit)’ VM, with a single socket, dual core processor, 4GB of RAM, a LSI SAS controller, with a 32GB disk, and 3 NIC’s with the VMXNET3 driver.  For some reason I had to change the CPU to explicitly allow for hardware MMU emulation.

Hardware MMU emulation

It’s also worth mentioning that when you are going to run ESXi on ESXi that any physical adapters that you want your virtual ESXi host to be able to lauch VMs and have them communicate onto you will need to enable the ‘Forged Transmits’ setting in the vSwitch properties.

vSwitch properties

With that in place, I was able to use my ESXi 5.5 ISO, and install into the VM.  There isn’t much to really say the installer will install ESXi, and reboot and you get the console.

ESXi running on ESXi

I went ahead and gave this ESXi server a static address, and that was pretty much it for the server.

Installing vCenter

I went ahead and used an eval copy of 2008 R2 that is available at Microsoft here.  The installation is really simple, not much to say but for my needs I gave the VM a static address, HOWEVER I did NOT install Active Directory as you cannot install vCenter onto a DC, and I really was not in the mood to have a DC along with a vCenter 2008 VMs as I’m only interested in doing API testing I don’t care about authentication plugins I’m only concerned with other aspects.  Obviously if you do care, then you’ll want to install 2 2008 servers.  I gave my 2008 server a static address of 192.168.1.12 so that I could easily find it on the network post-install.

Since I have no imagination, I renamed the server “vcenter” which of course will factor into the login credentials later on.

My ‘client’ gave me the ISO for vCenter “VMware-VIMSetup-all-5.5.0-3254792-20151201-update03.iso” which of course needs to be installed onto the 2008 server.

install vSphere vCenter

I did the easy install, as again I’m not building a real enterprise.  However I should point out that the easy install has a habit of popping alerts and prompts UNDER the current window so you could be waiting for quite a long long time for this to install when it really shouldn’t take all that long.  I also kept the option for the 2008 Express edition as I’m not going to try to stress the count of VMs 50 is far more than I require as again I’m only interested in limited stuff at this point.

vCenter ‘simple install’ ports

Again I just selected the defaults for the install as I’m not all that worried.  Phew with all of that done, it’s now ready to use!

Logging in

Now I went ahead and hit the following site:

https://192.168.1.12:9443/vsphere-client/#

I went ahead and installed the pluggin for a more ‘full’ experience with the web client.  The link is on the bottom of the page.

download this!

With all of that in place, I finally could use the web/flash site to login, using the local administrator account.  However upon logging in the domain was barren, no vcenters no data centers, nothing.

Empty data-center!

Obviously I must have screwed something up!  Even worse using the ‘fat’ C#/J# client (that I still love…) I would simply get this fun error:

You do not have permission to login to the server!

What? I don’t have permission?

A little digging around, and I found out of course, that it is because I don’t have an Active Directory, and that for ‘workgroup’ installs like this, you need to simply login as administrator@machine.local or ‘administrator@vsphere.local’ in my case, using the password that had to be set during the SSO installation.

Login

And now I was prompted to create my data-center, and add in the ESXi server into the virtual data-center.

vCenter is now operational

So now I have a virtual vCenter, along with an ESXi host to deploy stuff onto, and destroy all I want.  Even better VMWare Player & VMWare Fusion can also run ESXi nested, so you can take your virtual data-center with you on a laptop!  You can boot 2008 with vCenter in 2GB of RAM, although it really should have 4GB if not more, and with 4GB for an ESXi server that would be 8GB+ on any laptop or desktop.  However I’m lucky my ESXi server has 32GB of RAM, and my laptop has 16.  So it’s also a great excuse to upgrade!

End of the line for VMware Player, Workstation and Fusion?

It looks like in the wake of a declining stock price EMC/VMware is already laying off divisions, to ‘cut costs’ and I just received word from a friend that the “Hosted UI” group responsible for all these great products, and the former VMware Server/GSX products were all let go.

A Tribute to VMware Workstation, Fusion, and Hosted UI

Which to me is kind of crazy as this eliminates the only desktop product that could run VMware ESX on the desk for building virtual clusters.  I further guess it means that for what I like to do, I’ll eventually have to find one of those super expensive video cards that works with ESX to passthrough.  Or just drop any and all VMware stuff, and head straight into KVM territory and just get used to OpenStack being a fragmented disaster.

In addition they also closed the Burlington tech support centre.

Oh well, nothing lasts forever.

Getting dot1q to work between VMware and GNS3

So I had this fun episode where I was using Qemu to emulate an ASA, and it worked OK but it was incredibly slow, and I couldn’t put in multiple gigabytes of RAM.  So I thought I’d just dump Qemu and load it up on VMWare.

Well simple ethernet connections work just fine, but the dot1q interface (as this setup has about 50 different connections) doesn’t work at all.

The closest thing I could find was this interesting post, which states:

As I have attached previously there are 802.1q packets leaving the GNS emulated 7200 router but they are not being interpreted by the HOST-ONLY Adapter that is installed with workstation 11 nor does the HOST-ONLY adapter then TAG the l2 frames with the 802.1q ID.

So the host only adapters that I’m creating to give VMWare interfaces that GNS3 can latch onto, strip dot1q!

Well this is no good!

So I thought I’d try the older standby solution, which is the MS Loopback adapter, and try it that way.

Adding the adapter wasn’t too hard in 10, but they renamed it to the KM-TEST Loopback Adapter for some reason.  Anyways with the adapter installed, I removed all the bindings other than the VMware Bridge Protocol.

bindings for the loopback

bindings for the loopback

With that done, the next thing to do was run vmnetcfg, and bind the tunnel interface to a VMnet interface but not in the Host-only connection but bridged directly to the loopback adapter.

vmnetcfg

vmnetcfg

Now with the VMware part configured, it’s a matter of configuring a Cloud object in GNS3, and binding it to the loopback adapter, which in my case has the great name of ‘Ethernet 2’.

vmnetcfg

GNS3 bindings

From there I just attach the cloud to a dot1q ‘trunk’ interface on a GNS3 virtual ethernet switch.

With this proverbial house of cards built up, I can fire-up another VMware machine, in this case a Windows 2000 computer that is bound to a ‘normal’ VMnet adapter, with no fancy dot1a and..

It works!

It works!

I can get IE6 and all it’s glory on the internet.

Update to Windows 10

x

Get ready!

I know I’m crazy, but for some reason the update didn’t kick off automagically on my 7 box, so I fished around and found the direct download here.

From what I’ve read VMware Player 7 updates should work with 10.

Time to see what breaks, and what works!

The first issue I had is that after the upgrade, VMware Player couldn’t connect to the bridge adapters.  Luckily the fix is really easy.

Bring up your network connections, go to your physical Ethernet adapter, bring up it’s properties, and add in a ‘service’.

add

Restoring the VMware Bridge service

Then select the VMware Inc, vendor and the VMware Bridge Protocol.  Now with that done, all I had to do is then bind the bridge to the Ethernet adapter.

x

Configuring the Bridged virtual interface

And now my VM’s can talk to my network without any of that NAT nonsense.  And I didn’t have to re-install VMware Player to fix this either!

Some real fun came from upgrading my wife to 2015.  She uses Outlook 2013 to talk to an IMAP server.  No big deal right?  Well after upgrading when she tried to send an email she would get the ever so helpful error 0x800CCC13 .  So her server is setup to use SSL to talk to the outbound SMTP server.  It even has a valid certificate!  The best part is that verifying her account and IT WILL SEND THE TEST EMAIL.  Yes, that is right, Outlook 2013 cannot send to SMTP servers, but the test and diagnostics work.  And in the age of multigigabyte installations all the user is left with is a hexidecimal error code of 0x800CCC13.  Frankly this is totally inexcusable in 2015, let alone in the 1990s.  Hell even OS/2 had a system to look up cryptic error messages.  I guess that was an IBM thing.

So anyways, the best part is the ‘fix’.  Apparently according to here, the upgrade to Windows 10 corrupts some DLL’s that are a part of Outlook 2013, and they need to be repaired.  Simply run the following command as administrator:

sfc /scannow

It can take upwards of 10 minutes to complete.  After we ran this, we re-ran Outlook 2013, and all of our dozens of attempted test messages sent.

Another possible problem is that the Exchange server pluggin is interfering with the IMAP/SMTP plugins, and it needs to be disabled/deleted.  I haven’t had to go there since she can send emails now.

Nothing worse than a firewall crash

So for my email setup I use an OpenBSD firewall behind a hardware firewall (provided by the telecom), and from there I use OpenVPN to connect up to the VPS that in turn forwards email to my Exchange server.

It works great.

Except that the OpenBSD VM just crashed.  And to top it off I had no other way of accessing inwards except for some test machine that luckily was still on, and I had SSH enabled, along with port redirection.

So a few seconds with putty and you can redirect a local port on your computer to connect to a port on the remote network.  Dangerous as hell but, it certainly can save the day! (Yes you can even SSH to a machine, and then OpenVPN to it….)

Checking VMware KB 1012382 details a list of what ports are needed by which versions of their products to do what.

Product Port Protocol Source Target Purpose
ESXi 5.x 443 TCP VI / vSphere Client ESXi/ESX Host VI / vSphere Client to ESXi/ESX Host management connection
ESXi 5.x 902 TCP vSphere Client ESXi 5.x vSphere Client access to virtual machine consoles (MKS)

 

Putty port redirection

Putty port redirection

These are the two ports needed for basic checking in on the status of a standalone ESXi machine. So in this case I can point the VMware fat client to attach to 127.0.0.1, and add in redirects for TCP ports 443 & 902, which let me login, and start a remote console to see how the VMs are doing.

In later versions, you need to use a proper host name.  To set this up edit your %windir%\system32\drivers\etc\hosts file, and make sure you have something like this:

127.0.0.1       localhost esxiloop

And then point the client to esxiloop, and it ought to connect.

Critical flaw with VMware & Windows 10 preview 9879

As much as I’ve been enjoying 10, there is one issue, which is that I use a lot of VMs.  And I didn’t notice this until it was time to run updates on the Windows & Linux VMs.

As they went to reboot the system locked up hard.  Event viewer gave me this…

Event 1001, BugCheck

Event 1001, BugCheck

I tried updating one VM at a time… crash, updated my BIOS for the heck of it, crash.  Downgraded from Player 7 to 6.0.1 and crash. crash crash crash!

So I had to look to the user forums where more people seem to be greiving for their Pentium 3’s with 256MB of ram.  This issue was effects both Workstation & Player, as they have the same core tech. Since I’m cheap this hits Player 6 & 7.  I saw this buried at the bottom of the Workstation 11 release notes (workstation & player have the same core)

Shutting down a virtual machine on a host running Windows 10 Tech Preview can cause a blue screen.

If you have Workstation installed on a host that runs Windows 10 Tech Preview, occasionally when you shut down the guest operating system in a virtual machine, the host computer might restart unexpectedly. In this case, you see the following error code on a blue screen: DPC_WATCHDOG_VIOLATION. This issue can sometimes also occur with power-off and suspend operations.

Workaround: If a newer build of Windows 10 Tech Preview is available, try updating to the newest version.

Fantastic.

Unfortunately, a new full build isn’t expected until after the first of the year.

This will be the last new build of the year for Windows Insiders

So there is no workaround.

A’int that a kick in the head?

One crazy thing I’ve found is that MS-DOS & Novel Netware 3.12 work fine.  You can reboot/turn off/pause them without any issues.  But if you think about installing NT/2000/XP or Linux onto a MS-DOS VM something that it does to the virtual hardware sets it up for the same issue where a reboot or shutdown will cause 10 to lock up.

VMWare Player 7 is out!

First the pro version is still on sale, ending on December 5th

VMware Player 7 pro

VMware Player 7 pro

And for those wondering, where is the regular VMware Player 7? Buried down in the downloads section.. Right HERE!

VMware Player 7

VMware Player 7

Or if you prefer:

First of all support for 32bit hosts is gone.  This build should have better support for Windows 10.  For those who are worried, 7 still supports Windows 9x, NT 4.0. My nested ESXi still works fine, as does my Novel Netware 3.12!

And for Mac users, VMware Fusion 7 is on sale, which is great for OS X users who’ve upgraded to OS X 10.10

What is the difference between Pro and Free? From the FAQ:

  • VMware Player Pro is designed to be used commercially. It is licensed for use by employees, training organizations, contractors and can be transferred to partners or potential customers.
  • VMware Player Pro is enabled to run restricted virtual machines created by VMware Fusion Pro or VMware Workstation

For me, Player 6 removed the ability to easily create VMnet adapters easily, allowing elaborate network configurations with GNS3.  Minus a few UI issues I’m kind of liking Windows 10.  I may switch out my main OS, although I’ll certainly need the newer player.

Anyways, no I don’t get paid for this kind of thing, but I know people who use this stuff won’t want to miss out on a sale.  ESX starter is on sale as well, but only 15% off.

Using IDE hard disks on VMWare ESXi 5.5

I’m sure this applies to other versions as well.

VMWare stores the configs, and files on the UNIX file system, which you have to edit by hand.  You can’t do this in the UI, as VMWare tries to be SCSI only.  But if you manually edit the files, you too can enjoy a virtual IDE disk.

You need to enable SSH access to the ESX server, or do this change on the console.  Configs live in:

/vmfs/volumes/<storage group name>/<virtual machine>/

So for example, mine is:

/vmfs/volumes/datastore1/UnixWare 7.1.1/

First, edit the .vmdk file.  You are looking for the line:

ddb.adapterType = “buslogic”

And you change the buslogic to ide like this:

# Extent description

RW 16777216 VMFS “UnixWare 7.1.1-flat.vmdk”

# The Disk Data Base
#DDB

ddb.adapterType = “ide”
ddb.geometry.cylinders = “1044”
ddb.geometry.heads = “255”
ddb.geometry.sectors = “63”
ddb.longContentID = “8f535bb60df8d73a86c24853fffffff

The next thing is to alter the vmx file.  By default the hard disk will be on scsi0 and you’ll see something like this:

scsi0.present = “TRUE”
scsi0:0.deviceType = “scsi-hardDisk”
scsi0:0.fileName = “UnixWare 7.1.1.vmdk”
scsi0:0.present = “TRUE”

And what we do is rename the scsi0:0 to ide0:0

ide0:0.present = “TRUE”
ide0:0.deviceType = “disk”
ide0:0.fileName = “UnixWare 7.1.1.vmdk”
scsi0:0.present = “TRUE”

And you should be good to go.  The other (much easier) alternative is to try to use the legacy OS/2 profile as it’ll install an IDE disk by default.

IDE hard disk on VMWare ESX

IDE hard disk on VMWare ESX

Yay!

You should check out sanbarrow’s guide to more options you can configure in your VMX file.