A Windows Active Directory test environment is a great thing in itself. It can be used to test new settings, to train in hacking techniques and for research. From time to time it also inspires us to write a Labs article like BloodHound – Sniffing out domain admins. I built the first version of my test environment some time ago on a notebook running VMware Workstation with two domain controllers, two member servers and two clients. I used this environment as the basis for Area41, which will take place on June 15 and 16, 2018, and the Adversary Simulation workshop and have continued developing it. And there began a tale of woe.

The hypervisor

For the Area41 workshop, our IT admin provided me with a server that we had used for internal testing purposes in the past. The machine was running VMware ESXi 6.0, and the server’s uptime was 220 days. I decided to upgrade to vSphere 6.5, installed all updates and then began uploading my existing virtual machines (VM).

Switching the hardware invalidates the Windows Evaluation License for the Windows 10 clients. Because the license renewal with slmgr /rearm no longer worked under Windows 10, I could not continue using the clients in this way. That isn’t a big problem; a client can be easily replaced, since this is for the most part configured using group policies.

The server performance was a greater problem when all VMs were running at the same time. Disk utilization was so high that it was only possible to work in the VM at a snail’s pace. So I went back to the IT admin to scrounge up an SSD.

I installed it and then during the initialization process I began to wonder whether I should format the SSD with VMFS6 or with VMFS5 like the other two disks. After much hesitation, I opted for VMFS6.

I expected a performance boost with the SSD, but I was disappointed. The system was even more sluggish than before. And now working with the VMs had become impossible; they would only respond after long delays and took 15 minutes just to boot. It turned out that VMware had developed a new AHCI driver (vmw_ahci) with vSphere 6.5, resulting in performance problems with certain controllers. Monitoring the disk performance showed that the read and write I/O dropped shortly after starting and then remained at just a few KBps:

The AHCI driver can be disabled with the SSH command esxcli system module set --enabled=false --module=vmw_ahci. Rolling back to the legacy driver and/or updating to build 5969303 appears to have solved the problem for others. Performance remained slow on my system, however. Even trying various settings in the BIOS did not bring about any improvement.

I decided to downgrade to ESXi 6.0. During the downgrade, however, I discovered that ESXi 6.0 cannot read VMFS6-formatted partitions. The tool vmfs-tools does not (yet) support VMFS6 either.

Most of my VMs were, of course, stored on the SSD. Yet because I had still been unable to work with the VMs up until this point due to the performance problems, and my local VMs were therefore still up-to-date, I decided to reformat the SSD. But the performance problem persisted even with ESXi 6.0. At least it did not appear to be caused by the new AHCI driver. After speaking with the IT admin, we decided to use XenServer.

The migration

When migrating to XenServer, VMware VMs can be exported into the OVF format, which can then be imported directly using XenCenter. To my surprise, the export/import process worked fine. When running XenServer, there were no performance problems with the SSD either. It was only the Windows 2000 VM that would no longer start due to a bluescreen during the boot process – but more on that later.

After the migration, I began uninstalling VMware Tools on all VMs and installing the XenServer Tools. That worked on all systems, except for one Windows Server 2016 Core VM. I could not find any way to uninstall VMware Tools. VMware itself recommends uninstalling this software in Programs and Features or running the VMware Tools setup file. There did not appear to be any other option. The Programs and Features option is not available in Windows Server Core, however, and I didn’t want to go through the extra step of copying the VMware Tools setup program to the VM. If anyone knows a simple way to uninstall VMware Tools, any recommendations are welcome.

For the first VMware Tools service, I disabled and started with the XenServer Tools installation, which is available as an MSI package. The installation was blocked by a software policy, however. Because I had configured several hardening settings on this server, I wasn’t particularly surprised. But I still could not find the cause even after several hours of combing through the AppLocker policy, software restrictions and various registry keys. It ultimately turned out that the installation would have worked fine with msiexec /i.

All of the restarts and shuffling back and forth of the VMs had now created an inconsistency in the AD replication of the two domain controllers. There was initially a 60-minute time difference between the secondary DC and the primary DC, but I managed to fix this. After restarting, however, the event log started filling up with Kerberos error messages – a bad sign. Here, too, it took me some time to analyze and try various workarounds before finding a solution to the problem. What finally worked was resetting the machine account of the secondary domain controller, making it possible to replicate the two domain controllers again. Luckily, there were no other problems on the member servers. So the migration was completed successfully, except for the Windows 2000 machine.

The return of Windows 2000

The Windows 2000 machine is one of the challenges planned for the Area41 workshop, so I was interested in getting it up and running again. The bluescreen cropped up during the boot process, because Windows 2000 could not locate the boot partition. The reason was probably that Windows 2000 did not have a driver for the disk controller. A workaround was to run the VM again under VMware Workstation, install XenServer Tools 5.5 and then migrate the VM again. Windows 2000 now started successfully:

A new version of XenServer tools cannot be installed under Windows 2000, however, because it has a .NET Framework dependency. At least there is a guide on how the .NET Framework 3.0 and 3.5 can also be installed under Windows 2000. But now that my VM was running smoothly, I decided to stop experimenting.

Incidentally, I also noticed that Windows Update no longer worked in Windows 2000. This is probably because the requirements for the TLS/SSL configuration on the Windows Update are too strict, preventing the Windows Update client from being able to connect anymore. At least most of the updates for Windows 2000 can still be found on the internet – the links on the Microsoft download page are a dead end, however. For anyone feeling nostalgic and wanting to run Windows 2000 from time to time, I recommend using the Windows 2000 JSLinux VM developed by Fabrice Bellard.

Moving forward

Migrating the AD test environment to XenServer ultimately succeeded, but cost me a lot more time and effort than I had expected. After temporarily changing roles between Red Team and Sysadmin, I once again saw how much expertise, work and “pain” is required to set up an IT infrastructure and keep it up and running. So it’s only right that there is a System Administrator Appreciation Day (last Friday in July)! The setups for the Area41 workshop are now ready, and I will use the remaining time to set up additional systems, prepare vulnerabilities and develop a monitoring scheme together with our Blue Team. See you soon at the Area41!

About the Author

Michael Schneider has been in IT since 2000. Since 2010 he is focused on information security. He is an expert at penetration testing, hardening and the detection of vulnerabilities in operating systems. He is well-known for a variety of tools written in PowerShell to find, exploit, and mitigate weaknesses. (ORCID 0000-0003-0772-9761)

Links

• http://orbisvitae.com/ubbthreads/ubbthreads.php?ubb=showflat&Number=69204
• http://sysadminday.com/
• http://www.nxhut.com/2016/11/fix-slow-disk-performance-vmwahci.html
• https://area41.io/
• https://area41.io/workshops/#2
• https://bellard.org/
• https://bellard.org/jslinux/vm.html?url=https://bellard.org/jslinux/win2k.cfg&mem=192&graphic=1&w=1024&h=768
• https://docs.citrix.com/en-us/xencenter/6-5.html
• https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-7552DAD4-1809-4687-B46E-ED9BB42CE277.html
• https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-esxi-651-release-notes.html
• https://glandium.org/projects/vmfs-tools/
• https://support.microsoft.com/en-gb/help/325850/how-to-use-netdom-exe-to-reset-machine-account-passwords-of-a-windows
• https://www.vmware.com/support/developer/ovf/
• https://xenserver.org/