Tag Archives: troubleshooting

Capture Windows VM memory dump in ESX

I’m working with Microsoft to identify a problem I’m seeing with LSASS, possibly related to the VSS snapshot created by our backup software. At this point, I need to be able to capture the memory state on the system, even if I can’t log into the box.

There are several ways to trigger a crash in order to collect a memory dump, but this system is a guest running in VMWare VSphere (ESX4). I asked VMWare support, and they pointed me to KB article 1009187, Generating a Windows core dump from an ESX virtual machine.

I configured my test system guest to crash and collect a memory dump on an NMI event, then used the vmdumper command to send the NMI to the guest.

It worked like a champ:

vm-crash-nmi

I verified the integrity of the dump file with dumpchk. It looks good. I’m setting the same thing up on my production guest.

Domain Controller Blue Screen

CDC02-STOP During an upgrade of our VMware ESX infrastructure, I ran into an issue with our domain controllers. As part of the process we needed to upgrade the virtual hardware that is part of the guest vm. After updating the domain controller guest’s VMware Tools software, I shut down the guest and select Upgrade virtual hardware. Things looked good, so then I powered on the guest and it started to boot. After a moment, it threw a quick bluescreen and auto-rebooted.

My first order of business was to change the auto-reboot option so I could actually see the error message. I know now that I could just hit F8 during the boot process. However, I did it the hard way.

 

I booted to the Windows 2008 install media, and ran regedit from the recovery environment. I loaded the HKLM hive, and set the HKLMSYSTEMCurrentControlSetControlCrashControlAutoReboot DWORD value to 0 (zero). I see in the KB 307973 article that there’s also a wmic command that will set the option. I don’t know if wmic is available in the WinPE recovery mode.

Continue reading

Is that program running as administrator?

Using Process Explorer to view process integrity levels

A friend asked me how to open a Control Panel applet As Administrator. In Windows Vista, when you see a little shield icon as part of a button or shortcut, that would indicate that you would get prompted by the User Account Control (UAC) facility to elevate the process Integrity Level, that is, to run it as an administrator with full rights to muck with the system.

In Windows 7, the frequency of UAC prompts has been reduced. You will still see the shield icon, but sometimes there’s no UAC prompt.

You can use Microsoft SysInternals Process Explorer tool to view the integrity levels of running processes. On campus, you can run the tool from \filessoftwareutilitiessysinternalsprocexp.exe. Once you’ve started Process Explorer, there are two things you’ll want to do:

  1. From the File menu, select the Show Details for All Processes option (you noted the shield icon, yes?).
  2. From the View menu, choose Select Columns… and check Integrity Level item (on the Process Image tab; see below)

procexp-show-integrity

  Continue reading

re-enabling ESET NOD32

ESET has fixed the problem that caused widespread system hangs. If you followed my instructions to disable NOD32, you can re-enable it by repeating those steps and changing one word: replace disabled with auto.

To recapitulate:

1. Boot into safe mode

2. In either the Run dialog or the Vista Start Menu search box, type the following:

cmd /k "sc config ekrn start= auto"

(Please note that the space after start= is required; goodness knows why…)

start-run-enable

start-box-enable

3. Watch for the success message, and reboot.

ESET NOD32 making many systems hang

I’ve spent most of the day trying identify a systematic way to work around the campus antivirus solution, which is causing widespread system hangs. Our vendor has tentatively identified a problematic recent update, and is recommending that affected users temporarily disable the Eset Service service until a patch is available.

Disabling ESET NOD32 / ekrn Service.

If your system become unresponsive, in most cases soon after logging into the system, you may be affected. Please follow these instructions to disable the ESET service:

1. Restart your system in safe mode

2. In either the Run command ( Start->Run or [Windows Key]+R)

start-run

 

OR in the Vista Start menu search box…

start-box

 

3. …Enter the command below

cmd /k "sc config ekrn start= disabled"

(Please note that the space after start= is required; goodness knows why…)

 

4. Watch for the success message:

sc-success

 

Reboot and stay tuned to your friendly neighborhood technical support resources for updates.

PS. for what it’s worth, here’s my current ESET version info, which hangs my system.

eset-about

List folder contents – XP vs. Vista

Yesterday, a client called me complaining that, after installing Vista SP2, she couldn’t access a folder on a file share. She could access that same folder from her XP workstation, logged in with the same account.

I paid a service call (across the parking lot; any excuse to get up and walk outside 🙂 ), and after some poking around confirmed her claim. We did determine that she might not have attempted to access that folder from her new Vista system before.

So I started digging deeper. The folder granted her (via a group)  the “List Folder/Read data” permission. So I created a test folder and granted an analogous group this specific permission to the folder. This is displayed in the output of icacls thas “(S,RD)”.

C:>icacls s:citZTest
s:citZTest CAMPUSETS-FileServices-Browse:(S,RD)
             BUILTINAdministrators:(OI)(CI)(F)

This permission alone allows Windows XP workstations to browse the folder, but Windows Vista or later give an “Access in denied” error.

When creating a “browse” permission for a single folder, I start by granting the “List Folder Contents” standard permission, which assigns the following permissions to the folder and subfolders (not to files):

  • Traverse folder/execute file
  • List folder/read data
  • Read attributes
  • Read extended attributes
  • Read permissions

With icacls, this permission looks like this:

C:>icacls s:citZTest
s:citZTest BUILTINAdministrators:(OI)(CI)(F)
             CAMPUSETS-FileServices-Browse:(CI)(RX)

The (CI) indicates “Container inherit,” which means that permission (ACE) will be inherited by subfolders. Now I open the advenced security dialog, and edit the ACE to change the “Apply to” control to “This folder only.” Now the browse permission applies only to the particular folder. In icacls, it looks like this:

C:>icacls s:citZTest
s:citZTest BUILTINAdministrators:(OI)(CI)(F)
             CAMPUSETS-FileServices-Browse:(RX)

I changed the permissions on the client’s folder, and her access was restored.

See also:

Changing Boot drive with BCDBoot

Scott Hanselman is a consistently good source of useful info and commentary. Recently, he needed to change which drive his computer used as its System drive, which is to say the drive containing the boot loader and configuration.

( N.B. For some reason, the “System Drive” contains the boot info, and the “Boot Drive” contains the operating system. Why could this not have been corrected?!)

Scott points out his options:

Approach 1: Nuclear Option. Wipe and Start Over.

Approach 2: Copy the Hidden/System Boot Manager and Boot Folder over to the C: drive and run a tool called BCDEdit to move things around in 12 short steps. 😉

This was a scary prospect for me, because from my point of view, while this was a fairly advanced operation, I just wanted to switch where the boot info comes from.

Turns out there is a new (profoundly advanced, you have been warned) command line tool called BCDBoot.

See Scott’s blog post for more info. /me wonders if one could copy the bcdboot executable to a Vista system and perform the same operation.

Tuesday – May 5

Spurred by some recent traffic on the Windows-HiEd list, I have looked into the Windows Update process on some of our Server 2008 Core systems. The thread was specifically with regard to KB article 953631, and that some folks have found that it installs repeatedly on Server Core instances and blocks other updates.

In examining the event logs on a couple of our Server Core system, I found that the update is indeed re-installing repeatedly, but it doesn’t appear to be blocking other updates.

First, I ran the systeminfo command to display the installed updates. KB953631 was not listed. I grabbed the WUA_SearchDownloadInstall.vbs script from Microsoft (I renamed it to Get-WindowsUpdates.vbs, in keeping with the sound PowerShell naming conventions). When I ran the script, it found and downloaded two updates, the KB953631 update in question, and KB955430. I confirmed that I wanted the updates installed, and the first update installed successfully, but the second failed (my initial searches didn’t explain the 0x800f082f error code). I reproduced the same behavior on another server core instance.

I tried rebooting the host, and running the Get-WindowsUpdates.vbs script again, and this time both updates installed successfully. (yes, the KB953631 update installed again). I reproduced this success on the other host as well.

So it appears that in our environment, the KB953631 update isn’t blocking other updates. I’ll confirm this after Patch Tuesday.

At the very end of the KB article is the following:

Note for WSUS administrators
If you approve this update for deployment in a WSUS environment, be aware that after you run the update, it will not be reported as "Installed." The update itself is not installed on client computers. The update scans for missing files and replaces them as appropriate. If a computer requires a missing file, the 953631 update will be reported as "Needed.”

Also, Server Core is not mentioned specifically in the list of affected operating systems. It might be worth asking what the expected correct behavior should be in this situation.

In my investigating, I also found an article in the Scripting Center the describes a PowerShell approach to manipulating Windows Updates. This might be nice when Server 2008 R2 is availabel and .NET and PowerShell are included, or other update-wrangling tasks.

Wednesday – April 22

I’ve been working on revising and refactoring a Perl application that I wrote about four years ago to handle our domain account provisioning. Originally, it was a monolithic application, running on ActiveState Perl. Now it needs to run on a Windows Server 2008 x64 host. I use a couple of additional modules that are available from the excellent repository at UWinnipeg that include some compiled code. Rather than run the Perl64 version, and then having to compile my own DLLs, I decided to just install the 32-bit version of Perl, and continue using the modules.

The application is feature-complete, I believe, and is ready to be tried in production. When I attempted to run it under a service account, though, I encountered an error that I hadn’t received running it under my working account. I could repro the error with a simple one-liner:

C:Perlbin>perl -MNet::SSLeay
Can’t load ‘C:/Perl/site/lib/auto/Net/SSLeay/SSLeay.dll’ for module Net::SSLeay:
load_file:Access is denied at C:/Perl/lib/DynaLoader.pm line 202.
at – line 0
Compilation failed in require.
BEGIN failed–compilation aborted.

I checked my PATH, and verified rights to the file indicated. Things were in order and I was stumped. Some google searches turned up advice to check my PATH variable and confirm permissions. OK.

I used Process Monitor from SysInternals and filtered on the perl command line. Toward the end I found a couple lines indicating ACCESS DENIED to C:Perlbinlibeay32.dll.

procmon-perl-libeay32

Now this is not the file that was mentioned in the error, but I checked this one, and the SSLeay.dll that was there, too, and wouldn’t you know? They had different ACLs than the rest of the files. Perhaps the ppm installer didn’t assign the rights when it installed them? Whatever. I granted the service account appropriate access and that fixed the problem.

Huzzah!

Recalcitrant Vista SP1 install

I have spent a number of hours troubleshooting the installation of Windows Vista Service Pack 1 on a particular Dell Optiplex 755, but I finally succeeded.

Symptoms: The SP1 update was downloaded and queued for installation via the Automatic Updates engine. When initiated, the install would look like it had completed (going to 100% through the third configuration step). But upon reboot, the installation would rollback all changes.

Troubleshooting: I did some of the normal troubleshooting steps, including downloading the full installer, disabling AV, etc. Each attempt to install the service pack was time consuming, but ended the same way. Eventually, I tried running the system file checker,but that didn’t change anything.

SP1 installation support: Microsoft offers free support for SP1 installation, so I decided to give that a try. The first suggestion was that I try the installation in a clean boot environment, created using MSCONFIG:

  1. Click "Start", type: MSCONFIG in the search box and press Enter.
    Note: Please click "Continue" if the "User Account Control" window pops up.
  2. Click "Services", check the "Hide All Microsoft Services" box and click "Disable All" (if it is not gray).
  3. Click "Startup", click "Disable All", click "OK" and restart your computer.

I made the changes and attempted the install again, but without success. I bundled up some log info and sent it off to my support technician. Her next request was that I repair Vista, using installation media of the same Vista version that was currently running on the system, by running setup and selecting the upgrade option.

I ran into some issues with unreadable media and had to perform a firmware update on my DVD drive, but I got that working. The upgrade/repair took hours on a relatively powerful system. It did complete, however, and then I was able to run the SP1 install from the version I had downloaded.

Since this was a very significant system update, it would be prudent to back up any important data before performing this procedure on any system. All my software and data appears to have been preserved, but this is a lightly used system, and my important data is on my network account.