Kicking and Screaming I am Bloggin

Let me share a little story about a dirty little secret I have recently stumbled across.It just goes to confirm what Vlad is trying to convince you to do and why he is blowing up his Dallas NOC.

I just built a new server for one of my client’s in the bay area and I will be installing it right after SMB Nation. Always liking to be prepared I decided to do a trial run at the migration of the old server to the new server. Actually more of a image and restore than a migration. I was not overly concerned about the process, I have done it on other systems. I just wanted a dry run to have the steps fresh in my mind, it has been a couple of months since I did one. I had an image from my last trip out to the Bay Area, so I created some arrays on the new server and partitioned them. I booted up my StorageCraft ShadowProtect CD and low and behold, there were no drives showing up from either array. I hit the load driver link, it said it loaded successfully, but still no drives. I have seen this before, no problem, I will just use legacy mode.  Well same problem, the driver loads but no drives show up.

I contact StorageCraft thinking I have found a bug they should no about. That’s when I am told that it is a know issue with WinPE, well it seems it is known to the imaging vendors, but not to us mere mortals. Seems Microsoft is also well aware of it, but no one seems to be doing anything about it. When MS does their WHQL testing they only test for Windows installation, not WinPE. There is a guy named Bart who has instructions on how to build a different version of PE, he says it is legal, MS says it isn’t and has tried to stop him, I don’t know the current status of the legal battle. I do know that if you follow Bart’s instructions you can build a PE that boot’s with new drivers, so if Bart can do it why can’t MS?Anyways had I have just blindly followed my plan, sand shipped the server and waited until I got there to do the image, I would have been one very unhappy HA, I would have one very unhappy client while I spent days trying to get him back online. As it is now I had time to rethink my process ands try another option. Remember in Jeff’s DR presentation last year in NOLA, where he says it is not enough to have a DR Plan, you need many options? Well this is just another example of that comment. The best DR plan is a clam mind and a bunch of options you are familiar with, because you have tried them under non-stressed conditions.

What is my new plan you ask, well it is something I have done before, stick a mass storage card I know will work with the software I want to use, in the old server before I do the image. In this case it will be the RocketRaid 2320 because I have already proven it will show up when booted to StorageCraft’s WinPE and using the load driver option in ShadowProtect.I tried to save the client a few bucks and use the Intel Embedded Raid, I should have listened to my own advice, always use a plug in card, it makes your arrays portable.

So in my previous post I menetioned a slight problem, I thought I would share my log of the repair process in case you find yourself in a similar situation.

Disable Exchange Service, let’s not lose any mail in the process

Restore C Drive to alternate location on usb drive, just to have these files on hand, we know C: is hosed

Fly to Sunny California

Download update bios & firmware, burn to cd’s

Flash Intel Serverboard bios

Flash Intel Raid controller firmware

Boot to Shadow Protect IT Edition, to use it’s tools
Unfortunatley I had not had a chance to image this system yet.
Btw Storage Craft’s great support bailed me out in this disaster, as I lost my IT edition disk and they were able to provide an ISO first thing Monday morning.

Delete C partition

Create C partition bootable (sorry can’t do bootable)

Boot to sbs cd1 repair console

Bootfix, sucessful

Bootcfg, fails

Verify boot.ini

Copy files from alternate restore location on usb drive

Copy root files from backup to xp machine, remove attributes

Copy to root of server from repair console, ntldr snd ntdetect will still not copy

Boot to IT Edition again, find clean spot to install w2k3

Start gold w2k3 install to W:, not going to be time efficient

Contact kb get download of SBS R2 w/slipstream beta I just tested for them, Thanks Kevin !

Do Time bombed install to G:, no key

Boot to DSRM

Do system state from 6/24/07, pre original corruption

Disconnect Internet Cable

Boot to OS I hope, yippee it boots

Do online exchange restore of 7/27/07 bu, it fails

Do offline restore of the 3 chunks of mdbdata to original locations

Reboot, stores mounted :>)

Breathe

Verify mail via owa locally, we got mail

Connect Internet Cable let mail flow

Send and receive test message, we are talking to the world

Logon to bookkeeping station, ok

Test 4 QB Data Files, one needs to be rebuilt

Rebuild successful.

Logon to Kenny’s station, logon denied

Take off domain

Put on domain

Logon OK profiles intact

Logmein to remote, verify mail flows via Outlook Anywhere

Image Server with SP IT

Set backup to run tonight

Go to hotel for the evening, 11 hours is enough for one day

Logon to marketing station, logon denied

Take off domain

Put on domain

Logon OK profiles intact, CRM will not open, credential problem

Duh moment, I rolled system state back to before CRM rollup 2 was applied, system grumpy

Apply CRM rollup2 and reboot, CRM opens fine now

Logon to sales station, logon denied

Take off domain

Put on domain

Logon OK profiles intact

Logon to process station, logon denied

Take off domain

Put on domain

Logon OK profiles intact

Logon to lab station, Logon OK profiles intact

Logon to shipping station, logon denied

Take off domain

Put on domain

Logon OK profiles intact

Logon to accounting station, Logon OK profiles intact

Need 4 updates on server from wsus allow all 4 and reboot

Delete demo install and gold install used to restore system state

Reconfigure backup to run regularly

Re-enable offsite copy scheduled task

Verify Hazard Waste database is operational

Clear Logs

Grab jpg of drive layout save to C:\

Grab ipconfig /all save to C:\

Image again with Shadow Protect IT

Install Shadow Protect Server

Give Kenny the good news, no data lost everyone back online

Goto Bella Vista tonight to celebrate :>)

Tomorrow back to hot and humid North Carolina

Tonight I did a demo of Storage Craft Shadow Protect IT Edition 3.0 for my IT Pro Group, Triad SBS Group and they asked me to post the steps I followed during the demo. So for all of them and anyone else who may be interested here is what I did to image an SBS 2003 Premium with CRM SBE from a single processor Intel desktop class machine to a dual processor Intel Xeon server. This system had 6 partitions and was about 24 Gb of used disk space. I completed the entire process in a little over an hour enlarging all the partitions in the process. What follows are my notes to myself to do the demo.

1. Make sure DSRM password is known!!
2. Make sure drive lettering is known!!
3. Make sure IP Addressing is know and which nic will be which!!
4. Backup from live SBS (20 min) (did ya notice nothing is installed to the system)
5. Mention issue with vpc and usb drives, recommend VM Ware for testing to avoid usb problems
6. Restore 6 partitions and run HIR (20 min) (don’t forget, refresh volumes as you create partitions)
7. Boot DSRM load Intel Inf’s & fix drive lettering on this pass (10 min)
8. Make sure nics are connected to active hub/switch
9. Boot Safe w/network (10 min) (will need to wait for network timeout)
10. Check if nic drivers loaded (load if necessary) configure IP addresses and order (5 min)
11. Final Boot (5 min)
12. Rerun CEICW, check logs (5 min)