The Day the G5 Died
Keys Out Of Order

The Day the G5 Died – Keys Out Of Order

There are some mornings when just the simple act of getting out of bed foretells a bad day. It was Thursday morning, and surprisingly, my fifteen-month old daughter actually slept the entire night, so I actually got something close to eight hours of sleep. When I was younger, say in my teens or twenties, I could go days without sleep. Not anymore, I need as much as I can get. My youngest daughter, however, feels differently.

Waking up that morning, I cracked my knee on a drawer I had left open the night before on the dresser. It didn’t hurt too badly, but it is a hell of a thing to do in the first five seconds of wakefulness. Wake up, turn off annoying alarm, swing legs off nice, warm, comfy bed, stand up, and CRACK. Sigh. It’s going to be one of those days.

Actually, it only got worse.

As I am wont to do, I put my G5 to sleep every night rather than leaving it on or turning it completely off. I don’t leave it on because the thermostat for the furnace is actually right next to my G5 and monitor, and running all night, blowing warm air right at the thermostat, is a sure way to wake up to a chilly house. It’s hard enough to drag my butt out of bed, let alone trying to do so in a cold house.

So I headed downstairs, clicked on a few lights, and fired up the G5 by giving the spacebar a swift click. I hear the Mac start to churn up, the hard drive motors on, the monitor light brightens, and then… nothing. Again.

As Owen Rubin wrote about in a blog here, there seems to be a problem with Mac OS X 10.3 in which a Mac, regardless of its type, refuses to wake from sleep. It seems to get about halfway there, and then nothing. The result is a hard reset. A hard reset on the G5 means holding down the power button and wait for the machine to turn itself off. Then restart.

So I did that on that Thursday morning. Only this time, there was no start up. The Mac got as far as the grey screen with the Apple logo on it, and there it sat. Doing nothing. Oh-oh!

So I restart, hold down Command-S to boot into Unix, and run FSCK. With 10.3 and a journaled drive, you have to use the FSCK –F to force the disk repair utility to run. And I did. The result?

 

Gaaahhh! My hard drive is toast! And NOT the cool Roxio Toast kinda way, either! Toast as in “Hope ya got a backup, this hard drive is dead. D-E-A-D!”

Well, I know that the drive is physically fine. There is nothing wrong with the disk itself. The problem is the software ON the drive has gone bad. Still, I figured, I can rescue it.

Step one: Boot from the CD.
I have a number of CD’s I can boot the G5 from, including the Restore disk that came with the G5, 10.1 Install CD, 10.2, or 10.3. I grab the Mac OS X 10.3 Disc 1 from its case, pop it into my Super-Drive, and reboot.

I get to, surprise, the gray screen again, and no further. What the hey?? Yes, I held down the C key when I rebooted, telling the G5 to boot from the CD-ROM. It started to, but as soon as the OS X on the CD-ROM tried to read the internal hard drive, blam! No go.

Next up, make sure that the Mac not starting is actually related to the bad drive. So I shut down, unplug the drive, and reboot from the CD-ROM. No problem, the Mac starts right up. In fact, I have two internal ATA drives in the G5, and the 10.3 Install CD-ROM saw the other drive. This other drive does nothing more than hold video files and other large format graphic files I am in the process of working on. It’s a 250 GB Serial ATA drive, nice and fast. Thankfully, it also has around 200GB of free space. So I go ahead and install 10.3 on that drive.

Step two: Reboot from second drive into 10.3
This time, the Mac starts right up. Opps! I forgot to plug the original drive back in! Since I had already restarted, I let the new drive update everything before shutting down.

Once the new OS is up to date on the secondary drive, I shut the machine down, plug in the original drive, and reboot… Nothing.

Okay, maybe the G5 is trying to boot from the older, bad drive. This time I reboot and hold down the Option key, which brings up a handy feature of OS X in which you can select which OS you want to boot to. This was especially helpful during the early days of OS X when users needed to restart their machines into OS 9 without first having to go to the Start Up Disk control panel. It still works, and will let you designate which OS X on which drive you want to boot from. So I tell the Mac to boot from the newer drive, restart, and… Nothing.

Seems that if the bad drive is plugged in at all, the machine will not start. Nada. Nothing. It simply WILL NOT BOOT.

Okay…

Step Three: Firewire Target Disk Mode!
Next, connect the G5 to my G4 via the handy-dandy Firewire Target Disk Mode. This is pretty simply. Shut down the G5, connect a Firewire cable between the two machines, and restart the G5 by holding down the “t” on my keyboard.

The result? The G4 froze as soon as the bad hard drive tried to mount.

To make sure that this should work, I unplug the bad drive, but leave the good drive plugged in, and try again. Viola! The good G5 drive mounts on the G4’s desktop (Which is also running 10.3.5) as it should. So I know that the G5 wants to do it, just not with the bad HD plugged in.

At this point, I am a little frustrated. I make some stupid decisions, such as hot-plugging the bad hard drive in once the G5 is already up and running. This does nothing, the G5 does not see the drive. I try the latest copy of Norton Disk Doctor, it does not see the drive. I try Disk Warrior, it does not see it, either. I even tried Data Rescue from Prosoft, which really does not fix a drive, but helps you copy data off bad drives. In this case, it was useless. I am sure all three products do their job well, but with this problem, none could touch it.

Step Four: Try OS 9!
So I have yet another G4, not actually my machine but it has been here over a year now, that is running Mac OS 9.1. I decided that perhaps it may be something with OS X and the way it reads a hard drive software drivers that sit at the beginning of the hard drive spin-up cycle. So perhaps OS 9 would do a better job?

Connecting via Firewire Target Mode again, I booted up the G5, and the G4 with OS 9… crashed. Yup, not even mighty OS 9 could mount this puppy.

This was worrisome. The drive is FINE, just the software drivers are bad, or whatever “keys out of order” are. At this point, all I really want to do is copy my music files off the drive, or failing that, simply erasing the entire thing, reformat, and use it as a secondary drive. But no, you cannot even erase a drive if you cannot mount it, or start from a CD if the problem prevents ANY OS from starting up.

I had emailed the MyMac.com staff, explaining the problem and seeking help. Most suggested things I had tried on Thursday night or Friday, but it was still helpful to get a bunch of suggestions. Bruce, Chris, Adam, Roger, Russ, Jim Nemo, and Jeffery McPheeters all had good advice.

The thing is, I have been repairing Mac’s for ten years. I was an IT Manager in an all-Mac studio for almost five years. This is what I do. But I was stuck…

Owen Rubin emailed me a helpful idea:

Was your disk “protected” or encrypted?

Sounds like the driver in the boot blocks of that drive is dead, which is a bad thing. Sometimes you can reload the driver without hurting the disk to at least get it to load. USE to be able to do this with HARD DISK TOOL KIT because they ignored the driver on the disk. I do not know of a similar tool for OS-X.

Have you tried using the disk tools to update the driver on the drive and see if that works?

While most people will tell you NOT to connect a drive hot, I have done this before. Connect the cable but NOT the power. After the machine is up, CAREFULLY connect the drive power. I am not sure if OS-X will immediately try and load the drive or not, because if it does, and the driver is corrupted, then when it loads it will crash!

Do you have a machine that can boot OS-9? If so, boot that machine, connect the drive and run Hard Disk Tool Kit to mount the drive. At least you will be able to get your files off it.

These ideas might work unless the drive is physically damaged, in which case, you may be out of luck. Of course, drive savers can recover the data, but they are SO DAMN EXPENSIVE. Maybe you can offer to do a review of them saving your drive and post an ad in exchange? Just another thought.

-Owen-

Well, I had actually done all that. I was still not convinced that the drive was physically damaged, though I am sure my hot-plugging was not helping matters. And OS 9 would crash just as quickly as OS X would as soon as that drive tried to mount.

Owen emailed this follow-up as well:

To explain my last message a bit more, we use to see this problem when I first write bootable CD for the Mac. When a drive spins up, the system loads the devices driver from the device, and then executes it. Unfortunately, if the driver blocks contain crap, guess what, you die.

When I did bootable CD originally, I treated a CD in the drive like a floppy and tried to boot it if it had boot blocks. That means any bootable CD (which at that time only I had created) was not a problem. It turned out if you crashed with a music CD in your CD drive and restarted, some music CDs LOOK like they have boot blocks, the Mac would load the CD drivers, and crash immediately. This is why we added the “C” key to boot a CD now.

OK, this means during boot time that any and ALL devices that have boot blocks and want to mount will load their drivers early on and the system will execute them to mount the device. Not good if the device has a corrupted driver. This can be bad data in the boot blocks, a bad driver, a damaged drive, or any of a number of problems that causes the driver to get loaded. This is TYPICALLY why a machine crashes early on in the boot sequence and runs fine if you remove that device.

The goal is to mount the drive without using its drivers. See my last message for ways to do that.

-Owen-

This got me thinking. If the machine is already on, and the OS tries to mount the disk, but the disk driver is damaged, well, bad things happen. Also when he writes, “When the drive spins up, the system loads the drivers from the device, and then executes it.” Really got me thinking. So obviously I cannot have the OS load the driver. But how do you make the Mac NOT load the driver? And more, even if you could, would the Mac even “see” the drive and mount it?

In OS 7, 8, and 9, a common work around to get your Mac to boot when it was having problems was to hold down the Shift key to turn off extensions. Most problems in OS 9 and earlier were caused by extensions, so it was a very handy tool to use when troubleshooting problems on your Mac. But there is no equivalent in OS X that I am aware of. So what to do?

Step Five: OS 9 with no extensions.
I remembered from my IT days that if I booted Mac OS 9 without extensions on, any attached FireWire drive would also appear on the desktop. So I shut down the G4 running OS 9.1, fired up the G5 in Target mode, waited for it to get up and running, and restarted the G4 9.1 machine with extensions off.

Victory! There was the bad drive, as well as the good one, in the G5 mounted on the G4’s desktop!

Step Six: Copy! Copy! Copy!
Before I did anything else, I quickly copied all the files I was missing since my last backup. Unfortunately, that also meant copying around 40GB of music files. So late Saturday night, I started the copying, and by Sunday morning, all my files were now safely on the good G5 250GB hard drive.

Step Seven: Repair Time!
The first thing I did was run the OS 9 version of Disk Tools on the bad drive. As expected, it could not fix the problem. But I had, prior to connecting the G5 the first time around, installed the OS 9 version of Alsoft’s Disk Warrior on the G4.

I ran Disk Warrior on the bad drive. It took a LONG time. What Disk Warrior does is builds a new directory on a drive, which I was hoping would cure my problems. After three hours, Disk Warrior was done. It found a BUNCH of problems. To be on the safe side, I ran it again. This time it took only a little more than an hour, and did not find any more trouble.

Shutting everything down, I rebooted the G5. I held down the Option key so that I could tell the G5 which drive to start from. It only saw the new drive as a viable boot disk. Oh-oh…

I held down Command-S to bring up UNIX after the next boot, and ran FSCK-F on the bad drive. It worked, fixing a few thousand items. Yikes! What did Disk Warrior do? I ran it again, and it repaired some more. Three times, and it was done.

Restart.
Welcome to Macintosh!

And all was well in the world.

A few things to take away from this:

Not all problems are as bad as they seem. There are usually always solutions to your computer problem. Some involved thinking way outside the box. How many people would have figured that Mac OS 9.1 with extensions off would be the solution to this Keys out of Order problem? Certainly not I.

In most cases, Disk Warrior would have fixed this problem without all the run around I had to do. If you don’t already own it, you may want to go pick up a copy. It’s well worth the small investment.

FSCK is not always going to repair or fix your problems. So be sure to back-up your data often.

Having two hard drives is WAY better than having only one. If you have an iMac or Portable in which you cannot cram in another drive, look at external Firewire hard drives.

It is also helpful to have another Mac laying around for the Target Mode trick. Can’t afford one? I see them on eBay all the time, REALLY cheap!

Hope this article helps someone else out there if you ever run into a similar problem.