Stephenryan.ie

Windows 7 brought around a number of different improvements for my system administrator job, most notably the fact that it connects to the Win2k8 servers while the old XP system didn’t. However this morning during a mass patching, the Aero Peek feature really has shown its true nature.

Right now I’m installing patches on 6 identical machines to bring them up to date with the patches from our WSUS server. (There are reasons why we don’t clone the boxes to get patches on.) But picture the only, XP way, of doing this. 6 remote desktop sessions and randomly switching between them to see the progress. Switch to the Windows 7 way and it becomes open 6 Hyper-V connections, then mouse over the taskbar icon to get the image below.

Completely wonderful. Total time saver.

In the time normally spent flicking between machines, I’ve written this post, added some more nagios alerts in, and checked on a few other servers.

Long time users of Komplett already know that they switches warehouses and had a few account problems during the switchover. The whole thing seemed a little rushed from the experience on the website, but then management could easily have been pushing them. Things such as not being able to open multiple tabs by control clicking was when I first started getting worried.

Don’t get me wrong now, I wouldn’t be a massive Komplett purchaser, at least not on a person level. We did put a little bit of the company purchases through from time to time although it was about to expand out. However when I could no long login in to the site and read about missing orders, I decided to pass on placing the order at the time, go somewhere else and then return when all the problems were fixed.

Today is the day when all that changed. The following email arrived in this morning marking the end to my use of Komplett.

So what now? Komplett have always been a consistent company and decent to deal with. Well basically any company incapable of transferring account or even telling customers opening about the issues isn’t a company I want to deal with. (I had to google it to find out what was going on – to komplett, a graphic at the top of the homepage isn’t enough, correct place is a link on the login page)

In the end it has worked out better for me. We ended up with a large account with one of the supplier that the likes of Komplett use. Basically we are ending up saving another 2-5% of komplett prices, and we get things quite a bit faster, generally next day delivery.

I had heard of this message, but this is the first time I’ve seen it. I’d say what site, but even though it was only 5 minutes ago, I honestly can’t remember what site it was. The site lost a visitor, because simply, I’m not going to adjust my habits for you, and neither will many others.

Acronis Backup, a great tool. Backs up files. Lets you set a real schedule (Windows take note). New one seems to compress your older backup files to save space. Forces you to buy new products with every new version. Does not reply to emails outside of maintenance contracts (which you can not renew). Makes it hard to upgrade.

What? Hold on a second. Maybe something is amiss here.

Some Background

I’m a user off Acronis Backup products for close on 18 months now. I’ve been using the TrueImage corporate product. And I’ve been reasonably happy with it, but then I haven’t used it to do any full restores. I did try to pull a large file from a backup in the past. But that didn’t work. it repeatedly crashed while trying to do so. Support, who responded at the time since my maintenance contract was in date, weren’t much help. If the file was critical, I’d have been pissed. Luckily it wasn’t, and a few days and another disk later, I managed a full disk restore, mounted the new disk as an external drive, and then copied the file off. Painful.

Wonderful Search…but only for home users

The lately problem with restoring came from me needing to find a folder on my desktop that I knew existing sometime 5 months ago. All fine I thought, I’ll do a search of the files and it’ll show up. The product doesn’t support searching? Oh your HOME product does but your business one does not. Fine so. I’ll “test” the home version and see if I can switch to that. So in goes the trail version on a blank machine. The initial search looks promising. Finds a few text files in the first backup. I’ll leave it search and see if it can find the files I’m after.

A day later, Windows is still searching, still reading files, sort of, but no results. Watching the accesses, it is bouncing through the individual files in sequence, but then repeating. The Windows search service had also crashed and was unable to index inside the TIB files.

How about Google? They can search everything! And yes, there is an iFilter for Google Desktop. But, and there always is a but, it won’t search the archives either. It craps out.

Again, luckily, I managed to find the files I needed by browsing through a few days in June looking at the desktop each time. Can’t say I’d like to do that for something that went missing randomly.

Upgrades

I don’t know if they make this purposefully hard or not, but finding an option to upgrade took time. Hell I can’t even remember where it was now. All I remember was that the upgrade from the old TrueImage to the new Backup10 costs MORE than buying the home version outright as a new customer. And to add insult, the Home Version has more features, not that you can be sure they work either.

Looking at the upgrades began due to the fact that TrueImage does not work on Windows 7. It has just been added to the DoNotExecute list that Microsoft have in the OS. And it was added by Acronis. There is a way around this list, and everyone who has bypassed it has had NO problems with the program. We can only guess that Acronis have had some reason as to why they did this. Of course it would have nothing to do with their new product coming out. A new product that may or may not actually support Windows 7. The Home one does, but I wanted business style products.

When I didn’t find the upgrade option or any mention of it, I did email Acronis via my customer area. They even replied with one of those auto replies to say the email had been received. All I wanted to know was if the product was supported on Windows 7 and if it could be fixed. If not, what was the upgrade path so I could do it.

Conclusions

Having to wait over a week and counting for a reply to upgrade has left a bad taste in my mouth. It reminded me of the past experiences of the product and it failing.  Also being unable to renew the maintenance support after the initial 12 months is worse. But more so since they operate in a 14-16 month product release cycle going on other people I’ve talked to. That would only be the case to case people to not get product updates under their maintenance agreement and be forced to upgrade or buy the brand new product each time.

It is the lack of support that as sealed the deal for me. TrueImage has been uninstalled from my laptop (not that it worked since I swapped to Windows 7). I’ve already begun searching for alternatives. The inBuilt Windows Backup may do the trick if I can get it to run on weekdays only. One I am itching to trial is inSync by a company called Druvaa. I’ve been testing their new Server backup tool and am VERY impressed with it. So something like this would be overkill for personal use, but it may work. Especially with the dedupe, it may make it a good option. Time will tell. All that is sure is Acronis will not be getting any more of my money, or any from any of the companies I work for. Their loss.

Anyone who has worked with storage systems, or even large personal installs has heard of them. Double disk failures. Words you never say. Ever! You can be banished from the server room for even suggesting it is possible!

But the reality is it can, and does happen. It is why we have hot swappable disks, or even hot swappable drives. I’m even looking at some array by NetApp which has something called DP or Dual Parity which, they say, can handle two separate disk failures without taking down the array. Something that sounds very interesting really. The Dell / Equallogic array we have on test currently runs in a type of raid 50 so you can lose two disks but only from separate arrays. The other two disks are running as hot standby disks to allow for online rebuilding.

The setup

My current, dilemma we’ll call it, is with a much simpler setup. Intel based server with 8 SATA disks connected to a 3ware card doing Raid-5. It is a high end 3ware card too, a 9650. (I do NOT recommend these cards. We have numerous other performance issues with the cards in both Windows and Linux, the Windows ones being much worse, currently stopping me copying backups). Anyway, to make things a little more challenging, something every admin loves in their day is a challenge, the server is remote. In another country remote.

Anyway, this machine has been running fine for nearly a year. Raid array sitting there taking files happily enough. When I started testing some further backups recently, I ran into some troubles. Most of it looked to be Windows related so the usual apply the updates, reboot the machine and see what happens. Only on the first reboot, wham, disk 8 offline. Ok, so I’ll finish the updates and then worry about getting another disks over to be put into the machine. Next reboot, disk comes magically back online but in a degraded state. Strange, we’ll let this rebuild and return tomorrow, see if live has returned to normal.

Normal is normal is just a cover

Sleeping on things and letting the array rebuild and everything looks to be great and just a temporary problem that we can forget about and move on. Never a good idea but when you are overworked, what can you do?

Another day passes trying to move backups across and we hit another windows error. This time requiring a registry fix to increase the IRQStackSize. So I bang in the first change and reboot. Login and strange, the system is locking up it appears. Open the 3ware webpage and get prompted with something  I’d not seen until now.

Raid Status: Inoperable

Luckily these are backups, no live data lost. We can fix this. Hell lets try a reboot and see. Can’t do anymore damage can it?

The Recovery?

Rebooting fixes disks, magically. Both disks back online. Array in a consistent state. Why not leave well enough alone?

…

More windows problems and another reboot. Back to two disks offline. Reboot again and one disk gone. Useless, useless, useless.

Solutions…

If this was a live server, with live data? I’d probably cry. There’d not be much else to do. You could probably have it rebuild by replacing the disk that was going offline the most, but I’d move as much off as quick as possible. In this case, since it is a backup server, I’ll be getting the guys local to the machine to remove and reseat all the drives. And check the cables inside the case. And then destroy and reformat the array, and the filesystem, with full formats all around.

And then to top it off, 10 reboots, minimum, when the server isn’t looking! If they all work, then maybe, just maybe I’ll look at trusting it again. Any problems and I guess I’m on a plane

Lessons learned

Well I think I’ll be putting the really critical data onto more than one backup server in future. At least more of the fileserver data anyway. The massive exchange backups will need to be looked at.

Enterprise level SANs are cheaper than you think when you factor in the cost of fixing setups like this. Okay so you aren’t going to be able to get a SAN for twice the price of a server with 16x1TB drives in it, or even three times. You may get a low spec’d one however, and if it gives more piece of mind, maybe that is worth the cost? I know that if faced with the decision in future, I’ll probably recommend a SAN and attached server for a file server assuming it is above the 1TB mark. Lower than that, you can probably get anyway with the multiple servers, replication software AND backups. Replication software is NOT backup software. Delete from live, deletes from backup.

And what nows?

I don’t know. All I can hope is that reseating disks and cables fixes the array, gets it online and lets me start transferring backups offsite. Another box is going to be added to give more backups, hopefully point in time backups too.

Backups really are the largest cost for something you never will use. I do honestly hope I never have to pull any data from backups, ever. It is possible what with Volume Shadow Copies on file servers and raid disks for servers. And maybe real permissions for applications, but that is another day!

I’ve recently started using the In Private browsing feature of IE8 more and more, and no not for Porn!

For testing sites I’m developing or doing a clean Google search, it would usually involved closing the browser, clearing cookies / cache etc. and then restarting. It is now reduced to Tools -> InPrivate Browsing, and bang you’ve got a clean browser session. And I know Firefox supports this, but they really make it unusable if you run Firefox with lots of open tab (currently I’ve 36 and it isn’t a busy day) because it shuts the browser down, opens the private browsing mode and then restores things after it is finished.

I guess I’ve long since used two browsers. Firefox for personal stuff and general web development tasks. IE for Intranet net applications and not the InPrivate feature.

If only someone would invent some proper work spaces for a browser. And some better way of storing/organising favourites.

Some things don’t change, and in computers, while it may look to be moving really fast, a lot of the ideas have been floating around for a while. Technology enables technology.

I read about the book “Coders at Work” which the author kindly posted up a few interviews online. Bill Gates has one from 1986 (found here). It really is a truly insightful read. Bare in mind that this is pre windows, and not just gui windows. Hell it is pre CD Rom.

While talking about 4k memory limits, the thoughts of 650meg of data accessible to a program must have seemed surreal. Our minds aren’t capable of understanding numbers like this anymore. Best way I can explain it is like moving from only having data available from your harddisk to having data available from a Google data centre.

One of parts that really hit home was when Bill Gates starts talking about what is a effectively a Google Maps mash up.

GATES: CD ROM is totally different. We hope with CD ROM you’ll be able to
 look at a map of the United States, point somewhere, click, zoom in and
 say, “Hey, what hotels are around here?” And the program will tell you.

Quotes like that really do show how far ahead he really was. The internet was only being born but here he is thinking on new ways to display data.

If 1986 can give thoughts that only come to live today, what ways will people start displaying data in the future. Sure todays BIG computer problem is more a data problem than a technical one.

I’m not ignorant and I do like talking to you, honest. But getting into the flow takes time and noise interrupts in. Music in the background has always seemed to help block out the annoying thing. Its why in the early morning or late evening, so much more work is done. Sure you know yourself, you may be in work from 9-6, but you really only get about 2 hours of good work done. And now here is why.

http://softwarenation.blogspot.com/2009/01/importance-of.html provided a pretty good but basic enough explanation of the brain and how it works in sections. Combine this with what I knew previously, and it makes a lot of sense.

While programming part A, the voice in your head part,  is running away with itself, swapping variables and the like, part B, the thinking part, is looking ahead and coming up with that creative code block you are about to write. Part C, the hearing part, is being kept nice and busy by the music coming in stopping other noise coming in and ending up in the thinking part. Because when it gets to the thinking part, it is like a big freight train hitting a mountain, it derails and looses all its cargo. You start down the hill on another track, what it was you heard and the further you go, the more you’ve left behind. And just like that big freight train, it takes a long time to get back up to speed on the hill, and you have to pick up all those information bits that you lost before.

So today while working through some of the normal morning tasks, I started checking on the latests news. If you aren’t using it already, Google Reader is quite invaluable. I wish I could keep all the articles in it forever as you never know when you will need to find something older. It is why some of the tech newsletters still come through via email, but I’ll probably switch some of them soon.

Anyway, I decided I’d subscribe to my own blogs, both this one and the web hosting company one I write for. Low and behold, I noticed that one of them was only displaying a summary of the articles, a big no no if you ever want anyone to actually subscribe to your blog.

Even more shamefully, I had to Google to find the option to fix this.

For reference, it is in Settings -> Reading. “For each article in a feed, show”.

Through a few of the sites I have interactions with, there are a number of adwords campaigns that are running. Strangely, why tracking some of the incoming clicks, we noticed that at last one of the clicks came from within Google itself. And what is more, it appears to be legit going on the browser type of “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”.

Perhaps Google tests these things manually, who knows. Guess I’ll just have to continue digging through the logs and see what shows up.

For reference, IP in question came from

OrgName:    Google Inc.
OrgID:      GOGL
Address:    1600 Amphitheatre Parkway
City:       Mountain View
StateProv:  CA
PostalCode: 94043
Country:    US

NetRange:   74.125.0.0 - 74.125.255.255
CIDR:       74.125.0.0/16
NetName:    GOOGLE
NetHandle:  NET-74-125-0-0-1
Parent:     NET-74-0-0-0-0
NetType:    Direct Allocation
NameServer: NS1.GOOGLE.COM
NameServer: NS2.GOOGLE.COM
NameServer: NS3.GOOGLE.COM
NameServer: NS4.GOOGLE.COM
Comment:
RegDate:    2007-03-13
Updated:    2007-05-22

OrgTechHandle: ZG39-ARIN
OrgTechName:   Google Inc.
OrgTechPhone:  +1-650-318-0200
OrgTechEmail:  arin-contact@google.com