Stephenryan.ie

I’ve been used Dirvish now for just over a year. It replaced a number of rsync replication scripts that I had running that were doing rolling backups. While moving to Dirvish has required a few extra scripts to be written, it has been a worthwhile experience. My own scripts weren’t able to handle holding backups for longer periods, at least not gracefully. The biggest issue we had was trying to get Dirvish to do different backups on different schedules. The Dirvish config, while it may look like it allows this at first glance, it really isn’t setup for it. Backups once per day is its bread and butter.

Hopefully this will help clear up a few minor issues with Dirvish and get you running with multiple independent schedules.

Credit where credit is due, some of this is a result of a different sources on Google. We have modified this a number of times over the last year to fit our needs so I’m not totally sure how much of the original remains.

Note: This is from a Debian based system. Paths reflect same.

Initial Dirvish Configuration

For this guide, out setup consists of 1 host which we backup once per day, and the same host which has a directory which gets backed up once per hour. Backups are being stored under /storage/Backups/dirvish.

Our master.conf file – notice that no hosts are actually defined here.

bank:
     /storage/Backups/dirvish
exclude:
     lost+found/
     *~
     .nfs*
expire-default: +15 days
expire-rule:
#       MIN HR    DOM MON       DOW  STRFTIME_FMT
    *   *     *   *         1    +3 months
    *   *     1-7 *         1    +6 months
    *   *     1-7 1,4,7,10  1    +6 month
    *   10-20 *   *         *    +4 days
#   *   *     *   *         2-7  +15 days

runall-daily.conf

Runall:
     host     02:00

runall-hourly.conf

Runall:
      host/hourly/folder

/etc/cron.d/dirvish – This is what calls the jobs

00 01 * * *     root    /usr/sbin/dirvish-expire --quiet   # Expire old images
00 02 * * *     root    /usr/sbin/dirvish-runall --quiet --config runall-daily.conf
00 * * * *      root    /usr/sbin/dirvish-runall --quiet --config runall-hourly.conf

With those in place, our host backs up at 2am every day. The hourly script kicks in every hour. We setup the vaults as normal in the folders defined about. So the hourly is /storage/Backups/dirvish/host/hourly.

Only thing that needs changing is the image-default option in the configs.

Daily vault: image-default: %Y%m%d
Hourly vault: image-default: %Y%m%d%H

Living with it

This setup has run great for us. We get what we need backed up when we need it backed up. There has been a few notable excepts however.

First, one of our hosts started locking up during backup windows. Dirvish then went nuts and started marking incomplete backups correct somehow. We noticed when our bandwidth shot up as it was recopying full machine images across.

Second, our expire rules tend to leave too much data. Yes we could probably fix this, and we probably will when space becomes an issue. I guess the first thing is for our hourlys to reduce down to one a day on the older sets instead of keeping the whole day. But since Dirvish is so good with space, a few months of hourlys isn’t taking too much space.

Overall however, I can’t say we’ll be moving from Dirvish anytime soon for backups, at least for our linux machines.

Windows 7 brought around a number of different improvements for my system administrator job, most notably the fact that it connects to the Win2k8 servers while the old XP system didn’t. However this morning during a mass patching, the Aero Peek feature really has shown its true nature.

Right now I’m installing patches on 6 identical machines to bring them up to date with the patches from our WSUS server. (There are reasons why we don’t clone the boxes to get patches on.) But picture the only, XP way, of doing this. 6 remote desktop sessions and randomly switching between them to see the progress. Switch to the Windows 7 way and it becomes open 6 Hyper-V connections, then mouse over the taskbar icon to get the image below.

Completely wonderful. Total time saver.

In the time normally spent flicking between machines, I’ve written this post, added some more nagios alerts in, and checked on a few other servers.

Long time users of Komplett already know that they switched warehouses and had a few account problems during the switchover. The whole thing seemed a little rushed from the experience on the website, but then management could easily have been pushing them. Things such as not being able to open multiple tabs by control clicking was when I first started getting worried.

Don’t get me wrong now, I wouldn’t be a massive Komplett purchaser, at least not on a person level. We did put a little bit of the company purchases through from time to time although it was about to expand out. However when I could no longer login in to the site and read about missing orders, I decided to pass on placing the order at the time, go somewhere else and then return when all the problems were fixed.

Today is the day when all that changed. The following email arrived in this morning marking the end to my use of Komplett.

So what now? Komplett have always been a consistent company and decent to deal with. Well basically any company incapable of transferring account or even telling customers openly about the issues isn’t a company I want to deal with. (I had to google it to find out what was going on – to komplett, a graphic at the top of the homepage isn’t enough, especially a graphic that isn’t the default graphic, correct place is a link on the login page)

In the end it has worked out better for me. We ended up with a large account with one of the supplier that the likes of Komplett use. Basically we are ending up saving another 2-5% of komplett prices, and we get things quite a bit faster, generally next day delivery.

I had heard of this message, but this is the first time I’ve seen it. I’d say what site, but even though it was only 5 minutes ago, I honestly can’t remember what site it was. The site lost a visitor, because simply, I’m not going to adjust my habits for you, and neither will many others.

Acronis Backup, a great tool. Backs up files. Lets you set a real schedule (Windows take note). New one seems to compress your older backup files to save space. Forces you to buy new products with every new version. Does not reply to emails outside of maintenance contracts (which you can not renew). Makes it hard to upgrade.

What? Hold on a second. Maybe something is amiss here.

Some Background

I’m a user off Acronis Backup products for close on 18 months now. I’ve been using the TrueImage corporate product. And I’ve been reasonably happy with it, but then I haven’t used it to do any full restores. I did try to pull a large file from a backup in the past. But that didn’t work. it repeatedly crashed while trying to do so. Support, who responded at the time since my maintenance contract was in date, weren’t much help. If the file was critical, I’d have been pissed. Luckily it wasn’t, and a few days and another disk later, I managed a full disk restore, mounted the new disk as an external drive, and then copied the file off. Painful.

Wonderful Search…but only for home users

The lately problem with restoring came from me needing to find a folder on my desktop that I knew existing sometime 5 months ago. All fine I thought, I’ll do a search of the files and it’ll show up. The product doesn’t support searching? Oh your HOME product does but your business one does not. Fine so. I’ll “test” the home version and see if I can switch to that. So in goes the trail version on a blank machine. The initial search looks promising. Finds a few text files in the first backup. I’ll leave it search and see if it can find the files I’m after.

A day later, Windows is still searching, still reading files, sort of, but no results. Watching the accesses, it is bouncing through the individual files in sequence, but then repeating. The Windows search service had also crashed and was unable to index inside the TIB files.

How about Google? They can search everything! And yes, there is an iFilter for Google Desktop. But, and there always is a but, it won’t search the archives either. It craps out.

Again, luckily, I managed to find the files I needed by browsing through a few days in June looking at the desktop each time. Can’t say I’d like to do that for something that went missing randomly.

Upgrades

I don’t know if they make this purposefully hard or not, but finding an option to upgrade took time. Hell I can’t even remember where it was now. All I remember was that the upgrade from the old TrueImage to the new Backup10 costs MORE than buying the home version outright as a new customer. And to add insult, the Home Version has more features, not that you can be sure they work either.

Looking at the upgrades began due to the fact that TrueImage does not work on Windows 7. It has just been added to the DoNotExecute list that Microsoft have in the OS. And it was added by Acronis. There is a way around this list, and everyone who has bypassed it has had NO problems with the program. We can only guess that Acronis have had some reason as to why they did this. Of course it would have nothing to do with their new product coming out. A new product that may or may not actually support Windows 7. The Home one does, but I wanted business style products.

When I didn’t find the upgrade option or any mention of it, I did email Acronis via my customer area. They even replied with one of those auto replies to say the email had been received. All I wanted to know was if the product was supported on Windows 7 and if it could be fixed. If not, what was the upgrade path so I could do it.

Conclusions

Having to wait over a week and counting for a reply to upgrade has left a bad taste in my mouth. It reminded me of the past experiences of the product and it failing.  Also being unable to renew the maintenance support after the initial 12 months is worse. But more so since they operate in a 14-16 month product release cycle going on other people I’ve talked to. That would only be the case to case people to not get product updates under their maintenance agreement and be forced to upgrade or buy the brand new product each time.

It is the lack of support that as sealed the deal for me. TrueImage has been uninstalled from my laptop (not that it worked since I swapped to Windows 7). I’ve already begun searching for alternatives. The inBuilt Windows Backup may do the trick if I can get it to run on weekdays only. One I am itching to trial is inSync by a company called Druvaa. I’ve been testing their new Server backup tool and am VERY impressed with it. So something like this would be overkill for personal use, but it may work. Especially with the dedupe, it may make it a good option. Time will tell. All that is sure is Acronis will not be getting any more of my money, or any from any of the companies I work for. Their loss.

Anyone who has worked with storage systems, or even large personal installs has heard of them. Double disk failures. Words you never say. Ever! You can be banished from the server room for even suggesting it is possible!

But the reality is it can, and does happen. It is why we have hot swappable disks, or even hot swappable drives. I’m even looking at some array by NetApp which has something called DP or Dual Parity which, they say, can handle two separate disk failures without taking down the array. Something that sounds very interesting really. The Dell / Equallogic array we have on test currently runs in a type of raid 50 so you can lose two disks but only from separate arrays. The other two disks are running as hot standby disks to allow for online rebuilding.

The setup

My current, dilemma we’ll call it, is with a much simpler setup. Intel based server with 8 SATA disks connected to a 3ware card doing Raid-5. It is a high end 3ware card too, a 9650. (I do NOT recommend these cards. We have numerous other performance issues with the cards in both Windows and Linux, the Windows ones being much worse, currently stopping me copying backups). Anyway, to make things a little more challenging, something every admin loves in their day is a challenge, the server is remote. In another country remote.

Anyway, this machine has been running fine for nearly a year. Raid array sitting there taking files happily enough. When I started testing some further backups recently, I ran into some troubles. Most of it looked to be Windows related so the usual apply the updates, reboot the machine and see what happens. Only on the first reboot, wham, disk 8 offline. Ok, so I’ll finish the updates and then worry about getting another disks over to be put into the machine. Next reboot, disk comes magically back online but in a degraded state. Strange, we’ll let this rebuild and return tomorrow, see if live has returned to normal.

Normal is normal is just a cover

Sleeping on things and letting the array rebuild and everything looks to be great and just a temporary problem that we can forget about and move on. Never a good idea but when you are overworked, what can you do?

Another day passes trying to move backups across and we hit another windows error. This time requiring a registry fix to increase the IRQStackSize. So I bang in the first change and reboot. Login and strange, the system is locking up it appears. Open the 3ware webpage and get prompted with something  I’d not seen until now.

Raid Status: Inoperable

Luckily these are backups, no live data lost. We can fix this. Hell lets try a reboot and see. Can’t do anymore damage can it?

The Recovery?

Rebooting fixes disks, magically. Both disks back online. Array in a consistent state. Why not leave well enough alone?

…

More windows problems and another reboot. Back to two disks offline. Reboot again and one disk gone. Useless, useless, useless.

Solutions…

If this was a live server, with live data? I’d probably cry. There’d not be much else to do. You could probably have it rebuild by replacing the disk that was going offline the most, but I’d move as much off as quick as possible. In this case, since it is a backup server, I’ll be getting the guys local to the machine to remove and reseat all the drives. And check the cables inside the case. And then destroy and reformat the array, and the filesystem, with full formats all around.

And then to top it off, 10 reboots, minimum, when the server isn’t looking! If they all work, then maybe, just maybe I’ll look at trusting it again. Any problems and I guess I’m on a plane

Lessons learned

Well I think I’ll be putting the really critical data onto more than one backup server in future. At least more of the fileserver data anyway. The massive exchange backups will need to be looked at.

Enterprise level SANs are cheaper than you think when you factor in the cost of fixing setups like this. Okay so you aren’t going to be able to get a SAN for twice the price of a server with 16x1TB drives in it, or even three times. You may get a low spec’d one however, and if it gives more piece of mind, maybe that is worth the cost? I know that if faced with the decision in future, I’ll probably recommend a SAN and attached server for a file server assuming it is above the 1TB mark. Lower than that, you can probably get anyway with the multiple servers, replication software AND backups. Replication software is NOT backup software. Delete from live, deletes from backup.

And what nows?

I don’t know. All I can hope is that reseating disks and cables fixes the array, gets it online and lets me start transferring backups offsite. Another box is going to be added to give more backups, hopefully point in time backups too.

Backups really are the largest cost for something you never will use. I do honestly hope I never have to pull any data from backups, ever. It is possible what with Volume Shadow Copies on file servers and raid disks for servers. And maybe real permissions for applications, but that is another day!

I’ve recently started using the In Private browsing feature of IE8 more and more, and no not for Porn!

For testing sites I’m developing or doing a clean Google search, it would usually involved closing the browser, clearing cookies / cache etc. and then restarting. It is now reduced to Tools -> InPrivate Browsing, and bang you’ve got a clean browser session. And I know Firefox supports this, but they really make it unusable if you run Firefox with lots of open tab (currently I’ve 36 and it isn’t a busy day) because it shuts the browser down, opens the private browsing mode and then restores things after it is finished.

I guess I’ve long since used two browsers. Firefox for personal stuff and general web development tasks. IE for Intranet net applications and not the InPrivate feature.

If only someone would invent some proper work spaces for a browser. And some better way of storing/organising favourites.

Some things don’t change, and in computers, while it may look to be moving really fast, a lot of the ideas have been floating around for a while. Technology enables technology.

I read about the book “Coders at Work” which the author kindly posted up a few interviews online. Bill Gates has one from 1986 (found here). It really is a truly insightful read. Bare in mind that this is pre windows, and not just gui windows. Hell it is pre CD Rom.

While talking about 4k memory limits, the thoughts of 650meg of data accessible to a program must have seemed surreal. Our minds aren’t capable of understanding numbers like this anymore. Best way I can explain it is like moving from only having data available from your harddisk to having data available from a Google data centre.

One of parts that really hit home was when Bill Gates starts talking about what is a effectively a Google Maps mash up.

GATES: CD ROM is totally different. We hope with CD ROM you’ll be able to
 look at a map of the United States, point somewhere, click, zoom in and
 say, “Hey, what hotels are around here?” And the program will tell you.

Quotes like that really do show how far ahead he really was. The internet was only being born but here he is thinking on new ways to display data.

If 1986 can give thoughts that only come to live today, what ways will people start displaying data in the future. Sure todays BIG computer problem is more a data problem than a technical one.

I’m not ignorant and I do like talking to you, honest. But getting into the flow takes time and noise interrupts in. Music in the background has always seemed to help block out the annoying thing. Its why in the early morning or late evening, so much more work is done. Sure you know yourself, you may be in work from 9-6, but you really only get about 2 hours of good work done. And now here is why.

http://softwarenation.blogspot.com/2009/01/importance-of.html provided a pretty good but basic enough explanation of the brain and how it works in sections. Combine this with what I knew previously, and it makes a lot of sense.

While programming part A, the voice in your head part,  is running away with itself, swapping variables and the like, part B, the thinking part, is looking ahead and coming up with that creative code block you are about to write. Part C, the hearing part, is being kept nice and busy by the music coming in stopping other noise coming in and ending up in the thinking part. Because when it gets to the thinking part, it is like a big freight train hitting a mountain, it derails and looses all its cargo. You start down the hill on another track, what it was you heard and the further you go, the more you’ve left behind. And just like that big freight train, it takes a long time to get back up to speed on the hill, and you have to pick up all those information bits that you lost before.

So today while working through some of the normal morning tasks, I started checking on the latests news. If you aren’t using it already, Google Reader is quite invaluable. I wish I could keep all the articles in it forever as you never know when you will need to find something older. It is why some of the tech newsletters still come through via email, but I’ll probably switch some of them soon.

Anyway, I decided I’d subscribe to my own blogs, both this one and the web hosting company one I write for. Low and behold, I noticed that one of them was only displaying a summary of the articles, a big no no if you ever want anyone to actually subscribe to your blog.

Even more shamefully, I had to Google to find the option to fix this.

For reference, it is in Settings -> Reading. “For each article in a feed, show”.