A Highly Available system is usually defined as a system that is up at least 99.999% of the time.

A “Highly Available” hosting environment is automated.  Webservers are virtualized on physical servers using VMWare™ or some other, higher end virtualization platform.  An HA environment detects failed servers, reboots them automatically, or in the event of failed hardware moves running virtual servers around.  A true highly available system can suffer multiple hardware failures before data becomes unavailable.

Backups:

The cloud is a great place to host your web gizmos and the like, but are you prepared for a failure?  Are you prepared to put the whole of your business in someone else’s hands?  *THAT* is the question to ask yourself when choosing a hosting provider.

CATBytes was built with a “What if the primary datacenter (located in Springfield, VA) becomes a smoking hole in the ground.”

RPO & RTO:

These are terms that consumers don’t hear too often, but they’re important.  RTO stands for “Recovery Time Objective”, RPO stands for “Recovery Point Objective”

RTO

Recovery Time Objective is, quite simply put, “How long will my system be down in the event of a catastrophic failure.”  In most cases, as RTO increases, so does cost.

RPO

Recovery Point Objective is, “How much data am I going to lose in the event of a catastrophic failure.”  Again, the lower the number, the higher the cost.

Many hosts will try to impress you with things like Network connection speed, but ask them these questions and you’ll not only impress them with your knowledge, you’ll gain a better understanding of theirs:

Q:  What is the RTO in the event of a catastrophic failure?   (The answer should never be more than 24 hours)

Q: What is the RPO in the event of a catastrophic failure?  (The answer should never be more than 24 hours)

Q: Are backups stored remotely?  (Backups that are kept locally on site are useless in the event of a physical problem at the datacenter.)

Q: Is the physical site staffed 24×7?  (If not simple problems such as power failures can’t be dealt with in a reasonable amount of time.)

We plan, we prepare…and yet still sometimes it happens.

An outage.

On Friday 12/23 at about 8:30pm EST CATBytes went dark.  All servers/services were down for approximately 2 hours.  For the record, this was the first such full system outage (not counting simple network outages) since moving into the new datacenter.

Why did it happen?

Because for all of my planning, all of my preparation, I just couldn’t conceive that a datacenter UPS failing could take out two of the three redundant power circuits.

Yeah, my two.

In all it behaved rather well, the internal batteries took over and the systems powered themselves down gracefully.  It’s also shown me where my weak spots are and those will be fixed, starting with a 5KVA Rack battery that’s going in on Tuesday.  (This will provide the entire rack with about 2.5 hours of run-time in the event of a datacenter failure)

We live and we learn.  Thankfully the damage wasn’t too horrid…at least not from the users’ standpoint.  No data was lost, no corruption.  All is good. :)

Read *ANY* tech article these days and you’ll find the word “Cloud” thrown around quite a lot. And I mean a *LOT*.

So what is this cloud we’re all so enamored off all of a sudden? Do you want it? Is it something that can help me?

Let’s take these questions in order.

What is the “Cloud”?

The term cloud is actually a marketing term for something you already know.   The Cloud is Managed Services.  More specifically, Virtualized, Managed Services.

Paying someone else to host the content and applications so that you don’t have to maintain the infrastructure yourself.

“Cloud” is a fancy term for “I dont know where it is, but it’s out there somewhere.”

The theory being that it is always there when you need it right?

Do I want / need the “Cloud”?

If you use Mozy for backups or DropBox for filestorage (and backup) you already do.  If you lease a Virtual Private Server from CATBytes or anyone else, you already do.  Even a standard webhosting account is technically “in the cloud”.  If you can’t point to the server, it’s “In the cloud.”  :)

There is an old joke “I used to complain about paying my taxes until I realized that I don’t own a fire engine of my own.”

Cloud services are great for consumers who want to run a small business, are starting a small business, or just hobbyists.  You pay someone else a small monthly fee, and they take care of ALL that goes on behind the scenes.   Most people don’t realize all that goes into maintaining a simple website..

Among these:

• Servers to handle DNS Resolution
• Servers to handle email (Inlcuding Anti-Spam)
• High-End / Redundant Webservers
• Backup Servers
• Internet Connection
• Cooling
• Power

Can the Cloud help me?

Of course it can.  But there is a price.  You have to trust who you’re hosting with.  We learned with The Amazon AWS failure in 2010 and other high-visibility cloud failures that you’re only as secure as your host.

Just make sure you ask the right questions.

Checkpoints

When it comes to computers, a checkpoint is a snapshot of your data as it was at a certain time.  Apple users familiar with the functionality of Apple’s “Time Machine” understand what I’m referring to.

The ability to go back to your harddrive at any given point in time in the past and recover data that you may have accidentally deleted.

At CATBytes, your data is what matters most.  So we’ve devised a way to make this technology available to you.

Four times a day, our system will automatically take a ‘snapshot’ of your data as it is at that time.  This will then be made available to you.

A few tidbits about checkpoints:

**Checkpoints are READ ONLY.  Even you can’t delete the data within a checkpoint.  If you want to permanently delete something, you have to wait for all checkpoints to expire.

**Checkpoints are stored OFF-HOST – even if an enterprising young hacker manages to wipe your site clean and lay waste to the server, content is protected.

** Databases are NOT covered by checkpoints.  By their nature, database files are always in use, so checkpointing a filesystem doesn’t work on them.  We back up all database files daily and store for 60 days, along with a content backup.

** All daily backups are replicated off-site every day.  Even if we lose the datacenter, your data can be recovered.  (Of course, if that happens, you’ll find me living on the corner of 2nd and Pennsylvania SE in Washington.  There’s a Starbucks there.)

Nuts and Bolts

If you’re geeky like me, you’re using a command-line (text-only) FTP client.  The easiest way for you to access one of your checkpoints is simply to type ‘cd .ckpt’ from any directory.  A subsequent directory listing will give you a list of dates and times to choose from.  Choose the more recent, or if you’re looking for a specific version of a file, go further back in time.

(Click images to see full-size)

If you’re a non-geek, you’ll find you can still easily access your checkpointed data through your graphical FTP client: I use “Filezilla” when I’m lazy….

The file ‘wp-signup.php must die…

Highlight and click “Delete”

Of course I’m sure, I clicked it didn’t I?

It’s gone!  Whatever shall I do?  Oh the humanity… well..sort of.

But wait. I have CHECKPOINTS!.   (Note – you (HAVE) to manually enter the /.ckpt at the end of the path, it doesn’t show up unless you check “show hidden” in preferences)

In most cases, you’ll go to the last checkpoint (most recent) unless you’re looking for a file you changed several doezen times in the last few days, then you can narrow it down to a 6-hour period.

First we copy it to our local PC…

Then we copy it back to the site in it’s original location.

And Poof!- it’s back.

Of course, if you have time to kill and don’t want to be bothered, you can still email us at support@catbytes.com and we’ll happily take care of the restore for you, or if the file isn’t avaialble, we’ll arrange to pull a copy of the file off of our BURA (Backup, Recovery, Archive) subsystem.

Feel free to email if you have ANY questions.

Good lord, it’s finally here.

I’ve been working on the relaunch of the site, as well as the relaunch of the business, for a few months now.

I’m not sure when it happened, when my hobby turned into a business, when I started realizing that people actually needed the service.

I’ve often said that what is missing in this day and age is the personal service.  All of my competitors in the webhosting arena want fire & forget customers.  They want you to sign up, give them your credit card number, and then they really never want to hear from you again.

I’m not like that.  If I don’t hear from my customers from time to time I’ll send a tweet, or an IM, and figure out where they are.  Maintianing the relationship is key.

I’m a blogger.  Maybe not a great one, but a blogger.  I like to think I have a better idea of what you’re looking for.  Maximum uptime and protection are key.  When your livelihood is based on your site being visible to others, you can’t afford for it to disappear at random intervals for random reasons.  All of my servers average 99+ percent uptime. Outages are usually brief, dealt-with quickly, and apologized for profusely.

Finally, I spend more time/effort on backups than is really warranted.  My “day” job consists of planning for keeping a business up in a ‘worst-case-scenario’ situation (IE – datacenter becomes a smoking hole in the ground) and I tend to apply those same principles to CATBytes.  Even if hackers were to penetrate the dual firewalls and lay waste to a server, my recovery time is less than 3 hours.  If the primary datacenter goes offline I can rebuild in under 24 hours at an alternate location.

It will likely never happen, but I can’t not plan for it. :)