Redundancy, sharding and high availability
This page answers one real problem that every growing website eventually runs into: how do you keep a website working, and keep its information safe, once it has grown too big and too important to trust to a single computer? Below is exactly how that problem gets solved, one real piece at a time.
The problem being solved
A website is, underneath everything, a computer sitting in a building somewhere, doing work and storing information. If that one computer stops working, because its part breaks, because the building loses electrical power, or because someone by accident deletes something, the website stops working too, and the information on it can be lost forever. A hard drive, which is the part of a computer that actually stores information, is a physical object, and every physical object can eventually break. Relying on exactly one copy of anything important, stored on exactly one machine, is called having a single point of failure, meaning one single thing that, if it fails, brings everything else down with it. Everything on this page is a different way of making sure a single point of failure never exists.
1 Redundancy — keeping more than one copy running at the same time
Redundancy means keeping extra, real, working copies of something on purpose, so that if one copy stops working, another one is already there and ready to take over immediately. For a website, this usually means running the same website on more than one computer at the same time, often in buildings in different cities, so that a problem in one building, such as a power failure, a fire, or a flood, does not take the whole website down. This is different from a backup, explained further down this page, because these extra copies are not sitting in storage waiting to be used one day. They are already switched on, already working, and already able to answer a visitor's request right now.
2 Sharding — splitting the information into smaller pieces
A very large website can end up with more information than any single computer can comfortably hold or search through quickly. Sharding is the answer to that: instead of keeping every visitor's information in one enormous pile, the information is split into many smaller piles, called shards, and each shard is small enough for one computer to manage well. A website might, for example, put everyone whose account was made in a certain year into one shard, and everyone who lives in a certain part of the world into another. Splitting the work this way turns one job that would be too big for any single machine into many smaller jobs that many machines can each do well, at the same time.
3 Snapshots and continuous backup
A snapshot is a frozen picture of exactly what a piece of information looked like at one specific moment in time, saved separately so it can be looked at again later even if the original changes or is lost. Rather than making a whole new copy of everything each time, a well-built system is clever about only saving what changed since the last snapshot, which makes snapshots fast and cheap enough to take often, sometimes many times every hour.
4 A safety log for every single change
Before a well-built system changes any piece of information, it first writes down, in order, exactly what it is about to do, in something called a write-ahead log, often shortened to WAL. If the computer crashes in the middle of making a change, it can read that log back afterward and either finish the change correctly or undo it cleanly, instead of being left with information that is half correct and half wrong. This is the same idea as writing down a plan before starting a task, so that if something interrupts you half way through, you can look at the plan and know exactly where you left off.
5 Actually testing it, on purpose, before an emergency happens
A backup, a snapshot, or an extra copy that has never actually been used to restore a real, working website is not proven to work at all — it is only assumed to work. A well-run system is tested for real, on a regular schedule, by actually restoring service from a backup and checking that it works correctly. Some companies go further and deliberately switch off a real, live building on purpose, just to prove that everything keeps working without it, rather than waiting to find out for certain during a real emergency.
6 High availability — all of it, working together
High availability is the name for the end result of everything above being done well at the same time: a website that stays working correctly almost all of the time, because it was deliberately built with no single point of failure anywhere in it. It is not one single tool or one single setting. It is what you get when redundancy, sharding, snapshots, the safety log, and real testing are all built in together, and kept working, for as long as the website exists.
This is exactly the stage a growing website reaches once it has outgrown a single server. See how a real website's hosting grows, one real, concrete step at a time, from one server all the way to this point.
See how a website's hosting grows over time