How redundant is your network, really?
The well publicised GoDaddy outage on Sept 11 this year should be a huge wakeup for everyone to find out, just how good their network is designed.So, you have multiple data links to multiple carriers, redundancy, failover, the works, even a wireless/microwave link in case someone digs up the entire block including your fibre. You've also got dual power supplies in your routers, switches, and servers, even redundant routers and switches, they are all protected by dual channel mains supplies, each with their own dedicated UPS and battery bank, both connecting into the grid by two separate diverse paths, backed up by two generators.
You can go to bed tonight and sleep easy can't you... Or can you?
A bored shitless teenager, after getting banned from some gaming server on your network, has decided to seek revenge, so initiates a more, real life battle, he attacks the name servers that host the game server network since they may be slightly easier to take down, than the actual server.
Most kids don't care about collateral damage, then again, maybe they are smarter than many give them credit for, attacking the game server, only peeves off the gamers, attacking and taking out DNS peeves off the network admins for affecting the entire network, that would be a quick way to having the ISP kick off the game server, and with word of mouth, that game network soon becomes known as a high risk target, and may find it very hard to get another network to stick their gear in.
OK, back to our story, so, your primary DNS server 10.1.1.1 is massively DDoS'd, but wait, your secondary slave server should respond and save the day, so no problems, right?
Oh, you umm, what? You made 10.1.1.2 as your slave DNS server? So, both your primary and secondary DNS servers are in the same /24 netblock route path... and you wonder why not only is the game network status tango down, but so is your domain, and the domains of every single customer on your network, yes, every single one of them, all because you ignored DNS 101.
Lets not laugh at the above scenario, it happens all too often - as GoDaddy just found out, still, not as much as it could though, and that's mostly because most of us are not targets at all, let alone high risk targets. It is allowed to happen because some network operators just chuck it all together and hope it works and stays working, if you have two or more diverse paths, use them to your maximum advantage.
Default route and prefer for example 10.1.1.0/24 via Link A, so your primary name server along with any other critical stuff like mail servers and company web site will always use link A for internet traffic.
Then default route and prefer 10.1.2.0/24 via Link B your secondary name servers, your secondary backup MX mail server, and whatever else is secondary importance, then let BGP use best path for all other ranges.
Ideally your BGP configuration would be such that if one link is down, the other takes up slack. find a BGP expert to help you there, trust me, I'm not one
Further to this, it's also advantageous to have additional slaves (and secondary MX's) geographically located, if you're a national network with PoP's in different states, that's good enough, but personally, I prefer my external DNS slaves in another country, for instance, my private domain has slaves in the U.S. and U.K, and I have been considering adding a third slave, which will be in Frankfurt, Germany, (given it's fast becoming a European super hub), so now I'm pretty much covered.
There is no reason a service provider can not also do same, dedicated servers, in well known and reliable overseas data centres, will pay for themselves in no time.
One thing I do see a lot of these days, with VPS's becoming cheap, a lot of people go register a domain, get a VPS host which is issued with only one IP address, then in their registrars settings they set ns1 and ns2 to the same damn IP on the same damn server, seriously? I mean, WTF? If your only copy of bind exits (because you're probably running a debian or centos image that's so old and outdated using exploitable versions of softwares), or your VPS goes belly up, even with an external secondary MX configured, how the hell do you expect DNS lookups to succeed to lookup the mail backup I'll never know. Folks, this is DNS #1, never ever do it, its worse than putting both separate servers in the same immediate IP range.
There are services out there that offer secondary NS/MX services, google them, hell, I'll even do it for you for free if we have ever met, MSN'd/IRC'd/Tweeted/emailed, or even friended onfuckbook facebook, (OK so that's not so likely given I rarely use it given my dislike of that privacy invading cesspit of a ...REDACTED FOR LEGAL REASONS...) and, of course, if you're nice to me
If you screw up your DNS, your vanished off the face of the net, and unreachable, if not immediately, within 24 hours, which is the typical TTL (time to live) for DNS records. Some however use less than 24 hours, those using cheap load balancing methods may set Zero on the record for your MX servers, and require DNS to be available at all times for mail to succeed. You should also never allow your server to be an open recursive server, serving anyone, this can introduce local cache poising risks.
DNS is not something to be taken lightly, if you do not understand it, don't touch it at all, or, be prepared to spend a good few hours and learn it, it's not really that hard, but be warned, there is much, much wrong and outdated information found via search engines, too many blogs are written by people with three hours experience, and a hell of a lot of assumptions and guesses.
Most kids don't care about collateral damage, then again, maybe they are smarter than many give them credit for, attacking the game server, only peeves off the gamers, attacking and taking out DNS peeves off the network admins for affecting the entire network, that would be a quick way to having the ISP kick off the game server, and with word of mouth, that game network soon becomes known as a high risk target, and may find it very hard to get another network to stick their gear in.
OK, back to our story, so, your primary DNS server 10.1.1.1 is massively DDoS'd, but wait, your secondary slave server should respond and save the day, so no problems, right?
Oh, you umm, what? You made 10.1.1.2 as your slave DNS server? So, both your primary and secondary DNS servers are in the same /24 netblock route path... and you wonder why not only is the game network status tango down, but so is your domain, and the domains of every single customer on your network, yes, every single one of them, all because you ignored DNS 101.
Lets not laugh at the above scenario, it happens all too often - as GoDaddy just found out, still, not as much as it could though, and that's mostly because most of us are not targets at all, let alone high risk targets. It is allowed to happen because some network operators just chuck it all together and hope it works and stays working, if you have two or more diverse paths, use them to your maximum advantage.
Default route and prefer for example 10.1.1.0/24 via Link A, so your primary name server along with any other critical stuff like mail servers and company web site will always use link A for internet traffic.
Then default route and prefer 10.1.2.0/24 via Link B your secondary name servers, your secondary backup MX mail server, and whatever else is secondary importance, then let BGP use best path for all other ranges.
Ideally your BGP configuration would be such that if one link is down, the other takes up slack. find a BGP expert to help you there, trust me, I'm not one
Further to this, it's also advantageous to have additional slaves (and secondary MX's) geographically located, if you're a national network with PoP's in different states, that's good enough, but personally, I prefer my external DNS slaves in another country, for instance, my private domain has slaves in the U.S. and U.K, and I have been considering adding a third slave, which will be in Frankfurt, Germany, (given it's fast becoming a European super hub), so now I'm pretty much covered.
There is no reason a service provider can not also do same, dedicated servers, in well known and reliable overseas data centres, will pay for themselves in no time.
One thing I do see a lot of these days, with VPS's becoming cheap, a lot of people go register a domain, get a VPS host which is issued with only one IP address, then in their registrars settings they set ns1 and ns2 to the same damn IP on the same damn server, seriously? I mean, WTF? If your only copy of bind exits (because you're probably running a debian or centos image that's so old and outdated using exploitable versions of softwares), or your VPS goes belly up, even with an external secondary MX configured, how the hell do you expect DNS lookups to succeed to lookup the mail backup I'll never know. Folks, this is DNS #1, never ever do it, its worse than putting both separate servers in the same immediate IP range.There are services out there that offer secondary NS/MX services, google them, hell, I'll even do it for you for free if we have ever met, MSN'd/IRC'd/Tweeted/emailed, or even friended on
If you screw up your DNS, your vanished off the face of the net, and unreachable, if not immediately, within 24 hours, which is the typical TTL (time to live) for DNS records. Some however use less than 24 hours, those using cheap load balancing methods may set Zero on the record for your MX servers, and require DNS to be available at all times for mail to succeed. You should also never allow your server to be an open recursive server, serving anyone, this can introduce local cache poising risks.
DNS is not something to be taken lightly, if you do not understand it, don't touch it at all, or, be prepared to spend a good few hours and learn it, it's not really that hard, but be warned, there is much, much wrong and outdated information found via search engines, too many blogs are written by people with three hours experience, and a hell of a lot of assumptions and guesses.
Trackbacks
No Trackbacks
Comments
Display comments as Linear | Threaded