APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

A Tale of Two Routers

© June 2010 Anthony Lawrence

Early next week, I plan on correcting a mistake I made some time ago. This will take a bit of explaining, so bear with me.

A customer has a branch office store. Originally, back in the mists of history, a T1 line connected the two stores telephone systems and provided some data transfer. I can vaguely remember tying into those data channels many years ago. That was just dumb terminals talking to the Unix box at the main store; we did it through Digiboard multiport serial boxes.

Time went on, and pcs replaced terminals. It was still telnet across the T1, but it evolved to TCP/IP and of course a few shared folders were added. To my surprise, this didn't choke the data lines until the Internet started to be important. The pcs at the branch stores were getting their Internet access from the main store through the T1, and it was too slow.

To fix that, we gave the branch its own internet access, and that's where I made the mistake.

The pcs in the branch were currently getting DHCP from the router in the main store. Of course that gave them that router as their default gateway and we needed them to have their local router serve that function. There are a few things I could have done at that point.

I could have put the branch pcs in a different subnet and routed them through the T1 for access to the original LAN. That would have required more advanced routers than we were currently using.

I could have put those pcs into another subnet and created a VPN back to the store, bypassing the T1. This was easy enough to do, but the owner didn't like it because Internet access was far from rock solid then and he rightfully felt that the T1 was much more reliable. He still feels that is true, and although the Internet has become more reliable in the passing years, he's right.

I could have put static ip addresses and static gateways on all the pcs at the branch. That's annoying, so I went for the only other choice.

I enabled DHCP on their local router.

Of course I used a different scope. My reasoning went like this: DHCP works by the pcs broadcasting a request to be given an address. A server that can hand out addresses responds, and whatever response the pc sees first is what it will take.

There's nothing wrong with having two DHCP servers. People will insist otherwise, but they are wrong. Another server to failover is fine. You'd usually configure them with different scopes as I did, but it is even possible to configure the same scope. That is is unusual and tricky stuff though; usually you'd split the scope.

Of course in the usual case, you'd be handing out the same gateway and DNS from each server, but that's not what I was doing. What would prevent a pc in the branch from getting its address and gateway from the main store or vice versa?

Well, the T1 should keep that in order, I thought. Being a relatively slow path, the local pcs should get the local address offer before seeing an offer from the branch and vice versa.

And so it seemed to be. I deployed the router, rebooted the pcs, and yes, they got the gateway I wanted them to have. Life was good.

The first hint of trouble didn't turn up until we deployed OpenDNS in the main store. I configured their router to use OpenDNS, but we noticed that a few machines were not being restricted. I was shocked when I saw why - they were getting DHCP from the branch!

How could that be? Well, maybe at some point we had to reboot the router just as these machines were booting. Not getting a local response, of course they took what they got from the branch. Simple fix, I thought, repair the ip or reboot.

Nope. Did not work. Well, ok, that makes sense, it has an ip and it wants to renew with that same ip if it can, so it must prefer that server.

I tried giving it a static ip from the scope I wanted and then changing back to DHCP. Nope, it went right back to the branch. Does it store this stuff somewhere? I don't know. Is there any place you can tell it to prefer one server over others? Not that I could find.

It seems that the only way to force this was to disconnect the branch router and renew ip's while it was non-functional. That's a long way from a good solution.

Upon further investigation, I found one pc at the branch that had latched on to the main router at some point. Enough - this has to be fixed.

So, next week I will be putting static info on the branch office pcs. Then I will disable DHCP on its router and reorient the main office pcs that are incorrectly using it. That will put things back as they should be.

Got something to add? Send me email.

(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> Two DHCP servers cause a problem


Inexpensive and informative Apple related e-books:

Take Control of IOS 11

Take Control of Numbers

El Capitan: A Take Control Crash Course

Take Control of the Mac Command Line with Terminal, Second Edition

Take Control of iCloud

More Articles by © Anthony Lawrence

Thu Jun 17 14:45:15 2010: 8707   NickBarron


So the OpenDNS was not related to the issue, it just so happens that it was what you were doing when you discovered the problem?

Thu Jun 17 15:03:17 2010: 8708   TonyLawrence


Correct. When I examined the machines to see why they were not using the DNS I wanted them to use, I saw that they had obtained their ip from the branch router which had not yet been reprogrammed to use that.

Thu Jun 17 15:58:00 2010: 8709   DaveGillam


Is there not a firewall between the branch and main networks? I'd think doing a firewall block on DHCP between networks (and just for the record, blocking DHCP from the Internet) would be a simple solution to this.

Thu Jun 17 16:38:56 2010: 8710   TonyLawrence


Why would you expect a firewall? It's the same LAN subnet.

It would have to be a firewall transparent bridge. Yes, we could do that but there are only a few pcs in the branch.

Thu Jun 17 17:40:35 2010: 8711   DaveGillam


Ah, size... ok, for only a handful of PCs, I guess it's not such a big thing. I tend to work in larger environments, so segmenting sites is a big thing for security (don't want a virus infection at one site affecting another site, etc). Also, I assume that the DHCP server your customer is using cannot serve different pools of IPs depending on the source client. That would be too easy (if request if coming from siteA, use poolA, siteB = poolB, etc.).

Thu Jun 17 18:12:13 2010: 8712   TonyLawrence


My druthers would be to put it on a different subnet, but he won't go for the VPN (and I can understand that) so this is the simplest thing.

Fri Jun 18 01:13:10 2010: 8713   Sledge


Are the T1 routers configured to pass broadcast traffic to the other network segment? If so, is there a specific need for that? Certainly what you propose solves the problem, however, moving the branch to another network (subnet) and configuring the firewall with a static route to the main location using the IP of the T1 router would provide the solution you want. I don't recall working with a router that would pass the DHCP discovery packet since it is addressed to IP and FF:FF:FF:FF:FF:FF MAC.
I must have missed something in the description.

Fri Jun 18 12:41:56 2010: 8714   TonyLawrence


Hmm.. good point -I tend to forget about that because I dont have any involvement with the T1.

But the statics are done, so no need now.

Fri Jun 18 14:25:32 2010: 8715   AndrewSmallshaw


I read this last night and it wasn't making any sense whatsoever but it was late, I was tired and I assumed I must be overlooking something obvious.

My initial thought was the same thing Sledge has commented on - why on Earth would the router be forwarding DHCP broadcasts? Then this morning I noticed something else - you can't configure a local subnet because the router doesn't allow it.

That means one thing - what you have is not a router by definition. It is a bridge. The local network is only half the subnet and the difficulty lies in the fact you have no control over the other half.

Without extra hardware or access to the remote DHCP server reconfiguring the machines to static IPs is probably the option I woudl have gone for but it is hardly elegant. Throw in a new subnet and you get localised control, the lack of which is essentially what you are fighting against here.

But as you say it is done now so there isn't much point altering it now for the hell of it.

Fri Jun 18 14:30:55 2010: 8716   TonyLawrence


No, no, no.

Yes, of course the T1 is acting as a bridge. But I *do* have a local router and I could configure a subnet there. It would be faster and better to do that and pass it through a VPN.

But the owner still doesn't trust the internet to stay up.

Fri Jun 18 14:53:17 2010: 8717   AndrewSmallshaw


I understood the presence of the real router for Internet access. I take it this is a domestic grade model - one WAN interface and one LAN interface (albeit probably expanded to a few ports with a built in switch). With two real routers (even decent domestic units) you could put both on the same local subnet and configure a route on the one that would become the default gateway to the other one. You'd probably want to activate NAT on the T1 router so that everything on the local subnet appears to be coming from within the remote network once it gets there. Assuming that the routers support even that of course.

I know I don't need to explain this to you it is just the topology explanation was less than clear to begin with.

Fri Jun 18 14:58:54 2010: 8718   DaveGillam


At risk of beating a dead horse....
I think Andrew, et al, is talking about doing the following:

InternetA - firewall/NATrouter - private subnetA - firewall/router - private subnetB - firewall/NATrouter - InternetB

subnetA = 10.1.x.x (static route to 10.2.x.x = internal router to other site)
subnetB = 10.2.x.x (static route to 10.1.x.x = internal router to other site)

The two subnets get their own DHCP pools.
Traffic between the two subnets stays on the private T1. (no VPN through Internet needed)
Traffic within a subnet stays local.
Other traffic goes via Internet.

Fri Jun 18 15:02:44 2010: 8719   TonyLawrence


Yes. That's why I said "That would have required more advanced routers than we were currently using."

Fri Jun 18 15:06:01 2010: 8720   TonyLawrence


And then, after doing this, their existing router failed and I replaced it with something that could do exactly that :-)

But I'm not going to bother now. The whole place is in flux because of new software firing up on Monday so it's not the time to be changing topology.

Fri Jun 18 18:51:23 2010: 8723   anonymous


Re use of "scope," perhaps this means your point of view or are you using an instrument?

Fri Jun 18 19:00:45 2010: 8724   TonyLawrence


No. (link)

Fri Jun 18 22:14:33 2010: 8726   Patric


Transparent bridges (I'm an OpenBSD and PF fan, but whatever floats your boat) have huge benefits in keeping traffic you don't want going over the relatively low-bandwidth connection. I'm leaning towards managing them from a serial port (you probably have all the hardware laying around that paticular site to do that) as installing a 3rd nic in one (which is cleanest, IMO, I am aware that one or the other nic sould have an ip and run sshd on that IP, but I don't find that very clean in terms of pf rules). Anyway, being able to watch traffic go across that connection at the outset of this venture would have kept you from leaving a dodgy setup to start out with. Also, your wording on dhcp servers is pretty scary, there are some horrible implementations, and do not expect them all to act and play nice, stating that it "works just fine" is a bit of an exageration.

Mon Jun 28 01:05:10 2010: 8751   MattJohnson


Thanks for your story. It's nice to find others that deal with the same stuff I do. I'm wondering what OS were the clients and router using? The example you gave on clustered DHCP servers was for Windows Server 2008, so I was just wondering was the router at the client site running Windows Server 2008? As I was reading I was thinking you were using one of the many types of embeded Linux routers. Have a good one. -Matt

Mon Jun 28 01:24:30 2010: 8752   TonyLawrence


Mulitech routers. Win Xp clients, 2003,2008 and Unix servers.

Sat Jul 17 08:21:11 2010: 8824   anonymous


I know its probably too late since you went and configured all of the branch offices with static IPs. However an easier solution would have been to use something called a MAC reservation (host declaration) on your DHCP servers. This would ensure that the PCs in the other branch would always receive the required information (different Gateway but same dns) no matter which DHCP server they were talking to. This gives you not only failover for the DHCP server, but also uses the faster local router for internet access.

Sat Jul 17 09:55:42 2010: 8825   TonyLawrence


You can do MAC reservation on those simple rioters, but can't specify specific gateways, but yes with better dhcp servers, that would work.


Printer Friendly Version

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us

Printer Friendly Version

One in a million is next Tuesday. (Gordon Letwin)

Linux posts

Troubleshooting posts

This post tagged:



Unix/Linux Consultants

Skills Tests

Unix/Linux Book Reviews

My Unix/Linux Troubleshooting Book

This site runs on Linode