Not surprisingly, advanced routing is required on a multi-WAN system. Like most seasoned system administrators, the route command is part of the standard tool set. With multi-WAN, you need to forget about using route and start becoming friends with the ip route command! Every WAN interface on a multi-WAN system has its own routing table. The default table (shown in the “route” command) is just not relevant for WANs in multi-WAN. We will start to dig a bit deeper below.

The following is the output from running the old route command on a simple ClearOS gateway:

# route
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
206.248.154.122 0.0.0.0         255.255.255.255 UH    0      0        0 ppp0
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth2
0.0.0.0         206.248.154.122 0.0.0.0         UG    0      0        0 ppp0

There's a LAN network on eth2/192.168.2.x and a PPPoE/DSL connection on ppp0. Here is the same routing table using the ip route command:

# ip route
206.248.154.122 dev ppp0  proto kernel  scope link  src 69.165.220.249 
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.1 
default via 206.248.154.122 dev ppp0 

As soon as we have more than two WANs, routing becomes a little more complicated. So where do routes for each WAN interface go? The powerful (and sparsely documented) ip command is the key. With 2 WANs running, you can list all the routing tables using ip rule:

# ip rule
0:    from all lookup 255 
50:   from all lookup 50 
100:  from all fwmark 0x8000 lookup 100 
101:  from all fwmark 0x8001 lookup 101 
200:  from 69.165.220.249 lookup 200 
201:  from 69.165.220.248 lookup 201 
250:  from all lookup 250 
32766:  from all lookup main 
32767:  from all lookup default 

For comparison, take a look at the output from the ip rule command on a simple 1-WAN gateway:

# ip rule
0:	from all lookup 255 
32766:	from all lookup main 
32767:	from all lookup default 

The first column is the table number. For now, we are interested in the tables below:

Using the ip command again, you can examine the routing table for WAN #1 / ppp0:

# ip route show table 100
206.248.154.122 dev ppp0  proto kernel  scope link  src 69.165.220.249
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.1 
default via 206.248.154.122 dev ppp0 

The routing table for WAN #2 looks similar, but ppp1 information is used:

# ip route show table 101
206.248.154.122 dev ppp1  proto kernel  scope link  src 69.165.220.248
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.1 
default via 206.248.154.122 dev ppp1 

Table 250 is an important one. It shows the default route including the weight for each of the WANs:

 # ip route show table 250
 default  proto static 
   nexthop via 206.248.154.122  dev ppp0 weight 2
   nexthop via 206.248.154.122  dev ppp1 weight 1

When you add routes to a multi-WAN system, it is necessary to add routes to the routing table for each WAN interface. Fortunately, the ClearOS firewall simplifies the process. In our example, we are going to add a route for 192.168.55.0/24 on ppp1:

# ip route add 192.168.55.0/24 dev ppp1

Our default routing table looks like:

# ip route
206.248.154.122 dev ppp0  proto kernel  scope link  src 69.165.220.249 
206.248.154.122 dev ppp1  proto kernel  scope link  src 69.165.220.248 
192.168.55.0/24 dev ppp1  scope link 
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.1 
default via 206.248.154.122 dev ppp0 

Sure enough, our route has been added to the default table. Unfortunately, the routing table for each of the WAN interfaces has not been updated:

# ip route show table 100
206.248.154.122 dev ppp0  proto kernel  scope link  src 69.165.220.249 
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.1 
default via 206.248.154.122 dev ppp0 

For all intents and purposes, our 192.168.55.0/24 route is not active. If you restart the firewall, it will automatically rebuild the multi-WAN routing tables:

# service firewall restart
# ip route show table 100
206.248.154.122 dev ppp0  proto kernel  scope link  src 69.165.220.249 
192.168.55.0/24 dev ppp1  scope link 
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.1 
default via 206.248.154.122 dev ppp0 

If you are adding static routes to your ClearOS system, please make sure you follow the configuration the Managing Static Routes document. search?q=clearos%2C%20clearos%20content%2C%20dev%2C%20apps%2C%20maintainer_dloper%2C%20maintainerreview_x%2C%20keywordfix&btnI=lucky