The topic of this post is the evaluation and testing of open source network routing software for a wireless MANET. This post describes the selection of an open source network routing protocol, and the implementation of a network routing software test framework to test this protocol using the Extended Mobile Ad-hoc Network Emulator (EMANE) network emulation framework. Testing is performed using standard text based Linux commands issued within a terminal session coupled with custom visualization tools.
A mobile ad-hoc network (MANET) is a continuously self-configuring, infrastructure-less network of mobile devices connected with wireless links. Each device in a MANET is free to move independently in any direction, and will therefore change its links to other devices frequently. Each node must forward traffic unrelated to its own use (forwarding), and therefore must be a router.
The primary challenge in building a MANET is equipping each device to continuously maintain the information required to properly route traffic. There may be one or multiple different transceivers between nodes, each representing a. hop. This results in a highly dynamic, autonomous topology.
The use of open source MANET network routing software in embedded systems based communications equipment is highly prevalent. For the type of wireless network we are targeting, connections can established point-to-point between radios, but it does permit both data losses and small duration disruptions.
The EMANE test framework is used to evaluate a selected network routing protocol, verify it’s basic functionality relative to our requirements, to perform analysis for various test conditions, and perform network and configuration optimization.
Network Routing Protocol Types
A routing protocol uses software and routing algorithms to determine optimal network data transfer and communication paths between network nodes. Routing protocols facilitate router communication and overall network topology understanding. A routing protocol is also known as a routing policy.
Existing routing protocols in MANETs can be classified into three basic categories as follows.
Figure 1: Classification of Routing Protocols
Proactive (Table-Driven) Routing Protocols
In this family of protocols, nodes maintain one or more routing tables about nodes in the network. These routing protocols update the routing table information either periodically or in response to changes in the network topology.
The advantage of proactive routing protocols is that a source node does not need a route discovery procedures to find a route to a destination node.
On the other hand the drawback of these protocols is that maintaining a consistent and up-to-date routing table can require substantial messaging overhead, which consumes bandwidth and power, and decreases throughput, especially in the case of a very large number of high node mobility.
There are various types of Table Driven Protocols: Destination Sequenced Distance Vector routing (DSDV), Wireless routing protocol (WRP), Fish eye State Routing protocol (FSR), Optimized Link State Routing protocol (OLSR), Cluster Gateway Switch Routing protocol (CGSR), and Topology Dissemination Based on Reverse Path Forwarding (TBRPF).
Reactive (On-Demand) Routing Protocols
For protocols in this category, there is an initialization of a route discovery mechanism by the source node to find the route to the destination node when the source node has data packets to send. When a route is found, route maintenance is initiated to maintain this route until it is no longer required or the destination is not reachable.
The advantage of reactive protocols is that overhead messaging is reduced.
One of the drawbacks of these protocols is the delay in discovering a new route leading to high latency times in route finding. For reactive protocols, a large number of control packets are generated when a link breakout occurs, increasing the congestion in the active route.
The different types of reactive routing protocols are: Dynamic Source Routing (DSR), Ad-hoc On-Demand Distance Vector routing (AODV), Ad-hoc On-demand Multi-path Distance Vector Routing Algorithm (AOMDV), and Temporally Ordered Routing Algorithm (TORA).
Hybrid Routing Protocols
Both of the proactive and reactive routing methods have some advantages and shortcomings. In hybrid routing a combination of proactive and reactive routing methods are used which are better than the both used in isolation. It includes the advantages of both protocols.
As an example, a reactive routing protocol such as AODV can be combined with some proactive features by refreshing routes of active destinations to reduce the delay and overhead and improve the performance of the network and the node.
Hybrid protocols can incorporate the facility of other protocols without compromising its own advantages. Examples of hybrid protocols are Zone Routing Protocol, Hazy Sighted Link State.
MANET Routing Protocol Specifications
The following is a list of the specifications that generally govern the decision on routing protocol suitability:
- The maximum number of nodes in the MANET.
- The maximum latency permitted end-to-end.
- The maximum link outage time duration.
- The maximum hop count permitted (related to maximum latency).
- The maximum data rate required along with host traffic type such as protocol (e.g. UDP protocol) and MTU (e.g. 1500 bytes).
- The maximum percentage of network bandwidth permitted for network management messages.
- The packet delivery ratio required (ratio of packets received/packets sent).
- The mobility model for network nodes (including layout, speed, direction and pause values):
- Random Waypoint Mobility Model (random speed, direction, and pause values).
- Random Direction Mobility Model (avoids clustering of Random Waypoint).
- Random Walk (Brownian motion, random speed and direction, no pauses).
- Gauss-Marcov Mobility Model.
- Other Mobility Models (Manhattan, Freeway Mobile, etc.)
Open Source Routing Software
The following is a list of the most prevalent contemporary open source network routing software available for an embedded Linux distribution. This list is not entirely comprehensive. Some of these protocol stacks combine a number of different routing protocols into a bundle.
- Ad hoc On-demand Distance Vector (AODV)
- BIRD (BGP, Babel, BFD, OSPF, PIM, and RIP)
- Destination Sequence Distance Vector (DSDV)*
- Distance Routing Effect Algorithm for Mobility (DREAM)*
- Fisheye State Routing Protocol (FSP)*
- FRRouting (Quagga fork)*
- Optimized Link State Routing Protocol*
- Quagga (BGP, IS-IS, LDP, OSPF, PIM, and RIP)*
- Zebra (Quagga original)*
* Proactive routing protocols
Our Basic MANET Routing Software Requirements
For our ambitions, the fundamental requirements of the network routing software suite is providing support for the Self-X’s:
- Self Configuration – wireless mesh network formation is automatic without user interaction providing on-the-fly network formation (host-to-host).
- Self Optimization – routing is dynamic and based on link metrics for optimum communications path routing.
- Self Healing – repairing the network due to failures is automatic, using alternate routes on dropped nodes with dropped node recovery.
In addition to the basic Self-Xs, we require the following:
- Solution provided as open source with seamless integration into an embedded Linux based IP stack.
- Provides support for a small to medium sized networks where the node count is 50 or less.
- Provides route repair, suitable network convergence speed, and acceptable user traffic performance in a mobile environment.
- Proactive protocols are favored over reactive ones so that paths are already established for immediate packet delivery to reduce delays and latencies.
- Provides support for host-to-host traffic flows without manual configuration.
Network Routing Protocol Architectures
Most routing protocols explicitly separate the routing and forwarding functions (e.g. OLSR, OSPF, Quagga, Zebra, AODV-UIUC) as shown in the following figure.
Packet forwarding refers to the process of taking a packet, consulting a table (the forwarding table), and sending the packet towards its destination as determined by that table. It is implemented within the Linux kernel and is therefore efficient as fewer packets traverse the kernel to user-space boundary.
Packet routing, on the other hand, refers to the process of building the forwarding table. It is most often implemented in the user space as a daemon process (routing daemon).
Forwarding is a well-defined process performed locally at each node whereas routing involves a complex distributed decision making process commonly referred to as the routing algorithm or the routing protocol.
Some protocols (e.g. BATMAN, and some AODV variants) use custom kernel loadable modules or kernel modifications which are tied to the OS kernel version.
Figure 2: Routing Architectures
Network Routing via OLSR
The OLSR routing protocol is an optimization of the classical link state algorithm tailored to the requirements of a mobile wireless LAN. The key concept used in the protocol is that of multi point relays (MPRs). MPRs are selected nodes which forward broadcast messages during the flooding process.
This technique substantially reduces the message overhead as compared to a classical flooding mechanism, where every node re transmits each message when it receives the first copy of the message.
In OLSR, link state information is generated only by nodes elected as MPRs. Thus, a second optimization is achieved by minimizing the number of control messages flooded in the network. As a third optimization, an MPR node may chose to report only links between itself and its MPR selectors. Hence, as contrary to the classic link state algorithm, partial link state information is distributed in the network. This information is then used for route calculation.
OLSR provides optimal routes in terms of reducing the number of hops in a link. The protocol is particularly suitable for large and dense networks as the technique of MPRs works well in this context.
OLSR makes use of multi point relays (MPRs) which act as intermediate routers in route discovery procedure. OLSR MPRs limit the number of message retransmissions during the necessary flooding operations.
OLSR performs packet forwarding, neighbor sensing, and topology discovery procedures.
OLSR uses four message types:
- Hello message.
- Topology control.
- Multiple Interface Declaration (MID).
- Host and Network Association (HNA).
OLSR is the “standard” and most widely used routing protocol used for embedded communications systems. It is the most distinguished and dominant routing protocol for MANETs, for both commercial and military use.
OLSR is a proactive open source network routing software suite for embedded Linux. The current version implemented is 0.9.9
(described as the version 1 track, i.e. OLSRv1). The current RFC is RFC 3626 for the OLSRv1 track.
A version 2 track of OLSR exists as OLSRDv2 per RFC 7181. This build provides a dimensionless metric for determining link metric based routing.
Advanced research is ongoing modifying and tuning OLSR for the ambitions of MANET as a function of node speed: VANET (vehicular), and FANET (flying).
OLSR Customized Message Timing
OLSR does neighbor-sensing by emitting regular HELLO messages. In the absence of HELLO messages for a specified duration is used as an
indication of link failure.
A node announces its link-set by flooding TOPOLOGY CONTROL (TC) messages. The default interval is 2 seconds for HELLO messages, and 5 seconds for TC messages.
The HELLO validity time is set to 20 seconds for a node disconnection.
Route updates in OLSR occur by default upon reception of 10 messages, or 20 seconds for a 2 second HELLO interval.
For our research, we increased rate of HELLO message to 1/2 second intervals to increase system responsiveness (at the expense of network bandwidth). Route updates then occur at 5 second intervals, rather than 20 second intervals.
The OLSR daemon olsrd executes only within a radio, and not the host device acting as the source and sink of user data.
Host and Network Association (HNA) is an option to OLSR for telling all nodes on the MANET the IPs of the hosts that a radio node, running olsrd, manages.
A node can announce itself as a gateway from the OLSR network to other networks using HNA messages.
HNA is used in the research to allow radio nodes to announce the IPs of the host connected to the primary wired data interface of the radio (eth0). This is necessary for host-to-host data transfers.
Using HNA message, sufficient information is provided to all OLSR recipients to construct an appropriate routing table for host IPs.
OLSR Expected Transmit Count (ETX)
In dense networks, there may exist a number of different paths to reach a destination with different cost metrics. Therefore some quality of service (QoS) constraints are necessary to help choose a better route to a destination. QoS considerations for a routing protocol is essential for successful communication of nodes in the network. Typical QoS metrics includes throughput, packet loss, jitter, delay, error rate, etc.
OLSR version 1 (referenced also as simply OLSR-RFC) makes routing decisions simply to minimize the hop count. This may not be an effective QoS metric to use for managing wireless network links for certain network topologies and conditions. For example, this method will stick with a single hop connection between two nodes even if the link is extremely degraded while alternate multi-hop routes exist with significantly less or even zero packet drops, that is better overall packet completion rates (PCR) with longer latencies.
In basic OLSR, the QoS constraint is hop count whereas in OLSR with ETX extensions enabled (OLSR-ETX), it is expected transmission count.
Expected Transmit Count (ETX) extensions to OLSR provide for routing decisions to be made dynamically on link metrics rather than just minimizing the hop count.
ETX is an estimate of the number of transmissions expected to successfully transmit a packet with acknowledgment to a neighbor.
ETX consists of two components:
- Link Quality (LQ): If we receive from a neighbor to ourselves 7 out of 10 packets = 0.7 = 70%, this probability is what we call the Link Quality
- Neighbor Link Quality (NLQ): In the opposite direction, the neighbor will tell the node the number of packets received as a percentage from this node, called Neighbor Link Quality (node-to-neighbor).
ETX is the inverse of the probability of successful transmission per the following equation:
ETX Calculation: ETX = 1 / (LQ) * (NLQ)
With ETX extensions enabled, optimum routes will be selected by OLSR to minimize the sum total of all ETX values for the available routes. An ETX value of 1.0 is a zero loss network link or route. Any larger number represents a more loss prone link that is more costly to use.
OLSR Advanced Research
ETX is an efficient measure of the quality of a link in an wireless ad-hoc networks, but it may not be reactive enough to cope with very fast moving dynamic wireless ad-hoc networks, such as UAV networks. OLSR is the subject of a large number of research efforts in government and industry.
Research areas include but are not limited to the following:
- Predictive-OLSR: a custom OSLR release that exploits GPS information, where the weight of the ETX metric is scaled by a factor that takes into account the relative speed between the nodes (speed weighted ETX).
- Location awareness favoring forwarding nodes which are closer.
- Advanced security against OLSR specific security threats and cyber analysis.
- Energy efficient implementations.
- OLSR version 2 as a complete rewrite to OLSR version 1 which provides a dimensionless link metric.
- Link metrics based on Estimated Transmit Time, Estimated Throughput, etc.
- Leveraging host traffic for routing decisions (rather than just OLSR control messages).
- The FishEye Routing Protocol Extension for OLSR for bandwidth efficiency.
- Various ETX link quality algorithms (fixed and floating point).
- Overlays on OLSR to support DTN ambitions (store and forward capabilities).
Introduction to EMANE
EMANE is a next-generation framework for real-time modeling of mobile network systems created and maintained by Adjacent Link. The EMANE components focus on real-time modeling of link and physical layer connectivity so that network protocol and application software can be experimentally subjected to the same conditions that are expected to occur in real-world mobile, wireless network systems.
The EMANE emulator consists of three major subsystems, as shown in the following figure:
- Emulation Processing
- Application/Emulation Transport Boundary Processing
- Event Processing
Figure 3: EMANE Emulator Subsystems
The EMANE framework is particular well suited for evaluating real time performance of open source network routing software. Using EMANE, the actual software to be used for an embedded system can be executed real time on a Linux based hardware platform. In a practical sense, network emulation using EMANE requires the use of light weight virtualization such as Linux Containers (LXC) to provide independent IP stacks.
EMANE as a network emulator is a real-time application running native Linux applications within the context of a Linux container (LXC).
EMANE is used extensively in academic and industry circles to test out various routing algorithms and implementations.
EMANE is C++/XML based.
The following are some of the benefits of the use of EMANE for network emulation:
- Uses virtualization to emulate any number of network nodes.
- Scalability is limited only to platform resources.
- Cost effective.
- Real-time and faithful to protocols and implementations (overcomes limits on Modeling and Simulation platforms).
- Supports various common radio models (RF Pipe, IEEE 802.11abg, TDMA, LTE).
- Open Source with a large evolving code base.
- Supports Centralized or distributed deployments.
- High-fidelity emulation and/or actual implementation of lower-layer protocols & services.
- Government and Industry standard.
- No RF emissions.
- Does not depend on uncontrollable factors such as wireless channel interference.
- Includes mechanisms to add interference to abstracted wireless channel.
- Includes mechanisms for frame injection via abstracted wireless channel.
- Includes mechanism for driving the emulation with scripted or manual location, pathloss, or antenna position events.
- Extendable by way of either custom shims or the development of custom models.
- Allows for communications stack layer development in advance of a fully functioning PHY Layer of an SDR.
- Integrates with the open source CORE framework to provide a GUI tool for defining network topologies.
EMANE and LXC
EMANE depends upon virtualization to provide separate network IP stacks to implement independent Network Emulation Modules (NEMs). This virtualization is most often implemented using LinuX Containers (LXCs).
LXC is an operating system-level very light weight virtualization method for running multiple isolated Linux systems (“containers” or “guests”)
on a single control host workstation (i.e. the LXC “host” computer).
Linux containers provide independent network IP stacks for executing networking applications under a single version of the Linux kernel using virtual network interfaces.
EMANE when based on light-weight Linux containers facilitates separate network stack emulation, and is more scalable than a system built on heavier weight virtual machines (VM).
The EMANE binaries require a virtualization framework external to EMANE proper.
EMANE containerized network stacks encapsulate the functionality necessary to transmit, receive, and operate on data routed through the emulation space. This allows for heterogeneous network emulation using a pluggable Media Access Control (MAC) and Physical (PHY) Layer architecture.
EMANE inherently provides a method for changing waveforms.
The EMANE framework is the preferred test environment used widely in academia and industry for evaluating different routing algorithms,
implementations, and performance.
EMANE Receive Power Calculation
For each received packet, the EMANE physical layer computes the receiver power associated with that packet using the following calculation:
rxPower = txPower + txAntennaGain + rxAntennaGain − pathloss
pathloss is the pathloss between transmitter and receiver determined based on the EMANE propagationmodel configuration parameter as either precomputed or freespace.
With zero values for txPower, txAntennaGain and rxAntennaGain, rxPower equals the pathloss.
EMANE receive sensitivity is based on the following calculation:
rxSensitivity = −174 + noiseFigure + 10log(bandWidth)
- bandWidth is the transmit waveform bandwidth.
- noiseFigure is defined by the configuration parameter systemnoisefigure as receive chain uncertainty.
If the rxPower is less than the rxSensitivity, the packet is silently discarded.
Using the default noiseFigure of 4 dB and a bandWidth of 10 MHz, rxSensitivity is -100 dB.
EMANE Propagation Models
Pathloss within the EMANE physical layer model is based on location or precomputed pathloss events.
- Freespace: pathloss is dynamically calculated based on GPS node locations and the emulation is driven with location events. This represents a real world distribution of radios.
- Precomputed: pathloss is precomputed for network links and the emulation is driven with precomputed pathloss events. This allows for more theoretical analysis of routing algorithms.
Both of these propagation models were used for this research.
EMANE RF Pipe Radio Model PCR
The RF Pipe Radio Model Packet Completion Rate (PCR) is specified as a curve defined via XML as a series of SINR values along with their corresponding probability of reception.
The default PCR curve requires a 20 dB SINR for 100 percent loss less data reception, and degrades linearly by SINR from 20 dB to 0 dB as 100 to 0 percent reception.
Using an receive sensitivity of -100 dB with a 0 value for transmit power and transmit and receive antenna gains:
- For a pathloss of 80 dB, 20 dB of SINR, no data losses.
- For a pathloss of 85 dB, 15 db of SINR, 25% data losses.
- For a pathloss of 90 dB, 10 dB of SINR, 50% data losses.
- For a pathloss of 95 dB, 5 dB of SINR, 75% data losses.
- For a pathloss of 100 dB, 0 dB or SINR, 100% data losses.
Our research and testing used these 5 pathloss values as test points to examine network routing behavior when stressed and forced to make dynamic routing decisions as a function of 5 data loss percentages.
The pathloss values per link are specified by direct assignment when using the precomputed pathloss propagation model, or by separation distance when using the freespace propagation model.
EMANE Test Network
Our test network is based on a paired set of two containers representing a logical “node”, as a radio container and a host container as shown in the following figure. This figure shows 2 logical nodes of a 30 node network where the other 28 nodes follow the conventions shown for the 2 nodes.
Figure 4: Test Network
The radio container has EMANE running within along with olsrd, gpsd, and other software components. Each radio container is paired with a host container to act as the source and sink of network traffic.
The host container is connected to the primary eth0 interface of the radio container representing the data interface to the radio.
Each container has a control interface eth1 which is used exclusively for external connections for driving or monitoring the emulation. This control interface is used for SSH user sessions, as the target for EMANE events, and for the EMANE shell (emanesh).
A common Linux bridge device is used to represent the over-the-air (OTA) interface and is driven by the EMANE network emulation stack. This interface is connected to the eth2 interface for all radio containers.
Host-to-host data flows from host container to radio container to over the air use “hostxdata” as IP endpoints.
The following are our testing goals that were evaluated during this research:
- Demonstrate all Self-X’s.
- Host-to-host level data exchanges using dynamic wireless network routing.
- Basic routing functionality using ping as a function of network routing and trace routing.
- Maximum Data Rate Testing for a continuous stream of data using the standard nuttcp and iperf test tools.
- Routing Performance as a function of ETX values dynamically changed by path loss (precomputed or freespace location based events).
- One-Hop Direct Routing.
- Multi-hop Routing based on ETX mechanisms.
- Broken Links and Dropped Node Recovery.
- 30 node network support (both GPS location based freespace and precomputed pathloss propagation models).
The following test methods were used in this research:
- Host names are used to identify network resources and interfaces (e.g. “host1data”) via population of the etc/host Linux file.
- Data transfers use host container IP endpoints to test out the entire emulated wireless network host-to-host.
- Visibility into an LXC Container uses Secure Shell (SSH) connection to the container’s control interface (Linux terminal).
- SSH allows for common Linux functions to be performed, such as dumping the routing table of the radio container using standard Linux commands.
- Dumping of EMANE specific tables is provided via emanesh through the control interface to the radio container (eth1).
- An HTTP interface is provided by olsrd executing within a radio container to view OLSR tables.
- 2D view of network topology is provided using the olsrviewer graphical application.
- Real-time spectrum and waterfall plots are provided via the custom eView application.
- Wireshark can be used to view Ethernet frames on any of the container emulation virtual interfaces between containers.
- Tshark can be used within the container to view container internal interfaces.
OLSR ETX Test Values
For our testing, we choose path loss values for the following ETX values with the corresponding data loss rates as testing standards:
- ETX is 1 – 100% success, no losses.
- ETX is 1.78 – 75% success, 25% losses.
- ETX is 4 – 50% success, 50% losses.
- ETX is 16 – 25% success, 75% losses.
- ETX is 400 – 0% success, 100% losses.
A sample OLSR Links Table dump showing ETX as the Cost column for a 5 node network configuration is as follows:
Local IP Remote IP Hyst. LQ NLQ Cost
10.100.0.1 10.100.0.2 0.000 1.000 1.000 1.000
10.100.0.1 10.100.0.3 0.000 1.000 1.000 1.000
10.100.0.1 10.100.0.5 0.000 0.596 0.635 2.641
10.100.0.1 10.100.0.4 0.000 0.843 0.897 1.320
Viewing Routing Decisions
We were able to view the routing decisions made by olsrd running within each radio container as the emulation was driven by path loss and location based events using both standard Linux terminal based commands as well as custom graphical user interface applications as described in the following paragraphs.
Radio Container Route Table Dump
OLSR populates the Linux kernel routing table with routes to various destinations learned by OLSR messaging. In this test setup, olsrd is bound as a socket to the emane0 interface created by a running instance of the EMANE binary within a radio container. The routing table contents dictate the kernel mode routing of packets based on destination IP.
To view the routing table contents from within a radio container, use one of the following Linux commands within the radio container after establishing an SSH session with the radio container.
- netstat -rn
- ip route
- route -v
For example, the following is a routing table dump for radio #1 for a simple 3 node network with a 2-hop route route route for the Link 1→3 based on ETX values.
The results show that radio #3 is accessible over the air from the emane2 interface of radio #2, that is radio #2 is acting as a forwarding node for radio #3.
Host address for host #3 are on the network IP subnet 10.98.3.0/24. The routing table example shows that host #3 IP addresses are likewise accessible from emane2, the relay or forwarding node in the network.
[user@radio-001 ~]$ route -v
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.100.0.0 0.0.0.0 255.255.255.0 U 0 0 0 emane0
emane2 emane2 255.255.255.255 UGH 2 0 0 emane0
emane3 emane2 255.255.255.255 UGH 2 0 0 emane0
10.98.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.98.2.0 emane2 255.255.255.0 UG 2 0 0 emane0
10.98.3.0 emane2 255.255.255.0 UG 2 0 0 emane0
10.99.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.100.0.0 0.0.0.0 255.255.255.0 U 0 0 0 emane0
10.101.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
188.8.131.52 0.0.0.0 240.0.0.0 U 0 0 0 bmf0
PING measures the round-trip time (RTT) for messages sent from the originating host to a destination computer that are echoed back to the source.
For our network emulation testing, we used the standard PING command to test connectivity and latencies for the various interfaces provided in the radio and host containers.
For testing host-to-host latencies, we used PING with the -R option to display the PING routes.
This example is for a 3 node network with a 2-hop route alternative for 1-hop direct Link as 1→3 based on ETX values.
For this network topology and ETX values, radio #2 will act as a forwarder for communications between host #1 and host #3. Packet forwarding is performed in the kernel based on routing table population, so is very rapid avoiding any Linux kernel to user space activity.
[user@host-001 ~]$ ping -R host3data -c 1
PING host3data (10.98.3.2) 56(124) bytes of data.
64 bytes from host3data (10.98.3.2): icmp_seq=1 ttl=61 time=5.93 ms
RR: host1data (10.98.1.2)
--- host3data ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.936/5.936/5.936/0.000 ms
These results indicate the PING request routing from host#1 to host#2 follows the following network paths:
- The PING REQUEST originate from the host #1 container’s data interface as host1data,
- The PING REQUEST next travels to the radio #1 container’s EMANE interface as emane1,
- The PING REQUEST flows over-the-air using the EMANE framework to radio #2’s EMANE interface as emane2,
- The PING REQUEST is then forwarded by radio #2 to radio #3’s data interface as radio3data,
- The PING REQUEST is then received by host #3, which is the PING REQUEST destination.
The PING REPLY flows through the following network path:
- The PING REPLY originate from the host #3 container’s data interface as host3data,
- The PING REPLY next travels to the radio #3 container’s EMANE interface as emane3,
- The PING REPLY flows over-the-air using the EMANE framework to radio #2’s EMANE interface as emane2,
- The PING REPLY is then forwarded by radio #2 to radio #1’s data interface as radio1data,
- The PING REPLY is then routed to host #1, which is the PING REPLY destination.
The PING duration as 5.93 milliseconds for a two-hop route between host #1 and host #3 represents approximately twice the value of a single hop latency for the direct path between host #1 and host #3.
Similar results occur for a network of 30 nodes that we tested but are omitted in this post for brevity.
The Linux traceroute command prints the route that a packet takes to reach the host, similar to the results in the previous section. This command is useful when you want to know about the route and about all the hops that a packet takes.
The first column in the traceroute output corresponds to the hop count. The second column represents the address of that hop and after that, you see three space-separated time in milliseconds.
traceroute command sends three packets to the hop destination and each of the time values displayed refers to the time taken by the packet to reach the hop.
The following depicts how traceroute command is used to reach the data interface of host #3 from host #1 for a simple test network of 3 nodes with a 2-hop route route for the Link 1→3 based on ETX values assigned.
[user@host-001 ~]$ traceroute host3data
traceroute to host3data (10.98.3.2), 30 hops max, 60 byte packets
1 radio1data (10.98.1.1) 0.244 ms 0.210 ms 0.194 ms
2 emane2 (10.100.0.2) 2.700 ms 3.036 ms 3.128 ms
3 emane3 (10.100.0.3) 5.928 ms 6.226 ms 6.450 ms
4 host3data (10.98.3.2) 6.526 ms 6.685 ms 6.871 ms
The results are similar to the previous paragraph, where radio #2 is acting as a packet forwarder for a two-hop route between host #1 and host #3.
The following depicts how traceroute command is used to reach the data interface of host #1 from host #3 for this same simple test network of 3 nodes with a 2-hop route for the Link 1→3 based on ETX values assigned.
[user@host-003 ~]$ traceroute host1data
traceroute to host1data (10.98.1.2), 30 hops max, 60 byte packets
1 radio3data (10.98.3.1) 0.019 ms 0.007 ms 0.005 ms
2 emane2 (10.100.0.2) 2.667 ms 3.081 ms 3.074 ms
3 emane1 (10.100.0.1) 5.884 ms 6.181 ms 6.376 ms
4 host1data (10.98.1.2) 6.567 ms 6.600 ms 6.751 ms
The results are similar to the previous paragraph, where radio #2 is acting as a packet forwarder for a two-hop route between host #3 and host #1.
OLSR HTTP Interface
olsrd is designed to be a modular an extensible implementation. It features a plugin interface, allowing for developers to extend OLSR operation without interfering with the core code.
For our research, we enabled the txt_info olsrd plugin to olsrd. This plug-in provides an HTTP based interface that allows external access to internal olsrd tables, and allows the user to view all of the olsrd tables using a simple browser interface using the HTTP protocol.
To view all the olsrd tables, we entered into a web browser: http://radioip:2006/all, where radioip is the IP of the radio container’s interface (or host name specified in etc/hosts) used as the EMANE control interface.
The output in the browser will show the links of the node, its neighbors, the topology table, HNA (Host and Network Association) and MID (Multiple Interface Declaration) tables and the actual routing table.
olsrd tables include the following, which can be accessed individually by specifying the listed URL table specification:
- Config: “/config”
- Gateways: “/gateway”
- HNA: “/hna”
- Interfaces: “/interface”
- Links: “/link”
- MID: “/mid”
- Neighbors: “/neigh”
- Routes: “/route”
- Topology: “/topo”
- 2-hop neighbors: “/2hop”
The txt_info plugin allows the user to view real-time the olsrd tables, and ETX values used for route calculations.
As an example, consider the following olsrd table dumps for the sample simple 3 node network with a 2-hop route for the Link 1→3.
Destination Gateway IP Metric ETX Interface
10.100.0.2/32 10.100.0.2 1 1.000 emane0
10.100.0.3/32 10.100.0.2 2 2.000 emane0
10.98.2.0/24 10.100.0.2 1 1.000 emane0
10.98.3.0/24 10.100.0.2 2 2.000 emane0
The routes table of this example illustrates that the route for radio #1 to host #3 (10.98.3.0/24) uses radio #2 as the gateway or relay node. The ETX metric is listed as a value of 2 which corresponds to the lossless 2-hop route from 1->2, followed by the route from 2->3.
Local IP Remote IP Hyst. LQ NLQ Cost
10.100.0.1 10.100.0.2 0.000 1.000 1.000 1.000
10.100.0.1 10.100.0.3 0.000 0.446 0.438 5.093
The links table for radio #1 shows:
- The ETX value for the single hop route to radio #2 as having both a Link Quality and Neighbor Link Quality of 1, which is both links are lossless. This results in an ETX value of 1 for each direction corresponding to a symmetric lossless link.
- The ETX value for the single hop route to radio #3 as approximately 5, where the Link Quality is 0.446 (corresponding to approximately a 44 percent packet completion rate) and a Neighbor Link Quality of .438 (corresponding likewise to approximately a 44 percent packet completion rate).
With the direct link from radio #1 to radio #3 having an ETX value of 5, olsrd routed traffic to the 2-hop route instead which has a composite ETX value of 2, which is less than the direct link.
OLSR Network Topology Viewer
The research effort utilized a customized version of an OLSR 2D network topology viewer provided with EMANE, authored by Adjacent Link, the creator’s and maintainers of the EMANE framework.
The following is an example 2D network topology view for a 3 node network with a 2-hop route for the Link 1→3.
Figure 5: OLSR Network Topology Viewer Example for 3 Node Network
This network topology viewer presents a 2D graphical representation of communications nodes along with the node’s IP associated with the EMANE transport/emulation TAP interface within the radio container, the number of active links per node, and GPS position information such as latitude degrees, longitude degrees, and altitude.
GPS information is displayed only when using the freespace propagation model.
The rightmost table will show ‘red’ grid row for a radio container for which a connection to olsrd was not obtained, otherwise the row will be ‘white’.
The location of a radio on the display does not necessarily correspond to a dynamically assigned GPS location.
Link colors represent link quality but may overlay directed line segments of other links spatially.
Content for this viewer is provided by the following communications interfaces:
- An HTTP connection to the text info plugin for olsrd for obtaining network information.
- A Telnet connection to gpsd for obtaining position information.
Link qualities are shown as a directed link segment between two nodes. This link segment is split into two segments as either the Link Quality (LQ) or Neighbor Link Quality (NLQ).
In the referenced figure, the labels LQ and NLQ are listed from the perspective of Node #1.
The Link Quality says how good a given link between a neighbor and ourselves is in the direction from the neighbor to ourselves.
The Neighbor Link Quality says how good a given link between a neighbor and ourselves is in the direction from ourselves to the neighbor.
The line segment between nodes color is based on the ETX calculation and displayed in separate colors according the to the following color
- LQ >= 1.0: color = ‘black’
- LQ >= .75: color = ‘green’
- LQ >= .5: color = ‘blue’
- LQ >= .25: color = ‘yellow’
- LQ < .25: color = ‘red’
ETX values are obtained by mining information from the running instance of olsrd using the HTTP based external interface.
The rightmost table provides information about each node indicating:
- The EMANE transport/emulation boundary assigned IP.
- The number of OLSR links for this node.
- The latitude, longitude, and altitude assigned to the node.
The number of links for a node is obtained from mining information from the running instance of olsrd running within the radio container using the HTTP protocol interface.
The GPS position is obtained through the external Telnet interface to the running instance of gpsd from within the radio container. It is only populated when the emulation utilizes the freespace propagation model.
The following figure shows the 2D network topology viewer for a network of 30 nodes positioned dynamically according to GPS location using the freespace EMANE propagation model.
The nodes for this figure are separated so that each adjacent single hop route has 75 percent data loss rate (or 25 percent completion rate), which corresponds to a symmetric ETX value of 16 for each directly adjacent link. Nodes that are not directly adjacent, that is spaced further apart, and will have larger ETX values for connected links.
Figure 6: OLSR Network Topology Viewer Example for 30 Node Network
Multi-hop links will have a composite ETX value for all nodes in the route, a summation of the ETX values for all nodes in the route. For example, for this test configuration, the 8-hop route from NEM #1 to NEM #30 (which are the most physically separated nodes in the network topology) has a composite route ETX value of approximately 33.
The Links column of the rightmost table indicates the connection count for each node, which is obtained from the Links table managed by olsrd.
This table column will have a value of 29 for less degraded links, which is for a network topology where nodes are spaced more closely.
Despite not having active connections or links between all nodes from the perspective of the olsrd Links table, the routing tables populated by olsrd within the radio container still contain routes to each node to preserve paths until a disconnection occurs.
EMANE Viewer (eView)
We developed a custom real-time spectrum and waterfall viewer for the EMANE framework as eView. This graphical tool provides an instantaneous and historical view of EMANE over-the-air (OTA) network packets.
For our test configuration, a time-division multiple access (TDMA) channel access method for shared-medium networks using a common center frequency.
To allow for viewing the OTA transmissions from each NEM spatially separately without them laying atop each other on the same center frequency, we utilized an eView option to physically separate each NEM’s OTA activity horizontally to prevent overlaying content.
For example, the following image shows an EMANE OTA capture for a 30 node network, with unicast continuous network traffic being sent from host #1 to host #30, where NEMs are arranged from the leftmost as NEM #1, to NEM #30 as the rightmost.
Figure 7: eView Display of 30 Node Network from NEM #30’s perspective
Each NEM OTA transmission are shown vertically in an isolated manner to prevent display overlay, both the spectrum waveform as a Power Spectral Density (PSD) plot, and the waterfall plot .
This eView option is particular useful when analyzing network routing, where the waterfall view shows the history and frequency of route changes by the network routing software.
For example, in Figure 7, continuous host-to-host data is being sent from NEM #1 to NEM #30, using multiple hop routing. The waterfall viewer shows a 5 hop route at the bottom of the screen as the oldest routing history, and a 4 hop route as the more recent routing decision. The vertical length of each data column indicates the time duration of the route selection, based on the Update Rate setting in use. In our test configuration for olsrd, route changes are not made more frequently than 5 seconds, so the vertical size of the data column before a route change corresponds to a 5 second duration.
In Figure 7, the size of the signal from the perspective of NEM #30, that is the spectrum as NEM #30 views the world. Therefore, the signal level displayed in the spectrum analyzer view and waterfall viewer is the largest for rightmost NEM #30, and decreases to the left of the image as the physical distance increases between a relay node relative to NEM #30.
When operating using the GPS location based EMANE propagation model, eView can act as a virtual NEM and can be targeted by the EMANE event subsystem to assign eView a GPS location. This modality allows eView to operate as a “sniffer” node, which can be moved to any GPS location in the network topology to provide signal measurements from that particular location. In the reference image, eView was positioned 10 meters below the NEM #30 location. When eView is not being operated in “sniffer” mode, it operates in “omniscient” mode, where it provides a global view of the network pathlosses.
Based on values used for tx and rx antenna gains, and tx power, and the PCR curve used with the RF Pipe EMANE Model, 100 percent or zero loss requires a 20 dB SINR margin, which is an -80 dB or greater receive signal level. A 100 percent loss link with 0 dB of SINR margin corresponds to a receive signal strength of -100 dB or less, where -100 dBm is the receiver sensitivity level.
The reference image shows that for this particular 30 node test configuration, that no route in the multi-hop route from NEM #1 to NEM #30 is lossless, with all signals measured below the -80 dB signal level necessary for zero losses.
GPS Position Information
gpsd is a service daemon that monitors one or more GPSes or AIS receivers attached to a host computer through serial or USB ports. It is a common Linux based daemon that operates on embedded systems.
For this research, for network topologies using the EMANE freespace propagation model, gpsd is running within the radio container. gpsd would nominally be interfacing with a USB based GPS device providing data in NMEA 0183 format.
The EMANE event daemon executes within each radio container and converts location events from the EMANE event subsystem into NMEA formatted strings simulating a GPS device.
The OLSR 2D network topology viewer application uses a Telnet interface to gpsd to retrieve position information from it in the simpler JSON format. This information is displayed to the network topology viewer user interface.
EMANE NEM Stack Customization
The EMANE Network Emulation Module (NEM) protocol stack supports customization through the development of custom “shim” layers that exist as part of the executing NEM software stack.
An EMANE shim provides the following functionality:
- Allows monitoring and modification of OTA and cross-layer messaging exchanged between contiguous layers.
- May generate OTA messages for communication with respective shim Layers contained in different NEMs.
- May append and strip layer specific headers.
- May be inserted between the layers of a NEM stack or at the top and bottom of a layer stack.
- Implements the same generic interface as a MAC and PHY which allows insertion between components without requiring those components to have knowledge of their presence.
For this research effort, two custom “shims” were required to customize the EMANE NEM stack.
One custom shim was inserted between the EMANE MAC and PHY framework to process transmit metadata for downstream over-the-air packets. Transmit metadata consisted of transmit attributes including time of transmission start, transmission duration, transmit power, transmit center frequency, etc. This design approach allowed for external to the EMANE stack transmit packet attributes.
An addition custom shim was inserted in the NEM stack below the EMANE PHY layer to customize the EMANE over-the-air network packets for display in the custom eView application.
Network Emulation Hardware Platform and Configuration
The following is the network emulation hardware platform, operating system, and configurations.
The EMANE emulation was performed on a single hardware platform, an Intel NUC 8 Mini Desktop Computer operating under the general CentOS 7 desktop operating system distribution. With Intel’s Hyper-Threading, this quad core platform provided 8 logical cores to the operating system, where each CPU splits each of its physical cores into 2 virtual cores.
The full system specs for the Intel NUC 8 platform are as follows:
- Intel NUC 8 Enthusiast Mini Desktop Computer.
- 3.1 GHz Intel Core i7-8809G Quad-Core.
- 32GB of 2400 DDR4 MHz RAM.
- AMD Radeon RX Vega M GH Graphics (4GB).
The following is a diagram generated by the Portable Hardware Locality (hwloc) application showing the graphical representation of the hardware test platform showing the number of cores, the processing units within each core, the core caches, and the hardware interfaces.
Figure 8: Hardware Platform Topology
For the network of 30 nodes, where each node is a paired host and radio LXC container, 60 total containers are used. The emulation setup will utilize CPU pinning within each LXC to distribute the load from the containers to the platform’s cores to exploit parallel processing.
Empirical Test Results
The following are the test results for a network of 30 nodes, using the EMANE RF Pipe Model for the radio NEM stack, and driving the emulation with either pathloss events for the EMANE precomputed propagation model, or with GPS location based events using the EMANE freespace propagation model. OLSR HELLO messages were generated at 0.5 second intervals, with TOPOLOGY CONTROL messages generated at 2 second intervals. Both ETX and HSA options were used with the OLSR configuration as previously described.
Routing Decisions Timeline
Routing decisions were made at approximately 5 second intervals based on the configured HELLO message rate.
Support for Self X’s
OLSR provided an implementation that satisfied all of the Self X’s: self configuration, optimization, and healing.
Network Formation Timing
Network startup time from zero connections to a network of 30 radios, where each radio has 29 peer connections, is approximately 5 seconds.
A link connection occurs when one HELLO messages is received by a recipient. When a link connection occurs, the route to the destination within the Linux routing table is added on both peers of the connection. For the test network of 30 nodes, all nodes will have 29 connections to all other peers within 5 seconds from a fully disconnected state.
A link disconnection occurs when zero HELLO messages were received by a recipient after approximately 20 seconds based on the text OLSR configuration, which corresponds to approximately 40 HELLO packets not being received. When a link is disconnected, the route to the destination within the Linux routing table is removed on both peers of the link, and no data flows to this link node is possible until a complementary connection is made.
Population of Linux Routing Table Routes
The Linux routing table contents remained populated for all network nodes that at some previous time had a point-to-point connection to allow for routing up until the point where a node is disconnected. This allows packet forwarding under highly degraded link metrics until a full disconnection occurs.
Dropped Node Recovery
Recovery of a dropped node into the network is approximately 5 seconds. For example, for the 30 node network, restarting OLSR within a radio container, would result in recovery of this dropped node within 5 seconds. The ETX values calculated by OLSR for the links to this node process at approximately 5 second intervals from a value of INFINITE to an ETX value 1 in 4 steps over a 20 second span. While the link connection resulting in a Linux table route addition occurs upon reception of a single packet, the link confidence building procedure for a lossless link to an ETX value of 1 is not instantaneous.
QoS Policy for OLSR Traffic
Link routing decisions by OLSR to utilize a multi-hop route rather than a single hop route, or to additional hops relative to an existing multi-hop route, are driven only by data losses of the OLSR control messages.
This is independent of host traffic, such that OLSR control traffic losses drive the ETX mechanisms, and as such the routing decisions.
If a packet retry or QoS policy is implemented in the radio stack to ensure reliable delivery on a lossless link, it would be necessary to not apply this to OLSR traffic, which must have losses to drive the OLSR route decision making.
Latencies based on PING times for a single hop link is approximately 3 seconds based on the test configuration for olsrd and the EMANE NEM stack. For multiple hop links, the latency is approximately a multiple of the single link latencies based on the number of hops in the route. For example, for a 3 hop link, the latency was approximately 9 milliseconds, that is 3 times the latency for a single hop. This illustrates that the kernel mode forwarding performed by intermediate forwarding nodes is very rapid, and does not contribute significantly to the overall point-to-point latency.
Host-to-Host Throughput Test Results
Continuous host-to-host data transfer throughputs using standard test tools like iperf and nuttcp indicated loss rates approximately equal to the pathloss provided to the emulation either through pathloss events, or calculated by EMANE based on GPS position using the freespace propagation model. For example, on a link where the pathloss corresponded to 25 percent packet losses, continuous host-to-host data losses were approximately 25 percent when measured over several seconds. Due to the random drop mechanism built into the EMANE PHY framework, there is some amount of variance in the empirically measured values.
OLSR ETX Based Routing
OLSR routing decisions to choose a multi-hop route from a single hop, or from a route with a lower hop count to one with a larger number of nodes in the route, were based on the ETX calculations for the links. This decision is based on losses of the reception of periodic OLSR control packets, and the change in routing from less to more hops is lossy relative to host-to-host data flows.
OLSR ETX Based Route Flapping Change Frequency
In computer networking and telecommunications, route flapping occurs when a router alternately advertises a destination network via one route then another (or as unavailable, and then available again) in quick sequence.
In networks with link-state routing protocols, route flapping will force frequent recalculation of the topology by all participating routers. In networks with distance-vector routing protocols, route flapping can trigger routing updates with every state change. In both cases, it prevents the network from converging.
Route flapping can be contained to a smaller area of the network if route aggregation is used. As an aggregate route will not be withdrawn as long as at least one of the aggregated subnets is still valid, a flapping route that is part of an aggregate will not disturb the routers that receive the aggregate.
There are settings for path loss for a compromised link where the ETX value, due to the random drop implementation of the EMANE PHY framework, would cause the OLSR routing decision to choose either a single hop or multi-hop route as route flapping.
For example, consider a network of 3 nodes, where the links from nodes nodes 1->2 and nodes 2->3 have zero losses, which is a combined ETX value of 2. As the single hop link from node 1->3 is degraded, a 25 percent loss rate provides a theoretical calculated ETX value of approximately 1.79. Due to the randomness of the EMANE PHY framework, this ETX value can grow to over a value of 2, causing OLSR to select a multi-hop route, or below 2 to select the single hop route.
For this network topology condition, where the ETX value is close to the threshold requiring a route change, route changes or
route flapping is not more frequent than 5 seconds avoiding a frequent toggling or oscillating condition.
OLSR ETX Based Multi-hop to Single Hop Progression and Timing
Routing decisions to progress from a zero loss multi-hop link to one with fewer hops occurs only after approximate 20 seconds. This process empirically appears to be broadly lossless relative to host-to-host data transfers. The ETX values calculated by OLSR change at approximate 5 seconds intervals based on the HELLO message intervals of 0.5 seconds. Due to the inherent hysteresis in the OLSR ETX logic, ETX values for a link which has become lossless, progress from values beginning with INFINITE, to a value that correspond to a 75 percent loss link, to a 50 percent loss link, to 25 percent losses, and finally to a zero loss (ETX value of 1.0) at 5 second intervals. This is a confidence building process wherein OLSR will not switch to the lower hop count route from a multi-hop lossless route until a sufficient number of OLSR control packets are received successfully.
OLSR Hop Count Limitation
The number of links a multi-hop route is not capped to any upper-bound, other than what is possible for a particular network topology and number of nodes. For our testing, we configured network topologies that included up to 8 hops in a link, which resulted in a latency of approximately 24 ms, or 8 times the latency associated with a single hop.
EMANE Limitation on Number of Test Nodes
The number of nodes, where a node consisted of a paired host and radio container, were limited only to the resources available on the platform. Our initial testing on a quad core Intel NUC platform used a network of 30 nodes, or 60 total containers. This supported continuous data testing at 10 Mbps over an 8 hop link, which consisted of 16 simultaneously active containers, before exhausting the platform resources resulting in host-to-host continuous data transfer losses. Moving the test platform to a higher performance server with more CPU cores would permit additional test nodes in excess of 30 nodes and perhaps increased data rates over multi-hop links that were not CPU bound.
In this post, the performance of proactive routing protocols as OLSR in a MANET in different network topologies of 30 nodes was evaluated. It was determined that OLSR provides the basic capabilities to manage a MANET.
It was also determined that various OLSR options, extensions, and message frequency values benefited our design goals of a responsive network permitting host-to-host data flows.
- OLSR default message timing was changed to make routing decision more responsive, but at the expense of increased network bandwidth.
- Host and network association (HNA) messages were enabled in the OLSR configuration to disseminate network route advertisements to allow for traffic routing using host-to-host addresses.
- It was necessary to enable Estimated Transmit Count (ETX) extensions to OLSR to allow for routing decisions to be based on link metrics rather than simply to reduce link hop counts.
The research performed represents an introduction into the use of OLSR as a MANET routing engine. There are a number of advanced research ongoing into modifications to OLSR to provide improved behavior for different QoS metrics including throughput, packet loss, jitter, delay, error rate, network overhead, power efficiency, DTN ambitions, etc.
The research performed indicated that the EMANE network emulation framework provided a well suited test bed for evaluating the real-time performance of the OLSR routing software in a MANET network of 30 nodes. The actual source code for the OLSR daemon (olsrd) anticipated for deployment to an embedded system was used as well as customized OLSR configurations.
The EMANE RF Pipe Model coupled with the RF Pipe Packet Completion Rate (PCR) customizable curve (defined via XML) allowed for defining link packet losses as a function of pathloss. This permitted testing OLSR routing behavior as a function of per link packet completion (or conversely loss) rates.
The observed routing decisions and timing made by OLSR as a function of number of nodes and dynamically configured link path losses were consistent with the expectations based on using the ETX as the QoS constraint.
The EMANE freespace or precomputed RF propagation models allowed for defining network topologies and link losses based either on node GPS position assigned dynamically using EMANE location events, or through precomputed values using EMANE pathloss events. Precomputed pathloss value assignment can be asymmetric, while GPS position based calculated pathloss values are symmetric with some variance based on the EMANE PHY Framework random drop implementation.
Route decision made were observed and measured using standard Linux terminal based commands, as well as graphical user interfaces including a 2D network topology viewer and the custom eView real-time spectrum and waterfall plotting application.
Additional research topics associated with enhancements to the OLSR protocol have been identified in this post and suggest future areas of exploration. The ambitions of this research is to reduce packet loss rates during the time period where a route change occurs, to add GPS location or speed awareness to the routing algorithm, to add store-carry-forward capabilities to address the issues associated with a hybrid DTN solution, etc.
The choice of a routing protocol depends on a full range of factors, so while OLSR fit our requirements, there will certainly be circumstances where a different routing engine is a better fit for different requirements such as control overhead, route acquisition delays, bandwidth, energy efficiency, and power requirements.
The architecture of EMANE representing the MAC and PHY layers of a communications stack allows for a number of different routing software applications to be tested in a modular manner, where the routing software operates at the application layer above the EMANE application to emulation boundary. Therefore the test procedures discussed in this post apply to a broad range of proactive, reactive, and hybrid network routing software that follow the standard Layer 3 routing software architecture previously described.