-
Notifications
You must be signed in to change notification settings - Fork 129
NetFPGA 1G CML Reference Router
reference_router_nf1_cml
projects/reference_router_nf1_cml
-
NetFPGA Specific cores
-
Xilinx IP
The Java GUI allows the user to change entries in the Routing Table and ARP cache as well as the router's MAC and IP addresses. It also provides updates on counter values and graphs of throughput and much more. The GUI has a part that is written in C that provides the interface between the Java binaries and the driver. This native library is compiled from the nf10util.c file that contains the readReg and writeReg utilities. The library connects to the GUI using the Java Native Access (jna) library.
To build the GUI, first make sure that you have Sun's Java Development Kit
(version >=1.6.0) installed and make sure the java, javac, and jar binaries
are in your path (otherwise, you can edit the Makefile under
projects/reference_router_nf1_cml/sw/host/gui to reflect the actual locations).
Then cd into projects/reference_router_nf1_cml/sw/host/gui and type make clean. Then type make. You should get output similar to below:
make[1]: Entering directory `~/NetFPGA-10G-live/projects/reference_router_nf1_cml/sw/host/common'
gcc -fpic -c nf10util.c nf_util.c
gcc -shared nf10util.o nf_util.o -o libnf10.so
make[1]: Leaving directory `~/NetFPGA-10G-live/projects/reference_router_nf1_cml/sw/host/common'
Building java...
Done
Writing router gui manifest...
Building router jar...
Writing script to start router gui...
To run the GUI for the router, cd into
NetFPGA-10G-live/projects/reference_router_nf1_cml/sw/host/gui and type
./router.sh. The GUI should pop-up. The GUI constantly polls the data it
reads from the hardware.
The Quickstart Panel provides a summary of things that can be done with the hardware. There is also a tab for viewing statistics and a tab for details.
A standalone small CLI that allows you to change routing table entries, ARP
cache entries, and other settings is also provided in case the user doesn't
want to run SCONE. The CLI is under
NetFPGA-10G-live/projects/reference_router_nf1_cml/sw/host/cli. To build it,
type make in that directory. You should see output similar to the following:
gcc -g -c -o cli.o cli.c
gcc -g -c -o ../common/util.o ../common/util.c
gcc -lncurses cli.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h -o cli
gcc -g -c -o regdump.o regdump.c
gcc -lncurses regdump.o ../common/nf2util.o ../common/reg_defines.h -o regdump
gcc -g -c -o show_stats.o show_stats.c
gcc -lncurses show_stats.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h -o show_stats</pre>
For help on using the CLI, start it by typing ./cli and then type help in the CLI.
We invite you to extend this CLI and any of our software tools and contribute them back so we can expand our library and make it easier for anybody to use NetFPGA.
The router SCONE ( '''S''' oftware '''C'''omponent '''O'''f '''Ne'''tFPGA) is a user level router that performs IPv4 forwarding, handles ARPs and various ICMP messages, has telnet (port 23) and web (port 8080) interfaces to handle router control, and also implements a subset of OSPF named PW-OSPF. SCONE mirrors a copy of its MAC addresses, IP addresses, routing table, and ARP table to the NetFPGA card which hardware accelerates the forwarding path.
- Ensure your NETFPGA is programmed with the Reference Router bitfile
- Check that your nf0, nf1, nf2, nf3 interfaces are up and do not have an assigned IPv4 address
- To remove an IPv4 address from an interface run 'ifconfig nfX 0.0.0.0' replacing X with the interface name
- Setup the cpuhw file
- A text file named cpuhw must exist in the sw directory that you execute the
scone binary. The format is as follows:
- <interface name> <ip address> <ip mask> <mac address>
- An example:
- eth0 192.168.0.2 255.255.255.0 00:00:00:00:00:01
- Inside SCONE the interfaces are typically named and referred to as eth0-3, which correspond to the nf0-3 ports.
- (Optional) Specify a text file containing static routes
- A text file containing static routes to be added on startup can be
specified from the command line using the '-r' parameter, ie '-r
rtable.netfpga'. The format is as follows:
- <net> <next hop> <mask> <outgoing interface name>
- An example:
- 192.168.130.14 192.168.130.9 255.255.255.254 eth1
- (Optional) Specify a file to log packets in pcap format
- Specify a file using the '-l' command line parameter, ie '-l scone.log', all packets received by SCONE will be logged using pcap format to this file. This file can be opened and examined using Wireshark. Note packets that take the hardware forwarding path will not appear in this log file.
To modify the way SCONE operates after launch, use the telnet command line interface. Connect to one of the IP addresses specified in the cpuhw file on port 23. To get a list of commands using telnet issue the ? command. For help with a command enter the command and put ? for its arguments. IE 'show ip ?'.
For an incoming packet to take the hardware forwarding path the packet must get a hit on the destination MAC address table, the routing table, and the ARP table for the next hop. SCONE builds and maintains the routing table containing static and dynamic routes, and the ARP table in software. When changes are detected in the software tables, SCONE copies them down into the NetFPGA's hardware tables. If you want packets to be passed up to software that match the IP addresses assigned to the interfaces you must also push these IP addresses down into a hardware table. Following are some code snippets for writing to these various tables. In general to write you first write the registers representing the data for the row, then you write the row number into the corresponding WR register. To read a row, first write the row number you want into the RD register, then read the data from the corresponding registers.
uint8_t mac_addr[6];
unsigned int mac_hi = 0;
unsigned int mac_lo = 0;
mac_hi |= ((unsigned int)mac_addr[0]) << 8;
mac_hi |= ((unsigned int)mac_addr[1]);
mac_lo |= ((unsigned int)mac_addr[2]) << 24;
mac_lo |= ((unsigned int)mac_addr[3]) << 16;
mac_lo |= ((unsigned int)mac_addr[4]) << 8;
mac_lo |= ((unsigned int)mac_addr[5]);
writeReg(&netfpga, ROUTER_OP_LUT_MAC_0_HI_REG, mac_hi);
writeReg(&netfpga, ROUTER_OP_LUT_MAC_0_LO_REG, mac_lo);
// 1,2,3 can be substituted in for 0 to set the MAC for the other ports
NOTE : One confusing aspect of using a user level process to control the router is that the MAC addresses of the router will not match the kernel MAC addresses. That is, the MAC addresses of the nfX interfaces do not necessarily reflect those that are actually in the hardware. The nfX interfaces are software independent entities that do not necessarily reflect the state of the hardware. In the next section we will describe the Router Kit which reflects the Linux's view of the network to HCORR (Hardware Component of Reference Router). This includes the ports' MAC addresses, IP addresses, the routing table, and the ARP cache.
The HCORR has a table that stores IP addresses called the destination IP filter table. Packets whose destination IP address matches an entry in this table will be sent to the hardware. The number of entries in this table can be found in the RegisterMap. We use this table to filter out the IP addresses of the router and the IP addresses of PW-OSPF multicast packets. We write into the table as follows:
struct in_addr ip;
writeReg(&rs->netfpga, ROUTER_OP_LUT_DST_IP_FILTER_IP_REG, ntohl(ip.s_addr));
// i is the row to write the IP address to
writeReg(&rs->netfpga, ROUTER_OP_LUT_DST_IP_FILTER_WR_ADDR_REG, i);
Writing to the Routing table is similar to writing to the Destination IP filter table. Note that the output port(s) is specified in one-hot-encoded format corresponding to the output port of the User Data Path to the Tx Queues. The function getOneHotPortNumber returns the values (in decimal): 1 , 4, 16, 64 which correspond to physical output ports 0-3 on the NetFPGA card. You can also tell a route to specifically send to software by specifying 2, 8, 32, 128 corresponding to the nf0,1,2,3 interfaces. To send out of MAC ports 0 and 1, write 1+4=5 as the output port entry.
struct in_addr ip, mask, gw;</br>
int i;
char* iface;
/* write the ip */
writeReg(&netfpga, ROUTER_OP_LUT_RT_IP_REG, ntohl(ip.s_addr));
/* write the mask */
writeReg(&netfpga, ROUTER_OP_LUT_RT_MASK_REG, ntohl(mask.s_addr));
/* write the next hop */
writeReg(&netfpga, ROUTER_OP_LUT_RT_NEXT_HOP_IP_REG, ntohl(gw.s_addr));
/* write the port */
writeReg(&netfpga, ROUTER_OP_LUT_RT_OUTPUT_PORT_REG, getOneHotPortNumber(iface));
/* write the row number */
writeReg(&netfpga, ROUTER_OP_LUT_RT_LUT_WR_ADDR_REG, i);
uint8_t mac_addr[6];
unsigned int mac_hi = 0;
unsigned int mac_lo = 0;
struct in_addr ip;
int i;
mac_hi |= ((unsigned int)mac_addr[0]) << 8;
mac_hi |= ((unsigned int)mac_addr[1]);
mac_lo |= ((unsigned int)mac_addr[2]) << 24;
mac_lo |= ((unsigned int)mac_addr[3]) << 16;
mac_lo |= ((unsigned int)mac_addr[4]) << 8;
mac_lo |= ((unsigned int)mac_addr[5]);
writeReg(&netfpga, ROUTER_OP_LUT_ARP_MAC_HI_REG, mac_hi);
writeReg(&netfpga, ROUTER_OP_LUT_ARP_MAC_LO_REG, mac_lo);
/* write the next hop ip data */
writeReg(&netfpga, ROUTER_OP_LUT_ARP_NEXT_HOP_IP_REG, ntohl(ip.s_addr));
/* set the row */
writeReg(&netfpga, ROUTER_OP_LUT_ARP_LUT_WR_ADDR_REG, i);
Router Kit is a simple approach to providing hardware acceleration to an unmodified Linux system. It is comprised of a single program, rkd (Router Kit Daemon), which monitors the Linux routing table and ARP cache and mirrors it down to the NetFPGA IPv4 reference router implementation.
rkd [-h||--help} [-d||--daemon} [-i||--interval]
rkd should work from the command line without any external configuration options. Simply run (./rkd). To run the process in the background use -d. You may specify the polling time in milliseconds using the -i option.
Router Kit is only useful on a Linux host with the NetFPGA installed, and the ipv4 reference router bitfile loaded. Given this setup each port on the NetFPGA card is available to Linux via a nf* interface (i.e. nf0, nf1, nf2, and nf3 assuming a single card is installed).
rkd will attempt to mirror all ARP cache and routing table entries associate with a NetFPGA interface into hardware. This provides a very simple (and familiar) method of adding entries to the hardware. For example, to add a static ARP entry, simply use the arp(8) command. The following command will add a static ARP entry.
arp -s 1.2.3.4 ca:fe:de:ad:d0:d0 -i nf0
To add an entry into the routing table, use route(8) (or ip(8)). For example, adding a default entry with a next hop of 10.0.0.1 out of the first port would look something like:
route add default gw 10.0.0.1 dev nf0
Router kit is not limited to manual manipulation from the command line. All state (including dynamic state) is mirrored. To wit, running rkd alongside a standard routing daemon, such as XoRP, or Zebra/Quagga, should provide hardware acceleration of the forwarding table without any further configuration (provided the routing software is using the NetFPGA interfaces for forwarding).
rkd continuously polls the routing table and ARP cache state from /proc/net/route and /proc/net/arp respectively. When a change in state is detected, ./rkd writes the updated state to the NetFPGA through the register interface. All traffic not handled by the hardware is DMA'd to software where it is processed by the Linux kernel.
The division of the hardware into modules was hinted in the previous section. Understanding these modules is essential in making the most of the available designs. The distributed projects in the NFP, including the Router, all follow the same modular structure. This design is a pipeline where each stage is a separate module. Please, refer to the Reference NIC project for more information.
The first stage in the pipeline consists of several queues which we call the Rx queues. These queues receive packets from IO physical ports (i.e., 10GMAC) and provide a unified interface (AXI) to the rest of the system.
In the main datapath, the first module a packet passes through is the Input Arbiter. The input arbiter decides which Rx queue to service next, and pulls the packet from that Rx queue and hands it to the next module in the pipeline: The output port lookup module. The output port lookup module is responsible for deciding which port a packet goes out of. Here it is actually implemented the router forwarding logic. The 32-entries TCAM where the LPM table is stored is consulted to find the next-hop-ip value. Such a value is then used in the 32-entried CAM (ARP table) to find the right destination MAC address. After that decision is made, the packet is then handed to the output queues module which stores the packet in the output queues corresponding to the output port until the Tx queue is ready to accept the packet for transmission.
The Tx queues are analogous to the Rx queues and they send packets out of the IO ports instead of receiving.
Note: Actually, the reference router is not able to support the fully 40G line rate in the worst case scenario.
All the information could be found here.
Each projects has some features that are verified by doing Simulation
tests and HW tests. The test
infrastructure is based on the python. You can find the tests inside the
projects/{project_name}/test folder.