ReferenceNICWalkthrough
The reference NIC walkthrough will go through an example of using some of the tools that are distributed with the release, and an example of how to write a simple C program to interface to the hardware.
Assuming the regression tests have been completed successfully and the NFP is installed correctly, we can now proceed to use one of the distributed projects: the NIC. In the rest of the exercises, we assume that NFP is installed in the user's home directory. Replace the '~' with the full path to the installation location if not. To run the tools, IP addresses must be assigned to the nf2cX interfaces. To do that, run the following command as root after replacing all the x's:
/sbin/ifconfig nf2cX x.x.x.x
Next, we need to download the NIC bitfile onto the NetFPGA. As root, run the following:
~/netfpga/lib/C/download/nf_download ~/netfpga/bitfiles/reference_nic.bit
You should see output similar to the following:
Found net device: nf2c0 Bit file built from: nf2_top_par.ncd Part: 2vp50ff1152 Date: 2007/11/21 Time: 11: 0: 3 Error Registers: 1000000 Good, after resetting programming interface the FIFO is empty Download completed - 2377668 bytes. (expected 2377668). DONE went high - chip has been successfully programmed.
To compile the utilities type:
cd ~/netfpga/lib/C/nic make
One of the built tools is called counterdump. This simple tool reads several hardware counters and dumps the counts to the terminal. To use it, type:
./counterdump
You should see an output similar to this:
Found net device: nf2c0 Num pkts received on port 0: 0 Num pkts dropped (rx queue 0 full): 0 Num pkts dropped (bad fcs q 0): 0 Num bytes received on port 0: 0 Num pkts sent from port 0: 0 Num bytes sent from port 0: 0 Num pkts received on port 1: 0 Num pkts dropped (rx queue 1 full): 0 Num pkts dropped (bad fcs q 1): 0 Num bytes received on port 1: 0 Num pkts sent from port 1: 0 Num bytes sent from port 1: 0 Num pkts received on port 2: 0 Num pkts dropped (rx queue 2 full): 0 Num pkts dropped (bad fcs q 2): 0 Num bytes received on port 2: 0 Num pkts sent from port 2: 0 Num bytes sent from port 2: 0 Num pkts received on port 3: 0 Num pkts dropped (rx queue 3 full): 0 Num pkts dropped (bad fcs q 3): 0 Num bytes received on port 3: 0 Num pkts sent from port 3: 0 Num bytes sent from port 3: 0
Now let's try to send and receive some packets and check the counter outputs again. One of the tools that are included in the NFP is a tool called send_pkts. This tool can send arbitrary Ethernet packets from any given port, but you will need root access to use it. To use this tool:
cd ~/netfpga/lib/C/tools/send_pkts make
If everything goes correctly, you should see an output similar to this:
gcc `libnet-config --defines --cflags` -O2 -o send_pkts send_pkts.c `libnet-config --libs` -L/usr/lib-lnet -lpcap --static
For the next part, we will test sending a few packets from one of the ports. To send packets, issue the following commands:
cd ~/netfpga/lib/C/tools/send_pkts sudo ./send_pkts -i nf2c0 -s 10 -l 100
The last command sends 10 100-byte packets out of port 0 on the NetFPGA (port 0 is the port closest to the PCI connector on the NetFPGA). If you have a machine connected to the same LAN section as that port, you should be able to capture the packets using Wireshark.
Now check the counters again:
~/netfpga/lib/C/nic/counterdump
You should see an output similar to this:
Found net device: nf2c0 Num pkts received on port 0: 0 Num pkts dropped (rx queue 0 full): 0 Num pkts dropped (bad fcs q 0): 0 Num bytes received on port 0: 0 Num pkts sent from port 0: 10 Num bytes sent from port 0: 1000 Num pkts received on port 1: 0 Num pkts dropped (rx queue 1 full): 0 Num pkts dropped (bad fcs q 1): 0 Num bytes received on port 1: 0 Num pkts sent from port 1: 0 Num bytes sent from port 1: 0 Num pkts received on port 2: 0 Num pkts dropped (rx queue 2 full): 0 Num pkts dropped (bad fcs q 2): 0 Num bytes received on port 2: 0 Num pkts sent from port 2: 0 Num bytes sent from port 2: 0 Num pkts received on port 3: 0 Num pkts dropped (rx queue 3 full): 0 Num pkts dropped (bad fcs q 3): 0 Num bytes received on port 3: 0 Num pkts sent from port 3: 0 Num bytes sent from port 3: 0
If your NetFPGA ports are not connected to a quiet network, then you'll probably see different results.
The counters that have been dumped using counterdump are actually memory-mapped I/O registers. These are counters that exist in the NetFPGA FPGA hardware and are exported via the PCI interface. The software uses ioctl calls to do reads and writes into these registers. The ioctl calls are wrapped in two simple functions readReg and writeReg. In this section, we will open up counterdump.c and understand the software/hardware interface. counterdump.c is shown below with line numbers for reference.
1 /****************************************************************************
2 * vim:set shiftwidth=2 softtabstop=2 expandtab:
3 * $Id: ReferenceNICWalkthrough.txt,v 1.2 2011/10/13 20:08:56 AdamC Exp $
4 *
5 * Module: counterdump.c
6 * Project: NetFPGA NIC
7 * Description: dumps the MAC Rx/Tx counters to stdout
8 * Author: Jad Naous
9 *
10 * Change history:
11 *
12 */
13
14 #include <stdio.h>
15 #include <stdlib.h>
16 #include <unistd.h>
17
18 #include <net/if.h>
19
20 #include "../common/reg_defines.h"
21 #include "../common/nf2.h"
22 #include "../common/nf2util.h"
23
24 #define PATHLEN 80
25
26 #define DEFAULT_IFACE "nf2c0"
27
28 /* Global vars */
29 static struct nf2device nf2;
30
31 /* Function declarations */
32 void dumpCounts();
33 void processArgs (int , char **);
34 void usage (void);
35
36 int main(int argc, char *argv[])
37 {
38 nf2.device_name = DEFAULT_IFACE;
39
40 processArgs(argc, argv);
41
42 // Open the interface if possible
43 if (check_iface(&nf2))
44 {
45 exit(1);
46 }
47 if (openDescriptor(&nf2))
48 {
49 exit(1);
50 }
51
52 dumpCounts();
53
54 closeDescriptor(&nf2);
55
56 return 0;
57 }
58
59 void dumpCounts()
60 {
61 unsigned val;
62
63 readReg(&nf2, RX_QUEUE_0_NUM_PKTS_STORED_REG, &val);
64 printf("Num pkts received on port 0: %u\n", val);
65 readReg(&nf2, RX_QUEUE_0_NUM_PKTS_DROPPED_FULL_REG, &val);
66 printf("Num pkts dropped (rx queue 0 full): %u\n", val);
67 readReg(&nf2, RX_QUEUE_0_NUM_PKTS_DROPPED_BAD_REG, &val);
68 printf("Num pkts dropped (bad fcs q 0): %u\n", val);
69 readReg(&nf2, RX_QUEUE_0_NUM_BYTES_PUSHED_REG, &val);
70 printf("Num bytes received on port 0: %u\n", val);
71 readReg(&nf2, TX_QUEUE_0_NUM_PKTS_SENT_REG, &val);
72 printf("Num pkts sent from port 0: %u\n", val);
73 readReg(&nf2, TX_QUEUE_0_NUM_BYTES_PUSHED_REG, &val);
74 printf("Num bytes sent from port 0: %u\n\n", val);
75
76 readReg(&nf2, RX_QUEUE_1_NUM_PKTS_STORED_REG, &val);
77 printf("Num pkts received on port 1: %u\n", val);
78 readReg(&nf2, RX_QUEUE_1_NUM_PKTS_DROPPED_FULL_REG, &val);
79 printf("Num pkts dropped (rx queue 1 full): %u\n", val);
80 readReg(&nf2, RX_QUEUE_1_NUM_PKTS_DROPPED_BAD_REG, &val);
81 printf("Num pkts dropped (bad fcs q 1): %u\n", val);
82 readReg(&nf2, RX_QUEUE_1_NUM_BYTES_PUSHED_REG, &val);
83 printf("Num bytes received on port 1: %u\n", val);
84 readReg(&nf2, TX_QUEUE_1_NUM_PKTS_SENT_REG, &val);
85 printf("Num pkts sent from port 1: %u\n", val);
86 readReg(&nf2, TX_QUEUE_1_NUM_BYTES_PUSHED_REG, &val);
87 printf("Num bytes sent from port 1: %u\n\n", val);
88
89 readReg(&nf2, RX_QUEUE_2_NUM_PKTS_STORED_REG, &val);
90 printf("Num pkts received on port 2: %u\n", val);
91 readReg(&nf2, RX_QUEUE_2_NUM_PKTS_DROPPED_FULL_REG, &val);
92 printf("Num pkts dropped (rx queue 2 full): %u\n", val);
93 readReg(&nf2, RX_QUEUE_2_NUM_PKTS_DROPPED_BAD_REG, &val);
94 printf("Num pkts dropped (bad fcs q 2): %u\n", val);
95 readReg(&nf2, RX_QUEUE_2_NUM_BYTES_PUSHED_REG, &val);
96 printf("Num bytes received on port 2: %u\n", val);
97 readReg(&nf2, TX_QUEUE_2_NUM_PKTS_SENT_REG, &val);
98 printf("Num pkts sent from port 2: %u\n", val);
99 readReg(&nf2, TX_QUEUE_2_NUM_BYTES_PUSHED_REG, &val);
100 printf("Num bytes sent from port 2: %u\n\n", val);
101
102 readReg(&nf2, RX_QUEUE_3_NUM_PKTS_STORED_REG, &val);
103 printf("Num pkts received on port 3: %u\n", val);
104 readReg(&nf2, RX_QUEUE_3_NUM_PKTS_DROPPED_FULL_REG, &val);
105 printf("Num pkts dropped (rx queue 3 full): %u\n", val);
106 readReg(&nf2, RX_QUEUE_3_NUM_PKTS_DROPPED_BAD_REG, &val);
107 printf("Num pkts dropped (bad fcs q 3): %u\n", val);
108 readReg(&nf2, RX_QUEUE_3_NUM_BYTES_PUSHED_REG, &val);
109 printf("Num bytes received on port 3: %u\n", val);
110 readReg(&nf2, TX_QUEUE_3_NUM_PKTS_SENT_REG, &val);
111 printf("Num pkts sent from port 3: %u\n", val);
112 readReg(&nf2, TX_QUEUE_3_NUM_BYTES_PUSHED_REG, &val);
113 printf("Num bytes sent from port 3: %u\n\n", val);
114 }
115
116 /*
117 * Process the arguments.
118 */
119 void processArgs (int argc, char **argv )
120 {
121 char c;
122
123 /* don't want getopt to moan - I can do that just fine thanks! */
124 opterr = 0;
125
126 while ((c = getopt (argc, argv, "i:h")) != -1)
127 {
128 switch (c)
129 {
130 case 'i': /* interface name */
131 nf2.device_name = optarg;
132 break;
133 case '?':
134 if (isprint (optopt))
135 fprintf (stderr, "Unknown option `-%c'.\n", optopt);
136 else
137 fprintf (stderr,
138 "Unknown option character `\\x%x'.\n",
139 optopt);
140 case 'h':
141 default:
142 usage();
143 exit(1);
144 }
145 }
146 }
147
148
149 /*
150 * Describe usage of this program.
151 */
152 void usage (void)
153 {
154 printf("Usage: ./counterdump <options> \n\n");
155 printf("Options: -i <iface> : interface name (default nf2c0)\n");
156 printf(" -h : Print this message and exit.\n");
157 }
Let's go through the code.
20 #include "../common/reg_defines.h" 21 #include "../common/nf2.h" 22 #include "../common/nf2util.h"
Line 20 includes the header file that contains all the register addresses on the NetFPGA. This is needed to refer to register addresses as constant names rather than numeric addresses. Lines 21 and 22 include macros to access registers. These functions are used later in the code.
29 static struct nf2device nf2;
The nf2 struct will hold information about the device we are trying to access.
Now let's go into our main function.
38 nf2.device_name = DEFAULT_IFACE;
Set a default name for the device we are trying to access. This is useful so that the user of this program doesn't have to specify the name if she is using the default device (which is true in most cases).
40 processArgs(argc, argv);
Parses the command line options. In this simple program, the only command line option is to change the name of the interface we are trying to access.
43 if (check_iface(&nf2))
Checks that the interface exists and can be reached.
47 if (openDescriptor(&nf2))
Tries to open the interface for reading/writing using ioctl calls. The interface has to be up and assigned an IP address for a non-root user to be able to access it using ioctl calls.
52 dumpCounts();
calls the function to dump all the counts.
54 closeDescriptor(&nf2);
Closes the interface after we are done using it to be polite.
Reading and writing registers uses two functions:
int readReg(nf2device *nf2, unsigned int addr, unsigned int *val): Reads the register at address addr from device nf2 and writes the value in *val. Returns 1 on fail, 0 on success.
int writeReg(nf2device *nf2, unsigned int addr, unsigned int val): Writes val into the register at address addr from device nf2 . Returns 1 on fail, 0 on success.
As an example we will look at two lines in the dumpCounts() function. The rest of the lines are similar:
63 readReg(&nf2, RX_QUEUE_0_NUM_PKTS_STORED_REG, &val);
64 printf("Num pkts received on port 0: %u\n", val);
Line 63 reads the number of packets received into Rx Queue 0 into val, and line 64 prints out the result.
The registers available for access are documented in the Register Map. Unfortunately, registers keep getting added and removed as the design evolves, so the most current list of registers is in the reg_defines.h file that defines the register addresses. This file is generated automatically from the Verilog source code when it is simulated so it always has the most recent list of registers and their addresses. The registers in the Register Map are divided into groups corresponding to the modules described in the Verilog. The next section will go into more details on these modules.
Next, we will modify this file to dump the device ID which is assigned at implementation time. To do that, open file netfpga/lib/C/common/reg_defines.h file and copy the macro that defines the device ID to replace the XXXX in the following lines, and copy the lines into the start of dumpCounts() function after the declaration of val.
readReg(&nf2, XXXX, &val);
printf("Device ID: %u\n\n", val);
Then after saving the file, type make in the directory containing counterdump.c and run the program again. You should now see an output similar to the following:
Found net device: nf2c0 Device ID: 1 Num pkts received on port 0: 0 Num pkts dropped (rx queue 0 full): 0 Num pkts dropped (bad fcs q 0): 0 Num bytes received on port 0: 0 Num pkts sent from port 0: 10 Num bytes sent from port 0: 1000 Num pkts received on port 1: 0 Num pkts dropped (rx queue 1 full): 0 Num pkts dropped (bad fcs q 1): 0 Num bytes received on port 1: 0 Num pkts sent from port 1: 0 Num bytes sent from port 1: 0 Num pkts received on port 2: 0 Num pkts dropped (rx queue 2 full): 0 Num pkts dropped (bad fcs q 2): 0 Num bytes received on port 2: 0 Num pkts sent from port 2: 0 Num bytes sent from port 2: 0 Num pkts received on port 3: 0 Num pkts dropped (rx queue 3 full): 0 Num pkts dropped (bad fcs q 3): 0 Num bytes received on port 3: 0 Num pkts sent from port 3: 0 Num bytes sent from port 3: 0
"Diagram of the reference pipeline"
The division of the hardware into modules was hinted at in the previous section. Understanding these modules is essential in making the most of the available designs. The distributed projects in the NFP, including the NIC, all follow the same modular structure. This design is a pipeline where each stage is a separate module. A diagram of the pipeline is shown on the right.
The first stage in the pipeline consists of several queues which we call the Rx queues. These queues receive packets from IO ports such as the Ethernet ports and the PCI over DMA and provide a unified interface to the rest of the system. These ports are connected into a wrapper called the User Data Path which contains the processing stages. The current design (version 1.0 Beta) has 4 Ethernet Rx queues and 4 CPU DMA queues. The Ethernet and DMA queues are interleaved so that to the User Data Path, Rx Queue 0 is Ethernet Port 0, Rx Queue 1 is DMA port 0, Rx Queue 2 is Ethernet Port 1, and so on. Packets that arrive into CPU DMA Rx Queue X are packets that have been sent by the software out of interface nf2cX.
In the User Data Path, the first module a packet passes through is the Input Arbiter. The input arbiter decides which Rx queue to service next, and pulls the packet from that Rx queue and hands it to the next module in the pipeline: The output port lookup module. The output port lookup module is responsible for deciding which port a packet goes out of. After that decision is made, the packet is then handed to the output queues module which stores the packet in the output queues corresponding to the output port until the Tx queue is ready to accept the packet for transmission.
The Tx queues are analogous to the Rx queues and they send packets out of the IO ports instead of receiving. Tx queues are also interleaved so that packets sent out of User Data Path port 0 are sent to Ethernet Tx queue 0, and packets sent out of User Data Path port 1 are sent to CPU DMA Tx queue 0, and so on. Packets that are handed to DMA Tx queue X pop out of interface nf2cX.
For almost each of these modules, there is a set of registers that access status information and set control signals for the module. These registers are described in the Register Map.
- You can go through the SCONE Software Walkthrough if you are interested in writing software for the NetFPGA to run in the host
- You can go through the Linux Router Kit Walkthrough if you are interested in using the NetFPGA as a high performance router
- You can go through the Reference Router Walkthrough if you are interested in modifying the gateware on the FPGA
- You can go through the Buffer Monitoring System Walkthrough if you are interested in how to test/measure the lengths of buffers.
- Return to the Main Guide