forked from jingliu9/mtcp
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README
341 lines (269 loc) · 13.8 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
========================================================================
README for mTCP
========================================================================
mTCP is a highly scalable user-level TCP stack for multicore systems.
mTCP source code is distributed under the Modified BSD License. For
more detail, please refer to the LICENSE. The license term of io_engine
driver and ported applications may differ from the mTCP’s.
========================================================================
PREREQUISITE
========================================================================
We require the following libraries to run mTCP.
- libps (PacketShader I/O engine library) OR libdpdk (Intel's DPDK package*) OR netmap driver
- libnuma
- libpthread
- librt
- libgmp (for DPDK/ONVM driver)
Compling PSIO/DPDK/NETMAP/ONVM driver requires kernel headers.
- For Debian/Ubuntu, try apt-get install linux-headers-$(uname -r)
* We have modified the dpdk-17.08 package to export net_device stat data
(for Intel-based Ethernet adapters only) to the OS. To achieve this, the
dpdk-17.08/lib/librte_eal/linuxapp/igb_uio/ directory was updated. We also
modified mk/rte.app.mk and rte_cpuflags.mk files to ease the
compilation process of mTCP applications. We recommend using our package
for DPDK installation.
========================================================================
INCLUDED DIRECTORIES
========================================================================
mtcp - mtcp source code directory
mtcp/src - source code
mtcp/src/include - mTCP’s internal header files
mtcp/lib - library file
mtcp/include - header files that applications will use
io_engine - event-driven packet I/O engine (io_engine)
io_engine/driver - driver source code
io_engine/lib - io_engine library
io_engine/include - io_engine header files
io_engine/samples - sample io_engine applications (not mTCP’s)
dpdk-17.08 - Intel's Data Plane Development Kit* (modified)
dpdk-17.08/...
dpdk/ - Holds soft links to the compiled dpdk-17.08 include/ and lib/ paths
dpdk/include - the header files
dpdk/lib - the libraries that need to be linked
apps - mTCP applications
apps/example - example applications (see README)
apps/lighttpd-1.4.32 - mTCP-ported lighttpd (see INSTALL)
apps/apache_benchmark - mTCP-ported apache benchmark (ab) (see README-mtcp)
util - useful source code for applications
config - sample mTCP configuration files (may not be necessary)
========================================================================
INSTALL GUIDES
========================================================================
mTCP can be prepared in three ways.
- PSIO VERSION -
----------------
1. make in io_engine/driver:
# make
- check ps_ixgbe.ko
- please note that psio only runs on linux-2.6.x kernels
(linux-2.6.32 ~ linux-2.6.38)
2. install the driver:
# ./install.py <# cores> <# cores>
- refer to http://shader.kaist.edu/packetshader/io_engine/
- you may need to change the ip address in install.py:46
3. Setup mtcp library:
# ./configure --with-psio-lib=<$path_to_ioengine>
## e.g. ./configure --with-psio-lib=`echo $PWD`/io_engine
# make
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 8-core system, by using the following command:
# ./configure --with-psio-lib=`echo $PWD`/io_engine CFLAGS="-DMAX_CPUS=8"
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
following command; and then re-do step-3 (configure again):
# autoreconf -ivf
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
4. Check the configurations in apps/example
- epserver.conf for server-side configuration
- epwget.conf for client-side configuration
- you may write your own configuration file for your application
5. Run the applications!
- DPDK VERSION -
----------------
1. Set up Intel's DPDK driver. Please use our version of DPDK.
We have changed the lib/igb_uio/ submodule and a few makefiles
in DPDK's mk/ subdirectory. The best method to compile DPDK
package is to use DPDK's tools/setup.sh script. Please compile
your package based on your own hardware configuration. We
tested the mTCP stack on Intel Xeon E5-2690 (x86_64) machine
with Intel 82599 Ethernet adapters (10G). We used the following
steps in the dpdk-setup.sh script for our setup:
- Press [13] to compile the package
- Press [16] to install the driver
- Press [20] to setup 1024 2MB hugepages
- Press [22] to register the Ethernet ports
- Press [33] to quit the tool
- check that DPDK package creates a new directory of compiled
libraries. For x86_64 machines, the new subdirectory should be
*dpdk-17.08/x86_64-native-linuxapp-gcc*
- only those devices will work with DPDK drivers that are listed
on this page: http://dpdk.org/doc/nics. Please make sure that your
NIC is compatible before moving on to the next step.
- recent Linux kernels tend to rename Ethernet interfaces based on
their PCI addresses. If you are using Intel based Ethernet adapters,
please rename the interface with a ``dpdk`` prefix. You can do so
using the following command:
# sudo ifconfig <iface> down
## where X is an integer value.
## e.g. sudo ip link set ens786f0 name dpdk0
# sudo ip link set <iface> name dpdkX
2. Next bring the dpdk-registered interfaces up. Please use the
setup_iface_single_process.sh script file present in dpdk-17.08/tools/
directory for this purpose. Please change lines 49-51 to change the IP
address. Under default settings, run the script as:
# ./setup_iface_single_process.sh 4
This sets the IP address of your interfaces as 10.0.x.4.
3. Create soft links for include/ and lib/ directories inside
empty dpdk/ directory:
# cd dpdk/
# ln -s <path_to_dpdk_17_08_directory>/x86_64-native-linuxapp-gcc/lib lib
# ln -s <path_to_dpdk_17_08_directory>/x86_64-native-linuxapp-gcc/include include
4. Setup mtcp library:
# ./configure --with-dpdk-lib=$<path_to_mtcp_release_v3>/dpdk
## And not dpdk-17.08!
## e.g. ./configure --with-dpdk-lib=`echo $PWD`/dpdk
# make
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command:
# ./configure --with-dpdk-lib=$<path_to_mtcp_release_v3>/dpdk CFLAGS="-DMAX_CPUS=32"
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
following command; and then re-do step-4 (configure again):
# autoreconf -ivf
- checksum offloading in the NIC is now ENABLED (by default)!!!
- this only works for dpdk at the moment
- use
./configure --with-dpdk-lib=`echo $PWD`/dpdk --disable-hwcsum
to disable checksum offloading.
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
5. Check the configurations in apps/example
- epserver.conf for server-side configuration
- epwget.conf for client-side configuration
- you may write your own configuration file for your application
6. Run the applications!
- ONVM VERSION -
----------------
~NEW~: Now you can run mTCP applications (server + client) locally.
A local setup is useful when only 1 machine is available for the experiment.
ONVM configurations are placed as `.conf` files in apps/example directory.
ONVM basics are explained in https://github.com/sdnfv/openNetVM.
**Before running the applications make sure that onvm_mgr is running.**
Also, no core overlap between applications and onvm_mgr is allowed.
1. Install openNetVM using the following instructions
https://github.com/sdnfv/openNetVM/blob/master/docs/Install.md
2. Next bring the dpdk-registered interfaces up. This can be setup using:
# sudo $RTE_SDK/tools/dpdk-setup-iface.sh
3. Create soft links for include/ and lib/ directories inside
empty dpdk/ directory:
# cd dpdk/
# ln -s $RTE_SDK/$RTE_TARGET/lib lib
# ln -s $RTE_SDK/$RTE_TARGET/include include
4. Setup mtcp library
# ./configure --with-dpdk-lib=$<path_to_dpdk> --with-onvm-lib=$<path_to_onvm_lib>
# e.g. ./configure --with-dpdk-lib=`echo $PWD`/dpdk --with-onvm-lib=`echo $ONVM_HOME`/onvm
# make
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command:
# ./configure --with-dpdk-lib=$<path_to_mtcp_release_v3>/dpdk --with-onvm-lib=$<path_to_onvm_lib> CFLAGS="-DMAX_CPUS=32"
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
following command; and then re-do step-4 (configure again):
# autoreconf -ivf
- checksum offloading in the NIC is now ENABLED (by default)!!!
- this only works for dpdk at the moment
- use
./configure --with-dpdk-lib=`echo $PWD`/dpdk --disable-hwcsum
to disable checksum offloading.
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
5. Check the configurations in apps/example
- epserver.conf for server-side configuration
- epwget.conf for client-side configuration
- you may write your own configuration file for your application
6. Run the applications!
- Notes -
Once you have started onvm_mgr, sometimes an mTCP application may fail to get launched due
to a error(s) resembling the one mentioned below:
# EAL: FATAL: Cannot init memory
OR
# Cannot mmap memory for rte_config at [0x7ffff7fb6000], got [0x7ffff7e74000] - please use '--base-virtaddr' option
OR
# EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:06:00.0/resource3 to address: 0x7ffff7ff1000
To prevent this, use the base virtual address parameter to run the ONVM manager, e.g.:
cd openNetVM/onvm
./go.sh 1,2,3 1 -s stdout -v 0x7f000000000
- NETMAP VERSION -
------------------
See README.netmap for details.
========================================================================
TESTED ENVIRONMENTS
========================================================================
mTCP runs on Linux-based operating systems (2.6.x for PSIO) with generic
x86_64 CPUs, but to help evaluation, we provide our tested environments
as follows.
Intel Xeon E5-2690 octacore CPU @ 2.90 GHz
32 GB of RAM (4 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Debian 6.0.7 (Linux 2.6.32-5-amd64)
Intel Core i7-3770 quadcore CPU @ 3.40 GHz
16 GB of RAM (2 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Ubuntu 10.04 (Linux 2.6.32-47)
Event-driven PacketShader I/O engine (extended io_engine-0.2)
- PSIO is currently only compatible with Linux-2.6.
We tested the DPDK version (polling driver) with Linux-3.13.0 kernel.
========================================================================
NOTES
========================================================================
1. mTCP currently runs with fixed memory pools. That means, the size of
TCP receive and send buffers are fixed at the startup and does not
increase dynamically. This could be performance limit to the large
long-lived connections. Be sure to configure the buffer size
appropriately to your size of workload.
2. The client side of mTCP supports mtcp_init_rss() to create an
address pool that can be used to fetch available address space in
O(1). To easily congest the server side, this function should be
called at the application startup.
3. The supported socket options are limited for right now. Please refer
to the mtcp/src/api.c for more detail.
4. The counterpart of mTCP should enable TCP timestamp.
5. mTCP has been tested with the following Ethernet adapters:
1. Intel-82598 ixgbe (Max-queue-limit: 16)
2. Intel-82599 ixgbe (Max-queue-limit: 16)
3. Intel-I350 igb (Max-queue-limit: 08)
4. Intel-X710 i40e (Max-queue-limit: ~)
========================================================================
FREQUENTLY ASKED QUESTIONS
========================================================================
1. How can I quit the application?
- Use ^C to gracefully shutdown the application. Two consecutive
^C (separated by 1 sec) will force quit.
2. My application keeps printing "No route to 0.0.0.0"
- Try to turn off your network-manager for xge*. The network manager
can override the IP configuration set by install.py in PSIO driver.
3. Can I statically set the routing or arp table?
- Yes, mTCP allows static route and arp configuration. Go to the
config directory and see sample_route.conf or sample_arp.conf.
Copy and adapt it to your condition and link (ln -s) the config
directory to the application directory. mTCP will find
config/route.conf and config/arp.conf for static configuration.
========================================================================
CAUTION
========================================================================
1. Do not remove I/O driver (ps_ixgbe/igb_uio) while running mTCP
applications. The application will panic!
2. Use the ps_ixgbe/dpdk driver contained in this package, not the one
from some other place (e.g., from io_engine github).
========================================================================
Contact: mtcp-user at list.ndsl.kaist.edu
April 2, 2015.
EunYoung Jeong <notav at ndsl.kaist.edu>
M. Asim Jamshed <ajamshed at ndsl.kaist.edu>