UpdateNetwork
From Adapt
Baseline Throughput
Throughput for single-stream, using nuttcp. Hosts may not be storage resources, but machines w/ similar connections as mcat/storage nodes.
Untuned transmit/receive numbers in mbits/s. Raw Data
| xmit \ recv | naraapp10.umiacs.umd.edu | erasrb01.nara.gov | srbbrick11.sdsc.edu |
|---|---|---|---|
| naraapp10.umiacs.umd.edu | X | 298, 299, 299 | 3.5, 3.5, 3.5 |
| erasrb01.nara.gov | 426, 417, 415 | X | 3.6, 3.6, 3.6 |
| srbbrick11.sdsc.edu | 7.8, 8.0, 8.0 | 8.4, 8.2, 8.2 | X |
Tuned transmit/receive numbers in mbits/s. (using settings below) Raw Data
| xmit \ recv | naraapp10.umiacs.umd.edu | erasrb01.nara.gov | srbbrick11.sdsc.edu |
|---|---|---|---|
| naraapp10.umiacs.umd.edu | X | 454.35, 461.0, 461.16 | 168.79, 84.53, 168.37 |
| erasrb01.nara.gov | 373.67, 409.15, 339.57 | X | 35.53, 29.93, 29.71 |
| srbbrick11.sdsc.edu | 49.79, 57.83, 43.41 | 28.06, 29.34, 29.00 | X |
SRB Speed Tests
Speed tests were performed using three samples, 1000 x 1k files, 300 x 10m files, and 100 x 100m files. Tests were run starting w/ 5 threads and continuing until total plateaued and per-file time started increading. After this point, a bottleneck was reached, either mcat/srb or network limit.
- Total - Total time in ms for transfer, from directory listing to last thread exiting
- avg - Average per-file time (ms)
- throughput - Aggregate throughput size / time.
1k test
1000 files in 10 directories, each directory contains 100 1-k zero-filled files.
| Test Case \ Threads | 5 | 10 | 15 | 20 | 25 | 35 | |
|---|---|---|---|---|---|---|---|
| narasrb03-unix to sfs-tape-nara | total | 774262 | 595247 | ||||
| avg | 3846 | 5876 | |||||
| throughput | 5.22KB/s | 6.79KB/s | |||||
| total | 788026 | 506796 | |||||
| avg | 3902 | 4870 | |||||
| throughput | 5.13KB/s | 7.98KB/s | |||||
| total | 746006 | ||||||
| avg | 3713 | ||||||
| throughput | 5.42KB/s | ||||||
| narasrb03-unix to ux-rocket3 | total | ||||||
| avg | |||||||
| throughput | |||||||
| total | |||||||
| avg | |||||||
| throughput | |||||||
| total | |||||||
| avg | |||||||
| throughput | |||||||
| narasrb03-unix to sfs-tape-nara | total | 251424 | 221271 | 253956 | 182190 | ||
| avg | 1248 | 1964 | 3001 | 3506 | |||
| throughput | 16.08KB/s | 18.28KB/s | 15.92KB/s | ||||
| total | 193504 | 183960 | |||||
| avg | 1890 | 2712 | |||||
| throughput | 20.90KB/s | ||||||
| total | |||||||
| avg | |||||||
| throughput |
- to sfs-tape: For 10 threads there were 3 failed attempts where 1-2 files would not finish.
- Load average on narasrb01 at umd was fairly high (2.0+) during the tests, w/ postgres running hot
10meg test
300 files in 3 directories, each directory contains 100 10-m zero-filled files.
| Test Case \ Threads | 5 | 10 | 15 | 20 | 25 | 35 | |
|---|---|---|---|---|---|---|---|
| narasrb03-unix to sfs-tape-nara | total | 1243620 | 671664 | 473424 | 324716 | ||
| avg | 20461 | 21511 | 22905 | 25467 | |||
| throughput | 2.42MB/s | 4.47MB/s | 9.25MB/s | ||||
| total | 1245558 | 669506 | 282670 | ||||
| avg | 20616 | 22064 | 21832 | ||||
| throughput | 2.41MB/s | 4.49MB/s | 10.63MB/s | ||||
| total | 1249714 | 703995 | 315643 | ||||
| avg | 20671 | 21920 | 21619 | ||||
| throughput | 2.40MB/s | 4.27MB/s | 9.52MB/s | ||||
| narasrb03-unix to ux-rocket3 | total | ||||||
| avg | |||||||
| throughput | |||||||
| total | |||||||
| avg | |||||||
| throughput | |||||||
| total | |||||||
| avg | |||||||
| throughput | |||||||
| narasrb03-unix to sfs-tape-nara | total | 297814 | 272686 | ||||
| avg | 4928 | 9006 | |||||
| throughput | 11.02MB/s | ||||||
| total | 271900 | ||||||
| avg | 8958 | ||||||
| throughput | 11.05MB/s | ||||||
| total | 274309 | ||||||
| avg | 9043 | ||||||
| throughput | 11.0MB/s |
Through 25 threads load average on umiacs mcat was negligable
- erasrb02 is only on a 100mbit link, so 10MB shows saturation. Further testing showed saturation was achieved w/ only 2-3 threads.
100meg test
Ping Reports
NARA to UMD
~1ms
-bash-2.05b$ ./nuttcp -xt -p5005 naraapp10.umiacs.umd.edu traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets 1 207.245.162.1 (207.245.162.1) 0.109 ms 0.206 ms 0.129 ms 2 clpk-nara.maxgigapop.net (206.196.177.13) 0.343 ms 0.346 ms 0.362 ms 3 umd-i2-rtr.maxgigapop.net (206.196.177.126) 0.367 ms 0.350 ms 0.366 ms 4 Gi3-5.ptx-fw-r1.net.umd.edu (129.2.0.233) 0.462 ms 0.470 ms 0.487 ms 5 128.8.7.202 (128.8.7.202) 0.737 ms 0.850 ms 0.616 ms 6 128.8.7.193 (128.8.7.193) 0.738 ms 0.842 ms 0.490 ms 7 Gi5-8.css-core-r1.net.umd.edu (128.8.0.85) 0.988 ms 0.595 ms 0.740 ms 8 Gi0-1.css-priv-r1.net.umd.edu (128.8.0.14) 1.211 ms 1.222 ms 0.866 ms 9 128.8.6.138 (128.8.6.138) 1.488 ms 4.728 ms 1.613 ms 10 naraapp10.umiacs.umd.edu (128.8.132.109) 0.987 ms 0.849 ms 0.865 ms traceroute to 207.245.162.27 (207.245.162.27), 30 hops max, 38 byte packets 1 cfar-gw.cfar.umd.edu (128.8.132.1) 0.703 ms 0.532 ms 0.555 ms 2 Vlan8.css-priv-r1.net.umd.edu (128.8.6.129) 0.672 ms 0.622 ms 0.522 ms 3 Gi5-5.css-core-r1.net.umd.edu (128.8.0.13) 0.618 ms 0.369 ms 0.624 ms 4 Gi3-2.ptx-fw-r1.net.umd.edu (128.8.0.86) 0.541 ms 0.413 ms 0.588 ms 5 128.8.7.194 (128.8.7.194) 0.698 ms 0.669 ms 0.657 ms 6 128.8.7.201 (128.8.7.201) 0.738 ms 0.516 ms 0.479 ms 7 Gi2-8.css-max-r1.net.umd.edu (129.2.0.234) 1.153 ms 1.322 ms 1.255 ms 8 clpk-umd-i2.maxgigapop.net (206.196.177.125) 1.098 ms 0.988 ms 1.049 ms 9 nara-rtr.maxgigapop.net (206.196.177.14) 0.829 ms 0.740 ms 0.896 ms 10 erasrb01.nara.gov (207.245.162.27) 1.088 ms 1.030 ms 1.138 ms
NARA to SDSC
~75ms
-bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu
traceroute to srbbrick11.sdsc.edu (198.202.115.45), 30 hops max, 38 byte packets
1 207.245.162.1 (207.245.162.1) 0.240 ms 0.211 ms 0.127 ms
2 clpk-nara.maxgigapop.net (206.196.177.13) 6.966 ms 0.347 ms 0.365 ms
3 lvl3-clpk-10gigbb.maxgigapop.net (206.196.178.90) 1.235 ms 1.106 ms 1.116 ms
4 i2-lvl3.maxgigapop.net (206.196.178.46) 1.230 ms 1.226 ms 1.241 ms
5 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 14.726 ms 14.842 ms 14.722 ms
6 hstnng-atlang.abilene.ucaid.edu (198.32.8.33) 36.339 ms 36.197 ms 36.222 ms
7 losang-hstnng.abilene.ucaid.edu (198.32.8.21) 67.811 ms 67.806 ms 67.820 ms
8 hpr-lax-gsr1--abilene-LA-10ge.cenic.net (137.164.25.2) 69.061 ms 68.805 ms 68.819 ms
9 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 75.183 ms 74.947 ms 75.033 ms
10 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 74.804 ms 85.787 ms 74.691 ms
11 lightning.sdsc.edu (132.249.30.6) 74.932 ms 74.928 ms 74.814 ms
12 srbbrick11.sdsc.edu (198.202.115.45) 75.050 ms 74.924 ms 74.940 ms
traceroute to 207.245.162.27 (207.245.162.27), 30 hops max, 38 byte packets
1 stingray (198.202.112.15) 0.481 ms 0.458 ms 0.368 ms
2 198.202.112.6 (198.202.112.6) 22.859 ms 10.463 ms 0.858 ms
3 dolphin (132.249.30.17) 5.868 ms 0.596 ms 0.995 ms
4 hpr-lax-hpr--sdsc-10ge.cenic.net (137.164.27.168) 6.868 ms 6.723 ms *
5 nlr-packetnet--hpr-lax-hpr.cenic.net (137.164.26.131) 7.496 ms 7.218 ms 7.236 ms
6 hous-losa-87.layer3.nlr.net (216.24.186.31) 38.222 ms 38.085 ms 37.961 ms
7 atla-hous-70.layer3.nlr.net (216.24.186.9) 62.205 ms 73.809 ms 68.442 ms
8 wash-atla-64.layer3.nlr.net (216.24.186.21) 75.339 ms 75.792 ms 75.427 ms
9 216.24.184.11 (216.24.184.11) 74.816 ms 74.833 ms 74.823 ms
10 clpk-lvl3-10gigbb.maxgigapop.net (206.196.178.89) 75.064 ms 75.054 ms 75.070 ms
MPLS Label=101776 CoS=0 TTL=1 S=1
11 nara-rtr.maxgigapop.net (206.196.177.14) 74.939 ms 75.057 ms 74.947 ms
12 erasrb01.nara.gov (207.245.162.27) 75.077 ms 75.053 ms 74.949 ms
UMD to SDSC
~75ms
-bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu naraapp10.umiacs.umd.edu
traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets
1 stingray (198.202.112.15) 0.600 ms 0.459 ms 0.367 ms
2 198.202.112.6 (198.202.112.6) 0.615 ms 0.600 ms 0.487 ms
3 dolphin (132.249.30.17) 9.862 ms 0.596 ms 0.993 ms
4 hpr-lax-hpr--sdsc-10ge.cenic.net (137.164.27.168) 6.867 ms 6.850 ms 6.744 ms
5 nlr-packetnet--hpr-lax-hpr.cenic.net (137.164.26.131) 7.234 ms 7.100 ms 7.243 ms
6 hous-losa-87.layer3.nlr.net (216.24.186.31) 38.346 ms 38.328 ms 38.092 ms
7 atla-hous-70.layer3.nlr.net (216.24.186.9) 62.706 ms 62.061 ms 63.334 ms
8 wash-atla-64.layer3.nlr.net (216.24.186.21) 75.558 ms 75.551 ms 75.566 ms
9 216.24.184.11 (216.24.184.11) 74.813 ms 74.757 ms 74.939 ms
10 so-2-3-0.clpk-t640.maxgigapop.net (206.196.178.9) 74.936 ms 74.941 ms 75.052 ms
MPLS Label=101296 CoS=0 TTL=1 S=1
11 umd-i2-rtr.maxgigapop.net (206.196.177.126) 74.930 ms 74.930 ms 74.942 ms
12 Gi3-5.ptx-fw-r1.net.umd.edu (129.2.0.233) 75.062 ms 75.185 ms 75.181 ms
13 128.8.7.202 (128.8.7.202) 75.436 ms 75.501 ms 75.471 ms
14 128.8.7.193 (128.8.7.193) 75.479 ms 75.286 ms 75.480 ms
15 Gi5-8.css-core-r1.net.umd.edu (128.8.0.85) 75.471 ms 75.970 ms 75.478 ms
16 Gi0-1.css-priv-r1.net.umd.edu (128.8.0.14) 75.977 ms 75.970 ms 75.985 ms
17 128.8.6.138 (128.8.6.138) 76.468 ms 76.329 ms 76.478 ms
18 naraapp10.umiacs.umd.edu (128.8.132.109) 76.477 ms 75.967 ms 75.983 ms
traceroute to 198.202.115.45 (198.202.115.45), 30 hops max, 38 byte packets
1 cfar-gw.cfar.umd.edu (128.8.132.1) 0.597 ms 0.528 ms 0.550 ms
2 Vlan8.css-priv-r1.net.umd.edu (128.8.6.129) 0.724 ms 0.600 ms 0.683 ms
3 Gi5-5.css-core-r1.net.umd.edu (128.8.0.13) 0.404 ms 0.446 ms 0.561 ms
4 Gi3-2.ptx-fw-r1.net.umd.edu (128.8.0.86) 0.527 ms 0.399 ms 0.628 ms
5 128.8.7.194 (128.8.7.194) 0.712 ms 0.928 ms 0.655 ms
6 128.8.7.201 (128.8.7.201) 0.636 ms 0.653 ms 0.488 ms
7 Gi2-8.css-max-r1.net.umd.edu (129.2.0.234) 2.408 ms 2.002 ms 2.559 ms
8 clpk-umd-i2.maxgigapop.net (206.196.177.125) 1.048 ms 0.907 ms 1.114 ms
9 abilene-backup-gige-rtr.maxgigapop.net (206.196.177.106) 2.342 ms 2.069 ms 1.923 ms
10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 15.201 ms 15.545 ms 15.445 ms
11 hstnng-atlang.abilene.ucaid.edu (198.32.8.33) 36.749 ms 36.474 ms 44.498 ms
12 losang-hstnng.abilene.ucaid.edu (198.32.8.21) 95.274 ms 68.257 ms 76.072 ms
13 hpr-lax-gsr1--abilene-LA-10ge.cenic.net (137.164.25.2) 69.657 ms 72.062 ms 69.617 ms
14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 75.632 ms 75.593 ms 75.947 ms
15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 75.558 ms 75.335 ms 75.272 ms
16 lightning.sdsc.edu (132.249.30.6) 75.501 ms 75.401 ms 75.473 ms
17 srbbrick11.sdsc.edu (198.202.115.45) 76.059 ms 75.745 ms 75.715 ms
Performance Settings
[root@naraapp10 tmp]# cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 9375000 [root@naraapp10 tmp]# cat /proc/sys/net/ipv4/tcp_wmem 4096 16384 9375000 [root@naraapp10 tmp]# cat /proc/sys/net/core/rmem_max 9375000 [root@naraapp10 tmp]# cat /proc/sys/net/core/wmem_max 9375000
-- Main.MikeSmorul - 05 Sep 2007
