UpdateNetwork

Baseline Throughput

Throughput for single-stream, using nuttcp. Hosts may not be storage resources, but machines w/ similar connections as mcat/storage nodes.

Untuned transmit/receive numbers in mbits/s. Raw Data

xmit \ recv	naraapp10.umiacs.umd.edu	erasrb01.nara.gov	srbbrick11.sdsc.edu
naraapp10.umiacs.umd.edu	X	298, 299, 299	3.5, 3.5, 3.5
erasrb01.nara.gov	426, 417, 415	X	3.6, 3.6, 3.6
srbbrick11.sdsc.edu	7.8, 8.0, 8.0	8.4, 8.2, 8.2	X

Tuned transmit/receive numbers in mbits/s. (using settings below) Raw Data

xmit \ recv	naraapp10.umiacs.umd.edu	erasrb01.nara.gov	srbbrick11.sdsc.edu
naraapp10.umiacs.umd.edu	X	454.35, 461.0, 461.16	168.79, 84.53, 168.37
erasrb01.nara.gov	373.67, 409.15, 339.57	X	35.53, 29.93, 29.71
srbbrick11.sdsc.edu	49.79, 57.83, 43.41	28.06, 29.34, 29.00	X

SRB Speed Tests

Speed tests were performed using three samples, 1000 x 1k files, 300 x 10m files, and 100 x 100m files. Tests were run starting w/ 5 threads and continuing until total plateaued and per-file time started increading. After this point, a bottleneck was reached, either mcat/srb or network limit.

• Total - Total time in ms for transfer, from directory listing to last thread exiting
• avg - Average per-file time (ms)
• throughput - Aggregate throughput size / time.

1k test

1000 files in 10 directories, each directory contains 100 1-k zero-filled files.

Test Case \ Threads		5	10	15	20	25	35
narasrb03-unix to sfs-tape-nara	total	774262	595247
	avg	3846	5876
	throughput	5.22KB/s	6.79KB/s
	total	788026	506796
	avg	3902	4870
	throughput	5.13KB/s	7.98KB/s
	total	746006
	avg	3713
	throughput	5.42KB/s
narasrb03-unix to ux-rocket3	total
	avg
	throughput
	total
	avg
	throughput
	total
	avg
	throughput
narasrb03-unix to sfs-tape-nara	total	251424	221271	253956	182190
	avg	1248	1964	3001	3506
	throughput	16.08KB/s	18.28KB/s	15.92KB/s
	total		193504	183960
	avg		1890	2712
	throughput		20.90KB/s
	total
	avg
	throughput

• to sfs-tape: For 10 threads there were 3 failed attempts where 1-2 files would not finish.
• Load average on narasrb01 at umd was fairly high (2.0+) during the tests, w/ postgres running hot

• 10meg test*

300 files in 3 directories, each directory contains 100 10-m zero-filled files.

| Test Case \ Threads | | 5 | 10 | 15 | 20 | 25 | 35 | | narasrb03-unix to sfs-tape-nara | total | 1243620 | 671664 | 473424 | | 324716 | | | avg | 20461 | 21511 | 22905 | | 25467 | | | throughput | 2.42MB/s | 4.47MB/s | | | 9.25MB/s | | | total | 1245558 | 669506 | | | 282670 | | | avg | 20616| 22064 | | | 21832 | | | throughput | 2.41MB/s | 4.49MB/s | | | 10.63MB/s | | | total | 1249714 | 703995 | | | 315643 | | | avg | 20671 | 21920 | | | 21619 | | | throughput | 2.40MB/s | 4.27MB/s | | | 9.52MB/s | | narasrb03-unix to ux-rocket03 | total | | | | | | | | avg | | | | | | | | throughput | | | | | | | | total | | | | | | | | avg | | | | | | | | throughput | | | | | | | | total | | | | | | | | avg | | | | | | | | throughput | | | | | | | narasrb03-unix to erasrb2-unix2* | total | 297814| 272686 | | | avg | 4928 | 9006 | | | throughput | | 11.02MB/s | | | total | | 271900 | | | avg | | 8958 | | | throughput | | 11.05MB/s | | | total | | 274309 | | | avg | | 9043 | | | throughput | | 11.0MB/s |

Through 25 threads load average on umiacs mcat was negligable

• erasrb02 is only on a 100mbit link, so 10MB shows saturation. Further testing showed saturation was achieved w/ only 2-3 threads.

• 100meg test*

---++ Ping Reports

• NARA to UMD* - ~1ms

<verbatim> -bash-2.05b$ ./nuttcp -xt -p5005 naraapp10.umiacs.umd.edu traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets

1  207.245.162.1 (207.245.162.1)  0.109 ms  0.206 ms  0.129 ms
2  clpk-nara.maxgigapop.net (206.196.177.13)  0.343 ms  0.346 ms  0.362 ms
3  umd-i2-rtr.maxgigapop.net (206.196.177.126)  0.367 ms  0.350 ms  0.366 ms
4  Gi3-5.ptx-fw-r1.net.umd.edu (129.2.0.233)  0.462 ms  0.470 ms  0.487 ms
5  128.8.7.202 (128.8.7.202)  0.737 ms  0.850 ms  0.616 ms
6  128.8.7.193 (128.8.7.193)  0.738 ms  0.842 ms  0.490 ms
7  Gi5-8.css-core-r1.net.umd.edu (128.8.0.85)  0.988 ms  0.595 ms  0.740 ms
8  Gi0-1.css-priv-r1.net.umd.edu (128.8.0.14)  1.211 ms  1.222 ms  0.866 ms
9  128.8.6.138 (128.8.6.138)  1.488 ms  4.728 ms  1.613 ms

10 naraapp10.umiacs.umd.edu (128.8.132.109) 0.987 ms 0.849 ms 0.865 ms

traceroute to 207.245.162.27 (207.245.162.27), 30 hops max, 38 byte packets

1  cfar-gw.cfar.umd.edu (128.8.132.1)  0.703 ms  0.532 ms  0.555 ms
2  Vlan8.css-priv-r1.net.umd.edu (128.8.6.129)  0.672 ms  0.622 ms  0.522 ms
3  Gi5-5.css-core-r1.net.umd.edu (128.8.0.13)  0.618 ms  0.369 ms  0.624 ms
4  Gi3-2.ptx-fw-r1.net.umd.edu (128.8.0.86)  0.541 ms  0.413 ms  0.588 ms
5  128.8.7.194 (128.8.7.194)  0.698 ms  0.669 ms  0.657 ms
6  128.8.7.201 (128.8.7.201)  0.738 ms  0.516 ms  0.479 ms
7  Gi2-8.css-max-r1.net.umd.edu (129.2.0.234)  1.153 ms  1.322 ms  1.255 ms
8  clpk-umd-i2.maxgigapop.net (206.196.177.125)  1.098 ms  0.988 ms  1.049 ms
9  nara-rtr.maxgigapop.net (206.196.177.14)  0.829 ms  0.740 ms  0.896 ms

10 erasrb01.nara.gov (207.245.162.27) 1.088 ms 1.030 ms 1.138 ms </verbatim>

• NARA to SDSC* - ~75ms

<verbatim> -bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu traceroute to srbbrick11.sdsc.edu (198.202.115.45), 30 hops max, 38 byte packets

1  207.245.162.1 (207.245.162.1)  0.240 ms  0.211 ms  0.127 ms
2  clpk-nara.maxgigapop.net (206.196.177.13)  6.966 ms  0.347 ms  0.365 ms
3  lvl3-clpk-10gigbb.maxgigapop.net (206.196.178.90)  1.235 ms  1.106 ms  1.116 ms
4  i2-lvl3.maxgigapop.net (206.196.178.46)  1.230 ms  1.226 ms  1.241 ms
5  so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6)  14.726 ms  14.842 ms  14.722 ms
6  hstnng-atlang.abilene.ucaid.edu (198.32.8.33)  36.339 ms  36.197 ms  36.222 ms
7  losang-hstnng.abilene.ucaid.edu (198.32.8.21)  67.811 ms  67.806 ms  67.820 ms
8  hpr-lax-gsr1--abilene-LA-10ge.cenic.net (137.164.25.2)  69.061 ms  68.805 ms  68.819 ms
9  riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5)  75.183 ms  74.947 ms  75.033 ms

10 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 74.804 ms 85.787 ms 74.691 ms 11 lightning.sdsc.edu (132.249.30.6) 74.932 ms 74.928 ms 74.814 ms 12 srbbrick11.sdsc.edu (198.202.115.45) 75.050 ms 74.924 ms 74.940 ms

traceroute to 207.245.162.27 (207.245.162.27), 30 hops max, 38 byte packets

1  stingray (198.202.112.15)  0.481 ms  0.458 ms  0.368 ms
2  198.202.112.6 (198.202.112.6)  22.859 ms  10.463 ms  0.858 ms
3  dolphin (132.249.30.17)  5.868 ms  0.596 ms  0.995 ms
4  hpr-lax-hpr--sdsc-10ge.cenic.net (137.164.27.168)  6.868 ms  6.723 ms *
5  nlr-packetnet--hpr-lax-hpr.cenic.net (137.164.26.131)  7.496 ms  7.218 ms  7.236 ms
6  hous-losa-87.layer3.nlr.net (216.24.186.31)  38.222 ms  38.085 ms  37.961 ms
7  atla-hous-70.layer3.nlr.net (216.24.186.9)  62.205 ms  73.809 ms  68.442 ms
8  wash-atla-64.layer3.nlr.net (216.24.186.21)  75.339 ms  75.792 ms  75.427 ms
9  216.24.184.11 (216.24.184.11)  74.816 ms  74.833 ms  74.823 ms

10 clpk-lvl3-10gigbb.maxgigapop.net (206.196.178.89) 75.064 ms 75.054 ms 75.070 ms

    MPLS Label=101776 CoS=0 TTL=1 S=1

11 nara-rtr.maxgigapop.net (206.196.177.14) 74.939 ms 75.057 ms 74.947 ms 12 erasrb01.nara.gov (207.245.162.27) 75.077 ms 75.053 ms 74.949 ms

</verbatim>

• UMD to SDSC* - ~75ms

<verbatim> -bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu naraapp10.umiacs.umd.edu traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets

1  stingray (198.202.112.15)  0.600 ms  0.459 ms  0.367 ms
2  198.202.112.6 (198.202.112.6)  0.615 ms  0.600 ms  0.487 ms
3  dolphin (132.249.30.17)  9.862 ms  0.596 ms  0.993 ms
4  hpr-lax-hpr--sdsc-10ge.cenic.net (137.164.27.168)  6.867 ms  6.850 ms  6.744 ms
5  nlr-packetnet--hpr-lax-hpr.cenic.net (137.164.26.131)  7.234 ms  7.100 ms  7.243 ms
6  hous-losa-87.layer3.nlr.net (216.24.186.31)  38.346 ms  38.328 ms  38.092 ms
7  atla-hous-70.layer3.nlr.net (216.24.186.9)  62.706 ms  62.061 ms  63.334 ms
8  wash-atla-64.layer3.nlr.net (216.24.186.21)  75.558 ms  75.551 ms  75.566 ms
9  216.24.184.11 (216.24.184.11)  74.813 ms  74.757 ms  74.939 ms

10 so-2-3-0.clpk-t640.maxgigapop.net (206.196.178.9) 74.936 ms 74.941 ms 75.052 ms

    MPLS Label=101296 CoS=0 TTL=1 S=1

11 umd-i2-rtr.maxgigapop.net (206.196.177.126) 74.930 ms 74.930 ms 74.942 ms 12 Gi3-5.ptx-fw-r1.net.umd.edu (129.2.0.233) 75.062 ms 75.185 ms 75.181 ms 13 128.8.7.202 (128.8.7.202) 75.436 ms 75.501 ms 75.471 ms 14 128.8.7.193 (128.8.7.193) 75.479 ms 75.286 ms 75.480 ms 15 Gi5-8.css-core-r1.net.umd.edu (128.8.0.85) 75.471 ms 75.970 ms 75.478 ms 16 Gi0-1.css-priv-r1.net.umd.edu (128.8.0.14) 75.977 ms 75.970 ms 75.985 ms 17 128.8.6.138 (128.8.6.138) 76.468 ms 76.329 ms 76.478 ms 18 naraapp10.umiacs.umd.edu (128.8.132.109) 76.477 ms 75.967 ms 75.983 ms

traceroute to 198.202.115.45 (198.202.115.45), 30 hops max, 38 byte packets

1  cfar-gw.cfar.umd.edu (128.8.132.1)  0.597 ms  0.528 ms  0.550 ms
2  Vlan8.css-priv-r1.net.umd.edu (128.8.6.129)  0.724 ms  0.600 ms  0.683 ms
3  Gi5-5.css-core-r1.net.umd.edu (128.8.0.13)  0.404 ms  0.446 ms  0.561 ms
4  Gi3-2.ptx-fw-r1.net.umd.edu (128.8.0.86)  0.527 ms  0.399 ms  0.628 ms
5  128.8.7.194 (128.8.7.194)  0.712 ms  0.928 ms  0.655 ms
6  128.8.7.201 (128.8.7.201)  0.636 ms  0.653 ms  0.488 ms
7  Gi2-8.css-max-r1.net.umd.edu (129.2.0.234)  2.408 ms  2.002 ms  2.559 ms
8  clpk-umd-i2.maxgigapop.net (206.196.177.125)  1.048 ms  0.907 ms  1.114 ms
9  abilene-backup-gige-rtr.maxgigapop.net (206.196.177.106)  2.342 ms  2.069 ms  1.923 ms

10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 15.201 ms 15.545 ms 15.445 ms 11 hstnng-atlang.abilene.ucaid.edu (198.32.8.33) 36.749 ms 36.474 ms 44.498 ms 12 losang-hstnng.abilene.ucaid.edu (198.32.8.21) 95.274 ms 68.257 ms 76.072 ms 13 hpr-lax-gsr1--abilene-LA-10ge.cenic.net (137.164.25.2) 69.657 ms 72.062 ms 69.617 ms 14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 75.632 ms 75.593 ms 75.947 ms 15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 75.558 ms 75.335 ms 75.272 ms 16 lightning.sdsc.edu (132.249.30.6) 75.501 ms 75.401 ms 75.473 ms 17 srbbrick11.sdsc.edu (198.202.115.45) 76.059 ms 75.745 ms 75.715 ms

</verbatim>

---++ Performance Settings

<verbatim> [root@naraapp10 tmp]# cat /proc/sys/net/ipv4/tcp_rmem 4096 87380 9375000 [root@naraapp10 tmp]# cat /proc/sys/net/ipv4/tcp_wmem 4096 16384 9375000 [root@naraapp10 tmp]# cat /proc/sys/net/core/rmem_max 9375000 [root@naraapp10 tmp]# cat /proc/sys/net/core/wmem_max 9375000 </verbatim>

-- Main.MikeSmorul - 05 Sep 2007

VM Name	Description
pawnrecv01	Primary pawn receiving server, scheduler, and chronopolis manager
lcpawn	LC Pawn demo server
fedoravm01	Fedora repository installation
naradev07	Yet another devel server
narawks05	winxp remote desktop box
eramonitor	srb-monitor webapp and mysql db