Personal tools

UpdateNetwork: Difference between revisions

From Adapt

Jump to: navigation, search
No edit summary
 
No edit summary
 
Line 104: Line 104:
| || throughput ||          ||          ||          ||        || ||
| || throughput ||          ||          ||          ||        || ||
|}
|}


* to sfs-tape: For 10 threads there were 3 failed attempts where 1-2 files would not finish.
* to sfs-tape: For 10 threads there were 3 failed attempts where 1-2 files would not finish.
* Load average on narasrb01 at umd was fairly high (2.0+) during the tests, w/ postgres running hot
* Load average on narasrb01 at umd was fairly high (2.0+) during the tests, w/ postgres running hot


 
===10meg test===
*10meg test*


300 files in 3 directories, each directory contains 100 10-m zero-filled files.
300 files in 3 directories, each directory contains 100 10-m zero-filled files.
 
{| class="wikitable" style="margin: 1em auto 1em auto" border="1"
| Test Case \ Threads | | 5  | 10  | 15  | 20  | 25  | 35 |
! Test Case \ Threads !! !! 5  !! 10  !! 15  !! 20  !! 25  !! 35
| narasrb03-unix to sfs-tape-nara | total |  1243620 | 671664 | 473424 | | 324716 |
|-
| | avg | 20461 | 21511 |  22905 | | 25467 |
! narasrb03-unix to sfs-tape-nara
| | throughput | 2.42MB/s | 4.47MB/s | | |  9.25MB/s |
|         total ||  1243620 ||  671664 || 473424 || ||    324716 ||
| | total |  1245558 | 669506 | | | 282670 |
|-
| | avg | 20616| 22064 | | | 21832 |
| ||        avg ||    20461 ||    21511 ||  22905 || ||    25467 ||
| | throughput | 2.41MB/s | 4.49MB/s | | | 10.63MB/s |
|-
| | total |  1249714 | 703995 | | | 315643 |
| || throughput || 2.42MB/s || 4.47MB/s ||        || ||  9.25MB/s ||
| | avg | 20671 | 21920 | | | 21619 |
|-
| | throughput | 2.40MB/s | 4.27MB/s | | |  9.52MB/s |
| ||      total ||  1245558 ||  669506 ||       || ||    282670 ||
| narasrb03-unix to ux-rocket03 | total | |   | | |   |
|-
| | avg | |   | | |   |
| ||        avg ||    20616 ||    22064 ||        || ||     21832 ||
| | throughput | |   | | |   |
|-
| | total | |   | | |   |
| || throughput || 2.41MB/s || 4.49MB/s ||        || || 10.63MB/s ||
| | avg | |   | | |   |
|-
| | throughput | |   | | |   |
| ||      total ||  1249714 ||  703995 ||        || ||    315643 ||
| | total | |   | | |   |
|-
| | avg | |   | | |   |
| ||        avg ||    20671 ||    21920 ||       || ||    21619 ||
| | throughput | |   | | |   |
|-
| narasrb03-unix to erasrb2-unix2* | total | 297814| 272686 |
| || throughput || 2.40MB/s || 4.27MB/s ||        || ||  9.52MB/s ||
| | avg | 4928 | 9006 |
|-
| | throughput | | 11.02MB/s |
! narasrb03-unix to ux-rocket3
| | total | | 271900 |
|         total || || || || || ||
| | avg | | 8958 |
|-
| | throughput | | 11.05MB/s |
| ||        avg || || || || || ||
| | total | | 274309 |
|-
| | avg | | 9043 |
| || throughput || || || || || ||
| | throughput | |  11.0MB/s |
|-
| ||      total || || || || || ||
|-
| ||        avg || || || || || ||
|-
| || throughput || || || || || ||
|-
| ||      total || || || || || ||
|-
| ||        avg || || || || || ||
|-
| || throughput || || || || || ||
|-
! narasrb03-unix to sfs-tape-nara
|         total || 297814 ||    272686 || || || ||
|-
| ||       avg ||  4928 ||      9006 || || || ||
|-
| || throughput ||       || 11.02MB/s || || || ||
|-
| ||      total ||       ||    271900 || || || ||
|-
| ||        avg ||       ||      8958 || || || ||
|-
| || throughput ||       || 11.05MB/s || || || ||
|-
| ||      total ||       ||    274309 || || || ||
|-
| ||        avg ||       ||      9043 || || || ||
|-
| || throughput ||        ||  11.0MB/s || || || ||
|}


Through 25 threads load average on umiacs mcat was negligable
Through 25 threads load average on umiacs mcat was negligable
Line 147: Line 176:
* erasrb02 is only on a 100mbit link, so 10MB shows saturation. Further testing showed saturation was achieved w/ only 2-3 threads.
* erasrb02 is only on a 100mbit link, so 10MB shows saturation. Further testing showed saturation was achieved w/ only 2-3 threads.


*100meg test*
===100meg test===


---++ Ping Reports
==Ping Reports==
*NARA to UMD* - ~1ms
===NARA to UMD===
<verbatim>
~1ms
<pre>
-bash-2.05b$ ./nuttcp -xt -p5005 naraapp10.umiacs.umd.edu
-bash-2.05b$ ./nuttcp -xt -p5005 naraapp10.umiacs.umd.edu
traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets
traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets
Line 176: Line 206:
  9  nara-rtr.maxgigapop.net (206.196.177.14)  0.829 ms  0.740 ms  0.896 ms
  9  nara-rtr.maxgigapop.net (206.196.177.14)  0.829 ms  0.740 ms  0.896 ms
10  erasrb01.nara.gov (207.245.162.27)  1.088 ms  1.030 ms  1.138 ms
10  erasrb01.nara.gov (207.245.162.27)  1.088 ms  1.030 ms  1.138 ms
</verbatim>
</pre>


*NARA to SDSC* - ~75ms
===NARA to SDSC===
<verbatim>
~75ms
<pre>
-bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu
-bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu
traceroute to srbbrick11.sdsc.edu (198.202.115.45), 30 hops max, 38 byte packets
traceroute to srbbrick11.sdsc.edu (198.202.115.45), 30 hops max, 38 byte packets
Line 210: Line 241:
12  erasrb01.nara.gov (207.245.162.27)  75.077 ms  75.053 ms  74.949 ms
12  erasrb01.nara.gov (207.245.162.27)  75.077 ms  75.053 ms  74.949 ms


</verbatim>
</pre>


*UMD to SDSC* - ~75ms
===UMD to SDSC===
<verbatim>
~75ms
<pre>
-bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu naraapp10.umiacs.umd.edu
-bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu naraapp10.umiacs.umd.edu
traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets
traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets
Line 254: Line 286:
16  lightning.sdsc.edu (132.249.30.6)  75.501 ms  75.401 ms  75.473 ms
16  lightning.sdsc.edu (132.249.30.6)  75.501 ms  75.401 ms  75.473 ms
17  srbbrick11.sdsc.edu (198.202.115.45)  76.059 ms  75.745 ms  75.715 ms
17  srbbrick11.sdsc.edu (198.202.115.45)  76.059 ms  75.745 ms  75.715 ms
</pre>


</verbatim>
==Performance Settings==


---++ Performance Settings
<pre>
 
<verbatim>
[root@naraapp10 tmp]# cat /proc/sys/net/ipv4/tcp_rmem  
[root@naraapp10 tmp]# cat /proc/sys/net/ipv4/tcp_rmem  
4096    87380  9375000
4096    87380  9375000
Line 268: Line 299:
[root@naraapp10 tmp]# cat /proc/sys/net/core/wmem_max  
[root@naraapp10 tmp]# cat /proc/sys/net/core/wmem_max  
9375000
9375000
</verbatim>
</pre>


-- Main.MikeSmorul - 05 Sep 2007
-- Main.MikeSmorul - 05 Sep 2007
{| class="wikitable" style="margin: 1em auto 1em auto" border="1"
|-
! VM Name || Description
|-
| pawnrecv01 || Primary pawn receiving server, scheduler, and chronopolis manager
|-
| lcpawn || LC Pawn demo server
|-
| fedoravm01 || Fedora repository installation
|-
| naradev07 || Yet another devel server
|-
| narawks05 || winxp remote desktop box
|-
| eramonitor || srb-monitor webapp and mysql db
|}

Latest revision as of 15:23, 12 September 2008

Baseline Throughput

Throughput for single-stream, using nuttcp. Hosts may not be storage resources, but machines w/ similar connections as mcat/storage nodes.

Untuned transmit/receive numbers in mbits/s. Raw Data

xmit \ recv naraapp10.umiacs.umd.edu erasrb01.nara.gov srbbrick11.sdsc.edu
naraapp10.umiacs.umd.edu X 298, 299, 299 3.5, 3.5, 3.5
erasrb01.nara.gov 426, 417, 415 X 3.6, 3.6, 3.6
srbbrick11.sdsc.edu 7.8, 8.0, 8.0 8.4, 8.2, 8.2 X

Tuned transmit/receive numbers in mbits/s. (using settings below) Raw Data

xmit \ recv naraapp10.umiacs.umd.edu erasrb01.nara.gov srbbrick11.sdsc.edu
naraapp10.umiacs.umd.edu X 454.35, 461.0, 461.16 168.79, 84.53, 168.37
erasrb01.nara.gov 373.67, 409.15, 339.57 X 35.53, 29.93, 29.71
srbbrick11.sdsc.edu 49.79, 57.83, 43.41 28.06, 29.34, 29.00 X

SRB Speed Tests

Speed tests were performed using three samples, 1000 x 1k files, 300 x 10m files, and 100 x 100m files. Tests were run starting w/ 5 threads and continuing until total plateaued and per-file time started increading. After this point, a bottleneck was reached, either mcat/srb or network limit.

  • Total - Total time in ms for transfer, from directory listing to last thread exiting
  • avg - Average per-file time (ms)
  • throughput - Aggregate throughput size / time.

1k test

1000 files in 10 directories, each directory contains 100 1-k zero-filled files.

Test Case \ Threads 5 10 15 20 25 35
narasrb03-unix to sfs-tape-nara total 774262 595247
avg 3846 5876
throughput 5.22KB/s 6.79KB/s
total 788026 506796
avg 3902 4870
throughput 5.13KB/s 7.98KB/s
total 746006
avg 3713
throughput 5.42KB/s
narasrb03-unix to ux-rocket3 total
avg
throughput
total
avg
throughput
total
avg
throughput
narasrb03-unix to sfs-tape-nara total 251424 221271 253956 182190
avg 1248 1964 3001 3506
throughput 16.08KB/s 18.28KB/s 15.92KB/s
total 193504 183960
avg 1890 2712
throughput 20.90KB/s
total
avg
throughput
  • to sfs-tape: For 10 threads there were 3 failed attempts where 1-2 files would not finish.
  • Load average on narasrb01 at umd was fairly high (2.0+) during the tests, w/ postgres running hot

10meg test

300 files in 3 directories, each directory contains 100 10-m zero-filled files.

Test Case \ Threads 5 10 15 20 25 35
narasrb03-unix to sfs-tape-nara total 1243620 671664 473424 324716
avg 20461 21511 22905 25467
throughput 2.42MB/s 4.47MB/s 9.25MB/s
total 1245558 669506 282670
avg 20616 22064 21832
throughput 2.41MB/s 4.49MB/s 10.63MB/s
total 1249714 703995 315643
avg 20671 21920 21619
throughput 2.40MB/s 4.27MB/s 9.52MB/s
narasrb03-unix to ux-rocket3 total
avg
throughput
total
avg
throughput
total
avg
throughput
narasrb03-unix to sfs-tape-nara total 297814 272686
avg 4928 9006
throughput 11.02MB/s
total 271900
avg 8958
throughput 11.05MB/s
total 274309
avg 9043
throughput 11.0MB/s

Through 25 threads load average on umiacs mcat was negligable

  • erasrb02 is only on a 100mbit link, so 10MB shows saturation. Further testing showed saturation was achieved w/ only 2-3 threads.

100meg test

Ping Reports

NARA to UMD

~1ms 

-bash-2.05b$ ./nuttcp -xt -p5005 naraapp10.umiacs.umd.edu
traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets
 1  207.245.162.1 (207.245.162.1)  0.109 ms  0.206 ms  0.129 ms
 2  clpk-nara.maxgigapop.net (206.196.177.13)  0.343 ms  0.346 ms  0.362 ms
 3  umd-i2-rtr.maxgigapop.net (206.196.177.126)  0.367 ms  0.350 ms  0.366 ms
 4  Gi3-5.ptx-fw-r1.net.umd.edu (129.2.0.233)  0.462 ms  0.470 ms  0.487 ms
 5  128.8.7.202 (128.8.7.202)  0.737 ms  0.850 ms  0.616 ms
 6  128.8.7.193 (128.8.7.193)  0.738 ms  0.842 ms  0.490 ms
 7  Gi5-8.css-core-r1.net.umd.edu (128.8.0.85)  0.988 ms  0.595 ms  0.740 ms
 8  Gi0-1.css-priv-r1.net.umd.edu (128.8.0.14)  1.211 ms  1.222 ms  0.866 ms
 9  128.8.6.138 (128.8.6.138)  1.488 ms  4.728 ms  1.613 ms
10  naraapp10.umiacs.umd.edu (128.8.132.109)  0.987 ms  0.849 ms  0.865 ms

traceroute to 207.245.162.27 (207.245.162.27), 30 hops max, 38 byte packets
 1  cfar-gw.cfar.umd.edu (128.8.132.1)  0.703 ms  0.532 ms  0.555 ms
 2  Vlan8.css-priv-r1.net.umd.edu (128.8.6.129)  0.672 ms  0.622 ms  0.522 ms
 3  Gi5-5.css-core-r1.net.umd.edu (128.8.0.13)  0.618 ms  0.369 ms  0.624 ms
 4  Gi3-2.ptx-fw-r1.net.umd.edu (128.8.0.86)  0.541 ms  0.413 ms  0.588 ms
 5  128.8.7.194 (128.8.7.194)  0.698 ms  0.669 ms  0.657 ms
 6  128.8.7.201 (128.8.7.201)  0.738 ms  0.516 ms  0.479 ms
 7  Gi2-8.css-max-r1.net.umd.edu (129.2.0.234)  1.153 ms  1.322 ms  1.255 ms
 8  clpk-umd-i2.maxgigapop.net (206.196.177.125)  1.098 ms  0.988 ms  1.049 ms
 9  nara-rtr.maxgigapop.net (206.196.177.14)  0.829 ms  0.740 ms  0.896 ms
10  erasrb01.nara.gov (207.245.162.27)  1.088 ms  1.030 ms  1.138 ms

NARA to SDSC

~75ms 

-bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu
traceroute to srbbrick11.sdsc.edu (198.202.115.45), 30 hops max, 38 byte packets
 1  207.245.162.1 (207.245.162.1)  0.240 ms  0.211 ms  0.127 ms
 2  clpk-nara.maxgigapop.net (206.196.177.13)  6.966 ms  0.347 ms  0.365 ms
 3  lvl3-clpk-10gigbb.maxgigapop.net (206.196.178.90)  1.235 ms  1.106 ms  1.116 ms
 4  i2-lvl3.maxgigapop.net (206.196.178.46)  1.230 ms  1.226 ms  1.241 ms
 5  so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6)  14.726 ms  14.842 ms  14.722 ms
 6  hstnng-atlang.abilene.ucaid.edu (198.32.8.33)  36.339 ms  36.197 ms  36.222 ms
 7  losang-hstnng.abilene.ucaid.edu (198.32.8.21)  67.811 ms  67.806 ms  67.820 ms
 8  hpr-lax-gsr1--abilene-LA-10ge.cenic.net (137.164.25.2)  69.061 ms  68.805 ms  68.819 ms
 9  riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5)  75.183 ms  74.947 ms  75.033 ms
10  hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50)  74.804 ms  85.787 ms  74.691 ms
11  lightning.sdsc.edu (132.249.30.6)  74.932 ms  74.928 ms  74.814 ms
12  srbbrick11.sdsc.edu (198.202.115.45)  75.050 ms  74.924 ms  74.940 ms

traceroute to 207.245.162.27 (207.245.162.27), 30 hops max, 38 byte packets
 1  stingray (198.202.112.15)  0.481 ms  0.458 ms  0.368 ms
 2  198.202.112.6 (198.202.112.6)  22.859 ms  10.463 ms  0.858 ms
 3  dolphin (132.249.30.17)  5.868 ms  0.596 ms  0.995 ms
 4  hpr-lax-hpr--sdsc-10ge.cenic.net (137.164.27.168)  6.868 ms  6.723 ms *
 5  nlr-packetnet--hpr-lax-hpr.cenic.net (137.164.26.131)  7.496 ms  7.218 ms  7.236 ms
 6  hous-losa-87.layer3.nlr.net (216.24.186.31)  38.222 ms  38.085 ms  37.961 ms
 7  atla-hous-70.layer3.nlr.net (216.24.186.9)  62.205 ms  73.809 ms  68.442 ms
 8  wash-atla-64.layer3.nlr.net (216.24.186.21)  75.339 ms  75.792 ms  75.427 ms
 9  216.24.184.11 (216.24.184.11)  74.816 ms  74.833 ms  74.823 ms
10  clpk-lvl3-10gigbb.maxgigapop.net (206.196.178.89)  75.064 ms  75.054 ms  75.070 ms
     MPLS Label=101776 CoS=0 TTL=1 S=1
11  nara-rtr.maxgigapop.net (206.196.177.14)  74.939 ms  75.057 ms  74.947 ms
12  erasrb01.nara.gov (207.245.162.27)  75.077 ms  75.053 ms  74.949 ms

UMD to SDSC

~75ms 

-bash-2.05b$ ./nuttcp -xt -p5005 srbbrick11.sdsc.edu naraapp10.umiacs.umd.edu
traceroute to naraapp10.umiacs.umd.edu (128.8.132.109), 30 hops max, 38 byte packets
 1  stingray (198.202.112.15)  0.600 ms  0.459 ms  0.367 ms
 2  198.202.112.6 (198.202.112.6)  0.615 ms  0.600 ms  0.487 ms
 3  dolphin (132.249.30.17)  9.862 ms  0.596 ms  0.993 ms
 4  hpr-lax-hpr--sdsc-10ge.cenic.net (137.164.27.168)  6.867 ms  6.850 ms  6.744 ms
 5  nlr-packetnet--hpr-lax-hpr.cenic.net (137.164.26.131)  7.234 ms  7.100 ms  7.243 ms
 6  hous-losa-87.layer3.nlr.net (216.24.186.31)  38.346 ms  38.328 ms  38.092 ms
 7  atla-hous-70.layer3.nlr.net (216.24.186.9)  62.706 ms  62.061 ms  63.334 ms
 8  wash-atla-64.layer3.nlr.net (216.24.186.21)  75.558 ms  75.551 ms  75.566 ms
 9  216.24.184.11 (216.24.184.11)  74.813 ms  74.757 ms  74.939 ms
10  so-2-3-0.clpk-t640.maxgigapop.net (206.196.178.9)  74.936 ms  74.941 ms  75.052 ms
     MPLS Label=101296 CoS=0 TTL=1 S=1
11  umd-i2-rtr.maxgigapop.net (206.196.177.126)  74.930 ms  74.930 ms  74.942 ms
12  Gi3-5.ptx-fw-r1.net.umd.edu (129.2.0.233)  75.062 ms  75.185 ms  75.181 ms
13  128.8.7.202 (128.8.7.202)  75.436 ms  75.501 ms  75.471 ms
14  128.8.7.193 (128.8.7.193)  75.479 ms  75.286 ms  75.480 ms
15  Gi5-8.css-core-r1.net.umd.edu (128.8.0.85)  75.471 ms  75.970 ms  75.478 ms
16  Gi0-1.css-priv-r1.net.umd.edu (128.8.0.14)  75.977 ms  75.970 ms  75.985 ms
17  128.8.6.138 (128.8.6.138)  76.468 ms  76.329 ms  76.478 ms
18  naraapp10.umiacs.umd.edu (128.8.132.109)  76.477 ms  75.967 ms  75.983 ms

traceroute to 198.202.115.45 (198.202.115.45), 30 hops max, 38 byte packets
 1  cfar-gw.cfar.umd.edu (128.8.132.1)  0.597 ms  0.528 ms  0.550 ms
 2  Vlan8.css-priv-r1.net.umd.edu (128.8.6.129)  0.724 ms  0.600 ms  0.683 ms
 3  Gi5-5.css-core-r1.net.umd.edu (128.8.0.13)  0.404 ms  0.446 ms  0.561 ms
 4  Gi3-2.ptx-fw-r1.net.umd.edu (128.8.0.86)  0.527 ms  0.399 ms  0.628 ms
 5  128.8.7.194 (128.8.7.194)  0.712 ms  0.928 ms  0.655 ms
 6  128.8.7.201 (128.8.7.201)  0.636 ms  0.653 ms  0.488 ms
 7  Gi2-8.css-max-r1.net.umd.edu (129.2.0.234)  2.408 ms  2.002 ms  2.559 ms
 8  clpk-umd-i2.maxgigapop.net (206.196.177.125)  1.048 ms  0.907 ms  1.114 ms
 9  abilene-backup-gige-rtr.maxgigapop.net (206.196.177.106)  2.342 ms  2.069 ms  1.923 ms
10  so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6)  15.201 ms  15.545 ms  15.445 ms
11  hstnng-atlang.abilene.ucaid.edu (198.32.8.33)  36.749 ms  36.474 ms  44.498 ms
12  losang-hstnng.abilene.ucaid.edu (198.32.8.21)  95.274 ms  68.257 ms  76.072 ms
13  hpr-lax-gsr1--abilene-LA-10ge.cenic.net (137.164.25.2)  69.657 ms  72.062 ms  69.617 ms
14  riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5)  75.632 ms  75.593 ms  75.947 ms
15  hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50)  75.558 ms  75.335 ms  75.272 ms
16  lightning.sdsc.edu (132.249.30.6)  75.501 ms  75.401 ms  75.473 ms
17  srbbrick11.sdsc.edu (198.202.115.45)  76.059 ms  75.745 ms  75.715 ms

Performance Settings

[root@naraapp10 tmp]# cat /proc/sys/net/ipv4/tcp_rmem 
4096    87380   9375000
[root@naraapp10 tmp]# cat /proc/sys/net/ipv4/tcp_wmem 
4096    16384   9375000
[root@naraapp10 tmp]# cat /proc/sys/net/core/rmem_max 
9375000
[root@naraapp10 tmp]# cat /proc/sys/net/core/wmem_max 
9375000

-- Main.MikeSmorul - 05 Sep 2007

Retrieved from "https://wiki.umiacs.umd.edu/adapt/index.php?title=UpdateNetwork&oldid=2023"