Note: This web page was automatically created from a PalmOS "pedit32" memo.
Optiputer notes
Garrett to Charlie Z, 17 Aug 2005:
Hi Charlie,
> I'd also like to know the status of the switch which connects
> ESMF to the campus backbone, and hence ESS. I believe Dana Roode
> agreed that NACS would buy that. Are you ordering it at the same
> time as the jumbo frame 3750s?
For the switch which Dana agreed to pay for to connect ESMF to the
backbone at GE speed, we took one out of spares stock and installed
it, and a new one is being ordered to replace it on NACS dime.
Regarding the installation of it, we had to put in a new rack and
run some new fiber, but all of this work is now complete.
I just spoke to Francisco Lopez and asked him if DCS had yet
received the gigabit ethernet card needed to connect ESMF to the
new switch, and the answer was "no." I've asked him to see if he
can get that moving.
By the way, so people are clear about which switch we are talking
about relative to the ESMF system we are calling this one the
"co-location switch" and the one for the optiputer connection the
"optiputer switch." And yes, the co- location switch does support
jumbo frames.
I now understand Harry's confusion about the optiputer connection
being at 100 mbps. While it is actually at 1,000 mbps (gigabit) the
co-location connection--e.g., campus backbone connection--is still
at 100. This is because ESMF currently has only one GE NIC (network
interface card). That is what Francisco needs to take care of.
Regards,
Garrett
Hi Dan,
I am not aware of any resources/documentation.
We have access (or we used to have) to two blocks of IP addresses.
137.110.247.210 - 137.110.247.222 with a netmask of 255.255.255.240
and a gateway of 137.110.247.209
137.110.247.242-246 gateway 137.110.247.241 netmask 255.255.255.248
These blocks of IP addresses used to be on different waves and thus
could not talk to each other but I believe that due to the new switch
at UCI that should not matter any more (but I may be wrong).
Steve Jenks got SPDS working using 137.110.247.211 and the first
netmask and gateway.
If the new switch means that which wave you are on does not matter
then you should be able to use another address from this block.
However, if they are still set up the way they were then you would
have to use an address from the second block as SPDS is in Calit2 and
is on the wave that ESMF is not (I think).
But if addresses from neither of these blocks are working then the
issues are beyond my experience. Apart from 211 I do not think that
any of the other addresses are being used currently.
Steve might be able to give you an account on SPDS - I have added him
to the cc list on this email.
Hiperwall is not currently on the optiputer as I need to ask Garrett
to change the physical location of our optiputer outlet. I am about
to email that request to him. So I will be able to get an account for
you on hiperwall with which you can test optiputer is there is an
issue with SPDS. SPDS would be preferable as it is a little more stable.
I hope that helps,
Chris
Dan:
This is somewhat more like it. I added those 137.110.*.* routes via
smit on the ESMF. :) Strangely, the traceroute'ing didn't work at
first, nor did ping'ing, but after retrying a couple of times, they both
did. I got the correct address to traceroute to by logging into the
system and running "ifconfig -a", and then sifting briefly through the
addresses listed to find one that looked like an optiputer address.
esmf04m-root> netstat -nr
Routing tables
Destination Gateway Flags Refs Use If
PMTU Exp Groups
Route Tree for Protocol Family 2 (Internet):
default 128.200.197.129 UG 59 197674627 en1
1500 - -
127/8 127.0.0.1 U 11 2361266 lo0 -
- -
128.200.197.128/25 128.200.197.165 U 1 0 en1
- - -
128.200.197.165 127.0.0.1 UGHS 0 47404 lo0 -
- -
137.110.243/24 137.110.247.209 UG 0 73 en3 -
- -
137.110.244/24 137.110.247.209 UG 0 0 en3 -
- -
137.110.245/24 137.110.247.209 UG 0 0 en3 -
- -
137.110.246/25 137.110.247.209 UG 0 0 en3 -
- -
137.110.246.128/25 137.110.247.209 UG 0 0 en3
- - -
137.110.247/24 137.110.247.209 UG 1 0 en3 -
- -
192.168.1/24 192.168.1.12 U 55 29143815 en0 -
- -
192.168.1.12 127.0.0.1 UGHS 39 30502762 lo0 -
- -
192.168.2/24 192.168.2.12 U 7 945544387 en2
- - -
192.168.2.12 127.0.0.1 UGHS 0 57200 lo0 -
- -
192.168.3/24 192.168.3.12 U 2 5532884 css0 -
- -
192.168.3.12 127.0.0.1 UGHS 0 912133 lo0 -
- -
Route Tree for Protocol Family 24 (Internet v6):
::1 ::1 UH 0 0 lo0
16896 - -
Thu Aug 04 16:42:38
esmf04m-root> traceroute 137.110.243.3
trying to get source for 137.110.243.3
source should be 137.110.247.222
traceroute to 137.110.243.3 (137.110.243.3) from 137.110.247.222
(137.110.247.222), 30 hops max
outgoing MTU = 1500
1 137.110.247.209 (137.110.247.209) 4 ms 4 ms 4 ms
2 137.110.247.193 (137.110.247.193) 4 ms 4 ms 4 ms
3 rockstar.sdsc.optiputer.net (137.110.243.3) 4 ms 4 ms 4 ms
Thu Aug 04 16:42:41
esmf04m-root>
Entered on 08/04/2005 at 15:44:09 by Sean OConnell:
Dan-
We are currently using static routes for the optiputer network:
add -net 137.110.243.0 netmask 255.255.255.0 gw <your_opitputer_gateway>
add -net 137.110.244.0 netmask 255.255.255.0 gw ..
add -net 137.110.245.0 netmask 255.255.255.0 gw ..
add -net 137.110.246.0 netmask 255.255.255.128 gw ..
add -net 137.110.246.128 netmask 255.255.255.128 gw
add -net 137.110.247.0 netmask 255.255.255.0
137.110.243 -> SDSC (rockstar is 137.110.243.4)
137.110.244 -> SIO
137.110.245 -> CSE
137.110.246.0 -> JSOE (storage)
137.110.246.128 -> NCMIR (UCSD, School of Medicine)
137.110.247.0 -> others
Dan: ESMF has the following adapters, 2005-08-26:
sa0 Available 01-S1 Standard I/O Serial Port
sisscsia0 Available 1H-08 PCI-X Dual Channel Ultra320 SCSI Adapter
ent0 Available 1Z-08 10/100 Mbps Ethernet PCI Adapter II (1410ff01)
ent1 Available 1c-08 10/100 Mbps Ethernet PCI Adapter II (1410ff01)
ent2 Available 1n-08 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
ent3 Available 1n-09 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
scsi0 Available 2s-08 Wide/Ultra-3 SCSI I/O Controller
scsi1 Available 37-08 Wide/Ultra-3 SCSI I/O Controller
scsi2 Available 3b-08 Wide/Ultra-3 SCSI I/O Controller
scsi3 Available 3s-08 Wide/Ultra-3 SCSI I/O Controller
css0 Available 1V-08 SP Switch2 Communications Adapter
en0 100BaseT "management"
en1 100BaseT "campus"
en2 1000BaseT "data"
en3 1000BaseT "optiputer"
css0 SP2 "s"
en0:
flags=7e080863,10<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHECKSUM_SUPPORT,PSEG>
inet 192.168.1.12 netmask 0xffffff00 broadcast 192.168.1.255
en1:
flags=7e080863,10<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHECKSUM_SUPPORT,PSEG>
inet 128.200.197.165 netmask 0xffffff80 broadcast 128.200.197.255
en2:
flags=7e080863,50<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHECKSUM_SUPPORT,PSEG>
inet 192.168.2.12 netmask 0xffffff00 broadcast 192.168.2.255
tcp_sendspace 262144 tcp_recvspace 131072 rfc1323 1
css0: flags=800843<UP,BROADCAST,RUNNING,SIMPLEX>
inet 192.168.3.12 netmask 0xffffff00 broadcast 192.168.3.255
en3:
flags=7e080863,50<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHECKSUM_SUPPORT,PSEG>
inet 137.110.247.222 netmask 0xfffffff0 broadcast 137.110.247.223
tcp_sendspace 262144 tcp_recvspace 131072 rfc1323 1
lo0:
flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1/0
tcp_sendspace 65536 tcp_recvspace 65536
esmf04m-root> mmtu -s
Common MTUs:
65535 32000 17914 8166 4352 2002 1492 1006 508 296 68
Sat Aug 27 14:46:45
esmf04m-root> netstat -i -f inet
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 0.9.6b.e8.9.aa 85182316 0 44799410 0 0
en0 1500 192.168.1 esmf04m 85182316 0 44799410 0 0
en1 1500 link#3 0.9.6b.18.a.a1 940563275 0 1776695114
0 0
en1 1500 128.200.197 esmf 940563275 0 1776695114
0 0
en2 9000 link#4 0.9.6b.ae.3d.da 737463585 0 470579856 4 0
en2 9000 192.168.2 esmf04d 737463585 0 470579856 4 0
css0 65504 link#5 8975155 0 8852492 0 0
css0 65504 192.168.3 esmf04s 8975155 0 8852492 0 0
en3 9000 link#6 0.9.6b.ae.3d.db 10342904 0 1600287 2 0
en3 9000 137.110.247 esmf04o 10342904 0 1600287 2 0
lo0 16896 link#1 44897389 0 44956729 0 0
lo0 16896 127 loopback 44897389 0 44956729 0 0
lo0 16896 ::1 44897389 0 44956729 0 0
On Tue, 16 Aug 2005 at 17:14 Charlie Zender wrote:
> Hi Dan,
>
> I know you're working on the DDSS. Regardless of the status of that
> we'd like you to come to the next NCO meeting, Monday August 29 at
> 11:00. The goal is to clear-up/strategize ensuring we understand the
> optiputer connection and know how to maximize and interpret our
> benchmarks, both ESMF<->SPDS and ESMF<->UCSD.
> Garrett, it might be helpful if you came, too.
As a reminder, that meeting is scheduled to be held in Calit2 2301
(Vizlab) so we can get tan from the glow of HIPerWall (and show ESS
data on it).
Steve (Jenks)
When benchmarking, be careful not to test with, for example,
spds1.calit2.uci.edu. It's important to instead use 137.110.247.211.
The latter is one of the former's IP addresses, and probably the only
one that's on the optiputer.
Benchmark Summary:
2 Gigabytes from ESMF to spds1 via ssh, measured with reblock: 45.8
megabits/second
679 Megabytes from ESMF to spds1 via iperf: 569 megabits/second
iperf from spds1 to ESMF failed because of routing problem?
Packet lengths from ESMF to spds1 during an ssh data transfer has packets
from 66 to 17554 bytes
2 Gigabyte netcats from ESMF to spds1. Many matlabs on ESMF04 at the time
356 megabits/second
121 megabits/second
92 megabits/second
netio got as much as 537 megabits/second from ESMF to spds1
traceroute appears useless on optiputer, at least from ESMF to spds1o
ssh appears to be using 16K blocks for read and write. iperf is using
8K packets.
Actual benchmark results (gory detail):
2 Gigabytes from ESMF to spds1 via ssh, measured with reblock:
esmf04m-strombrg> dd if=/dev/zero bs=1024k count=2048 | reblock -e
$[1024*2048] $[1024*1024] 300 | ssh 137.110.247.211 'cat > /dev/null'
stdin not seekable - will not give percentage
Estimated filetransfer size is 2147483648 bytes
Estimated percentages will only be as accurate as your size estimate
(estimate: 13.9% 10s 6m) Kbytes: 292864.0 Mbits/s: 38.0 Gbytes/hr:
16.7 min: 1.0
(estimate: 29.7% 42s 4m) Kbytes: 624640.0 Mbits/s: 40.6 Gbytes/hr:
17.8 min: 2.0
(estimate: 39.7% 32s 4m) Kbytes: 834560.0 Mbits/s: 36.2 Gbytes/hr:
15.9 min: 3.0
(estimate: 48.1% 18s 4m) Kbytes: 1010688.0 Mbits/s: 32.8 Gbytes/hr:
14.4 min: 4.0
(estimate: 72.1% 55s 1m) Kbytes: 1512448.0 Mbits/s: 39.3 Gbytes/hr:
17.3 min: 5.0
2048+0 records in) Kbytes: 2096128.0 Mbits/s: 45.8 Gbytes/hr: 20.1
min: 5.9
2048+0 records out
(estimate: 100.0% ) Kbytes: 2097152.0 Mbits/s: 45.8 Gbytes/hr: 20.1
min: 5.9
Fri Aug 26 15:27:51
679M from ESMF to spds1:
esmf04m-strombrg> iperf -c 137.110.247.211
Client connecting to 137.110.247.211, TCP port 5001
TCP window size: 256 KByte (default)
[ 3] local 137.110.247.222 port 41066 connected with 137.110.247.211 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 679 MBytes 569 Mbits/sec
Fri Aug 26 15:34:44
spds1 to ESMF (failed):
[strombrg@spds1 bin]$ dd if=/dev/zero bs=1024k count=2048 | reblock -e
$[1024*2048] $[1024*1024] 300 | ssh strombrg@137.110.247.223 'cat >
/dev/null'
ssh: connect to host 137.110.247.223 port 22: Network is unreachable
stdin not seekable - will not give percentage
Estimated filetransfer size is 2147483648 bytes
Estimated percentages will only be as accurate as your size estimate
2+0 records in
1+0 records out
[strombrg@spds1 bin]$ ping 137.110.247.223
Do you want to ping broadcast? Then -b
[strombrg@spds1 bin]$
Packet length counts from ESMF to spds1 during an ssh test. Incredible
packet sizes. I didn't think jumbo frames would exceed 9200 (a little
less than that, actually). Note the large number of tinygrams:
esmf04m-root> /usr/local/sbin/tcpdump -c 10000 -e -v -i en3 host
137.110.247.211 | grep ' length ' | sed 's/^.* length: \([0-9]*\)).*$/\1/'
| histogram | sort -n +1
tcpdump: listening on en3, link-type EN10MB (Ethernet), capture size 96 bytes
10000 packets captured
10007 packets received by filter
0 packets dropped by kernel
8193 52
299 100
55 556
10 604
19 1060
21 1108
753 15980
279 16436
134 16988
1 17100
1 17428
8 17492
227 17540
Sat Aug 27 14:38:11
Packet length counts during an iperf from ESMF to spds1o:
esmf04m-root> /usr/local/sbin/tcpdump -c 10000 -e -v -i en3 host
137.110.247.211 | grep ' length ' | sed 's/^.* length: \([0-9]*\)).*$/\1/'
| histogram | sort -n +1
tcpdump: listening on en3, link-type EN10MB (Ethernet), capture size 96 bytes
10000 packets captured
10002 packets received by filter
0 packets dropped by kernel
8826 52
279 33356
378 34804
356 36252
147 37700
14 39148
Sat Aug 27 14:39:28
Three netcats from ESMF to spds1:
[strombrg@spds1 bin]$ nc -l -p 5000 > /dev/null
esmf04m-strombrg> dd if=/dev/zero bs=1024k count=2048 | reblock -e
$[1024*2048] $[1024*1024] 300 | nc 137.110.247.211 5000
stdin not seekable - will not give percentage
Estimated filetransfer size is 2147483648 bytes
Estimated percentages will only be as accurate as your size estimate
2048+0 records in) Kbytes: 2096128.0 Mbits/s: 356.2 Gbytes/hr: 156.5
min: 0.7
2048+0 records out
(estimate: 100.0% ) Kbytes: 2097152.0 Mbits/s: 356.2 Gbytes/hr: 156.5
min: 0.7
^C punt!
^C
Fri Aug 26 16:04:14
esmf04m-strombrg> dd if=/dev/zero bs=1024k count=2048 2> /dev/null |
reblock -e $[1024*2048] $[1024*1024] 300 | nc 137.110.247.211 5000
stdin not seekable - will not give percentage
Estimated filetransfer size is 2147483648 bytes
Estimated percentages will only be as accurate as your size estimate
(estimate: 43.7% 17s 1m) Kbytes: 917504.0 Mbits/s: 119.2 Gbytes/hr:
52.4 min: 1.0
(estimate: 80.7% 28s) Kbytes: 1693696.0 Mbits/s: 109.9 Gbytes/hr:
48.3 min: 2.0
(estimate: 100.0% ) Kbytes: 2097152.0 Mbits/s: 121.6 Gbytes/hr: 53.4
min: 2.2
^C punt!
^C
Fri Aug 26 16:07:00
esmf04m-strombrg> dd if=/dev/zero bs=1024k count=2048 2> /dev/null |
reblock -e $[1024*2048] $[1024*1024] 300 | nc 137.110.247.211 5000
stdin not seekable - will not give percentage
Estimated filetransfer size is 2147483648 bytes
Estimated percentages will only be as accurate as your size estimate
(estimate: 56.5% 46s) Kbytes: 1185792.0 Mbits/s: 154.1 Gbytes/hr:
67.7 min: 1.0
(estimate: 84.2% 22s) Kbytes: 1767424.0 Mbits/s: 114.7 Gbytes/hr:
50.4 min: 2.0
(estimate: 100.0% ) Kbytes: 2097152.0 Mbits/s: 92.4 Gbytes/hr: 40.6
min: 2.9
esmf04m-strombrg> ./netio 137.110.247.211
NETIO - Network Throughput Benchmark, Version 1.14
(C) 1997-2001 Kai Uwe Rommel
TCP/IP connection established.
Packet size 1 KByte: 51156 KByte/s
Packet size 2 KByte: 66605 KByte/s
Packet size 4 KByte: 66743 KByte/s
Packet size 8 KByte: 67772 KByte/s
Packet size 16 KByte: 68200 KByte/s
Packet size 32 KByte: 67649 KByte/s
Packet size 64 KByte: 66962 KByte/s
Packet size 128 KByte: 65433 KByte/s
Packet size 256 KByte: 67686 KByte/s
Packet size 512 KByte: 68557 KByte/s
Packet size 1024 KByte: 68608 KByte/s
Packet size 2048 KByte: 68608 KByte/s
Packet size 4096 KByte: 68813 KByte/s
Fri Aug 26 17:08:27
During a netcat from ESMF to spds1o that got ~400 megabits/second:
esmf04m-root> /usr/local/sbin/tcpdump -c 10000 -e -v -i en3 host
137.110.247.211 | grep ' length ' | sed 's/^.* length: \([0-9]*\)).*$/\1/'
| histogram | sort -n +1
tcpdump: listening on en3, link-type EN10MB (Ethernet), capture size 96 bytes
10000 packets captured
23384 packets received by filter
13298 packets dropped by kernel
9153 52
1 468
5 772
1 964
1 996
1 1124
1 1308
1 1460
1 1500
1 1852
1 2444
2 2948
1 4084
2 4396
1 6020
5 7292
1 7508
34 8244
7 8740
1 9452
1 11636
1 14588
1 15252
1 16204
1 16820
1 17084
1 17108
1 17852
1 18876
3 23220
1 23524
2 24668
1 25044
1 29012
1 29524
1 29732
1 29956
1 30460
1 31180
4 31908
125 33356
572 34804
19 36252
3 37700
34 40596
Sat Aug 27 14:41:36
esmf04m-root> ./pchar -I 256 -R 32 -t 1 -m 18000 137.110.247.211
pchar to 137.110.247.211 (137.110.247.211) using UDP/IPv4
Using raw socket input
Packet size increments from 32 to 18000 by 256
70 test(s) per repetition
32 repetition(s) per hop
0: 137.110.247.222 (esmf04o)
Partial loss: 2080 / 2240 (92%)
Partial char: rtt = 0.264682 ms, (b = 0.000148 ms/B), r2 = 0.960070
stddev rtt = 0.022220, stddev b = 0.000017
Partial queueing: avg = 0.000761 ms (5156 bytes)
Hop char: rtt = 0.264682 ms, bw = 54192.920437 Kbps
Hop queueing: avg = 0.000761 ms (5156 bytes)
1: 137.110.247.211 (spds1o)
Path length: 1 hops
Path char: rtt = 0.264682 ms r2 = 0.960070
Path bottleneck: 54192.920437 Kbps
Path pipe: 1792 bytes
Path queueing: average = 0.000761 ms (5156 bytes)
Start time: Sat Aug 27 11:24:08 2005
End time: Sat Aug 27 12:29:39 2005
Sat Aug 27 12:29:40
From Praveen, 2005-09-14:
There are some problems with rockstar network. There is 50 % loss to
137.110.247.222. storage.optiputer.net is fine though.
137.110.247.211 seems to be filtered on our network. I will look at
cisco to see if i can unblock it.
DNS servers for optiputer (Heard from Frank, Frank says Garrett told him)
132.239.1.52
128.54.16.2
137.110.247.209 is the gateway for ESMF and hiperwall
Adding a Redhat Enterprise (including Rocks and CentOS) to the optiputer:
Defining your default route:
This is setting up the default gateway, to UCInet (but be sure
you verify that you're using the correct IP address):
1. My reference (obtained with google):
http://www.linux.com/guides/html/chapter07/network.shtml
2. What I did:
[root@hiperstore fallback-reboot]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hiperstore
[root@hiperstore fallback-reboot]# cat >> /etc/sysconfig/network
GATEWAY=128.195.169.1
GATEWAYDEV=eth0
GATEWAY_IF=eth0
[root@hiperstore fallback-reboot]#
3. Why I did it that way:
* The doc said to do GATEWAY and GATEWAY_IF
* The script seemed to want GATEWAY and GATEWAYDEV
* I didn't think it'd hurt to use both device variables
Defining your static routes for the optiputer:
I put the following content:
any net 137.110.243.0 netmask 255.255.255.0 gw 137.110.247.209
any net 137.110.244.0 netmask 255.255.255.0 gw 137.110.247.209
any net 137.110.245.0 netmask 255.255.255.0 gw 137.110.247.209
any net 137.110.246.0 netmask 255.255.255.128 gw 137.110.247.209
any net 137.110.246.128 netmask 255.255.255.128 gw 137.110.247.209
any net 137.110.247.0 netmask 255.255.255.0 gw 137.110.247.209
...in the file:
/etc/sysconfig/static-routes
...based on the information on static routes at
http://www.siliconvalleyccie.com/linux-hn/appendix-diff.htm
and a quick
glance at /etc/init.d/network .
Ideally this will mean that when hiperstore is rebooted, it'll come up
with some useful optiputer static routes.
Please let me know if any of this doesn't work for you so I can adjust
my notes accordingly.