Before all the stuff below started, an IBM field engineering installed the NIC in esmf04 for me. Later the same day, I went to ifconfig the NIC, and the whole machine panic'd, and it turned out that the machine had been configured with too little dump space to save a complete VM image, so we didn't get much in the way of information about what went wrong. 2006-01-31 In diag... Command: OK stdout: yes stderr: no Before command completion, additional instructions may appear below. # Slot Description Device(s) U1.5-P1-I1 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I2 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I3 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I4 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I5 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I6 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I7 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I8 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I9 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P1-I10 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I1 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I2 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I3 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I4 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I5 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I6 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I7 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I8 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I9 PCI-X capable, 64 bit, 133MHz slot Empty U1.5-P2-I10 PCI-X capable, 64 bit, 133MHz slot Empty U1.10-P1-I1 PCI-X capable, 64 bit, 133MHz slot ent2 ent3 U1.10-P1-I2 PCI-X capable, 64 bit, 133MHz slot ent5 ent4 U1.10-P1-I3 PCI-X capable, 64 bit, 133MHz slot css0 So it about has to be U1.10-P1-I2 I'm interested in. Using identify now. Selected remove option Crud. The diag screen went black, just like with identify, so I went and pulled the card. But after I pulled the card, I came back and hit enter, and the dang diag program said it couldn't remove voltage because the device hadn't been unconfigured. So I tried to unconfigure U1.10-P1-I2, but it said that wasn't a known device name Anyway, I have the card pulled, and esmf07m appears to be idle. 2006-01-31 2:55PM The slot we need on esmf07m about has to be: Command: OK stdout: yes stderr: no Before command completion, additional instructions may appear below. # Slot Description Device(s) U1.23-P1-I1 PCI-X capable, 64 bit, 133MHz slot ent2 ent3 U1.23-P1-I2 PCI-X capable, 64 bit, 133MHz slot Empty U1.23-P1-I3 PCI-X capable, 64 bit, 133MHz slot css0 U1.23-P1-I2 Trying to "configure device" now. F4 is one's friend. | [TOP] | | L2cache0 Available 00-00 L2 Cache | | aio0 Defined Asynchronous I/O | | css0 Available 1V-08 SP Switch2 Communications Adapter | | csspd0 Available SP Switch2 Pseudo Device | | en0 Available 1Z-08 Standard Ethernet Network Interfac | | en1 Defined 1c-08 Standard Ethernet Network Interfac | | en2 Available 1n-08 Standard Ethernet Network Interfac | | en3 Defined 1n-09 Standard Ethernet Network Interfac | | ent0 Available 1Z-08 10/100 Mbps Ethernet PCI Adapter I | | ent1 Available 1c-08 10/100 Mbps Ethernet PCI Adapter I | | ent2 Available 1n-08 2-Port 10/100/1000 Base-TX PCI-X A | | ent3 Available 1n-09 2-Port 10/100/1000 Base-TX PCI-X A | | et0 Defined 1Z-08 IEEE 802.3 Ethernet Network Interf | | et1 Defined 1c-08 IEEE 802.3 Ethernet Network Interf | | et2 Defined 1n-08 IEEE 802.3 Ethernet Network Interf | | et3 Defined 1n-09 IEEE 802.3 Ethernet Network Interf | 2006-01-31 Found a dangling esmf04 cable, plugged it into esmf07's new adapter (it has two ports). Configured the interface for 128.200.197.161. Need a route... esmf07m-root> route add -net 128.200.0.0 -netmask 255.255.0.0 128.200.197.165 128.200.197.165 net 128.200.0.0: gateway 128.200.197.165 Tue Jan 31 16:26:06 esmf07m-root> route delete 128.200/16 128.200.197.165 128.200.197.165 net 128.200: gateway 128.200.197.165 Tue Jan 31 16:38:14 esmf07m-root> netstat -nr Routing tables Destination Gateway Flags Refs Use If PMTU Exp Groups Route Tree for Protocol Family 2 (Internet): default 192.168.1.1 UG 0 27 en0 - - - 127/8 127.0.0.1 U 11 6117917 lo0 - - - 128.200/16 128.200.197.129 UG 0 0 en3 - - - 128.200.197.161 127.0.0.1 UGHS 0 5 lo0 - - - 192.168.1/24 192.168.1.15 U 14 31757118 en0 - - - 192.168.1.15 127.0.0.1 UGHS 5 6425902 lo0 - - - 192.168.2/24 192.168.2.15 U 9 259037126 en2 - - - 192.168.2.15 127.0.0.1 UGHS 2 43 lo0 - - - 192.168.3/24 192.168.3.15 U 2 16273696 css0 - - - 192.168.3.15 127.0.0.1 UGHS 0 2683763 lo0 - - - Route Tree for Protocol Family 24 (Internet v6): ::1 ::1 UH 0 0 lo0 16896 - - Tue Jan 31 16:38:15 Tried both ports, but neither are working... 06-2-8 4:50 pm looking at the back of esmf07 unlabeled cable is in top port, esmf04-1 cable is in bottom port bottom port is 100% unlit top port is green on top (means active/link, other values are off: no link and blinking: active), and the bottom port is orange (means gige. Other values are off: 10baset and 100: green) cable (unlabeled one) is cat 5e upper port appears to have an "A" next to it. Lower one is unreadable due to the card faceplate being in the way pulled esmf04-1 cable, let it fall. Pulled unlabeled cable, put it in the bottom port, and it's again greenn and orange. putting cable back in the top 2006-02-10 2:37PM I learned yesterday, that this has replaced OPeNDAP as Charlie's highest priority for DCS-ESMF work. 2006-02-10 3:43PM Just wrote errpt-analysis. The manual procedure I explored the script with yielded this very relevant result. As it happens, the 31st is the day I put the adapter in esmf07. File 00000966 LABEL: GOENT_LINK_DOWN IDENTIFIER: DED8E752 Machine Id: 0020D45A4C00 Node Id: esmf07m Class: H Type: TEMP Resource Name: ent3 Resource Class: adapter Resource Type: 14108902 Location: 1n-09 VPD: Product Specific.( ).......2-Port 10/100/1000 Base-TX PCI-X Adapter Part Number.................00P4289 FRU Number..................00P4289 EC Level....................H12460 Manufacture ID..............YL1021 Network Address.............00096BAE3E0D ROM Level.(alterable).......DVL001 Description ETHERNET DOWN Probable Causes CABLE CSMA/CD ADAPTER Failure Causes LINK TIMEOUT Recommended Actions CHECK CABLE AND ITS CONNECTIONS Detail Data FILE NAME line: 181 file: goent_limbo.c PCI ETHERNET STATISTICS 0000 0005 0063 8853 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 DEVICE DRIVER INTERNAL STATE 5555 5555 0000 0000 0000 0000 SOURCE ADDRESS 0009 6BAE 3E0D
This is kind of interesting, no less because we tend to hard code a lot of our network equipment to avoid autonegotiation: esmf07m-root> entstat -d ent0 |grep "Media Speed" Media Speed Selected: 100 Mbps Full Duplex Media Speed Running: 100 Mbps Full Duplex Fri Feb 10 16:07:49 esmf07m-root> entstat -d ent3 |grep "Media Speed" Media Speed Selected: Auto negotiation Media Speed Running: Unknown Fri Feb 10 16:07:52
More detail from the same command: esmf07m-root> entstat -d ent3 /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ ETHERNET STATISTICS (ent3) : Device Type: 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902) Hardware Address: 00:09:6b:ae:3e:0d Elapsed Time: 10 days 0 hours 9 minutes 39 seconds Transmit Statistics: Receive Statistics: /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ Packets: 0 Packets: 0 Bytes: 0 Bytes: 0 Interrupts: 0 Interrupts: 0 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 1 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 0 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 0 Broadcast Packets: 0 Multicast Packets: 0 Multicast Packets: 0 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 0 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 General Statistics: /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 2000 Driver Flags: Up Broadcast Simplex Limbo 64BitSupport ChecksumTCP ChecksumOffload PrivateSegment DataRateSet 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902) Specific Statistics: /\/\/\/\/\/\/\/\/\/\/\/\/\/\ Link Status: Down Media Speed Selected: Auto negotiation Media Speed Running: Unknown PCI Mode: PCI-X (100-133) PCI Bus Width: 64-bit Jumbo Frames: Disabled TCP Segmentation Offload: Enabled TCP Segmentation Offload Packets Transmitted: 0 TCP Segmentation Offload Packet Errors: 0 Transmit and Receive Flow Control Status: Disabled Transmit and Receive Flow Control Threshold (High): 32768 Transmit and Receive Flow Control Threshold (Low): 24576 Transmit and Receive Storage Allocation (TX/RX): 16/48 Fri Feb 10 16:09:02
2006-02-13 3:30PM I'm realizing that esmf07 may have 6 network ports, not 4. en0 and en1 are 10/100 en2 and en3 are 10/100/1000, but en2 is on the "d" network, so it must've been a preexisting NIC. It -looked- like a new NIC, because nothing was using it, but that's because Francisco moved the "d" network to one of the 100BaseT adapters a while back... That means I should be getting an en4 and en5 that are gigabit, but I'm not, so all this stuff with the IP address, routing table, moving cables from port to port, and later getting Iglesias to swap cables, are a bunch of red herrings. :)
2006-02-13 3:36PM I just scanned briefly back a bit in this memo, and noticed that diag -does- see ent4 and ent5. :) Or rather it used to. Now, when I got to the hotplug PCI screen in diag, and list the hotpluggable devices, I see: Command: OK stdout: yes stderr: no Before command completion, additional instructions may appear below. # Slot Description Device(s) U1.23-P1-I1 PCI-X capable, 64 bit, 133MHz slot ent2 ent3 U1.23-P1-I2 PCI-X capable, 64 bit, 133MHz slot Unknown Unknown U1.23-P1-I3 PCI-X capable, 64 bit, 133MHz slot css0 ...which seems to suggest a bad adapter.