20131107

Aquilion8 - will not boot

TSX-101A/FB, FDB0562035, V1.60*R002, Base: V2.06*R000
Console: CKCN-012C/1A, s/n: 1AA0562407

Phenomenon
After turning ON the power, the S-con won't complete the start-up.

Observations and interventions

It had already been attempted to tackle this issue by various means, checks and replacement of
the following parts, before on-site technical support was sought.

Parts replaced prior to seeking on-site support:

• RTM
• RTM-D
• MHR-FC in slots 19 and 18
• S-con PC
• FC card in S-con
• Recon Box back plane/chassis

Checks performed:
• Power supplies confirmed
• Running with minimum configuration: MHR-FC (slot 19), MHR-FC (slot 18), MHR-DP (slot 17)
• Connected fiber cable direct from S-con PC FC card to MHR-FC in slot18

The RTM boot log always indicated the same thing (after each intervention). Here is a copy of the
boot log in blue.
3
2¢É£¹J§©µ°”¿Íèj¿–º=³00°©!±«É¥¤¹©©!±«É¥¤¹©©”2£¹J'¤¹©©
0x1fffdf8 (tRootTask): miiPhyInit check cable connection
Attached TCP/IP interface to dc unit 0
Attaching interface lo0...done

Adding 3247 symbols for standalone.

]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]] ]]]] ]]]]]]]]]] ]] ]]]] (R)
] ]]]]]]]]] ]]]]]] ]]]]]]]] ]] ]]]]
]] ]]]]]]] ]]]]]]]] ]]]]]] ] ]] ]]]]
]]] ]]]]] ] ]]] ] ]]]] ]]] ]]]]]]]]] ]]]] ]] ]]]] ]] ]]]]]
]]]] ]]] ]] ] ]]] ]] ]]]]] ]]]]]] ]] ]]]]]]] ]]]] ]] ]]]]
]]]]] ] ]]]] ]]]]] ]]]]]]]] ]]]] ]] ]]]] ]]]]]]] ]]]]
]]]]]] ]]]]] ]]]]]] ] ]]]]] ]]]] ]] ]]]] ]]]]]]]] ]]]]
]]]]]]] ]]]]] ] ]]]]]] ] ]]] ]]]] ]] ]]]] ]]]] ]]]] ]]]]
]]]]]]]] ]]]]] ]]] ]]]]]]] ] ]]]]]]] ]]]] ]]]] ]]]] ]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]]]] Development System
]]]]]]]]]]]]]]]]]]]]]]]]]]]]
]]]]]]]]]]]]]]]]]]]]]]]]]]] VxWorks version 5.4.2
]]]]]]]]]]]]]]]]]]]]]]]]]] KERNEL: WIND version 2.5
]]]]]]]]]]]]]]]]]]]]]]]]] Copyright Wind River Systems, Inc., 1984-1999

CPU: Motorola MCP750 - MPC 750. Processor #0.

Memory Size: 0x2000000. BSP version 1.2/43.
WDB: Ready.
RTM Version: 1.2/1406.
->
Press any key for service menu...

2
1
0

Starting...

Last reset reason: Front Panel Button Reset (0x02)

RTM Tornado Target: T2CP4 for MCP750/MCP820/STAR Release 27
RTM Hardware: Motorola MCP750 - MPC 750
RTM Serial Number: <ff><ff><ff><ff>UUUUUUUU
RTM Software Load: 1406
Found Backplane Bridge Type INTEL : bus 1

Found 6 G4 processors(s):

PCI VxWorks Board Bridge Serial

CPU Slot Node Size Revision Type Type Number
--- ---- ---- ---- -------- ------- ------ --------
1 19 A 64 6 G4-DTB INTEL 13694y16
2 19 B 128 6 G4-DTB INTEL 13694y16
3 18 A 4 6 G4-DTB INTEL 08565928
4 18 B 4 6 G4-DTB INTEL 08565928
5 17 A 4 6 G4-Proc INTEL 05525414
6 17 B 4 6 G4-Proc INTEL 05525414

ISPStartup.c 95 tStart | Using firmware version 2.02.03 for ISP2200 Qlogic

ISPStartup.c 97 tStart | Using firmware version 4.00.26 for ISP2422 Qlogic
ISPStartup.c 99 tStart | Looking for ISP2200/EP2422 devices...
ISPStartup.c 142 tStart | ISP Device ID 2200 found in slot 19, node 1, interrupt 0x1b
ISPStartup.c 166 tStart |
DFH.c 452 tTS | DFH initialized OK
ISPCmds.c 429 tISPHost | ISP2200 Rev:01, RISC Rev:03, FB&FPM Rev:0405, RISC ROM Rev:04
ISPCmds.c 318 tISPHost | ISP2200 started successfully.
ISPHost.c 331 tISPHost | Waiting for Fibre Channel Communications to start.
ISPHost.c 1020 tISPHost | LIP (Loop Initialization Procedure Complete)
ISPHost.c 1103 tISPHost | Fibre Channel Communication started. FC loop is working.

ISPHost.c 1265 tISPHost | WARNING, Got SGI Loop ID 0x00, expected 0x40
ISPHost.c 1274 tISPHost | errno:0, 0x40040005: <<MISSING devices on Host Fibre Channel>>
ISPHost.c 1277 tISPHost | errno:0, 0x40040005: <<Expected 3 devices, Only 6 found >>
ISPHost.c 1290 tISPHost | 6 devices found: 0x78 0x79 0x7a 0x7b 0x7c 0x41
ISPHost.c 1343 tISPHost | Sent ISP ready message to RDD.
ISPHost.c 1037 tISPHost | Some port(s) changed ...
ISPHost.c 1265 tISPHost | WARNING, Got SGI Loop ID 0x00, expected 0x40
ISPHost.c 1274 tISPHost | errno:0, 0x40040005: <<MISSING devices on Host Fibre Channel>>
ISPHost.c 1277 tISPHost | errno:0, 0x40040005: <<Expected 3 devices, Only 6 found >>
ISPHost.c 1290 tISPHost | 6 devices found: 0x78 0x79 0x7a 0x7b 0x7c 0x41
0xfbaa78 (tNetTask): miiPhyInit check cable connection
The following lines of the log are not correct and were of concern:
PCI VxWorks Board Bridge Serial
CPU Slot Node Size Revision Type Type Number
--- ---- ---- ---- -------- ------- ------ --------
1 19 A 64 6 G4-DTB INTEL 13694y16
2 19 B 128 6 G4-DTB INTEL 13694y16
3 18 A 4 6 G4-DTB INTEL 08565928
4 18 B 4 6 G4-DTB INTEL 08565928

This should be:

PCI VxWorks Board Bridge Serial
CPU Slot Node Size Revision Type Type Number
--- ---- ---- ---- -------- ------- ------ --------
1 19 A 64 6 G4-DTB INTEL 01313x16
2 19 B 64 6 G4-DTB INTEL 01313x16
3 18 A 128 6 G4-DTB INTEL 01483y04
4 18 B 4 6 G4-DTB INTEL 01483y04

Apparently the wrong slot was getting configured by the RTM. This is also why the RTM was
replaced again as a test. Communication from the RTM to the other boards is via the back plane,
hence a spare back plane was also obtained for testing purposes.

The following line of the log is also not correct:

ISPStartup.c 142 tStart | ISP Device ID 2200 found in slot 19, node 1, interrupt 0x1b

This should be:

ISPStartup.c 120 tStart | ISP2200 found in slot 18, node 0

And:
ISPHost.c 1265 tISPHost | WARNING, Got SGI Loop ID 0x00, expected 0x40
ISPHost.c 1274 tISPHost | errno:0, 0x40040005: <<MISSING devices on Host Fibre Channel>>
ISPHost.c 1277 tISPHost | errno:0, 0x40040005: <<Expected 3 devices, Only 6 found >>
ISPHost.c 1290 tISPHost | 6 devices found: 0x78 0x79 0x7a 0x7b 0x7c 0x41
0xfbaa78 (tNetTask): miiPhyInit check cable connection

This should be:

ISPHost.c 875 tISPHost | 2 devices found: 0x40 0x41

The proper system configurations were confirmed.

The board jumpers and settings were checked.
The back plane jumpers and settings were checked.
Cable connections and fibre channel loops continuity were confirmed.
Confirmed correct RTM bootline.
Reconfigured the RTM as per FSM-CT4014.
Confirmed recon box power supplies with the specified tolerance.
Power supplies were:
24.43V (TB1-1 and 2)
3.33V (TB1-5 and 6)
5.09V (TB2-2 - TB1-5.

Diagnostics test were performed:

hwtest already failed on the first test, the fibre channel check. This indicates that there is no
communication between the RTM and S-con PC FC card.

FCDview showed only 1 device, where it should be 2.

Q-Logic test (performed by Ctrl-Q at system start-up): The system found and recognized the FC
card in the S-con PC. It also passed the loop back test from the S-con PC FC cCard to the Rec
Box MHR-FC.
This result confirms that the hardware connection between the S-con and Rec box is good.

It was decided to resort to parts replacement. Using the parts available on-site, the following parts
were replaced one at a time and returned to original when there was no change:

RTM & RTM-D

MHR-FC in slot 19
Swapped copper fiber cables between MHR-FC and RDD
FC card in S-con PC
Rdd RAID
Back plane/chassis

Based on the negative results the suspicion was raised that there could be more that one failed
part at this time. It was decided to start leaving in the new parts and look for any changes in the
boot log.

With the new back plane/chassis in the system and a new RTM, the system booted with the correct
chassis configuration and started configuring the RTM. After a few system self-resets, it booted
passed the point where it had stopped before (many times..) and continued all the way.
The system did not come up correctly, however this was because of all the self-resets, causing it to
not be in sync with the system software boot sequence. So, after a full system reboot, it came all
the way up and it was possible to do warm-up. A quick test also confirmed that scanning worked
fine.

The system software was reconfigured and the RDD data was deleted by doing a RDD config with
parity. This was done to ensure that there was a clean and fresh RDD config file. System
restoration was completed and the system was re-calibrated.

The new parts that were left in the system are: S-con PC, RTM, RTM-D, and back plane/chassis.
The S-con PC had actually already been replaced to tackle an earlier issue with a network /
motherboard problem. The other 3 parts were the cause of the system not booting.

Solution
Replacement of the back plane/cPCI chassis, the RTM and the RTM-D.

Notes
Knowing what information to expect in logs or boot logs is only possibly by keeping logs and boot
logs of normal system behaviour available for reference and comparison in the events of abnormal
system behaviour. Such normal logs, for reference purposes, anticipating abnormal system
behaviour, can be kept available in the S: partition on the system (S-con)(recommended), or in a
folder on the service PC of the site-responsible service engineer.