Leading Network Transformation

L1 & L2 Log analysis

Halt ID
• L1/L2 crash logs indicate the reason of crash through a Halt Id

2 ALTIOSTAR PROPRIETARY & CONFIDENTIAL

Halt ID (Sample 1)
• Last moment logs with Halt ID will be available in the following logs,
 l1_spu_0_inst_0.log
 [Error Cause]: EC:02 A:00 Core:04 03308842495069300932 00126 BBUPool core 6 is STUCK!! Aborting..
 [Exit Cause]: [UTC: 3308842495072490368] L1 Abnormal exit!!, halt id(SPU_HALT_NONE)
 [Exit LocalTime: Fri Dec 6 13:34:42 2019
 l2_spu_0_inst_0.log
 [Error Cause]: EC:00 A:41 Core:02 03308842495069782664 00011 ===========================Stopping L2 on receiving L2_SUSPEND_REQUEST from
L1(halt_id:1)===========================
 [Exit Cause]: [UTC: 3308842495121215820] L2 Abnormal exit!!, halt id(SPU_HALT_NONE)
 [Exit LocalTime: Fri Dec 6 13:34:42 2019
 L1.log
 <E> [C-4] [EC-2] ed13 3308842495069300488 2019-12-06 13:34:43 366346752 366346 7846 [HEALTHMON_PROC 0025] BBUPool core 6 is STUCK!!
Aborting..
 L2.log
 C-02 <E-WLSROUTER-> EC-00 2019-12-06 13:34:43.366562816 000087ff ==Stopping L2 on receiving L2_SUSPEND_REQUEST from L1(halt_id:1)==
 vspu_init.log
 AppInRam: Start: pid=30114binName=</l2app> appname=<./l2app>AppInRam: End. 3 areas checked to be loaded in RAM
 AppInRam: Start: pid=30115binName=</l1app> appname=<./l1app>AppInRam: End. 3 areas checked to be loaded in RAM
 RMA_DEP::RMA_MSG_ID::MID_HALT_MSG_FROM_L2 msg received
 INFO :: update_spuagent_state :: SPU state update from SPUAGENT_READY_STATE to SPUAGENT_STOPPING_STATE
 SPU HALT :: SPU Agent is halting due to SPU_HALT_NONE
 ===== SPU Agent detected SIGCHLD. Parent PID: 30089, Child PID: 30114======
 SPU HALT :: SPU Agent is halting due to SPU_SIG_CHLD
 WARNING :: RMASysSpuAgtMsgHandler::receive_message :: Received SIG_CHLD from PID[30114]
 ===== SPU Agent detected SIGCHLD. Parent PID: 30089, Child PID: 30241======
 INFO :: update_spuagent_state :: SPU state update from SPUAGENT_STOPPING_STATE to SPUAGENT_STOPPED_STATE
 SPU HALT :: SPU Agent is already in halt process due to SPU_SIG_CHLD
 ===== SPU Agent detected SIGCHLD. Parent PID: 30089, Child PID: 30115======
3  SPU HALT& ::CONFIDENTIAL
ALTIOSTAR PROPRIETARY SPU Agent is already in halt process due to SPU_SIG_CHLD
 WARNING :: RMASysSpuAgtMsgHandler::receive_message :: Received SIG_CHLD from PID[30115]
Halt ID (Sample 2)
• Last moment logs with Halt ID will be available in the following logs,
 l1_spu_0_inst_0.log
 [Error Cause]: EC:02 A:00 Core:04 03308842124221623864 00171 Warning Difference is not as expected task gen:| Qid = 0, taskId = 14,taskSf = 5345,
readSf = 5340 | r_offset = 2,w_offset = 2 nTotalCell = 3 nDoneCell = 2
 [Exit Cause]: [UTC: 3308842187531297200] L1 Abnormal exit!!, halt id(SPU_HALT_NONE)
 [Exit LocalTime: Fri Dec 6 13:32:15 2019
 l2_spu_0_inst_0.log
 [Error Cause]: EC:00 A:41 Core:03 03308842124222042052 00038 ===========================Stopping L2 on receiving L2_SUSPEND_REQUEST from
L1(halt_id:1)===========================
 [Exit Cause]: [UTC: 3308842187580642829] L2 Abnormal exit!!, halt id(SPU_HALT_NONE)
 [Exit LocalTime: Fri Dec 6 13:32:16 2019
 L1.log
 <E> [C-4] [EC-2] bd95 3308842124221623104 2019-12-06 13:31:46 772214528 772214 5332 [DUMMY 0000] Warning Difference is not as
expected task gen:| Qid = 0, taskId = 14,taskSf = 5345, readSf = 5340 | r_offset = 2,w_offset = 2 nTotalCell = 3 nDoneCell = 2
 L2.log
 C-03 <E-WLSROUTER-> EC-00 2019-12-06 13:31:46.772400640 00001871 ==Stopping L2 on receiving L2_SUSPEND_REQUEST from L1(halt_id:1)==
 vspu_init.log
 AppInRam: Start: pid=29359binName=</l2app> appname=<./l2app>AppInRam: End. 3 areas checked to be loaded in RAM
 AppInRam: Start: pid=29360binName=</l1app> appname=<./l1app>AppInRam: End. 3 areas checked to be loaded in RAM
 RMA_DEP::RMA_MSG_ID::MID_HALT_MSG_FROM_L2 msg received
 INFO :: update_spuagent_state :: SPU state update from SPUAGENT_READY_STATE to SPUAGENT_STOPPING_STATE
 SPU HALT :: SPU Agent is halting due to SPU_HALT_NONE
 SPU HALT :: SPU Agent is halting due to SPU_HB_FAILURE
 ===== SPU Agent detected SIGCHLD. Parent PID: 29334, Child PID: 29448======
 INFO :: update_spuagent_state :: SPU state update from SPUAGENT_STOPPING_STATE to SPUAGENT_STOPPED_STATE
 SPU HALT :: SPU Agent is already in halt process due to SPU_HB_FAILURE
 ===== SPU Agent detected SIGCHLD. Parent PID: 29334, Child PID: 29359======
 SPU HALT :: SPU Agent is already in halt process due to SPU_HB_FAILURE
4  WARNING
ALTIOSTAR PROPRIETARY :: RMASysSpuAgtMsgHandler::receive_message :: Received SIG_CHLD from PID[29359]
& CONFIDENTIAL
Halt ID (Sample 3)
• Last moment logs with Halt ID will be available in the following logs,
 l1_spu_0_inst_0.log
 [Error Cause]: EC:00 A:00 Core:17 03325899753122117940 01665 None of the DL messages received for TTI 1121 for phyInstance 0 yet!!!
 [Exit Cause]: [UTC: 3325899753123837236] L1 Abnormal exit!!, halt id(SPU_HALT_NONE) spu restart reason(REASON_UNKNOWN)
 [Exit LocalTime: Mon Mar 9 13:50:17 2020
 l2_spu_0_inst_0.log
 [Error Cause]: EC:00 A:41 Core:15 03325899753122318628 03410 ===========================Stopping L2 on receiving L2_SUSPEND_REQUEST from
L1(halt_id:6)===========================
 [Exit Cause]: [UTC: 3325899753152073288] L2 Abnormal exit due to L1 Crash/exit!!, halt id(SPU_SIG_CHLD) spu restart reason(REASON_UNKNOWN)
 [Exit LocalTime: Mon Mar 9 13:50:17 2020
 L1.log
 <E> [C-4] [EC-2] bd95 3308842124221623104 2019-12-06 13:31:46 772214528 772214 5332 [DUMMY 0000] None of the DL messages
received for TTI 1121 for phyInstance 0 yet
 L2.log
 C-03 <E-WLSROUTER-> EC-00 2019-12-06 13:31:46.772400640 00001871 ==Stopping L2 on receiving L2_SUSPEND_REQUEST from L1(halt_id:1)==
 vspu_init.log
 AppInRam: Start: pid=3425binName=</l2app> appname=<./l2app>AppInRam: End. 3 areas checked to be loaded in RAM
 AppInRam: Start: pid=3426binName=</l1app> appname=<./l1app>AppInRam: End. 3 areas checked to be loaded in RAM
 RMA_DEP::RMA_MSG_ID::MID_HALT_MSG_FROM_L2 msg received
 INFO :: update_spuagent_state :: SPU state update from SPUAGENT_READY_STATE to SPUAGENT_STOPPING_STATE
 SPU HALT :: SPU Agent is halting due to Halt id(SPU_SIG_CHLD) Restart reason(REASON_UNKNOWN)
 Received restart request with reason as RADIO_FAILURE
 ===== SPU Agent detected SIGCHLD. Parent PID: 3411, Child PID: 3426======
 SPU HALT :: SPU Agent is already in halt process due to Halt id(SPU_SIG_CHLD) Restart reason(REASON_UNKNOWN)
 WARNING :: RMASysSpuAgtMsgHandler::receive_message :: Received SIG_CHLD from PID[3426]
 INFO :: update_spuagent_state :: SPU state update from SPUAGENT_STOPPING_STATE to SPUAGENT_STOPPED_STATE
 SPU HALT :: SPU Agent is already in halt process due to Halt id(SPU_SIG_CHLD) Restart reason(REASON_UNKNOWN)
 ===== SPU Agent detected SIGCHLD. Parent PID: 3411, Child PID: 3425======
5  SPU HALT& ::CONFIDENTIAL
ALTIOSTAR PROPRIETARY SPU Agent is already in halt process due to Halt id(SPU_SIG_CHLD) Restart reason(REASON_UNKNOWN)
 SPU Agent Weird SIGCHLD with PID:WARNING :: RMASysSpuAgtMsgHandler::receive_message :: Received SIG_CHLD from PID[3425]
Halt ID (Sample 4)
• Last moment logs with Halt ID will be available in the following logs,
 l1_spu_0_inst_0.log
 [Error Cause]: EC:01 A:00 Core:06 03871851026357106984 01726 Halting L1 on receiving PHY_SUSPEND_REQUEST from L2(halt_id:6)!!
 [Exit Cause]: [UTC: 3871851026550103888] L1 Abnormal exit due to L2 Crash/exit!!, halt id(SPU_SIG_CHLD) spu restart reason(REASON_UNKNOWN)
 [Exit LocalTime: Sun Feb 14 02:54:20 2021
 l2_spu_0_inst_0.log
 [Error Cause]: EC:00 A:57 Core:02 03871851026356985296 04302 UDP msg_send(msg-id: 50310) failed buf(113983a0) len(1492) fd(221) dport(8388)
errno(1) eNetDown(0)
 [Exit Cause]: [UTC: 3871851026411049390] L2 Abnormal exit!!, halt id(SPU_HALT_NONE) spu restart reason(REASON_UNKNOWN)
 [Exit LocalTime: Sun Feb 14 02:54:20 2021
 L1.log
 EC:01 A:00 Core:06 03871851026357106984 01726 Halting L1 on receiving PHY_SUSPEND_REQUEST from L2(halt_id:6)!!
 EC:01 A:00 Core:06 03871851026357106984 01726 Halting L1 on receiving PHY_SUSPEND_REQUEST from L2(halt_id:6)!!
 L2.log
 EC:00 A:57 Core:02 03871851026356985296 04302 UDP msg_send(msg-id: 50310) failed buf(113983a0) len(1492) fd(221) dport(8388) errno(1)
eNetDown(0)
 EC:00 A:57 Core:02 03871851026356985296 04302 UDP msg_send(msg-id: 50310) failed buf(113983a0) len(1492) fd(221) dport(8388) errno(1)
eNetDown(0)
 vspu_init.log
 AppInRam: Start: pid=6408binName=</l2app> appname=<./l2app>AppInRam: End. 3 areas checked to be loaded in RAM
 AppInRam: Start: pid=6409binName=</l1app> appname=<./l1app>AppInRam: End. 3 areas checked to be loaded in RAM
 RMA_DEP::RMA_MSG_ID::MID_HALT_MSG_FROM_L2 msg received
 INFO :: update_spuagent_state :: SPU state update from SPUAGENT_READY_STATE to SPUAGENT_STOPPING_STATE
 SPU HALT :: SPU Agent is halting due to Halt id(SPU_HALT_NONE) Restart reason(REASON_UNKNOWN)
 ===== SPU Agent detected SIGCHLD. Parent PID: 5911, Child PID: 6409======
 SPU HALT :: SPU Agent is halting due to Halt id(SPU_SIG_CHLD) Restart reason(REASON_UNKNOWN)
 WARNING :: RMASysSpuAgtMsgHandler::receive_message :: Received SIG_CHLD from PID[6409]
 ===== SPU Agent detected SIGCHLD. Parent PID: 5911, Child PID: 9502======
6  INFO :: update_spuagent_state
ALTIOSTAR PROPRIETARY & CONFIDENTIAL :: SPU state update from SPUAGENT_STOPPING_STATE to SPUAGENT_STOPPED_STATE
 SPU HALT :: SPU Agent is already in halt process due to Halt id(SPU_SIG_CHLD) Restart reason(REASON_UNKNOWN)
vDU Initialization/Add or Delete carrier (Flexran)
• SPU Initialization Procedure
 https://
intranet.rma.lan/engineering/software/SFSs/System%20Services/4G%20Initialization%20Sequence/SPU%20Init
ialization%20Procedure.pdf
• Add Carrier Procedure
 https://
intranet.rma.lan/engineering/software/SFSs/System%20Services/4G%20Initialization%20Sequence/Add%20Car
rier%20Procedure.pdf
• Delete Carrier Procedure
 https://
intranet.rma.lan/engineering/software/SFSs/System%20Services/4G%20Initialization%20Sequence/Delete%20
Carrier%20Procedure.pdf
• END To END Sector Initialization Procedure
 https
://intranet.rma.lan/engineering/software/SADs/System%20Services/SAD%20-%20System%20Initialization%20o
f%20(v)DU%20with%20RIM_RIU.docx

7 ALTIOSTAR PROPRIETARY & CONFIDENTIAL

vDU Site Fluctuation
• TTI Stretch due to DMAR error
• TTI Stretch due “i40e VF 0 still in reset. Try again” error
• TTI Stretch due to neighbour vDU reboot
• TTI Stretch due to neighbour vDU creation/deletion
• TTI Stretch due to MH/AR NIC port down/up
• TTI Stretch due to Power Issues
• TTI Stretch due to NIC PF reset
• TTI Stretch due to CPU Frequency throttling
• TTI Stretch due to none of the above errors
• http://wiki.rma.lan/wiki/doku.php?id=site_fluctuation
• http://wiki.rma.lan/wiki/doku.php?id=abrupt_vcu_vdu_reboot

8 ALTIOSTAR PROPRIETARY & CONFIDENTIAL

K8s Help
• http://wiki.rma.lan/wiki/doku.php?id=k8s_help
• http://wiki.rma.lan/wiki/doku.php?id=troubleshooting_network_function_deployments

9 ALTIOSTAR PROPRIETARY & CONFIDENTIAL

Thank You