Standard Case Report

Priority 3 - Medium
Number CS00012301526
Subject Wi-Fi 2.4G fatal error log constantly displayed
Category Issue
Requester Donghoon Lee
Date/Time Created 21-June-2023 11:40:11
Status Closed
Company Kaon Media

Broadcom Confidential 1 16-August-2023 22:16:25

 Description Notes about some mandatory fields in BCA WLAN CSP project:
"Product" field: Please select the actual chip number which the issue is on
"Customer project": The customer product/project name this CSP case is for
"Sub Item": This field has these selections: 49xx/67xx-Retail, 49xx/67xx-
Operator, 63xxx-DSL, 68xxx-PON, 33xx-CM, 72xx-STB, Bluetooth, which are
based on the SoC used in the customer product/project, independent of where
the issue is on. In the case when 49xx/67xx SoC is used, 49xx/67xx-Retail
should be selected for retail router/extender designs; 49xx/67xx-Operator
should be selected for designs targeting for operators. Bluetooth should be set
when the case is bluetooth related on 43569/570 in STBs.
"Driver Version Found In": Please enter the wifi driver version as indicated in wl
ver, e.g, 17.10.188.2501
*Software/Platform Version Found In": Please enter the SDK version, e.g.,
5.04L02P1 for DSL/PON/ethernet (63xxx/68xxx/49xx/67xx) SoC platforms,
OpenBFC-RDKM 21.2 for cable (33xx) platforms, URSR 20.1 for STB (72xx)
platforms

[Problem Description]
If the initial state of the sample or factory reset (restoredefault) is performed,

the error log as shown in the figure will continue to be displayed.

(wl1: fatal error, reinitializing, total count of reinit's)

[Expected Behavior]
The log should not be output.

[Reproduction Steps]

booting complete after restoredefault

[Platform Description]
Please always provide all the SoC and wifi chip numbers on the product:
Soc chip number: BCM6756
2G Wifi chip number: BCM6756
5G (dualband) or 5GL (triband) Wifi chip number: BCM6715X
5GH (triband) Wifi chip number: N/A
6G Wifi chip number: BCM6715X

[SW Description]
Please always provide both the SDK and wifi SW versions
SDK version: bdk_5.04l.04p1
Wifi version: ? (wl -i wlX ver and wl -i wlX revinfo output, where wlX is the
interface name of the problematic wifi)

WiFi driver : Impl87

# wl -i wl0 ver
Broadcom BCA: 17.10 RC188.75
wl0: Jun 21 2023 09:48:58 version 17.10.188.75 (r811021)-PvB0.W7 FWID 01-
84ce61a1
# wl -i wl1 ver
Broadcom BCA: 17.10 RC188.75
wl1: Jun 21 2023 09:52:48 version 17.10.188.75 (r811021)-PvB0.W7
Broadcom Confidential 2 16-August-2023 22:16:25
 # wl -i wl2 ver
Broadcom BCA: 17.10 RC188.75
wl2: Jun 21 2023 09:52:48 version 17.10.188.75 (r811021)-PvB0.W7

# wl -i wl1 status
SSID: "BrcmAP1"
Mode: Managed RSSI: 0 dBm SNR: 0 dB noise: -78 dBm Channel: 6u
BSSID: 00:90:4C:40:41:11 Capability: ESS ShortSlot RRM
Beacon Interval: 100 msecs
Supported Rates: [ 1(b) 2(b) 5.5(b) 6 9 11(b) 12 18 24 36 48 54 ]
HE Capable:
Chanspec: 2.4GHz channel 4 40MHz (0x1904)
Primary channel: 6
HT Capabilities: 40MHz SGI20 SGI40
Supported HT MCS : 0-15 32
Supported VHT MCS:
NSS1 Tx: 0-11 Rx: 0-11
NSS2 Tx: 0-11 Rx: 0-11
Supported HE MCS:
20/40/80 MHz:
NSS1 Tx: 0-11 Rx: 0-11
NSS2 Tx: 0-11 Rx: 0-11
QBSS Channel Utilization: 0xdb (85 %)

[Other Supporting Logs]if available, attach the following to the CSP

... WlGetDriverCfg.sh dump; particular if there is a good case and bad case,
provide the WlGetDriverCfg.sh dump from both good and bad cases ...
... WlGetDriverStats.sh for performance/throughput related issues ...
... in case of wifi dongle crash, attach the crash dump file (in the form of debug-
YYYY_MM_DD_HH_MM_SS.tgz in /tmp/confmtd/crash_logs or in the backup
directory specified by the "crash_log_backup_dir" NVRAM param) and the
rtecdc.exe matching the wifi FW rtecdc.bin being used ...

P.S : SW produced by enabling BUILD_HND_MFG (WLTEST) does not have

this problem.

CC List
External CC b230203@kaonmedia.com,jeremiah.bae@kaonmedia.com

Additional Fields
Customer Project AR1840
Software/Platform Version bdk_5.04l.04p1
Found In
Sub Item 49xx/67xx-Operator
Driver Version Found In 17.10 RC188.75

Comments
User Description Status Date
21-
Dongho This is the nvram file being used for that model Assign June-
2023
on Lee ed 11:42:
50
Broadcom Confidential 3 16-August-2023 22:16:25
 Hi Donghoon,

Please attach the full logs from bootup. 21-

Kevin Assign June-
This issue is observed at 2.4G with prod build only but not shown with mfg build ? 2023
Yeo ed 15:19:
51
Thanks

kevin.
Hi Kevin

Thank you for your quick response.

21-
Dongho I am attaching the bootlog. Assign June-
2023
on Lee ed 15:54:
Also, the corresponding error log is not displayed in MFG FW. 21

thanks

Broadcom Confidential 4 16-August-2023 22:16:25

 Hello Donghoon,

Unlucky, we could not decide the reason why it has fatal error with this limited
information.

At least, we need to know the reason for fatal. It seems like WL_ERROR log does not
display now.

Could you provide two things ?

1) Kernel Log

2) Replace WL_ERROR -> WL_PRINT like below example.

wlc_fatal_error(wlc_info_t *wlc)
{
uint32 gptime = 0;
#ifdef STA
int idx;
wlc_bsscfg_t *bsscfg;
bool PMenabled[WLC_MAXBSSCFG] = {0};
#endif
#ifndef BCMNODOWN
if (wlc->down_override) {
WL_PRINT(("wl%d: %s don't reinit if attempted to bring driver down\n", 22-
Henry wlc->pub->unit, __FUNCTION__)); Assign June-
2023
Ma return; ed 09:08:
} 52
#endif /* BCMNODOWN */
if ((wlc->hw->need_reinit != WL_REINIT_RC_USER_FORCED) &&
(g_assert_type == 3)) {
WL_PRINT(("wl%d: %s don't reinit if assert_type is 3 unless forced by user\n",
wlc->pub->unit, __FUNCTION__));
return;
}
WLCNTINCR(wlc->pub->_cnt->reinit);
WL_ERROR(("wl%d: fatal error, reinitializing\n", wlc->pub->unit));
WL_EAP_TRC_ERROR(("wl%d: fatal error, reinitializing\n", wlc->pub->unit));
WL_PRINT(("wl%d: fatal error, reinitializing, total count of reinit's[%d]\n",
wlc->pub->unit, wlc->pub->_cnt->reinit));
wlc->pub->reinitrsn->rsn[REINITRSNIDX(wlc->hw->need_reinit)]++;
WL_PRINT(("wl%d: 802.11 reinit reason[%d], count[%d]\n",
wlc->pub->unit, wlc->hw->need_reinit,
wlc->pub->reinitrsn->rsn[REINITRSNIDX(wlc->hw->need_reinit)]));
#if defined(DONGLEBUILD) && (defined(BCMDBG) || defined(WL_MACDBG))
if ((wlc->hw->need_reinit != WL_REINIT_RC_NONE) &&
(wlc->hw->need_reinit != WL_REINIT_RC_USER_FORCED))
{
/* this routine should never return to the caller for the dongle environment */
OSL_SYS_HALT();
}
#endif /* defined(DONGLEBUILD) && (defined(BCMDBG) || defined(WL_MACDBG)) */
if (DEVICEREMOVED(wlc)) {
/* Powercycle WiFi module in case of OSX */
if (wlc->hw->need_reinit == WL_REINIT_RC_NONE) {
wlc->hw->need_reinit = WL_REINIT_RC_DEVICE_REMOVED;
}
Broadcom Confidential 5 16-August-2023 22:16:25
 }
wlc->state = WLC_STATE_GOING_UP;
.......

We will try to reproduce on BRCM reference board then let you know the result if we can
reproduce it or not.

In my review, the steps are like below. If I'm wrong, please correct me.

6756 (SOC) + 504L04P1 + 17.10.188.75

Reboot -> RestoreDefault -> Check Fatal

Thanks,

Henry
Hello Donghoon,

This rb194056 has forced fatal error print.

22-
Please merge this patch and try to reproduce then let us get logs.
Henry Assign June-
2023
Ma ed
If WL_ERROR does not display like before, please change WL_ERROR to WL_PRINT. 11:02:
37

Thanks,

Henry

Broadcom Confidential 6 16-August-2023 22:16:25

 The replay scenario is the same as you said.

6756 (SOC) + 504L04P1 + 17.10.188.75

Reboot -> RestoreDefault -> Check Fatal

22-
Also, I attach the kernel log (dmesg_log.txt) you mentioned below
Dongho Assign June-
2023
on Lee ed
and the boot_log replaced with WL_PRINT (modify_code__boot_log.txt). 11:59:
48

in boot_log. "802.11 reinit reason[16]," is being output, I think it probably means

"WL_REINIT_RC_AP_BEACON = 16". (Reinit_resaon_code.txt)

I'll also apply the rb194056.patch you gave me, and I'll give you the results soon.

Hi Donghoon,

Thanks for your confirmation to check reason 16 : WL_REINIT_RC_AP_BEACON.

It occurs when there is not chance to send beacon. To make sure that we want to
check "chanim_stats" for this condition.

Unlucky, we don't reproduce it on my environment. What is frequency to reproduce on

your setup ?

Test#1.
22-
Henry When it is going to reinit repeatedly, please get chanim_stats about 20 seconds. Assign June-
2023
Ma ed 13:06:
(ex. while true;do wl chanim_stats;sleep 1;done) 14

Test#2.

If you can move setup into shieldroom, please check it again.

At this time, chanim_stats needs together for comparison with Test#1 although it doesn't
occur issue.

Thanks,

Henry
Additionally, if you can print counters for txbcnfrm, it will be grate to make sure that 22-
Henry issue. Assign June-
2023
Ma ed 13:10:
(ex. while true;do wl counters | grep txbcnfrm;sleep 1;done) 27

Broadcom Confidential 7 16-August-2023 22:16:25

 Hi Henry

First of all, attach logs for chanim_stats and txbcnfrm in normal environment (#Test1). 22-
Dongho Assign June-
2023
on Lee Information from the measurements in the shield room will be delivered soon. ed 14:06:
12

thanks

Hi Henry

The data measured in the shield room is attached.

22-
Dongho The peculiarity is that when measured in the shield room, there is no fatal error log... Assign June-
2023
on Lee ed 14:58:
Please reply if you need additional data. 27

thanks

Hi Donghoon,

The interface is wl1 but we got logs about "while true;do wl chanim_stats;sleep 1;done"
(wl0)

So, it has 50%-60% for txop in my thought. Could you add "-i wl1" for chanim_stats and
txbcnfrm ?

(Ex; while true;do wl -i wl1 chanim_stats; wl -i wl1 counters | grep txbcnfrm; sleep
1;done)

23-
Henry Assign June-
2023
Ma Anyway, let me explain the reason why it has reinitialization. ed 10:10:
01
If there are heavy noisy environment, we don't have a chance to send beacon.

It has timeout 4 seconds default and it has fatal error if it has timed out.

To relieve this behavior, we could extend timeout value from below. (ex. 4->10-20)

#define WLC_AP_TXBCN_TIMEOUT 4 /**< # seconds: interval to detect beacon loss */

Thanks,

Henry

Broadcom Confidential 8 16-August-2023 22:16:25

 Hi Henry

thanks for your answer

23-
Dongho I attach wl1 data of normal environment and shielded room environment. Assign June-
2023
on Lee ed 12:14:
I'll try adjusting the WLC_AP_TXBCN_TIMEOUT value. 14

Thanks

Hi Donghoon,

As we expected, there are extremely lower txop (about 1%) and there are not chance to
send beacon.
23-
Henry - txbcnfrm stuck at 39 count 15 times from log Assign June-
2023
Ma ed 12:28:
- txop 1% 44

Thanks,

Henry
Hi. Henry

yeah..then...I just would like to ask...

23-
Is just adjusting the WLC_AP_TXBCN_TIMEOUT value the only way to solve this
Dongho Assign June-
2023
on Lee problem? ed 13:07:
52

thanks

Hi Donghoon,

Currently, we don't have much thing to do like extremely heavy noisy environment.

If you don't want to reinitialize, you can disable this as "0". 23-
Henry Assign June-
(WLC_AP_TXBCN_TIMEOUT 0) 2023
Ma ed 13:55:
56
However, it depends on your environment.

Thanks,

Henry

Broadcom Confidential 9 16-August-2023 22:16:25

 Case Modified. Status changed from Open-Assigned to Awaiting Confirmation-Awaiting
Confirmation 23-
Awaitin June-
Henry g 2023
Ma Notes: Confir 13:56:
mation 13
.

Hi Donghoon,
29-
Any update on this? Please let us know if you have any question further. Awaitin June-
Kevin g 2023
Yeo Confir
Thanks 07:08:
mation 50
Kevin.
hi kevin henry

During various tests related to the 2.4ghz reinit problem of the AR1840 model,
it was confirmed that the BDK_5.04L.02p1 did not have this problem in the same
environment.

I have attached the log, can you please check it??

29-
-reference Awaitin June-
Dongho g 2023
on Lee BDK_5.04L.02p1 : wifi driver 17.10.188.6401 Confir 12:33:
mation 11
BDK_5.04L.04p1 : wifi driver 17.10.188.75

thanks

Hi Donghoon,

Awaitin 29-
Do you have experienced this situation at same environment and same time ? June-
Henry g 2023
Ma Confir 12:48:
mation 05
Thanks,

Henry
Additionally,
29-
It should be same channel. Awaitin June-
Henry g 2023
Ma Confir
We could see that they are different channel (11u vs 1) 12:49:
mation 59

Broadcom Confidential 10 16-August-2023 22:16:25

 Hi henry

I'm new to bdk this time, so I don't know... I'm sorry... 29-
Awaitin June-
Dongho However, this time I made another model (BCM6750) using BDK_5.04L.04p1, g 2023
on Lee and it outputs the same fatal error log as the AR1840 model. Confir 12:55:
mation 51

thanks...

Hi Donghoon,

Let me explain a little bit more.

AR1840_BDK_5.04L.02p1.txt (no reinit) => txop : 1%, channel : 1, txbcnfrm is

increasing

AR1840_BDK_5.04L.04p1.txt (reinit) => txop : 0%-1%, channel 11u, txbcnfrm is stuck

(20 times then reinit)

29-
Awaitin June-
Henry g
For comparison, we want to verify on same channel. 2023
Ma Confir 13:25:
mation 17
Please change channel #1. (ex. wl -i wlx csa 1 10 1) : 2G

Then, please run the test at same time/location.

Thanks,

Henry

p.s. If you have questions, please contact me (010-2086-1868)

Broadcom Confidential 11 16-August-2023 22:16:25

 hi henry

I tested with the same channel, bandwidth, same time and same environment.

Could you please check the attached file?

I analyzed the text file and got the following result. 29-
Awaitin June-
Dongho g 2023
on Lee Confir 13:32:
mation 30
AR1840_BDK_5.04L.02p1_same_ch.txt (no reinit) => txop : 0%-1%, channel : 11u,
txbcnfrm is stuck

AR1840_BDK_5.04L.04p1_same_ch.txt (reinit) => txop : 0%-1%, channel 11u, txbcnfrm

is stuck

thanks

Hi Henry

I have one more question.

(Sorry, the question goes against the title)

MAX Throughput test was performed using AR1840 HW + BDK_5.04L.04p1(WiFi:

17.10.188.75).
Due to the board temperature exceeding 120°C, the board attempted to reboot.

However,
when 120°C is recognized with the combination of AR1840 HW + 29-
Awaitin June-
Dongho BDK_5.04L.02p1(WiFi: 17.10.188.6401), g 2023
on Lee I saw a scenario where the temperature is lowered by lowering the TP instead of Confir 16:03:
rebooting. mation 55

I am attaching the log to this.(This was tested by changing only SW in the same
sample.)

I think BDK_5.04L.04p1 should behave like BDK_5.04L.02p1.

Is there any way to do this?

thanks

Hi Donghoo,

For the new question, Please create new CSP due to our CSP support policy since we 29-
Awaitin June-
Kevin may need another engineer to support this new issue. g 2023
Yeo Confir 16:19:
Thanks mation 09

Kevin.

Broadcom Confidential 12 16-August-2023 22:16:25

 Hi Donghoon,

As you see, there is only 1 incensement txbcnfrm from

"AR1840_BDK_5.04L.02p1_same_ch.txt".

So, it does not have reinitialization. Unlucky, we have stuck txbcnfrm

over 33 seconds from "AR1840_BDK_5.04L.04p1_same_ch.txt".
29-
Awaitin
We cannot tell that it has SW deviation between BDK_5.04L.04p1(WiFi: June-
Henry g 2023
Ma 17.10.188.75) and BDK_5.04L.02p1(WiFi: 17.10.188.6401). Confir 16:33:
mation 03
(As you know, if it was default 4 seconds, we already meet the reinitialization at
BDK_5.04L.02p1 either. Because we confirm that "txbcnfrm 29361" was repeated
over 20 times.

Thanks,

Herny

Hi Donghoon,

Fyi, Regarding, please have a look at the "CPE-AN3601.pdf" in SDK release package.
(CPE Power Management) before creating new CSP.

The bcm_thermal Linux driver utilizes the Linux thermal framework to track both the chip
temperature and the ARM CPU
temperature; if the maximum of these two temperatures goes beyond 115°C, the bcm_thermal
Linux driver will take the one
CPU offline (CPU3 for BCM4908, BCM63158, BCM47622, and BCM6755, CPU2 for CM63178
and BCM6750). If the
temperature goes beyond 125°C, the bcm_thermal Linux driver will take all the possible CPUs
except CPU0 offline. When
the temperature returns below 110°C (accounting 5°C for hysteresis), the bcm_thermal Linux
driver will return all the offline 30-
Awaitin
CPUs back online. For test and other experimental purposes, the Linux shell command "rmmod - June-
Kevin g
w bcm_thermal" can be 2023
Yeo Confir
used to disable this feature by unloading the bcm_thermal Linux driver module from the running 10:01:
mation
kernel. 38

On BCM63178, BCM6750, BCM47622, and BCM6755, the WLAN driver switches the Wi-Fi radio
off (reducing 50% of
chains) when the Wi-Fi core temperature goes beyond 110°C

For reinit problem, As Henry mentioned, this is due to test environment(extreme

noisy )and cannot see any significant difference between SDK versions.

So, Please close this ticket or call me if you need to clarify more. (010-2093-
6178)

Thanks

Kevin.

Broadcom Confidential 13 16-August-2023 22:16:25

 Case Modified. Status changed from Awaiting Confirmation-Awaiting Confirmation to
Resolved-Resolved

Notes:

Hi Donghoon, 13-
Kevin Resolv July-
2023
Yeo Set to resolved since there is no activity for a while. ed 10:43:
37
Please close if you don't have any further question.

Thanks

Kevin.

Case Modified. Status changed from Resolved-Resolved to Closed-Closed

14-
July-
Henry
Notes: Closed 2023
Ma 14:35:
. 48

Attachments
Sl.No Attachment Name Public Date
1 Magyar_Telekom_temp_BDK_5.04L.04p1_Reboot_.txt True 29-June-2023
16:03:55
2 Magyar_Telekom_temp_BDK_5.04L.02p1_PASS.txt True 29-June-2023
16:03:55
3 AR1840_BDK_5.04L.04p1_same_ch.txt True 29-June-2023
13:32:30
4 AR1840_BDK_5.04L.02p1_same_ch.txt True 29-June-2023
13:32:30
5 AR1840_BDK_5.04L.04p1.txt True 29-June-2023
12:33:11
6 AR1840_BDK_5.04L.02p1.txt True 29-June-2023
12:33:11
7 [normal place] wl1_data.txt True 23-June-2023
12:14:14
8 [shieldroom]_wl1_data.txt True 23-June-2023
12:14:14
9 on_shield_room_data.txt True 22-June-2023
14:58:27
10 test1_normal_place__wl counters.txt True 22-June-2023
14:06:12
11 test1_normal_place_wl chanim_stats.txt True 22-June-2023
14:06:12
12 Reinit_reason_code.txt True 22-June-2023
11:59:48
13 dmesg_log.txt True 22-June-2023
11:59:48
14 Modfiy code.PNG True 22-June-2023
11:59:48
15 modify_code__boot_log.txt True 22-June-2023
11:59:48
16 rb194056.patch True 22-June-2023
11:02:37
17 error_sample_bootlog.txt True 21-June-2023
15:54:21
Broadcom Confidential 14 16-August-2023 22:16:25
 18 AR1840_M_1.nvm True 21-June-2023
11:42:50

Broadcom Confidential 15 16-August-2023 22:16:25