Need Help for IOS XRv9k CPU soft lockup issue

[terencefung]

I am running IOS XRv 9000 device in EVE-NG Professional version. When I set up 4 CPU in the device and run it, I got the following error when the device being loaded. It kept trying but was stuck in the process. Before, I set up 1 CPU in the device, this issue did not happen but the device was very slow and showed 100% CPU usage.

Starting cgroup-init
Network ieobc_br defined from /etc/init/ieobc_br_network.xml

Network local_br defined from /etc/init/local_br_network.xml


Network ieobc_br started

Network local_br started

Network xr_local_br started

mcelog start/running, process 3662
diskmon start/running, process 3724

Sat Sep 23 14:32:39 UTC 2023 (/proc/self/fd/9): Hardware profile: vrr
Sat Sep 23 14:32:39 UTC 2023 (/proc/self/fd/9): Host has 15.51GB RAM / 4 vCPUs
Sat Sep 23 14:32:39 UTC 2023 (/proc/self/fd/9): Management plane: 1024MB RAM
Sat Sep 23 14:32:39 UTC 2023 (/proc/self/fd/9): XR control plane: 7168MB RAM
Sat Sep 23 14:32:39 UTC 2023 (/proc/self/fd/9): XR packet memory: 128MB RAM
Sat Sep 23 14:32:39 UTC 2023 (/proc/self/fd/9): Centralized LC: 7168MB RAM
Sat Sep 23 14:32:39 UTC 2023 (/proc/self/fd/9): Data plane core assignment: 2-3
Sat Sep 23 14:32:39 UTC 2023 (/proc/self/fd/9): Control plane core assignment: 0-1

################################################################################
# #
# Welcome to the Cisco IOS XRv9k platform #
# #
# Please wait for Cisco IOS XR to start. #
# #
# Copyright (c) 2014-2019 by Cisco Systems, Inc. #
# #
################################################################################

Cisco IOS XR console will start on the 1st serial port
Cisco IOS XR aux console will start on the 2nd serial port
Cisco Calvados console will start on the 3rd serial port
Cisco Calvados aux will start on the 4th serial port
[ 196.247011] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 224.247009] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 256.247006] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 284.247008] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 312.247008] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 340.247010] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 368.247008] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 396.247010] BUG: soft lockup - CPU#3 stuck for 23s! [loop4:15070]
[ 436.247007] BUG: soft lockup - CPU#3 stuck for 23s! [loop4:15070]
[ 464.247010] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 492.247007] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 520.247009] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 548.247006] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 576.247017] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 616.247008] BUG: soft lockup - CPU#3 stuck for 22s! [loop4:15070]
[ 644.247009] BUG: soft lockup - CPU#3 stuck for 23s! [loop4:15070]

[rusty725]

soft lock ups usually mean that cpu is being overloaded. you need to check using this file https://www.eve-ng.net/index.php/download/#CALC how many cores and ram you need per lab. However, if you're running only 1 xrv9k then add more cores /ram.

[terencefung]

Thank you for the reply. I used the spreadsheet to calculate the resources for my project as 146432 MB RAM and 37.1 vCPU. My VM running EVE-NG Pro has 128 GB RAM and 80 vCPU. It should be enough. I checked the CPU usage and memory usage in Nodes in EVE-NG and verified that the memory usage for each node is never more than 50%. I tried setting different CPU in the IOS XRv9k devices (1, 2, 4, 8). The only way to avoid the "BUG: soft lockup - CUP#3 stuck for 22s!" problem is by setting CPU to 1. I think EVE-NG may have some kind of thread scheduling problems working with multi-CPU devices. Somehow, the CPU thread that got suspended was waiting for too long to get the next time to run. Is there a way to avoid that except for setting CPU to 1?

[rusty725]

come to our chat https://www.eve-ng.net/index.php/live-helpdesk/ and give me anydesk