Image starting oddities after upgrade to 4.0.1-8-PRO

[agrajag]

Hi all,

I have been upgrading from 3.0.1-21 through to 4.0.1-8 PRO versions and had no problems until 4.0.1-3

I am now unable to start any image from existing labs first time. I am getting "Failed to create network (11)" messages in the HTML console. This is happening IOL, qemu and dockers.

If I keep on starting the image it will eventually start. Stopping the image then restart will result in same error message. I have today tried uipgrading to the latest 4.0.1-8, but that has made no difference.
Also tried a new lab, but this also has the same problems.

Here are the details on the system and very brief investigations that I have done so far

Hardware
2 x CPU 20 cores (40 vCPU)
model name : Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
64Gb RAM
1Tb NVME mounted on /opt
1Tb root disk
2Tb spare disk
Single physical NIC

EVE-NG 4.0.1-8-PRO Running on Bare Metal

Linux eve-ng 5.3.0-1039-gke #42-Ubuntu SMP Thu Oct 22 00:23:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@eve-ng:/opt/unetlab/addons/iol/bin# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and- ... acy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

VT-X enabled



From unl_wrapper.txt

Feb 08 10:17:09 Feb 08 10:17:09 Online Check state: Valid
Feb 08 10:17:09 Feb 08 10:17:09 ERROR: sat=0
Feb 08 10:17:09 ERROR: echo 65535 > /sys/class/net/vnet00200000059/bridge/group_fwd_mask 2>&1 --- Failed to set group_fwd_mask on bridge (80028).
Feb 08 10:17:09 net.ipv6.conf.vnet00200000059.disable_ipv6 = 1
Feb 08 10:17:09 Feb 08 10:17:09 ERROR: Failed to set group_fwd_mask on bridge (80028).
Feb 08 10:17:09 Feb 08 10:17:09 ERROR: Failed to create network (11).


From /var/log/auth.log

Feb 8 10:17:06 eve-ng sudo: www-data : TTY=unknown ; PWD=/opt/unetlab/html ; USER=root ; COMMAND=/opt/unetlab/wrappers/unl_wrapper -a clearstopped
Feb 8 10:17:06 eve-ng sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 8 10:17:07 eve-ng sudo: pam_unix(sudo:session): session closed for user root
Feb 8 10:17:08 eve-ng sudo: www-data : TTY=unknown ; PWD=/opt/unetlab/html ; USER=root ; COMMAND=/opt/unetlab/wrappers/unl_wrapper -a start -T 0 -U admin -D 69 -F /opt/unetlab/labs/KNOX/Knoxlab_UD.unl
Feb 8 10:17:08 eve-ng sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Feb 8 10:17:09 eve-ng sudo: pam_unix(sudo:session): session closed for user root




And here is the log entry from a finally running node

Feb 8 10:20:53 eve-ng sudo: root : TTY=unknown ; PWD=/opt/unetlab/tmp/0/a781569b-c681-483b-baee-35606b9124a2/14 ; USER=root ; COMMAND=/usr/bin/systemd-run -G --no-block --property=WorkingDirectory=/opt/unetlab/tmp/0/a781569b-c681-483b-baee-35606b9124a2/14 --service-type=simple --uid=0 --gid=32768 --unit=eveng_0@a781569b-c681-483b-baee-35606b9124a2@14 /opt/unetlab/wrappers/iol_wrapper -C 39319 -L 2 -T 0 -D 14 -t R14 -F /opt/unetlab/addons/iol/bin/i86bi_LinuxL3-AdvEnterpriseK9-M2_157_3_May_2018.bin -d 0 -e 1 -s 0 -- -n 1024 -q -m 1024


Only difference is the working directory. Failure starts in /opt/unetlab/html, but the working one starts in the proper lab directory /opt/unetlab/tmp/0/......../<IMAGE>


Regards,

Andy

[Uldis (UD)]

You forget standard advice clear your browser cookies and CACHE !!!

[agrajag]

Hi UD,

Also speaking on live chat about this.

Tried with several browsers, all with same result

Tried wiping whole lab and one IOL node would start first time, but others will not. Similar problems with QEMU and dockers

Sorry about the hassle. let me know if can get you more information.

Cheers,

Andy

[Uldis (UD)]

Come to eve live chat mate
here I will not be able to help you
https://www.eve-ng.net/index.php/live-helpdesk/
it is fastest and best way to reach me ot other experts
use you gmail account to join in chat
or create new account using own email

https://www.eve-ng.net/index.php/live-helpdesk/

It is 100% sortable..
just some thing is forgotten to do

i hope after upgrade you did reboot eve?

[agrajag]

Hi UD,

Yep, rebooted as soon as it was upgraded.

Just checked again and any node that has no network connection will start immediately. Connect to another node or network and the error "Failed to create network(11)" appears.

Can go over to live helpdesk, but whats the best time?
UK office hours best for me, but can arrange time out of hours if necessary.

Thanks for taking the time to help.

Cheers,

Andy

[Uldis (UD)]

Andy please in live chat ok

[agrajag]

Thanks to Alain and UD for the quick diagnostics over on the live chat.

Problem was incorrect kernel version
Wrong - Linux eve-ng 5.3.0-1039-gke
Right - Linux eve-ng 4.20.17.2-eve-ng-uksm-wg+

Working fantastic now