Problem starting nodes

Before posting something, READ the changelog, WATCH the videos, howto and provide following:
Your install is: Bare metal, ESXi, what CPU model, RAM, HD, what EVE version you have, output of the uname -a and any other info that might help us faster.

Moderator: mike

bmwl
Posts: 10
Joined: Sat Feb 27, 2021 4:52 am

Problem starting nodes

Post by bmwl » Fri Dec 30, 2022 7:04 pm

Hello, we have a few eve-ng pro lab boxes at our company, and one of them is suddenly unable to start any qemu VMs
I have verified the normal things: license is valid, virtualization extensions are still on in the BIOS, all mounted volumes have lots of space available, fixpermissions script has been run and shows no errors, eve-ng-pro package version is 5.0.1-20, apt update/apt upgrade have completed successfully and there are no waiting upgrades and the machine has been rebooted multiple times. There are no satellite nodes.

Here is the log snippet from trying to start a Windows VM, but Linux VMs have the same behaviour:

Code: Select all

Dec 30 18:49:53 Dec 30 18:49:53 Online Check state: Valid
Dec 30 18:49:53 Dec 30 18:49:53 ERROR: sat=0
Dec 30 18:49:53 Dec 30 18:49:53 LOG: will start node 2 0 300 1 58171
Dec 30 18:49:54 INFO: tunctl -u 33794 -g root -t vun001000000200 2>&1
Dec 30 18:49:54 INFO: sysctl -w net.ipv6.conf.vun001000000200.disable_ipv6=1
Dec 30 18:49:54 INFO:  vun001000000200 -> 0,0,0,0
Dec 30 18:49:54 LOG: exec : brctl addif pnet1 vun001000000200 2>&1
sh: 1: ovs-vsctl: not found
Dec 30 18:49:54 ERROR: QEMU Arch is not set (80015).
Dec 30 18:49:54 cmd = mountpoint -q /opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2/jail || mount -B -o ro /opt/unetlab/jail /opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2/jail
Dec 30 18:49:54 cmd = for i in /opt/qemu-*; do ( mountpoint -q /opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2/$i || mount -B -o ro $i /opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2/$i ) ; done
Dec 30 18:49:54 cmd =  mountpoint -q /opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2/opt/unetlab/addons/qemu || mount -B -o ro /opt/unetlab/addons/qemu /opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2/opt/unetlab/addons/qemu
Dec 30 18:49:54 INFO: CWD is /opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2
Dec 30 18:49:54 INFO: starting /opt/unetlab/wrappers/qemu_wrapper -C 58171 -T 0 -D 2 -t "LVThinManagerTest" -F /opt/qemu-4.1.0/bin/qemu-system-x86_64 -d 0 -x -- -device e1000,netdev=net0,mac=50:0c:00:02:00:00 -netdev tap,id=net0,ifname=vun001000000200,script=no -smp 4,sockets=4,cores=1,threads=1 -m 8192 -name LVThinManagerTest -uuid 2d753488-ea1e-46e5-9227-d0340cc1bf94 -qmp unix:./qmp-sock,server,nowait  -monitor unix:./mon-sock,server,nowait  -monitor unix:./mon2-sock,server,nowait  -vnc :52271 -cdrom /opt/unetlab/addons/qemu/winserver-2019gui/cdrom.iso -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -cpu host,+fsgsbase -vga std -usbdevice tablet -boot order=dc -drive file=/opt/qemu/share/qemu/virtio-win-drivers.img,index=1,if=floppy,readonly
Dec 30 18:49:54 INFO: starting sudo systemd-run -G  --no-block   --property=ExecStopPost=/opt/unetlab/go/eve-clean --property=Environment=WDIR="/opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2" --property=Environment=TYPE="qemu" --property=WorkingDirectory=/opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2 --service-type=simple --uid=0 --gid=32768  --unit=eveng_0@fa62387e-eb99-4672-b902-c3c43a507538@2 /opt/unetlab/wrappers/qemu_wrapper -C 58171 -T 0 -D 2 -t "LVThinManagerTest" -F /opt/qemu-4.1.0/bin/qemu-system-x86_64 -d 0 -x -- -device e1000,netdev=net0,mac=50:0c:00:02:00:00 -netdev tap,id=net0,ifname=vun001000000200,script=no -smp 4,sockets=4,cores=1,threads=1 -m 8192 -name LVThinManagerTest -uuid 2d753488-ea1e-46e5-9227-d0340cc1bf94 -qmp unix:./qmp-sock,server,nowait  -monitor unix:./mon-sock,server,nowait  -monitor unix:./mon2-sock,server,nowait  -vnc :52271 -cdrom /opt/unetlab/addons/qemu/winserver-2019gui/cdrom.iso -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -cpu host,+fsgsbase -vga std -usbdevice tablet -boot order=dc -drive file=/opt/qemu/share/qemu/virtio-win-drivers.img,index=1,if=floppy,readonly
Running as unit: eveng_0@fa62387e-eb99-4672-b902-c3c43a507538@2.service
Dec 30 18:49:54 INFO: starting systemctl show --property MainPID --value eveng_0@fa62387e-eb99-4672-b902-c3c43a507538@2
Dec 30 18:49:54 INFO: started process is 62354
PHP Warning:  unlink(/opt/unetlab/tmp/0/fa62387e-eb99-4672-b902-c3c43a507538/2/l1down_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1174
Dec 30 18:49:54 INFO:  vun001000000200 -> 0,0,0,0
Dec 30 18:49:54 Dec 30 18:49:54 ERROR: node 2
Please help!

-Brian

Uldis (UD)
Posts: 5068
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: Problem starting nodes

Post by Uldis (UD) » Fri Dec 30, 2022 8:32 pm

Show output of you EVE

Code: Select all

eve-info

bmwl
Posts: 10
Joined: Sat Feb 27, 2021 4:52 am

Re: Problem starting nodes

Post by bmwl » Sun Jan 01, 2023 10:15 am

Code: Select all

Sun 01 Jan 2023 10:14:19 AM UTC
---------------Packages Installed----------------
ii eve-ng-chrome 5-eve-ng
ii eve-ng-dind 5-eve-ng
ii eve-ng-dockers 5.0-eve-ng
ii eve-ng-dynamips 5.0.1-1
ii eve-ng-kali-large 1.6-eve-ng
ii eve-ng-ostinato 5.1.1-eve-ng
ii eve-ng-pro 5.0.1-20
ii eve-ng-pro-guacamole 5.0.1-1
ii eve-ng-qemu 5.0.1-1
ii eve-ng-schema 5.0.1-1
ii eve-ng-vpcs 5.1-eve-ng
ii linux-image-5.17.8-eve-ng-uksm-wg+ 1

---------------Hostname--------------------------
   Static hostname: wlnetlab
  Operating System: Ubuntu 20.04.5 LTS
            Kernel: Linux 5.17.8-eve-ng-uksm-wg+
      Architecture: x86-64
---------------Disk Usage------------------------
Filesystem                         Size  Used Avail Use% Mounted on
udev                               189G     0  189G   0% /dev
tmpfs                               38G  2.2M   38G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  145G   57G   82G  41% /
tmpfs                              189G     0  189G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              189G     0  189G   0% /sys/fs/cgroup
/dev/loop0                          64M   64M     0 100% /snap/core20/1738
/dev/loop4                          50M   50M     0 100% /snap/snapd/17883
/dev/loop2                          92M   92M     0 100% /snap/lxd/23991
/dev/loop1                          64M   64M     0 100% /snap/core20/1778
/dev/loop3                          92M   92M     0 100% /snap/lxd/24061
/dev/sda2                          1.5G  253M  1.2G  19% /boot
/dev/sdb1                          3.3T  893G  2.2T  29% /opt
tmpfs                               38G     0   38G   0% /run/user/0

---------------CPU Info--------------------------
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          32
On-line CPU(s) list:             0-31
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           45
Model name:                      Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
Stepping:                        7
CPU MHz:                         1200.000
CPU max MHz:                     3100.0000
CPU min MHz:                     1200.0000
BogoMIPS:                        4799.61
Virtualization:                  VT-x
L1d cache:                       512 KiB
L1i cache:                       512 KiB
L2 cache:                        4 MiB
L3 cache:                        40 MiB
NUMA node0 CPU(s):               0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
NUMA node1 CPU(s):               1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX vulnerable
Vulnerability Mds:               Vulnerable; SMT vulnerable
Vulnerability Meltdown:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and use
                                 rcopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtr
                                 r pge mca cmov pat pse36 clflush dts acpi mmx f
                                 xsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rd
                                 tscp lm constant_tsc arch_perfmon pebs bts rep_
                                 good nopl xtopology nonstop_tsc cpuid aperfmper
                                 f pni pclmulqdq dtes64 monitor ds_cpl vmx smx e
                                 st tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse
                                 4_2 x2apic popcnt tsc_deadline_timer aes xsave
                                 avx lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnm
                                 i flexpriority ept vpid xsaveopt dtherm ida ara
                                 t pln pts md_clear flush_l1d

---------------Memory Info-----------------------
              total        used        free      shared  buff/cache   available
Mem:          377Gi       2.3Gi       361Gi       9.0Mi        14Gi       373Gi
Swap:         8.0Gi          0B       8.0Gi

---------------Nic Info--------------------------
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet0 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet1 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet2 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet3 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
6: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet4 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet5 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
8: eth6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet6 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
9: eth7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet7 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
10: eth8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet8 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
11: eth9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet9 state                                                                                                                                                              UP mode DEFAULT group default qlen 1000
12: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet10 sta                                                                                                                                                             te UP mode DEFAULT group default qlen 1000
13: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet11 sta                                                                                                                                                             te UP mode DEFAULT group default qlen 1000

---------------IP Info---------------------------
●   State: routable
           172.29.129.254 on nat0
           172.17.0.1 on docker0
           172.29.130.254 on wg0

Dec 30 18:47:18 wlnetlab systemd-networkd[1236]: rtnl: received neighbor for link '30' we don't know about, ignoring.
Dec 30 18:47:18 wlnetlab systemd-networkd[1236]: vun001000000200: Link DOWN
Dec 30 18:49:54 wlnetlab systemd-networkd[1236]: vun001000000200: Link UP
Dec 30 18:49:58 wlnetlab systemd-networkd[1236]: rtnl: received neighbor for link '32' we don't know about, ignoring.
Dec 30 18:49:58 wlnetlab systemd-networkd[1236]: rtnl: received neighbor for link '32' we don't know about, ignoring.
Dec 30 18:49:58 wlnetlab systemd-networkd[1236]: vun001000000200: Link DOWN
Dec 30 18:50:47 wlnetlab systemd-networkd[1236]: vun001000000200: Link UP
Dec 30 18:50:54 wlnetlab systemd-networkd[1236]: rtnl: received neighbor for link '34' we don't know about, ignoring.
Dec 30 18:50:54 wlnetlab systemd-networkd[1236]: rtnl: received neighbor for link '34' we don't know about, ignoring.
Dec 30 18:50:54 wlnetlab systemd-networkd[1236]: vun001000000200: Link DOWN

---------------Bridge Info-----------------------
pnet0           8000.001018a43330       no              eth0
pnet1           8000.001018a43332       no              eth1
pnet10          8000.d4ae52aaf34b       no              eth10
pnet11          8000.d4ae52aaf34c       no              eth11
pnet2           8000.001018a42b10       no              eth2
pnet3           8000.001018a42b12       no              eth3
pnet4           8000.d4ae52aaf349       no              eth4
pnet5           8000.001018a42b4c       no              eth5
pnet6           8000.001018a42b4e       no              eth6
pnet7           8000.001018a42b68       no              eth7
pnet8           8000.001018a42b6a       no              eth8
pnet9           8000.d4ae52aaf34a       no              eth9

---------------H/W Accel-------------------------
INFO: /dev/kvm exists
KVM acceleration can be used

---------------Service Info----------------------
-------------------------------------------------
--------------Guacamole--------------------------
● guacd.service - LSB: Guacamole proxy daemon
     Loaded: loaded (/etc/init.d/guacd; generated)
     Active: active (running) since Fri 2022-12-30 18:32:56 UTC; 1 day 15h ago

--------------Tomcat-----------------------------
Unit tomcat8.service could not be found.

--------------Mysql------------------------------
● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-12-30 18:33:21 UTC; 1 day 15h ago

--------------Apache-----------------------------
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-12-30 18:33:07 UTC; 1 day 15h ago


Uldis (UD)
Posts: 5068
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: Problem starting nodes

Post by Uldis (UD) » Sun Jan 01, 2023 7:54 pm

And what is that node which won't start?

bmwl
Posts: 10
Joined: Sat Feb 27, 2021 4:52 am

Re: Problem starting nodes

Post by bmwl » Sun Jan 01, 2023 10:00 pm

No nodes will start. I've tried dozen across labs, and also created new windows/linux/openbsd and nothing will start

Uldis (UD)
Posts: 5068
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: Problem starting nodes

Post by Uldis (UD) » Mon Jan 02, 2023 9:36 am

it is image creation issue then,,,
Winservers up to 2008, 2012, 2016, 2019
Windows 7,8,10
are fine
just need follow how to,,
https://www.eve-ng.net/index.php/docume ... n-the-eve/

even video is given how to create windows host

bmwl
Posts: 10
Joined: Sat Feb 27, 2021 4:52 am

Re: Problem starting nodes

Post by bmwl » Tue Jan 03, 2023 3:11 am

Hi, I don't think its a problem with windows image creation as the same problem is on windows, linux and bsd nodes
This server has been successfully running for a long time now, and now none of the nodes will start.
Last thing that was done before things stopped working was in install of a new clearpass image:

Code: Select all

  unzip CPPM-VM-x86_64-6.10.0.180076-KVM.raw.zip
  cd home/jenkins/KVMImage
  qemu-img convert -f raw -O qcow2 CPPM-VM-x86_64-6.10.0.180076-KVM.raw hda.qcow2
  /opt/qemu/bin/qemu-img create -f qcow2 hdb.qcow2 100G
  mkdir /opt/unetlab/addons/qemu/clearpass-6.10.0.clabv
  mv hda.qcow2 /opt/unetlab/addons/qemu/clearpass-6.10.0.clabv
  mv hdb.qcow2 /opt/unetlab/addons/qemu/clearpass-6.10.0.clabv
  /opt/unetlab/wrappers/unl_wrapper -a fixpermissions
Any help would be appreciated!

Uldis (UD)
Posts: 5068
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: Problem starting nodes

Post by Uldis (UD) » Tue Jan 03, 2023 1:19 pm

Create new lab and simply add some vIOS or CSR node and try, if it is same then your EVE needs to reinstall

bmwl
Posts: 10
Joined: Sat Feb 27, 2021 4:52 am

Re: Problem starting nodes

Post by bmwl » Wed Jan 04, 2023 10:23 pm

I don't have any Cisco gear, so can't test ios, but both VPC and docker apps work
Seems to narrow it down to a qemu issue? I tried, windows, linux, android, openbsd and esxi images and none of them work.

I found a couple of things trying to run the qemu command directly from the CLI :

Code: Select all

chroot . /opt/qemu-4.1.0/bin/qemu-system-x86_64 -device e1000,netdev=net0,mac=50:01:00:01:00:00 -netdev tap,id=net0,ifname=vun001000100100,script=no -smp 1,sockets=1,cores=1,threads=1 -m 8192 -name Winserver -uuid 19280543-ae2b-46fb-8ff3-97dbcdd8ee7d -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor unix:./mon2-sock,server,nowait -vnc :47127 -cdrom /opt/unetlab/addons/qemu/winserver-2019gui/cdrom.iso -drive file=virtioa.qcow2, if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -cpu host,+fsgsbase -vga std -usbdevice tablet -boot order=dc -drive file=/opt/qemu/share/qemu/virtio-win-drivers.img,index=1,if=floppy,readonly


first, the libsdl2-2.0-0 package seemed to be missing
also, the path in the -drive file=virtio.qcow2 parameter seems to be missing for some reason (but not on the -cdrom parameter)

once I fixed those 2 things I was able to start the VM from the command line

Code: Select all

apt-get install libsdl2-2.0-0

root@wlnetlab:/# chroot . /opt/qemu-4.1.0/bin/qemu-system-x86_64 -device e1000,netdev=net0,mac=50:01:00:01:00:00 -netdev tap,id=net0,ifname=vun001000100100,script=no -smp 1,sockets=1,cores=1,threads=1 -m 8192 -name Winserver -uuid 19280543-ae2b-46fb-8ff3-97dbcdd8ee7d -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor unix:./mon2-sock,server,nowait -vnc :47127 -cdrom /opt/unetlab/addons/qemu/winserver-2019gui/cdrom.iso -drive file=/opt/unetlab/addons/qemu/winserver-2019gui/virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -cpu host,+fsgsbase -vga std -usbdevice tablet -boot order=dc -drive file=/opt/qemu/share/qemu/virtio-win-drivers.img,index=1,if=floppy,readonly
qemu-system-x86_64: -usbdevice tablet: '-usbdevice' is deprecated, please use '-device usb-...' instead
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.fsgsbase [bit 0]

The VM then kept running until I did a ctrl-c

Note : I did a wipe and reinstall of this server a few months back when the upgrade to the new major Ubuntu release uninstalled the eve-ng-pro package and my system was broken, so the whole system should be very fresh.
also, maybe change the tomcat8 to tomcat9 in the eve-info script? It was a bit of a red herring when troubleshooting

bmwl
Posts: 10
Joined: Sat Feb 27, 2021 4:52 am

Re: Problem starting nodes

Post by bmwl » Sat Jan 07, 2023 5:45 am

I found a couple of things trying to run the qemu command directly from the CLI :
clarification: I ran the command that I found in the logs and found that the image was missing its full path, which I assume is a mistake since you can't invoke the chroot directly from the image's path (there isn't a usable root beneath those folders)

Any ideas? I'd love to be able to get this running for Monday when the rest of our IT team is back from holidays!

Post Reply