EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Before posting something, READ the changelog, WATCH the videos, howto and provide following:
Your install is: Bare metal, ESXi, what CPU model, RAM, HD, what EVE version you have, output of the uname -a and any other info that might help us faster.

Moderator: mike

SirStephanikus
Posts: 10
Joined: Tue May 19, 2020 5:14 pm

EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by SirStephanikus » Mon Aug 03, 2020 7:56 pm

Hello to the EVE-NG Team

I've a serious problem with EVE-NG and VMWare Workstation 15.5.6. Each time, when I load my INE lab (10 CSR1000v nodes with a vSwitch), the whole systems freezes at ~50%. With "system" I mean even the Hypervisor.
So let me list what I got here and what I already did:

EVE-NG PRO latest patch level
OS: XUbuntu 20.04 LTS current patch level
CPU: i0-10980XE
RAM: 128 GB 2666 MHZ Corsair
HDD: 2 Samsung NVME's
GFX+BIOS do have the latest drivers/firmware
Hypervisor: VMWare Workstation Pro 15.5.6

I already re-installed EVE but the problem stays, same goes for Windows 10 Pro, even there I get those freezes.
I ran multiple RAM tests for many many hours and also stress tested my CPU...everything works fine.
Other VM's run fine.

For a test I used the beta of VMWare Workstation Zealot which support way more cores and RAM...same problem.

I have no idea what affects this. Please Uldis, help.

Uldis (UD)
Posts: 3351
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by Uldis (UD) » Tue Aug 04, 2020 3:41 pm

So, Native OS on your PC is Ubuntu 20?
First need try to reinstall VM ware

SirStephanikus
Posts: 10
Joined: Tue May 19, 2020 5:14 pm

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by SirStephanikus » Tue Aug 04, 2020 4:06 pm

Hi Uldis

OS is a native Xubuntu 20.04 LTS

I already re-installed VMWare Workstation.

Uldis (UD)
Posts: 3351
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by Uldis (UD) » Tue Aug 04, 2020 6:31 pm

Try my lab and make sure you are using:
csr1000vng-universalk9.16.06.07.Everest
The problem could be if lab is very old and created time ago using not bridge but OVS cloud
and CSR image, nan raise issues...

Such lab for sure need more than 12 CPU assgned for eve vm, and min 48GB RAM
You do not have the required permissions to view the files attached to this post.

SirStephanikus
Posts: 10
Joined: Tue May 19, 2020 5:14 pm

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by SirStephanikus » Sun Aug 09, 2020 8:25 pm

Hi Uldis

I tried your lab (which is the INE lab 1:1 to mine). Same result, after around 8GB of loaded RAM the whole system freezes (like I said, even my underlying OS).
I did some further research and found some hints that the INTEL TURBO MODE were responsible for some issues, but those hints are OLD !

The VM has 16 cores and 64 GB RAM.

However, I switched it off (CPU runs now with its native 3 GHZ instead of 4.6 GHZ). The labs do start now...BUT...I never had those issues with the 2.x EVE-NG version.
Another strange effect that I see now:
--> The loading of 10 CSR's take a while...much longer than I used to
--> When the CSR's are loaded, the total consumed RAM is around 14 GB (first it rises to ~ 27 GB), but it should be more than 40GB. How is that? HTOP shows that qemu has the memory set with -m 4096.
--> After I restarted EVE-NG (like another day), I can not start my CSR routers. EVE-NG writes the green messages at the top right corner that all images started but nothing happens. It doesn't matter if I start it via right click "start selected nodes" or via the more actions menu.
I'm forced to wipe 'em all and than the VM begins to start the nodes. Same goes when I simply switch off all nodes and want to directly restart them.

Uldis (UD)
Posts: 3351
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by Uldis (UD) » Sun Aug 09, 2020 10:07 pm

Iam afraid to ask, what CPU model you have please?
But please exact...

I cannot test your situation because I simply have not such stuff

Intel CPU can not and never did it
at least on windows/vmware

SirStephanikus
Posts: 10
Joined: Tue May 19, 2020 5:14 pm

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by SirStephanikus » Mon Aug 10, 2020 6:00 am

Hello Uldis

As mentioned in my 1st post:

CPU = Intel i9-10980XE

lscpu output

Code: Select all

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          36
On-line CPU(s) list:             0-35
Thread(s) per core:              2
Core(s) per socket:              18
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
Stepping:                        7
CPU MHz:                         1771.657
CPU max MHz:                     3000,0000
CPU min MHz:                     1200,0000
BogoMIPS:                        6000.00
Virtualization:                  VT-x
L1d cache:                       576 KiB
L1i cache:                       576 KiB
L2 cache:                        18 MiB
L3 cache:                        24,8 MiB
NUMA node0 CPU(s):               0-35
Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled v
                                 ia prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user
                                  pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RS
                                 B filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtr
                                 r pge mca cmov pat pse36 clflush dts acpi mmx f
                                 xsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rd
                                 tscp lm constant_tsc art arch_perfmon pebs bts 
                                 rep_good nopl xtopology nonstop_tsc cpuid aperf
                                 mperf pni pclmulqdq dtes64 monitor ds_cpl vmx e
                                 st tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca s
                                 se4_1 sse4_2 x2apic movbe popcnt tsc_deadline_t
                                 imer aes xsave avx f16c rdrand lahf_lm abm 3dno
                                 wprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid
                                 _single ssbd mba ibrs ibpb stibp ibrs_enhanced 
                                 tpr_shadow vnmi flexpriority ept vpid ept_ad fs
                                 gsbase tsc_adjust bmi1 avx2 smep bmi2 erms invp
                                 cid cqm mpx rdt_a avx512f avx512dq rdseed adx s
                                 map clflushopt clwb intel_pt avx512cd avx512bw 
                                 avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc
                                  cqm_occup_llc cqm_mbm_total cqm_mbm_local dthe
                                 rm arat pln pts hwp hwp_act_window hwp_epp hwp_
                                 pkg_req avx512_vnni md_clear flush_l1d arch_cap
                                 abilities

Uldis (UD)
Posts: 3351
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by Uldis (UD) » Mon Aug 10, 2020 9:14 am

first VM ware workstation does not support more that 16 CPU as I know and 64G RAM
try reduce on this frame, and test

SirStephanikus
Posts: 10
Joined: Tue May 19, 2020 5:14 pm

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by SirStephanikus » Mon Aug 10, 2020 10:13 am

Hello Uldis

Even if you have 10000 Cores, VMWare Workstation Pro limits the useable cores to each VM, it does not mean that you can't use VMWare Workstation.

I testet it with VMWare Workstation 15.5.6 (current version and it supports also Ubuntu 20.04 LTS) and I tried the new beta (see my 1st post) which also maximizes the usable CPU Core Count to 32 and RAM to 128 GB.
Again, I had not those issues with EVE-NG 2.x

My last lscpu output were from my main system NOT from inside the VM.

SirStephanikus
Posts: 10
Joined: Tue May 19, 2020 5:14 pm

Re: EVE-NG Pro on VMWare Workstation 15.5.6 System-Freezes

Post by SirStephanikus » Sun Aug 16, 2020 5:05 pm

I fixed it ... or better say, I found the problems.

My first problem:
System freezes
I turned off INTEL TURBO MODE in my BIOS which can speed up my CPU with +1.5 GHZ with ease. After it, all freezes were gone. BUT I NEVER HAD THESE ISSUES WITH EVE 2.x !!! @Uldis, does the Kernel from Ubuntu 18.04 miss some modules ??? It is definitive a serious bug.

My second problem:
--> When the CSR's are loaded, the total consumed RAM is around 14 GB (first it rises to ~ 27 GB), but it should be more than 40GB. How is that? HTOP shows that qemu has the memory set with -m 4096.
Here the bad guy is UKSM --> Once it is switched off, EVE use all the RAM AND also I can boot up a 2nd time without wiping the node. I switched it off globally, I guess, there are some bugs in it !

Another problem (minor):
systemctl status ovfstartup.service threw a error in the output --> tunctl can't be found

A quick look into /opt/ovf/ovfstartup.sh show me a wrong path:

Code: Select all

# Fixing setcap
setcap cap_net_admin+ep /usr/sbin/tunctl
The location is wrong, provable with "which tunctl" --> /usr/bin/tunctl (sbin is wrong, bin is correct)
Once I corrected the path, the message were gone too

Code: Select all

# Fixing setcap
setcap cap_net_admin+ep /usr/bin/tunctl

Post Reply