Community edition VM on ESXi 6.7 running on Dual Processor

[mathewfer]

Hi

UPDATE:

I Installed fresh ESXi 6.7 with licencing "Enterprise Plus" on a new 2T SATA SSD before bare-metal install/test and found that the earlier reported errors are still there, after 24hrs of testing.

While moving to bare-metal install within the next few days. can I please know what should be the ESXi licencing for ver 6.7 with two physical processors, each physical processor with 18 cores?
I can test both ESXi 6.7 with the required licencing and bare-metal install to test this.

Currently I have "Enterprise Plus" licence running on ESXi 6.7

NOTE - more testing: One of the Linux VM under test on EVE-ng runs on ESXi directly without any errors but the same QEUM image version on EVE-ng gives errors. Further the same QEUM image tested on an EVE-ng Pro bare-metal (different hardware server from a friend) install runs without any errors.

[mathewfer]

Hi,

UPDATE:

I installed bare-metal and setup the same Linux VM/LAB. All are working very well and stable - no issues at all so far - no CPU errors with freezing effects.
I believe issue could be related to "nested virtualisation" and it is possible that it is on my ESXi setup.

Also when I ran this Linux VM on ESXi, other nodes (Cisco/Juniper/Arista) also get freezing experience.

I am now testing a full/few LABs now on bare-metal setup to verify that my server hardware has no issues at all.

[mathewfer]

Final update:

I will have to use EVE on bare-metal setup as it seems EVE does not work on my Dual CPU based ESXi 6.7 with Enterprise license.

It is very strange that EVE works very well with single CPU and after adding the 2nd CPU EVE did not like on ESXi.

EVE works very well on bare-metal on this same hardware.

[Uldis (UD)]

It works,
set CPU settings

OPTION for ESXi 7.x or ESXi 8.x.
Set Processors “Number of processors” Set “Cores per Socket”. If your server has dual CPU, then Cores per socket will be divided by 2.
Example below, show VM setup if you have dual CPU Server with 2x24 CPUs.
Set Expose hardware assisted virtualization to the guest OS to ON (checked) and set Expose IOMMU to the guest OS to ON (checked)

[mathewfer]

Hi Uldis,

Thank you for the reply.
I have set all the options as per your screen capture before, EXCEPT for "set Expose IOMMU to the guest OS to ON (checked)". I did that yesterday on my ESXi 6.7 (which started this feature). With that option enabled, I see no CPU errors anymore for the last 16hrs. But I will monitor running the full LAB for another 24hrs to see how the performance goes. If I do not see CPU errors/freezing (for few seconds, like in those errors) for about 48hrs or so, then I can say this setting resolves this issue.

I am not really clear what this "Expose IOMMU to the guest OS" does internally.

I thought I have exhausted all the available options.
It is still very early to conclude and will update after about 48hrs from now.

One question, can I please know what should be the minimum licensing required on ESXi 7.x/ESXi 8.x to run EVE-ng?

Thanks again for your reply.

[Uldis (UD)]

Literally this IOMMU is gate/bridge from virtual nested node to HW physical server resource, which makse stability for your lab nodes.
Furthermore, in a virtualized system, IOMMU enables a guest OS to directly access IO and allows direct interrupt delivery without the host's intervention. Similarly, IOMMU enables safe and secure user-space access IO in an unvirtualized system.

[mathewfer]

Hi Uldis,

Thanks for the replies/support on this.
I ran the full LAB with the IOMMU settings ticked but I still get issues in nodes and no CPU errors in logs of EVE-ng VM. See below. Supermicro box/board (6018R-MTR/MBD-X10DRL-i) is also ESXi 6 certified.

On LAB Nodes, I also noticed BGP bounces - random and frequent. Between CiscoIOL nodes, it could hold the BGP peer from start to end but all others such as Juniper vMX/vSRX, Arista etc has got BGP / LACP flaps/drops frequently.

LAB RAM and CPU requirements from your resource calculator:

RAM - 61952MB
CPU virtual cores total - 36

This LAB works 100% well on bare-metal on the same server.
As a solution, I have now setup dual boot with two SSDs, each with 2T, one with ESXi 6.7 and other bare-metal running EVE-ng. This helps with both EXi and EVE-ng learning LAB requirements. I am not sure I have left any more tests to be done with ESXi 6.7.
If there are other sugguests to test, please let me know. If I find any fixes later on, I will update here.

Board-ESXi-cert.png

Errors logs on devops Linux VM:

Code: Select all

user1@devops-srv:~$ cat /var/log/kern.log | grep watchdog
Jan 14 09:24:55 tools-srv kernel: NMI watchdog: Perf NMI watchdog permanently disabled
Jan 14 09:47:21 tools-srv kernel: NMI watchdog: Perf NMI watchdog permanently disabled
Jan 14 09:53:31 tools-srv kernel: watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:102]
Jan 14 09:53:31 tools-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
Jan 14 09:53:31 tools-srv kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 54s! [ps:2176]
Jan 14 09:53:32 tools-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
Jan 15 00:25:47 tools-srv kernel: NMI watchdog: Perf NMI watchdog permanently disabled
Jan 15 03:11:46 tools-srv kernel: NMI watchdog: Perf NMI watchdog permanently disabled
user1@devops-srv:~$ cat /var/log/kern.log.1 | grep watchdog
Jan 11 06:15:31 tools-srv kernel: NMI watchdog: Perf NMI watchdog permanently disabled
user1@devops-srv:~$

No CPU errors in EVE-ng logs

Code: Select all

root@eve-ng:/opt/unetlab# cat /var/log/kern.log | grep watchdog
Jan 14 09:46:08 eve-ng kernel: [    1.146508] NMI watchdog: Perf NMI watchdog permanently disabled
root@eve-ng:/opt/unetlab# cat /var/log/kern.log | grep CPU
Jan 14 09:46:08 eve-ng kernel: [    0.605635] smpboot: Allowing 70 CPUs, 0 hotplug CPUs
Jan 14 09:46:08 eve-ng kernel: [    0.605698] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:70 nr_cpu_ids:70 nr_node_ids:2
Jan 14 09:46:08 eve-ng kernel: [    1.030383] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=70, Nodes=2
Jan 14 09:46:08 eve-ng kernel: [    1.041869] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=70.
Jan 14 09:46:08 eve-ng kernel: [    1.045947] random: crng init done (trusting CPU's manufacturer)
Jan 14 09:46:08 eve-ng kernel: [    1.137952] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
Jan 14 09:46:08 eve-ng kernel: [    1.137953] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
Jan 14 09:46:08 eve-ng kernel: [    1.145111] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz (family: 0x6, model: 0x4f, stepping: 0x1)
Jan 14 09:46:08 eve-ng kernel: [    1.145857] core: CPUID marked event: 'cpu cycles' unavailable
Jan 14 09:46:08 eve-ng kernel: [    1.145859] core: CPUID marked event: 'instructions' unavailable
Jan 14 09:46:08 eve-ng kernel: [    1.145860] core: CPUID marked event: 'bus cycles' unavailable
Jan 14 09:46:08 eve-ng kernel: [    1.145861] core: CPUID marked event: 'cache references' unavailable
Jan 14 09:46:08 eve-ng kernel: [    1.145861] core: CPUID marked event: 'cache misses' unavailable
Jan 14 09:46:08 eve-ng kernel: [    1.145862] core: CPUID marked event: 'branch instructions' unavailable
Jan 14 09:46:08 eve-ng kernel: [    1.145863] core: CPUID marked event: 'branch misses' unavailable
Jan 14 09:46:08 eve-ng kernel: [    1.147612] smp: Bringing up secondary CPUs ...
Jan 14 09:46:08 eve-ng kernel: [    1.147732] .... node  #0, CPUs:        #1
Jan 14 09:46:08 eve-ng kernel: [    1.194155] .... node  #1, CPUs:   #35
Jan 14 09:46:08 eve-ng kernel: [    0.050970] smpboot: CPU 35 Converting physical 0 to logical die 1
Jan 14 09:46:08 eve-ng kernel: [    1.245183] smp: Brought up 2 nodes, 70 CPUs
Jan 14 09:46:08 eve-ng kernel: [    2.281456] intel_pstate: CPU model not supported
Jan 14 09:46:08 eve-ng kernel: [    2.286148] ledtrig-cpu: registered to indicate activity on CPUs
Jan 14 09:55:59 eve-ng kernel: [  614.631398] WARNING: CPU: 53 PID: 592 at arch/x86/kvm/../../../virt/kvm/kvm_main.c:649 kvm_mmu_notifier_change_pte+0x28a/0x2b0 [kvm]
Jan 14 09:55:59 eve-ng kernel: [  614.631628] CPU: 53 PID: 592 Comm: uksmd Not tainted 5.17.8-eve-ng-uksm-wg+ #1
root@eve-ng:/opt/unetlab# cat /var/log/kern.log.1 | grep CPU
Jan 10 08:36:27 eve-ng kernel: [    0.597700] smpboot: Allowing 70 CPUs, 0 hotplug CPUs
Jan 10 08:36:27 eve-ng kernel: [    0.597739] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:70 nr_cpu_ids:70 nr_node_ids:2
Jan 10 08:36:27 eve-ng kernel: [    1.022474] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=70, Nodes=2
Jan 10 08:36:27 eve-ng kernel: [    1.031884] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=70.
Jan 10 08:36:27 eve-ng kernel: [    1.035849] random: crng init done (trusting CPU's manufacturer)
Jan 10 08:36:27 eve-ng kernel: [    1.075471] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
Jan 10 08:36:27 eve-ng kernel: [    1.075472] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
Jan 10 08:36:27 eve-ng kernel: [    1.105224] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz (family: 0x6, model: 0x4f, stepping: 0x1)
Jan 10 08:36:27 eve-ng kernel: [    1.105224] core: CPUID marked event: 'cpu cycles' unavailable
Jan 10 08:36:27 eve-ng kernel: [    1.105224] core: CPUID marked event: 'instructions' unavailable
Jan 10 08:36:27 eve-ng kernel: [    1.105224] core: CPUID marked event: 'bus cycles' unavailable
Jan 10 08:36:27 eve-ng kernel: [    1.105224] core: CPUID marked event: 'cache references' unavailable
Jan 10 08:36:27 eve-ng kernel: [    1.105224] core: CPUID marked event: 'cache misses' unavailable
Jan 10 08:36:27 eve-ng kernel: [    1.105224] core: CPUID marked event: 'branch instructions' unavailable
Jan 10 08:36:27 eve-ng kernel: [    1.105224] core: CPUID marked event: 'branch misses' unavailable
Jan 10 08:36:27 eve-ng kernel: [    1.105224] smp: Bringing up secondary CPUs ...
Jan 10 08:36:27 eve-ng kernel: [    1.105224] .... node  #0, CPUs:        #1
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 1 Converting physical 2 to logical package 1
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 1 Converting physical 0 to logical die 1
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 2 Converting physical 4 to logical package 2
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 2 Converting physical 0 to logical die 2
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 3 Converting physical 6 to logical package 3
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 3 Converting physical 0 to logical die 3
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 4 Converting physical 8 to logical package 4
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 4 Converting physical 0 to logical die 4
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 5 Converting physical 10 to logical package 5
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 5 Converting physical 0 to logical die 5
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 6 Converting physical 12 to logical package 6
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 6 Converting physical 0 to logical die 6
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 7 Converting physical 14 to logical package 7
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 7 Converting physical 0 to logical die 7
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 8 Converting physical 16 to logical package 8
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 8 Converting physical 0 to logical die 8
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 9 Converting physical 18 to logical package 9
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 9 Converting physical 0 to logical die 9
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 10 Converting physical 20 to logical package 10
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 10 Converting physical 0 to logical die 10
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 11 Converting physical 22 to logical package 11
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 11 Converting physical 0 to logical die 11
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 12 Converting physical 24 to logical package 12
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 12 Converting physical 0 to logical die 12
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 13 Converting physical 26 to logical package 13
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 13 Converting physical 0 to logical die 13
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 14 Converting physical 28 to logical package 14
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 14 Converting physical 0 to logical die 14
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 15 Converting physical 30 to logical package 15
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 15 Converting physical 0 to logical die 15
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 16 Converting physical 32 to logical package 16
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 16 Converting physical 0 to logical die 16
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 17 Converting physical 34 to logical package 17
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 17 Converting physical 0 to logical die 17
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 18 Converting physical 36 to logical package 18
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 18 Converting physical 0 to logical die 18
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 19 Converting physical 38 to logical package 19
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 19 Converting physical 0 to logical die 19
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 20 Converting physical 40 to logical package 20
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 20 Converting physical 0 to logical die 20
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 21 Converting physical 42 to logical package 21
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 21 Converting physical 0 to logical die 21
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 22 Converting physical 44 to logical package 22
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 22 Converting physical 0 to logical die 22
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 23 Converting physical 46 to logical package 23
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 23 Converting physical 0 to logical die 23
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 24 Converting physical 48 to logical package 24
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 24 Converting physical 0 to logical die 24
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 25 Converting physical 50 to logical package 25
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 25 Converting physical 0 to logical die 25
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 26 Converting physical 52 to logical package 26
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 26 Converting physical 0 to logical die 26
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 27 Converting physical 54 to logical package 27
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 27 Converting physical 0 to logical die 27
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 28 Converting physical 56 to logical package 28
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 28 Converting physical 0 to logical die 28
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 29 Converting physical 58 to logical package 29
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 29 Converting physical 0 to logical die 29
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 30 Converting physical 60 to logical package 30
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 30 Converting physical 0 to logical die 30
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 31 Converting physical 62 to logical package 31
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 31 Converting physical 0 to logical die 31
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 32 Converting physical 64 to logical package 32
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 32 Converting physical 0 to logical die 32
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 33 Converting physical 66 to logical package 33
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 33 Converting physical 0 to logical die 33
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 34 Converting physical 68 to logical package 34
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 34 Converting physical 0 to logical die 34
Jan 10 08:36:27 eve-ng kernel: [    1.150525] .... node  #1, CPUs:   #35
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 35 Converting physical 70 to logical package 35
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 35 Converting physical 0 to logical die 35
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 36 Converting physical 72 to logical package 36
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 36 Converting physical 0 to logical die 36
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 37 Converting physical 74 to logical package 37
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 37 Converting physical 0 to logical die 37
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 38 Converting physical 76 to logical package 38
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 38 Converting physical 0 to logical die 38
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 39 Converting physical 78 to logical package 39
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 39 Converting physical 0 to logical die 39
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 40 Converting physical 80 to logical package 40
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 40 Converting physical 0 to logical die 40
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 41 Converting physical 82 to logical package 41
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 41 Converting physical 0 to logical die 41
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 42 Converting physical 84 to logical package 42
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 42 Converting physical 0 to logical die 42
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 43 Converting physical 86 to logical package 43
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 43 Converting physical 0 to logical die 43
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 44 Converting physical 88 to logical package 44
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 44 Converting physical 0 to logical die 44
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 45 Converting physical 90 to logical package 45
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 45 Converting physical 0 to logical die 45
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 46 Converting physical 92 to logical package 46
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 46 Converting physical 0 to logical die 46
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 47 Converting physical 94 to logical package 47
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 47 Converting physical 0 to logical die 47
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 48 Converting physical 96 to logical package 48
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 48 Converting physical 0 to logical die 48
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 49 Converting physical 98 to logical package 49
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 49 Converting physical 0 to logical die 49
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 50 Converting physical 100 to logical package 50
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 50 Converting physical 0 to logical die 50
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 51 Converting physical 102 to logical package 51
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 51 Converting physical 0 to logical die 51
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 52 Converting physical 104 to logical package 52
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 52 Converting physical 0 to logical die 52
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 53 Converting physical 106 to logical package 53
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 53 Converting physical 0 to logical die 53
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 54 Converting physical 108 to logical package 54
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 54 Converting physical 0 to logical die 54
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 55 Converting physical 110 to logical package 55
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 55 Converting physical 0 to logical die 55
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 56 Converting physical 112 to logical package 56
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 56 Converting physical 0 to logical die 56
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 57 Converting physical 114 to logical package 57
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 57 Converting physical 0 to logical die 57
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 58 Converting physical 116 to logical package 58
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 58 Converting physical 0 to logical die 58
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 59 Converting physical 118 to logical package 59
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 59 Converting physical 0 to logical die 59
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 60 Converting physical 120 to logical package 60
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 60 Converting physical 0 to logical die 60
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 61 Converting physical 122 to logical package 61
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 61 Converting physical 0 to logical die 61
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 62 Converting physical 124 to logical package 62
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 62 Converting physical 0 to logical die 62
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 63 Converting physical 126 to logical package 63
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 63 Converting physical 0 to logical die 63
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 64 Converting physical 128 to logical package 64
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 64 Converting physical 0 to logical die 64
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 65 Converting physical 130 to logical package 65
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 65 Converting physical 0 to logical die 65
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 66 Converting physical 132 to logical package 66
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 66 Converting physical 0 to logical die 66
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 67 Converting physical 134 to logical package 67
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 67 Converting physical 0 to logical die 67
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 68 Converting physical 136 to logical package 68
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 68 Converting physical 0 to logical die 68
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 69 Converting physical 138 to logical package 69
Jan 10 08:36:27 eve-ng kernel: [    0.009413] smpboot: CPU 69 Converting physical 0 to logical die 69
Jan 10 08:36:27 eve-ng kernel: [    1.206180] smp: Brought up 2 nodes, 70 CPUs
Jan 10 08:36:27 eve-ng kernel: [    2.230333] intel_pstate: CPU model not supported
Jan 10 08:36:27 eve-ng kernel: [    2.234866] ledtrig-cpu: registered to indicate activity on CPUs
Jan 10 08:42:39 eve-ng kernel: [  390.231431] WARNING: CPU: 22 PID: 592 at arch/x86/kvm/../../../virt/kvm/kvm_main.c:649 kvm_mmu_notifier_change_pte+0x28a/0x2b0 [kvm]
Jan 10 08:42:39 eve-ng kernel: [  390.231693] CPU: 22 PID: 592 Comm: uksmd Not tainted 5.17.8-eve-ng-uksm-wg+ #1
Jan 10 23:50:47 eve-ng kernel: [    0.621062] smpboot: Allowing 70 CPUs, 0 hotplug CPUs
Jan 10 23:50:47 eve-ng kernel: [    0.621110] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:70 nr_cpu_ids:70 nr_node_ids:2
Jan 10 23:50:47 eve-ng kernel: [    1.053669] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=70, Nodes=2
Jan 10 23:50:47 eve-ng kernel: [    1.063963] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=70.
Jan 10 23:50:47 eve-ng kernel: [    1.068808] random: crng init done (trusting CPU's manufacturer)
Jan 10 23:50:47 eve-ng kernel: [    1.160797] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
Jan 10 23:50:47 eve-ng kernel: [    1.160799] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
Jan 10 23:50:47 eve-ng kernel: [    1.166810] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz (family: 0x6, model: 0x4f, stepping: 0x1)
Jan 10 23:50:47 eve-ng kernel: [    1.167641] core: CPUID marked event: 'cpu cycles' unavailable
Jan 10 23:50:47 eve-ng kernel: [    1.167643] core: CPUID marked event: 'instructions' unavailable
Jan 10 23:50:47 eve-ng kernel: [    1.167644] core: CPUID marked event: 'bus cycles' unavailable
Jan 10 23:50:47 eve-ng kernel: [    1.167645] core: CPUID marked event: 'cache references' unavailable
Jan 10 23:50:47 eve-ng kernel: [    1.167647] core: CPUID marked event: 'cache misses' unavailable
Jan 10 23:50:47 eve-ng kernel: [    1.167648] core: CPUID marked event: 'branch instructions' unavailable
Jan 10 23:50:47 eve-ng kernel: [    1.167649] core: CPUID marked event: 'branch misses' unavailable
Jan 10 23:50:47 eve-ng kernel: [    1.169526] smp: Bringing up secondary CPUs ...
Jan 10 23:50:47 eve-ng kernel: [    1.169697] .... node  #0, CPUs:        #1
Jan 10 23:50:47 eve-ng kernel: [    1.215866] .... node  #1, CPUs:   #35
Jan 10 23:50:47 eve-ng kernel: [    0.053821] smpboot: CPU 35 Converting physical 0 to logical die 1
Jan 10 23:50:47 eve-ng kernel: [    1.267797] smp: Brought up 2 nodes, 70 CPUs
Jan 10 23:50:47 eve-ng kernel: [    2.295572] intel_pstate: CPU model not supported
Jan 10 23:50:47 eve-ng kernel: [    2.299345] ledtrig-cpu: registered to indicate activity on CPUs
Jan 11 00:02:15 eve-ng kernel: [  707.600182] WARNING: CPU: 9 PID: 594 at arch/x86/kvm/../../../virt/kvm/kvm_main.c:649 kvm_mmu_notifier_change_pte+0x28a/0x2b0 [kvm]
Jan 11 00:02:15 eve-ng kernel: [  707.600310] CPU: 9 PID: 594 Comm: uksmd Not tainted 5.17.8-eve-ng-uksm-wg+ #1
Jan 11 06:03:02 eve-ng kernel: [    0.627628] smpboot: Allowing 70 CPUs, 0 hotplug CPUs
Jan 11 06:03:02 eve-ng kernel: [    0.627674] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:70 nr_cpu_ids:70 nr_node_ids:2
Jan 11 06:03:02 eve-ng kernel: [    1.049572] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=70, Nodes=2
Jan 11 06:03:02 eve-ng kernel: [    1.059434] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=70.
Jan 11 06:03:02 eve-ng kernel: [    1.063455] random: crng init done (trusting CPU's manufacturer)
Jan 11 06:03:02 eve-ng kernel: [    1.121226] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
Jan 11 06:03:02 eve-ng kernel: [    1.121227] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
Jan 11 06:03:02 eve-ng kernel: [    1.139809] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz (family: 0x6, model: 0x4f, stepping: 0x1)
Jan 11 06:03:02 eve-ng kernel: [    1.140322] core: CPUID marked event: 'cpu cycles' unavailable
Jan 11 06:03:02 eve-ng kernel: [    1.140324] core: CPUID marked event: 'instructions' unavailable
Jan 11 06:03:02 eve-ng kernel: [    1.140325] core: CPUID marked event: 'bus cycles' unavailable
Jan 11 06:03:02 eve-ng kernel: [    1.140326] core: CPUID marked event: 'cache references' unavailable
Jan 11 06:03:02 eve-ng kernel: [    1.140327] core: CPUID marked event: 'cache misses' unavailable
Jan 11 06:03:02 eve-ng kernel: [    1.140329] core: CPUID marked event: 'branch instructions' unavailable
Jan 11 06:03:02 eve-ng kernel: [    1.140330] core: CPUID marked event: 'branch misses' unavailable
Jan 11 06:03:02 eve-ng kernel: [    1.143097] smp: Bringing up secondary CPUs ...
Jan 11 06:03:02 eve-ng kernel: [    1.143449] .... node  #0, CPUs:        #1
Jan 11 06:03:02 eve-ng kernel: [    1.191921] .... node  #1, CPUs:   #35
Jan 11 06:03:02 eve-ng kernel: [    0.020274] smpboot: CPU 35 Converting physical 0 to logical die 1
Jan 11 06:03:02 eve-ng kernel: [    1.244026] smp: Brought up 2 nodes, 70 CPUs
Jan 11 06:03:02 eve-ng kernel: [    2.271539] intel_pstate: CPU model not supported
Jan 11 06:03:02 eve-ng kernel: [    2.275119] ledtrig-cpu: registered to indicate activity on CPUs
Jan 11 06:09:28 eve-ng kernel: [  405.706599] WARNING: CPU: 2 PID: 591 at arch/x86/kvm/../../../virt/kvm/kvm_main.c:649 kvm_mmu_notifier_change_pte+0x28a/0x2b0 [kvm]
Jan 11 06:09:28 eve-ng kernel: [  405.706806] CPU: 2 PID: 591 Comm: uksmd Not tainted 5.17.8-eve-ng-uksm-wg+ #1
root@eve-ng:/opt/unetlab#

[Uldis (UD)]

show list of nodes loaded in la and how many

[mathewfer]

Hi Uldis,

Thanks for the reply and I am sending the node list via a PM. The CPU and RAM requirement summary is same.

[mathewfer]

Hi Uldis,

UPDATE:

I did the below on a new 2T SSD SATA

1. Installed ESXi 7.0.3
2. Installed EVE-ng CE from ISO
3. Copied QUEM images for Juniper, Cisco, Cumulus, Linux images etc
4. Imported the test LAB which worked 100% without any CPU errors, freezes, BGP/LACP/interface bounces for 3+ days on bare-metal EVE-ng CE on the same server hardware
5. Copied the device templates (.yml) files under "intel" as per Uldis's PM
6. ESXi settings as per Uldis's post and PM - all CPU related settings for 2 CPU are done
7. ESXi 7 is with the default evaluation full licence for 6 days which is good enough first to test.

Ran the LAB for 6hrs and observed the below:

1. There are no CPU freeze errors (maybe so far) in EVE-ng and as well as devops Linux instance
2. Linux image experiences few seconds freezing feel but no logs of CPU errors
3. There are many BGP and LACP flaps on Juniper and Cumulus. There were very bad as they are frequent - every 3min flapping like.

In summary, with ESXi 7.x, there are no improvements when compared to ESXi 6.7.
Next, I will install ESXi 8.x to try out the same LAB.

If ESXi 8 fails to maintain a stable LAB, I will have a dual boot - ESXi for other LAB work and EVE-ng on bare-metal install on 2nd SSD.

Code: Select all

root@eve-ng:~# uptime
 10:37:03 up  6:37,  1 user,  load average: 10.91, 12.65, 14.15
root@eve-ng:~# cat /var/log/kern.log | grep watchdog
Jan 22 03:56:16 eve-ng kernel: [    0.786343] NMI watchdog: Perf NMI watchdog permanently disabled
Jan 22 04:00:01 eve-ng kernel: [    0.784581] NMI watchdog: Perf NMI watchdog permanently disabled
root@eve-ng:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                                59G     0   59G   0% /dev
tmpfs                               12G  6.4M   12G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  1.2T  104G  1.1T   9% /
tmpfs                               60G     0   60G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                               60G     0   60G   0% /sys/fs/cgroup
/dev/sda2                          2.0G  257M  1.6G  15% /boot
/dev/loop0                          64M   64M     0 100% /snap/core20/1828
/dev/loop1                          92M   92M     0 100% /snap/lxd/24061
/dev/loop2                          50M   50M     0 100% /snap/snapd/18357
/dev/loop3                          41M   41M     0 100% /snap/snapd/20671
tmpfs                               12G     0   12G   0% /run/user/0
root@eve-ng:~# cat /proc/cpuinfo | grep processor | wc -l
70
root@eve-ng:~#