Poor performance

Before posting something, READ the changelog, WATCH the videos, howto and provide following:
Your install is: Bare metal, ESXi, what CPU model, RAM, HD, what EVE version you have, output of the uname -a and any other info that might help us faster.

Moderator: mike

Post Reply
hfeveuser
Posts: 3
Joined: Wed Aug 07, 2019 2:38 pm

Poor performance

Post by hfeveuser » Mon Aug 12, 2019 2:20 pm

We installed the system about a month ago and the installation looked fine and also the provisioning of the images and machines. However, after we started building labs we noticed that the machines got really slow to the point that we could not work with them anymore. It happens with Windows server, workstation, Cisco devices and others.

After we restart the server, everything works ok for some minutes and then start getting slow again.

Can you please, advise on what other information you may require to help us and what can we do to sort out this issue?


The server is installed on ESXi 6.5.0 Update 2 (Build 13635690) which is installed on a server with the following specs:

Manufacturer: Dell Inc.
Model: PowerEdge R730
CPU: 20 CPUs x Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Memory: 255.91 GB

Storage:
Adapter/Model/Driver
1/PERC H730 Mini/lsi_mr3
2/Wellsburg AHCI Controller/vmw_ahci
3/Wellsburg AHCI Controller/vmw_ahci
HD Information
Name: Local DELL Disk
Type: Disk
Capacity: 3.82 TB
Queue Depth: 448
Vendor: DELL
Status: Normal

Virtual machine information:
Guest OS: Ubuntu Linux (64-bit)
Used space: 3.23 TB
Host CPU: 7.5 GHz
Host memory: 218.15 GB


The details about Eve are shown below:
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 4.20.17-eve-ng-ukms+ x86_64)
Last login: Thu Aug 8 16:33:42 2019 from XX.XX.XX.XX
# uname -a
Linux eve-ng-pro 4.20.17-eve-ng-ukms+ #2 SMP Wed Jun 5 08:18:06 CEST 2019 x86_64 x86_64 x86_64 GNU/Linux

Uldis (UD)
Posts: 5067
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: Poor performance

Post by Uldis (UD) » Mon Aug 12, 2019 7:42 pm

first: how many CPU are assigned for eve vm?
RAM?
How many nodes per lab?

hfeveuser
Posts: 3
Joined: Wed Aug 07, 2019 2:38 pm

Re: Poor performance

Post by hfeveuser » Tue Aug 13, 2019 11:37 am

Hi,

Thank you for your response.

the EVE VM has assigned: 40 vCPU, 240 GB RAM, 3 TB HD.
Currently, there are 9 labs running one with 27 nodes and the rest only 5 or 6 nodes. However, it happened from the beginning before building the one with 27.

We have powered down some solutions like Cisco ISE and Palo Alto firewalls hoping that they are affecting the system negatively but it happens even when they are shut down and either way the eve VM does not to show any CPU or memory spike.

Uldis (UD)
Posts: 5067
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: Poor performance

Post by Uldis (UD) » Tue Aug 13, 2019 7:43 pm

I have similar EVE bare server, but I do not afford such heavy load for eve :)
Bare server x2 E5-2650v3 (40 vcpu) and 128 Gb ram.

Usually we run about 3-4 labs (3 eve users..)
each consume about 15% ram and cpu..
Normally eve hold that amount.

do not overcoount your CPU cores mate even your CPU is showing small percantage.. Your 9 labs probably already eating all 40 vcpu !!
Average CPU calculator is here:

https://docs.google.com/spreadsheets/d/ ... edit#gid=0

Download this sheet to use it !!!!!

Uldis

hfeveuser
Posts: 3
Joined: Wed Aug 07, 2019 2:38 pm

Re: Poor performance

Post by hfeveuser » Wed Aug 14, 2019 9:39 am

Hi Uldis,

Thank you for your replay.

I understand that with all the 9 labs running they would eat al the CPUs so a low performance can be expected.

However, the reason why I am seeking some help is that even with one or two labs running with no more than 5 or 6 devices each the performance drop and not in a coherent way, I mean. Let's say I have these to labs running all day then sometimes can be very slow and ten minutes after seems to be working normally, so it is not consistent. It led us to think that might be something wrong with the images I am running or a memory leak.

Are there any logs, alerts or further troubleshooting that I can carry out to find out if there is a problem?

Thanks,

Post Reply