Some IOL nodes crashing after start

[zillow32]

I have a lab with 4 IOL L2 nodes. Two of the nodes continue to work, however, the other 2 are crashing right after starting. The only differences are the crashing nodes do have a LOT of config compared to the other nodes (BGP, VRFs, Interfaces, etc). The nodes seemed to be crashing once the config was put in place, however, when starting them with No startup config they continue to crash. Not sure what's going on. I know this isn't a licensing issue as 2 nodes with the same firmware load run just fine.

Code: Select all

***** EVE-NG INFORMATION TOOL - CREATED BY R! ******
Wed Jan 31 01:58:29 EET 2018
***** EVE-NG VERSION INSTALLED  *****
eve-ng 2.0.3-86
***** HOSTNAME INFORMATION *****
   Static hostname: eve-ng
    Virtualization: vmware
  Operating System: Ubuntu 16.04.3 LTS
            Kernel: Linux 4.9.40-eve-ng-ukms-2+
      Architecture: x86-64
***** FILE SYSTEM DISK SPACE USAGE *****
Filesystem                    Size  Used Avail Use% Mounted on
udev                          9.8G     0  9.8G   0% /dev
tmpfs                         2.0G   15M  2.0G   1% /run
/dev/mapper/eve--ng--vg-root   73G   20G   50G  29% /
tmpfs                         9.9G     0  9.9G   0% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         9.9G     0  9.9G   0% /sys/fs/cgroup
/dev/sda1                     472M  217M  231M  49% /boot
 ***** CPU INFORMATION  *****
model name      : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
 ***** FREE AND USED MEMORY *****
              total        used        free      shared  buff/cache   available
Mem:            19G        473M         18G         25M        579M         18G
Swap:          6.0G          0B        6.0G
 ***** INTERFACE INFORMATION *****
pnet0           8000.000c2962ba5b       no              eth0
pnet1           8000.000000000000       no
pnet2           8000.000000000000       no
pnet3           8000.000000000000       no
pnet4           8000.000000000000       no
pnet5           8000.000000000000       no
pnet6           8000.000000000000       no
pnet7           8000.000000000000       no
pnet8           8000.000000000000       no
pnet9           8000.000000000000       no
 ***** KVM CHECK INFORMATION  *****
INFO: /dev/kvm exists
KVM acceleration can be used
 ***** APACHE STATUS  *****
● apache2.service - LSB: Apache2 web server
   Active: active (running) since Wed 2018-01-31 01:12:07 EET; 46min ago
 ***** CHECKING INERNET STATUS  *****
PING www.google.com (172.217.11.36) 56(84) bytes of data.
64 bytes from lga25s61-in-f4.1e100.net (172.217.11.36): icmp_seq=1 ttl=55 time=21.5 ms
--- www.google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 21.517/21.517/21.517/0.000 ms

root@eve-ng:/opt/unetlab/data/Logs# cat unl_wrapper.txt

Code: Select all

Jan 31 00:54:32 INFO: sysctl -w net.ipv6.conf.vunl0_15_23.disable_ipv6=1
Jan 31 00:54:32 INFO: tunctl -u unl0 -g root -t vunl0_15_39 2>&1
Jan 31 00:54:32 INFO: sysctl -w net.ipv6.conf.vunl0_15_39.disable_ipv6=1
Jan 31 00:54:32 INFO: tunctl -u unl0 -g root -t vunl0_15_55 2>&1
Jan 31 00:54:32 INFO: sysctl -w net.ipv6.conf.vunl0_15_55.disable_ipv6=1
Jan 31 00:54:32 INFO: CWD is /opt/unetlab/tmp/0/bf879c37-0a65-47a0-96be-90215439504d/15
Jan 31 00:54:32 INFO: starting /opt/unetlab/wrappers/iol_wrapper -T 0 -D 15 -t "test-node-01" -F /opt/unetlab/addons/iol/bin/i86bi-linux-l2-adventerprise-15.1b.bin -d 0 -e 8 -s 0 --  -n 1024 -q -m 1024 > /opt/unetlab/tmp/0/bf879c37-0a65-47a0-96be-90215439504d/15/wrapper.txt 2>&1 &

root@eve-ng:/opt/unetlab/data/Logs# cat /opt/unetlab/tmp/0/bf879c37-0a65-47a0-96be-90215439504d/15/wrapper.txt

Code: Select all

30/0 23:54:32.397 INF   Tennant_id = 0
30/0 23:54:32.397 INF   Device_id = 15
30/0 23:54:32.397 INF   NETMAP file created.
30/0 23:54:32.397 INF   TS configured.
30/0 23:54:32.397 INF   TAP interface configured (s=8, n=vunl0_15_0).
30/0 23:54:32.397 INF   TAP interface configured (s=10, n=vunl0_15_16).
30/0 23:54:32.397 INF   TAP interface configured (s=12, n=vunl0_15_32).
30/0 23:54:32.397 INF   TAP interface configured (s=14, n=vunl0_15_48).
30/0 23:54:32.397 INF   TAP interface configured (s=16, n=vunl0_15_1).
30/0 23:54:32.397 INF   TAP interface configured (s=18, n=vunl0_15_17).
30/0 23:54:32.397 INF   TAP interface configured (s=20, n=vunl0_15_33).
30/0 23:54:32.397 INF   TAP interface configured (s=22, n=vunl0_15_49).
30/0 23:54:32.397 INF   TAP interface configured (s=24, n=vunl0_15_2).
30/0 23:54:32.398 INF   TAP interface configured (s=26, n=vunl0_15_18).
30/0 23:54:32.398 INF   TAP interface configured (s=28, n=vunl0_15_34).
30/0 23:54:32.398 INF   TAP interface configured (s=30, n=vunl0_15_50).
30/0 23:54:32.398 INF   TAP interface configured (s=32, n=vunl0_15_3).
30/0 23:54:32.398 INF   TAP interface configured (s=34, n=vunl0_15_19).
30/0 23:54:32.398 INF   TAP interface configured (s=36, n=vunl0_15_35).
30/0 23:54:32.398 INF   TAP interface configured (s=38, n=vunl0_15_51).
30/0 23:54:32.398 INF   TAP interface configured (s=40, n=vunl0_15_4).
30/0 23:54:32.398 INF   TAP interface configured (s=42, n=vunl0_15_20).
30/0 23:54:32.398 INF   TAP interface configured (s=44, n=vunl0_15_36).
30/0 23:54:32.398 INF   TAP interface configured (s=46, n=vunl0_15_52).
30/0 23:54:32.398 INF   TAP interface configured (s=48, n=vunl0_15_5).
30/0 23:54:32.398 INF   TAP interface configured (s=50, n=vunl0_15_21).
30/0 23:54:32.398 INF   TAP interface configured (s=52, n=vunl0_15_37).
30/0 23:54:32.398 INF   TAP interface configured (s=54, n=vunl0_15_53).
30/0 23:54:32.398 INF   TAP interface configured (s=56, n=vunl0_15_6).
30/0 23:54:32.398 INF   TAP interface configured (s=58, n=vunl0_15_22).
30/0 23:54:32.398 INF   TAP interface configured (s=60, n=vunl0_15_38).
30/0 23:54:32.398 INF   TAP interface configured (s=62, n=vunl0_15_54).
30/0 23:54:32.398 INF   TAP interface configured (s=64, n=vunl0_15_7).
30/0 23:54:32.398 INF   TAP interface configured (s=66, n=vunl0_15_23).
30/0 23:54:32.398 INF   TAP interface configured (s=68, n=vunl0_15_39).
30/0 23:54:32.398 INF   TAP interface configured (s=70, n=vunl0_15_55).
30/0 23:54:32.398 INF   Adding subprocess stdout descriptor (5).
30/0 23:54:32.398 INF   Adding telnet socket descriptor (7).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (8).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (16).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (24).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (32).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (40).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (48).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (56).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (64).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (10).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (18).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (26).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (34).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (42).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (50).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (58).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (66).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (12).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (20).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (28).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (36).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (44).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (52).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (60).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (68).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (14).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (22).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (30).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (38).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (46).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (54).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (62).
30/0 23:54:32.398 INF   Adding TAP interface descriptor (70).
30/0 23:54:35.399 ERR   Error while connecting local AF_UNIX: Connection refused (2)
30/0 23:54:35.399 ERR   Cannot listen at AF_UNIX (240). ERR: Cannot open AF_UNIX sockets (2).
30/0 23:54:35.399 ERR   Failed to create AF_UNIX socket file (2).
30/0 23:54:35.399 INF   Caught SIGTERM, killing child.
30/0 23:54:35.399 INF   Child is no more running.

This is as far as I've been able to track the issue down so I'm hoping for some direction on the following errors.
30/0 23:54:35.399 ERR Error while connecting local AF_UNIX: Connection refused (2)
30/0 23:54:35.399 ERR Cannot listen at AF_UNIX (240). ERR: Cannot open AF_UNIX sockets (2).
30/0 23:54:35.399 ERR Failed to create AF_UNIX socket file (2).

Thanks!

[Uldis (UD)]

Try other image and give more RAM.
Like 2048 for L2

But it is image issue..
Some older IOL were very unstable and sensible for heavy configs, specially VRFs

[zillow32]

Will give the more ram a try. Is there a newer more stable L2 image you recommend?

[Uldis (UD)]

YES, L2 IRON from 09 Aug, 2017 is very good
search arround, google

Here is not allowed share anything like images...

UD

[zillow32]

No crashing with the same config on high iron. Thanks!

[dennis_alfaris]

Hi

I have kind of similar issue. After setting up the eve-ng as vm in esxi and i put the iol images and licenses in the correct path. But iol are stopping immmediatly after starting.