RevPi Crash

[EcoMan]

Hi,

I'm running a revPi with Linux version 4.19.95-rt38-v7 (gcc version 8.3.0 (Debian 8.3.0-2)) .
The revPi is running a python based program we have written.

I have recently encountered a problem logging in to my revPi. When connecting a micro-hdmi cable, I'm seeing the stack trace attached below.
Cant be sure when it happened, it was definitely in the last 12 hours.

********** EDIT:
I've just realized I haven't supplied much details, so here it is:

• RevPi Core3.

• Connected to network via wifi using a dongle.

• Wasn't sampling any devices during the crash (development RevPi).

Any other details needed ?

The latest kernel log is this :

Code: Select all

Feb 13 22:33:14 00000000006d99bf kernel: rtl8192cu: MAC auto ON okay!
Feb 13 22:33:15 00000000006d99bf kernel: rtl8192cu: Tx queue select: 0x05
Feb 13 22:33:15 00000000006d99bf kernel: Adding 102396k swap on /var/swap.  Priority:-2 extents:1 across:102396k SSFS
Feb 13 22:33:16 00000000006d99bf kernel: smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
Feb 13 22:33:17 00000000006d99bf kernel: wlan0: authenticate with XX:XX:XX:XX:XX
Feb 13 22:33:17 00000000006d99bf kernel: wlan0: send auth to XX:XX:XX:XX:XX(try 1/3)
Feb 13 22:33:18 00000000006d99bf kernel: wlan0: authenticated
Feb 13 22:33:18 00000000006d99bf kernel: wlan0: associate with XX:XX:XX:XX:XX (try 1/3)
Feb 13 22:33:18 00000000006d99bf kernel: wlan0: RX AssocResp from XX:XX:XX:XX:XX (capab=0x431 status=0 aid=7)
Feb 13 22:33:18 00000000006d99bf kernel: wlan0: associated
Feb 13 22:33:35 00000000006d99bf systemd-fstab-generator[769]: Failed to create mount unit file /run/systemd/generator/run-ecobox.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:35 00000000006d99bf systemd-fstab-generator[769]: Failed to create mount unit file /run/systemd/generator/run-ecobox.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:35 00000000006d99bf systemd-fstab-generator[769]: Failed to create mount unit file /run/systemd/generator/var-log.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:35 00000000006d99bf systemd-fstab-generator[769]: Failed to create mount unit file /run/systemd/generator/run-ecobox.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:35 00000000006d99bf systemd-fstab-generator[769]: Failed to create mount unit file /run/systemd/generator/var-log.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:35 00000000006d99bf systemd-fstab-generator[769]: Failed to create mount unit file /run/systemd/generator/run-ecobox.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:35 00000000006d99bf systemd-fstab-generator[769]: Failed to create mount unit file /run/systemd/generator/var-log.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:35 00000000006d99bf systemd[1]: apt-daily-upgrade.timer: Adding 6min 13.892271s random time.
Feb 13 22:33:35 00000000006d99bf systemd[1]: apt-daily.timer: Adding 8h 35min 35.734260s random time.
Feb 13 22:33:35 00000000006d99bf kernel: wlan0: deauthenticating from XX:XX:XX:XX:XX by local choice (Reason: 3=DEAUTH_LEAVING)
Feb 13 22:33:36 00000000006d99bf kernel: rtl8192cu: MAC auto ON okay!
Feb 13 22:33:36 00000000006d99bf kernel: rtl8192cu: Tx queue select: 0x05
Feb 13 22:33:39 00000000006d99bf kernel: rtl8192cu: MAC auto ON okay!
Feb 13 22:33:39 00000000006d99bf kernel: rtl8192cu: Tx queue select: 0x05
Feb 13 22:33:41 00000000006d99bf kernel: wlan0: authenticate with XX:XX:XX:XX:XX
Feb 13 22:33:41 00000000006d99bf kernel: wlan0: send auth to XX:XX:XX:XX:XX (try 1/3)
Feb 13 22:33:41 00000000006d99bf kernel: wlan0: authenticated
Feb 13 22:33:41 00000000006d99bf kernel: wlan0: associate with XX:XX:XX:XX:XX (try 1/3)
Feb 13 22:33:41 00000000006d99bf kernel: wlan0: RX AssocResp from XX:XX:XX:XX:XX (capab=0x431 status=0 aid=7)
Feb 13 22:33:41 00000000006d99bf kernel: wlan0: associated
Feb 13 22:33:49 00000000006d99bf systemd-fstab-generator[920]: Failed to create mount unit file /run/systemd/generator/run-ecobox.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:49 00000000006d99bf systemd-fstab-generator[920]: Failed to create mount unit file /run/systemd/generator/run-ecobox.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:49 00000000006d99bf systemd-fstab-generator[920]: Failed to create mount unit file /run/systemd/generator/var-log.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:49 00000000006d99bf systemd-fstab-generator[920]: Failed to create mount unit file /run/systemd/generator/run-ecobox.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:49 00000000006d99bf systemd-fstab-generator[920]: Failed to create mount unit file /run/systemd/generator/var-log.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:49 00000000006d99bf systemd-fstab-generator[920]: Failed to create mount unit file /run/systemd/generator/run-ecobox.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:49 00000000006d99bf systemd-fstab-generator[920]: Failed to create mount unit file /run/systemd/generator/var-log.mount, as it already exists. Duplicate entry in /etc/fstab?
Feb 13 22:33:49 00000000006d99bf systemd[1]: apt-daily-upgrade.timer: Adding 39.892997s random time.
Feb 13 22:33:49 00000000006d99bf systemd[1]: apt-daily.timer: Adding 10h 41min 5.439792s random time.

Any chance someone here can point me in a direction to figure out what went wrong ?

[Rapanui]

A memory leak in a running python-script similar to the issue in viewtopic.php?f=8&t=2966&p=11854#p11854 ?
Not more than an idea, I'm just a simple user.

[EcoMan]

Thanks for trying to help but doesn't seem like a mem leak. I happen to have a component in my app that monitors that and I'm seeing that mem usage was at most around 27% :\

[EcoMan]

Anyone has any other directions i can explore or any ideas how to handle those crashes

[j.magnano@kunbus.com]

Hello Ecoman,

can it be that the problem existed arleady before you connected the monitor to the RevPi via the HDMI port?

Thanks, Julian

[EcoMan]

Hey, sorry for the late response.
Yes, the revpi was suddenly unavailable via ssh so I've connected the hdmi to see whats wrong, and then saw this trace.

[kjkoster]

Dear EcoMan,

Your kernel log seems to have a few ominous messages about `/etc/fstab`. Can you please post the contents of that file for us to look at? Have you considered file system corruption?

Kees Jan

[EcoMan]

Thats an interesting observation.
I use the fstab to mount my log paths to RAM, so it doesn't wear the storage with tons of read/write cycles.
But I did notice that on some occasion, my mount commands got duplicated a lot. do you think that could have caused the issue?
this is the current content :

Code: Select all

proc            /proc           proc    defaults          0       0
PARTUUID=27cd3d9d-01  /boot           vfat    defaults          0       2
PARTUUID=27cd3d9d-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
# EcoBox Configuration
tmpfs /run/ecobox tmpfs nodev,nosuid 0  0
tmpfs /var/log tmpfs nodev,nosuid,size=50M 0  0

thank you !

[nicolaiB]

Hi EcoMan,

could you please share the output of "df -h"? This command will print the filesystem usage.

Nicolai

[EcoMan]

There it is:

Code: Select all

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.6G  2.6G  815M  77% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M   40K  463M   1% /dev/shm
tmpfs           463M   46M  417M  10% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
tmpfs           463M  516K  463M   1% /run/ecobox
/dev/mmcblk0p1   43M   24M   19M  56% /boot
tmpfs            93M     0   93M   0% /run/user/1000