RevPi Connect+ freezes suddenly
Posted: 06 Sep 2022, 12:23
Good morning!
I'm experiencing sudden hangs on at least one of our RevPi's. First thought was the temperature, but on the core it's not going over 60'C. I made a test with other unit and heat it up over 82'C and everything was fine.
Second thought - RAM. I'm pretty sure that it could cause problems, because it's only 1GB of total memory and I ended up with almost 20MB of free RAM when it did crash. I reduced the amount of memory for GUI to 16MB, change the screen resolution to lowest, enable SWAP, uninstalled TeamViewer. Right now the Pi is using less than 300MB out of available 972MB. Unfortunately it still freezes suddenly after some time (different every time).
After it freezes, the ethernet port seems to work (blinking), but no respond from the Pi (no ping). Once it happened that the A3 LED was still orange like in shutdown mode. Only the gateway LED's are blinking in the way like the console service would be disabled (no communication). The only solution is to turn it off and on again, but after that the log files for CODESYS are cleared.
To sumirize:
1. RevPi Connect+ 32GB with
2. Image 2022-05-25-revpi-buster.img
3. Debian 10.12
4. Kernel 5.10.120-rt70-v7
5. Running CODESYS V3.5 SP18 Patch 2 project with PROFIBUS Gateway - average cycle time 159us, max 800us.
6. Running service which is reading the data from CODESYS via OPC UA and sends them to the cloud
7. Temp. on the core <60'C
8. RAM usage < 300MB
9. CPU usage average 40-50%
10. Updated and upgraded
What can I check to investigate where the problem is? Which log files?
Edit 07.09.2022 - RevPi stucked again
I was connected since yesterday remotely with VNC and CODESYS to this RevPi. It disconnected me somewhere in the night or in the morning, but I connected again with VNC and everything was ok, but after a couple of minutes it disconnected me for good.
User LOG
[Auth LOG
Messages LOG
I'm experiencing sudden hangs on at least one of our RevPi's. First thought was the temperature, but on the core it's not going over 60'C. I made a test with other unit and heat it up over 82'C and everything was fine.
Second thought - RAM. I'm pretty sure that it could cause problems, because it's only 1GB of total memory and I ended up with almost 20MB of free RAM when it did crash. I reduced the amount of memory for GUI to 16MB, change the screen resolution to lowest, enable SWAP, uninstalled TeamViewer. Right now the Pi is using less than 300MB out of available 972MB. Unfortunately it still freezes suddenly after some time (different every time).
After it freezes, the ethernet port seems to work (blinking), but no respond from the Pi (no ping). Once it happened that the A3 LED was still orange like in shutdown mode. Only the gateway LED's are blinking in the way like the console service would be disabled (no communication). The only solution is to turn it off and on again, but after that the log files for CODESYS are cleared.
To sumirize:
1. RevPi Connect+ 32GB with
2. Image 2022-05-25-revpi-buster.img
3. Debian 10.12
4. Kernel 5.10.120-rt70-v7
5. Running CODESYS V3.5 SP18 Patch 2 project with PROFIBUS Gateway - average cycle time 159us, max 800us.
6. Running service which is reading the data from CODESYS via OPC UA and sends them to the cloud
7. Temp. on the core <60'C
8. RAM usage < 300MB
9. CPU usage average 40-50%
10. Updated and upgraded
What can I check to investigate where the problem is? Which log files?
Edit 07.09.2022 - RevPi stucked again
I was connected since yesterday remotely with VNC and CODESYS to this RevPi. It disconnected me somewhere in the night or in the morning, but I connected again with VNC and everything was ok, but after a couple of minutes it disconnected me for good.
User LOG
Code: Select all
Sep 7 05:56:09 RevPi10030476771 vncserver-x11[1646,root]: Connections: connected: 10.242.2.2::51622 (TCP)
Sep 7 05:56:15 RevPi10030476771 vncserver-x11[1646,root]: Connections: authenticated: 10.242.2.2::51622 (TCP), as pi (f permissions)
Sep 7 05:56:15 RevPi10030476771 colord[1640]: failed to get session [pid 1183]: No data available
Sep 7 06:30:40 RevPi10030476771 mtp-probe: checking bus 1, device 7: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.3"
Sep 7 06:30:40 RevPi10030476771 mtp-probe: bus: 1, device: 7 was not an MTP device
Sep 7 06:30:40 RevPi10030476771 mtp-probe: checking bus 1, device 7: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.3"
Sep 7 06:30:40 RevPi10030476771 mtp-probe: bus: 1, device: 7 was not an MTP device
Sep 7 06:30:40 RevPi10030476771 mtp-probe: checking bus 1, device 3: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1"
Sep 7 06:30:40 RevPi10030476771 mtp-probe: checking bus 1, device 6: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.2"
Sep 7 06:30:40 RevPi10030476771 mtp-probe: bus: 1, device: 6 was not an MTP device
Sep 7 06:30:40 RevPi10030476771 mtp-probe: bus: 1, device: 3 was not an MTP device
Sep 7 06:30:40 RevPi10030476771 mtp-probe: checking bus 1, device 5: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.1"
Sep 7 06:30:40 RevPi10030476771 mtp-probe: checking bus 1, device 7: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.3"
Sep 7 06:30:40 RevPi10030476771 mtp-probe: bus: 1, device: 7 was not an MTP device
Sep 7 06:30:40 RevPi10030476771 mtp-probe: bus: 1, device: 5 was not an MTP device
Sep 7 06:30:41 RevPi10030476771 udisksd[1311]: udisks daemon version 2.8.1 starting
Sep 7 06:30:41 RevPi10030476771 udisksd[1311]: failed to load module crypto: libbd_crypto.so.2: cannot open shared object file: No such file or directory
Sep 7 06:30:41 RevPi10030476771 udisksd[1311]: failed to load module mdraid: libbd_mdraid.so.2: cannot open shared object file: No such file or directory
Sep 7 06:30:42 RevPi10030476771 udisksd[1311]: Failed to load the 'mdraid' libblockdev plugin
Sep 7 06:30:42 RevPi10030476771 udisksd[1311]: Failed to load the 'crypto' libblockdev plugin
Sep 7 06:30:43 RevPi10030476771 vncserver-x11[1614,root]: LogDestMem: Start replaying stored messages
Sep 7 06:30:43 RevPi10030476771 vncserver-x11[1614,root]: VendorConfig: Error in Certificate "CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US": X.509 Error: Certificate expired
Sep 7 06:30:43 RevPi10030476771 vncserver-x11[1614,root]: VendorConfig: Error in Certificate "CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R2": X.509 Error: Certificate expired
Sep 7 06:30:43 RevPi10030476771 vncserver-x11[1614,root]: LogDestMem: Finished replaying stored messages
Sep 7 06:30:43 RevPi10030476771 vncserver-x11[1614,root]: ServerManager: Server started
Sep 7 06:30:44 RevPi10030476771 lightdm[1662]: Error getting user list from org.freedesktop.Accounts: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Accounts was not provided by any .service files
Sep 7 06:30:49 RevPi10030476771 colord[1566]: failed to get session [pid 1345]: No data available
Sep 7 06:30:49 RevPi10030476771 vncserver-x11[1614,root]: ConsoleDisplay: Found running X server (pid=1726, binary=/usr/lib/xorg/Xorg)
Sep 7 06:30:49 RevPi10030476771 udisksd[1311]: Acquired the name org.freedesktop.UDisks2 on the system message bus
Sep 7 06:30:50 RevPi10030476771 colord-sane: [bjnp] create_broadcast_socket: ERROR - bind socket to local address failed - Cannot assign requested address
Sep 7 06:30:50 RevPi10030476771 lightdm[2129]: Error getting user list from org.freedesktop.Accounts: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Accounts was not provided by any .service files
Sep 7 06:30:50 RevPi10030476771 vncserver-x11[1614,root]: HostedRendezvous: Rendezvous lookup failed: Hosted Bootstrap error: Network failure: Error connecting: getaddrinfo: 'Temporary failure in name resolution' (-3)
[Auth LOG
Code: Select all
Sep 7 00:09:01 RevPi10030476771 CRON[3205]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 00:09:01 RevPi10030476771 CRON[3205]: pam_unix(cron:session): session closed for user root
Sep 7 00:17:01 RevPi10030476771 CRON[4697]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 00:17:01 RevPi10030476771 CRON[4697]: pam_unix(cron:session): session closed for user root
Sep 7 00:39:01 RevPi10030476771 CRON[9002]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 00:39:01 RevPi10030476771 CRON[9002]: pam_unix(cron:session): session closed for user root
Sep 7 01:09:01 RevPi10030476771 CRON[14912]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 01:09:01 RevPi10030476771 CRON[14912]: pam_unix(cron:session): session closed for user root
Sep 7 01:17:01 RevPi10030476771 CRON[16641]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 01:17:01 RevPi10030476771 CRON[16641]: pam_unix(cron:session): session closed for user root
Sep 7 01:39:01 RevPi10030476771 CRON[20906]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 01:39:01 RevPi10030476771 CRON[20906]: pam_unix(cron:session): session closed for user root
Sep 7 02:09:01 RevPi10030476771 CRON[26858]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 02:09:01 RevPi10030476771 CRON[26858]: pam_unix(cron:session): session closed for user root
Sep 7 02:17:01 RevPi10030476771 CRON[28439]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 02:17:01 RevPi10030476771 CRON[28439]: pam_unix(cron:session): session closed for user root
Sep 7 02:39:01 RevPi10030476771 CRON[32748]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 02:39:01 RevPi10030476771 CRON[32748]: pam_unix(cron:session): session closed for user root
Sep 7 03:09:01 RevPi10030476771 CRON[6305]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 03:09:01 RevPi10030476771 CRON[6305]: pam_unix(cron:session): session closed for user root
Sep 7 03:17:01 RevPi10030476771 CRON[8020]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 03:17:01 RevPi10030476771 CRON[8020]: pam_unix(cron:session): session closed for user root
Sep 7 03:39:01 RevPi10030476771 CRON[12326]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 03:39:01 RevPi10030476771 CRON[12326]: pam_unix(cron:session): session closed for user root
Sep 7 04:09:01 RevPi10030476771 CRON[18360]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 04:09:01 RevPi10030476771 CRON[18360]: pam_unix(cron:session): session closed for user root
Sep 7 04:17:01 RevPi10030476771 CRON[19858]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 04:17:01 RevPi10030476771 CRON[19858]: pam_unix(cron:session): session closed for user root
Sep 7 04:39:01 RevPi10030476771 CRON[24211]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 04:39:01 RevPi10030476771 CRON[24211]: pam_unix(cron:session): session closed for user root
Sep 7 05:09:01 RevPi10030476771 CRON[30075]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 05:09:01 RevPi10030476771 CRON[30075]: pam_unix(cron:session): session closed for user root
Sep 7 05:17:01 RevPi10030476771 CRON[31765]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 05:17:01 RevPi10030476771 CRON[31765]: pam_unix(cron:session): session closed for user root
Sep 7 05:39:01 RevPi10030476771 CRON[3830]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 05:39:01 RevPi10030476771 CRON[3830]: pam_unix(cron:session): session closed for user root
Sep 7 05:56:15 RevPi10030476771 vncserver-x11[1646,root]: session started: user pi permissions f
Sep 7 06:30:40 RevPi10030476771 systemd-logind[1318]: New seat seat0.
Sep 7 06:30:43 RevPi10030476771 sshd[1659]: Server listening on 0.0.0.0 port 22.
Sep 7 06:30:43 RevPi10030476771 sshd[1659]: Server listening on :: port 22.
Sep 7 06:30:49 RevPi10030476771 login[1686]: pam_unix(login:session): session opened for user pi by LOGIN(uid=0)
Sep 7 06:30:49 RevPi10030476771 systemd-logind[1318]: New session 1 of user pi.
Sep 7 06:30:50 RevPi10030476771 systemd: pam_unix(systemd-user:session): session opened for user pi by (uid=0)
Sep 7 06:30:50 RevPi10030476771 lightdm: pam_unix(lightdm-autologin:session): session opened for user pi by (uid=0)
Sep 7 06:30:50 RevPi10030476771 systemd-logind[1318]: New session 3 of user pi.
Sep 7 06:30:53 RevPi10030476771 polkitd(authority=local): Registered Authentication Agent for unix-session:3 (system bus name :1.24 [lxpolkit], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB)
Sep 7 06:31:18 RevPi10030476771 dbus-daemon[1303]: [system] Failed to activate service 'org.bluez': timed out (service_start_timeout=25000ms)
Sep 7 06:34:48 RevPi10030476771 vncserver-x11[1614,root]: session started: user pi permissions f
Sep 7 06:39:01 RevPi10030476771 CRON[2534]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 7 06:39:01 RevPi10030476771 CRON[2534]: pam_unix(cron:session): session closed for user root
Messages LOG
Code: Select all
Sep 7 00:09:01 RevPi10030476771 CRON[3206]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 00:17:01 RevPi10030476771 CRON[4698]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 7 00:39:01 RevPi10030476771 CRON[9003]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 01:09:01 RevPi10030476771 CRON[14913]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 01:17:01 RevPi10030476771 CRON[16642]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 7 01:39:01 RevPi10030476771 CRON[20907]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 02:09:01 RevPi10030476771 CRON[26859]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 02:17:01 RevPi10030476771 CRON[28440]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 7 02:39:01 RevPi10030476771 CRON[32749]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 03:09:01 RevPi10030476771 CRON[6306]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 03:17:01 RevPi10030476771 CRON[8021]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 7 03:39:01 RevPi10030476771 CRON[12327]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 04:09:01 RevPi10030476771 CRON[18361]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 04:17:01 RevPi10030476771 CRON[19859]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 7 04:39:01 RevPi10030476771 CRON[24212]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 05:09:01 RevPi10030476771 CRON[30076]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 05:17:01 RevPi10030476771 CRON[31766]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 7 05:39:01 RevPi10030476771 CRON[3831]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 7 06:30:40 RevPi10030476771 cron[1322]: (CRON) INFO (pidfile fd = 3)
Sep 7 06:30:40 RevPi10030476771 cron[1322]: (CRON) INFO (Running @reboot jobs)
Sep 7 06:30:40 RevPi10030476771 rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.1901.0]
Sep 7 06:30:40 RevPi10030476771 rsyslogd: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="1314" x-info="https://www.rsyslog.com"] start
Sep 7 06:31:08 RevPi10030476771 systemd-timesyncd[1173]: Synchronized to time server for the first time 162.159.200.123:123 (2.debian.pool.ntp.org).
Sep 7 06:39:01 RevPi10030476771 CRON[2535]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)