Revolution Pi Crash

[mezz]

Hi,

Frank, yes the first time it happened the revpi 3 was freshly unpackaged, the original image had been flashed.
I just opened an ssh session, launched an "apt-get update, apt-get upgrade", then I left the ssh window open without closing the client session (and perhaps Pictory browser page), after some hours the revpi was down (red LED).

The second time I think I had just installed the "screen" tool, connected via ssh, opened a screen session and the issue appeared again after some hours.

I thought it was due the power supply, so I added the UPS, 3 DIO modules and 1 DI module and I made sure not ssh sessions (screen or terminal) were running by logging out every time. It ran longer, probably 3-4 days but then it crashed again.
All modules had RED leds too.

Then another crash on 15.03, I think then I did an update for Pictory (viewtopic.php?f=17&t=646) and the system has been running since.
I took a snapshot of the logs and analysed them all finding nothing strange except the picontrol line I posted.
Since last time I have kept a couple of screen sessions detached for testing purposed and a program of mine is running taking up 2,7% of the CPU.
I am monitoring and profiling this piece of software and writing the logs to an external USB flash drive, there are no program error logs, no kernel error logs.
No outages so far, but I am bracing myself for the next one since it seems weird to me that a Pictory update makes a difference.

I'll try to take a picture and post more details.

Regards

Enrico

[volker]

Hi Enrico,
so if I take your description in detail I assume that in all cases there had already been an upgrade to the new kernel version, right? I'm asking because there is a slight suspicion that the new kernel version might be unstable with our kernel driver piControl. It is hard to reproduce and therefore it has not really been proofed but as i said there is a slight suspicion that this could be the case. So if possible I would ask you to take the original kernel and only update PiCtory but not everything (including the kernel). We are actually hard working to find out if there is really a problem with the new kernel and piControl. It will take some time...

[mezz]

Hi Volker,

the current kernel version of the RevPi 3 is (uname -a)
Linux RevPi4858 4.4.50-rt66-v7+ #1 SMP PREEMPT RT Wed, 20 Sep 2017 07:09:15 +0200 armv7l GNU/Linux

The same kernel on a RevPi 1 does not have issues (Linux RevPi100102 4.4.50-rt66+ #1 PREEMPT RT Wed, 20 Sep 2017 07:09:15 +0200 armv6l GNU/Linux).

How shall I proceed? do I need to flash the whole system to the factory image and update Pictory or shall I try to downgrade the kernel to a specific version, any suggestion how I can do that and where I can find the kernel in this case?
The reason being that I need to set up everything again from scratch, it doesn't take long though.

Thanks

Enrico

[volker]

please try and flash the original image and only upgrade the PiCtory package... we'll see if that changes anything...

[f.fallon]

Hi,

Replaced with a brand new revolution pi, same results it crashed sometime last night but the good thing is I have something on the screen that might help with determining the cause.

Photo of the screen:
https://imgur.com/a/8Rmcd

I have also attached the logs.

Kind Regards,
Frank.

[mezz]

Hi,

if it helps, this is the configuration that went completely "red" with the revpi3.

https://imgur.com/jpVWSQx

But the issue appeared also without any modules, usb etc. attached.

I have replaced the revpi3 with a revpi1 in the meantime to rule out the power supply is causing the issue. On a side note this revpi1 configuration is taking up plenty of CPU, I don't know if this is normal, this revpi1 has all the latest updates (apt-get upgrade).
Below a "top" in case you see something suspicious with the processes with the latest updates.

https://imgur.com/dtlSfav

revpid is the only extra program of mine, the rest belongs to the system default.

Cheers
Enrico

[f.fallon]

Hi,

if it helps, this is the configuration that went completely "red" with the revpi3.

https://imgur.com/jpVWSQx

But the issue appeared also without any modules, usb etc. attached.

I have replaced the revpi3 with a revpi1 in the meantime to rule out the power supply is causing the issue. On a side note this revpi1 configuration is taking up plenty of CPU, I don't know if this is normal, this revpi1 has all the latest updates (apt-get upgrade).
Below a "top" in case you see something suspicious with the processes with the latest updates.

https://imgur.com/dtlSfav

revpid is the only extra program of mine, the rest belongs to the system default.

Cheers
Enrico

Interesting, Mine stays green when it crashes,I get no red LED's. I just put in a brand new Rev Pi 3 yesterday and installed our software on it. It crashed sometime over night (no logs indicated our software caused the crash). I am now leaving it over the weekend without our software running to see if it still crashes.

Kind Regards,

[f.fallon]

Update, no crash with our software not running. Have gotten onto our software vendor as to the cause.

Kind Regards,
Frank.

[mezz]

Hi,

I need to report news in the opposite direction unfortunately. A premise, I haven't flashed the revpi back to the factory settings because I wanted to make sure the power source was not the problem.

Frank's issue seems solved and of different nature, let me know if I should move the conversation to a different thread at this stage.

I set up a simple environment with the revpi3 and a DIO module and a different power supply (all new, forgive the temporary wiring), stopped any custom software and I still got several crashes trying to ssh the revpi, so several logins and logouts.
This time I have enabled syslog journalling storage and I will try to collect more information before I flash everything back.

This is my "red" state:

https://imgur.com/ZIeS8IX

The power led of the DIO module goes red, the revpi3 stays green but irresponsive. My configuration:
- revpi3 with the latest apt update packages
- a modbus master in Pictory
- a DIO module
- no custom software, I just installed screen
- no UI attached or X running. only networking
- the issue happens apparently more frequently when repeatedly accessing the revpi3 with an ssh session, but happens every now and then while the system is running with no ssh sessions open
- no kern error logs
- once the system crashes, the system heats up apparently, below a snapshot of the temperature after a crash
https://imgur.com/GFod81P
- a "last -x" shows the crash:
pi@RevPi4858:~ $ last -x

Code: Select all

pi       pts/0        itvic-eet-debian Sun Mar 25 21:20   still logged in   
runlevel (to lvl 3)   4.4.50-rt66-v7+  Sun Mar 25 21:19 - 21:31  (00:11)    
reboot   system boot  4.4.50-rt66-v7+  Sun Mar 25 21:19 - 21:31  (00:11)    
pi       pts/0        itvic-eet-debian Sun Mar 25 19:00 - crash  (02:19)    
pi       pts/0        itvic-eet-debian Sun Mar 25 18:57 - 18:59  (00:01)    

- attacched the logs

The revpi1 on the same live system works flawlessly up till now, so there seems to be definitely something wrong with my revpi3.

I am going to try to collect some syslogs if I manage, otherwise try to flash the image and test it for say two weeks without manual intervention and ping back the results.

Regards

Enrico

[f.fallon]

Hi,

I need to report news in the opposite direction unfortunately. A premise, I haven't flashed the revpi back to the factory settings because I wanted to make sure the power source was not the problem.

Frank's issue seems solved and of different nature, let me know if I should move the conversation to a different thread at this stage.

I set up a simple environment with the revpi3 and a DIO module and a different power supply (all new, forgive the temporary wiring), stopped any custom software and I still got several crashes trying to ssh the revpi, so several logins and logouts.
This time I have enabled syslog journalling storage and I will try to collect more information before I flash everything back.

This is my "red" state:

https://imgur.com/ZIeS8IX

The power led of the DIO module goes red, the revpi3 stays green but irresponsive. My configuration:
- revpi3 with the latest apt update packages
- a modbus master in Pictory
- a DIO module
- no custom software, I just installed screen
- no UI attached or X running. only networking
- the issue happens apparently more frequently when repeatedly accessing the revpi3 with an ssh session, but happens every now and then while the system is running with no ssh sessions open
- no kern error logs
- once the system crashes, the system heats up apparently, below a snapshot of the temperature after a crash
https://imgur.com/GFod81P
- a "last -x" shows the crash:
pi@RevPi4858:~ $ last -x

Code: Select all

pi       pts/0        itvic-eet-debian Sun Mar 25 21:20   still logged in   
runlevel (to lvl 3)   4.4.50-rt66-v7+  Sun Mar 25 21:19 - 21:31  (00:11)    
reboot   system boot  4.4.50-rt66-v7+  Sun Mar 25 21:19 - 21:31  (00:11)    
pi       pts/0        itvic-eet-debian Sun Mar 25 19:00 - crash  (02:19)    
pi       pts/0        itvic-eet-debian Sun Mar 25 18:57 - 18:59  (00:01)    

- attacched the logs

The revpi1 on the same live system works flawlessly up till now, so there seems to be definitely something wrong with my revpi3.

I am going to try to collect some syslogs if I manage, otherwise try to flash the image and test it for say two weeks without manual intervention and ping back the results.

Regards

Enrico

Hi Enrico,

The command last -x is exactly what I needed! I now know when the crash occured.

https://imgur.com/a/C3d27

I am wondering is it a temperature problem? The revpi device is rated for 55 degrees C, I was logging earlier before the crash and it reached 52 degrees C. What happens if the temperature reach 55 degrees C, does it shutdown the device? @volker can you chime in on this?