Page 1 of 4

Revolution Pi Crash

Posted: 21 Mar 2018, 11:10
by f.fallon
Hi,

In the last few days our revolution pi has crashed a couple of times (2 times in the last week) I have checked the messages and kernal log files and have not found anything obvious, could you help me troubleshoot the cause?

Kind Regards,
Frank.

Re: Revolution Pi Crash

Posted: 21 Mar 2018, 18:14
by mezz
I am experiencing an analogous issue with a RevPi core 3 I received a month ago.
At times it shuts down by itself and all LEDs (revpi and attached modules) are red, so in shutdown/fault mode I guess.
There was no custom software running at fault time the first time the issue appeared and the revpi was freshly unpacked and all updates installed.
Everything had been working with custom software on a RevPi core 1 for months before without a single crash, same power supply and IO modules.

I have noticed that a non-closed remote ssh login seems to cause more frequent crashes.

After the the system was correctly powered through a UPS and earthed the issue appeared again, a normal raspberrypi connected on the same line did not report any shutdown events or faults, so the main line (UPS) is stable.
This time an additional program was running but was constantly profiled and the logs are written to a mounted external USB flash device. The CPU has an average of 2.5% usage per core and RAM is available.
There were no hardware changes, I just left some screen sessions open for testing.
I have analysed the kern.log, auth.log, checked the last logins and nothing strange shows up, I still have them in case.
As said a RevPi core 1 with the same Jessy image and updates had not had any issues with modules attached.
This last crash was on 15th March at about 8:30am guessing by the missing cron events in auth.log. The system has been running since and I am constantly monitoring it.

One suspicious entry was this:

Code: Select all

Mar 11 22:06:29 RevPi4858 kernel: [123702.816477] piControl: too many communication errors -> set inputs to default 0 2 2 3   3 3 0 0
But it happened to appear days before the last crash. The annoying part is that this has happened 4 times now and there is no obvious error source.

Any suggestions as to why the system shuts down under normal conditions? all status LEDs of the RevPi core 3 and of the connected modules are red, as if it had been shut down via software, a full power loss and a reboot would have restarted the system correctly.
Could it be the 24V power source? But there is a UPS and the other pieces of hardware don't go down and it has been running for 6 days now (sign of power stability).

Enrico

Re: Revolution Pi Crash

Posted: 21 Mar 2018, 21:59
by volker
I'm sorry to hear that you both have experienced trouble . The facts you have reported do not give any clue why your system acts as it does. We could try finding out reasons but this will be a difficult task to be done by email and forum. All I can offer is the following:
1) Please do download the current production image (if you have not saved your original image) and upload it to the device. Then let your system run without any additions and applications but configured with PiControl in a proper way. If the system runs stable then you might have a trouble with your application or system setup parameters.
2) Get in contact with me by PM so we can exchange details and mail addresses so our experts could may be tell you which logs we need to analyze the crash from a deeper point of view.
3) If you do not want to invest the time for 1) and 2) then simply send back the device and we will replace it and hope that things will get working with the replacement.

As we do have thousands of systems out there running without trouble I can only guess that either any kind of user application or data or system setup causes the trouble or your individual RevPi hardware has some kind of trouble which was not detected by our product end tests. In case a) number 1) would help and in case b) number 3) will help. So please choose whatever you think is the most likely way to solve your problems without waisting too much time.

Re: Revolution Pi Crash

Posted: 22 Mar 2018, 09:56
by f.fallon
mezz wrote: 21 Mar 2018, 18:14 I am experiencing an analogous issue with a RevPi core 3 I received a month ago.
At times it shuts down by itself and all LEDs (revpi and attached modules) are red, so in shutdown/fault mode I guess.
There was no custom software running at fault time the first time the issue appeared and the revpi was freshly unpacked and all updates installed.
Everything had been working with custom software on a RevPi core 1 for months before without a single crash, same power supply and IO modules.

I have noticed that a non-closed remote ssh login seems to cause more frequent crashes.

After the the system was correctly powered through a UPS and earthed the issue appeared again, a normal raspberrypi connected on the same line did not report any shutdown events or faults, so the main line (UPS) is stable.
This time an additional program was running but was constantly profiled and the logs are written to a mounted external USB flash device. The CPU has an average of 2.5% usage per core and RAM is available.
There were no hardware changes, I just left some screen sessions open for testing.
I have analysed the kern.log, auth.log, checked the last logins and nothing strange shows up, I still have them in case.
As said a RevPi core 1 with the same Jessy image and updates had not had any issues with modules attached.
This last crash was on 15th March at about 8:30am guessing by the missing cron events in auth.log. The system has been running since and I am constantly monitoring it.

One suspicious entry was this:

Code: Select all

Mar 11 22:06:29 RevPi4858 kernel: [123702.816477] piControl: too many communication errors -> set inputs to default 0 2 2 3   3 3 0 0
But it happened to appear days before the last crash. The annoying part is that this has happened 4 times now and there is no obvious error source.

Any suggestions as to why the system shuts down under normal conditions? all status LEDs of the RevPi core 3 and of the connected modules are red, as if it had been shut down via software, a full power loss and a reboot would have restarted the system correctly.
Could it be the 24V power source? But there is a UPS and the other pieces of hardware don't go down and it has been running for 6 days now (sign of power stability).

Enrico
Enrico, sorry but could you please create your own thread and not hijack mine please. If you were trying to contribute to my post I apologise but I don't see that anywhere?

Re: Revolution Pi Crash

Posted: 22 Mar 2018, 09:59
by f.fallon
volker wrote: 21 Mar 2018, 21:59 I'm sorry to hear that you both have experienced trouble . The facts you have reported do not give any clue why your system acts as it does. We could try finding out reasons but this will be a difficult task to be done by email and forum. All I can offer is the following:
1) Please do download the current production image (if you have not saved your original image) and upload it to the device. Then let your system run without any additions and applications but configured with PiControl in a proper way. If the system runs stable then you might have a trouble with your application or system setup parameters.
2) Get in contact with me by PM so we can exchange details and mail addresses so our experts could may be tell you which logs we need to analyze the crash from a deeper point of view.
3) If you do not want to invest the time for 1) and 2) then simply send back the device and we will replace it and hope that things will get working with the replacement.

As we do have thousands of systems out there running without trouble I can only guess that either any kind of user application or data or system setup causes the trouble or your individual RevPi hardware has some kind of trouble which was not detected by our product end tests. In case a) number 1) would help and in case b) number 3) will help. So please choose whatever you think is the most likely way to solve your problems without waisting too much time.
Hi Volker,

Do you have a thread or guide to flash the current production image to the device? I think I will try it from a blank image and check does it crash then. If it still does I will contact you via PM and we can try troubleshoot further.

Kind Regards,
Frank.

Re: Revolution Pi Crash

Posted: 22 Mar 2018, 10:51
by volker
Hi Frank,
there is a tutorial about that:
https://revolution.kunbus.com/tutorials ... ng-images/
To download the current installation image please register in our shop and then go to "software" and "Jessie" and you will find a link to download the image.
Once you have downloaded and installed the image during the very first boot sequence (and only once) you need to input the serial number and MAC address printed on the device (on the front, serial number is the middle number of the 3 numbers printed right of the QR code) to give the system a chance to individualize.
Thanks for your patience and please contact me by PM if things do not work out well with the original image.
Good luck!

Re: Revolution Pi Crash

Posted: 22 Mar 2018, 12:20
by mezz
Hi,

Frank I meant to contribute with some more information since my RevPi raised the issue immediately after the unpacking and updating and I have been investigating it for a while.
My contribution was meant to provide help for both and not hijack the attention to two different threads, apologies if you felt otherwise, I won't barge in any more.

Volker, I did flash the original image, will try to reinstate it and replace the RevPi 3 with the old one in the mean time, I will send it back in case.

Enrico

Re: Revolution Pi Crash

Posted: 22 Mar 2018, 13:02
by volker
Hi Enrico,
when sending it back please do me a favor and add a little comment that Volker has asked you to do so. otherwise things might get in wrong hands and will thus take longer as necessary.

Re: Revolution Pi Crash

Posted: 22 Mar 2018, 14:32
by f.fallon
mezz wrote: 22 Mar 2018, 12:20 Hi,

Frank I meant to contribute with some more information since my RevPi raised the issue immediately after the unpacking and updating and I have been investigating it for a while.
My contribution was meant to provide help for both and not hijack the attention to two different threads, apologies if you felt otherwise, I won't barge in any more.

Volker, I did flash the original image, will try to reinstate it and replace the RevPi 3 with the old one in the mean time, I will send it back in case.

Enrico
Hi Enrico,

Ah I understand sorry that was not obvious to me my apologies. So after unpacking and updating you get crashing issues as well?

Kind Regards,
Frank.

Re: Revolution Pi Crash

Posted: 22 Mar 2018, 17:02
by volker
If you don't mind you could both explain your hw configuration in more detail: Which modules are you using? How is the wiring (best would be to post a foto of this)?
And also: have you updated your system with apt upgrade during the last 4 weeks or are you running the original image's kernel?