piControl: AIO addr 31: communication failed (req:20,ret:-5)

[nicolaiB]

You cannot use ` rpi-update` as it will install the stock Raspberry Pi kernel over our kernel. With this the kernel driver vor piControl won't work.

The bootloader firmware is part of the package `raspberrypi-bootloader`. Please check the version, it should look like this:

Code: Select all

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                   Version        Architecture Description
+++-======================-==============-============-=================================
ii  raspberrypi-bootloader 1:1.20230405-1 arm64        Raspberry Pi bootloader

[in06khattab]

Thank you for your reply. It looks like the bootloader is indeed older.

Code: Select all

dpkg -l |more
ii  raspberrypi-bootloader            1:1.20230509~buster-1             armhf        Raspberry Pi bootloader

I upgraded the Kernel using the instructions below and vcgencmd is now correct, but the issue still appears.

Download the latest Bullseye image here https://revolutionpi.com/de/tutorials/d ... evpiimages
Follow the instructions here: https://pip.raspberrypi.com/categories/ ... rmware.pdf

Code: Select all

sudo apt-get install --reinstall raspberrypi-kernel 	

Now, vcgencmd is correct

Code: Select all

root@Pi1:~# vcgencmd version
Mar 17 2023 10:50:39
Copyright (c) 2012 Broadcom
version 82f3750a65fadae9a38077e3c2e217ad158c8d54 (clean) (release) (start)

There are some other errors here in bold.

Code: Select all

Apr 29 20:45:05 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-5)
Apr 29 20:45:05 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req
Apr 29 20:45:05 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-74)
[b]Apr 29 20:45:05 Pi1 kernel: pi-bridge serial1-0: invalid checksum (expected: 0xc7, got 0x80[/b]
Apr 29 20:45:05 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-74)
[b]Apr 29 20:45:05 Pi1 kernel: pi-bridge serial1-0: unexpected response addr 0x1e[/b]
Apr 29 20:45:05 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-5)
Apr 29 20:45:05 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req
Apr 29 20:44:56 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-5)
Apr 29 20:44:56 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req
Apr 29 20:44:56 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-5)
Apr 29 20:44:56 Pi1 kernel: pi-bridge serial1-0: received packet truncated (9 bytes missing)
Apr 29 20:44:56 Pi1 kernel: pi-bridge serial1-0: failed to discard 9 bytes within timeout
Apr 29 20:44:56 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-5)
Apr 29 20:44:56 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req
Apr 29 20:44:40 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-5)
Apr 29 20:44:40 Pi1 kernel: pi-bridge serial1-0: receive data error in io-req(len: 20)
Apr 29 20:44:40 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-74)
[b]Apr 29 20:44:40 Pi1 kernel: pi-bridge serial1-0: unexpected response addr 0x1f[/b]
Apr 29 20:44:40 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-5)
Apr 29 20:44:40 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req

Shall I try updating the bootloader with the following commands from https://github.com/evilsocket/pwnagotchi/issues/1175?

Code: Select all

wget https://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-bootloader_1.20230405-1_armhf.deb
dpkg -i ./raspberrypi-bootloader_1.20230405-1_armhf.deb

[in06khattab]

The above commands worked and I have updated the bootloader to 1:1.20230405-1.

The vcgencmd is also up to date.

I am still getting errors in journalctl. I have also noticed that /var/log will get filled up after ~2 days due to the error messages. Then network is affected, it disconnected after too many errors.

Attached is sos report, but after a reboot, so network was ok.

sosreport-Pi1-2024-05-01-fhasqkw.tar

(1.52 MiB) Downloaded 10 times

Example errors:

Code: Select all

May 01 19:33:05 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-5)
May 01 19:33:05 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req
May 01 19:29:26 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-74)
May 01 19:29:26 Pi1 kernel: pi-bridge serial1-0: received packet truncated (9 bytes missing)
May 01 19:29:26 Pi1 kernel: pi-bridge serial1-0: failed to discard 9 bytes within timeout
May 01 19:30:45 Pi1 kernel: pi-bridge serial1-0: receive data error in io-req(len: 20)
May 01 19:30:45 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-74)

Could it be that this is a 32 bit image?

[in06khattab]

I ran piTest -f and the last line could be a clue. The lines are printed in reverse order as I used journalctl -r.

Code: Select all

May 01 21:16:00 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req
May 01 21:15:58 Pi1 kernel: piControl: piAIOComm_Init(30) done 0
May 01 21:15:58 Pi1 kernel: piControl: piAIOComm_Init(31) done 0
May 01 21:15:58 Pi1 kernel: piControl: start data exchange
May 01 21:15:58 Pi1 kernel: piControl: update 3 addr 64 ret -95
May 01 21:15:58 Pi1 kernel: piControl: Virtual modules don't have firmware to update
May 01 21:15:58 Pi1 kernel: piControl: update 2 addr 30 ret 0
May 01 21:15:58 Pi1 kernel: piControl: firmware is up to date: 1.5 >= 1.5
May 01 21:15:58 Pi1 kernel: piControl: firmware file length: 63540
May 01 21:15:58 Pi1 kernel: piControl: update 1 addr 31 ret 0
May 01 21:15:58 Pi1 kernel: piControl: firmware is up to date: 1.5 >= 1.5
May 01 21:15:58 Pi1 kernel: piControl: firmware file length: 63540
May 01 21:15:58 Pi1 kernel: piControl: update 0 addr 0 ret -95
May 01 21:15:58 Pi1 kernel: piControl: RevPi has no firmware to update
May 01 21:15:58 Pi1 kernel: piControl: piIoComm_gotoGateProtocol returned 0
May 01 21:15:58 Pi1 kernel: piControl: stop data exchange
May 01 21:15:58 Pi1 kernel: piControl: Note: ioctl KB_UPDATE_DEVICE_FIRMWARE is deprecated. Use PICONTROL_UPLOAD_FIRMWARE instead

[in06khattab]

Here is the full output of dpkg -l |more.
Hope it helps.

dkpg.7z

(27.62 KiB) Downloaded 11 times

[in06khattab]

Downgrading to kernel 5.10.152 removed the errors with AIO. When upgrading again to the latest kernel, the errors came back.
I am happy to provide an img file of the system.

Code: Select all

sudo apt install raspberrypi-kernel=1:9.20230926+1-5.10.152-1+revpi11+1

[in06khattab]

I think I have figured out the source of the issue. When I delete all rows from Modbus Extended data, the errors go away from journalctl. When I add the rows back in, the errors come back. I tried a brand new Pictory project and as soon as I add all the rows, the errors appear.

Here is the RSC file:

Picotry_V2.zip

(4.93 KiB) Downloaded 13 times

I also noticed that all modbus data returns 0 when the driver is reset (piTest -x) and I receive the following messages in journalctl -r.
I have to restart the pi in order to receive modbus messages.

Code: Select all

May 05 09:39:01 Pi1 CRON[4833]: pam_unix(cron:session): session closed for user root
May 05 09:39:01 Pi1 CRON[4834]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
May 05 09:39:01 Pi1 CRON[4833]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 05 09:17:01 Pi1 CRON[4810]: pam_unix(cron:session): session closed for user root
May 05 09:17:01 Pi1 CRON[4811]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 05 09:17:01 Pi1 CRON[4810]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 05 09:09:10 Pi1 systemd[1]: Finished Clean php session files.
May 05 09:09:10 Pi1 systemd[1]: phpsessionclean.service: Succeeded.
May 05 09:09:10 Pi1 systemd[1]: Starting Clean php session files...
May 05 09:09:01 Pi1 CRON[4748]: pam_unix(cron:session): session closed for user root
May 05 09:09:01 Pi1 CRON[4749]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
May 05 09:09:01 Pi1 CRON[4748]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 05 08:39:10 Pi1 systemd[1]: Finished Clean php session files.
May 05 08:39:10 Pi1 systemd[1]: phpsessionclean.service: Succeeded.
May 05 08:39:09 Pi1 systemd[1]: Starting Clean php session files...
May 05 08:39:01 Pi1 CRON[4649]: pam_unix(cron:session): session closed for user root
May 05 08:39:01 Pi1 CRON[4650]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
May 05 08:39:01 Pi1 CRON[4649]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 05 08:17:01 Pi1 CRON[4624]: pam_unix(cron:session): session closed for user root
May 05 08:17:01 Pi1 CRON[4625]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 05 08:17:01 Pi1 CRON[4624]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 05 08:09:10 Pi1 systemd[1]: Finished Clean php session files.
May 05 08:09:10 Pi1 systemd[1]: phpsessionclean.service: Succeeded.
May 05 08:09:09 Pi1 systemd[1]: Starting Clean php session files...
May 05 08:09:01 Pi1 CRON[4568]: pam_unix(cron:session): session closed for user root
May 05 08:09:01 Pi1 CRON[4569]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
May 05 08:09:01 Pi1 CRON[4568]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 05 07:39:10 Pi1 systemd[1]: Finished Clean php session files.
May 05 07:39:10 Pi1 systemd[1]: phpsessionclean.service: Succeeded.
May 05 07:39:09 Pi1 systemd[1]: Starting Clean php session files...

Code: Select all

pi@Pi1:~ $ uname -a
Linux Pi1 6.1.46-rt13-v7l #1 SMP PREEMPT_RT Mon, 22 Apr 2024 08:37:46 +0200 armv7l GNU/Linux

Errors in journalctl -r

Code: Select all

May 05 12:31:57 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-5)
May 05 12:31:57 Pi1 kernel: pi-bridge serial1-0: receive data error in io-req(len: 20)
May 05 12:31:57 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-74)
May 05 12:31:57 Pi1 kernel: pi-bridge serial1-0: unexpected response addr 0x1f
May 05 12:31:57 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-5)
May 05 12:31:57 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req

Code: Select all

May 05 12:42:33 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-5)
May 05 12:42:33 Pi1 kernel: pi-bridge serial1-0: receive data error in io-req(len: 20)
May 05 12:42:33 Pi1 kernel: piControl: AIO addr 30: communication failed (req:20,ret:-74)
May 05 12:42:33 Pi1 kernel: pi-bridge serial1-0: unexpected response addr 0x1f
May 05 12:42:33 Pi1 kernel: piControl: AIO addr 31: communication failed (req:20,ret:-5)
May 05 12:42:33 Pi1 kernel: piAIOComm_sendCyclicTelegram: 3 callbacks suppressed
May 05 12:42:33 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req
May 05 12:42:22 Pi1 kernel: pi-bridge serial1-0: receive data error in io-req(len: 20)
May 05 12:42:22 Pi1 kernel: pi-bridge serial1-0: unexpected response addr 0x1f
May 05 12:42:22 Pi1 kernel: pi-bridge serial1-0: receive head error in io-req

I think the errors are somehow related to modbus.

[in06khattab]

I think I found another bug in Pictory. When I changed the parity from even to none, saved the project and restarted the driver using the GUI. I then restarted the pi. After opening Pictory again, I noticed the following boxes missing numbers.

All the text boxes that are missing had the number 1 inside.
Browser used was chromium running locally on the RevPi

[nicolaiB]

Thanks for the investigation! We will look into this. If you can provide an image of your system this would be great. Please contact me via private message for details where to send the image to.

Nicolai

[nicolaiB]

I can also see from the SOS report you provided, that the /boot/config.txt contains custom parameters which might interfer with our overlays. Please remove them and check if it will make any difference:

enable_uart=1
dtparam=i2c_arm=on

Nicolai