No communication between Connect+ and DIO Module after distribution upgrade to bullseye

[energiemanagement]

Hi,
I upgraded my 4year old Kunbus RevolutionPi Connect+ (8GB) manually from stretch to buster. And afterwards from buster to bullseye. After restarting the system I can't access the DIO Module anymore. I got the following errors (note I removed lines in between for better readability, see attached file for full dmesg output):

Code: Select all

[   10.600913] pi-bridge serial0-0: receive head error in io-req
[   10.600935] piControl: DIO addr 31: communication failed (req:6,ret:-5)
[   14.630231] pi-bridge serial0-0: receive head error in io-req
[   16.481912] pi-bridge serial0-0: receive crc error in io-req
[   16.481934] piControl: DIO addr 31: communication failed (req:6,ret:-5)
[   16.494485] pi-bridge serial0-0: failed to discard 25 bytes within timeout
[   16.494510] pi-bridge serial0-0: received packet truncated (25 bytes missing)
[   16.494518] piControl: DIO addr 31: communication failed (req:6,ret:-74)
[   16.764784] pi-bridge serial0-0: receive head error in io-req
[   16.764803] piControl: DIO addr 31: communication failed (req:6,ret:-5)
[   19.745151] pi-bridge serial0-0: receive head error in io-req
[   19.748290] pi-bridge serial0-0: invalid checksum (expected: 0xb9, got 0x99
[   20.337827] pi-bridge serial0-0: failed to discard 25 bytes within timeout

I do not believe that this is a hardware problem, because everything runs fine before the distribution upgrade. I am aware that I may have done some mistakes, when doing the distribution upgrade (for example the default dist-upgrade path has removed systemd-timesyncd and broke the rtc, which I needed to fix manually) . So I hope that I simply run some wrong/old packages.

Note also, that unfortunately it is not an option to flash a new default image. We have a ton of systems and I need to find an upgrade path without physically accessing the devices.
Any Ideas what I can do to isolate/fix the problem?

Here is what I already did:

Code: Select all

>piTest -f
firmware is up to date: 1.5 >= 1.5

Unpowered connect+ and DIO module for 5minutes and restartet it again

Code: Select all

> piTest -d
Found 2 devices:

Address: 0 module type: 105 (0x69) RevPi Connect V1.0
Module is present
     input offset: 113 length: 6
    output offset: 119 length: 5

Address: 31 module type: 96 (0x60) RevPi DIO V1.5
Module is present
     input offset: 0 length: 70
    output offset: 70 length: 18

I am up to date with the following repos:

Code: Select all

deb http://raspbian.raspberrypi.org/raspbian/ bullseye main contrib non-free rpi
deb http://archive.raspberrypi.org/debian/ bullseye main
deb http://packages.revolutionpi.de/ bullseye main contrib
deb http://packages.revolutionpi.de/ bullseye-backports main contrib

[Update]

Code: Select all

> cat revpi/image-release
2019-03-14-revpi-stretch.img

>uname -a
Linux oemsbox-100000001 6.1.46-rt13-v7 #1 SMP PREEMPT_RT Tue, 28 Nov 2023 11:37:39 +0100 armv7l GNU/Linux

The "2019-03-14-revpi-stretch.img" was probably the image base before applying the distribution update.

[nicolaiB]

Could you please share the complete kernel log in order to check if there are any errors which might help the search? Also please share the /boot/config.txt

Nicolai

[energiemanagement]

Hello Nicolai,
here are some boot params

Code: Select all

>cat config.txt |grep -v "^#"|awk NF
hdmi_force_hotplug=1
hdmi_drive=2
dtparam=audio=on
dtoverlay=revpi-connect
dtparam=eth0_mac_hi=0xc83ea701
dtparam=eth0_mac_lo=0x4a07
dtparam=eth1_mac_hi=0xc83ea701
dtparam=eth1_mac_lo=0x4a08

>cat cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 fsck.repair=yes rootwait nosplash plymouth.ignore-serial-consoles ipv6.disable=1

I also attached a kern.log from a new reboot.
Thanks for your support.
Chris

[p.rosenberger]

Hi,

this looks pretty strange. Could you temporarily flash an up to date image?

1. create an image from your device
2. flash a current image
3. test if this also happens with this image
4. reflash your old image

This way we could check if this might be an H/W issue (unlikely, but possible) or if this is a bug in our current software. If it is neither, we need to figure out what is the difference between your image and our current image.

Best regards,
Philipp

[p.rosenberger]

A different idea, we released the 6.1 kernel for RevPi a short while ago. We needed to change how the communication between RevPi and IO modules is implemented. We didn't get any bug reports about it, but you might test the 5.10 kernel.

Code: Select all

sudo apt install raspberrypi-kernel=1:9.20230926+1-5.10.152-1+revpi11+1

If this works, it would be great to get as much information about your setup as possible. So we could recreate this and then fix the issue.

Best regards,
Philipp

[energiemanagement]

Hi Philip,
thanks for the support. I installed the old kernel 5.10.152-rt75-v7 with the command you provided. And it worked. Now the errors are gone and I can communicate with the DATA IO Module. Attached you will find the kernel log from the most recent start (with linux kernel 5.10).

The kernel which produces the error was:

Code: Select all

>apt list --installed|grep raspberrypi-kernel
raspberrypi-kernel/bullseye,now 1:9.20231128+1-6.1.46-1+revpi11+1 armhf [installed]
>uname -a
Linux oemsbox-100000001 6.1.46-rt13-v7 #1 SMP PREEMPT_RT Tue, 28 Nov 2023 11:37:39 +0100 armv7l GNU/Linux

I haven't changed anything else in the system. Could I give you any support on the next steps? E.g. installing kernel 6.1 again and dumping the logs here.

Regards, Chris

[p.rosenberger]

Hi Chris,

thanks for the feedback. It would be great if we could get as much information about your setup as possible. If you don't want to share this in public you can send this to support@kunbus.com. Please put a link to this forum post, then we know it is related to this conversation.

• A full sos-report would be good: https://kunbus-gmbh.atlassian.net/servi ... 2036400208
• details about H/W revision from all devices
• details about your H/W setup including power supply

And anything which might help us to rebuild this setup and reproduce the issue.

Best regards,
Philipp

[dirk]

Hi Chris, have you sent us anything by e-mail yet?