Problems with security upgrade

[tor]

Last week, 8th of November, we released a security upgrade from Debian upstream to OPI and Keep.

It seems like some systems failed to perform this upgrade leaving the units inaccessible. We are currently investigating this. If you are affected by this please contact OpenProducts support at support@openproducts.com.

/Tor

[tor]

An update on this issue.

It seems that some OPIs did run out of storage during the upgrade and thus entered a failed state. This explains why we unfortunately missed this during testing when we mainly test on newly installed units.

We are currently investigating more in detail on causes of this and working on a solutions to the problem.

/Tor

[DarS]

Hello,
I am afraid I could get hit this error just now. Three days ago (Friday) all of the sudden the mail server started to malfunction, with 'no memory' errors. Self-addressing mail sending attempt ended with:

Code: Select all

SMTP Error (452): Failed to set sender "darek@mail..." (4.3.1 Insufficient system storage).

Strange, because the SD card was far from being full: 9.2 GB / 13.8 GB.

I tried to debug this further. The device was shut down, the SD card removed and checked on Linux notebook - everything looked fine, no signs of fs errors.
So this afternoon the device was turned ON again (with the SD card in it, of course), and the presence of the 'no memory' error was re-checked (still present), but ... New error just appeared in the admin GUI

Code: Select all

Date 2019-12-01 20:14 
Upgrade failed, log in /var/log/dist-upgrade.log. Please run upgrade manually from command line.

Are these two issues related and interconnected? How can I troubleshoot them further?
Regards,
-Darek

[DarS]

Yeah, it looks that my system is really running out of memory.
I managed to access the device via ssh, and this is what I saw:

Code: Select all

df -h
Filesystem       Size  Used Avail Use% Mounted on
udev             231M     0  231M   0% /dev
tmpfs             50M  2.1M   48M   5% /run
/dev/mmcblk1p2   1.7G  1.6G   37M  98% /
tmpfs            247M     0  247M   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
tmpfs            247M     0  247M   0% /sys/fs/cgroup
/dev/mapper/opi   15G  9.3G  4.6G  67% /var/opi

The on-board mmcblk1p2 has almost no space left.
Regards,
-Darek

[tor]

Hi Darek,

Sorry that you also got struck by this problem. Since you have ssh access lets see if we can fix this.

Start by trying to free up some space by issuing:

Code: Select all

apt autoclean

and then

Code: Select all

apt autoremove

For most users this have managed to get the upgrade to complete. If not, contact us at support@openproducts.com.

To check the system after this try with

Code: Select all

dpkg -l | grep -v ^ii

This should now hopefully not list any half installed packages, ie. status iU
Also check the installed kernels by issuing

Code: Select all

dpkg -l | grep linux-image

If installed correctly it should show something like

Code: Select all

ii  linux-image-4.9.0-11-armmp     4.9.189-3+deb9u1               armhf        Linux 4.9 for ARMv7 multiplatform compatible SoCs
ii  linux-image-4.9.0-6-armmp      4.9.88-1op1                    armhf        Linux 4.9 for ARMv7 multiplatform compatible SoCs
ii  linux-image-4.9.0-9-armmp      4.9.168-1+deb9u2               armhf        Linux 4.9 for ARMv7 multiplatform compatible SoCs
ii  linux-image-armmp              4.9+80+deb9u9                  armhf        Linux for ARMv7 multiplatform compatible SoCs (meta-package)

If it does show all ii as status, restart the system to make sure you run the latest kernel and then remove the two old ones by issuing

Code: Select all

apt purge linux-image-4.9.0-6-armmp

and then

Code: Select all

apt purge linux-image-4.9.0-9-armmp

This should hopefully free up some 200MB leaving the system fully operational again.

Should something go wrong during this please get in contact with us using the support email.

/Tor

[DarS]

Yeah, I wanted to be a smartie ... I didn't focused on clearing cache or obsolete updates, just thought about manually finding the largest files and removing them temporarily out of the crowded file system. And yeah, I did it... I managed to locate the large size files. The initial root file system images (initrd) were of course among the biggest:

Code: Select all

-rw-r--r--  1 root root  2'973'529 Sep 20 11:03 System.map-4.9.0-11-armmp
-rw-r--r--  1 root root  2'963'628 May  4  2018 System.map-4.9.0-6-armmp
-rw-r--r--  1 root root  2'970'976 May 13  2019 System.map-4.9.0-9-armmp
-rw-r--r--  1 root root 17'091'145 May 14  2018 initrd.img-4.9.0-6-armmp
-rw-r--r--  1 root root 17'460'453 Jun  6 01:24 initrd.img-4.9.0-9-armmp

Seeing the versions and dates, I thought I don't need 'initrd.img-4.9.0-6-armmp' anymore (ver. 4.9.0-9 was present anyway, and 4.9.0-11 was coming). So it was moved to SD card. Now my device is dead, with just one green LED lit.
I guess I will need to re-flash the device again (assuming the 'opi-kgp-19.06.img' install image will work). Or otherwise try to restore the removed image (ISP ? any other way of accessing internal flash ?).
Gosh, hard lesson, but still a lesson.

Regards!
-Darek

PS. I did not realize, how resource-hungry are some applications. 214MB just for 'nextcloud': 43MB for it's core and ... 113MB for it's apps. If you're on the modern Linux desktop (not mentioning the server), these are peanuts. On embeded, every MB counts. Congratulation for your mastery in keeping everything in a small footprint!

[tor]

Hi Darek,

If the system does not boot you in principal only have two options, either to reinstall the system as you suggest, or convert an installer image to a rescue device and then use that to chroot into the installed system and reinstall the kernel image. The latter however is on the advanced scale of operation but if you are interested i could give advice on that.

/Tor

[DarS]

Hi Tor,
Yeah, my attempts to revitalize the OPI failed so far. I tried both:
- Kinguard image installation
- OPI rescue image
Unfortunately no joy. One LED goes on for a moment, but then no sign of further activity.
I managed to connect the serial console to read the boot messages. See below what was shown...
I had to keep the POWER button (on PCB) pressed at poweron, otherwise OPI just seemed not to notice the presence of SD card.
The rescue image was the most successful so far. Comparing the U-Boot dates (2018 vs 2014) it is visible that the boot loader from the rescue SD card was executed. But then it complained Unable to read file uEnv.txt and ceased it's operations. No further messages on the serial console.

How can I tailor the rescue image to be useful for further debugging? Can you help me with crafting proper uEnv.txt file to have at least a working U-Boot console?

Best regards,
-Darek

A. ATTEMPT TO BOOT OPI WITHOUT ANY SD CARD INSERTED

Code: Select all

U-Boot SPL 2018.01 (Mar 07 2018 - 11:50:50)
Trying to boot from MMC2
*** Warning - bad CRC, using default environment

reading u-boot.img
reading u-boot.img

U-Boot 2018.01 (Mar 07 2018 - 11:50:50 +0100)
CPU  : AM335X-GP rev 2.1
I2C:   ready
DRAM:  512 MiB

B. ATTEMPT TO BOOT WITH OPI RESCUE IMAGE ON SD CARD

Code: Select all

U-Boot 2014.01 (Jun 16 2014 - 23:48:24)
I2C:   ready
DRAM:  512 MiB
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Using default environment
Net:   <ethaddr> not set. Validating first E-fuse MAC
cpsw, usb_ether
Hit any key to stop autoboot:  0 
mmc0 is current device
SD/MMC found on device 0
reading uEnv.txt
** Unable to read file uEnv.txt **

C. ATTEMPT TO BOOT WITH KINGUARD IMAGE ON SD CARD

Code: Select all

U-Boot SPL 2018.01 (Mar 07 2018 - 11:50:50)
Trying to boot from MMC1
*** Warning - bad CRC, using default environment

reading u-boot.img
reading u-boot.img

U-Boot 2018.01 (Mar 07 2018 - 11:50:50 +0100)

CPU  : AM335X-GP rev 2.1
I2C:   ready
DRAM:  512 MiB

[tor]

Hi DarS,

Sorry for the long delay in communication. (I have been busy and on vacation during christmas.)

A pity that you couldn't recover your unit A first question, do the log output all end this early? That looks really strange.

If that is all the output you get could you try a forced reinstall and see if that makes any difference?

Best Regards,

/Tor