OPI half-bricked after the upgrade

DarS · Post by **DarS** » Thu Nov 20, 2014 4:52 am

Hello,
I am quite busy last months so I did not notice the OPI automatic upgrade up until this Monday, when I realized issues with ownCloud client on my notebook. It turned out that OPI has been half-bricked after the upgrade.
WHAT WORKS
I can log in into the device (both as 'admin' and as regular 'user') and then use 'Settings' web interface. Here everything seems to work well. I can log out and log in several times, I can reconfigure various parameters, etc. I even enabled 'ssh' service but ... could not log in, because the password sits somewhere in email inside OPI (and it's web mail interface does not work - see next point).

WHAT DOES NOT WORK
Once I log into the device and try to access Calendar, Mail or Files, I see nothing but a rotating circle...

Moreover, ownCloud clients also display signs of errors:

Of course OPI worked fine before the upgrade, so I guess that what I see is the result of the upgrade itself and not misconfiguration of the device.

I just performed simple troubleshooting so far. OPI was rebooted ('nice' MS Windows habit...) once, but this did not help. After the initialization OPI begun to behave the same way as described above. I've checked that ownCloud clients on MS Windows and Linux display similar errors. I did not check e-mail access via IMAP or POP yet - I don't have email clients configured for OPI. But I will if it is needed for troubleshooting

My question: what should be the best way to troubleshoot further my malfunctioning OPI? I wish I could have at least 'ssh' access, things could be much easier then

Regards,
-DarS

Post by **tor** » Thu Nov 20, 2014 9:32 am

Hi DarS,

That was a strange error we have not encountered yet. We have noticed that there where a regression on OC and group management and that there under some circumstances the upgrade of OC could fail.

This however does not explain why your email isn't working.

One possible cause could be that something have stopped MySQL from working as it should.

First off, can you access email some other way than via Web-mail. F.x. with a desktop client such as Thunderbird, outlook, Apple Mail or something similar. That way you should be able to retrieve the ssh password.

If this isn't an option or if this does nor work we could try to use our newly released rescue image. The big limitation with that is that it can only be used to examine and fix the internal storage. Not the external SD-card.

It can be found here: Rescue image 1.0 follow the instructions on (Re)Installing OPI to create the rescue card.

Use the card to boot from. I.e. power down OPI and change SD card to the rescue card and power up OPI. Unfortunately this image wont tell you when its booted. It just "hangs" with the middle led blinking. It should however be up and running in a minute or two.

When that is done you should be able to ssh into OPI using username root and password op237.

You should then always start by doing a filesystem check on the internal storage. When logged in issue:

fsck.ext4 -y /dev/mmcblk1p2

Preferably twice if there where any errors. You should get a message that the filesystem where clean when all is good.

When that is done you should mount the internal storage to hopefully get any clue on what is wrong. Do:
mkdir -p /mnt
mount -text4 /dev/mmcblk1p2 /mnt

Then the interesting stuff is in the /mnt/var/log directories. Take a look into those and see if they might reveal why things aren’t working.

And as always, contact us at support@openproducts.com if you need deeper assistance.

/Tor

DarS · Post by **DarS** » Fri Nov 21, 2014 5:28 am

Tor,
Thanks for great hints on troubleshooting! I am glad I learned about 'rescue image', it is good to know that you can rely on such tooling when in emergency.

I was wrong saying that Email does not work. You inspired me to configure the desktop email client and ... voila! It worked. So I can confirm that both engines: Postfix and Dovecot do work. It was a malfunctioning web front-end which caused the impression that 'Email did not work'.

Few comments/hints (perhaps someone might find them useful):
- I noticed that 'ssh password' is being send to OPI ADMIN mail account. So if you created several mail accounts/users AND for daily work you use your regular account and not the admin account (desired practice), then you will not find the 'ssh password' in your email. You need to log-off and on as OPI ADMIN instead.
- whatever your OPI ADMIN user name is (let's assume JOE_ADM), your ssh login is always 'root' and not OPI ADMIN (JOE_ADM in our example).

Further debugging of broken ownCloud and email web frontend will be a breeze with 'ssh' enabled

Btw - I cannot see any ownCloud process on 'ps aux' list. How can I start it manually?

Regards,
-DarS

DarS · Post by **DarS** » Fri Nov 21, 2014 6:25 am

Sorry... I completely forgot that ownCloud is written in PHP. Why on Earth I expected any daemon to run?

But anyway, I am curious how to troubleshoot further the dead email frontend and OC.

Regards,
-DarS

DarS · Post by **DarS** » Sat Nov 22, 2014 8:14 am

Hi Tor,
Detailed examination of log files revealed, that the issue I experienced is related to web frontend.

MISSING FILES
'/var/log/nginx/error.log' file indicated clearly what went wrong.
For user web access:
"/usr/share/opi-webadmin/frontend/js/modernizr-2.8.0.min.js" failed (2: No such file or directory)
"/var/www/static/core/templates/404.php" failed (2: No such file or directory)
"/usr/share/opi-webadmin/frontend/skins/opi/images/OPI_RGB.png" failed (2: No such file or directory)

For ownCloud client access:
PHP Fatal error: Call to a member function getGID() on a non-object in /usr/share/owncloud/lib/private/group.php on line 193

And indeed, manual check revealed, that
- 'modernizr-2.8.0.min.js' file is missing, although other files (like 'jquery.cookie.js' or 'apiadmin.js') are in the '/usr/share/opi-webadmin/frontend/js/' directory,
- '/usr/share/opi-webadmin/frontend/skins' directory is missing, although other dirs (like '/css' or '/js') are there.

POTENTIAL CAUSE ?
'OPI Admin Webinterface' package includes majority (all?) of files placed in '/usr/share/opi-webadmin/frontend/' directory. The version installed on my OPI was 'opi-webfrontend_1.34_all.deb'. I thought that there was perhaps some failure in the OPI upgrade process and therefore my 'opi-webfrontend' was not updated to the latest version.
However, manual upgrade:
apt-get update
apt-get upgrade

surprised me with the output:
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
The source for 'apt' seemed to be correct: 'repo.op237.se/opi'.

SUMMARY
'apt-get' suggests that my OPI runs the latest version. However, certain files are missing and therefore web front end and ownCloud do not work.

Could I ask for hints how to move things forward?

Regards,
-DarS

Post by **tor** » Mon Nov 24, 2014 7:49 am

Hi again DarS,

Sorry for the delayed answer.

It definetly seems like there is something that went wrong with the upgrade. Which is the cause of the initial problem as well. We have noticed that the upgrade have failed for some customers.

It seemed like the device driver for the storage system ran out of memory and then terminated the upgrade as well. This unfortunately could leave the system in a somewhat inconsistent state

The owncloud access problem is however a different problem. It is a bug that where revealed when we upgraded OC. We have a fix for that one but unfortunately we are a bit hesitant to release that one until we fully have resolved the upgrade problem that you and a few other customers have been hit by.

There is however a temporary workaround for the OC client problem. Remove the users that are affected from any groups. Not pretty but it should solve that problem temporarily until the fix is released.

Regarding the missing files on the upgrade. I would recommend trying to reinstall opi-frontend.

Issue:
apt-get clean
followed by
apt-get install --reinstall opi-webfrontend

Hopefully this should resolve the problems.

Best Regards,

/Tor

DarS · Post by **DarS** » Tue Nov 25, 2014 6:14 am

Hi Tor,
I failed again

I am in a hurry, so cannot dig deeper at this time (in the evening perhaps). But this is the outcome of 're-install'

apt-get install --reinstall opi-webfrontend
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/863 kB of archives.
After this operation, 0 B of additional disk space will be used.
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)
debconf: falling back to frontend: Readline
dpkg: warning: 'ldconfig' not found in PATH or not executable
dpkg: warning: 'start-stop-daemon' not found in PATH or not executable
dpkg: error: 2 expected programs not found in PATH or not executable
Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin
E: Sub-process /usr/bin/dpkg returned an error code (2)

First thoughts - my OPI clearly is missing something (more screws up during upgrade?). Or the file system was corrupted as well.

And this is what I saw when I tried to install the 'missing Dialog'
apt-get install dialog
[...]
Package dialog is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'dialog' has no installation candidate

Regards,
-Darek

harrijer · Post by **harrijer** » Tue Nov 25, 2014 8:49 am

Not sure how helpful this is but:

I had the same problem and did the following:

export PATH=/sbin:/usr/sbin:$PATH

Then ran the:

apt-get clean
apt-get install --reinstall opi-webfrontend

Also ran:

apt-get install --reinstall opi-themes

However this did not cure my problem of not being able to access the owncloud parts of the web interface.

As Tor advised, I logged in to the web interface as an admin user and removed all members of a group I had setup to share calendars. This brought back access to owncloud. However, obviously my shared calendars are now no longer working. Luckily, our setup is not complicated and we can access the shared calendar using the credentials of the user that created it. Not ideal, but workable for us.

The reinstall of opi-webfrontend did not cure messages like:

2014/11/25 08:28:18 [error] 6124#0: *3804 open() "/usr/share/opi-webadmin/frontend/js/modernizr-2.8.0.min.js" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: localhost, request: "GET /admin/js/modernizr-2.8.0.min.js HTTP/1.1", host: "xxxxx.op-i.me", referrer: "https://xxxxx.op-i.me/admin/admin.html"

Not sure what the siginifcance of this is though as the webinterface seems to be working fine but then I don't use it a lot.

/Jeremy

Post by **tor** » Tue Nov 25, 2014 2:10 pm

Hi DarS and Jeremy,

I looked a bit deeper into the problem with the missing files mentioned above. It seems like these are actually false indications. I.e. they are referenced but seems not to be needed. (I have the same error-messages here.)

Hence those can be safely ignored and i have added them to our internal bug-tracker. This also means that reinstalling opi-webfrontend won't solve any problems as Jeremy also found out.

Regarding the other problem with the group problem this is a known issue and as i said before i have a fix ready to be released here. Unfortunately we have encountered other problems that got a higher priority. (We have a memory problem leaving some users units bricked after the last upgrade. I will write a separate note on this shortly.)

But as soon as that fix is released my intentions is to release an upgrade to OC with the group fix.

DarS, you however said that you also where unable to access web-mail? Is this still an issue? And Jeremy, your roundcube install works?

Finally, DarS the reason that issueing the commands failed where due to the missing PATHs as Jeremy found out. Just follow the advice and start by issue:

export PATH=/sbin:/usr/sbin:$PATH

And the commands should execute nicely.

/Tor

harrijer · Post by **harrijer** » Tue Nov 25, 2014 5:33 pm

Just to be sure, I have just done the following:

Logged in as a user that is not a member of a group and all panels work fine.

Logged in as admin, I added the same user to a group.

Logged back in as the user, now the only panel that works is the user admin, all others leave me at the spinning clock.

Remove the user from the group, all working again.

That bascially confirms your thoughts on the oc bug.

/Jeremy