Backup takes long time

Put support questions regarding KEEP here. Examples are hardware related issues and other questions that are specific to KEEP.
Post Reply
haraldc
Posts: 12
Joined: Sun Oct 05, 2014 9:45 am

Backup takes long time

Post by haraldc » Thu Sep 13, 2018 11:57 am

The backup takes bout 11 hours to complete. Currently I have 65GByte of data with very little changes of the data from day to day. The backup is made to a USB connected external disk. Is this long backup time normal? What will happen when I increase the data to 200GByte? How can I speed up the backup?

User avatar
pa
Posts: 66
Joined: Thu Aug 14, 2014 7:44 am

Re: Backup takes long time

Post by pa » Thu Sep 13, 2018 12:23 pm

Hi Harald,

I am assuming that you are using OPI with a pre 18.06 release?

The best way to speed up the backup is to upgrade the software to 18.06. In the old software there were a couple of unnecessary time-consuming recursive calls to index the data that has been removed.

Also we are currently testing further improvements that we will be released in a not do distant future that improves how Nextcloud updates its database, which has a significant impact on time spent for the backup.

Or to make it easier, upgrade to KEEP.... :)

User avatar
pa
Posts: 66
Joined: Thu Aug 14, 2014 7:44 am

Re: Backup takes long time

Post by pa » Thu Sep 13, 2018 12:27 pm

I just realized that you probably are using KEEP, since this is posted in "KEEP support"...

Still the second part of my answer above applies, the Nextcloud scan will take forever on the current stable version, but that will be significantly improved with the software currently in testing.

/PA

haraldc
Posts: 12
Joined: Sun Oct 05, 2014 9:45 am

Re: Backup takes long time

Post by haraldc » Thu Sep 13, 2018 3:29 pm

Hi,
I am running on Keep and the version is 18.06.3.
--Harald

haraldc
Posts: 12
Joined: Sun Oct 05, 2014 9:45 am

Re: Backup takes long time

Post by haraldc » Sun Jan 06, 2019 9:35 pm

Hi,
I have done some measurements and I have tried to guess what happens in the backup process.

It turns out that the actual backup seems to take only 4 minutes. It is the scanning of files after the backup to synchronize Nextcloud database (?) that takes more than 10 hours. The scanning is done on folder and file level only and does not involve the data. The scanning is done for the current file names and the file names in seven backup versions.

In the backup log (/var/log/opi-backup/backup.log) the following summary for the scanning is presented by the script /usr/share/opi-backup/link_backup.sh

+---------+--------+--------------+
| Folders | Files | Elapsed time |
+---------+--------+--------------+
| 11457 | 220349 | 10:50:10 |
+---------+--------+--------------+

Is this scan necessary to do after each backup?
The active files should be OK since they are only modified through the Nextcloud interface.
It is only the new backup and the removed backup that need to be scanned to update the database as far as I understand.
Is it possible to limit the scan to the new and the removed backups? That would cut the scan time by a factor four.

User avatar
pa
Posts: 66
Joined: Thu Aug 14, 2014 7:44 am

Re: Backup takes long time

Post by pa » Mon Jan 07, 2019 8:39 am

Hi Harald,

You have done some good digging, and you are correct in your assumptions.

The need for the scan of data is a dependency that is related to that Nextcloud needed the scan, otherwise the backup folders would not be shown in the file manager.

With the upcoming release that is now in testing, we have managed to limit the scan to just the base folder, so that it is not recursively done for every backup.
The penalty of this is that the listing of the files in the backup folders are a bit slower, since NC does not have an updated data base of these files, but it is a very limited penalty for saving lots of time in the backup process, just as you say.

For a "home" folder with 61892 files this is what it will look like:

Starting scan for user 1 out of 1 (pa)
Folder /pa/files/backup
File /pa/files/backup/.nomedia

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 1 | 1 | 00:00:00 |
+---------+-------+--------------+

/PA

haraldc
Posts: 12
Joined: Sun Oct 05, 2014 9:45 am

Re: Backup takes long time

Post by haraldc » Mon Jan 07, 2019 5:10 pm

Hi pa,
Sounds good. Does that mean that the the folders <user>/files/Documents and <user>/files/Photos do not need to be scanned?
If that is the case I will turn off the scanning for the time being since I can access the backups through ssh if I need them. I can wait for the next release to have them accessible through the Nextcloud interface again.

User avatar
pa
Posts: 66
Joined: Thu Aug 14, 2014 7:44 am

Re: Backup takes long time

Post by pa » Tue Jan 08, 2019 9:29 am

Hi Harald,

If you comment the lines 64-71 in the link_backup script, you will effectively remove the scan that is performed after the backup.

https://github.com/Kinguard/kgp-backup/ ... kup.sh#L64

Just note that the UI probably will not work then for the backups, since the backup folder will not be updated when the backups run, so it will have outdated data displayed.

/PA

haraldc
Posts: 12
Joined: Sun Oct 05, 2014 9:45 am

Re: Backup takes long time

Post by haraldc » Fri Jan 11, 2019 7:53 am

I did as you wrote and now the backup script completes in less than six minutes :) I look forward to the new release so that I can access the backup files through the Nexcloud interface again.

Post Reply