Cannot download or synchronize large files

Put support questions regarding KEEP here. Examples are hardware related issues and other questions that are specific to KEEP.
Post Reply
haraldc
Posts: 29
Joined: Sun Oct 05, 2014 9:45 am

Cannot download or synchronize large files

Post by haraldc » Thu Aug 15, 2019 5:07 pm

There seems to be a limit of 2G on the file size now. I have previously uploaded 3G and 7G files to the Keep server. Now I cannot synchronize them or download them.

User avatar
tor
Posts: 120
Joined: Thu Aug 14, 2014 3:42 am
Contact:

Re: Cannot download or synchronize large files

Post by tor » Mon Aug 19, 2019 12:02 pm

Hi haraldc,

Could you provide a bit more information regarding this. Is it that you cant upload files larger than 2GB using nextclouds web based filemanager? And when you say synchronize, i assume that this is the nextcloud sync app?

(It sounds like we might have missed some PHP/Nginx setting on the latest upgrade.)

/Tor

haraldc
Posts: 29
Joined: Sun Oct 05, 2014 9:45 am

Re: Cannot download or synchronize large files

Post by haraldc » Wed Aug 21, 2019 7:25 pm

It is downloading of the large files that do not work. Using the web interface I have tried Chrome on Windows 10 and Firefox on Linux.

When I set up a new client machine with synchronization software it refuses to synchronize the large files, larger than 2GByte. I have tried with a Linux machine with Nextcloud client (I think) and a Windows 10 with Owncloud client v2.5.1.

I am currently testing uploading a 3G file using a Chrome browser on Windows 10 and it seems to work. It will take a while and I let you know if it fails.

I do not know if the download of the large files worked before or not. The large files I have stored on the server were uploaded by synchronization software two years ago and it is not until now I have had a reason to access them.

haraldc
Posts: 29
Joined: Sun Oct 05, 2014 9:45 am

Re: Cannot download or synchronize large files

Post by haraldc » Wed Aug 21, 2019 9:14 pm

There seem to be a problem with the uploading of large files as well using the browser. I monitored the 3G file upload using SSH. A temporary file was created on the Keep server which grew steadily up to the size 2.0G. After that the file disappeared and the upload started again from scratch. The second time it disappeared again after it had grown to the size of 2126140160 bytes. The file was named <filename>.ocTransferId<number>.part

I then tested to move the 3G file into a folder that is synchronized by an Owncloud client with Keep. The behavior is different. In Keep a temporary folder with smaller files are created. The files are all less than 80M. The small files are merged to successfully create a copy on the Keep server of the local 3G file.

Conclusion: Upload to Keep server of 3G file using sync software works, but not upload using the browser.

User avatar
tor
Posts: 120
Joined: Thu Aug 14, 2014 3:42 am
Contact:

Re: Cannot download or synchronize large files

Post by tor » Thu Aug 22, 2019 1:04 pm

Hi haraldc,

I checked the web server configuration on keep/opi. And there is a hard limit on transfers set at 2GB which seem to match what you experience. That limit is set due to the limited storage on OPI.

If you want to experiment with that and don't mind getting your hands dirty. Use ssh and login to keep. As root, add a new file named 40-uploadlimit.ini to the folder /etc/php/7.0/fpm/conf.d containing something like this:

Code: Select all

max_execution_time = 600
max_input_time = 600
post_max_size = 4G
upload_max_filesize = 4G

Were you can experiment with sizes and timeouts.

To make changes take effect issue:

systemctl reload php7.0-fpm.service

It is however strange that there seem to have became a limit in the sync client. If i remember correctly that usually split up transfers and thus shouldnt be affected by the web server limit.

Check if altering above makes any difference on the sync issue. If not i think you should check in the Nextcloud support channels. (You do run the nextcloud client, not owncloud?)

Best!
/Tor

haraldc
Posts: 29
Joined: Sun Oct 05, 2014 9:45 am

Re: Cannot download or synchronize large files

Post by haraldc » Sun Aug 25, 2019 7:35 pm

Unfortunately no change. The fix does not seem to make any difference. After creating the new .ini file and starting up php again I tried to download a 2.4GByte file using the web interface from a web browser (not using the sync software) and nothing happens.

I have been using OwnCloud clients for synchronization. As you recommended I tried to switch to NextCloud on a Windows 10 machine, but I cannot make the client software connect with the server. It gets stuck on a Redirecting Window (see below). Searching NextCloud forums I am not alone with this problem. A remedy is to put 'overwriteprotocol' => 'https' in the config.php file. I have been searching my Keep server, but I have not been able to identify such a config.php file.
NextCloud-window.JPG
NextCloud-window.JPG (92.61 KiB) Viewed 21809 times
So I am quite stuck. The fix does not work and I am unable to switch to NextCloud that may work better.

haraldc
Posts: 29
Joined: Sun Oct 05, 2014 9:45 am

Re: Cannot download or synchronize large files

Post by haraldc » Sun Aug 25, 2019 8:42 pm

Correction - the fix with the .ini file works for file synchronization. I reinstalled the Owncloud client on the Windows 10 machine and it managed to synchronize the large files. The 2G+ files on the Keep server were downloaded to the client folders with the OwnCloud sync software. It is only the web download that does not work for large files after the fix.

User avatar
tor
Posts: 120
Joined: Thu Aug 14, 2014 3:42 am
Contact:

Re: Cannot download or synchronize large files

Post by tor » Tue Aug 27, 2019 8:06 am

Hi again haraldc,

Nice to hear that sync started working with the ini-change.

I have investigated this a bit more, and i now recollect that i have investigated this before. Downloading files larger than 2GB does not work with our version of Nextcloud, it is an issue fixed in a newer version. You can read a bit about it here: https://github.com/sabre-io/http/issues/77 and the fix is here https://github.com/sabre-io/http/pull/7 ... f711040e62. I tested this and with it i can download files larger than 2GB.

We will upgrade Nextcloud in a not to distant future if everything goes as planned but currently we are working on an upgrade to use Debian Buster and some other large changes.

Then to the lesser good news. Uploading files larger than 2GB using DAV or the web based filemanager is not as easy fix. It seems that this is more of a PHP-problem when running on a 32 bit OS/Architecture, which is the case with Keep, and there is, as i said, no easy fix for this.

But it fortunately seems like that isn't the main use case for you.

Best
/Tor

Post Reply