File Plugin | Questions about performance

Hi there!

Long time no see!

 

Anyways, I'm about to use elgg for a big project, which will require big uploads through the file plugin. Right now I'm using an Ubuntu server with PHP 7.x that I set up with Vagrant. 

I was doing some testing, and I uploaded a 200+ MB file. It uploaded just fine, but when it reached 100% elgg was doing something, there was a big pause. Now, here's are questions:

What is elgg during after upload and during that pause? 

Will I have a longer pause with bigger files?

Can elgg handle bigger files? 1GB? 2GB? 

Can the upload be optimized?

  • I don't know what Elgg does exactly after the upload in that pause. To be honest I never tried uploading any file larger than maybe 100M. My guess is that Elgg might write the uploaded file to the data directory during the pause (while the upload was done only into memory in the first place).

    Dealing with very large files is probably a matter of the server hardware and its configuration. Elgg seems not to use any sophisticated upload code but relies on PHP core code, i.e. what PHP can (and is not limited for example by PHP variable values and RAM of the server and lastly maybe by filesystem limits on the server) would then also work with Elgg.

    If this functionality is reliable and comfortable is probably another matter (e.g. it might be much better if interrupted uploads could be resumed especially for large files and probably give better feedback like a progress bar during upload and a definite message once the upload is complete for the user to know he can leave the upload page or would have to keep it open for some reason).

    I guess there are optimizations and other improvements possible. For example the Tidypics plugin uses the plupload uploader to provide an html5 compatible uploader. Plupload would support uploads in chunks, gives some feedback with regards to progress etc. It would also allow for multiple uploads at once. I guess the overall performance would not necessarily be any better automatically. And automatically it wouldn't work anyway as the Files plugin would have to be modified to make use of Plupload and probably also to add other improvements (and if only for larger files).

    • I guess there are optimizations and other improvements possible. For example the Tidypics plugin uses the plupload uploader to provide an html5 compatible uploader. Plupload would support uploads in chunks, gives some feedback with regards to progress etc. It would also allow for multiple uploads at once. I guess the overall performance would not necessarily be any better automatically. And automatically it wouldn't work anyway as the Files plugin would have to be modified to make use of Plupload and probably also to add other improvements (and if only for larger files).

     

    Right now, I'm looking to a Drop Zone plugin integration, which might solve some problems.

    Dealing with very large files is probably a matter of the server hardware and its configuration. 

    I was thinking the same thing. Due to be a server with limited resources, that might be the cause of that pause I mentioned.

     

  • I was thinking the same thing. Due to be a server with limited resources, that might be the cause of that pause I mentioned.

    Well, maybe testing on a server with less limitated resources or testing with different file sizes (specifically taking into account the limits of RAM/SWAP, free memory and limits set with php variables) might help to better understand when and why a pause/delay occurs.

  • I had similar problem when I uploaded images with too many pixels (width X height).

    The problem was with GD but it gone with larger memory.

    No problem with imagic.

  • Well, maybe testing on a server with less limitated resources or testing with different file sizes 

    I did. No issues uploading large files. I am impressed by the robustness of elgg.

    The problem was with GD but it gone with larger memory

    It seems like the way to solve this is by having a good amount of Memory. 

    Thanks for the feedback!

Performance and Scalability

Performance and Scalability

If you've got a need for speed, this group is for you.