error in elggbatch? items being skipped?

i am using elggbatch here as part of an upgrade process for videolist. i am grabbing larger thumbnails from the various video websites and replacing the small format that videolist supports by default.

the whole process is working, except for that the batch processing appears to be skipping some items for unknown reasons.

i have added log entries for each of the steps in the process and also counters, so i can see what is occurring during the upgrade. essentially, the count increases from 1 to 360 (the total amount of videos on the site) and at several points, the logs are cut off and then resume further into the process..
so there appears to be a nginx/server fault of some kind here - the logs should not be cutting off mid-line of output.

there are some videos that are not being updated and they appear to be connected to the log-skipping issue..

i continue to debug this here.. and am wondering if anyone has encountered this before and has any tips?
thanks

 

  • Enable logging in php.ini, not on the fast cgi output in nginx log. I've seen that logs from fast cgi tend to be trimmed sometimes for some reason..

  • thanks, though as far as i am aware, most of the logging options that i can enable in php.ini already are enabled.
    though whether they are or they aren't, that doesn't explain why the logs are being cut short.

    i just switched on a couple of minor options in php.ini that were disabled.. re-ran the upgrade and the results are the same as before.

  • If you re-run it is it always the same entities being skipped?  If so try dumping information about them in your loop to see if they are actually entering the loop.  It may be that there's a data problem preventing them from being processed.  Regarding logs - I can't help you much there, other than my limited work on the community site here I've never used nginx

  • i haven't compared every line in the process.. however, from what i have seen, the patterns are repeating the same each time i run the upgrade.

    i am dumping information about them already, yes..
    the loop must be functioning since the counters are increasing accurately.. however, there are log entries missing. some images are not being processed due to permissions issues at youtube, for example.. i know that much.

    the only other hint i have is that the upgrade process ends with:

    if ($batch->callbackResult) {
        error_log("Elgg videolist upgrade (2014041001) succeeded");
    } else {
        error_log("Elgg videolist upgrade (2014041001) failed");
    }

    and the 'failed' message is being logged.
    i have set the return value of the upgrade function that is called in the batch to be true or false, depending on the outcome of each iteration.. i am not sure on the intended purpose of 'callbackResult'.. without looking at the code, i think maybe a 'failed' msg will be outputted if any of the iterations return false.

  • I can't follow the explanation regarding a possible permission issue at youtube (or other external sites). Aren't you trying to update entries in your database only? Then the only permission issue that could occur would be with these entries. Maybe it would help to add a few things before fetching the items from the database with an ElggBatch:

    // Ignore access to make sure all items get updated
    $ia = elgg_set_ignore_access(true);

    elgg_register_plugin_hook_handler('permissions_check', 'all', 'elgg_override_permissions');
    elgg_register_plugin_hook_handler('container_permissions_check', 'all', 'elgg_override_permissions');

    // Make sure that entries for disabled entities also get upgraded
    $access_status = access_get_show_hidden_status();
    access_show_hidden_entities(true);

    Now the fetching and upgrade code and afterwards:

    elgg_set_ignore_access($ia);
    access_show_hidden_entities($access_status);

  • the permissions from youtube appears to be related to videos that are no longer active on youtube, yet are still in the elgg database - or perhaps have been set to private on youtube, after having already been added to elgg when the video was public on youtube... as far as i presently know.

    i copied the upgrade script from videolist that was already in there, so i already am including the lines to handle the ignoring of access in elgg.
    the hidden entities is not an issue i knew to exist in elgg at all.. i will look into that, though that would not be causing the failure of the log file.

  • well, i'm still not sure what occurred there. however, i have run the upgrade on my production site and so far i haven't found any images that weren't upgraded. so now i have high quality thumbnails for videolist. :)
    i will most likely fork videolist and include this upgrade, along with the projekktor player - so that relevant media files stored in the files plugin are shown alongside external videos by videolist.

  • oh, no.. i found some that weren't processed. hmm.

  • if i locate a specific video that has 'disappeared' from the process - and ammend the paramaters of the elgg batch, so that only that single video is included in the batch, then the process completes successfully.
    seems like some kind of limit is being reached and not declared.. such as a timeout or even a memory limit.. but i am not aware of any that are likely to be being exceeded.
    maybe youtube has a threshold to prevent scraping.

  • ah ok, i needed to use curl to check if the thumbnails exist, before grabbing them - since some videos are removed or set to private. now the upgrade appears to have completed correctly.