Elgg with Amazon EC2, AM Load balancer and Amazon RDS

Hello,

We've setup a small social services APIs with elgg and its REST services framework.

We are running everything on Amazon's infrastructure and please note that we are using only REST services from Elgg.

My question is, what should we do in order to optimize our service WITHOUT heavy pages-view processing?

Thank you.

  • You can, but I recommend that you should declare a new service handler and remove whatever you don't need.

  • Sounds like you would just want make sure core is loading only the parts you need. Since you only use the api, I think maybe the views system is the heavier subsystem that you aren't using. Page routing and actions may be the others.

    This would require a core patch so working with us is probably your best bet long term here.

  • You can also read this article, it describes or lists some optimizations that are not rest-specific. You should also optimize particular methods, making sure process of using them is well planned and that they are well written. It's sometimes possible to significantly limit amount of calls to REST. I know this seems obvious, although from my experience this is the most common problem in services we were optimizing so far - it's worth to recheck.

    You should also take care of fluent scalability using Amazon services, although you probably did this already :)

  • Hi Evan and rubycell,

    Any update on how to patch the core in order to remove the view subsystem and any other parts not needed when loading the Elgg engine?

    We are using only REST requests in one project I am working on and I would need this optimization.

    Thanks!

     

  • @rubycell & @SocialPlus: Are you actually running into performance trouble or just wondering how to make it faster?

    You can cut a bunch of code loading, but I don't see that helping all that much. By far the biggest gains are going to come from reducing queries, like the improved entity caching (just added to 1.8) and metadata prefetching (hope to get in 1.8 very soon).

    If you do do some cutting, please consider posting your experiments to github (work off a branch so you can merge new Elgg code) and let us know to take a look. 

    Do you expect a ton of unauthenticated clients? If so you could significantly benefit from a proxy cache like varnish and making sure your responses have good cache headers.

     

  • Many thanks Steve for your feedback.  We exepect tons of mobile authenticated client requests (unauthenticated requests are not allowed).

    For our project this is an architectural issue that has to do mainly with distribution and scalability than with performance. The architectural principles we are following are:

    • Splitting the application into separate application and presentation layers.
    • The Application layer can have one or more Application nodes which process application logic, own, store and manage all entities.
    • Mobile clients (native apps) and Web clients

             - reside on the Presentation layer
             - control their own UI workflow
             - communicate with Application node(s) via REST calls which resturn JSON results
             - work locally with 'proxy' entities

    Therefore, App nodes don't need a view system and Web clients don't need a blown up appliction layer (e.g. entities CRUID) anymore. So we need to do these cuttings.


    What we want to achieve with this architecture is:

        - Creating a distributed and cloud oriented architecture.
        - Scaling out on the application layer by adding more App nodes
          (specialized for different tasks).
        - Scaling out any time for Web clients (e.g. load balancing)
        - Clients are not bound to a specific UI technology
        - Caching on diferent layers
            - memcached for App nodes (caching Web service call results)
            - Proxy caching for Web clients (we are using Varnish)

    Regarding search, we will use Elasticsearch as it fits very well in such a distributed architecture.

    We are working on a proof-of-concept, and, I will report later with some results.

  • you need to identify parts of the core which serve your app layer "nodes'" purposes and then 'build' your own 'elgg' engine shells to satisfy those functions. the elgg engine by itself is = procesing + presentation. you want to split those aspects ? do exactly that - in order to have your elgg as pulled-apart and put together again into different func pkgs - for yr diff func 'layers'.

  • @DhrupDeScoop

    For App nodes I will have to pull out an Elgg core version without the view system. My starting point is going to be start.php:

    // load the rest of the library files from engine/lib/
    $lib_files = array(
        'access.php', 'actions.php', 'admin.php', 'annotations.php', 'cache.php',
        'calendar.php', 'configuration.php', 'cron.php', 'database.php',
        'entities.php', 'export.php', 'extender.php', 'filestore.php', 'group.php',
        'input.php', 'languages.php', 'location.php', 'mb_wrapper.php',
        'memcache.php', 'metadata.php', 'metastrings.php', 'navigation.php',
        'notification.php', 'objects.php', 'opendd.php', 'output.php',
        'pagehandler.php', 'pageowner.php', 'pam.php', 'plugins.php',
        'private_settings.php', 'relationships.php', 'river.php', 'sessions.php',
        'sites.php', 'statistics.php', 'system_log.php', 'tags.php',
        'user_settings.php', 'users.php', 'upgrade.php', 'views.php',
        'web_services.php', 'widgets.php', 'xml.php', 'xml-rpc.php',

        // backward compatibility
        'deprecated-1.7.php', 'deprecated-1.8.php',
    );


    So I will remove the view related libs from this array one by one. It all depends on the dependencies between the core subsystems and whether the view subsystem is somehow loosely coupled form the other core subsystems. So I will have to experiment with this until I find out the core parts dependencies.

    The experimental core weight loss & diet plan is as follow:

    1. Removing view libs/handlers: views.php, widgets, navigation, pagehandler, languages.
    2. Removing view related classes like ElggMenu, ElggWidget etc.
    3. Hmm...what else? what do you think?
  • " pull out an Elgg core version without the view system" YES! & that sounded like what you wanted  /needed for yr Apps/Layers 'nodes' - if u r demaracating via node|function - the 'split' would have been inevitable. start.php may be fine.. but you'll also find script files (trbl spots?;) where e.g. elgg_list  and elgg_get functions are mixed - the _list_ will invoke *views which u do not need.. --> 'split code files' ?? u might be looking at some sort of re-structuring of elgg along the way ;-) if you do - i am cofident that a fair bit if what you encounter and resolve will end uo being useful for elgg design in general.what u r doing kinda vaguely resembles my 'distributed elgg directions - presented @elgg hootenany some years ago @Boston) - same sentiments..

     

  • You can safely remove the following libraries too,

    • calendar, 
    • export, 
    • opendd, 
    • and if you are sure that you are not using any deprecated functions in your plugins, just remove those two massive deprecated libraries too.

    In our install, we have removed the above libraries with web_services too.