pointing extended elgg_objects to a new database table

Hi Everyone,

I've been sussing out elgg for a new project we might take on at work. Our client is creating a website with social networking functionalities which can potentially include a large number of objects and products .

I have some questions and would really appreciate if you could share you knowledge and expertise in th ematter.

a) has elgg been tested for large database sizes ( potentially 10 million objects and products ) previously ? and based on your experiences can it easily handle such loads ?

b) Is it easy to change the database to Oracle or SQL Server ?

c) Is it easy to extend elgg objects to write their data to separate database tables, as oppose to writing it to the metadata tables ?

Thanks,

Shaktus

 

  • a) Not just 'tested' - we run the largest Elgg-based SocNet website with numbers such as you're quoting.

    b) Oracle/SqlServer - you'll have quite a fair bit of code re-working cut out for you.

    c) This can be done but is generally not advisable once you've studied the Elgg Data Model thotoughly.

  • a) Do you have 10 million products now, or are you hoping to get there eventually? Can you afford the hardware and IT costs?  Also keep in mind that any non-customized system will require tweaking eventually.

    b) Database agnosticism has not been worked into the system yet.  It will be a while.  There is a ticket on Trac for this and IIRC it's slated for 1.9.  Any development that you could contribute in this direction would be appreciated by the community!

    c) As Dhrup explained, this is not advisable and rarely necessary.  I have never seen performance metrics that were so compelling that you would change the data model rather than approach the issue from another direction (e.g. database duplication/splitting, better hardware, etc.).

    Finally, you may find this article helpful: http://docs.elgg.org/wiki/Scalability.

  • I actually have a question about the comment above... Has ELGG ever been scaled horizontally or has it ever been studied and shown to be easily doable to duplicate databases and split the userbase/functions of the website horizontally across servers in order to scale ELGG into a multi-system backend social network (similar to the way Twitter does it)?

  • Duplicating/ Splitting Databases is not that much of a chore given the prerequisite technical know-how with manipulating MySQL databases and access to a seasoned Database Administrator - the sort of advice that will be more difficult to gather from most of the 'freebie open source' forums. Typically organizations that venture into "scaling" web applications get their expertise from professional sources - where the required technology experience is more readily available. The level of know-how and the time needed to study and derive workable solutions will take quite some effort in time and effort and such know-how cannot be explained in a few short sentences such as on this forum post ;-)

  • Yes, obviously. I know several professional level network engineers that do nothing besides scale backend networks for websites and for business/university level computer networks. I know that it is not simple or that someone can tell me everything there is to know about it. I am just asking, has ELGG ever been studied in this way/actually modelled in the way that makes it capable of splitting the database and scripts into separate servers to create a horizontal scaling model that would allow for just simple script reproduction to split users into separate user bases connected by a single backend?

  • Yes - but not in the simplistic script manner you have described ;) Have a browse/ study through my ElggCamp Boston 2009 presentation "Distributed Elgg Directions - A Conceptual Design"...

  • Thanks Dhrup and Evan, for your input.

    the website will essentially go live with just about 100,000 entities, but the client's optimistic that it will eventually get up to 10 million.  And the client can afford beefy servers and hardware/database redundency. These are what we usually call 'good problems'.  If their sales and services go up to that level we can afford to do major refactoring in the system.

    Thanks for the link to the Scalability Doco Evan. 

    Dhrup, could you fill me in on the details of the SocNet website testing, are these published/non-confidential ?

    Cheers,

    Shaktus