Duplicating / Migrating Elgg Install

This wiki doc http://docs.elgg.org/wiki/DuplicateInstallation does cover *sufficient instructions for duplicating / migrating your Elgg site to another domain.

In the process of  migrating Elgg sites for some of my clients, I have discovered some problems with the size of the database and some other places where Elgg stores the Domain name.

E.G.

If the MySql dump is rather large(er) -- for example 8MB - we're stuffed because PhpMyAdmin limits the Sql file size (2MB).

To cater for this I have created some php utilities to scan and split the mysql dump into smaller files which can be handled my PhpMyAdmin and also to zap the other Domain name text occurences.

The process takes several hours of work effort -- in order to retain all site data, i.e. Users, Groups, Topics, Files..... from the old Elgg site to be carried over to the new Elgg site.

Whew !!! ;-(

  • Hi,

    It shouldn't take several hours, at most 1, if you have shell access, you could run three scripts:

    mysqldump -u username -h mysqlservername -p databasename > elgg2009.sql
    tar -cvvf elgg_code.tar elggphpfolder/
    tar -cvvf elgg_data.tar elggfilesuploads/

    Doing the string replacement is easy is you work directly on the SQL file and do a simple search and replace, and then reimport it.

    Then changing data root in the database, and that's it.

    Regards,

    Uddhava

  • What is your file size limit for import in phpMyAdmin ?

  • Hi Dhrup, although i have phpmyAdmin i don't use it for duplicating sites, its just not good for importing huge files, simply use command line again (changind the > to < to import):

    mysqldump -u username -h mysqlservername -p databasename < elgg2009.sql

    also many other desktop applications let you import huge files, without being as slow or troublesome as phpmyadmin import.

    Regards
    Uddhava

  • Need to get full perstective. My reference to PhpMyAdmin was for a *client who does not have ssh access nor direct access to the MySq, and their dump was hugh. So in such a situation, would you not take the business ? ( some other clients have ssh )

  • My experience with exporting/importing with phpMyadmin has always been bad for huge files. It works nice for tables, or small databases. But really, any shared host will let you enable mysql from external IP, and for that you can use any desktop software available.

    I wouldn't recommend using phpmyadmin at all for this task. Better invest the several hours in finding a new set of tools to work with, instead of depending on phpmyadmin.

    Regards,
    Uddhava

  • WoW... we are now demeaning PhpMyAdmin..???!!!

     

    Never had a glitch with it.. and by the way.. I am also talking large db.

     

    Before we start talking PhpMyAdmin -v- local environment tools with mySql access, we need to remember that there are other things that come into play besides what piece of software you use, your server capabilities, your host and even speed all play a role. It has identical markings as ftp. Sure, manipulating SQL from a local environment, with permitted access of course, works.. and yes there may be some advanced tools out there that will do that, but that don't mean that a web based prog is not good. Hell.. it is in fact the common thing. I had enough of sitting in coffee shops arguing with other developers that notepad can produce the same quality website as dreamweaver.. and I wont do the same for db migration protocols.

     

    Now, I'm not an expert on db migration, but as a developer designer, from a business perspective, I'd exactly what Dhrup is suggesting.

  • Made some more progress on this Site Migration.

    Downloaded my client's whole Elgg install to my localhost on PC.

    Split the mega-size mysql dump into 26 smaller files.

    Patched the Domain name, HomeDir, DataDir directory name occurences (automatically) - there are more occurences of the doman name -> depends on certain PlugIns  being in use who use other database tables not generally known.. "cave mysqlanis !", fixed the engine settings and voila => looks exactly like the original server-based site.

    Tried to login and...

    POOF !!! ;-(

    Error = " No has podido iniciar sesión. Es posible que no tengas una cuenta todavía, los datos de registro son incorrectos o has intentado acceder sin éxito demasiadas veces. Asegúrate de que todos los datos de registro facilitados son correctos y por favor "

    ==> " You could not log on. You may not have an account yet, registration data is incorrect or you have tried unsuccessfully many times. Make sure that all registration details provided are correct and please try again." ;-(

    What the heck was my password on the server site ???

    If I do not remember soon.. I'm gonna do a double by-pass PhpMyAdmin on the database to Patch in a new password for me UserID LOL.. or run my Load_New_User Utlity... ;-)

    Once I have completed this migrate / duplicate / copy of the Elgg site onto my localhost PC. I will repeat the process on the new server. When all done I'll post some more notes here esp regarding the "other" places where domain name is held by PlugIns and some other caveat thingys to watch out for...

     

     

     

     

     

  • I patched the database with a new password.. 1.25 mins.. I'm in ;-) more chit-chat later...;-O

  • Ok,

    Now the site has been relocated to a new host, using the same mysql splitter utility as for the PC install. So we have taken the website from old Host, migrated to my development PC's server at Home, then moved to a new Host. What I have fiund somewhat ratther useful is the mickey-mouse utility I coded to so the split and also the text changes for old domain, etc to new domain, etc.

    Here is the output from running the utlity. You will see that there are other occurences of the domain.com and such strings which also need changing. The extra domain.com you will note is usually created by some PlugIns, e.g. PhpMailer, etc...

    The whole process becomes is a little tedious, but safe...

    The system log and the sessions tables do not really need to be migrated, though the empty tables will to be created.

     

    MYSQL_2_CREATE.php

       [www.MyDomain.com]&nbsp;
    ==>[www.MyDomain.com]&nbsp;
       [/homepages/AAA/htdocs/AAA/]
    ==>[/home/MyDocRoot/public_html/]
       [/homepages/21/d237884458/htdocs/data/]
    ==>[/home/MyDocRoot/elggdata/]

    1  :DROP TABLE IF EXISTS `elggaccess_collection_membership`;
       (elggaccess_collection_membership)
       LINES = 7
    2  :DROP TABLE IF EXISTS `elggaccess_collections`;
       (elggaccess_collections)
       LINES = 11
    3  :DROP TABLE IF EXISTS `elggannotations`;
       (elggannotations)
       LINES = 94
    4  :DROP TABLE IF EXISTS `elggapi_users`;
       (elggapi_users)
       LINES = 10
    5  :DROP TABLE IF EXISTS `elggconfig`;
       (elggconfig)
       LINES = 13
    6  :DROP TABLE IF EXISTS `elggdatalists`;
       (elggdatalists)
       ('path', '/homepages/AAA/htdocs/AAA/'),
    ==>('path', '/home/MyDocRoot/public_html/'),
       ('dataroot', '/homepages/21/d237884458/htdocs/data/'),
    ==>('dataroot', '/home/MyDocRoot/elggdata/'),
       LINES = 50
    7  :DROP TABLE IF EXISTS `elggentities`;
       (elggentities)
       LINES = 2465
    8  :DROP TABLE IF EXISTS `elggentity_relationships`;
       (elggentity_relationships)
       LINES = 12
    9  :DROP TABLE IF EXISTS `elggentity_subtypes`;
       (elggentity_subtypes)
       LINES = 43
    10 :DROP TABLE IF EXISTS `elgggeocode_cache`;
       (elgggeocode_cache)
       LINES = 9
    11 :DROP TABLE IF EXISTS `elgggroups_entity`;
       (elgggroups_entity)
       LINES = 9
    12 :DROP TABLE IF EXISTS `elgghmac_cache`;
       (elgghmac_cache)
       LINES = 7
    13 :DROP TABLE IF EXISTS `elggmetadata`;
       (elggmetadata)
       LINES = 2338
    14 :DROP TABLE IF EXISTS `elggmetastrings`;
       (elggmetastrings)
       (279, '/homepages/AAA/htdocs/data/'),
    ==>(279, '/home/MyDocRoot/elggdata/'),
       (298, 'www.MyDomain.com'),
    ==>(298, 'www.MyDomain.com'),
       LINES = 1414
    15 :DROP TABLE IF EXISTS `elggobjects_entity`;
       (elggobjects_entity)
          INSERT INTO `elggobjects_entity` (`guid`, `title`, `description`)
          VALUES     
          (3, 'announcements', ''),
       (3190, '[%site_name%] http://www.MyDomain.com/pg/siteaccess/activate... %site_url%'),
    ==>(3190, '[%site_name%] http://www.MyDomain.com/pg/siteaccess/activate... %site_url%'),
       (3191, '[%site_name%] http://www.MyDomain.com/action/siteaccess/confirm...'),
    ==>(3191, '[%site_name%] http://www.MyDomain.com/action/siteaccess/confirm...'),
       LINES = 2410
    16 :DROP TABLE IF EXISTS `elggprivate_settings`;
       (elggprivate_settings)
          INSERT INTO `elggprivate_settings` (`id`, `entity_guid`, `name`, `value`)
          VALUES     
          (1, 3, 'display_count', '10'),
       (9122, 2472, 'feed_url', 'http://www.MyDomain.com/'),
    ==>(9122, 2472, 'feed_url', 'http://www.MyDomain.com/'),
       (9187, 2527, 'feed_url', 'http://www.MyDomain.com/'),
    ==>(9187, 2527, 'feed_url', 'http://www.MyDomain.com/'),
       (9474, 2614, 'feed_url', 'http://www.MyDomain.com/'),
    ==>(9474, 2614, 'feed_url', 'http://www.MyDomain.com/'),
       (9731, 2683, 'feed_url', 'http://www.MyDomain.com/'),
    ==>(9731, 2683, 'feed_url', 'http://www.MyDomain.com/'),
       (10734, 3020, 'feed_url', 'http://www.MyDomain.com/'),
    ==>(10734, 3020, 'feed_url', 'http://www.MyDomain.com/'),
       (10783, 3047, 'feed_url', 'http://www.MyDomain.com/'),
    ==>(10783, 3047, 'feed_url', 'http://www.MyDomain.com/'),
       (10852, 3064, 'feed_url', 'http://www.MyDomain.com/'),
    ==>(10852, 3064, 'feed_url', 'http://www.MyDomain.com/'),
       (10906, 3081, 'feed_url', 'http://www.MyDomain.com/'),
    ==>(10906, 3081, 'feed_url', 'http://www.MyDomain.com/'),
       LINES = 13029
    17 :DROP TABLE IF EXISTS `elggriver`;
       (elggriver)
          INSERT INTO `elggriver` (`id`, `type`, `subtype`, `action_type`, `access_id`, `view`, `subject_guid`, `object_guid`, `posted`)
          VALUES     
       LINES = 275
    18 :DROP TABLE IF EXISTS `elggsites_entity`;
       (elggsites_entity)
          INSERT INTO `elggsites_entity` (`guid`, `name`, `description`, `url`)
          VALUES     
          (1, 'MyDomain.com', '...', 'http://www.MyDomain.com/');
       INSERT INTO `elggsites_entity` (`guid`, `name`, `description`, `url`)
          VALUES     
          (1, 'MyDomain.com', '...', 'http://www.MyDomain.com/');
    ==>INSERT INTO `elggsites_entity` (`guid`, `name`, `description`, `url`)
          VALUES     
          (1, 'MyDomain.com', '...', 'http://www.MyDomain.com/');
       LINES = 11
    19 :DROP TABLE IF EXISTS `elggsystem_log`;
       (elggsystem_log)
       LINES = 50793
    20 :DROP TABLE IF EXISTS `elggsystem_log_1245866874`;
       (elggsystem_log_1245866874)
       LINES = 10482
    21 :DROP TABLE IF EXISTS `elggsystem_log_1245970429`;
       (elggsystem_log_1245970429)
       LINES = 11428
    22 :DROP TABLE IF EXISTS `elggsystem_log_1245970844`;
       (elggsystem_log_1245970844)
       LINES = 3405
    23 :DROP TABLE IF EXISTS `elggsystem_log_1245977554`;
       (elggsystem_log_1245977554)
       LINES = 14
    24 :DROP TABLE IF EXISTS `elggusers_apisessions`;
       (elggusers_apisessions)
       LINES = 11
    25 :DROP TABLE IF EXISTS `elggusers_entity`;
       (elggusers_entity)
       LINES = 64
    26 :DROP TABLE IF EXISTS `elggusers_sessions`;
       (elggusers_sessions)
       LINES = 26

    TOTALS:
       TBLS = 26 
       HTTP = 12
       HOME = 1
       DATA = 2

    EOF

  • Some qualification on the technique(s) to use..

    As Uddhava Dasa has explained earlier above, --

    Uusing SSH & linux commands **is much quicker - **if you are comfortable at that level using linux shell commands. The main rationale for the ssh linux command line approach is that we are working on the server directly (after uploading tgz files) and the commands will usualy take a few mere seconds to execute as compared to the slower response times using WHM/ CPanel/ PhpMyAdmin.

    The approach I took with PhpMyAdmin would be more suitable for the "average" user who does not want to "fool around" with programmer type commands and prefers the mostly point-and-click UI that CPanel and PhpMyAdmin offers.

    The few hours I spent on the first try was worth it because now I have a tool which does actually bring the total time down to < 1 hour ++ whatever extra time one may want to send in checking, rechecking, testing, etc to be certain and happy that they have done everything correctly. Then we smile ;-)

    PS:

    Next I might tackle another little pet project - Copying User Avatars frrom Joomla over to Elgg - My client actually dumped their Joomla-base SocialNet and crossed over to Elgg, but the user's would be more pleased if their avatars can be copied over automatically..