Elgg at a crossroads

I'm excited about the upcoming Hackathon (Sept 23, 24), but this temporary burst can't make up for the fact that core dev contributions are down and have been for a long time.

This isn't anyone's fault. To an extent, this is normal cyclic behavior as people get busy with other things, but I rather think it's that Elgg is no longer exciting or fun enough to maintain developer interest.

Sure, we've made huge improvements in DX and performance, and 2.x provides a lot of value, but the product remains highly tethered to trends, technologies, and architectural decisions made 8 years ago. We have some loud voices in our community telling us that Elgg should change even more slowly (or not at all), and I've personally over-romanticized the idea of extreme backwards compatibility, but if we continue this path I think developer interest will keep dropping off.

Personally I really enjoy working with Elgg's contributors and don't want to stop, but for my career's sake I can't focus on reworking 2008-era APIs much longer. I considered quitting in 2014 and didn't out of inertia and because family health problems made it comforting work to bury myself in.

What can we do?

We could do nothing. This is just my perspective and I could be wrong or it could really not matter. There's nothing wrong with stable software that nudges along slowly. My prediction is that high maintenance chores will start to go by the wayside. Elgg 3.0.x will get dozens of releases and plugins will thrive as they did on 1.8. It's not a bad outcome for lots of people, but I'd be bummed.

In the short term, Ismayil and I should stop stalling new development in order to wait for a 3 member consensus. If a PR on 2.x/master gets a thumbs up and no one expresses concern or a desire to review for weeks, it goes in. Those branches give plenty of opportunity to revert or fix concerns that come up later. When reviewing involvement goes up, we can scale the need for consensus accordingly.

Businesses relying on Elgg could offer developer time. That won't help if devs really don't want to work on it though. And IMO business owners are under no moral obligation to contribute, but it would of course help.

We could seek out and destroy any low-return hassles that make working on Elgg less fun:

  • Commit messages are a constant headache, but may be worth it for our awesome changelog.
  • LTS releases: if so many sites want to run 1.12, one of their developers should put in the work of backporting fixes to it; merging this stuff into 3 other branches is a headache. Maybe the foundation can pay someone to make these releases occasionally.
  • Community spam: Maybe you can't register until you to contribute some sort of content; then we can just delete several thousand spam profiles.
  • RST is still !@#$. I'm continuously looking up how to do basic things and even the docs about RST are bad. Markdown is inferior tech with miles ahead UX. Really I want to do like Drupal and build more docs from source code.
  • ??

We should identify roadblocks that make more radical changes seem impossible. It was really disheartening to turn away a good 3rd party effort at moving to UTF8MB4 columns. Schema changes in general are almost unimaginable due to the expectation that Elgg run its own upgrades.

Personally I would love to--at least temporarily--set aside the requirements of upgrade paths and BC and start over.

  • How can Elgg be many magnitudes faster, not a few ms.?
  • How can it have the real-time functionality users now expect?
  • Are we sacrificing anything by giving plugins too much power?
  • What Elgg-specific dev concepts are really cool or not worth the overhead for new devs to learn?

In general I want Elgg to take a great leap forward to become something I'd be proud to put my name on, not filled with things I have to apologize about.

  • My main concern is Elgg's UI. Perhaps we could start by scraping the theme and forgetting about BC for a minute and building a modern UX/UI suite with real-time features in mind. We could branch off the project and if anyone cares about the TLS they can put effort into maintaining them. My patience and motivation to work with Elgg is runnjng out and I feel like I am a mindless robot sometimes doing and redoing the same things. We did make a great leap but BC concerns over ancient code hold us back too much.
    Maybe we should stop trying to squeeze in features into stable releases and just focus on a next major. But again I spend to much time on core sacrificing my free time and financial stability, so it might as well be good time to explore new technologies, though it will be hard to abandon all the work that has been done and the ideas that have piled up.

  • My vote (small one indeed) would be to march bravely forward, and not worry too much BC. If a plugin is written well, then it should work. However, the current state of Elgg, IMHO, is not sustainable in the long-term. It is dying of a thousand cuts.

    The lack of "expected" features is always a major concern, and I find I spend way too much time either trying to "jury-rig" something or building plugins that should be in core, just to get a project out the door. And my understanding is that because of the legacy code, it is near impossible to easily add these features.

    Again, my small vote is to "cut the cord" at "Elgg 3 or Elgg 4 or whatever is practical. will have "practical."

    I can't start to count the number of times prospective clients have balked at:

    • no video system. In a modern software, this is unacceptable;
    • no decent mobile experience;
    • groups are confusing (all content created in a group, should stay in that group when clicked. I should not be taken to the file, page, blog plugin. Which causes the user to have to navigate back to the group.);
    • comment system is awful;
    • UX/UI outdated;
    • no real-time;
    • no membership levels: basic, premium etc.;
    • or administrator levels: moderators, site admins etc.;
    • no way to "easily add CMS" pages that are not stripped of content because htmlawed will not allow it. I am talking trusted members (admins) adding content, not users; and
    • etc.. etc.

    Many of the features of Elgg currently seemed to be cobbled together. For instance, look at the page structure of the Blog plugin versus the Page plugin. This is pretty hard to explain to a client.

    These are core features that will help people start making a real investment in Elgg. I want to start using more Elgg-capable freelancers for my project. I am just hesitant at the moment to invest a lot of time/effort and money in features that should be core, and might not work a year from now. 

    I do understand that Elgg ultimately is a framework and volunteer based. Therefore it's not meant to be everything to everyone; that's actually what I like about it. However, when I buy a car, I do want ubiquitous features. For instance, I know a car does not HAVE to have a radio, but I am not going to buy one without it.

    Why not take the best of what is there, make it as BC as possible, and start creating something that people really want/can use. Enough work has been put into the 1.x/2.x branches that someone would not be left immediately out in the cold if they did not want to upgrade. And people are going to complain, that's their right.

    Change is inevitable; you can either get in front of it, or get rolled over by it.

    This "change" is very necessary and loooooong overdue.

    P.S. Hopefully it will not be a five-year change like Magento.

    P.S.S Hopefully this did not come off as a rant (not my intention) - I would love to contribute to Elgg to move forward, not just fix things.

  • Taking into consideration the current developer activity, I'd be fine with the following:

    • Let's reduce the needed approval for PR to two developers (but be careful out there!)
    • Semver allows to break stuff in Elgg 3, so knock yourself out! :)
    • If someone wants to keep using older LTS versios, they should start backporting the features. Site owners cannot expect core devs to spend their free time maintaining legacy code out of kindness.

    Personally I would love to--at least temporarily--set aside the requirements of upgrade paths and BC and start over.

    I'd be fine with putting dynamite to the APIs without providing exact upgrade instructions (e.g. "Passing a string to elgg_do_something() is deprecated, pass in an ElggSomething instead"). But we must however provide upgrade paths for the database and dataroot data. If those cannot be run by Elgg itself (booting engine depends on the data that must first be migrated), we should provide a separate CLI script or something to do the job.

    About the crossroads, IMO it wouldn't be only "nice to have" that Elgg would make a huge and brave leap forward, Elgg *must* the take leap. Elgg *must* evolve. I also cannot see myself working with Elgg in the future unless we get rid of all the ancient architecture and make Elgg truly a modern framework.

  • Since I am commenting via a cellphone I will not say much. I think it is time for egg to start as soon as possible to build a modern UX/UI suite with real-time features!

  • I am going to address invididual bullet points made:

    • Commit messages are a constant headache, but may be worth it for our awesome changelog.

    I don't have any issues with commit messages. I use them all across my projects/plugins now.

    • LTS releases: if so many sites want to run 1.12, one of their developers should put in the work of backporting fixes to it; merging this stuff into 3 other branches is a headache. Maybe the foundation can pay someone to make these releases occasionally.

    Do we really need to merge them up? I would say we could just make a fix in an LTS release, and do the same if needed in higher branches. I don't see a reason to merge up.​

    • RST is still !@#$. I'm continuously looking up how to do basic things and even the docs about RST are bad. Markdown is inferior tech with miles ahead UX. Really I want to do like Drupal and build more docs from source code.

    +1. I hate RST markup. Sphinx is a pain in the ass to use, compiling docs for each release is just an extra burden.

    • How can Elgg be many magnitudes faster, not a few ms.?

    We need a thorough inspection and optimization of DB queries. 

    Fingers crossed, Brett will denormalize the metadata table. I really hope we can then loose objects_entity, users_entity, and groups_entity. That will make so many things so much easier.

    • How can it have the real-time functionality users now expect?

    I have been doing a lot of work in this area, and it's hard. Ideally we would make a switch to a JS framework, but none of them will be compatible with our view system. One way would be to loose the view system, or switch it over to handlebars or similar that can be rendered both server/client side. 

    If we go down the PJAX route, which is also fine, we would need to hack our way through elgg_list_* functions and make them autonomous. I have been doing some work on sortable lists: https://github.com/hypeJunction/hypeLists#sortable-list-views - in theory if we loose type specific tables, we could build a unified interface for all lists with search/sort capabilities.

    AJAX tabbing would be a great addition, but we would have to decomposition page/layout views. Some work has been done here, but whenever a third-party plugin is being a smart-ass things break: https://github.com/hypejunction/Elgg-ui_tabs

    Stateful elements would make life much easier: https://github.com/hypejunction/Elgg-ui_element_states. Current toggling of menu items is just bizarre. New AJAX API allows us to feed data to these stateful elements, and it would be easy to integrate with websockets if we ever get there.

    Menu items should be standardized. You wouldn't imagine the length of hooks that I have written just to replace all the stupid icon menu items with their text equivalent. We should better leverage menu item data params and pass things like icon, counter/indicator etc. Menu section markup needs work, unwrapped multiple ul tags are just plain bad. https://github.com/hypeJunction/Elgg-menus_api

    • Are we sacrificing anything by giving plugins too much power?

    Yes and no. Having recently worked with WordPress, Elgg's major advantage is that plugins can play well together. Unlike WordPress plugins, where code is breeding and producing more bad code, Elgg's code base is clean - plugins can work well together. We have standardized many things, and Elgg 2.x plugins are much easier to work with, so I think we should continue moving in that direction - we need more generic UI/UX elements that plugins can recycle and collaborate on.

    One area that is lagging behind is plugin integration testing. Hopefully, with all the work I have been doing on tests, we will get there eventually.

    • What Elgg-specific dev concepts are really cool or not worth the overhead for new devs to learn?

    We have too many intermediary steps for everything. I feel like I keep doing the same thing over and over again. We have too loose the pagehandlers, we have to automate breadcrumbs.

    Files API was a major source of annoyance, but it's a lot better now and makes it much easier to work with files.

    We need a better Forms/Actions API. Too much code duplication. Actions API especially is a permanent copy/paste. We should move towards reusable API shared by actions, web services, CLI et al scripts. We need better validation/sanity checks within entity classes, we need to standardize when river items are generated and when the notifications are sent. Currently there are too many examples of too many ways to do it. I want to be able to just save an entity and not worry about having to validate its properties, creating river items, sending notifications etc.

    • no video system. In a modern software, this is unacceptable;

    Not sure what a "video system" entails. You can't have a decent streaming service without a CDN and a dedicated server to transcode the videos. Current Files API allows you stream videos in a much more efficient way. https://github.com/hypeJunction/elgg_file_viewer

    • no decent mobile experience;

    We need a mobile first theme. We need to loose a lot of menus - there are too many menus with too many menu items that grow and grow. We need to rethink the navigation entirely.

    We need a proper responsive grid. elgg-col- madness is useless. 

    Switch to flexbox is essential.

    • groups are confusing (all content created in a group, should stay in that group when clicked. I should not be taken to the file, page, blog plugin. Which causes the user to have to navigate back to the group.);

    Same as above - too many menu items. Group pages are cluttered.

    • comment system is awful;

    https://github.com/hypeJunction/hypeInteractions

    • no membership levels: basic, premium etc.;
    • or administrator levels: moderators, site admins etc.;

    This is a general problem that we need to address. We need a better Capabilities API. Current ElggEntity::can* APIs are limited. We need a better role/capabilities system.

    • no way to "easily add CMS" pages that are not stripped of content because htmlawed will not allow it. I am talking trusted members (admins) adding content, not users; and

    It's a trade off between security and usability. We choose security, hence Elgg sites are not easily hacked - in fact I haven't heard of a single Elgg site that has been compromised.

    We need to revive ECML and implement a better shortcode system.

    • But we must however provide upgrade paths for the database and dataroot data. If those cannot be run by Elgg itself (booting engine depends on the data that must first be migrated), we should provide a separate CLI script or something to do the job.

    We should make CLI scripts a priority. I have started some work here: https://github.com/hypeJunction/elgg-cli

    In general Elgg upgrades are painful. Manually deleting obsolete files, upgrading one minor version at a time - all seems like a waste of time. We should start with upgrade scripts that can automatically wipe old files, upgrade scripts should be independent from releases etc.

    It should also be easier to upgrade the core and plugins from Admin interface. WP does a great job at that and we should learn. I have started some work here: https://github.com/hypeJunction/hypePluginInstaller. The idea is to allow various providers to hook into the upgrade process and provide info about available plugin versions and run the plugin install. We could do similar for Elgg core.

     

    As said before, too many idea, too little time, too few hands.

    After looking into WordPress, I am strongly convinced that Elgg is a superior PHP framework, and I don't understand why we lag behind so much. If we could implement at least some of the UX features found in WP, perhaps we would have a higher adoption rate and end up with more dev hands.

  • WordPress was good enough when Moveable Type imploded, it's easy for PHP beginners to write blog themes, its spam fighting system has always been steps ahead of anything else, wpautop() was magic, admin and upgrading continually improves, and in general Automattic have worked smart.

    But success to me would be a dozen devs excited to contribute, and for no one to wonder if it can scale.

  • Speaking of scalability - we need to get rid of that bloody friendspicker and rewrite the notifications settings pages.

  • i have been using the elgg fork in use at minds.com recently. currently minds is receiving something of an avalanche of traffic from people leaving facebook and is doing well. minds now has it's code on github and has already filled in many of the gaps in elgg such as realtime notifications and other UX issues. i raised the point that it may be a good idea to port some of those features back into elgg core from minds' code, but so far no-one replied to me - either here or in the github issues list in the minds repo. i know nothing about how challenging it would be to port these features from minds into elgg, but it surely looks like a good idea if it is do-able.

  • i raised the point that it may be a good idea to port some of those features back into elgg core from minds' code, but so far no-one replied to me

    OK, yes that would be nice. Without some overall guidance from the Minds team this would be extremely difficult, and in the very least it will require big new server requirements. Also their work is GPL and we would have to be able to make MIT releases without it (ouch).

  • i understand that help from their team would be needed and that's why i opened an issue in their repo. the issue with that, i feel, is that they are busy with bugfixing and other issues - however, they do apparently have some significant help from their own community with the coding now - so it may be possible to get the communication and assistance flowing.

    in the very least it will require big new server requirements

    i imagine you are talking about the software requirements rather than hardware requirements, since they are using several server technologies that elgg does not use at present. although it would be nice in some ways to avoid that, at the same time it may be impractical to expand elgg up to the level of 'modern day social network' without such an expansion.

    the licensing issue is something that is beyond my (imaginary) 'pay grade' ;) - though since we are discussing 'radical' changes to elgg (hypothetically) i will just add that ideally there would only be one license option in use.

This discussion is closed.

This discussion is closed and is not accepting new comments.

Feedback and Planning

Feedback and Planning

Discussions about the past, present, and future of Elgg and this community site.