Elgg at a crossroads

I'm excited about the upcoming Hackathon (Sept 23, 24), but this temporary burst can't make up for the fact that core dev contributions are down and have been for a long time.

This isn't anyone's fault. To an extent, this is normal cyclic behavior as people get busy with other things, but I rather think it's that Elgg is no longer exciting or fun enough to maintain developer interest.

Sure, we've made huge improvements in DX and performance, and 2.x provides a lot of value, but the product remains highly tethered to trends, technologies, and architectural decisions made 8 years ago. We have some loud voices in our community telling us that Elgg should change even more slowly (or not at all), and I've personally over-romanticized the idea of extreme backwards compatibility, but if we continue this path I think developer interest will keep dropping off.

Personally I really enjoy working with Elgg's contributors and don't want to stop, but for my career's sake I can't focus on reworking 2008-era APIs much longer. I considered quitting in 2014 and didn't out of inertia and because family health problems made it comforting work to bury myself in.

What can we do?

We could do nothing. This is just my perspective and I could be wrong or it could really not matter. There's nothing wrong with stable software that nudges along slowly. My prediction is that high maintenance chores will start to go by the wayside. Elgg 3.0.x will get dozens of releases and plugins will thrive as they did on 1.8. It's not a bad outcome for lots of people, but I'd be bummed.

In the short term, Ismayil and I should stop stalling new development in order to wait for a 3 member consensus. If a PR on 2.x/master gets a thumbs up and no one expresses concern or a desire to review for weeks, it goes in. Those branches give plenty of opportunity to revert or fix concerns that come up later. When reviewing involvement goes up, we can scale the need for consensus accordingly.

Businesses relying on Elgg could offer developer time. That won't help if devs really don't want to work on it though. And IMO business owners are under no moral obligation to contribute, but it would of course help.

We could seek out and destroy any low-return hassles that make working on Elgg less fun:

  • Commit messages are a constant headache, but may be worth it for our awesome changelog.
  • LTS releases: if so many sites want to run 1.12, one of their developers should put in the work of backporting fixes to it; merging this stuff into 3 other branches is a headache. Maybe the foundation can pay someone to make these releases occasionally.
  • Community spam: Maybe you can't register until you to contribute some sort of content; then we can just delete several thousand spam profiles.
  • RST is still !@#$. I'm continuously looking up how to do basic things and even the docs about RST are bad. Markdown is inferior tech with miles ahead UX. Really I want to do like Drupal and build more docs from source code.
  • ??

We should identify roadblocks that make more radical changes seem impossible. It was really disheartening to turn away a good 3rd party effort at moving to UTF8MB4 columns. Schema changes in general are almost unimaginable due to the expectation that Elgg run its own upgrades.

Personally I would love to--at least temporarily--set aside the requirements of upgrade paths and BC and start over.

  • How can Elgg be many magnitudes faster, not a few ms.?
  • How can it have the real-time functionality users now expect?
  • Are we sacrificing anything by giving plugins too much power?
  • What Elgg-specific dev concepts are really cool or not worth the overhead for new devs to learn?

In general I want Elgg to take a great leap forward to become something I'd be proud to put my name on, not filled with things I have to apologize about.

  • I think the main job of the core is to make ithings possible, to provide API for plugins to implement features and needs that vary from site to site. We can't stock everything into core - on the contrary there is too much overhead with the bundled plugins. Do we really need blogs and pages together? Do we really need the wire and the inferior messageboard? Why do we need bookmarks - can't people jusy use their browser bookmarks or write a blog post if they want to share an article?
    I agree that all these post types are ambiguous - we should instead develop a rich post interface that reflects the needs of modern social networks - ability to attach content and rich media, out of the box mentions (users, groups and other local "bookmarks") without absolute URLs (ECML to the rescue).
    We should focus on making admin experience better: there should be a visual toggle for all config values. It should be easier to modify site/footer menus and link to static pages (get rid of expages in favour of anypage and improve it).
    Instead of bundling plugins we could create recomended composer json variations in the starter project that would build a project for specific use cases.

  • If this is not the appropriate place, I apologize and let me know I'll move it or whatever.

    Not to continue beating a dead horse or continue bashing, but I am going back to reread all of these comments made here after I write this, but I am so frustrated (been so for almost a year now) that I felt a compulsion to immediately vent.  I apologize in advance, but this hit home. BTW, I am glad this discussion has been started because I have a very bitter taste in my mouth from my experience.  I am not sure of the exact problem in the elgg system causing my issues, but as a user, yes, we just want something that works and NOT AT A CRAWL to a page load of 4-5 minutes per Event Calendar query (before you start thinking its bad code, it was developed by a programmer here with a lot of elgg experience and plugins). Could it be the database structure? Maybe...in my limited knowledge of the elgg database, I am still inclined to think so.

    My project is of an Event Calendar with copious data and mapping.

    I was a very excited user because of this project, hired someone from the elgg group to customize his event calendar for me back when it was I think 1.18 or something.  I just wanted something to work and it did for small amounts of data.  But after spending over $17,000, through 2 programmers ($6000) and 1 mobile app company ($11000), I still DO NOT have a functioning site.  It has now been 3 years since I started.

    The version updates and lack of backwards compatibility is part of my frustration.  I am not able to keep the plugin up to version updates. UGH. But I surrendered to stopping at that version (because of the event calendar plugin in particular), which I didn't mind doing, because I just wanted something that worked.  It did work at first, with small amounts of data.  I didn't expect for the event input volume to gravely affect the download time of its pages.  More UGH. I am at a loss of where to turn.  My site(s), after several iterations, and plugin minimization, reinstalls, the event calendar pages take literally 4-5 minutes to load per request even just out of the box, after about 2 weeks of constant event inputting -- and the list of event being queried and mapped on the page is just 20 at a time! Even more UGH. Plus, new versions do not work with the event calendar plugin I had built, so do not know if it would improve anything.

    That prompted me to hire that company to do a mobile app to develop something that could work with the elgg database, workaround it's super sluggish front end, and instead going through the backend developing an apk to handle the data directly.  It is faster, but still slow.  I had to stop on the mobile app because I didn't want to spend any more money on something until I could figure out what is the problem, fix it or whatever.  I don't think I can use the elgg system except MAYBE as a core collection tool for event data, but it is still sluggish. 

    I guess this is more of a rant than usable input, I am by no means an elgg expert or elgg programmer, but I felt it was at least some feedback from a user that just want something to work. As rjcalifornia said, we just need something that works and isn't outdated like mine is after just a couple years because customized plugins fail to work and it is costly to maintain them every time there is an enhancement.

    So I guess I am in the market to discuss yet another hire from here to help me.  If interested shoot me a message.  I already went through Freelancer and that other one, but I guess it wouldn't hurt to try looking for a programmer here.  Perhaps over there they aren't as familiar with elgg (particularly the database) as they say they are.

    Peace, Kepa

  • @Kepa I'm sorry about your experience. We're happy to help you--or your developer--via the community, but let's leave it out of this thread. Post as much detail as possible to a new topic in the Performance group. Particularly helpful would be access to MySQL's slow query log. How long do requests take for pages with little overhead, like https://example.org/admin/statistics/server ? What's the Elgg version? Is 4-5 minutes under no load or with lots of users online? Again, post these answers elsewhere :) -- Edit: I'll just add that Elgg's data model is fine for some things, but generally you'll need to use custom tables to get better performance, either to denormalize, cache expensive operations, or just match the data needs better.

  • @Kepa could it be that you asked me once about your problem with the Event Calendar plugin? It rings some bells. My answer probably included that I don't do custom works as I only code as a hobby. But I think I also made some guessing about the reason for the long page loads (which seemed to happen only with your modified version but not with the unmodified version of Event Calendar). As far as I remember your problem you are using a modified version of the Event Calendar plugin with some Google Maps feature added. It worked fine with only a few events but no longer with many events. I think the problem is that there's a query made to the Google Maps server for every existing event on every Event Calendar page load. And this will simply not work in any way like this. The number of queries must be reduced significantely, either by only making a query when viewing a single event entry in full view of at least to limit the queries to events currently displayed in the event list of the current page.

  • for anyone who is thinking about UX in social sites, this new video from devtips is a detailed look at site design using the evolution of facebook as an example: https://www.youtube.com/watch?v=UmKD1ciQg2A

  • I am not able to keep the plugin up to version updates. UGH. But I surrendered to stopping at that version

    This. I know there are several elgg powered sites/apps that won't upgrade their elgg version. We have an SaaS app and we decided not to upgrade versions anymore. At one point we were keeping up with each release, but we were mismanaging development time when we focused on upgrading every time a new version was released.

     

    Actually I meant Ismayil's Wall+extended tabs we funded and shared with community. Unfortunately Elgg's (river? wire?) is not really enough for users looking for a modern interface/experience.

    @Michele Sorry, I think I didn't explain myself. The problem is that none of the features of Ismayil's Wall have been implemented in the core version, which is a shame. We are falling behind times.

     

     

     

  • Thank you all for your feedback! I took a while to give my feedback because if you react immediately you can not always look at the big picture. Luckily i do not disagree with any of the previous comments. All feedback is very valid. As a long time user/developer/administrator of Elgg we have seen Elgg develop from its early beginnings. And it surely developed... a lot. I also agree Elgg should move forward, but, like @juho already said,  within our BC policies we have all the room for it.

    In the last 5 years of Elgg development i can not say that there have been significant UX changes (other than the Aalborg Theme). The focus was merely on framework / API features and that helped me as a plugin developer a lot! So thanks for that, but end users will never see the beauty of my code :( They only see the end result, so if i spend a week making plugins work a lot more efficient, or better compatible, if there are no visual changes they will think i have wasted my time.

    This kind of thinking is similar to how people look at Elgg (out of the box). They see a non changing product. What happend under the hood... stays under the hood. And that is a shame. With all the great capabilities that Elgg core currently has, we could build much more features.

    I am not saying it should be part of core!

    This is a long running debate... should we pack Elgg with more features out of the box? Or should it even be less..

    I think we should do both! There is no need for a full packed installation if you can easily install additional plugins. So we should focus on distribution of those plugins and make them installable from the admin interface. How? Do not know it yet, but it will give site administrators a better feel that they themselves can make Elgg how they want it to be.

    If you install (almost) all plugins manually as a site admin, there is no need for the plugins to reside in the elgg/elgg core repository. Plugins should go in a separate repo. This will have the benefit that we can version manage plugins, give people access to write to plugin repos (but not core). The downside of this approach will be that related commits between core and the plugins loose their connection, but maybe that can be resolved and on the other hand, i do not think it is that big of a deal.

    Documentation is work

    I personally do not think RST is so bad and with Read the docs we have a fancy, multilingual/translatable and indexed documentation site. However... choosing something else could have its benefits. Currently the documentation of Elgg is part of the installation, but without a RST server it can not be read from within Elgg... This feels a bit dumb... I would like to see documentation to be readable from within your own Elgg installation. With MD files we can easily do this. I also would like to see a feature where plugins could append their own docs to the Elgg documentation. This would end up with a full documentation of everything installed. We need to come up with a way to integrate plugin docs into the main Elgg docs, but that could be arranged. I think in the end it would be better if we also move the documentation to a plugin of its own, with a structure of MD resource files... Downside of that approach is that it is probably hard to make it translatable. But is personally think i can accept not having translatable documentation in favor of a well documented installation. If we use MD docs, we could just rely on the github presentation of the docs...

    Performance

    Is this seriously a big issue? Yes you could make Elgg faster by doing some changes... and we will do that, but performance is really not a big issue in my opinion. If you have loads of data or loads of page views you need to analyse your site. We do it all the time and in almost all cases the most benefit is in optimizing the plugins we use, not in core. Surely core can optimize, and it should and it will, but performance gains are a side effect from those changes... Take for example the denormalisation of the database... this is expected to bring performance gains, but the real benefit for Elgg is in reducing the complexity of Core. There are a few proposed changes that are db related that will reduce complexity of core code by a lot and those are:

    • removing object/user/group/site entity tables
    • denormalizing metadata
    • dropping metadata access levels
    • replacing friends access suffix with ACL

    These changes will make Elgg less complex at its heart and will have effect on various other endeavours. I think tackeling these will help to make bigger changes (like multiple access levels, or caching or Query objects, changes is ege_* functions) in the future much easier as the datamodel is easier to use.

    Search

    With the use of Elgg's default features, there will still be one performance problem and that is search. That can not be sped up if we want to be able to have a search that could work on any shared host. Luckily there are 2 plugins available that allow integration with ElasticSearch or SOLR. Those plugins will reduce all your search problems to zero, but you will have to install additional software to get it to work.

    Plugins

    I need to tell more about the benefits of Elgg plugins in their own repository. Currently if we BREAK something in a plugin that is part of core, like for example we change the urls for groups to group (because we want every url to be singular), this would be a BREAKING change... for the whole of Elgg core... and it would require a major release. Plugins do not get much attention because breaking them would mean we break a lot of sites, but also core in the proces. Moving them to seperate repos will allow us to break plugins, but not core. This will also encourage (core)plugin development i hope as that is something that is lacking at the moment. Core plugins should be an example of best pratices and have beautiful code... however they are most of the time the worst example to use as they are not updated that much.. Elgg has always been very hesitant towards adding new features to plugins... i think that is bad for the long run. The argument always was... there is probably a plugin that does it better, or make your own. I think that is bad... it discourages core (plugin) contributions. Elgg is good because of its plugins, not because of its own plugins... that is very unfortunate... a lot of good ideas should just end up in the core plugins. With separate repos we could organize and control these features a lot better.

    UX/UI

    Don't want to add much more to this... yes it should be better. There have been suggested a few good ideas, Just naming a few to summarize:

    • ajaxed tabs / pagination
    • realtime updating content (thewire / site notifications)
    • improve the menus so they support icons
    • adopt bootstrap (will make theming a lot easier and also provides the use of existing (non elgg) bootstrap themes available)
    • rethink navigation (breadcrumbs are always an issue)

    Organisation

    I feel that currently there is noone "in control" of Elgg...Not the foundation, not the core developers and not the community. Previous comments about a community council of some sort let us know that there are community members that are willing to participate in such a thing. I think having a formalized Advisory Board is not such a bad thing. It could advise core developers to focus on certain developments, aggregate community discussion into valuable feedback for the general of Elgg. It would suggest that the community organize themselves in such an Advisory Board and try to give a 2 monthly report and publish that report on the Elgg community. At the same time Core Developers should talk about their plans with Elgg and formalize a roadmap. Not with dates... not with specific features... but with a general direction. With a global roadmap (inspiration here: http://stonehearth.net/roadmap/) we can keep focus as developers but also communicate easy about the long term plans... Maybe a big database migration is just around the corner... but if we do not communicate about it no one will know.... The github issue are just the detailed tasks for the roadmap.

    Communication

    While Elgg core teammembers are active on github or on the elgg community, there is a lack of communication. There should be more PR. The problem is that we have no good PR people on board AFAIK. This should be improved (together with the Elgg website). If there are community members that are willing to take on the challenge of being the first PR guy/girl for Elgg let us know.

    Elgg Plugin Shop

    A paid plugin repository is something Elgg was always very hesitant against. I can understand why, but i can also think of lots of reasons why this could be beneficial to Elgg.

    • Having a paid plugin repo will encourage devs to create plugins for Elgg
    • Having a repo on the elgg community will keep non-elgg sanctioned repos out of the picture
    • Elgg could get a commision for each sold plugin
    • Plugin devs would want core to be better, so they can easier maintain their plugins, thus core contribution could increase

    This would bring some extra challenges:

    • how to handle payments
    • how about quality control
    • plagiarisme
    • paid vs free

    I think this is really worth thinking about, but requires a better organisation first.

    Summary

    @ismayil said:

    I think the main job of the core is to make things possible

    With this i agree. The core code of Elgg should make things possible, but Elgg as an end product should be much better than in its current state, so there should be just as much attention go to UX/UI as it goes to core. In this we are lacking. Maybe it is because we are to much developers that care about the internals and like to delve deep into the Elgg dungeons to find some hidden gems. But without a nice frontend nobody will every see the beauty of Elgg. Because that is what i still believe, that from a developers perspective Elgg is still the best tool for building social networks, so i hope we can bring that developer experience all the way to the frontend. If there are people that are willing to help in this part, you are very welcome!

  • Elggers!

    The irony of this situation:
    - you want many wonderful real-time and other features but you complain about fast realease cycle
    - you want to have a plugin-based framework but you want to stuff the core of it with everything

    We can't build a new house without breaking the old foubdation it is built on.

    We can't give you a fully featured car that has extra room in the trunk.

    The way I see it, unless you jump in and do something to contribute, it will be Steve and I pushing through our vision, and this will be an endless discussion on trying to please everyone.

    So instead of us having this back and forth, please formulate your thoughts in a convincing set of arguments in favor of a specific feature, lay down your ideas about its implementation and design, how it fits into the bigger picture, why should it be in core and not a plugin, why should it break existing sites, and open on issue on Github. Both Steve and I get easily excited, so we might as well pick up on your reasoning and make it happen. Otherwise all this would be nice to have and Elgg sucks discussion creates a bad impression for people visiting Elgg community for the first time.

  • A paid plugin repository is something Elgg was always very hesitant against.

    One reason for this is that the community is hosted by http://osuosl.org/ and their terms of use allow only non-commercial use. Opening community to commercial services requires switching to a new host.

  • i think the decision to focus on UI/UX/visuals is partially for users of elgg sites and partially to draw in new technical people who are inspired to use elgg and who thus then become able to help expand the core too. i'm sure there are a few people like me who could put more commits in to core elgg code but who find it difficult, due to other commitments, to dedicate timespace to doing that - so who generally stick to releasing plugins from time to time.. however, i am also sure that by attracting more general desire on the internet for using elgg in general, there would be more developers available who do have enough timespace to work on core too.

    so i'm not really adding anything here, other than to just reiterate that i feel the most significant gains will come from improving the 'eye candy' end of things. that said, i cannot guarantee that in any way.. maybe people are just too preoccupied with pokemon go :/

This discussion is closed.

This discussion is closed and is not accepting new comments.

Feedback and Planning

Feedback and Planning

Discussions about the past, present, and future of Elgg and this community site.