Elgg at a crossroads

I'm excited about the upcoming Hackathon (Sept 23, 24), but this temporary burst can't make up for the fact that core dev contributions are down and have been for a long time.

This isn't anyone's fault. To an extent, this is normal cyclic behavior as people get busy with other things, but I rather think it's that Elgg is no longer exciting or fun enough to maintain developer interest.

Sure, we've made huge improvements in DX and performance, and 2.x provides a lot of value, but the product remains highly tethered to trends, technologies, and architectural decisions made 8 years ago. We have some loud voices in our community telling us that Elgg should change even more slowly (or not at all), and I've personally over-romanticized the idea of extreme backwards compatibility, but if we continue this path I think developer interest will keep dropping off.

Personally I really enjoy working with Elgg's contributors and don't want to stop, but for my career's sake I can't focus on reworking 2008-era APIs much longer. I considered quitting in 2014 and didn't out of inertia and because family health problems made it comforting work to bury myself in.

What can we do?

We could do nothing. This is just my perspective and I could be wrong or it could really not matter. There's nothing wrong with stable software that nudges along slowly. My prediction is that high maintenance chores will start to go by the wayside. Elgg 3.0.x will get dozens of releases and plugins will thrive as they did on 1.8. It's not a bad outcome for lots of people, but I'd be bummed.

In the short term, Ismayil and I should stop stalling new development in order to wait for a 3 member consensus. If a PR on 2.x/master gets a thumbs up and no one expresses concern or a desire to review for weeks, it goes in. Those branches give plenty of opportunity to revert or fix concerns that come up later. When reviewing involvement goes up, we can scale the need for consensus accordingly.

Businesses relying on Elgg could offer developer time. That won't help if devs really don't want to work on it though. And IMO business owners are under no moral obligation to contribute, but it would of course help.

We could seek out and destroy any low-return hassles that make working on Elgg less fun:

  • Commit messages are a constant headache, but may be worth it for our awesome changelog.
  • LTS releases: if so many sites want to run 1.12, one of their developers should put in the work of backporting fixes to it; merging this stuff into 3 other branches is a headache. Maybe the foundation can pay someone to make these releases occasionally.
  • Community spam: Maybe you can't register until you to contribute some sort of content; then we can just delete several thousand spam profiles.
  • RST is still !@#$. I'm continuously looking up how to do basic things and even the docs about RST are bad. Markdown is inferior tech with miles ahead UX. Really I want to do like Drupal and build more docs from source code.
  • ??

We should identify roadblocks that make more radical changes seem impossible. It was really disheartening to turn away a good 3rd party effort at moving to UTF8MB4 columns. Schema changes in general are almost unimaginable due to the expectation that Elgg run its own upgrades.

Personally I would love to--at least temporarily--set aside the requirements of upgrade paths and BC and start over.

  • How can Elgg be many magnitudes faster, not a few ms.?
  • How can it have the real-time functionality users now expect?
  • Are we sacrificing anything by giving plugins too much power?
  • What Elgg-specific dev concepts are really cool or not worth the overhead for new devs to learn?

In general I want Elgg to take a great leap forward to become something I'd be proud to put my name on, not filled with things I have to apologize about.

  • Thank you all for your feedback! I took a while to give my feedback because if you react immediately you can not always look at the big picture. Luckily i do not disagree with any of the previous comments. All feedback is very valid. As a long time user/developer/administrator of Elgg we have seen Elgg develop from its early beginnings. And it surely developed... a lot. I also agree Elgg should move forward, but, like @juho already said,  within our BC policies we have all the room for it.

    In the last 5 years of Elgg development i can not say that there have been significant UX changes (other than the Aalborg Theme). The focus was merely on framework / API features and that helped me as a plugin developer a lot! So thanks for that, but end users will never see the beauty of my code :( They only see the end result, so if i spend a week making plugins work a lot more efficient, or better compatible, if there are no visual changes they will think i have wasted my time.

    This kind of thinking is similar to how people look at Elgg (out of the box). They see a non changing product. What happend under the hood... stays under the hood. And that is a shame. With all the great capabilities that Elgg core currently has, we could build much more features.

    I am not saying it should be part of core!

    This is a long running debate... should we pack Elgg with more features out of the box? Or should it even be less..

    I think we should do both! There is no need for a full packed installation if you can easily install additional plugins. So we should focus on distribution of those plugins and make them installable from the admin interface. How? Do not know it yet, but it will give site administrators a better feel that they themselves can make Elgg how they want it to be.

    If you install (almost) all plugins manually as a site admin, there is no need for the plugins to reside in the elgg/elgg core repository. Plugins should go in a separate repo. This will have the benefit that we can version manage plugins, give people access to write to plugin repos (but not core). The downside of this approach will be that related commits between core and the plugins loose their connection, but maybe that can be resolved and on the other hand, i do not think it is that big of a deal.

    Documentation is work

    I personally do not think RST is so bad and with Read the docs we have a fancy, multilingual/translatable and indexed documentation site. However... choosing something else could have its benefits. Currently the documentation of Elgg is part of the installation, but without a RST server it can not be read from within Elgg... This feels a bit dumb... I would like to see documentation to be readable from within your own Elgg installation. With MD files we can easily do this. I also would like to see a feature where plugins could append their own docs to the Elgg documentation. This would end up with a full documentation of everything installed. We need to come up with a way to integrate plugin docs into the main Elgg docs, but that could be arranged. I think in the end it would be better if we also move the documentation to a plugin of its own, with a structure of MD resource files... Downside of that approach is that it is probably hard to make it translatable. But is personally think i can accept not having translatable documentation in favor of a well documented installation. If we use MD docs, we could just rely on the github presentation of the docs...

    Performance

    Is this seriously a big issue? Yes you could make Elgg faster by doing some changes... and we will do that, but performance is really not a big issue in my opinion. If you have loads of data or loads of page views you need to analyse your site. We do it all the time and in almost all cases the most benefit is in optimizing the plugins we use, not in core. Surely core can optimize, and it should and it will, but performance gains are a side effect from those changes... Take for example the denormalisation of the database... this is expected to bring performance gains, but the real benefit for Elgg is in reducing the complexity of Core. There are a few proposed changes that are db related that will reduce complexity of core code by a lot and those are:

    • removing object/user/group/site entity tables
    • denormalizing metadata
    • dropping metadata access levels
    • replacing friends access suffix with ACL

    These changes will make Elgg less complex at its heart and will have effect on various other endeavours. I think tackeling these will help to make bigger changes (like multiple access levels, or caching or Query objects, changes is ege_* functions) in the future much easier as the datamodel is easier to use.

    Search

    With the use of Elgg's default features, there will still be one performance problem and that is search. That can not be sped up if we want to be able to have a search that could work on any shared host. Luckily there are 2 plugins available that allow integration with ElasticSearch or SOLR. Those plugins will reduce all your search problems to zero, but you will have to install additional software to get it to work.

    Plugins

    I need to tell more about the benefits of Elgg plugins in their own repository. Currently if we BREAK something in a plugin that is part of core, like for example we change the urls for groups to group (because we want every url to be singular), this would be a BREAKING change... for the whole of Elgg core... and it would require a major release. Plugins do not get much attention because breaking them would mean we break a lot of sites, but also core in the proces. Moving them to seperate repos will allow us to break plugins, but not core. This will also encourage (core)plugin development i hope as that is something that is lacking at the moment. Core plugins should be an example of best pratices and have beautiful code... however they are most of the time the worst example to use as they are not updated that much.. Elgg has always been very hesitant towards adding new features to plugins... i think that is bad for the long run. The argument always was... there is probably a plugin that does it better, or make your own. I think that is bad... it discourages core (plugin) contributions. Elgg is good because of its plugins, not because of its own plugins... that is very unfortunate... a lot of good ideas should just end up in the core plugins. With separate repos we could organize and control these features a lot better.

    UX/UI

    Don't want to add much more to this... yes it should be better. There have been suggested a few good ideas, Just naming a few to summarize:

    • ajaxed tabs / pagination
    • realtime updating content (thewire / site notifications)
    • improve the menus so they support icons
    • adopt bootstrap (will make theming a lot easier and also provides the use of existing (non elgg) bootstrap themes available)
    • rethink navigation (breadcrumbs are always an issue)

    Organisation

    I feel that currently there is noone "in control" of Elgg...Not the foundation, not the core developers and not the community. Previous comments about a community council of some sort let us know that there are community members that are willing to participate in such a thing. I think having a formalized Advisory Board is not such a bad thing. It could advise core developers to focus on certain developments, aggregate community discussion into valuable feedback for the general of Elgg. It would suggest that the community organize themselves in such an Advisory Board and try to give a 2 monthly report and publish that report on the Elgg community. At the same time Core Developers should talk about their plans with Elgg and formalize a roadmap. Not with dates... not with specific features... but with a general direction. With a global roadmap (inspiration here: http://stonehearth.net/roadmap/) we can keep focus as developers but also communicate easy about the long term plans... Maybe a big database migration is just around the corner... but if we do not communicate about it no one will know.... The github issue are just the detailed tasks for the roadmap.

    Communication

    While Elgg core teammembers are active on github or on the elgg community, there is a lack of communication. There should be more PR. The problem is that we have no good PR people on board AFAIK. This should be improved (together with the Elgg website). If there are community members that are willing to take on the challenge of being the first PR guy/girl for Elgg let us know.

    Elgg Plugin Shop

    A paid plugin repository is something Elgg was always very hesitant against. I can understand why, but i can also think of lots of reasons why this could be beneficial to Elgg.

    • Having a paid plugin repo will encourage devs to create plugins for Elgg
    • Having a repo on the elgg community will keep non-elgg sanctioned repos out of the picture
    • Elgg could get a commision for each sold plugin
    • Plugin devs would want core to be better, so they can easier maintain their plugins, thus core contribution could increase

    This would bring some extra challenges:

    • how to handle payments
    • how about quality control
    • plagiarisme
    • paid vs free

    I think this is really worth thinking about, but requires a better organisation first.

    Summary

    @ismayil said:

    I think the main job of the core is to make things possible

    With this i agree. The core code of Elgg should make things possible, but Elgg as an end product should be much better than in its current state, so there should be just as much attention go to UX/UI as it goes to core. In this we are lacking. Maybe it is because we are to much developers that care about the internals and like to delve deep into the Elgg dungeons to find some hidden gems. But without a nice frontend nobody will every see the beauty of Elgg. Because that is what i still believe, that from a developers perspective Elgg is still the best tool for building social networks, so i hope we can bring that developer experience all the way to the frontend. If there are people that are willing to help in this part, you are very welcome!

  • Elggers!

    The irony of this situation:
    - you want many wonderful real-time and other features but you complain about fast realease cycle
    - you want to have a plugin-based framework but you want to stuff the core of it with everything

    We can't build a new house without breaking the old foubdation it is built on.

    We can't give you a fully featured car that has extra room in the trunk.

    The way I see it, unless you jump in and do something to contribute, it will be Steve and I pushing through our vision, and this will be an endless discussion on trying to please everyone.

    So instead of us having this back and forth, please formulate your thoughts in a convincing set of arguments in favor of a specific feature, lay down your ideas about its implementation and design, how it fits into the bigger picture, why should it be in core and not a plugin, why should it break existing sites, and open on issue on Github. Both Steve and I get easily excited, so we might as well pick up on your reasoning and make it happen. Otherwise all this would be nice to have and Elgg sucks discussion creates a bad impression for people visiting Elgg community for the first time.

  • A paid plugin repository is something Elgg was always very hesitant against.

    One reason for this is that the community is hosted by http://osuosl.org/ and their terms of use allow only non-commercial use. Opening community to commercial services requires switching to a new host.

  • i think the decision to focus on UI/UX/visuals is partially for users of elgg sites and partially to draw in new technical people who are inspired to use elgg and who thus then become able to help expand the core too. i'm sure there are a few people like me who could put more commits in to core elgg code but who find it difficult, due to other commitments, to dedicate timespace to doing that - so who generally stick to releasing plugins from time to time.. however, i am also sure that by attracting more general desire on the internet for using elgg in general, there would be more developers available who do have enough timespace to work on core too.

    so i'm not really adding anything here, other than to just reiterate that i feel the most significant gains will come from improving the 'eye candy' end of things. that said, i cannot guarantee that in any way.. maybe people are just too preoccupied with pokemon go :/

  • If someone wants to design/donate a new theme, put your graphic proposal on Github, let's take a vote and I will turn it into a theme. End of discussion.

  • - you want many wonderful real-time and other features but you complain about fast realease cycle
    - you want to have a plugin-based framework but you want to stuff the core of it with everything

    I like the fact that there are more releases than ever before. It is good. Just hear me out:

    I know there's a lot of work and effort with each release. Steve is right, I am only seeing the surface, but I am fully aware of the internal changes, which as of right now, elgg has become faster. It is not much of a problem, it is just... don't take it the wrong way, but from the outside we see constant releases but we don't see "surface features" such as real-time River (I know it has been accomplished with several plugins). So, it is more of a perception problem, and that perception can lead to the really bad "Why do I need to upgrade? It is not worth it" or the "You are releasing too fast, I don't have time for that"

    We have bootstrapped themes, and several high quality themes and some themes that completely reworked the site menu. Yet, it all stays within the plugins. I'm not saying it is bad. I'm not saying "Let's pour every shiny thing I can think of". Just start adding small things, one thing at a time.

    If we could come to an agreement and create a roadmap, what could possibly be the first goal for example?

    Real Time River Stream? When is gonna be added? Whenever is ready. I do not want things now. Coding is hard and maintaining elgg is an exhausting task and I'm glad you put time and effort for that. That's just one example, not suggesting to do that now.

     

  • Personally, it's pure the way Elgg is written. It doesn't have any pattern whatsoever, it's built from scratch without sociability and flexibility in mind. Although, this is currently chancing, it'll only take so long before it reaches that point where developers will start contributing/using it again, without a reason. Currently, most of the people I see around here are from the old ages, where Elgg was still "the shit", which it isn't, anymore.

    I would prefer to hop on a completely different path: get a framework (Laraval, CakePHP 3, Zend, Symfony) and start from scratch (although, with the current features in mind). This will have many benefits (a lot, and a lot, of plugins, if you choose the right framework, e.g. Laraval or CakePHP 3), event handling, job queues (Gearman? RabbitMQ?), notifications built-in, all kinds of mailing advantages and last but not least: a platform developers will built on. With ease. Oh, and the database structure normalized against the frameworks standards. ACL and denormalized metadata (and no metadata access levels).

    I've been really struggling to get started with Elgg, and still am in some points, like the way all files are organized or how certain things are implemented. I'm trying to contribute as much as I can, but most of the PR's/issues are still left open, denied or merged at some point, which doesn't really motivate to contribute.

    And the template, yeah. Same as above, adapt the modern tools, and developers will be more happy to contribute. The same for tests, way to important to skip those.

     

  • i think the issue goes further than just designing a new theme - although that would be good. there are probably already good enough themes in the community (although i don't know of any since i don't use them or review them often). the other issue is related to the upkeep of basic features. e.g. when i upgraded from 1.8/1.9 through to 2.1, there were many features that got lost due to plugins not being upgraded along with the core. it took me probably a month of solid coding to get them up to date (i use a lot of plugins), but since some of these plugins are for the kinds of features that are considered basic, core features for social software in 2016 (and since maybe 2013) i think we have some kind of agreement here already that the code would be ideally held within the elgg core. moving, say, galliComments (for ajaxed commenting) and hypeLists (for ajaxed lists) into core would tick several of the boxes this thread has raised:

    • elgg not being up to date with competing products
    • plugins for basic features not being known or used by all who would need them and so interest in elgg dropping off.
    • plugins not being upgraded, holding back core upgrades on live sites and thus limiting plugin development from those sources too.

    hypeInteractions may also be suitable, i haven't used it though, so i'm not sure. having these in core would ensure that the base level of functionality in elgg is raised to a new standard/level and i think that is the aim here. if this were combined with a new theme to package with elgg and maybe to use in the elgg community here, all the better.  n.b. i just went to scan through the themes but there are no screenshots available in the list view which makes navigating themes much more challenging.

  • I work on a hobby project, for 5 years now, originated in 1.8, running on 2.2 now. I still haven't launched. Because I have this urge to do as much as I can, alone and I want very specific things, but no budget.
    I couldn't code one line when I started but I am making my own plugins for my project myself now.This was a very bumpy road, since there were so few people to help out, and Elgg had so few of my requests in core. So I had to rely on github, the help I got here, learning PHP myself and I read cash's 1.8 book.

    I gave up multiple times, and took a restart multiple times. But although I don't post very often, I watch the discussions here every day.

    What I do want to say is that since we have proper docs, starting with 1.9 I think, I really started to understand things better, and I can say I'm able to do most things , though slowly, on my own now.

    If I may, I say a few things on what was written here and try to be constructive.

    First:
    My own contribution

    I am really, really willing to do a lot of work for Elgg, wherever I can. I'm thankful of what I got from this community
    The problem is that I am just beginning to feel comfortable in coding, for the first time I'm feeling that I can build a plugin from scratch and do most things myself. I use github a lot to see how core handles things.
    But I'm very afraid of doing things 'not the best, or most efficient way'. It's a burden for me to do something because I feel that the dev reviewing my code will probably do it different and has to rewrite it. If you're up with that. I'll start contributing.

    Community site

    I do think this is a problem for arriving newcomers.
    Although Aalborg was a big leap, it still looks a bit 'unfinished'. And I must say, adding the elgg.org topbar on top of the community topbar didn't do it good.
    You'd be surpised what some subtle gradations of one colour + some boxshadow, and a modern font could do in a here.
    I'l post a screenshot of the wall of my project. immature and unfinished, but this is basically Aalborg, but with a lot of small colour changes. But it's clean and looks more modern to me then the community.
    http://www.daltonstudio.be/garbage/screenshot.png

    If my help on this is wanted. I'd be glad to help, really.

    Elgg.org

    This is really not good, it looks like it's from the 90's. I can imagine people clicking away instantly when they see this and don't even bother to look how great the engine actually is.

    I'm am certainly willing to work on this. I think we should really integrate this into the community site, instead of the half-half solution we have now

    Instant notifications

    Although I agree it's a must-have, this is a difficult one to achieve,  don't we need a push server? Will this work for the numerous shared hosting people we have?
    I would rather get rid of the site-notifications and use something like Juho's Notifier instead.

    Leave the instant/push to plugin authors, like Juho is doing now with his notifier plugin.
    This could be wonderfull if plugin authors could easily hook into the notifications and deliver them the way they want (like it is now, basically) There are multiple push services who have a very good API to build on.

    Ajax

    • This can help to make it a smoother experience. Everything is in place to do this.
      The team already did a great job here. But there are just small things that could easily be ajaxified
      I'm trying to make every action in my project as ajaxified as I can with a normal action fallback.
    • Together with a new modern SVG spinner (instead of the basic gif elgg comes with) could give the user experience a big boost. People just like spinners, it feels everything happens on the background and it's working, not for pageload of course, but for saving actions.
    • Inplace editing, it's wonderful to a UX and not really difficult to implement + it would break anything.

    Plugins

    • A admin installer would be great. I know Pawel and Ismayil worked on this.
      We should think this over, how and what.  I love the way Wordpress handles this.

    Commercial plugins

    If the reason not to do this, is the hosting. We should host this shop somewhere different.
    I am definitely willing to pay for a quality plugin if this means that it will be updated.
    If Elgg could get a commission for every sale the hosting could be payed with this.
    I agree this could attract developers to make plugins that could matter and where Elgg could benefit from.

    Bundled plugins

    • The bundled plugins we have are a bit strange, like said before. I've never gotten why there is a wall and a wire and why they are not just one thing.
    • Bookmarks: I do not think many users use this, people want to bookmark in their browser and have it in sync with all their devices.
      If it's about sharing. We need a sharing mechanism, like facebook has. Just share an entity's link on the wall. That's it, could be a small lightbox where they could add a message and press submit ->action will post it on activity.
    • Pages: meh, I would know what to say about it other then that you could blog as well.

    Plugins that should be bundled
    or actually, be in core if you ask me

    • Hypewall or simmilar: This is just what users expect of a social network, just share your thoughts from the timeline, blame facebook.
    • A decent friendspicker.
    • Infinite listing (like hypelists provides)
    • Lazy loaded images with placeholders (good for UX too).

    Theming

    I'm pro moving to bootstrap. However, will this implicate that every elgg site will have to completely redesign their themes? If that's the case I don't know if I'm still pro.
    Breaking things is ok, but this is breaking a lot, if not everything. I don't know how you guys see this.

    Admin

    We got all 2.0 with Aalborg, but we forgot the admin.
    It makes you think you're working with 90's software, although it isn't.

    I can work on that if you'd like.

    I'll stop, don't think anyone is still reading anyway.
    Thanks for all so far Elggers.

  • Can you guys see all of the beauty of this discussion? Elgg is alive and the best devs are stil here willing to do better and more. Those who left...well they're missing it and maybe they'll come back.

    Love

This discussion is closed.

This discussion is closed and is not accepting new comments.

Feedback and Planning

Feedback and Planning

Discussions about the past, present, and future of Elgg and this community site.