Getting to a 0-overhead state

As I mentioned in my "Road Ahead" post, one of the main pain points I feel as a core developer is the amount of overhead that goes into managing the project. Cutting releases, push changes to the community site, uploading new versions of my plugins when I change something. I really don't care about doing any of these things. These are all overhead. Overhead is just so discouraging to me. Waiting time between releases due to overhead is maddening.

Is that just life? I don't think so. I think we can get to a place where the only thing developers have to do is develop software, and computers take care of the overhead. All of it.

Continuous Deployment

I mentioned managing the community site is a hassle. We have some scripts that make it nicer but currently the workflow is like so:

  1. Set up a local install of the community (yikes, see last point for details)
  2. Develop and test feature/fix on new branch
  3. Submit a PR
  4. Travis runs tests (maybe)
  5. Merge the PR when Travis is happy
  6. Log on to Elgg's servers and run an update script for the particular plugin you want updated (overhead)

If the process is upgrading the community's Elgg version, there's quite a lot more hassle. The 1.8 process was a nightmare and I would have ended up leaving the community in shambles if Cash hadn't been there. He definitely saved the day. I'm honestly not looking forward to 1.9 with the data migration.

Step 6 seems small but it's surprising how much of a hassle this actually is when you have to do it for every single change. We should definitely do this for the community's (custom) plugins. What I'd also very much like to see post-1.9 is putting the community site at Elgg's HEAD and keep master stable enough to auto-update the community with every change to Elgg. Brett, you mentioned this might be too risky:

The core team doesn't develop this way, so I don't think that this is a desirable goal currently.

I think we're closer to this goal than you realize. Travis is almost always happy on master, for example. Besides, I'm only suggesting that we do this for the, not (necessarily) all sites everywhere. I feel like "Continuous Deployment" is becoming more popular and for good reason:

  • Incentivizes good automated testing discipline
  • Immediate reward for developer contributions
  • Early production testing on == more stable tagged releases for the world

That last point is key, I think. We will be able to ship releases with higher confidence (not complete confidence by any means, since the community doesn't use all the features that come with Elgg, but higher confidence for sure).

For outside examples of this idea: The Angular team has been encouraging folks to live at Angular's HEAD so they can get early feedback on any unintentionally breaking changes. This is voluntary and those engaging with it are aware of the risks, but Angular has a good testing process so it's pretty stable. Every team at Google participates in this. It really does work and is great because there's no waiting for the latest fixes and features.

If we're not stable enough to continuously deploy to the community site I'd like to know what needs to change and make those changes. I think everyone will be very pleased with the results. The problem with letting master be "unstable" as we have in the past is that it allows us to check in unfinished features which then blocks releases because someone must go in and finish that feature before we ship. The motivation might not be there, unfortunately. This might have made sense when the project was funded and people were being paid to work on it, but I don't think we can sustain that model with an all-volunteer team.

Is there a positive argument for making master unstable that I'm missing?

Automated Releases

Releasing Elgg is hard, and takes a long time each time it feels like. It's work that isn't feature development or bugfixing. It's just overhead. We've changed our commit message format to take the pain of Changelog generation away. I'm extremely excited about that -- we just need to integrate the generator now. But we can do more. If we keep master as stable as I'm suggesting then we can be one step closer to having fully automated and regular releases. We would not have to wait years between stable releases for new features. That's just too long.

Ember is on a 6 week cycle. Angular does 1.2.x releases every week or two. Angular doesn't even blog about releases anymore because that's too much manual effort at that kind of regular pace and the name of the game is 0 overhead.

If we did automated (and unannounced) releases every two weeks, we could just do highlight blogs once a quarter to thank contributors and such. This is what the Angular team does monthy with their meetups. Would there be only a few changes in each release? Yup. But so what? If it's not worth it for sites to upgrade, they can wait for more changes to roll in on the following weeks (just like we're forcing them to do now by "blessing" releases far less often). There's nothing wrong with bumping version numbers.

Local environment setup

I mentioned developing for the community is not super duper fun because it's rather difficult to get an exact replica working locally. By "rather difficult" I simply mean "more work than `git clone ... && composer install`". If we could get our development setup time down, that would certainly be an exciting thing for me. I just wouldn't have to think anymore. Thinking is so hard...

Streamlining local development, I acknowledge, is probably a ways off. But I think we should shoot for it sooner rather than later.

Releasing plugins to the community

This was so painful that I eventually just stopped bothering. Seriously. I don't care that much about my plugins to zip the package correctly and come here and upload it and fill in duplicate information again and add the screenshots, etc etc. Back in the day when I was being paid to work on an Elgg project and was single without kids? Sure. But no more.

Upgrading Elgg for arbitrary Elgg-based sites

If we really do start releasing regularly, the upgrade process is going to be just horrific to do that often. Announce downtime, take it down for maintenance, copy zip file, ftp to server, unzip on top of existing release, hit "upgrade" button. Ugh. This is why things like softaculous exist. We should have a better story here of staying up-to-date with bugfix releases at least. I think we can do this more securely than wordpress, but they definitely have a good UX here.

Something I missed?

What do other folks think? Are there any other pain points regarding "overhead" that I haven't addressed?

Feedback and Planning

Feedback and Planning

Discussions about the past, present, and future of Elgg and this community site.