Evan Winslow Great thinking about the Elgg 2.0 .. consider reading it

I am keen to share with all you community members the Idea of this great guy Evan Winslow about Elgg 2.0 and its future . The following is taken From his guithub respo..

 

# Looking forward to Elgg 2.0

Evan's advice on how we can make Elgg a 10x better development and end-user
experience than Elgg 1.x and any of our existing competitors else currently out
there:

* Develop apps, not sites
* Code for the future
* Take advantage of modern tools
* Embrace dependencies

These are radical changes I'm proposing, but I don't think they are
fundamentally unreasonable. They are guided by the following, very pragmatic,
values:

* Performance
* Security
* Productivity

I want 2.0 to move Elgg forward leaps and bounds compared to the current
state of affairs and compared to other, similar projects (BuddyPress,
Drupal Commons, JomSocial, Boonex, SocialEngine, PHPFox, Idno, etc.). We have a
responsibility to bring something valuable to the table I think that's
only possible if we make radical, but reasonable, changes that do not carry
the baggage of legacy with us.

Let me expound on each bit of advice one at a time. Then I'll go over why I
think embracing this advice lends itself to the values outlined above.

Develop apps, not sites
----------------
I want Elgg 2.0 to completely split frontend and backend development. Anything
that requires PHP logic (sending emails, DBAL, etc.) should be in a separate
git repo from the UI code. Frontend logic (JS), templates (HTML), and theme
(CSS) should be completely static and **not mingled with PHP at all**. This
allows them to ship straight from a webserver that's great at managing static
files.

In other words, the Elgg 2.0 UI should be hostable on GitHub pages.

To do this we need a paradigm shift in our thinking away from the traditional
model of shipping HTML documents to the user and toward a model of delivering
applications. These applications then request data from an API and render a UI
to the user.


Code for the future
-------------
We should write code against the latest web standards and rely on tools and
polyfills to plug in the gaps for old-browser compatibility. A prime example
of this is writing css prefixes manually. We should never do that again.
Instead, we should just write against a stable standard and rely on tooling to
add whatever prefixes are necessary to support older browsers.

Working around browser compatibility issues should be considered out-of-scope
for the project. Any polyfills that we need to develop that don't yet exist
should be contributed to an existing project external project for whom these
issues are in scope.


Take advantage of modern tools
------------
We need to stop reinventing wheels and pawn off development effort to more
focused and well-funded communities.

Here is a list of tools that we are not using that could easily make us more
productive and help keep us on the cutting edge:

- r.js -- The optimizer for RequireJS
- grunt.js -- A JS-based build tool with a large community + many plugins
- bower -- A package manager for front end components
- composer -- The defacto PHP package manager


Embrace dependencies
--------
As long as we choose dependencies wisely and can remain in control of our API,
this can only result in a more modern framework that is easier to keep up to
date (because there's less code for us to maintain!):

- PHP
- Symfony
- Zend
- etc...
- JS
- AngularJS
- momentjs
- underscore
- etc...
- CSS
- Bootstrap
- Foundation
- etc...

Performance
------------
The performance of a system is second in importance only to correctness. I'll
focus mainly on frontend performance here because (IMHO) that's where most of
the gains can be had for the least investment.

### Startup latency
Client-rendered UI with aggressive caching (AppCache) must become the new normal
in order for us to keep up with user expectations. I'm shooting for a 10x
improvement in startup time. That means going from 3 seconds to 300ms. This
can't happen with our current model of full-page-refresh per action as the
standard.

Forcing all web client content to be static (i.e. not generated on the fly by
PHP templates) opens up the possibility of using AppCache to build an offline
experience on par with native mobile apps. Using AppCache in Elgg is impossible
now because we are delivering our HTML pages differently depending on who is
logged in/etc. Pushing the logic and templates to the client would allow us
eagerly ship all of this logic to the client and cache it there instead of
making the server do all that work.

There is a 0-latency startup time for web apps that have been AppCached. Like,
zero! This cannot be acheived with just HTTP caching because the index.html file
still needs to be fetched on refresh. Consider the implications of this
for apps which try to render quickly on first load by inlining their css/JS
content. Without AppCache, that content is uncacheable. With AppCache, it's
*still cacheable* even after a page refresh. Normally in SPA's you can get a lot
out of inlining content on the first page load because there is no full page
refresh after that (for the duration of the session), so you don't have to pay
that download tax for every navigation. However, if the user ever does a full
page refresh, you have to pay the download tax again because that inlined
content is not cached. With AppCache this isn't so. Even inlined content is
cached because all master files are served from cache and only update in the
background (think Chrome-style silent updates).

In the good old days, teams like Gmail came up with crazy hacks to download
scripts in comments or strings and then eval them only once necessary. This is
a really cool hack to shave a lot of parse time off the initial app load, but it
adds an insane amount of complexity both to your frontend code and requires
server cooperation. Instead, we can use AppCache to download javascript eagerly
without executing it. And with AMD managing dependencies, we can be sure that
only the minimum amount of javascript actually needed will be parsed and
evaluated. We actually have real browser support for this hack and nobody has
realized it because Jake Archibald destroyed AppCache's reputation with
[his 2012 "A List Apart" article][1] on the subject. This is a travesty, and I
think we should fix it (or at least rely on a framework that fixes it -- this
goes back to the "embracing dependencies" comment I made before).

[1]: http://alistapart.com/article/application-cache-is-a-douchebag

### Backend performance
Moving all rendering to the UI means we can ditch Elgg's views system. It has
served us well for a long time but its time has come. Elgg views are often slow,
hard to cache, insecure, and don't do a good job of separating concerns.

It has also been suggested that starting up PHP just to serve static content
hampers performance significantly after a certain load. See the discussion at:
http://community.elgg.org/discussion/view/880800/elgg-optimizations-report

Finally, pushing all UI logic to the client gets you free CPU cycles. The user's
machine is the one doing the work to render the UI instead of the server doing
that. It's faster for them and cheaper for you. Everybody wins, what's not to
love?


Security
--------
Currently it is possible to do the right thing with Elgg concerning security.
We need to move past that and make it really hard to do the wrong thing.

Lots of people have asked for CDN support in the past wherein static content
could be served from a different domain for performance. I propose going the
opposite direction: keep the static content on your vanity domain
(www.example.org) and move the API server to a different domain
(api.example.org). This has some nice security benefits.

### OAuth, OpenId, BrowserID (Authentication)
We can eliminate entire classes of authentication vulnerabilities by dropping
our reliance on cookies and pawning off auth management to someone else.

Authentication in Elgg 2.0 could happen ONLY via OAuth/OpenId/BrowserID. This
eliminates XSRF vulerabilities completely (!!!) and ensures we're set up to
interoperate with other clients (Android, iOS, third parties) in a
secure way (i.e. don't have to ask for passwords on the client).

If we can switch our default authentication to a standard like BrowserID,
Elgg sites by default won't be contributing to the yet-another-password problem,
which is a security issue (Elgg databases are surely less well-guarded than
Google or Facebook databases) in addition to a UX problem (users hate creating
new usernames and passwords and forget them all the time). Furthermore, when
Elgg is used in any existing community, there is typically already an identity
solution so getting Elgg and that provider to play well can be significant extra
work. If you did want your Elgg site to be an identity provider, that could
still be an option through a plugin or something, but the default should be an
auth-as-a-service model. At the end of the day, you just make your site one of
the services if you need to revert back to the legacy behavior of Elgg 1.x.

No more dealing with:
* email verification
* password resets
* storing user passwords

Letting someone else do that hard work and integrating with them
in a standards-based and federated way (BrowserID), seems like
exactly the right way to go. If there any more identity bridges that need
building, we can just support the BrowserID project instead of
defaulting to building/maintaining our own thing.

### Content-Security-Policy and strict contextual escaping (XSS)
With a ground-up rewrite of the frontend code, we're free to set up a strict
default CSP and only open up permissions as needed. That allows us to ban inline
scripts effectively eliminating XSS vulnerabilities for all browsers that
support the standard (Firefox, Chrome, Safari). Couple that with a great
security-conscious templating library (Angular with strict contextual escaping),
and we can have some a much higher confidence that XSS is behind us, rather than
hoping that any existing XSS vulnerability gets discovered and reported before
it gets exploited.


Community
-------
The Elgg community is enthusiastic, but rather small. The more stuff we try to
implement on our own, the thinner we'll be spread and the less we'll be able
to contribute to the things that actually make us unique. There are huge
communities out there solving a lot of the basic problems we're having and it
would simply be foolish not to depend on them. Elgg 2.0 should be less about
developing new features and more about pulling together all the latest best
practices for frontend development into one cohesive unit.

Take the developer tooling for example: great tools are out there these days for
managing (npm, bower), building (gruntjs) client-side apps that simply didn't
exist 2 or 3 years ago. Grunt seems to be the de-facto javascript task runner.
Bower is a great package manager built on npm. Composer does a perfectly fine
job of PHP package management. Do we really want to create our own packaging
system and plugin distribution system just for Elgg, or would we rather
interoperate with the rest of the ecosystem and take advantage of all the great
work that's going ithose efforts.

Future
------------
Furthermore, this split completely separates the frontend code from any
knowledge of how the backend is implemented. The fact that PHP is being used is
irrelevant and it could be easily replaced with Java, Ruby, Go, etc. This allows
the frontend code to stand on its own merits instead of standing or falling with
the PHP/MySQL choice.

### Development/developer convenience
This also provides some development conveniences in that we don't need to hack
around existing tools to support the dual-environment setup that we have. For
example, Travis, the continuous testing service, only really supports one
execution environment at a time and we have had to come up with workarounds to
get our tests to run correctly and efficiently in the dual PHP/JS environment.

### Server becomes client agnostic and vice-versa
Providing a server with a robust API means that we are not tied to
whatever client implementation we provide. If someone decides to reimplement the
client in Ember/Browserify instead of Angular/AMD, they are completely free to
do that without needing to give up on the server component. More likely is that
someone will want to implement a native iOS/Android app for their site, which we
would be able to wholeheartedly encourage since we would already have a
battle-tested API that they can draw from (and a standard way to contribute back
changes that they may have needed to make).

Finally, there are yet other app environments like Chrome packaged apps that
lend themselves to this kind of development model. With the current state of
affairs it would be non-trivial to create a Chrome packaged app for an
Elgg-based site. With the proposed model, creating such a thing should be a
fairly trivial matter of adding some extra build rules since all the assets are
already static and perhaps we just need to generate the manifest file or
whatever Chrome requires for a packaged app these days.

 

  • @Gerard - the approach suggested here is, I think, the opposite of lock-in: it allows for the possibility to have Elgg behaving much as it does now if we want but also for it to be accessed through other means with much more flexibility. Quite apart from the user-facing app possibilities, I'm particularly intrigued by the potential of this approach to allow a federation of Elgg services: an API server could easily be a client to several different servers or, equally, the front end could access multiple sites, or both. This might effectively give us multisite for free and then some. 

    I'd hate to be the one trying to make all of this backwards-compatible though! I think that is very important - the move from 0.x to 1.x was extremely and unnecessarily painful and I suspect a lot of Elgg momentum was lost as a result. I know that it had a disastrous effect on the University of Brighton's pioneering service and, based on lessons learned there, most of our developer effort for some months at Athabasca University went into trying to translate the old data and expected functionality from 0.x to 1.x, mostly fairly successfully. It would be a pity if we had to relive that again and we simply don't have the resources any more to even begin to try. This is such a big change that  I'm guessing it might need a compatibility layer a little like the Rosetta engine that Apple used when they moved to OS/X.

  • Well, if it is about extending the API. I am all for it. That can be used for mobile but also for extension to other websites or any other application wanting to use the backend. I am not sure how that would create a lot of legacy issues to you. In my perception it would only extend ELGG with webservices and therefore create a more open platform.

    But as you might read from my comment, I am still not clear what the app approach is really about.

  • @Gerard - my spider sense started tingling at the suggestion to do away with views altogether. That's not just extending the API but radically altering the architecture of the whole thing. The idea is very sound but there must be a compatibility layer to deal with old plugins and I'm guessing that's not a trivial thing to build.

  • @evan do me a favour please provide us good documentation with example .... its very tough learn ......... documentation is good but dont have example for all ....... for elgg 2.0

     

  • wow some good info exiting stuff : ) looking forward to the release though i see it may be a while yet

  • @gerard "Apps are of course great to mobile users, but they build in huge vendor lock-in."

    I think jondron clarified this well, but I'll reiterate that I'm talking about building a WebApp, not native apps. The API that necessarily comes out of building the WebApp could be used to build native apps, but I'm hoping the webapp is so compelling that building a native app feels like a waste of resources.

    @jondron "I'd hate to be the one trying to make all of this backwards-compatible though!"

    There will be b/c for data, but not UI. This will feel much like 1.7 => 1.8, but with way more value in the transition (and we'll probably try to stick with our existing CSS classes this time). Major version bump technically means no backwards compatibility promises. We really should have done a major bump with 1.8. That would have been more honest of us. I'm pretty sure we never want to do 0.x => 1.x again, since that's effectively killing one project and starting a new one, so twiddling version numbers would also be dishonest. Any data migrations will continue to come with upgrade scripts as they always have for the 1.x series.

    @jondron "my spider sense started tingling at the suggestion to do away with views altogether."

    My argument here is:

    • It is literally impossible to do a b/c layer for the new architecture. Not just hard: impossible. Elgg views are server-side and in the new world everything is offline-cached and rendered on the client.
    • Breaking b/c is justified by the 10x value provided by the new architecture.

    Also keep in mind none of this is set in stone by any means. In the short term, you can probably expect 1.x to gain a solid REST API while I go off and build this thick web client with perhaps a few fellow trailblazers.

    @sathishkumar, re: documentation

    Yes, documentation is critically important. I've already written a fair bit, but will continue with more as we move forward. Fortunately there are great services like readthedocs.com that make hosting and maintaining good documentation really easy.

    @ura soul, re: performance

    Performance will almost certainly improve, I suspect. We'll have to measure it, but here are my reasons for thinking it's going to get better:

    • view logic is rendered on the client, therefore not burdening server CPU or memory.
    • the app's assets will be cached offline, therefore not burdening the server with requests for the same assets on repeat visits.
    • the app's assets are cached offline, so users startup experience in many cases is bounded by JS parse and execution time, not network RTT and server response time. If you don't have the data cached offline yet, then you do still need to take the network hit, unfortunately.
    • The app's assets are static, so no need for PHP to run when they're first requested.
    • It will be trivially easy to make sure that all assets are minified and compressed, including optimizing images and minifying HTML templates.
    • Everything is executed async and as-needed with RequireJS/AMD to minimize JS parse/execution time, instead of loading the WHOLE app in 1 giant JS file up front, which would kill startup time on mobile because Elgg is HUUUGE. Only the minimum JS for the current page is run, then as you load more pages more JS is loaded to work in those pages. It's really quite cool if I do say so myself.
    • Even entities and lists of entities will be cached offline so users will be able to get stale data in a few milliseconds (effectively instantly). That data is then updated in the background from the server. Turns out this is quite similar to what you would get by doing mysql master/slave replicas, but without the server complexity.

    For metrics, we should compare:

    • user-perceived response time for cold-start with unprimed cache
    • user-perceived response time for page navigation with unprimed cache
    • user-perceived response time for cold-start with primed cache
    • user-perceived response time for page navigation with primed cache
    • user-perceived response time for various UI actions like commenting, liking, etc.
    • server memory usage for comparable pages
    • server response time (CPU usage) for comparable pages

    If people think of any others, please do suggest them.

  • Just a wild and ill-informed thought: could a 1.x version be built as an interim step that would call on a 2.x server for its heavy lifting, data storage, etc and, reciprocally, begin to do some of that offline cache stuff for newer 2.x components? In effect, a modded version of 1.x might become a client for 2.x while retaining compatibility with the thousands of existing plugins we rely on. Or is that just as impossible as building that into 2.x? 

  • I think we'd go the other way: a 2.x client could talk to a 1.x server that implements the rest API. That's basically all I mean by 2.x -- a thick web client. You could even use both clients in parallel for a while. Indeed, that's likely what I'll do on the Elgg site I'm currently managing. Users will be able to switch between both clients while the new one is being developed.

  • If it is possible to use a 2.x client with the new rest API on a 1.x server why is it then impossible to also keep the current Elgg view system in 2.x ? Plugin developers will than have time to upgrade their work to expose their functionility to the new client. It also seems simpler because from what I read, you will need to change the database structure too on 2.x That would be needed too for the 2.x client to work on 1.x I guess.

  • i have noticed in many ways/places that elgg as sorely lacking a formal design approach - which is partially why there have been so many tickets for issues which would have been coded automatically and thus never turned into bugs/tickets, had there been such designs.

    in my opinion no major revision of elgg should be initiated without full, formal design documentation. otherwise you are potentially wasting thousands of hours of work for many, many people.

    the benefits of full, formal design documentation are massively greater than most coders realise, since most coders have never seen such documentation - since most coding projects are coded by coders who do not produce such designs and documentations. you can build a house without plans, but to work with a team, the plans are essential.

Feedback and Planning

Feedback and Planning

Discussions about the past, present, and future of Elgg and this community site.