Release Cycle

Get yourself a cup of coffee or some Earl Grey - this is going to be rather long.

In our wiki, we have outlined our Release Cycle. I think it is time we revise and extend this to give greater clarity and allow us to move more quickly between stages of the release.

We have been using the bug tracker to report and classify bugs and features, combining them to determine our release phase. For example, once a final release is made available, new features begin to be added, based on our roadmap. Once all, or most, of those features have been added, we release an Alpha release as a preview and proof of concept. We then refine how those features work based on feedback from other developers and the community.

Once we have them functioning pretty much the way desired, we move on to the Beta phase, where our focus turns from features to bugs. Bug classifications could be very severe (Blockers, or show stoppers), Critical - basic functions don't work, Major, Minor, or Trivial. Our current definitions in the wiki allow us to release an RC only when all known Blockers and Criticals are fixed. Our current definitions only allow us to fix new Blockers or Criticals after we have reached an RC stage.

While this is a widely accepted practice in the software community, it is rather broad and fuzzy about some things.

What this all boils down to is - How do we define Code Freeze?

When is something a new feature or enhancement to an existing feature and not a bug?

Is all code equal? To strictly apply the rule of no code changes during an RC prevents any and all changes - even modifying a language constant to clarify a label or on screen instruction is forbidden.

I definitely agree with the feature freeze stage - and this goes along with my blog post - the new features added to an upcoming release need to be focused and clearly defined before we go into developing that release and nothing more needs to be added. Once those features are in - Alpha is launched.

Now comes the tricky part - determining what parts of the code to freeze and when. I am going to suggest we need to be more precise in this area and consider the following 'types' of code for a code freeze:

* Core functions (classes, methods, arguments - API stuff)
* Language constants and documentation
* User Experience (usability)
* User Interface (design - similar to the above, but this is the look and feel, not the steps followed to complete a task)
* and, Code Performance

This is taken from reviewing the processes of other successful open source projects, trying to learn some of the best practices they are following. One thing also to note - the other projects change their commit process as they move into their final stages. Instead of committing directly to the trunk during the stabilization phase (after Beta and before Final), patches are submitted by the developers, which are then tested by the other developers before being committed to the trunk.

We may not be a big project now, but in order to be one, we need to start acting like one and the sooner, the better I'd say.

Now - your turn!

Thanks for this excellent post Steve!

Some of you already know my position regarding this. Since the start of the project, I've always been pretty much "radical" when it comes releasing final software. My biggest priority when releasing a final is releasing a software that works. If users cannot install, cannot login, or cannot upgrade, chances are they will conclude our software does not work on move on to one of the hundreds other CMS out there...

By experience, I know that any change, as small as it can be, can have undesired side effects. Even the best coder can do a mistake, forget a semi-column, commit an undesired file (mainfile.php ? ) and break something. Now, if this something affects installation process, login or upgrade, we will loose potential users...

However, our project grew a lot in the last 2 years. We have gain more users, more developers, more contributors, and in general, more eyeballs! So perhaps it is time that I revisit my position about this.

One of the other key concept I truly believe in open source software is "Release early, release often". The only result of all our efforts is actually releasing! Without release, nothing happens: we don't get people testing our product, we don't gain new users, we don't gain new contributors. Release is golden, everything else does not matter.

So, in order to actually help us releasing more, it is indeed time we improve our release process. Steve's question is the core of the matter:

Quote:

I believe the following should be considered in a Code Freeze:

* Core functions (classes, methods, arguments - API stuff): because this can have multiple side effects anywhere in the code. This will be more and more true as we use IPF more and more in the code.
* Code Performance: as this can also impact multiple areas of our software

The rest can fall outside of the code freeze rule, but with some rules and direction. So the following can be changed before a final:

* Language constants and documentation: as even if we screw something up there, there are very little changes it will screw something up in install, login or upgrade functions
* User Experience (usability): again, very little chance of breaking something bad
* User Interface (design - similar to the above, but this is the look and feel, not the steps followed to complete a task): look is very important in gaining popularity, so anything we can do, even before a final to improve this area, let's do it.

But always keep in mind that a broken software is costing us users. So before a final, if you change something in the previous 3 areas, please make sure it doe snot break anything. We should always double-check that the change we make does not break something, but this is even more true before a final.

Also, about this;
Quote:
This is indeed a good way to do it, but the overhead cost would be just to big for now. I would say we don't go that way for now. We need more developers to be able to do this. Instead, we should educate our current team to respect our rules, review their commits, and respect what should be frozen and what should not.

So, in summary, from what I see, our release cycle process should be as follow:

* We list the features that we want to add in a new cycle (example, 1.3)
* When enough of those features have been added, we release Alpha
* If more features are needed, we release a new alpha
* When we are satisfied, we release a Beta
* Then we call a Feature Freeze. No more features are to be added before the next release cycle (example, 1.4)
* We fix all bugs that are reported in Beta phase
* If to many important bugs were found and our fixes need to be properly tested, we release a new Beta
* We are nearly ready for Final, so we first release an RC
* Then we call a Code Freeze
* No more bugs are fixed, except what has been listed as being outside the Code Freeze.
* When all of the previous are fixed and not being postponed to next release cycle, we release Final, we pass Go and claim our 200 $

Your thoughts will be more then welcomed here, so don't be shy and speak up !

This would also be much simpler if when we implement a testing framework to ensure all critical functions are working. SimpleTest and PHPUnit are 2 I would strongly recommend.

Quote:
That's exactly the point! The part marked in bold is very very important. The red part is MOST important. Everyone should review and do some deep testing before commiting his code. That's at least the way I do it and I think it's working for me.

To add a comment to your list Marc: If RC means not to fix any more bugs, please tell me why we have just done so? There are quite a few left on the list. Maybe I got you wrong on that point. Please clarify .

Quote:

It's not exactly what we did. We released Beta, then we fixed bugs and we released RC. Now, we don't fix anymore bugs before final, except for the areas explained before.

As I said in my previous post:
Quote:

So after RC, no more bugs should be fixed except for list of things being outside the Code Freeze, which would be:

* Language constants and documentation: as even if we screw something up there, there are very little changes it will screw something up in install, login or upgrade functions
* User Experience (usability): again, very little chance of breaking something bad
* User Interface (design - similar to the above, but this is the look and feel, not the steps followed to complete a task): look is very important in gaining popularity, so anything we can do, even before a final to improve this area, let's do it.

Everything else needs to wait for next release cycle, for example 1.2.1.

And we do understand some known bugs might be left for next release cycle, but it's the only way to go, because there will always be bugs. If we wait that no more bugs are found to release a final, it means we will never release

Please let me know if it's not clear. Thanks!

Well, it's clearer now. However there are some bugs on the list that would fall under the "code freeze" part but they definitly need some attention before 1.2 because they are extremely annoying for the end user. I'm talking about the text editors thing (line breaks - html conversion, ...). Therefore the decision for the RC or the priority for these bugs would have been wrong.

Further we need to have an agreement on "actual non-core features" such as bundled modules (for now: protector, content, profile). As they don't effect the core itself but only the module there might apply another rule. However, you have to distinguish between bugfixing that require changes in the core itself (such as IPF related bugfixes) and module-only bugfixes.

Thanks for your feedback Phoenyx. You are indeed bringing very valid discussion points and this is exactly what this thread is all about.

Quote:
Then if they are that annoying, they should have been set as critical or blocker and addressed before RC. We will need to take a quick decision about this. And if we do decide to not wait next release cycle, we will need to release an RC 2. Linebreaks/editors problems affects all areas of inputing content on a site and they need to be properly tested by users, through a proper release.

If we continue to change critical code like this after an RC, then RC does not mean anything. We could simply do beta1, beta2, etc... and then jump to final when we are ready. RC has absolutely no meaning if we keep changing critical code before final.

Quote:
True. But now RC is released and we can't go back. My vote would be to leave this to next release cycle and stick with the new plan we have about Code Freeze.

Quote:.
Agreed. I will open another thread for this is a different topic.

another opportunity to get some tea - no coffee - we need to breath slowly and relax as we discuss this

This is as much about culture (of our project and community) as it is about process.

One thing I want to stress about this thread - it is concerned about what we do in the future. We are still evolving as a project and as a team. We have open discussions about our practices and direction so we can benefit from the knowledge and experience of everyone who contributes - be it as a developer, a designer, a tester, a user, or someone who does marketing and PR.

One other thing to keep in mind is this - working by consensus is difficult and can take a long time. Working by dictatorship is also difficult. Since we do not want to distract ourselves with seemingly endless discussions and little productive work, we work to balance them both, which is why we have leadership teams for our primary task areas - development, community and promotion. They work together to set the practices we will follow and monitor them and assist others with them.

When it comes time for a release, one role we have not officially established is that of a Release Manager. For the 1.2 release cycle, I have had that role. The Release Manager oversees the feature list, the bug list and guides everyone between the release phases. They also are in charge of determining if the phase is sufficiently ready to package, oversees creating and testing of the packages, etc. While in this position, I have experienced what many other open source project managers have experienced - usually in quiet desperation, because there are few published resources for open source management. As I have studied other projects and the few documents I have found, I chose to start communicating here, so here we are.

Now, as Marc-Andre confirmed, the decision has already been made to go to RC and it has been released. While he and I don't completely agree about everything we discuss, we are of one mind on this. The opinions expressed in this thread will be considered as we establish our practices for the next release.

Code review is something Marc-Andre do completely agree on and we will be enhancing our tools and practices to enable more of this on a timely basis. This will also be part of our future release process, for sure.

Quote:
Oh my God! Thanks for writing the words that were in the back of my head for so long without being able to expressed them intelligibly.

This is exactly what we need. We are already a very open community, which often leads to endless discussion. Great things come out of those discussions, but at some point, we need a decision. The Release Manager does exactly that: listening to people, gathering people's feedback, but then making a decision. A decision which the dev team will accept. That way, actions are taken, and it's not just words. And in the next release, if most of us disagree with the decision taken by the Release Manager, then we'll name a new one!

I'm really happy to see this thread, and I'm glad it's so alive. As the project grows, everything will become more complicated, be it team coordination or code complexity. Having this discussion regularly will help the project in all kinds of ways.

As stated in other threads, we have a problem of missing deadlines. Doing this kind of 'audit' of the way ImpressCMS is working could help on that account.

This thread has many solutions for parts of the problem:
- release manager
- better prioritising of defects
- clear alpha-beta-rc-final procedures
- educate the team about the rules

and these are all very valid. Part of the problem I feel, is due to the fact that the releases are too big, and are monolithic. Given the fact that we aim to have a release every 6 months, the pressure is high to fix a certain defect or include a feature in the current release. If not, that functionality/fix will have to wait another 6 months, which is sometimes unacceptable. That means functionality is squeezed in at the last moment, and we all know that's asking for problems and delays.

I’m with Marc-André on that point, maybe with a twist: release earlier, release more often, release smaller.

1.2 is a huge release. In most projects, it would have been labeled 2.0, as there are many fundamental code changes. This is indicated by the number of modules that won't work without changes (there are quite a few of those).

I’m asking myself if most changes, except the fundamental core alterations, couldn’t have been released in smaller packages (1.1.3, 1.1.4, 1.1.5). For example rewriting the administration interface for some part, adding a smaller functionality, doing some code cleanup. That would mean two quick wins:
- we gain functionality much quicker,
- the momentum of the project is much more visible to the outside world
- release cycles are shorter, missing a release isn’t such a dramatic event
As a side effect, the test effort is spread over a longer period of time, and doesn’t have to be concentrated in one gigantic release.

When re-reading this, I seem to be promoting a more ‘agile’ form of working. That could be the case. However, Agile Development is mostly done with teams working in the same room, or in the same building. I’m aware that the daily stand-up meetings and the coach-pupil is difficult to do with a the loose organisation of an open source project. The concept of sprints and iterative development is something that could benefit ImpressCMS in my opinion.

Next to my previous rant (see my post a few minutes ago) I'm happy we are discussing the rules of the release process. Written, well defined rules will make it easier to decide next time which defect is included, and when to release what.

I would propose that the 'release responsible' of the next version is nominated right after the final release of the current version. I do think there is a difference between project lead and release responsible. a project lead has to tackle the issues that span more than one release, and is responsible for the long-term evolution of the project.

First of all it's good to see the discussion going on. Unfortunately it stoped a little to early for my taste. It's a very important point for an open source project. That's why I would like to respond to some points mentioned after my last post.

Marc, you're absolutely right that RC wouldn't mean anything if we would fix bugs now and release a final then. That's why I (personally) vote for fixing the real dirty bugs, release an RC2 and a final then. To be honest, the bug was there in 1.1.2 and right now it's still there in 1.2 RC. We have so many questions in the german board regarding this specific issue and every time we have to lead them to the database to fix this issue (which isn't a good solution though).

Leading to the next point from Steve that we need something like a release manager. I would appreciate this very much.

There are also a lot of valid points in Davids post(s). A clear definition for our release stages would be something really helpful. Together with our discussions beeing more streamlined a lot of things would be better as they are today.

Hopefully this will bring this discussion back rolling

I think what you're heading towards, Phoenyx, is more about what makes something 'critical' or 'trivial'. Bug classification is really a completely separate discussion - what the release cycle is all about is what types of bugs get fixed between each stage and what other things can be added. To keep this thread on topic, let's open another for bug classification, shall we?

What I hope is understood from this thread is that RC indicates the developers have no known critical or blocker bugs in the current trunk. There may still be other known bugs or unknown issues, but the code is stable enough to present to a broader audience for testing and verification. What we have introduced is allowing for bug fixes for things in areas that do not affect the stability of ImpressCMS. These areas are things like documentation, design and the user interface.

The other thing to realize is that as we improve, modify and move ahead, some things will no longer be supported. In other words, we will break things as we fix other things. The problem with the wysiwyg editors is really about legacy support. Not all modules or text areas have the problems described. So, do we change the core, or do we push the module developers to comply? Is there a way around the problem? Yes, there is, in this case. Can it be improved? Yes again. Do I think we should hold up the release because of it? No, I don't.

A seperat thread for bug classification would be great then.

Regarding your last paragraph: The problem with the text areas exists in the core (system preferences). I was never talking of any modules!

I will accept the decision, however, I won't use 1.2 then I'm afraid (or at least it's the first thing for me to fix before using it).

Quote:
A fix has been made. Now we need testing

Quote:
Following our chat today, I believe you will be able to use 1.2 finally