You are currently browsing the archives for the Configuration Management tag.

Releasing Code

June 20th, 2011

The Release process

The Release process defines the actions required to deliver a software product to an external customer. The external customer is any entity outside the development department. This may be a true (paying) customer, or may be another engineering department, for example Testing or Production.

The Release process is a triggered activity. The trigger events are scheduled as part of project planning. Defining a release is a project milestone which must define

  • What will be released
  • When it will be released
  • Who it will be released to

 

Release process relationships

The Release process is related to, but independent of, the Change Management, Revision Control and Build Processes.

 

image

Figure 21 – Release management is related to, but independent of, the other CM practices

Change Management

Defines the modifications and/or additions to the product, the order in which the changes are incorporated.

Revision Control

Ensures the configuration of the product is controlled and reproducible.

Build Process

Defines how to build the product.

Release Process

Defines the target recipient of the product.

 

Software release stages

During development the product may be released:

  • To different standards
  • To different customers

The different releases comprise a release lifecycle, with each stage representing an improvement in product quality (Figure 22).

 

image

Figure 22 – Each release type represents a different level of quality, and may be released to different customers
 
Development releases

Development releases are internal releases; usually to (independent) test. These releases are unlikely to be ‘feature-complete’; often the release represents one or more work packages (or, in the case of Agile projects, features or ‘sprints’).

It is not expected that these early releases are perfect. It is likely they have only undergone developer testing. A significant number of bugs can be expected in early releases.

Development releases may be produced at high frequency. Weekly releases would be expected at the beginning of development, possibly rising to daily as the project enters a debug phase.

Alpha and Beta

Alpha and Beta releases focus on usage and/or useability testing. Sometimes these are known as Technical Preview releases. The product may be feature-complete (or close) at this stage. Alpha/Beta releases are relatively stable and should contain no (known) critical bugs.

Alpha testing consists of simulated or actual operational testing. It is normally carried out in-house and performed by non-development users, for example internal proxy-customers (staff acting on behalf of the ‘real’ customers).

Beta testing is also operational testing. It is often performed out-house (that is, outside the control of the development organisation). It is carried out by focus groups, or specially selected users. Very often Beta releases are made available free to existing customers to use and test in their own environment.

It is important not to begin Alpha and Beta releases too early in the development cycle. Although allowing users to test the product is potentially very effective a product with many bugs (particularly in areas of key user functionality) can lead to a loss of confidence in the product that is very difficult to recover.

Production-ready releases

The term Release candidate refers to a version with the potential to be a final product. It is essentially ready to release unless fatal bugs emerge during final testing (or possibly Alpha or Beta testing). The product features all designed functions and no known critical bugs.

A Production release is very similar to a Release Candidate ( in fact, it could be argued the Production release is just the final release candidate!). Any last minute bugs fixed. The Production release represents final product quality and features, and it the release sent to Production engineering.

Change Management

June 13th, 2011

Change Management is concerned with the proposal, selection and scheduling of changes during the lifecycle of a project.

Change Management is interlinked with, but separate to, Revision Control.

Change Management is the core to controlling your development processes. Without effective Change Management the management of your project is subject to slavish adherence to a (fixed, and pre-determined) project plan, with no mechanism for dealing with inevitable changes in requirements, design, implementation or testing.

It is no surprise that Change Management is at the heart of so many Agile process, such as SCRUM.

Change Request

The core of Change Management is the Change Request (often abbreviated simply as CR) A Change Request has many different names, all meaning the same thing:

  • Change Note (CN)
  • Engineering Change (Order)
  • Engineering Change Request (ECR)
  • Action Request (AR)
  • Request For Change (RFC)

Essentially, a Change Request is a call for an adjustment to a system. Change requests typically originate from one of five sources (Dennis, Wixom, & Tegarden, 2002):

  • Bugs that must be fixed
  • System enhancement requests from users
  • Events in the development of other systems
  • Changes in underlying structure and or standards
  • Demands from senior management

 

The CR Artefact

A CR is a project artefact – that is, it is a entity that is created, worked on, stored and audited, just like every other artefact in the system. The CR represents the lifecycle of a change. As such it has a different lifecycle to other artefacts.

As an artefact the CR may (in fact, should) also be held under revision control.

The CR lifecycle is shown in Figure 19. There are three main parts to the lifecycle.

 

image

Figure 19 – The Change Request is an artefact with its own unique lifecycle

 
Opening the CR

Creating a CR records that some change to the system is requested; it does imply that the work will be performed. Once created the change must be reviewed before it can be worked on. The review is performed by the Change Control Board (CCB). The CCB consists of stakeholders who will be affected by the change, and those who can decide whether the change is worth doing. At the minimum this will be the Project Manager or Team Leader; but may include a multi-disciplinary group including engineering, senior management, marketing, customer support, etc.

The CR must be assessed for impact to the project. This work should ideally be done by the CR submitter. Points considered during the assessment of a change request include:

  • Technical feasibility
  • Timescales
  • Customer expectation
  • Resource
  • Quality
  • etc.

The CR may be Accepted (Opened for working), Rejected (Infeasible or invalid) or Deferred (delayed; therefore inducing technical debt to the project)

Open CRs

Once opened project artefacts can be modified. Each artefact follows its own Configuration Item lifecycle (Figure 20). The CR records artefacts modified. Each artefact records the changes made in support of the CR.

 

image

Figure 20 – Each artefact modified under the Change Request follows its own change lifecycle
 
Including the change

The completed change should be reviewed again by the CCB. The purpose of the review is to assess whether the change is valid – That is, do the modifications made to the system correctly addresses the change requested? An invalid change will be rejected for rework.

Once accepted the change can be integrated into the product.

 

Change Management is often overlooked in CM. Change Management controls precisely what is going to change in the project and when. Without Change Management a project is running on ad hoc and unrecorded decisions by the development team or project manager and runs a serious risk of heading out of control.  Although the Change Management presented here involves project artefacts (CRs) many Agile processes adopt similar principles using techniques such as Product Backlogs and Feature Lists (SCRUM), which are organised by customer priority. These mechanisms are, in effect, simple Change Management processes.

Baselines and Branching

June 6th, 2011

A baseline is an identified set of files and directories in which there is one and only one version of each file and directory.

A baseline identifies one particular configuration of the software (or a subset thereof)

The baseline represents a fixed point in the development; that may be recreated as required.

Specifying a Baseline

A baseline defines a set of files, each at a particular version. These need not be the latest (most recent) version. A baseline label uniquely identifies the configuration. Files may belong to one or more baselines.

image

Figure 12 – A baseline defines a set of files, each at a particular version.

 

In the example of Figure 12 baseline BL1.0 is the first baseline recorded. It consists of seven artefacts, each at a unique revision number. For this example, assume that BL1 records the most recent versions of each artefact. As development progresses each artefact is modified as required (that is, some artefact are modified, some are not). At some time later another baseline is taken – BL2.0. In this case BL2.0 records the current latest revisions of each file. Notice that artefact F is unchanged, so F v1.0 is included in both baseline BL1.0 and BL2.0.

In general each successive baseline contains more recent versions of files (but not always).

When to use baselines

Baselines are a key tool in managing and auditing the state of a project. Baselines should be created prior to any significant project activity:

Development

Baselines provide a ‘working’ point in the project’s configuration. Recording a baseline prior to making a major change to the system allows the development to start from a known working point and, in the event of something going wrong, to ‘roll-back’ to that working configuration.

Baselines should be made prior to starting any new package of work (as evidence of project progress); and new feature development; prior to changing or upgrading any tools; etc.

Release

A baseline records the complete configuration of the product that is being distributed from the development department (whether that is an external customer or just the test department).

Audit

Baselines provide process evidence for audits and also time-stamped evidence of the project’s progress (for example, starting a particular work package)

 

Baseline of baselines

In larger, more complex systems, we may choose to manage the system complexity using a ‘Components’ approach. Each ‘component’ represents some aspect of system functionality – for example a subsystem. Using this approach each component of the system has its own (unique) baselines, independent of any other component (Figure 13).

image

Figure 13 – each component of the system has its own unique set of baselines

Note, not shown in this example is the fact that components may share artefacts. For example, both component A and component B may share a common artefact H; with each component using a different revision.

image

Figure 14 – A complete system constructed from a  ‘baseline of baselines’

The (complete) product’s baselines are constructed from (unique) combinations of the component’s baselines (Figure 14). This method is known as a ‘Baseline of baselines.’

 

Branch Patterns

Branching is the (apparent) duplication of an artefact, or set of artefacts, so that they can be developed independently of the main development activity. Branching is a facility of all revision control systems. RCS systems incorporate mechanisms (such as lazy copying) to avoid the potentially massive overhead of copying artefacts between branches. Branches allow parallel development, which is essential in all but the most simple of software developments.

Conceptually, the artefacts that make up a product or system form the main branch or trunk of the development. The trunk represents the main line of development for the product. Ideally, the trunk represents the most recent (working) configuration of the project. Branches represent alternative paths of development. Motivations for branching include:

  • Maintenance of released products
  • Customer specific additions or modifications
  • New development work
  • Research and development projects

Note, with branching the unique identifier of an artefact must be extended to include the artefact’s branch. Thus artefact MyDoc v1.1 will be extended to be (something like) \main\MyDoc v1.0, which will be different to \main\branch1\MyDoc v1.0. In this case \main identifies the trunk branch and \main\branch1 identifies the first branch from the trunk branch.

Uncontrolled branching can lead to major administrative issues (many companies have a RCS administrator simply to keep the repository ‘healthy’)

The branching patterns presented below are a systematic approach to version tree usage. They are design to reduce the complexity of repository branch management and allow effective project management.

 

Sequential

The Sequential branching pattern (Figure 15) is the simplest model – in that it contains no branching! In essence, it is a ‘pseudo-branching’ pattern.

All development is performed on the trunk branch. All development is linear; no parallel development can be supported. This pattern requires, and enforces mutually exclusive changes to artefacts. By default, this is the pattern followed by all artefacts.

 

image

Figure 15 – The ‘pseudo-branching’ pattern, Sequential.  There is no branching!

The Sequential pattern is fine for projects where there is no parallel development required and the current release always has the most up-to-date developments. In general, this restricts the pattern’s use to simpler projects.

 

Off-Shoot

The Off-Shoot branching pattern allows a legacy version (the mainline development) to have derivative and independent versions created.

 

image

Figure 16 – The off-shoot branching pattern

The off-shoot branch can be created retrospectively – that is, development on the trunk branch could be well advanced before the off-shoot branch is created. The off-shoot is never merged into another branch. Off-shoots may also have their own off-shoots.

Note that the main trunk is baselined before the offshoot is created.

 

Loop

The Loop branch pattern is a variation on the Off-Shoot pattern. The Loop pattern allows basic managed development.

 

image

Figure 17 – The Loop branching pattern

In this example the trunk branch represents the release branch. That is, all releases are from the main branch. New development is performed in an Off-Shoot branch (called \Dev BL 2.0 in the example since it represents the new code that will appear in baseline 2.0 of the product). The new development work is independent of any release code.

Once development has been completed in the Off-Shoot branch the Loop is closed by merging the off-shoot back into its parent branch. Note that baselines in the main branch only represent completed development activities.

 

Integration

The Integration pattern extends the Loop pattern model to allow managed and concurrent development.

 

image

Figure 18 – the Integration branching pattern

As above, the main trunk branch represents the release branch for the code; no development is performed on the main branch.

For a new feature a new branch is created (\Integration). The new development consists of three work packages. In this example Work Package 1 (\WBS1) must be completed before Work Package 3 (\WBS3) can be started.

Within the \Integration branch two new Loops are started (\WBS1and \WBS2). The work on both packages continues independently and in parallel. When Work Package 1 is complete it is merged into the \Integration branch. At this point Work Package 3 can be started, so a new branch is created from the \Integration branch (after baselining the \Integration branch – not shown). Some time later Work Package 2 is completed and merged into the \Integration branch. Finally Work Package 3 is complete and merged. To complete development the \Integration branch is merged back into the mainline development.

The Integration pattern builds on the smaller patterns to form a comprehensive branch/merge strategy. The strength of this pattern is the RCS archive reflects the development project plan.

Configuration Items

May 30th, 2011

A Configuration Item (sometimes referred to as a Computer System Configuration Item, or CSCI) is an artefact (hardware and/or software) that has an end-user purpose – that is, it contributes, in some way, towards the attributes of a system or product, or the development processes followed to produce the system or product. A Configuration Item (CI) is an artefact that is treated as a self-contained unit for the purposes of identification and change control. For example:

  • Specifications
  • Design models
  • Source code
  • Make files
  • etc.

Configuration Item identification

Configuration identification is the process of identifying the attributes and procedures that define every aspect of a configuration item.

Configuration identification performs three distinct roles:

Uniquely identifying each Configuration Item.

CIs are the artefacts to be controlled under Configuration Management. Each CI must be uniquely identified by name, version (revision) and location (on the file system).

Establishing a change policy

The CI’s change policy defines how the artefact is revision controlled; and whether that control changes during the lifetime of the project.

Establishing a review policy

Ideally, every artefact should be reviewed. Configuration identification establishes how the artefact will be reviewed and when it will be reviewed.

Configuration Identification must be performed prior to establishing any revision control system. All CI attributes and policies must be recorded in the project’s CM plan.

Configuration lifecycles

Configuration lifecycles define how an artifact is checked into and out of the revision control system. The configuration lifecycle defines the ‘life’ of one particular change, as it applies to that artifact (c.f. Change Management, below). The lifecycle is typically shown as a state transition diagram.

For any real project of any complexity a number of lifecycles will be required. Also note, an artefact’s lifecycle may change through the lifetime of the project. This is defined by the artefact’s change policy.

Listed below are a number of common configuration lifecycles.

‘Archive’ lifecycle

Archive artefacts are configuration items that are simply stored in the CM system. Archive artefacts are never modified once they exist. For example, the binary (download) file for a particular release.

 

image

Figure 8 – The Archive lifecycle
 
‘Basic CI’ lifecycle

The ‘Basic’ CI lifecycle (Figure 9) is for artefacts that must be archive and may be modified over time. For example, the Compiler used on the project may be updated over the course of the project’s lifetime.

image

Figure 9 – the Basic lifecycle

In addition, this lifecycle is used for artefacts that have not yet been released (see Release Management, below). Such items should be stored in the project’s RCS repository (for completeness) and should have their modification history recorded but these items do not need to be subject to stringent review or quality checks before being checked in.

‘Reviewed CI’ lifecycle

A ‘Reviewed’ Configuration Item must pass a review (peer or formal) before it can be checked back in (Figure 10). For example, a Requirements Specification.

This lifecycle is sometimes known as a Document lifecycle.

image

Figure 10 – The Reviewed CI (or Document) lifecycle

Any artefact that has been released from development, or originates from outside the development department should be subject to this lifecycle.

‘Software CI’ lifecycle

The Software CI lifecycle (Figure 11) is an extension to the ‘Reviewed CI’ lifecycle (above). In addition to being reviewed, a software CI should also pass Static Analysis testing; and pass Unit (component) tests; before being checked in.

 

image

Figure 11 – The software item lifecycle.

The Software CI lifecycle can be implemented in environments that demand very high software quality, for example safety-critical applications.

 

To get the most out of your Revision Control System each project artefact must be defined, with its usage, review policy and, perhaps most importantly, its change policy. This information gives the developer the information needed when an artefact has to change – what is required and who must be involved. Having effective configuration identification in place also ensures that the project’s artefacts retain their quality (at whatever level that quality is set)

Revision Control

May 23rd, 2011

What is Revision Control?

Revision control is the management of multiple revisions of the same unit of information. The focus is on controlling access to the artefact; and recording the history of changes.

Revision Control is variously known as version control, source control, source code management and several other titles.

Revision control has its roots in the management of engineering blueprints and paper documents. Today, any practical application of Revision Control requires the use of specialist software tools.

Artefact Identification

The core to revision control is the artefact. An artefact is a unique unit of information that may change over time (or not – some artefacts are deliberately unchanging – see below). Every revision-controlled artefact is identified by a unique name made up of:

Artefact Name

The artefact’s name represents a human-readable identifier. This is typically how the artefact is referred to during the development process. The Artefact’s name is constant – that is, it does not change.

Revision identifier

The revision identifier is used to track the history of the information. It is usually a number (e.g. 1.1); the highest number is the most recent version (revision)

image

Figure 1 – The artefact identifier reflects the name of the unit of information and its history

As the artefact is modified (revised) the revision identifier is incremented and a new entity is created (see Figure 1). Thus, artefact MyDoc v1.0 is a different entity to MyDoc v1.1 since it contains different contents. One of the principles of revision control is that this process is hidden from the user, and they only see the most recent version, unless they explicity choose to access the artefact’s history (by accessing a previous revision).

Problems with ‘file system’ Revision Control

The simplest form of revision control is to use the directory system on your computer. Each revision of a product is kept as a separate directory on the disk (Figure 2). This is a simple technique but is rarely very effective.

image

Figure 2 – Storing file revisions using a file system is rarely effective

Since few file systems have a history facility built in, each revision must result in the storage of multiple (complete) copies of each artefact.

Normally, using a file system as a repository implies using a networked file server. Most modern file systems will recognise that a file is open for edited and restrict access (see File Locking, below). However, to improve responsiveness most users will make a local copy of their ‘working files’. This can lead to a problem known as ‘Latest-write wins’. This means, if multiple developers copy the same file to modify it (usually in different ways) then each copy their modified file back to the server this can lead to the loss of any previous changes; and only the last developer to save has their revisions included. Clearly, to control access to each file requires careful management to ensure consistency. Such systems, requiring process and procedures to retain consistency are easy to abuse, particularly in the heat of the rush to delivery.

Another problem comes when artefacts are shared between systems; especially if each product will make different demands, and require different modifications, to the artefact.

Using your file system for source control is, at best, adequate for one-man projects.

Revision Control Systems

For anything more than trivial revision control (for example, with more than one developer) then a Revision Control System (RCS) is required.

A Revision Control System (also known as a Version Control System – VCS) is a piece of software that acts as an archive for artefacts. The RCS stores the latest version of the artefact and all revisions. The developer can take latest revision (often called the ‘tip’) or any named revision.

Typically, the RCS stores first version of artefact, plus changes between one revision and the next (known as ‘deltas’). Artefact version numbering is usually automatically performed by the tool.

Access control

To ensure integrity of artefact revisions access control is required. Typically this involves locking the file against changes.

Acquiring the lock is referred to as Checking Out.

Committing the change (releasing the lock) is referred to as Checking In.

There are two common methods for achieving access control: File locking and Revision Merging

File locking

In a file locking system only one developer has write access to the artefact (Figure 4). Other developers will have read-only access to the current (stored) version. The file is only available again once it is checked back in.

image

Figure 4 – File locking allows only one developer at a time to modify an artefact.

File locking avoids complex merges due to large-scale changes since only one developer has access to the file and the ‘merges’ are essentially the new changes made by the developer.

Because a file may be checked out for a significant time developers may be tempted to simply circumvent the system

Revision Merging

Most systems allow multiple check-outs of the same file (see Figure 5).

If the file is checked-out multiple times the first developer to check-in always succeeds.

image

Figure 5 – Revision merging allows multiple check-outs on an object

Subsequent check-ins must merge their changes into the current revision. For simple merges this may be performed automatically by the RCS software. More complex merges may (and typically do) require human intervention.


Revision Control System Configurations

Revision control systems are designed to allow multiple users to access and modify artefacts from any location, either locally or across a network. There are two basic configurations of RCS – centralised and distributed. Each has its own merits; and the choice of RCS depends on the type of project being developed.

Centralised database systems

In a centralised system there is one master reference copy of all artefacts (Figure 6). Clients access the artefacts by making copies of a subset of central archive. This subset is a ‘view’ on the repository known as a Workspace. The workspace acts as an additional form of access control. The client is free to modify any artefact within their workspace, but may not modify anything not in their workspace (in fact, it should not be visible to them). Workspaces allow CM Managers to restrict access of staff to only the artefacts that are relevant to them.

image

Figure 6 – In a centralised system clients view a subset of the central repository

Centralised systems are best suited to geographically-close, commercial development projects, keeping all the company’s artefacts (which comprise their intellectual property) in a central location (for ease of back-up, etc.).

Distributed database systems

In a distributed system each client’s workspace is a bona fide repository. Each client has a full copy of the archive, complete with all revisions (Figure 7). The client is free to modify the database as they see fit.

image

Figure 7 – Multiple versions of the archive exist in a distributed system

Copies of the archives are kept synchronised with periodic updates (known as patches) sent between each of the RCSs.

Distributed database systems are widely used for open source software development. They are well suited to developments that may be geographically distant and being modify according to widely differing requirements.

Fundamentals of Configuration Management

May 16th, 2011

Configuration Management (CM) is a core process in any development activity. Software engineers realise this more than any other discipline, but for many software teams using Subversion as a source code repository is as far as they get. Configuration Management is a lot more than this.

These articles introduce the fundamental concepts of Configuration Management and what they mean to the development team. Revision control is the most well known and understood CM activity but few extend this to develop artefact change policies; and even fewer develop comprehensive review policies.

Change Management is the key to controlling your software development. Without Change Management projects rely on heroic efforts and (often!) blind luck to deliver.

I’ll introduce a new set of patterns to control branching within a source control system. Coupled with effective Change Management these patterns can save many hours of effort in maintaining the source control repository (a common problem in large repositories)

What is Configuration Management?

Revision Control basics

Defining your Configuration Items

Baselines and Branching

Change Management

Releasing Code

%d bloggers like this: