Advanced Procedures

Delivering Code

Expectations

It's very simple: code in the delivery branch is intended for release. Code that isn't intended for release shouldn't be in the delivery branch. Code that accidentally makes its way into the delivery branch but shouldn't be there should be removed, either manually or by using the ct-undo method.

This page will discuss techniques to fulfill this expectations and also expose the many advantages of doing so.

Delivery Process

Quick rehash of the change and delivery process:

First, the developer uses a development branch/view that looks like this:

  intbranch_username_suffix

intbranch refers to the delivery branch where the code will eventually be delivered. username is the developer's username. suffix may be anything, but a couple of special suffixes are recognized by the system:

Note that all three conventions can be combined. You could, for example, create a development view with this name:

  main_coder_id00323_shared_buddy_nt

This would denote a view that uses MS-DOS text mode, that addresses bug id ID00323 and that will be using a development branch called main_coder_id00323_shared.

The developer makes his edits, builds and tests them until he is convinced that his code will meet expectations. He runs ct-bringover to merge his changes with other people's changes. He rebuilds and retests until again he is convinced that his code will meet expectation. Finally, he runs the ct-putback command, which triggers the following events:

  1. Verification that bug "ID00323" is indeed assigned to coder and to the main delivery branch, and is in the Open state.
  2. Check in of all checked out elements in the development branch. Abort if there are checkouts by other people, into other views or unreserved checkouts.
  3. Copy merge of all modified files into the delivery branch
  4. Collection of all checkin comments since the branch creation or since the last ct-putback.
  5. Registration of a Fix in the bug DB. A Fix is a collection of pointers to those versions of those elements that have been copy-merged into the delivery branch. This Fix has an id, which can be used as argument to the ct-undo and ct-refo commands. Note that a bug may have multiple fixes, either intentionally or because QA detected problems the developer couldn't see himself.
  6. Bug "ID00323" set to the Fixed, but Unlabeled state.

Internally, a Fix is really a version of the special element /vob/history/history, which gets ClearCase hyperlinks attached to it pointing to the modified versions. The checkin comment of that history element is the putback comment, and the contents is used for various purposes.

Code Review

Code review serves two purposes:

The putback process described above is modified as follows:

The name of a development branch/view looks like this:

  intbranch_coder-reviewer_suffix

Note that the username portion now contains both the name of the developer and the reviewer, separated by a hyphen (not underscore!). The remaining conventions remain the same.

A developer running ct-putback will not trigger the copy merge, but instead run a putback preview. The results of this preview will be emailed to the reviewer, and a Review Fix will be appended to the bug, which is placed into the Review state.

The reviewer can inspect the fix either via the web interface or by running the script stored in the /vob/history/history special element. He may then decide to reject the fix, which will simply put the bug back into the Open state and notify the developer, or he can accept the fix, in which case he assumes the role of a developer in the normal process (i.e. he is responsible for merging other people's changes via ct-bringover).

A special case exists when a view is defined with the same reviewer as the developer. This may be useful for practising the review process. In this case, and if one wants to simulate a reviewed putback from a development view that doesn't specify a reviewer, ct-putback -accept must be used to actually deliver code into the delivery branch.

Partial Deliveries

It often happens that a complex bug fix can be decomposed into multiple steps, some of which meet the expectation of deliverable code whereas others are high risk or not urgent. Also, a partial fix to one bug may simplify fixing other bugs, making it convenient to deliver that partial fix.

Using the -partial option to the ct-putback command will register this as a partial fix and keep the bug state Open.

Sharing Code

Sharing between Different Development Branches

It often happens that code changes done by one developer are of interest to other developers. The easiest way to accomodate this is by dividing your work into multiple steps and using partial deliveries to place this code in the delivery branch.

Note: Resist the tempation to deliver. Don't ever violate the expectations set on a delivery, and don't use delivery for the sole purpose of sharing code!

If a partial delivery is not possible, then a developer wanting to use code from another developement branch can use the -from option of the ct-bringover command. This will transfer all the changes made in that development branch, merging as required. It is a good idea to do an ordinary ct-bringover prior to doing this, since this will ensure that changes coming from a third party are merged separately.

If a bringover over all changes is too much, a restricted bringover may be run by specifying the paths to the subdirectories over which the merge should run.

Finally, it is also possible to merge over one element at a time by using ClearCase's findmerge command, for example like this:

% ct findmerge file.c -fver '.../main_otherguy_id00324/LATEST' -nc -merge
Needs merge ...
...

Do not ever just copy files from somebody else's view. This will create gratuitous merge conflicts since you are denying ClearCase the information on where the changes came from. You are also likely to destroy other changes to that file, including yours. Arbitrary copying from other views can create a slew of other problems that have annoying and non-intuitive symptoms (e.g. eclipsed files). Please don't do it.

Sharing the same Development Branch

If a group of people knows that they will be working closely together on a particular change, they may elect to use a shared development branch. To do this, they first elect a captain that will be responsible for executing the ct-bringover and ct-putback commands. The captain starts the process by creating a view named, for example, like so:

% ct-mkview main_coderguru_id00345_shared
Creating view "main_coderguru_id00345_shared"...

The other developers latch on like so:

% ct-mkview main_coderguru_id00345_shared_juniorcoder
Creating view "main_coderguru_id00345_shared_juniorcoder"...

This will create a separate view for juniorcoder which will use the same development branch (main_coderguru_id00345_shared) as coderguru. This will have the following effects:

This process should be very familiar to users of systems like CVS.

Building and Labeling

Expectations

The expectations set for the code in the delivery branch are:

It is therefore not the responsibility of the build process to "make things work". If the build fails, then somebody violated a rule above and it needs to be fixed there, not at build time.

The importance of a regularly scheduled, automated nightly build is to expose violations above, and also to eliminate "tweaking" of the build configuration, making it trival for a developer to reproduce a particular problem because it will be trivial to understand how a particular build was produced.

Creating the Build View

To create a build view, use the ct-build command. This will create a view selecting time frozen versions from the delivery branch. The time is chosen to be one where no ct-putback is in progress, ensuring that a consistent set of versions is selected.

You can reuse an existing build view, allowing for incremental builds.

The name of the build view is, by default, the name of the delivery branch preceded by the Build_ prefix.

Labeling the Result

Labels used in this process serve two purposes:

Note that labels do not select the versions of elements participating in the build, and that one should not use them that way except to reproduce an old build.

Labeling after a build serves to emphasize this restriction, and also allows one to use the same label to label build results, if such are kept under version control.

If the build requires knowledge of the label that will be used to label the result, the -preview option of the ct-mklabel can be used from within the build script. This is particularly meaningful if you generate the labels via the -auto option of the ct-mklabel wrapper script.

In particular, labels should not be moved individually. If it becomes necessary to revert to an older version of an element, that older version should be restored via the normal change process. Not doing so violates the expectations set on an delivery branch and will end up causing considerable grief.

Labels are created and moved using the ct-mklabel command. This command will prompt you for a list of Fixed but Unlabeled bugs to associate to that label. To keep the situation consistent, all of these bugs should end up being associated with some label. It is perfectly OK to create multiple labels for a single build, but note that you cannot associate more than one label to a bug.

Running ct-mklabel with the -replace option will move the specified labels forward. For transient releases, this is the recommended process, as it will keep track of which bugs were added or removed from a release. No value should be placed on the ability to reproduce an old transient release.

Running ct-mklabel with the -release option will finalize the location of that label and create a bugfix branch.

A label name can be generated automatically by invoking ct-mklabel with the -auto option. The resulting label name is composed of the specified prefix, followed by the delivery branch name, followed by a version number. For example, if your project id is ID and your delivery branch name is "gold", the generated label name could be R-ID-gold-345.

Now suppose you would create a bug fix branch off that label. The name of that branch would be b_R-ID-gold-345. Generating a new label on that branch would look like this: R-ID-b_R-ID-gold-345-15. This is a triffle long and can be simplified easily to R-ID-gold-345-15, and that is exactly the name generated by ct-mklabel with the -auto option on a bug fix branch.

Package Labelings

Package labels exist to allow partial builds of one or more packages. Package labels are still applied to the complete source tree, but only those bugs in the bugDB that have a matching package name in their package field will be listed as Fixed.

A package label is specified by appending an unambigous substring of the package name consisting solely of upper or lower case letters to the project id at the beginning of the label name.

Maintaining Multiple Delivery Branches

Since the basic delivery process serializes deliveries, a single delivery branch can become a bottleneck if many developers deliver changes into it. Also, you may want to create multiple delivery branches that represent various stability levels of the code. Finally, if you use ClearCase MuliSite, you have no choice but to create separate delivery branches, one per site.

Communicating between Delivery Branches

The primary vehicel for transfering changes from one branch to another is the ct-bringover command with the -from option specifying the source delivery branch. This will merge over all changes from the source delivery branch to the current branch.

The strict way to do this is to create a development branch/view for this type of changes and run the ct-bringover -from command there, followed by the usual ct-bringover and ct-putback sequence. If for your particular project, merge conflicts are rare, then you can also run the ct-bringover -from command directly from within the delivery branch's view.

If you want to pick and choose among all the deliveries into the source delivery branch, you can use the ct-redo command. Note that it is a lot easier to replay putbacks in chronological order.

Bug Fix Branches

A bug fix branch is created by finalizing a label as described above. The name can be a little long, but it otherwise acts like any other delivery branch.

In practice, a bug fix branch is really a "pre-release" branch. One should create it whenever there is a conflict between delivering ongoing feature changes versus delivering stabilizing fixes towards the next release.

In time, a series of bug fix branches will sprout from the ongoing delivery branch, and fixes delivered into bug fix branches should be brought forward from the oldest to the newest branch by using successive ct-bringover -from commands.

The timing of the creation of a bug fix branch can be wrong on occasion. If it is too hard to determine the proper moment for branching, then the following strategy may work better:

  1. Create the bug fix branch when you're sure you need it;
  2. Deliver ct-undo's into the newly created bug fix branch to remove unwanted deliveries from the ongoing development branch;
  3. Use ct-redo to backport fixes from one branch to the other.

Note that you should not mix the ct-bringover -from strategy with the ct-undo/ct-redo strategy, since you may end up undoing fixes in the ongoing delivery branch and create very confusing merge situations.

The ct-undo/ct-redo strategy has the disadvantage of requiring an action for every delivery that needs to be back ported, but does have the advantage of not comitting you to any particular delivery branch before starting the change. Often, fixes that appear simple turn out to be very risky, and postponing the decision on where to fix it can be useful on occasion.

Variant Branches

A variant branch is a special kind of delivery branch. Views based on this delivery branch will select versions following a variant hierarchy. The intended usage model can be exemplified by, for example, considering language variants:

         generic
          /   \
    english   french
     /   \
    uk   us

By choosing the proper target delivery branch for a particular change, one can easily propagate this change down the hierarchy, using the ct-bringover -parent option. For elements that do not have a variant, this will not even involve a merge, but will be automatic.

A variant branch is created by running the ct-mkbranch command with the -variant option. Any delivery branch may serve as a basis for a variant.

Multisite Branches

When using multisite, every site must use their own set of delivery branches. It turns out that the best way to organize this is by making those delivery branches variant branches of some master branch. It is recommended that the variant branches are named using the master branch name as the prefix, followed by hyphen and a short site id (airport ids for example). This not only makes the parent child relationship visible, but also the mastership.

Changes in the variants are communicated by starting a normal change in a development branch, then executing a ct-bringover -parent.

Changes in the master branch are pulled from the variant branches by executing a ct-bringover -from. Effectively, changes will be crossmerged in the pattern depicted on the diagram to the right.

This technique has the advantage that the masterships of the delivery branches remain constant. No reqmaster business. It does introduce a little delay in the propagation of changes, though, since somebody needs to actually do the crossmerging. If the activities performed at the different sites are orthogonal, this crossmerging may be automated, but it is somewhat risky to do so in general. Humans being as they are, conflicting merges resulting from an automated process tend to get ignored, only to be resolved under time pressure at release time, greatly increasing the risk.

Fixed Branches

Fixed branches start off from a branch point label and do not have the automatic update property of variant branches. All changes must be explicitly merged over. This is the branch type the ct-mkbranch command will create by default.

Note that branches must be assigned to projects if they are to appear as an option for assigning bugs of that project.

Custom Branches

In rare occasions or when switching from a legacy system, it may become necessary to create a branch based on a special selection of versions. For example, a combination of imported labels may be needed to define a set of versions, or a combination from several branches:

  1. Use the administration view, usually called vobadm or cmadmin, and define a config spec that will select the appropriate versions of the elements to be labeled or branched: 
    % ct setview vobadm

% ct edcs
... edit config spec ...
  2. Run ct-mklabel or ct-mkbranch as required.

Note: Care should be taken to ensure that some version of all elements linked into a selected directory version is selected. You can usually ensure this by making the last rule in the config spec be element * /main/LATEST. It will usually be the case that too many elements are selected. If this is a problem, a cleanup change should be made, pruning the directories to contain only the elements that should be there. If this is done, do not use ct-bringover -from to merge over the changes done on the custom branch. Use ct-redo instead, otherwise the cleanup (deletions) will be done on the target branch, something you probably don't want.

Recovering from Failures

Please report all bugs, failures or weirdnesses so that they can be fixed. Please do not build process to work around bugs. Help fix the bugs instead!

Undoing a Removal

The important thing to remember is that renames, additions and removals of elements from a directory are all harmless and recoverable, so don't panic!

It helps to visualize additions, removals and renames as editing the containing directory. It's very much like adding, removing or changing lines in a file, and the recovery procedures are similar.

The easiest thing to do is to simply cancel the checkout of the directory. This will restore the previous version of the directory, and the removed elements will reappear, as if by magic:

% ct co -nc .
Checked out "." from version "/main/234".

% ct rm file  <=== OOOPS!!
Removed "file".

% ct unco .
Checkout cancelled for ".".

% ls file
file

%

If you don't notice the error immediately and have already checked in your directory, you can take advantage of the evil twin trigger and allow it to help you. Simply attempt to re-mkelem the deleted element and follow the instructions:

% ct co -nc .
Checked out "." from version "/main/234".

% ct mkelem file
ERROR:  An element named "file" already exists in
        in some other version of ".":

Instead of creating a new element, you probably want to
create a hard link to the existing element, like so:

cleartool ln .@@/main/LATEST/file .

% ct ln .@@/main/LATEST/file .
Link created: "./file".

%

In general, directories don't contain elements but contain named links to elements. Removing an element doesn't destroy the element, it just removes the link. It is therefore almost trivial to resurrect a removed element.

One consequence is that removing directories and removing files are really the same operation, and the recovery procedure is identical. In particular, if you accidentally removed a whole tree, you only need to recreate the link to the topmost directory of that tree.

Undoing a mkelem

Most sites disable the rmelem command, therefore you cannot undo a mkelem directly. Instead, you should use the rmname (rm for short) command.

Suppose, for example, you accidentally created the subdir element as a file element instead of a directory element. Use this procedure to correct the problem:

% ct mkelem subdir   <== error!
Created element "subdir" (type "compressed_file").
Checked out "subdir" from version "/main/0".

% ct ci -identical -nc subdir
Checked in "subdir".

% ct rm subdir
cleartool: Warning: Object "subdir" no longer referenced.
cleartool: Warning: Moving object to vob lost+found directory as "subdir.9e881d0d390711d5b3ee000180a933fe".
Removed "subdir".

% ct mkdir subdir
Created directory element "subdir".
Checked out "subdir" from version "/main/0".

%

The checkin is mainly for cosmetic reasons and to avoid confusion if this procedure is used in a snapshot view. In the end, the element isn't removed but just relocated into the trash bin. The ClearCase administrator will empty out the trash once in a while.

This procedure will not work as described if you don't notice the error immediatly and check in the containing directory. If you later check out the directory and remove the element, you will notice that it doesn't get relocated into lost+found, since the previous version of the directory still has a reference to that element. If you then attempt to create a directory element with the same name, the evil twin trigger will get in your way. You can fool the trigger, though, by first creating a directory element with a different name and then renaming it using the ct mv command.

Eclipsed Files

Symptoms of this failure are messages like this:

...
cleartool: Error: Not a vob object: "filename".

This can happen during ct-bringover, ct-putback or even during a simple ct checkout. You can diagnose the problem using ct ls, as follows:

% ct ls filename
filename
filename@@ [eclipsed]

This output indicates that a view private file is covering up an element under version control (eclipsing). This can happen in various ways, the most common being that the developer copied in some new files from some other developer's branch and later executed a ct-bringover that made those new files suddenly appear as elements under version control. The bottom line is: never copy stuff, always use ct-bringover.

It may happen on occasion, usually after someone's putback failed, that the special file called /vob/history/history becomes eclipsed. This will prevent ct-bringover and ct-putback from working. This may even take on the more interesting form that looks like this:

% ct ls /vob/history/history
/vob/history/history@@some-version
/vob/history/history [checked out but eclipsed]

In this case, cancelling the checkout is the proper recourse, but it may involve adding the CHECKEDOUT rule to the view's config spec.

Breaking Locks

Occasional failures may leave locks in place. Use the ct-unlock wrapper script to remove locks. Please use this with care, and double check that the lock is not active. Running concurrent ct-bringover or ct-putback operations will cause a lot of grief.

The locking process used by the wrapper scripts is as follows:

  1. Attempt to grab the METALOCK. This is actually a label type that only exists as an excuse to lock and unlock it.
  2. Rearrange branch locks.
  3. Release the METALOCK

Prior to removing any locks, you may want to browse the history of the locks in order to help figure out who left the locks in place and why. For example:

% ct lshistory -minor lbtype:METALOCK@/vob/admin
20-Apr.11:30   cg         unlock label type "METALOCK"
20-Apr.11:30   cg         lock label type "METALOCK"
  "Locked for all users."
20-Apr.11:08   cg         unlock label type "METALOCK"
20-Apr.11:08   cg         lock label type "METALOCK"
  "Locked for all users."
20-Apr.11:08   cg         unlock label type "METALOCK"
20-Apr.11:07   cg         lock label type "METALOCK"
  "Locked for all users."
...

Putback Failures

All the wrapper scripts are equiped with code to trap most of the common failures. You will see a message similar to this one:

### An error occured while processing the following command:
###  findmerge ...
### Please REPORT this error together with any context info to:
###  vobadm@digisle.net

The recovery code will usually do a good job of restoring the state before the operation. The proper procedure is to retry the operation, adding the -debug flag if so desired. If that operation fails again, please send the output to the email address specified in the message.

Please report these occurrences! They should be rare and are usually an indicator of serious problems.

Sometimes, though, the failure will occur past the point of no return, and the recovery code will attempt to restore a well defined state so that you can redo the operation but only end up catching up on the cleanup.

Should a ct-putback operation fail in some very unusual way, follow these steps to restore the state prior to restore sanity.

  1. Replace lock on the delivery branch with a write lock
  2. Cancel all checkouts in the delivery view
  3. Cancel the checkout of the history element in the development view
  4. Remove locks
Assume ib is the delivery branch and ib_user is the development branch. The commands you need to issue are as follows:

% ct-lock -replace -nuser user ib
Locked branch type "ib".
% cd /view/ib
% ct lsco -short -cview -avobs | sort -r | xargs cleartool unco -rm
...
% ct-unlock ib_user
Unlocked branch type "ib_user".
% ct unco -rm /view/ib_user/vob/history/history
...
% ct-unlock ib
Unlocked branch type "ib".

Sometimes further action is required, see above in the eclipsed files section.

©2002 CG software design