Log in / create account
Openbravo.com
Issues
Forge
University
View source | Discuss this page | Page history | Printable version   

SCM Migration Process

Contents

Introduction

Openbravo has been using a centralized Source Code Management system (SCM; Subversion) since April 2006. Now (November 2008), Openbravo has decided to migrate to a distributed SCM (DSCM; Mercurial). This page holds information related to this decision and to the migration process.

Requirements

Development:

Administration:

Others:

Why Mercurial

This section does not include an exhaustive analysis of all the alternatives. It is just an enumeration of the main reasons to use Mercurial.

Development:

Administration:

Others:

As of version 1.1, Mercurial has some limitations:

Authentication

There are basically two ways of publishing repositories in Mercurial: through HTTP and through SSH. Both options have their pros and cons. HTTP is easier to implement and is available everywhere (not filtered). SSH gives a more granular control.

The decision is to go for the simplest solution. So we decided to first try with HTTP and only try SSH if we found that HTTP was not enough. It seems that starting with Mercurial 1.1 HTTP is enough for our needs. However, we may need some simple scripts to manage access through groups.

Every project will have at most four different zones:

Structure of repositories

Project structure

URL scheme: http://code.openbravo.com/[project]/[stage]/[subproject]

List of projects:

Common stages:

Example Openbravo ERP URLs:

Example Openbravo POS URLs:


Release publishing repositories

The idea is to have some read-only repositories for the released versions. These repositories will have different access permissions depending on the type of release:

When Openbravo used Subversion, the tags were not really tags. Some were modified. Others did not reflect a state or snapshot of the parent repository path. Because of this, we cannot use the natural way of managing tags in Mercurial, at least with the old ones.

Considerations and criteria:

Chosen solution:

Release stabilization branches

Bulbgraph.png   Release stabilization branches are no longer used.

With Subversion, whenever we wanted to make a new release, we created a new branch to isolate release stabilization changes from further developments done in the original branch. For example, for the release of 2.50alpha-r4, we created a branch from trunk called branches/2.50alpha-r4. Developers could continue fixing bugs in trunk and should only fix bugs in the 2.50alpha-r4 branch if the bug is critical for the release. This meant that the final tag is a snapshot of the stabilization branch and not a snapshot of trunk. This was not a problem in Subversion, because Subversions tags are just new branches. However, a tag in Mercurial is just a label for a revision. And in the previous example, those labels should point to trunk.

Although in theory Mercurial allows doing repositories with just one revision, it's rather ugly and this is a good opportunity to change the release process to a cleaner approach. So from now on, instead of developing in the original branch and doing stabilization in a new one, the stabilization will be done in the original repository and new developments will be done in clones. After the tag has been created, the clones can be merged back.

Processes

From the book Distributed revision control with Mercurial:

"...it's often good practice to keep a 'pristine' copy of a remote repository around, which you can then make temporary clones of to create sandboxes for each task you want to work on. This lets you work on multiple tasks in parallel, each isolated from the others until it's complete and you're ready to integrate it back. Because local clones are so cheap, there's almost no overhead to cloning and destroying repositories whenever you want."

We recommend this process of using cheap local clones. The general process is:

A developer could have several temporary clones of the same repository to isolate different developments. For example, if a developer is fixing two bugs simultaneously but wants the commits and tests to be isolated, he would use two separate clones.

Developers

Fixing a bug

Example: the developer has to fix issue 6265 in main.

The developer starts by doing a local clone of the branch he wants to fix: 

hg clone ~/src/erp/devel/main ~/src/erp/devel/issue6265_main

Then the developer can start fixing and testing the issue in that clone, doing as many commits as needed. Each commit should be an atomic piece. 

cd ~/src/erp/devel/issue6265_main
(edit fixme.java)
hg ci -m "Fixes issue 6265."

When the developer is happy with the fix, the changesets can be pushed to Openbravo repository: 

hg push https://code.openbravo.com/erp/devel/main

When finished, he can delete the clone and update his pristine clone: 

rm -rf ~/src/erp/devel/issue6265_main
cd ~/src/erp/devel/main
hg pull -u

Backporting a bug fix

Example: the developer has to backport the fix of issue 6265 from devel/main to stable/2.40.

The developer has already fixed the bug in one branch and wants to repeat the fix in another branch. First he clones the branch he wants to fix: 

hg clone ~/src/erp/stable/2.40 ~/src/erp/stable/issue6265_2.40

Then he can use transplant to apply the same fix in this branch (4544 is the revision of the commit done to fix this bug in main): 

hg transplant -s ~/src/erp/devel/main 4544

After testing the fix, he can push the update to Openbravo repository and delete the temporary clone: 

hg push https://code.openbravo.com/erp/devel/main
rm -rf ~/src/erp/stable/issue6265_2.40

The transplant extension is distributed along with Mercurial. Another possibility to backport changes is to use hg export and hg import.

Developing a new feature

New features can be developed in local repositories. But if other core developers or community members are also contributing to this feature, you want a public repository to share your developments.

Example: the developer wants to start a community project to develop a new feature based on main. 

hg clone https://code.openbravo.com/erp/devel/modularity ~/src/erp/devel/modularity

hg clone ~/src/erp/devel/modularity ~/src/modularity

hg clone ~/src/erp/devel/main ~/src/erp/devel/merge_modularity

cd ~/src/erp/devel/merge_modularity
hg pull ~/src/erp/devel/modularity
hg merge
(conflict resolution)
hg ci -m "Merged modularity."

Instead of pull+merge, the fetch extension could be used.

Once the merge is done, the changesets can be pushed into the centralized main repository and the merge repository can be deleted: 

hg push https://code.openbravo.com/erp/devel/main
rm -rf ~/src/erp/devel/merge_modularity

Finally, the modularity repository can be deleted.

New repository for new feature

Sharing changes with other developers is an easy task with SSH or using the built-in web server. However, there are situations where developers want to share the work with other developers or with the community in a centralized manner.

There is no built-in support in Mercurial for server-side copy of a branch (i.e. remote to remote cloning). However, an interactive script could be added to the server to provide this feature. An example session could look like this: 

$ ssh $USERNAME@code.openbravo.com 'create erp feature-1'
Clone another ERP branch? Answering no will create an empty repository [Y/N]: Y
Name of the branch to clone: erp/devel/main
...
Your new branch is at http://code.openbravo.com/erp/devel/feature-1

Core developers are allowed to create remote clones of the main repository.

An alternative to SSH is to create our own web interface to manage these requests.

New developer & access controls

Every repository will have one or more developers in change of them (manager). The Project Managers decide who can pull/push and who cannot.

To add a new developer, an interactive script could be added to the server to provide this feature. An example session could look like this: 

$ ssh $USERNAME@code.openbravo.com 'addhguser'
Create a new account? [Y/N]: Y
Developer user name: johndoe
Developer e-mail: john@doe.com
...
New user created. An e-mail has been sent to john@doe.com with the developer credentials.

Equally, to delete a developer account: 

$ ssh $USERNAME@code.openbravo.com 'delhguser'
Delete an existing account? [Y/N]: Y
Developer user name: johndoe
This is permanent. Are you absolutely sure? [Y/N]: Y
...
User johndoe deleted.

Finally, the developer in charge of a repository is capable of managing the access controls. An example session could look like this: 

$ ssh $USERNAME@code.openbravo.com 'hgacl'
Manage access controls? [Y/N]: Y
You are the manager of the following repositories:
  * erp/devel/main
  * erp/devel/localization
  * erp/devel/my-feature-1
Select a repository: erp/devel/my-feature-1
Developer user name (use 'all' for anonymous): johndoe
Current permissions of johndoe in erp/devel/my-feature-1:
 +pull
 -push
Select action: +pull
...
Permissions modified for user johndoe in erp/devel/my-feature-1:
 +pull
 +push

Release Management

Creating new stable repository

To be done only by Release Management. This could be done directly on the server. Or a ssh script/web interface could be created.

Releasing a new version

Example: Release Management wants to release 2.50alpha-r5.

A release is based on a tag. A tag is just a name for a specific revision. The process to make the release is: 

hg tag 2.50alpha-r5

Deleting a repository

To be done only by Release Management. This could be done directly on the server. Or a ssh script/web interface could be created. Repository Managers will notify Release Management when a repository is no longer useful.

Server infrastructure

Until now, all the developments were done in a single server. And all the developments done related to projects were done in the same repository. For example, the main Openbravo ERP development was done in the openbravo repository, in the trunk path. A trunk feature branch as done in openbravo/branches/feature-1, and also a customer branch was locate in openbravo/branches/cus_name. Given the centralized nature of Subversion, we were forced to use this approach.

Now, with Mercurial we want to split this into different servers, depending on the purpose. This means that all the developments done by core developers of Openbravo ERP and Openbravo POS will be done in one server. And the repositories used by the Openbravo Global Partner Services will be located in a different server.

Server requirements:

The number one candidate is Amazon EC2.

Migration

Migration tool

The migration tool should satisfy the following requirements:

  1. Allow migrating an entire repository.
  2. Allow migrating till a specific revision.
  3. Allow migrating starting from a specific revision.
  4. Migrate the entire untouched history.
  5. Allow mapping user names to real names.
  6. Allow saving a revision - changeset equivalent table in a text file.
  7. Manage tags and branches.
  8. Be reasonably fast (not more than 1 day for a 5GB repository).

Viable tools:

So the Convert Extension will be used to do the migration.

This is the command used to migrate trunk: 

hg convert --config convert.svn.startrev=1063 file:///mnt/svn-repos/openbravo/trunk /mnt/trunk_migrated /mnt/trunk_migrated.revmap

To convert a tag without preserving the branch name: 

hg convert --config convert.hg.usebranchnames=False  file:///mnt/svn-repos/openbravo/tags/r2.35mp5 /mnt/hg-repos/erp/release/2.35mp5

It takes 45min. The maintenance branches, DBSourceManager and Openbravo POS take around 15 minutes.

What

What will be migrated:

Milestones and tentative schedule

ID Milestone Owner When Status
1 Prepare the test Mercurial machine Jpabloae 17/12/2008 Done
2 Test the processes with volunteered developers Jpabloae 19/12/2008 Done
3 Port the current Subversion-Mantis integration to Mercurial Jaimetorre 20/01/2009 Done
4 Test and update development documentation using Mercurial Eclipse Jaimetorre 10/02/2009 Done
5 Write a Mercurial at Openbravo guide Jpabloae 10/02/2009 Done
6 Set up final server in EU Jpabloae 12/02/2009 Done
7 Migrate ERP's trunk and stable branches Jaimetorre 24/02/2009 Done
8 Set up e-mail notification hooks in mailing lists and in CIA Jpabloae 24/02/2009 Done
9 Announce migration to Mercurial Jaimetorre 25/02/2009 Done
10 Do internal training sessions Jpabloae 25/02/2009 Done
11 Set up server remote backups Jpabloae 25/02/2009 Done
12 Migrate all GPS repositories to Mercurial Jpabloae undefined Not started

Open issues

Terminology

References

Retrieved from "http://wiki.openbravo.com/wiki/SCM_Migration_Process"

This page has been accessed 31,015 times. This page was last modified on 2 February 2010, at 08:03. Content is available under Creative Commons Attribution-ShareAlike 2.5 Spain License.