Release Process
Introduction
This document describes the different phases of the release engineering process leading up to the actual build and publication of a 3.0 release, both the Community Edition and the Professional Subscription.
Overall process
- package
- pass the machine to QA for testing
- QA will mail all fine to proceed with publish
- we do publish in QAApproved status
- for 17Q1.1 the plan is to put in CS
- CS: get okay with DME and PLU for final promotion to CS and do promote_cs for both erp and retail together
Package 1st of each month (normal)
Check issues target (normal and emergencies)
- Check in issues that all the issues targeted for the current mp are close (example for 17Q1.1)
- Check that it is not needed to wait for a specific commit.
Disable int-promote (normal)
Disable job http://ci.openbravo.com/job/int-promote/
Create backport repo (normal)
Create the backport repo for the current release, so pi is not blocked for next release. Note that backport repositories are created based on main, so they cannot be created at this stage.
ssh code.openbravo.com su - su - www-data ./make_erp_backport_repo.sh 3.0PRyyQn e.g: ./make_erp_backport_repo.sh 3.0PR17Q2
Update the release status in aha (normal & emergency)
- Check for the release status in aha and update it.
Login using ismael account Navigate to Releases -> Details Select the current release version in the top tab bar Check 'Status' and update it * For normal first package status must be "Send to QA" * On publish of normal release must be "QAA" * On first CS of the emergency release it must be "CS"
Email pi open for next mp (normal)
Send an email to release management list
PI is now open for PRyyQn commits. The dates for PRyyQn are the usual ones [1]. [1] http://wiki.openbravo.com/wiki/Release_Policy#ERP_release
e.g: for sending erp + retail email saying repos are opened
recipients: release.management@openbravo.com subject: erp & retail repos open for 17Q3 and Backports available for 17Q2 Hi erp/devel/pi and erp/pmods open for 17Q3 Backports for 17Q2 available at: erp: https://code.openbravo.com/erp/backports/3.0PR17Q2 retail: https://code.openbravo.com/retail/backports/3.0RR17Q2 Regards RM
Update the releaser14 (normal)
Update the packing machine and put the current date in last-update column for releaser14 in server-update-sheet
# connect to release14 sudo su - apt-get clean apt-get update apt-get dist-upgrade apt-get autoclean apt-get --purge autoremove reboot
Remove permissions to push backport repo (emergencies)
Before run the package of a emergency release is needed to remove push permissions to the repo , so no new commits come unexpectedly during the process.
- Comment the allow_push for the current release in hgrc file
ssh code.openbravo.com -p 19198 -lubuntu ubuntu@code ~ $ su - code ~ # vi /scm/hg/config/hgrc [web erp/backports/3.0PRxxQy.z] #allow_push = @core, @packagers allow_push = staff.rm
- reload apache2
code ~ # /etc/init.d/apache2 reload
commit the change to the repo e.g:
cd /scm/hg/ hg ci -m "Revoke access to 3.0PR16Q3.5 repo" -u "Rafa Alonso <ral@openbravo.com>"
Run the package (normal and emergencies)
Connect to the releaser
ssh packager@releaser-14.openbravo.com -p19198
Check for disk space
df -h
- required 9GB erp and 6GB for retail in ~/outputs
- ~/build has alteast 3GB)
Remove the old releases in ~/outputs/
cd ~/output # criteria to delete * Not latest emergency releases * The releases promoted to CS
Clean the folders from ~/repos/
cd ~/repos # criteria to delete * Remove old release repos. If the release is published remove it.
Clean the folders from ~/build/
cd build rm -rf 3.0*
For all packages it is needed to create the qa instances, except for the repackages of normal releases, and also it is needed to be careful to not delete the instance that they are using for this normal releases. This is pending to automate.
screen cd ~/releaser
Normal :
./package 3.0PRyyQn 2>&1 | tee ../logs/package-3.0PRyyQn.log
Emergency :
./package 3.0PRyyQn.o build-qa 2>&1 | tee ../logs/package-3.0PRyyQn.o.log note: build-qa is same as executing ./build_qa_instance 3.0PRyyQn.o after ./package
2 emails should be received at staff.rm@openbravo.com
- qa machine
- package done
Update qa machine (normal)
- qa-erp.openbravo.com (i-f0280b3c in ap-south-east-1)
ssh qa-erp.openbravo.com -lopenbravo sudo su - /etc/init.d/tomcat stop apt-get -qq update && echo '------- >> finished update' && apt-get -qq dist-upgrade && echo '------- >> finished dist-upgrade' && apt-get -qq autoclean && echo '------- >> finished autoclean' && apt-get -qq --purge autoremove && echo '------- >> finished --purge autoremove' && reboot
If needed, QA will ask for another machine to test update, check with QA if needed an instance with previous QAA or CS release
./build_qa_instance 3.0RRyyQn.o 2>&1 | tee ../logs/build_qa_instance-3.0PRyyQn.log
Modules reminder (normal)
Note: only normal releases, no emergencies. And in the first package. Important: Retail should be also packaged and the merge of release versions merged into pi (pmods) repos of retail should be done
cd ~/releaser # Do a first check without send mails , check that no appear personal emails, only should appear the usual teams ones (erp engineering support one should not appear also) ./modules-reminders.sh package 3.0PR17Q1.1 emails-no 2>&1 | tee ../logs/modules-reminders-package-3.0PR17Q1.1.log # real run sending the mails to teams ./modules-reminders.sh package 3.0PRyyQn.o emails-yes 2>&1 | tee ../logs/modules-reminders-package-3.0PRyyQn.o.log
After the first notification remind weekly to the teams till done.
Note : Add calendar entry in time span of one week to check of confirmation mail from teams.
Complete list of fixed issues in the PR (normal)
Add the list of issues/features fixed from last MP to the release changelog.
Complete list of fixed issues in the PR (emergency)
1. Get the text about the issues fixed in the release from:
$ ssh issues.openbravo.com -p 19198 -lstaff.rm $ su - # /rt/changelog_script/get_backport_issues.sh 3.0PRyyQn.x e.g: # /rt/changelog_script/get_backport_issues.sh 3.0PR16Q4.1
2. Edit http://wiki.openbravo.com/wiki/Release_Changelog
- add a new entry for the release.
- paste the text returned by the script above
Optionally can be checked with roadmap page to ensure that is correct: https://issues.openbravo.com/roadmap_page.php
Release Notes (normal and emergencies)
![]() | In order to allow developers to write the release notes when they commit some issue/project, the release notes for the next release should be also available. |
- (normal) Create the wiki document Release_Notes/3.0PRyyQn and copy the structure of the template.
- (emergencies) Create the wiki document copying the previous emergency or normal release
- (normal) Run the job https://ci.openbravo.com/view/rm/job/int-release-notes/ to get the 'what's new' and 'extension' sections.
- (normal and emergencies) Write the estimate publish date and the version
- (emergencies) Mark previous release as canceled (C)
- (normal and emergencies) Update online demo section text
- (normal) Prepare the Release Notes for the next release based on the this template
e.g, for PR16PR4.1
- create a wiki page: http://wiki.openbravo.com/wiki/Release_Notes/3.0PR16Q4.1
- pick up the text from http://wiki.openbravo.com/wiki/Release_Notes/3.0PR16Q3.1 and
- change every reference to PR16Q3 to PR16Q4
- update the Release date to today
- update the Module version to the module version of http://wiki.openbravo.com/wiki/Release_Notes/3.0PR16Q4 + 10 -> 3.0.30430 + 10 = 3.0.30450
- update Maturity Status to QAA
- update the fixed issues list with the one from the http://wiki.openbravo.com/wiki/Release_Process#Complete_list_of_fixed_issues_in_the_PR_.28emergency.29 step
- paste the text in the new http://wiki.openbravo.com/wiki/Release_Notes/3.0PR16Q4.1 page
Update API changes wiki (normal and emergencies)
Update the API changes list, as follows:
- Rename PI to PRyyQn, e.g. PR14Q2.
- Create a new empty section for PI.
Email of release notes (normal)
Send an email to team leaders to write the "Fixed issues" section with meaningful messages.
To: asier.lostale@openbravo.com eduardo.argal@openbravo.com egoitz.castillo@openbravo.com pablo.sarobe@openbravo.com victor.martinez@openbravo.com dmitry.mezentsev@openbravo.com staff.rm@openbravo.com Subject: 17Q2 ERP release notes to fill fixed issues Hi Please update the fixed issues section of 17Q2 Release Notes: http://wiki.openbravo.com/wiki/Release_Notes/3.0PR17Q2 List of issues: http://wiki.openbravo.com/wiki/Release_Changelog#3.0PR17Q2 Regards RM
Email of packaging (normal and emergencies)
Send an email to release.management@openbravo.com following this template example:
Subject: 17Q1 Final package (erp and retail) Hi 17Q1 repackaged, QA Machines can be updated via MMC with QA maturity. ERP Version: 3.0.31087 Changelog: 35371: Wrong orders shown in "Create Invoices From Orders" window Retail Version: 1.8.2502 Changelog: 35021: Copyright year extend to 2017 for pos hardware manager Regards RM
Second package (normal and emergencies)
Do the package (normal and emergencies)
Run the package
cd ~/releaser ./package 3.0PRyyQn 2>&1 | tee ../logs/package-3.0PRyyQn.v2.log Note : If QA request to rebuild the test machine pass 'build-qa' as parameter to package command ./package 3.0PRyyQn build-qa
Update the release changelog
Get the changelog
cat ~/logs/3.0PRyyQn-changelog.log Example: cat ~/logs/3.0PR17Q1-changelog.log Add the new issues to Release_Changelog Note: Check for the issues sorted in ascending order
Email of the package (normal and emergencies)
For announce this change search for previous release mail, and edit it accordingly:
- maintain the receivers of the mail
- if gmail create links in the copy paste, put the mouse in the link and remove it
- read the entire mail, to try to avoid copy paste errors
Template
To: release.management@openbravo.com, staff.qa@openbravo.com Subject: yyQn packaged (erp and retail) Hi YYQn for ERP and Retail packaged for testing ERP version : 3.0.<version> - PRyyQn Changelog : http://wiki.openbravo.com/wiki/ERP/3.0/Release_Changelog#3.0PRyyQn QA Machine : http://qa-erp.openbravo.com Retail version : 1.8.<version> - RRyyQn Changelog : http://wiki.openbravo.com/wiki/Retail:Release_Changelog#3.0RRyyQn QA Machine : http://qa-retail.openbravo.com/openbravo Regards RM
Publish (normal and emergencies)
Wait for the Product manager and QA to approve the publishing
Check for 'lost' commit done after last package (normal)
The repository was closed to further pushes when the packaging started, so this step can be skipped.
If for any reason new pushes have been added, stop all publish and check.
Run the publish script (normal and emergencies)
In the releaser machine
ssh packager@releaser-14.openbravo.com -p19198 cd ~/releaser ./publish 3.0PRyyQn 2>&1 | tee ../logs/publish-3.0PRyyQn.log
Verify that the process exits with status 0
Modules reminder (normal)
Note: Should be done only once when the version is first promoted to QAA maturity.
cd ~/releaser # Do a first check without send mails , check that no appear personal emails, only should appear the usual teams ones (erp engineering support one should not appear also) ./modules-reminders.sh publish 3.0PR17Q1.1 emails-no 2>&1 | tee ../logs/modules-reminders-publish-3.0PR17Q1.1.log # real run sending the mails to teams ./modules-reminders.sh publish 3.0PR17Q1.1 emails-yes 2>&1 | tee ../logs/modules-reminders-publish-3.0PR17Q1.1.log
After the first notification remind weekly to the teams till done.
API docs
This step is fully automated. The javadoc is created on packaging, uploaded to code.openbravo.com/docs and its index page is updated.
API check
Ths package/publish is also taking care of re-genearing the api-checks data on every (non-emergency) release and pushes it to api-checks.
Repositories
Merge to main. Rearrange tags (normal and emergencies)
This step sorts the tags in the repository. Sorted tags are required for other RM scripts to work. No steps required for normal release. No steps required for the last version. You can also verify if the tags must be rearranged, looking at the tags in the repository https://code.openbravo.com/erp/devel/main/tags. If the published version is not the first one, rearrange is required.
Partially automated for emergency release publish. Re-ordering of tag is required. Check of the publish script message "NEED to ORDER the releases"
ssh packager@releaser-14.openbravo.com -p19198 cd /home/packager/output/3.0PRxxQn.m/merge-main # Manually reorder the entries in .hgtags and .hgsigs hg resolve -a -m hg ci -m "Merge temporary head for $TAG" # Check that everything it is fine and push hg glog | more hg out hg push
Merge main to pi (normal and emergencies)
NOTE: if publishing a emergency release during release process of a normal release, then not needed to do the merge of tags form main to pi, since when done this merge for the normal release this will automatically done. Is possible to do the merge from main to pi of emergencies without wait to the normal release, but ensure when merging the normal release from main to pi that the tags are correctly merged and none is lost.
1. Merge tags from main to pi:
ssh ubuntu@code.openbravo.com -p 19198 su - su - www-data ./merge-main-pi-erp.sh 2>&1 | tee merge-main-pi-3.0PRyyQn.log
2. Verify the merge
cd /tmp/merge_main_pi_erp_temp
- tags are in order
hg tags
- changesets are correct (emergencies), verify that the expected changesets are present
hg log -G | less
3. Finally
hg push
Create new backport repo in code (emergencies)
Note that the backport repository is created based on main, so, once the publish script has finished, the backport can be created
- Close the current used backport repo and create a new one for the next versionck.
ssh ubuntu@code.openbravo.com -p 19198 su - su - www-data cd /var/www ./make_erp_backport_repo.sh 3.0PRyyQx.y e.g: if publishing PR16Q4.1 ./make_erp_backport_repo.sh 3.0PR16Q4.2
- Verify that the old release folder in erp/backports has been removed and that the old repository is no longer shown in the list of backports https://code.openbravo.com/erp/backports
Restore permissions to the repos
Change permissions
ssh ubuntu@code.openbravo.com -p 19198 su - su - www-data cd /scm/hg/config vi hgrc # e.g. for PR16Q4.1 [web erp/backports/3.0PR16Q4.1] allow_push = @core, @packagers
commit changes to the config repo
hg ci -m "Grant access to PR16Qx.y backport repo" -u <username <email>>
reload apache2 as root
ctrl+d /etc/init.d/apache2 reload
Push the automation tag (normal)
Verify that the changeset is the correct one, always the tip
- check the email of the push that passed try
- if any doubt ask QA
https://ci.openbravo.com/job/try-gui-pgsql
Then
cd ~/repos/automation_pi hg pull -u hg tag <3.0PR19Qn> hg out hg push
Update status in AHA (normal)
Refer to Update_the_release_status_in_aha
Mantis (normal and emergencies)
- Go to https://issues.openbravo.com , and select manage -> projects -> openbravo-erp.
- In the versions mark the current 3.0PRyyQn version as released.
- Create a new target versions of the type of the just published. Idea is that should be two actives branches in each thread. For example should be 2 normal released available, 2 target versions of q1.x and 2 target versions of q3.x.
- Verify that the order is correct. The order is date based, so modify the date of the new records so they are properly sorted
Release notes (normal and emergencies)
In http://wiki.openbravo.com/wiki/Release_Notes/3.0PRyyQn
- Update the date to today's date
- normal:
- Check the release notes is complete check with stefan/dmitry
- Add the community appliance amis
- Update the Maturity Status: QAA for the last release; CS for the rest
- Clean up the webpage
- Remove the WorkInProgress banner
- Remove the "Great feature" texts
- Remove text 'estimate date'
- emergencies:
- Update the Maturity Status: CS
- Update the previous Maturity Status: C; in release-history and actual release notes
In 3.0 release notes, add a new entry for the new release, copying another and updating the links
- If a transplant was requested within the process, remember to recreate the list of defects and update the release notes
Add CS date to calendar (normal)
Add to calendar the day to pass CS, it is fixed date, the last day of moth in two months after the publish in qaa.
Announcements
Pre-Sales (normal and emergencies)
need to be sent to
Xavier Places <xavier.places@openbravo.com>, Dmitry Mezentsev <dmitry.mezentsev@openbravo.com>, "Staff.RM OB" <staff.rm@openbravo.com>
Email template example:
Recipients: Xavier Places <xavier.places@openbravo.com>, Pablo Rueda <pablo.rueda@openbravo.com>, Dmitry Mezentsev <dmitry.mezentsev@openbravo.com>, "Staff.RM OB" <staff.rm@openbravo.com> Subject: Released 16Q4.1 in 'QA Approved' and 16Q3.5 'Confirm Stable' status New versions of ERP and Retail are published in 'QA Approved' and 'Confirm Stable' maturity. Also ready to announce 16Q4.1 QA Approved Maturity: http://wiki.openbravo.com/wiki/Release_Notes/3.0PR16Q4.1 http://wiki.openbravo.com/wiki/Retail:Release_Notes/3.0RR16Q4.1 16Q3.5 Confirm Stable Maturity http://wiki.openbravo.com/wiki/Release_Notes/3.0PR16Q3.5 http://wiki.openbravo.com/wiki/Retail:Release_Notes/3.0RR16Q3.5
- maintain the receivers of the mail
- if gmail create links in the copy paste, put the mouse in the link and remove it
- read the entire mail, to try to avoid copy paste errors
Email backports are available (normal and emergencies)
Email template
To: team.platform <team.platform@openbravo.com>, Team ERP Engineering <team.erpengineering.development@openbravo.com>, Team Retail <team.retail@openbravo.com>, Team StoreServer <team.storeserver@openbravo.com>, Staff.RM OB <staff.rm@openbravo.com>, Sofware Factory <staff.swf@openbravo.com> Subject: 17Q1.1 backport repositories available Hi 17Q1 is published in 'QA Approved' status and 17Q1.1 repos are available for backport. ERP https://code.openbravo.com/erp/backports/3.0PR17Q1.1 Retail https://code.openbravo.com/retail/backports/3.0RR17Q1.1 Regards
Enable int-promote (normal)
Enable job http://ci.openbravo.com/job/int-promote/
Make POST_PUBLISH_ERP=true in ci To disable the 'SystemValidatorTest and check.module.consistency' for modules and retail job.
Note: Remember to make POST_PUBLISH_ERP=false after one int-promote is completes.
Check log update (normal)
Update the "Check log" script in jenkins and replace all the 99999 with the version of the erp published.
Update demo (normal)
1. Run the demo package job
Retrieve the resulting demo ip from the https://ci.openbravo.com/job/int-demo-prepare-clone/ job. Note that both ERP and Retail execute this job
2. Check version in clone
login http://<ip>/openbravo Help -> about (check for version)
Send an email to team.erpengineering.development@openbravo.com, (get the ip from the mail sent by demo-pacakge job)
e.g.: Demo machine for erp version yyQn.m is ready for testing backend : http://<ip>/openbravo
3. If ok, run the demo publish job.
build_with_parametes CONTINUE="yes" SERVERS demo-eu1 demo-us1 DEMO = erp
NOTE: This job does a system update + reboot of the demo machines (update server_update_sheet for date of demo system update)
4. check if the login works correctly in demo-eu1 and demo-us1
Confirmed Stable promotion (normal and emergencies)
When the MP was for 40 days in QA Approved it's needed to promote it to Confirmed Stable, execute in releaser machine:
Note: erp can only be promoted to Cs only after retail is published in QAA
example : if 16Q3.3 to CS make sure 16Q3.3 is already in QAA and ready for CS promotion
cd ~/releaser ./promote_cs 3.0PRyyQn 2>&1 | tee ../logs/promote_cs-3.0PRyyQn.log INFO: Folders for 3.0PRyyQn in ~/repos, ~/output can be removed after promote to Confirmed stable maturity.
Update the status in the Release Notes page from QAA to CS
Update the Release Notes of the particular release, e.g: 3.0PR16Q4.3
Check update to QAA works
- Start an ec2 instance with old release version
- Change the maturity to QAA
- Scan for updates and check it updates all the modules to version released
Modules reminder (emergency)
Note: Should be done only once when the version is first promoted to CS maturity.
cd ~/releaser # Do a first check without send mails , check that no appear personal emails, only should appear the usual teams ones (erp engineering support one should not appear also) ./modules-reminders.sh cs 3.0PR17Q1.1 emails-no 2>&1 | tee ../logs/modules-reminders-cs-3.0PR17Q1.1.log # real run sending the mails to teams ./modules-reminders.sh cs 3.0PR17Q1.1 emails-yes 2>&1 | tee ../logs/modules-reminders-cs-3.0PR17Q1.1.log
After the first notification remind weekly to the teams till done.
Release Notes (normal and emergencies)
- Set Maturity level column to CS in the the general Release Notes page.
- Set Maturity Status to C in the release note of previous release.
- Update the Release Notes page of that release and remove the message about the module being available in QA Approved.
Announcement (normal and emergencies)
For announce this change search for previous release mail, and edit it accordingly:
- maintain the receivers of the mail
- if gmail create links in the copy paste, put the mouse in the link and remove it
- read the entire mail, to try to avoid copy paste errors
Cancel the pending previous confirm stable emergencies (normal and emergencies)
For example if q3.x is promoted to cs, all the q2.x not published needs to be canceled.
In issues:
- retire the backports repositories
- reject all the backports with that target version:
- Resolution status: suspended
- Message: "The newer version 15Q3.x has been promoted to Confirmed Stable (CS). So no new 15Q2.x releases will be published."
- If the issue is already closed, add the previous note to it
- make this target version unavailable for new backports, mark it as obsolete
Example email for send to release management:
Subject: Canceled erp >= 15q2.7 and retail >= 15q2.6 releases Hi all, ERP and retail 15q3.x have been promoted to confirmed stable (CS), so all the pending (not yet published) 15q2.x releases of erp and retail are canceled. As a consequence we have: - retired the backports repositories - rejected all the backports with that target version - made this target version unavailable for new backports Regards, --