Projects:Stack Configuration for Performance

Contents 1 Introduction 2 Summary 3 Openbravo (Version) 4 Browser (Version) 5 Network 6 PostgreSQL version 7 Application Server: Tomcat 8 Database Server 8.1 PostgreSQL 8.2 Oracle 9 Example Memory Settings 10 Openbravo (Configuration/Tools) 10.1 Openbravo Core Features 10.2 Addon Modules 11 Monitoring 11.1 Application Level Monitoring 11.2 System Level Monitoring 11.3 Database Level Monitoring (PostgreSQL)

Introduction

The aim of this project is to create documentation explaining how the stack (Tomcat, DB, etc...) should be configured in order to make Openbravo performant.

It should give a guideline which should be usable as a starting point for typical installations.

Summary

The following gives an overview about the rest of this document. It can serve a a quick checklist which can be used to sign-off on when configuring or reviewing installations.

The sections below detail on why and how to implement those recommendations.

1. Use latest Confirmed Stable (CS) Release from Openbravo: 3.0PR23Q3.1
2. Use latest available Browser version of a supported browser.
3. Use HTTP compression (automatic if using new Openbravo Appliance)
4. On PostgreSQL use major version 9.1 and not 8.3 or 8.4
5. Tomcat memory configuration
6. PostgreSQL configuration
7. Useful Openbravo features / Addon modules
8. Monitoring

Openbravo (Version)

To ensure good performance it is important to run a recent version of Openbravo. Ideally the latest Confirmed Stable (CS) release.

The current Confirmed Stable release is: 3.0PR23Q3.1

In addition of receiving bug-fixes nearly every Openbravo update is improving general performance by either:

• Fixing identified performance problems
• Delivering new projects improving performance.

This blog post on improving openbravo performance gives a good overview about some bigger performance related projects which have been delivered recently.

To get an overview about the numerous smaller performance fixes lookup the list of issues tagged with Performance in the Openbravo issue tracker. To know exactly in which release a fix is included please refer to the Release Notes and the Release Changelog.

Browser (Version)

As most current web-based software with Openbravo the Browser is doing the heavy-lifting regarding the User Interface.

It is important to run a recent version of your favorite Browser as those regularly have improved performance.

Also for security reasons you should always run the latest version of your browser.

Network

To optimize network transport only one configuration change is important which is to HTTP compression.

This compresses data send from the server to the browser transparently and thus can reduce data transferred and access time significantly.

To configure this manually please refer to the Compression section of the Custom Installation manual.

PostgreSQL version

On PostgreSQL based installation we strongly recommend to run version 9.1.

If your installation is still running with 8.3 or 8.4 we strongly recommend updating it.

That applies mostly to users of one of the following deployment options:

• Old Openbravo Appliance (based on rpath Technology)
• Openbravo package for Ubuntu (for any version lower then 12.04)
• Any custom install using PostgreSQL < 9.1

The recommended deployment is to use the Openbravo Appliance. Migrating to it is explained here.

Application Server: Tomcat

For the Tomcat part only one group of parameters is interesting to reconfigure which are 2 Java memory related parameters.

1. Java Permanent Generation (-XX:MaxPermSize): You normally never need to raise this parameter above the default value of 256MB. This parameter does not depend on load or data size but solely on the number of loaded java classes so in practice it does not need to be touched unless running multiple Openbravo copies inside one tomcat instance.
2. Java Heap (-Xmx parameter): This parameter is important to scale up with the system usage. However this is mostly about avoiding very bad system performance when using nearly all of the configured memory and not so much improving performance otherwise. It can be hard to estimate a good general value valid for all instances without measuring average heap memory used in an instance. So a few basic rules to help with selection
   1. Needs to be set high enough to avoid using >95% of the configured value. As that the system will essentially come to a complete halt
   2. It should cover average normal heap memory usage
      1. Current rough heap usage can be seen by running the following command and summing up numbers of Eden Space - used & PS Old Gen - used
      2. sudo jmap -heap `jps -l | grep 'Bootstrap' | cut -d' ' -f1`
   3. Plus some buffer to allow for big transactions complete which need more memory
   4. If a system does not use all it's heap memory then increasing the maximum does not increase system performance (but can give extra buffer to absorb big transaction or load spikes)
   5. The table below gives some values to start with for various available memory settings.

Database Server

PostgreSQL

There are many many parameters which can be tuned trying to improve performance. However there are few main ones which give already most of the benefit.

This guide focuses on those very few which concrete recommendations and points to further external documentation in case more tuning is needed. In most cases it should not be needed to further fine-tune the database system in a typical installation.

The following lists those few basic configuration parameters which are all found inside the main PostgreSQL configuration file. (in Openbravo Appliance based on Ubuntu: /etc/postgresql/9.1/main/postgresql.conf)

Again the table below gives some values to start with for various available memory settings. If a value is not listed for a specific configuration that means it can be left at its default setting.

1. shared_buffers: Main parameter affecting memory usage of the database. Raising this to use more memory for caching of frequently used data. This can speed up operations significantly. Raising this also requires to raise some kernel limits which is explained in more detail here
2. work_mem: This parameter affects memory usage of internal operations in queries (sort, unique and so on). The memory set here will be used several times per running query for every connection. This parameter can be very difficult to adjust safely so we recommend to keep it at a low setting < 4MB.
3. effective_cache_size: This parameter does not specify memory usage directly but rather how much memory the operating system is expected to have available for disk caching. On a linux based system this corresponds to the average number of 'cached Mem' on a loaded system.
4. checkpoint_segments: This value affects write performance of the database system. We recommend to raise to a value of 16 to get better performance.

Much more information about those parameters and more can be found in the Tuning your PostgreSQL Server and Resource Consumption documentation of the main PostgreSQL website.

Oracle

For Oracle we recommend to configure it with a memory target which is the amount of total system memory it should use.

That way Oracle will automatically adjust other parameters as necessary.

Example Memory Settings

This table gives example settings for both Tomcat and PostgreSQL based on available system memory for one server running both Tomcat & PostgreSQL.

The column "Unallocated" is the amount of not directly allocated system memory.

Leaving this memory not allocated is intentional as it will be used by the Operating System for caching.

This is important as it will directly benefit database performance as it also advised to PostgreSQL by setting the effective_cache_size parameter.

Amazon instance type	Available RAM	Tomcat PermGen	Tomcat Max Heap	PostgreSQL shared_buffers	Unallocated	Extra config
m1.small or c1.medium	1.7G	256M	1G	128M	332M
m3.medium or c3.large	3.75G	256M	1.5G	0.5G	1.5G	work_mem=4M effective_cache_size=1G
m3.large or c3.xlarge	7.5G	256M	2.5G	2.5G	2.25G	work_mem=4M effective_cache_size=1.5G
m3.xlarge or c3.2xlarge	15G	256M	5G	5G	4.75G	work_mem=4M effective_cache_size=2G
m3.2xlarge or c3.4xlarge	30G	256M	8G	12G	9.75	work_mem=4M effective_cache_size=6G

Openbravo (Configuration/Tools)

This sections highlights some useful Openbravo features or Addon modules related to

• Improving Performance
• Improving System stability with very large datasets

Openbravo Core Features

• Grid Configuration for High Volume Windows: Add new grid configurations to optimize work of windows with very large data:
  • This feature is available from MP31
  • Define fields to be non sortable or filterable
  • Define which criteria (iEquals, Equals, iStartsWith, StartsWith, iContains or Contains) is applied to fields when filtering
  • Define tabs to apply all filter criteria at once after user's request
• Query Timeout: Capability to define timeouts (so far implemented only for Oracle) for different types of queries, such as grid, web services
  • This feature is available from PR14Q2
• Configurable DB Pool: Allow to define and configure DB Pool through external modules
  • This feature is available from PR14Q2

Addon Modules

• Log Clean Up Utility: Module that provides a basic infrastructure to facilitate the clean up of log tables.

Monitoring

This section points to several tools which are very useful to monitor application and system performance.

Application Level Monitoring

Monitoring on this level can give good insight into application behavior for both analyzing errors as well as detecting performance problems.

From experience we can recommend the commercial software AppDynamics as it has proven very useful already in debugging performance related problems in several Openbravo installations.

System Level Monitoring

Additionally monitoring on system level is important to ensure that the server(s) which are running the application are running fine.

Some example for metrics to monitor here are:

• CPU Usage (by processes, waiting for UI)
• Memory usage (+ how that memory is used (Application, caching)
• Disk Usage
• Database Size evolution
• Number of database connections
• Java Heap-Usage inside the application server

The recommend tools on a linux based is Munin which is easy to install and flexible enough.

Database Level Monitoring (PostgreSQL)

When running a PostgreSQL based system the following two configuration options should be used to allow to gather important performance metrics

• track_functions=all
  • This enables tracking of runtime statistics of all PL code (functions & triggers) and allows to detect slow and frequently used functions.
• log_min_duration_statement=10000 (i.e. 10seconds)
  • This enables logging of slow queries in the database logfile and is a very easy but powerful tool to detect slow queries. This can be combined with i.e. the AppDynamic software mentioned above as it allows to correlate this slow query to the application level http-request which is causing it.

Category: Performance