Retail:MasterdataLoading
Contents |
Introduction
In order for the WebPOS to be able to keep working when the connection is unstable or it is not possible to reach the server, it is necessary to load a group of key data information. There are two main types of information:
- Terminal configuration related data: This information is retrieved using a single request to the backend (org.openbravo.retail.posterminal.term.Terminal request), and it is always retrieved in full every login or page refresh.
- Masterdata models: products, business partners, discounts, taxes, ...
This document will explain the second type, as it is a bit more complex, due to the fact that it is in most cases big, and cannot always be loaded in full and in one shot.
Masterdata load modes of WebPOS
There are two ways to load the masterdata in the Web POS:
- Full refresh: this mode loads every record of each model, and inserts it in the local database
- Incremental refresh: this mode loads only the records which have changed since the last refresh, and inserts (or updates) them in the database.
When first accessing the Web POS from a device, a full refresh is always triggered. This should be considered installation of the system, and it is mandatory.
After that, subsequent logins will do a full or incremental refresh depending on the system configuration. Additionally, also depending on the system configuration, an incremental refresh will automatically and periodically happen while the user is logged in the system.
Incremental refresh is in most cases much less demanding on the server, and also much quicker if the amount of changes in the data is not big.
Configuration
There are two fields on POS Terminal Type window which control this functionality:
- Time to fully refresh masterdata (in minutes): This field controls how frequently should a full refresh be done. Full refresh only happens on login; it will never happen while the user is logged in, as depending on the dataset size, it may take a significant amount of time.
- Time to incrementally refresh masterdata (in minutes): This field controls how frequently should an incremental refresh be done.
Additionally, since 17Q2.4, there is a preference called No Auto Incremental Load at Login. If this preference is set to 'Y', no refresh will happen during login unless it is time to do a full or incremental.
There is another preference which controls how masterdata is loaded: Masterdata models batch size. The masterdata is loaded using batching: data is divided in batches, to ensure that the browser is not overloaded due to having to load and insert too many records at the same time. Administrators can control the size of this batch with this preference. Bigger batches will in many cases decrease the loading time as it will reduce the number of requests to the backend server, but if the terminal devices are lower end machines, the browser may struggle and crash if the size is too big, so this preference should be configured taking this into account, and never in production before doing some performance testing with the real terminal hardware which will be used.
![]() | Note: Starting from RR18Q3, a new field called Time to Show incremental Refresh Popup (in minutes) has been created in the POS Terminal Type Window. |
The value in this new field enables an extra behavior for the Incremental Refresh: Once the timer for an incremental refresh has finished, instead of showing the standard incremental refresh popup the following warning will be shown on the top right of the Web POS: Master data refresh will happen after this ticket is closed. This message informs the user that if no ticket is finished during the time value of the new field, the standard Incremental Master Data Refresh popup will shown. However, if a ticket is closed during this new time window, the incremental will be done before starting the new ticket.
There are also other scenarios besides finishing a ticket which will trigger the incremental after the warning:
- Going back from a Cash Management to Web POS.
- Going back from a Cash Up to Web POS.
- Going back from SCO to Web POS.
In general, any action which brings the user back to Web POS will trigger the incremental Refresh.
The behavior for SCO has also been changed since the implementation of this new functionality. In this case, Incremental Refresh will only happen once a ticket is finished and no warning will be shown.
Operation of each refresh type
The type of masterdata load that will happen during login depends on how the configuration fields and preference are configured:
- If the terminal is used for the first time, or if the browser cache was deleted, a full refresh is always done on login
- If a full refresh was tried before, and failed (due to loss of connectivity, or server failure), another full refresh is always triggered in the next login (in fact, the Web POS will not complete the login in this case, and the application will show a message and force another login)
- If the incremental field is not defined and the preference is also not defined, a full refresh always happens on login, and no periodic refresh happens while the user is logged.
- If both fields are defined and the preference is not defined, then when the user logs in, if it's time to do a full refresh, a full refresh will happen. Otherwise, an incremental refresh will always be done.
- If the incremental field is not defined, and the preference is set, then during login, either a full refresh will happen (if it's time to do it), or nothing will happen.
- If both fields are defined, and the preference is set, then if it's time to do a full refresh, a full refresh will be done. Otherwise, if it's time to do an incremental refresh, an incremental refresh will be done, and otherwise, no refresh will be done.
As explained before, if a full refresh happens during login and the process fails for some reason, the Web POS cannot login, and another full refresh will automatically happen. On the other hand, if an incremental refresh is executed and the process fails for some reason, the login process will finish successfully. The only consequence here (apart from the fact that new data couldn't be loaded) is that during the next login, another incremental refresh will always be done.
Apart from this, once the user is logged in the Web POS, if the incremental refresh is configured, the Web POS will automatically launch an incremental refresh when it is time to do so. The application shows a warning before executing it, and the user can cancel this process:
![]() | Tip: You can temporarily set the incremental field to empty, to force the next refresh to be a full refresh. However, don't forget to set it again afterwards! |
Technical implementation of masterdata loading
In both modes, models are always loaded one by one, and every model starts only once the previous one has been successfully loaded.
As explained before, each model of masterdata executes a query to obtain necessary data. If the size of data is too large, the results are paginated to reduce the impact of the request and the insertion of data in the POS. Pages are also loaded one by one, and each page is only loaded after the previous one finished successfully.
As with every remote request, each request to load a masterdata page has a timeout. If this timeout is reached, the request is considered a failure, and the process finishes.
Full Masterdata Load
During full masterdata load, each local table is removed, and the complete record set is loaded and inserted again.
Once the full masterdata process has started, it must finish completely without errors. If there is an error, the Web POS will not be able to login, and another full refresh will automatically happen.
Incremental Masterdata Load
When data is loaded incrementally, the Web POS only loads the records which have been changed since the last refresh in this terminal.
This mode is designed to make successive masterdata refreshes more efficient than the first one, and it is very important to configure it to have reasonable POS login times if the masterdata size is reasonably big.
If this mode fails, the Web POS will be able to complete the login regardless. The reason for this is that unlike in the case of the full refresh, the previous data was never deleted from the tables, so it's still available, so the application can still run correctly.
The way this mode technically works is by saving inside the terminal cache the timestamp generated in the backend when the data was last loaded. This timestamp is sent as a parameter in the request, and the query retrieves all records whose "updated" column contains a value greater than this timestamp.
This mode has a specific limitation: it is not able to detect data removals, due to using the updated column to check which records changed. In general, masterdata should never be deleted, as transactions could be referencing it. Instead, records should always be disabled, using the "Active" field in the backend windows. Several masterdata models currently support the removal from the terminals of records which have been disabled using the "Active" column, during incremental refresh. These models are the following:
- Product
- Business Partner
- Discount
- Tax
This means that if any of the records of these models is disabled, and an incremental refresh happens, then this record will be removed from the terminal. This can be used to remove from the terminals products which are no longer in stock, or taxes which are no longer valid, without having to force a full refresh of the masterdata.
Monitoring masterdata refresh across terminals
The masterdata refresh leaves log client messages when it is executed. This means that it is possible to check in the log client to see if some particular terminal ran masterdata refresh, and when, and it is even possible to check which models were executed, and how many records they loaded.
Each time the masterdata refresh is executed, it generates two messages per data model:
[sdreresh-inc] The model Discount has started masterdata load [sdreresh-inc] The model Discount has finished loading with a total of 1 records. There was no pagination.
You can check the creation date of these messages to find out when the terminal refreshed its masterdata. The second message also explains how many records were retrieved.
These messages can be found in the Log Client window, in the backoffice.
Appendix: Masterdata models of Retail WebPOS
In the POS, all masterdata information are related to some models defined within the POS.
Masterdata models defined in Retail Pack for POS
- TaxRate
- TaxZone
- Product
- ProductCategory
- ProductCategoryTree
- PriceList
- ProductPrice
- OfferPriceList
- ServiceProduct
- ServiceProductCategory
- ServicePriceRule
- ServicePriceRuleRange
- ServicePriceRuleRangePrices
- ServicePriceRuleVersion
- BusinessPartner
- BPCategory
- BPLocation
- Order
- DocumentSequence
- ChangedBusinessPartners
- ChangedBPlocation
- ProductBOM
- TaxCategoryBOM
- CancelLayaway
- Discount
- DiscountFilterBusinessPartner
- DiscountFilterBusinessPartnerGroup
- DiscountFilterProduct
- DiscountFilterProductCategory
- DiscountFilterRole
- DiscountFilterCharacteristic
- CurrencyPanel
- SalesRepresentative
- Brand
- ProductCharacteristicValue
- CharacteristicValue
- Characteristic
- ReturnReason
- CashUp
- OfflinePrinter
- PaymentMethodCashUp
- TaxCashUp
- Country
- CashManagement
Masterdata models not defined in Retail Pack for POS
Some external modules (not included in Retail pack) implement new additional masterdata models to the previous list. Here are some examples
- ComplementaryProduct
- IssuingCompany
- Subtype
- OBDISX_offer_paymentmethod
- FreeProduct
- ComboFamily
- ComboProduct
- Coupon
- LevelProductPrice
- MultiUPC
- StockValuation
- TaxCategory
- UOM
- GiftCard
- OBRETSS_Sequence
- OBDSCAL_Scale
- DataCloseCashPaymentMethod
- OBPOS_CurrencyPanel