How Often Does Data Update?

Table of Contents 

  1. Introduction
  2. Fulfillment Processing
  3. Order Processing
    1. Real-Time Order Processing
  4. Product Processing
    1. Using the FarApp NetSuite Bundle
    2. Using the FarApp Scheduler
      1. Retrieve Full NetSuite SKU List
      2. Continuous Product Load
      3. Product-Load Priority Updates
      4. Inventory-Change Updates
      5. Map-and-Post to Storefront(s)
  5. Factors Affecting Processing Time
    1. Number of Orders
    2. Number of Open Orders
    3. Number of Posting Errors
    4. Data Quality
    5. Number of SKUs
    6. Frequency and Number of Changes
    7. Number of NetSuite Logins (for FarApp)
    8. Complexity of Mapping
    9. Number of Item Fields
    10. Storefront Flags

 

Introduction

FarApp updates three types of data: Product-, Order-, and Fulfillment-data. This document gives a general description of each process (or set of processes, in the case of product-data), and the implementation-goals and factors that affect the timing of the posting results.

The complexity of each process varies, with fulfillments being the simplest, and products being the most complex. Here, we go in that order. This discussion is specific to clients who are using NetSuite as their back-end system, though much of it applies to all systems.

We highly recommend that you install the FarApp NetSuite bundle (more info here: https://support.farapp.com/hc/en-us/articles/219375948-FarApp-NetSuite-Bundle) if you will be syncing product data with FarApp and using NetSuite as your backend system. When the bundle is installed on NetSuite, certain syncs will not rely on the scheduler processes described below and some data will be able to flow to your storefronts in real-time. Pricing of real-time price/qty data update varies by connector. Please reference your specific connectors here: https://www.farapp.com/products/.

All processes are initiated by the FarApp Scheduler. The Scheduler is an “interval scheduler,” meaning that it starts processes on a time-interval basis, not at specific times. Each time a process starts, the start-time is recorded, and the next scheduled start-time is set for a future time based on the start-time, not on the end-time of the process. This means that if a process is scheduled to run every 90 minutes, the next start-time will be 90 minutes after the process last started, regardless of how long the process ran. (If the process takes over 90 minutes to run, the next scheduled start-time will be immediately after the process finishes.)

 

Fulfillment Processing

Fulfillment processing runs as a single process. Fulfillment data is retrieved from NetSuite approximately every 90 minutes. FarApp only retrieves fulfillment data for orders it has processed. Retrieving fulfillment data from NetSuite is a fairly slow task, especially if there are many open orders, though posting fulfillments to storefronts is generally quick. (Subsequent processing by the storefronts, notably Amazon, can take a long time.) For each open order, FarApp retrieves the order, and possibly all of the fulfillment records associated with that order. If FarApp sees any unposted fulfillment activity for the order, it creates the posting record, and then posts to the storefront (immediately in most cases, but for clients doing large volumes of Amazon orders, the fulfillments will be lumped into a large posting file, and then posted at the end of the retrieval process). The goal of this process is to post fulfillments within a couple of hours of the data being available, but it’s the lowest priority of the three process types, so consideration is given to that point.

 

Order Processing

Order processing may run as either a single process, or as two separate processes, depending on the storefront and the FarApp configuration. In either case, there are two distinct parts to the process(es): the retrieval of the order data, and the order posting. The process is typically set to run every 20 minutes.

For order retrieval, FarApp connects to the storefront, and retrieves data for new, postable orders, and loads those into the FarApp database.

The order-posting stage is more complex. For each order that is still marked as unposted, FarApp first checks NetSuite to make sure the order doesn’t already exist. It then tries to match the customer to an existing NetSuite customer, updating that customer’s address book entries if necessary. If FarApp cannot find an existing customer, it adds a new customer to NetSuite. It then posts the order, and if there are no errors, marks it as posted (so that the fulfillment process can start checking it).

The implementation goal is to get orders posted within a half-hour of data availability from the storefront. If there are two processes involved, the intervals are staggered to account for the processes running independently of each other.

Real-Time Order Processing

Real-time order processing is a feature that is available for Shopify and Magento 1.0. Please follow the steps below to enable this feature.

 

Product Processing

Using the FarApp NetSuite Bundle

Again, we highly recommend that you install the FarApp NetSuite bundle (more info here: https://support.farapp.com/hc/en-us/articles/219375948-FarApp-NetSuite-Bundle) if you will be syncing product data with FarApp and using NetSuite as your backend system. When the bundle is installed on NetSuite, certain syncs will not rely on the scheduler processes described below and some data will be able to flow to your storefronts in real-time. Pricing of real-time price/qty data update varies by connector. Please reference your specific connectors here: https://www.farapp.com/products/.

Using the FarApp NetSuite bundle's product update features will enable the following:

  1. Changes made to data in any mapped product fields in NetSuite will cause the FarApp NetSuite bundle to push these changes to FarApp in real-time. This bypasses the Continuous Product Load process task described below eliminating a portion of the wait time for updating products in the storefront.
  2. Real-time price/quantity updates from NetSuite to the storefront. This bypasses all the processes described below for price and/or quantity field updates in NetSuite only. 

Using the FarApp Scheduler

The FarApp scheduler processes for product updates still run even when you are using the FarApp NetSuite bundle's real-time price/quantity sync feature. If you are using the FarApp NetSuite bundle the scheduler processes sync your non-price/quantity product data from FarApp to the storefront, and act as a backup periodic sync to your price/quantity product data.

There are several processes involved in product-data scheduler updates. Below is a list of the major process, with a description of each. Together, they work towards a goal of updating product-data to storefronts within 24 hours of changes/additions in NetSuite (often much faster, depending on several factors, which are the subject of the next section). Where needed, an inventory-only update can be run which targets a 3-hour update-window to avoid stock-outs.

Retrieve Full NetSuite SKU List

FarApp needs a complete list of all Item Internal IDs in NetSuite. The product-data is processed by other tasks, but among the reasons for needing the full list is to make sure that nothing escapes due to unexpected errors (which can be common when systems communicate over public networks – just think of how many times you’ve had a connection problem with your cell phone). This is primarily a back-up process, to make sure all bases are covered. It runs once-a-day.

Continuous Product Load

This is the process that actually loads the details product data from NetSuite to FarApp. It does a “batch” of products on each pass, running about once-an-hour. It cycles through all NetSuite items, only loading data for flagged items, but checking all items in the system to make sure that FarApp is in-sync with NetSuite. NetSuite is very slow when it comes to retrieving product data (between 3 and 30 seconds per item, when things are running smooth), so this is a lengthy, continual process. 

Product-Load Priority Updates

This process checks only flagged items, using a NetSuite search that returns items with recent changes. For any items with changes, it sets the priority in FarApp so that they will be loaded first by the Continuous Load process. This is one of the key processes in keeping data-syncs current. This is scheduled to run every 3 to 5 hours.

Inventory-Change Updates

If stock-outs are a concern, this process looks for recent inventory changes in NetSuite, and loads the inventory updates to FarApp. It then immediately posts them to the storefront (Amazon only, other storefronts post in batches as described below). If needed, this is set up to run every three hours.

Map-and-Post to Storefront(s)

For each storefront (Amazon, eBay, etc.), there is a separate process that does the data-mapping and data-posting to that storefront. When the product-load process(es) run, they mark whether an item’s data has changed or not. The Map-and-Post process(es) look for items that are either new or changed (by changed, we mean that data in ANY field is different from when last posted), and if so marked, the item is considered postable to the storefront.

The Map-and-Post processes also work in batches; i.e., they don’t attempt to post all items in one pass, but do small groups of items at one time. The processes are usually scheduled hourly, and depending on the mapping involved, typicall take between 30 minutes and an hour to run (this is the most compute-intensive process for FarApp). Depending on how many items have recent changes, all item updates may post within that hour, or only a portion.

 

Factors Affecting Processing Times

Below are some of the factors that affect processing times and performance. Many of them are obvious, in that the more work there is to do, the longer it takes. A few are more subtle or unfamiliar. In most cases, there isn’t much that can be done to change things; FarApp needs to do its job, and there’s a lot of work involved. There are a few tips, though, that can lead to better performance.

Number of Orders

Obviously, the more orders that are processed, the longer this process takes.

Number of Open Orders

Although this mainly affects the length of the fulfillment-posting process, since all of these processes must interact with NetSuite, there is an overall affect on system performance.

Number of Posting Errors

Unhandled product-posting errors can adversely affect product posting times. FarApp handles these differently for the various storefronts, but the basic problem is that FarApp will re-attempt to post unposted items, and the items with posting errors will continue to fail unless the underlying issue is fixed. Since FarApp is posting batches of items to the storefronts, the larger the number of failed postings detracts the posting of valid items and their updates, reducing the actual postings for any given batch.

Data Quality

Usually, the better maintained your product data, the more efficient it is to process. If there are frequent errors in your data, redundancies, corrections, duplicates, and issues like that, those will be reflected to some extent in processing time. This applies not only to your actual data, but to the organization of the data (for example, if you have 20 custom fields for Color, it will take 20 times longer for FarApp to map a color field than if you have a single custom field for that purpose).

Number of SKUs

The more SKUs you have, the longer it takes for FarApp to cycle through all of your items. NetSuite searches are also slower.

Frequency and Number of Changes

Like most of the above items, this is just a workload issue: the more work to do, the longer it takes. If you have workflows in NetSuite, please note that all workflows should have checks to make sure they are not already completed (ie: if a workflow flags a checkbox, it should have a check that confirms the checkbox is not already flagged). Otherwise these workflows may trigger unnecessarily frequent updates, which will consume processing power and degrade the performance of other tasks.

Number of NetSuite Logins (for FarApp)

Because there are numerous processes accessing NetSuite, they vie for resources on your NetSuite server(s). Since these processes share the same login(s), they cooperate with each other, interleaving their requests so that they can all function simultaneously, but for each concurrent process, some delay is introduced into the performance of all processes. If the conflicts and delays are too frequent, adding more NetSuite logins to your FarApp account can greatly reduce the process collisions, and improve performance considerably. We’ve found that adding one additional login has a major impact, going from 2 to 3 or 4 has a very significant impact, and 5 or more has a marginal impact, so there are diminishng returns when using this approach. For clients with over 500 orders/day, or more than 20,000 SKUs, having at least two NetSuite logins for FarApp is essential.

Complexity of Mapping

Product-mapping is the slowest task as FarApp processing is concerned. The more fields that must be mapped, and the more conditions and rules that are applied to each field, the longer it takes to map an item.

Number of Item Fields

FarApp stores virtually all of your product data in its own database. If you have a large number of fields for your items, then there is more to retrieve, store, and process.

Storefront Flags

FarApp is set to load only Flagged items. When you set storefront flags to N, FarApp will remove those items from the storefront, but continues to store, map, and process those items, which is a time-consuming process. Removing the flag values (setting the fields to blank) once they’ve been removed from the storefront(s) greatly reduces the amount of work that FarApp must do, and speeds up the remainder of your processing.

You can read more about the FarApp flag field here: https://support.farapp.com/hc/en-us/articles/217185977-Storefront-Flag-Fields

Comments