(in Xs) forecasting model development
This substep captures all the necessary activities to design forecasting models based on causal variables (economic drivers, input variables, exogenous variables, independent variables, Xs). One possible option is to develop the multivariate models as a time series model by using multiple regression. A limitation of this approach, however, is that the regression coefficients of the forecasting model are based on static relationships between the independent variables (Xs) and the dependent variable (Y). Another option is to use dynamic multiple regression that represents the dynamic dependencies between the independent variables (Xs) and the dependent variable (Y) with transfer functions. In both cases, the same modeling sequence, described in the previous section, is followed. However, different model structures, such as autoregressive integrated moving average with exogenous input model (ARIMAX) or unobserved components model (UCM), are selected. Note that the forecasted values for each independent variable, selected in the multivariate model, are required for calculating the dependent variable forecast. In most cases the forecasted values are delivered via univariate models for the corresponding input variables, that is, developing univariate models is a part of the multivariate forecasting model development substep.
Consensus planning
In one specific area of forecasting—demand-driven forecasting—it is of critical importance that each functional department (sales, planning, and marketing) reach consensus on the final demand forecast. In this case, consensus planning is a good practice. It takes into account the future trends, overrides, knowledge of future events, and so on that are not contained in the history.
Forecasting model development deliverables
The selected forecasting models with the best performance are the key deliverable not only of this step but of the whole project. In order to increase the performance, the final deliverable is often a combined forecast from several models, derived from different methods. In many applications the forecasting models are linked in a hierarchy, reflecting the business structure. In this case, reconciliation of the forecasts in the different hierarchical levels is recommended.
Another deliverable is the selected models performance. The document summarizing the model performance of the final models must include key statistics as well as a detailed description of the model validation and selection process. If sufficient data are available, it is recommended to test the performance robustness while changing key model process parameters, that is, test the size of in-sample and out-of sample sets.
The most important deliverable, however, is to convince the user to apply the forecasting models on a regular basis and to accomplish the business objectives. One option is to compare the model-generated and judgmental forecasts. Another option is to give the user the chance to test the model with different “What-If” scenarios. For final acceptance, however, a consistent record of forecasts within the expected performance metric for some specified time period is needed. It is also critical to prove the pre-defined business impact, that is, to demonstrate the value created by the improved forecasting.
Forecasting model deployment steps
This block of the work process includes the procedures for transferring the forecasting solution from development to production environment. The assumption is that beyond this phase the models will be put into the hands of the final users. Some users actively apply the forecasting models to accomplish the defined business objectives either in an interactive mode, by playing “What-If” scenarios, or by exploring optimal solutions. Other users are interested only in the forecasting reports delivered periodically or on demand. In both cases, a special version of the solution in a system-like production environment has to be developed and tested. The important substeps and deliverables for this block of the work process are discussed briefly below.
Production mode model deployment
It is assumed in production mode the selected forecasting models can deliver automatic forecasts from updated data when invoked by the user or by another program. In order to accomplish this, the necessary data collection scripts, data preprocessing programs, and model codes are combined in one entity. (In the SAS environment the entity is called a stored process.) In addition to the software during the model development cycle, some code for testing the data consistency in the future data collections has to be designed and integrated in the entity. Usually, the test checks for large differences between the new data sample and the current historical values in the data. By default, the new forecast is based on applying the selected models with the existing model parameters over the updated data. In most cases the user interface in production mode is a user-friendly environment.
Forecasting decision-making process definition
In the end, the results from the forecasting models are used in business decisions, which create the real value. Unfortunately, with the exception of demand-driven forecasting (see examples in Chase, 2009), this substep is usually either ignored or implemented in an ad hoc manner. It is strongly recommended to specify the decision-making process as precisely as possible. Then the quality of the decisions should be tracked in the same way as the forecasting performance. Using the method of forecast value analysis (FVA) is strongly recommended.5 Even the perfect forecast can be buried by a bad business decision.
Forecasting model deployment deliverables
The ideal deliverable from this block of the work process is a user interface designed for the final user in an environment that he likes. In most of the cases that environment is the ubiquitous Microsoft Excel. Fortunately, it is relatively easy to build such an interface with the SAS Microsoft Add-in, as shown in Section 2.3.4.
Documenting the forecasting decision-making process is a deliverable of equal importance. The purpose of such a document is to define specific business rules that determine how to use the forecasting results. Initially the rule base can be created via brainstorming sessions with the subject matter experts. Another source of business rules definition could be a well-planned set of “What-If” scenarios generated by the forecasting models and analyzed by the experts. The end result is a set of business rules that link the forecasting results with specific actions and a value metric.
Training the user is a deliverable, often forgotten by developers. The training includes demonstrating the production version of the software. It is also expected that a help menu is integrated into the software.
Forecasting model maintenance steps
The final block of the proposed work process includes the activities for model performance tracking and taking proper corrective actions if the performance deteriorates below some specified critical limit. This is one of the least developed areas in practical forecasting in terms of available tools and experience. It is strongly recommended to discuss the model support issue in advance during the model definition phase. In the best-case scenario the project sponsor signs a service contract for a specified period of time. The users must understand that due to continuous changes in the economic environment forecasting models deteriorate with time and professional service is needed to maintain high-quality forecasts. A short description of the corresponding substeps and deliverables is given below.
Statistical baseline definition
The necessary pre-condition for performance assessment is to define a statistical baseline. The accepted baseline is called the naïve forecast, which assumes that the current observation can be used as the future forecast. It is also very important to explain to the final user the meaning of a forecast since non-educated users are looking only at the predicted number at the end of the forecast horizon as the only performance metric. A forecast is defined as the combination of: (1) predictions, (2) prediction standard errors, and (3) confidence
