General Validation Approach
First we develop a complete map of the entire modeling process and client operations environment.
Below is a partial example for a flow chart we prepared for a utility company.
Next, our validation approach focuses on data inputs, then key assumptions, and sound theoretical formulation and documentation.
The most critical factor for an effective model is the validity of its input data. If this data is not correct, the output of the model could be severely flawed. Thus, our model validation process calls for a comprehensive audit of all data inputs to the model. Since models are based on contextual assumptions, these assumptions must also be evaluated and adjusted for changing environments and market conditions. An inadequate review of contextual assumptions is the most common shortcoming in model development.
Complex theoretical model development is more of an art than a science. It often depends on the model developers’ judgment and proficiency in understanding the context for which the model is designed. A primary source of model error is poor understanding or inappropriate usage of the theoretical tools used to develop the model within a business context. Prescio can identify such errors if they exist.
To ensure that the business context and model design are compatible, Prescio stresses proper documentation of all theoretical considerations that go into developing the models. This includes an evaluation of your model techniques and the strength and reliance of the model’s predictive power within clearly defined risk tolerance levels. These standards apply to both internally developed and vendor-supplied models.
Prescio model validation often includes comparison of outputs with similar in-house models. Further emphasis is placed on the following validation tests to identify issues in models:
- Back Testing
- Scenario / Stress Testing
Our team has extensive experience working with complex quantitative and statistical models at financial institutions and utility companies and they understand the business context where the models are applied. Our depth of experience and understanding allow us to make model improvements that allow our clients to reduce risk, minimize costs, and generate more revenue.