Naylor and Finger validation approach

Naylor and Finger (1967) proposed a three-step approach to validate a simulation model. It combines rationalism (the model must make sense), empiricism (the assumptions must agree with data) and positive economics (the model must predict the real system's output well). Banks, Carson, Nelson & Nicol use it as the standard framework.

Step 1: Build a model that has high face validity

Face validity means the model looks reasonable to people who know the real system. To achieve it:

• Discuss the model with managers, operators and users of the real system.
• Use all known facts: observations of the system, past studies, relevant theory, and the experience of experts.
• Show the model's structure, inputs and results to users and ask whether they are believable.
• Check that results change in the expected direction when inputs change (more arrivals should give longer queues).

Step 2: Validate the model assumptions

Assumptions are of two kinds.

• Structural assumptions: how the system works and is abstracted, such as queue discipline, number of servers, balking or breakdowns. These are checked by observing the real system and discussing with staff.
• Data assumptions: the input data and the distributions fitted to it. Check the data collection method, test independence and stationarity, and use goodness-of-fit tests (chi-square, K-S) on the fitted distributions.

Step 3: Compare input-output transformations

The model is treated as a black box that turns input data into output measures. Run the model with the same input conditions as the real system and compare the outputs.

• Use historical input data and compare model output with the real output.
• Use statistical tests, such as a t-test on the difference of means, or a confidence interval for the difference.
• If the real system does not exist, compare against another validated model or expert judgement.
• A "Turing test" can also be used: experts try to tell real reports from model reports.

Example

For a bank teller model, step 1 is confirming with the manager that tellers, one queue and lunch breaks are modelled correctly. Step 2 checks that the inter-arrival times really fit an exponential distribution. Step 3 runs the model with last week's arrivals and compares the average customer delay with the measured delay.

Real system --> observe --> Step 1: face validity
Data        --> test    --> Step 2: assumptions
Inputs --> [Model] --> Outputs  vs  real outputs (Step 3)