To be most effective, computational models of cognitive phenomena should meet the following three criteria. First, there should be a number of measures applied to both the model and the real-world phenomenon being modelled. Second, multiple models of the phenomenon must be evaluated. Third, the complete information about the model and the measurements must be communicated to others.
Developing a measure for comparison also involves precisely specifying the environment around the model. This is actually a computational model of the experimental situation being modelled, and should be completely independent of the cognitive model (i.e., the model of the participant in the experiment). Defining these measures is done by creating environment models (including exact details of the inputs and outputs between the environment and the participant) and specifying the measures taken (time between stimulus and response, number of correct answers, memory accuracy, etc.). For quantitative measures, model and reality can be compared by determining the maximum difference between their respective confidence intervals. This determines how much the model could be in error for each measure. Ideally, a wide variety of measures and different environmental situations should be used. Combining these measures should be by maximum error, not average error (which allows extreme variations to be hidden).
To simplify the creation of these measures, it is useful to create a highly simplistic naïve cognitive model (for example, a participant who presses buttons randomly). This also provides a baseline for comparison for more complex models. Model development should start with these simple models and become more complex only as needed.
Given any set of measures, it is always possible to create a model which exactly matches on those measures. A valid cognitive model, on the other hand, should be able to produce closely matching results without extensive customization to a particular situation. To demonstrate this, first determine what range of parameter values for a given model produce results of a desired level of accuracy. This range can be compared to the ranges for that model in other situations. A similar process can then be carried out for qualitatively different models (i.e., models using alternate cognitive architectures).
The most desirable result is finding a range of parameters for a model that are effective across many situations. If a model is found to require different parameters for different situations, then there should be further work in determining how that parameter variation can be predicted. In particular, we can identify collections of environmental situations which work for particular parameter ranges.
Given the complexity of models (and the environment created for the measures taken), it is infeasible to give a complete description within a publication. It is not even possible to give a complete description of the results of data analysis of the measures; researchers must decide on what they believe succinctly and accurately describes the results. Fortunately, since the computational model exists as computer code, it can be made to be freely available on-line. This availability must extend not only to the model, but the environment it is put within, and the measures taken. By using a simulation software package such as CCMSuite, the same software used for running the simulations also makes it available to others, along with the raw data and analysis tools. Furthermore, it facilitates others making modifications to the models, performing different analyses, exploring different parameter settings, or making different measures in different situations.
| Attachment | Size |
|---|---|
| One_Page_Modelling.pdf | 56.42 KB |