Several characteristics of this training domain lead us to structure assessment around a situation space. Due to the dynamic, unpredictable nature of the domain, there are no set plans for students to learn which could form the basis for student evaluation. Nevertheless, some structuring is necessary to ascertain, for instance, whether students are getting closer to their goals. Further, it was necessary to deal with the volume of information and to account for missing information.
By structuring the state space into a situation space, monitoring is organized top-down around the situation, which in turn usefully constrains interpretation and assessment. Behavioral trends can be monitored and assessed according to their appropriateness within a situation. Changes in behavioral trends within a situation can be used to infer missing information, such as whether the platoon has spotted the enemy and is going into action on contact. Unnecessary details about the state of the world are not monitored. And the transitions between situations provide a principled basis of coupling analysis of planned and reactive behaviors.
A key feature of our approach is that the high level analysis appropriate to the situation is based mainly on partial state descriptions and trends in those descriptions. This is quite different from work on student modeling and plan recognition which use a complete action model as the basis of checking/inferring a plan with an agent's actions. These action-based approaches are not as tolerant of incomplete knowledge nor can they easily infer from low level actions the kinds of higher level analysis that are important for the instructors here.
For instance, recognizing that a platoon is trying to travel in a wedge formation would be at best difficult if the recognition was based on the low level actions that the 16 crew members in the 4 tanks were executing. And evaluating this team behavior is best done at the level of the abstract, partial state descriptions, especially trends in those descriptions, and not at the level of individual discrete actions these various team members are performing. The relation between the levels is not strongly fixed. Moreover, recognition and evaluation is best done at the level at which this joint behavior has consequences in this dynamic environment. Still, causal analysis at the level of individual actions would be of diagnostic utility but only in the context of the higher level analysis and not as a way of deriving the higher level analysis. Some form of constrained plan recognition, for example, may be useful for determining why a wedge formation is falling apart.