The usability of IT security management (ITSM) tools is hard to evaluate by regular methods, making heuristic evaluation attractive. However, standard usability heuristics (e.g., Nielsen’s) are hard to apply, as IT security management occurs within a complex and collaborative context that involves diverse stakeholders. In a joint project with CA Technologies, my Ph.D. student Pooya Jaferian has proposed a set of ITSM usability heuristics that are based on activity theory, are supported by prior research, and consider the complex and cooperative nature of security management. The paper reporting the evaluation of the heuristics received Best Paper Award at SOUPS ’11.
In a between-subjects study, we compared the employment of the ITSM and Nielsen’s heuristics for evaluation of a commercial identity management system. Participants who used the ITSM set found more problems categorized as severe than those who used Nielsen’s. As evaluators identified different types of problems with the two sets of heuristics, we recommend employing both the ITSM and Nielsen’s heuristics during evaluation of ITSM tools.