Hannah Hoyt and Dr. Bret Lyman, Nursing Department

Introduction

Approximately 400,000 deaths occur each year due to preventable errors in hospitals.¹ Organizational learning² is the process through which organizations progress toward achieving their desired outcomes. Hospital units can use principles of organizational learning to reduce errors and achieve excellent performance outcomes.³ Unit managers have limited evidence-based resources to guide their organization’s development toward achieving desired outcomes. The purpose of this project was to develop two reliable survey instruments to measure the presence of certain contextual factors (environment) and mechanisms (actions) that are conducive to organizational learning on hospital units. These contextual factors and mechanisms emerged from a realist review of the literature.⁴ The third instrument will measure the developmental stage of the unit, based on a progression of developmental characteristics.⁵ To develop a reliable instrument, it is recommended to follow a rigorous process including cognitive interviews and expert reviews of each item to determine its content validity, as well as the content validity of the whole survey.⁶

Methodology

This project evolved from a pilot study to identify the presence of contextual factors on one hospital unit, to the preparation of three instruments for reliability testing and factor analysis. Preparation included reviewing literature on proper instrument development, gaining IRB approval, developing survey questions to align with the definition of each survey construct (sub-category), conducting 10 cognitive interviews with registered nurses for each survey, refining survey questions according to feedback received, and having 8 experts review each question to rate its relevance to the construct definition. Experts were selected because of their experience in the fields of organizational learning research, instrument development, and/or nurse management, and they rated the relevance of each question from 1 to 4. Scores of 3 and 4 were considered relevant, and the eight expert responses for each item were averaged, to yield the item content validity index (I-CVI) score. Next, these scores were averaged to yield the Survey-CVI (S-CVI) score for each of the 20 constructs. Afterreviewing feedback from expert reviewers, additional changes were made to the survey items in preparation for an additional round of cognitive interviews and expert reviews.

Results

From the cognitive interview process with registered nurses, we found that some words caused confusion. For example, reviewers suggested we change “collegial relationships” to “respectful peer relationships” and “team members have a space to discuss issues” to “team members have a place to discuss issues.” Adjustments were made accordingly. Two new questions were generated in this stage.

Eight expert reviewers rated each item on surveys 1 and 2 for content validity (scale: 1- 4), and seven reviewed survey 3. These scores had to exceed 0.78 to be considered relevant.6 Only 4 items (4.1% of total survey questions) received I-CVI scores of 0.75. Adjustments were made to these items, which will likely lead to I-CVI score improvement from the second round of expert reviews. The remainder of the items (95.9%) were within acceptable range for relevancy, receiving I-CVI scores from 0.86 to 1.00. We made changes, focusing on items whose scores were 0.86 to 0.88, and according to feedback on specific items and constructs. 16 of the 20 constructs received S-CVI scores >0.90, which is considered standard relevancy level. Constructs that received S-CVI scores <0.90 included: collaborative relationships (0.878), skills (0.896), identity (0.895), and ownership (0.86). Adjustments were made to these constructs, which will likely lead to improved S-CVI scores. Survey 1, on contextual factors, averaged 0.927 for all I-CVI scores. Survey 2, on mechanisms, averaged 0.9775. Survey 3, on developmental characteristics, averaged 0.9482. All three surveys combined averaged 0.951. Overall, the 98 survey items proved highly relevant. One item related to written shift-change reports was removed due to irrelevancy. Eight items were added, to clarify definitions of identity, outside peer groups, and leadership skills.

Discussion

Limitations for the cognitive interview process include that nurses were primarily alumni of the University’s Baccalaureate program. Many registered nurses nationwide attended Associate’s level programs, so our sample pool may have been slightly unrepresentative. The nurses interviewed have been working for two to three years, which could be viewed as insufficient time to know about some managerial aspects. Limitations for the content validity process include that only seven experts responded for survey instrument 3 (Developmental). Additionally, several organizational learning experts were unable to respond to any of the surveys, so feedback was primarily from nurse managers (6 of 8 respondents were managers).

The next step after the validation process is to send the surveys to thousands of nurses, to obtain 1500 responses. These responses will allow for an extensive factor analysis to distinguish which questions produced significant data. Non-distinguishable questions will be removed before final instrument publication and use.

Conclusion

Reliable instrument development requires a thorough process. This project included only the first few steps in the process. However, this careful process should produce reliable instruments that can guide hospital unit managers as they strive to become learning organizations. Dr. Lyman hopes to eventually develop a tool to guide nurse managers as they endeavor to improve outcomes on their units.

Works Cited

1. James, J. T. (2013). A new, evidence-based estimate of patient harms associated with hospital care. Journal of Patient Safety, 9, 122-128.
2. Senge, 1990.
3. Nembhard, I. M., Alexander, J. A., Hoff, T. J. & Ramanujam, R. (2009). Why does the quality of health care continue to lag? Insights from management research. Academy of Management Perspectives, 23(1), 24-42.
4. Lyman, B., & Rencher, B. (in preparation). Organizational learning in inpatient hospitals: A realist review of the literature.
5. Lyman, B., Shaw, L., & King, C. (under review). Organizational learning in an orthopedic unit: A learning history.
6. Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459-467.