Error message

Notice: Undefined index: und in citation_doi_content() (line 88 of /home/fulltxt/public_html/sites/all/modules/citation/citation.module).

Objective Assessment of Orthopaedic Skills

Singh, Kaur, and Singh: Objective Assessment of Orthopaedic Skills

Authors

INTRODUCTION

Medical education is moving away from facilitator driven to learner-centered, patient focused education.1 This has changed the emphasis to gaining practical skills as opposed to theoretical knowledge. Surgical disciplines call for psychomotor skills that are traditionally learned by practice on live patients. Simulation provides a good alternative without inherent risk to the patients.2 For example, task trainers have been used to teach/learn basic skills such as cannulation and venipuncture.3

Orthopedic office procedures are commonly used by many general practitioners and internists as part of their daily practice4,5 and are useful for medical students to learn. However it has been noted that these are not adequately taught during medical training,6 and that there is no organized training of such skills in most medical curricula. Realizing the need to teach these skills in an organized manner Vogelgesang et al. developed an instructional program to teach aspiration and injection techniques of the knee and shoulder to medical students and residents. They used didactic lecture and hands-on workshop using anatomical models made of synthetic materials. They found that didactic and workshop trained students outperformed the traditional group and were more confident in doing procedures.7

We have been using task trainers for teaching various techniques such as injections for carpal tunnel, trigger finger with tenosynovitis and knee joint lavage. At present these skills are being taught in a non-systematic manner and we do not have an objective method of evaluating them. We decided to systematize the delivery and evaluation of these skills using our existing task trainers in order to deliver standardized, uniform and comprehensive training in these skills for students posted in orthopaedics.

MATERIALS AND METHODS

The study was conducted during period of March-2016 to July-2016. Ethical clearance was obtained from the college ethics and research committee. A total of 80 MBBS students were recruited in the study on voluntary basis.

The students were divided into three groups for instruction - i.e. didactic (lecture), video and hands on training. The students were not randomized using any specific technique but were divided into aforementioned three groups according to their roll numbers i.e. the first one in group one, 2nd student in group two and so on and so forth. All the participants were lectured on the indications for knee joint injection and aspiration, the risks of the procedure, sterile technique, anatomic landmarks, needle position, and needle insertion through the superolateral approach, informed consent, post procedure management and instructions to the patient. The video group was shown a video of the procedure, in addition to this lecture. The hands-on practical group was shown the procedure on the anatomic model and was allowed to freely practice on the model for one hour every day. The lecture and the video group students did not have access to the model until the day of injection. On the day of assessment, each group of students was given 20 min to become familiar with the model and equipment. After 20 min each student performed the task of knee joint injection through superolateral approach on an anatomical knee model. The knee models used had moderate joint effusion so as to simplify the task.

Before the instruction (didactic, video or hands on training) a MCQ test (10 questions with single best answer) was used for base line assessment of student’s knowledge of the knee anatomy. Each participant was asked to rate his or her confidence in performing knee joint injections with use of a l0-point Likert scale, anchored with “No confidence at all” (1 point) and “Extremely confident” (10 points).

During the task the competency was assessed using newly developed rating scale for knee joint injection/arthrocentesis (Apeendix1) by a single rater.

A post procedure MCQ test (10 questions with single best answer) based on knee joint anatomy was conducted. Again, each participant was asked to rate his or her confidence in performing knee joint injections with use of a l0-point Likert scale, anchored with “No skill” (1 point) and “Extremely skillful” (10 points).

Implementation Planning

  • Permission was taken from the Dean of the Institute and Head of Department of Orthopaedics to conduct this project.

  • The proposed study project was presented before the ethics and research committee of the institute: Permission to carry out the project and ethical waiver was obtained.

  • The department colleagues were informed and persuaded to help with the study.

  • Informed consent forms were prepared.

  • Adequate copies of newly developed rating scale were made.

  • Two MCQ tests based on Knee joint anatomy with each comprising of 10 single best answer questions were prepared. The MCQs given before and after were same set of questions.

  • Adequate copies of self-confidence assessment forms were prepared.

The Assessment tool

The assessment tool for knee joint injection/arthrocentesis was prepared after discussion with experts. The assessment tool was a task specific rating scale that measures the components of the given procedure that the subject completes. The rating scale had nine items for which 0 to 4 marks were allotted. These items included explains procedure/ benefits/adverse effects to patient, orally obtains informed consent, selects/arranges the adequate materials, properly positions the limb, does sterile skin preparation, identifies and marks the landmarks, uses appropriate needle positioning, does post-procedural dressing, orally provides post-procedural instructions to the patient. The tenth item successful joint entry was indicated by joint fluid entry into the syringe and did not carry any marks. Higher scores (maximum 36) indicated greater knowledge and ability to complete the necessary procedural steps of knee joint injection. Four experts (3 from the institution and one external orthopaedic surgeon) were asked to rate each item as clearly representative, somewhat representative and not representative of the knee joint injection skills assessment. Out of the initial 11 items in the rating scale 2 items were omitted. The remaining 9 items were included based on the criteria that at least 3 out of 4 experts viewed them as clearly representative. A pilot study was done on 20 subjects. After the pilot study and with inputs from three internal and one external expert, the tool was further refined and finalized for the study. The components of the rating scale included are shown in Appendix 1.

Table 1:

Mean task specific rating scale scores

Instruction methodnMean ScoreSDP- value
Lecture2616.1±3.040.0001
Video2620.5±3.47
Hands-On Training2826.3±3.46

* Independent t-Test comparing mean scores between Lecture and video, video and hands-on training, lecture and hands-on training were found to be statistically significant (p<0.001).

Discussion with faculty, students and sensitization

The first session with the faculty members of department of Orthopaedics was a sensitization session wherein the faculty members were introduced to the need of objective assessment of orthopaedic skills, and its possible utility to students and teachers. The proposed study plan was also discussed with the faculty. One of the faculty from the hospital who was not associated with the institution acted as rater and was instructed on the use of assessment tool. The students who participated in the study were informed about the study. Written informed consent was obtained. In the following sessions the faculty helped in conducting the study.

RESULTS

A total of 80 fourth year MBBS students participated in the study. The study sample consisted of thirty-two men and forty-eight women, with a mean age of twenty-five years (range from twenty-two to sixty years). On descriptive analysis, the participants scored highest in “Explains procedure/ benefits/adverse effect to patient” (mean score=3.4) and “Orally obtains informed consent” (mean score=3.3) out of maximum possible of 4. The lowest mean score was for item “Orally provides post-procedural instructions to the patient” i.e. 1.1 out of maximum possible of 4. On comparing the means by one-way Anova, the mean score was highest for the hands on training group (26.3) followed by video group (20.5) and lecture group (16.2) respectively (Table 1). The difference in the scores between three groups was statistically significant (p<0.0001). The average score of 1st MCQ test was 3.2±1.50 out of maximum possible of 10. The average score of post procedural MCQ test was 6.6±0.78. The self-confidence in performing knee joint injections increased from 2.3±0.74 to 6.6±1.14. Both the increase in MCQ score and confidence level were statistically significant.

The overall internal consistency (cronbach’s alpha) of the assessment tool developed was found to be 0.63. The Cronbach’s alpha seems to be less which shows less internal consistency. However high convergent validity (inter-item correlation) was found for 8 out of 9 items in the assessment tool (Table 2).

DISCUSSION

Traditionally, senior surgeons do subjective assessment of surgical competency during intraoperative observation. The chief drawback of this type of assessment is the lack of consistency. As aptly put by Moorthy “Without objective, valid, and reliable assessment training programmes cannot ensure the learning of skill, tackle deficiencies in training, and implement remedial measures”.8 It has been noted that joint injection skills training is not a part of most medical curricula and the need for a systematic training and evaluation of these skills has been realized.6 The observer based assessment tools include task specific checklists/rating scales and global rating scales. These tools can allow for continuous assessment during the training period. It has been suggested that global rating scale is a better assessment tool than task specific checklist, either alone or in combination with a global rating scale. However, Insel et al.9 pointed out that task specific checklists are important since they “captures the ability to complete all of the necessary steps of the procedures, reflecting basic knowledge and technical ability.”

Table 2:

Convergent Validity (Item-total Correlation): Correlation of individual items of the new tool with its total score

ItemsSpearman’s Correlation Coefficient (r)P-value
Explains procedure/ benefits/adverse effects to patient+0.0360.754
Orally obtains informed consent+0.5560.0001*
Selects/arranges the adequate materials+0.7080.0001*
Properly positions the limb+0.5960.0001*
Does sterile skin preparation+0.5640.0001*
Identifies and marks the landmarks+0.7750.0001*
Uses appropriate needle positioning+0.4620.0001*
Does post-procedural dressing+0.2210.049*
Orally provides post-procedural instructions to the patient+0.5470.0001*

* Here, p-value <0.05 was considered as significant

The advantages stated for these tools are that they objectively assess the operative skills and are free from recall bias associated with direct observation by preceptors. Another advantage is that the skills can be tested both in a laboratory setting and during real time procedure. The disadvantage is the need for trained observers or experts for assessment.

Validity and Reliability

The tools used for objective assessment should be valid and reliable. Validity means the extent to which a test/measure/concept is well founded and correlates with the real world. It includes construct, content, predictive, face and concurrent validity.

Construct validity is the ability of a test to actually measure what it is supposed to measure. Convergent validity is considered as subcategory or subtype of construct validity. To establish convergent validity, we need to show that measures that should be related are in reality related. It was evident from the fact that inter-correlations for all items with the total score were very high. The item-total correlation in Table 2 shows that the convergent validity eight out of the nine items of the assessment tool was high.

Content validity means the extent to which the test measures the domain, which is being measured. During the development of the tool, four orthopaedic experts graded the various items of the rating scale according to their relevance. The final tool contained all the essential items as agreed upon after discussion with the experts. That proves the content validity of the tool, however content validity ratio (CVR) was not calculated.10

Face validity means the extent to which a test resembles a real life situation. It is tested by ‘users’ opinion about the functionality and realism of a test”.11 In this study the face validity of the assessment tool was assessed from the overall impressions of the orthopaedic experts. Concurrent validity of a test means the extent of correlation with a previously validated tool or a gold standard test for the same trait. Concurrent validity applies to validation studies in which the two measures are administered at approximately the same time. However, since no other previously validated tool was used for comparison in this study, the concurrent validity of this newly developed tool could not be established. Other variants of validity like Predictive, discriminant, criterion validity etc. were not assessed.

Reliability means the extent to which the test to gives consistent results. Inter-rater reliability means the extent to which test results are consistent between different observers whereas the test-retest reliability measures the extent of consistency in results when undertaken over a period of time. Since only single rater was used and the skills were not assessed before the educational intervention, both inter-rater reliability and test-retest reliability of this tool are unknown at present. Internal consistency refers to the correlation between different items in the same test and is measured by Cronbach’s alpha. The overall internal consistency (Cronbach’s alpha) of the assessment tool developed was 0.63. Besides having validity and reliability, the tool/test should be convenient to apply (i.e. feasible) and cost-effective. The limiting factor of use of this tool is that it requires an expert assessor.

The hands on training group scored highest in the knee injection task as assessed by the new tool. This further proves that the psychomotor skills are best learned through hands on training. Change in confidence is a crude marker of competence. Nevertheless, all the instruction groups showed increase in confidence and knowledge of joint anatomy. However, it was not the primary aim of the study.

Outcomes: What this study adds

In this study we attempted to devise an objective tool for assessment of joint injection skills. After a review of the literature, no validated evaluation method of joint injection skills was found. The other positive results that emerged from the study were increase in student’s knowledge of anatomy and confidence level as assessed by MCQ test and the confidence questionnaires.

Limitations

  • Lack of randomization makes this study lower on hierarchy of evidence.12

  • In this study the skills were not tested on live patients. How well the skills performed on model transfers to practice is not known. Also, we can’t predict that for how long the skill will be retained without further practice.

  • A single rater assessed all the subjects, hence inter-rater reliability was not assessed.

  • No test-retest reliability was tested.

  • Another interpretation of construct validity is the ability to discriminate between experts and novice. If this tool succeeds at differentiating between experts and novice, it will further confirm its construct validity, that however remains to be seen.

CONCLUSION

Availability of an objective assessment tool for orthopaedic skills will systemize the delivery and evaluation of these skills in our setting. Further the experience gained through this study will be used to develop objective assessment tools to evaluate the skills in other medical disciplines.

In conclusion, this newly developed tool is a valid and practical tool to evaluate knee joint injection technique. As medical students/ professionals gain proficiency in joint injection technique by practice, this tool will capture the gain in skill.

ACKNOWLEDGEMENT

We would like to thank Prof Dr Dinker Pai, Head of Department, Surgery for his guidance and support.

Notes

[3] Conflicts of interest CONFLICT OF INTEREST

The authors declare no conflict of interest.

ABBREVIATION USED

MCQ:

Multiple Choice Question

CVR:

content validity ratio

MBBS:

Bachelor of Medicine, Bachelor of Surgery.

REFERENCES

1. 

Wang AT, Cook DA , authors. Patient outcomes in simulationbased medical education: A systematic review. J Gen Intern Med. 2013;28(8):107889.

2. 

Ziv A, Small SD, Wolpe P , authors. Patient safety and simulationbased medical education. Med Teach. 2000;22(5):48996.

3. 

Maran NJ, Glavin RJ , authors. Low-to high-fidelity simulation- a continuum of medical education? Med Educ. 2003;37(s1):228

4. 

Wigton RS, Niolas JA, Blank LL , authors. Procedural skills of the general internist: A survey of 2500 physicians. Ann Intern Med. 1989;111(12):1023–34

5. 

Norris TE, Felmar E, Tolleson G , authors. Which procedures should be currently taught in family practice residency programs. Fam Med. 1997;29(2):99–104

6. 

Elnicki DM, Shumway JM, Halbritter KA, Morris DK , authors. Interpretive and procedural skills of the internal medicine clerkship: performance and supervision. South Med J. 1996;89(6):603–8

7. 

Vogelgesang SA, Karplus TM, Kreiter CD , authors. An instructional program to facilitate teaching joint/soft-tissue injection and aspiration. J Gen Intern Med. 2002;17(6):441–5

8. 

Insel A, Carofino B, Leger R, Arciero R, Mazzocca AD , authors. The development of an objective model to assess arthroscopic performance. J Bone Joint Surg Am. 2009;91(9):2287–95

9. 

Moorthy K, Munz Y, Sarker SK, Darzi A , authors. Objective assessment of technical skills in surgery. BMJ. 2003;327(7422):1032–7

10. 

Lawshe CH , author. A quantitative approach to content validity. Personnel Psychology. 1975;28(4):563–75

11. 

Van Hove PD, Tuijthof GJM, Verdaasdonk EGG, Stassen LPS, Dankelman J , authors. Objective assessment of technical surgical skills. British J Surg. 2010;97(7):972–87

12. 

Cebm.net.[http://www.cebm.net/]. Oxford, Centre for Evidence-Based Medicine. [updated 2016 June 16; cited 2016 Nov 9] Available from: http://www.cebm.net/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/.

Appendix I:

Knee Joint Injection rating scale

Roll number:

Gender:

Age:

Prior training in injection techniques: Yes/No

Handedness (right or left):

Table 2

Steps

Explains procedure/benefits/adverse effects to patient01234
Orally obtains informed consent
Selects/arranges the adequate materials
Properly positions the limb
Does sterile skin preparation
Identifies and marks the landmarks
Uses appropriate needle positioning
Does post-procedural dressing
Orally provides post-procedural instructions to the patient
Successful joint entryNOYES