Content Validation by Subject Matter Expert

 A master student asked me to rate the relevance of items in a questionnaire that she wants to use in her research project. I’m not really sure what kind of validation that she wants. Ratings of relevance is usually done with subject matter experts to test face and content validity. However, the questionnaire in question is already published in a journal. Is it ‘cultural equivalence’ that she wants? So, I read the original journal article to understand the questionnaire.

Agudo, M., & de Dios, J. (2013). An investigation into how EFL learners emotionally respond to teachers' oral corrective feedback. Colombian Applied Linguistics Journal, 15(2), 265-278.

“A short 10-item questionnaire was specially designed for this purpose to gather information on how EFL learners emotionally respond to the oral feedback process in classroom situations.” (p 270). So, the questionnaire is meant to measure emotional responses among students after receiving oral feedback in the classroom. It would be more helpful is a proper operational definition is given in addition to the ‘construct statement’ given. For example,

• a.   whose feedback do matters? Only teachers? Why not include feedback from other students? Should we assume constructive and collaborative learning do not occur in the classroom?
• b.   What kind of emotions matter? Only negative emotions? It seems that there are a lot more negative than positive emotion items.
• c.   Does the time frame matter? Is the questionnaire meant to capture the emotions in the classroom only? Or also hours or days after the teachers gave oral feedback in the classroom?
• d.  Does it matter how the feedback was targeted? Does feedback to the whole class count?
• e.   What kind of emotional reactions are measured? Only the ‘feelings’? Or also the behavior and thoughts?

It looks like nit-picking isn’t it? Do we really need to be that detailed with the operational definition? Well, without a proper operational definition, it would be difficult to judge whether a question is relevant or not. It is like being asked whether a dish is suitable or not. If we don’t know who it is for, we might ‘approve’ a high-sugar dish for consumption by people with diabetes. That would be an erroneous approval, isn’t it?

Now let us stick to the construct statement “emotional responses among students after receiving oral feedback in the classroom.” Are the following questions (items) relevant to the construct?

1. I feel I have learnt a lot from being orally corrected.

Is ‘learning a lot’ a type of emotion? Or is it the case that, in this sentence, ‘feel’ is used to mean ‘think’?

2. I think that the oral feedback provided is necessary and helpful.

The items sounds more like a cognitive evaluation (attitude) statement rather than emotional response.

3. I resent it when I make oral mistakes.

YES! ‘Resent’ is a type of emotion. However, it is NOT an emotional reactions after being corrected by the teacher.

4. I worry about making oral mistakes in language class.

Again, while ‘worry’ could be accepted as an emotion, the feedback from teacher is absent.

5. I hate making oral mistakes because they make me doubt myself.

Where is the teacher?

6. I resent being orally corrected by the teacher in the classroom.

This ticks all the boxes: I would rate this item as highly relevant.

7. I get upset when I don´t understand what the teacher is correcting.

This item has all the important component. However, it does highlight the ‘feedback’ component in the construct. Are we concerned with feedback that are meaningful or understood by students, or any type of feedback. When a student say an incorrect sentence, and the teacher goes “Pfffftttt! You should go back to kindergarden”, does that count as an oral feedback? Or do you want to restrict it ‘understandable constructive or corrective oral feedback’

8. I am afraid that my language teacher is ready to orally correct every mistake that I make in class

Where is the feedback? Still has not happened yet, right?

9. How do you feel when the teacher immediately corrects your mistakes?

a. I feel angry

b. I feel embarrassed

c. I feel sorry

d. I feel happy

e. I feel satisfied

f. I feel bothered

g. I feel indifference

 h. I feel nervous

i. I feel overwhelmed

Yeay for the very direct questions. Shall we keep this item only? I’d add the ‘oral’ part though.

 

10. What do you think and what do you do after the teacher´s immediate correction?

a. I believe that `I wish I had not spoken anything´.

 b. I just listen, not speak anymore!

 c. I think the reasons why I make mistakes.

d. I think the teacher is not patient enough to wait for the end of my sentences.

 

This is quite a strange item.

• Not all options are about emotional responses.
• The questionnaire description says that ALL items are to be answered with a 5-point rating scale. However, the analysis in the article seem to suggest that students were asked to choose only 1 option from among the 4.
• The list of options is quite a narrow range of possible response. It sends a sense of arbitrariness. Are we assuming students don’t respond positively to the correction?

If I were the subject matter expert doing content validation of the questionnaire, only 3 items (6,7, and 9) would be recommended as relevant. This experience strengthens my conviction that researchers need to be careful in adopting and adapting questionnaires. Just because a questionnaire appears in a journal, it does not mean the questionnaire is fit for OUR purpose. So, if you want to validate the content of the questionnaire, the operational definition should be clear and relevant to your research objectives.