How to Develop a Questionnaire
A good and relatively concise guide to developing questionnaires for social scientists. Do give it a read
My comments:
1. I would add Pre-testing before Recruiting the Participants. Pre-testing has a different aim than a Pilot study. As described in the article, a pilot study is focused on the feasibility, practical, and technical issues related to the administration of the new instrument.
In contrast, pre-testing is for process response validation. It involves asking a group (or two) of target respondents to answer the questionnaire. Then, cognitive debriefing interviews are conducted to check on the respondents' understanding (cognitive processes - hence the name process response validation) of the items and instructions in the instrument.
2. The MEASURE approach is limited to the application of Classical Test Theory. EFA and CFA are stated for structural validation. An instrument may be developed using Item Response Theory and similar steps; however there are different details in the processes.
For example
a. In the A stage, a construct map should be presented by describing the expected responses of individuals with different levels of ability.
b. Sample size calculation may be determined by identifying the desired level of logit's precision.
c. In the second E stage, the suitability of the rating scale (5-point vs 4-point, for example) can be tested through rating scale analysis.
3. When I deliver a workshop on Questionnaire Development, invariably I would get questions about the process for adapting and translating questionnaires. These topics are not covered in this article. There are no suggestions or comments on which of the stages would be suggested for translating questionnaires. I guess, the title should be taken as "Development AND validation" but not "Development OR validation".
4. Something that is also missing from the article is suggestions on the procedure or sequence of validation. Which validity should be tested first? Criterion-related or Convergent validity? Over the lifetime of an instrument, many of the suggested validation should be carried out. But what about a single project? Which of the validities should be included?
5. Related to point 4 above, the article discussed steps that seems to range from "must-have" to "good to have". However, there are no clear explicit recommendations for the degree or level of neediness of those steps. Here, I would like to draw upon the Islamic jurisprudence concepts related to human needs. In this case, it would be modified to be psychometric needs.
The fuqaha distinguish between daruriyat (must have; will cause chaos if absent or not fulfilled), hajiyyat (good to have; desirable to have, but no serious negative consequences if not fulfilled) and tahsiniyyat (it is a bonus to have; no negative consequences if not fulfilled) needs.
To use this classification of psychometric needs, it would be beneficial to state the context or specific purpose. For example, we may talk about the psychometric needs for a PhD project. So, we may identify the daruriyat, hajiyyat and tahsiniyyat to be included in the PhD thesis. As to what psychometric characteristics belong to which category, we need to psychometric fuqaha to state their ijmak (consensus).
Comments