Layla Zhang

As a consumer of research reports, the most important things for me in the methods and results section of a high-quality qualitative research report are correctly identifying the type of the research design and applying own critical thinking to the data collection and the themes.

Reflecting on the knowledge I have gained this week, I find the hardest part for me is to identify the qualitative research design when the study does not explicitly state the type. Occasionally, I will mix the case study research design with the other types. Although there is a great difference between qualitative research and quantitative research, making sure the research design used in the study is the primary step. Unlike quantitative research which is quite easy to address the research design, the qualitative research has a much wider design list, especially some of them are quite similar. Table 9.1 on page 289 of the textbook has provided a useful summary of each type and can be used as a standard to identify the design of a study as well. Furthermore, preparing a close understanding of the key characteristics is extremely helpful for both recognizing the type and the quality of the research design, such as how the research problem has been raised, how the data is gathered and analyzed, and how the findings are illustrated.

Since the end of last week, I have realized the significant role of own critical thinking in reviewing the study. I think the procedures for data collection and analysis of the qualitative research are easier to understand but more complicated than those of the quantitative research. One’s own critical thinking is especially crucial for understanding a qualitative research because the purpose of it is to explore a central phenomenon. In addition, it is impossible for the researchers to be completely objective during the whole process of the study. For example, different researchers will not extract the same codes based on the words of the participants. Therefore, the findings built from the codes and coded data will be distinctive. Also, the themes emerged from the analysis of the data may address different major aspects. The readers have to be aware of the accuracy of the data and whether the coding procedures and the findings are justified and unbiased all the time.

Actually, keeping critical thinking is the most interesting part for me when reading a qualitative study. It is like a dialogue between me and the researchers in my mind. I would explore the ways they are using for understanding and studying the problem initially, then compare to my own feelings and suggestions about what they have found in the procedures of the exploration. And my question is:

How to balance the judgments generated by you in the process of reading the research article with the findings presented by the researchers, especially when they are apparently different?

Reference

Plano-Clark, V., & Creswell, J. (2015). Understanding research: A consumer’s guide (2nd ed.). Boston, MA: Pearson.

Unit 6 Learning Activities

Article

The Influence of Servant Leadership on Restaurant Employee Engagement

Quality Criteria

1.The analysis process used rigorous qualitative procedures.

Rate

3=Excellent

Evidence and/or Reasoning

a) The researchers stated in the study that transcription of the responses were derived from audio recording and a professional transcriber transferred verbal data to written form.

b) Rather than using computerized software that may prohibit the detection of subtle themes, the researchers adopted the strategy of manual coding through key-words-in-text to get a deeper insight to the meaning of words and themes.

c) Given the major concern of separating out personal bias or perceived conclusions, Epoche was applied to address the concern.

d) The researchers used the modified van Kaam method to analyze the data, The seven steps were presented thoroughly and this process led to the findings and themes based on the analyzed data.

e) Although the researchers did not claim clearly, I deemed the list of servant leader quality of Table 1 was the list of the in vivo codes summarized by the researchers.

Quality Criteria

2.Strategies were used to validate the findings.

Rate

2=Good

Evidence and/or Reasoning

Only bracketing and triangulation were identified to ensure the accuracy and credibility of the findings.

Quality Criteria

3.The findings include a description of the people, places, or events in the study.

Rate

3=Excellent

Evidence and/or Reasoning

a) The description of the detailed rendering of participants and places is in the method section.

b) The six questions are asked in a broad-to-narrow way.

c) The authors focus on facts and simply report the questions and answers in the results section, which are different from the themes representing the authors’ interpretations.

d) There are participants quotes under each question that provide emphasis and realism in the account.

Quality Criteria

4.The findings include appropriate themes about the central phenomenon.

Rate

2=Good

Evidence and/or Reasoning

a) There are five themes reported.

b) The themes convey both major ideas about and the complexity of the central phenomenon. Theme 1 uses sub-themes and multiple perspectives; theme 2 uses literary devices and contrary opinion; theme 3 uses multiple perspectives; theme 4 uses sub-themes and multiple perspectives; theme 5 uses multiple perspectives.

c) There is no evidence of participant quotes appearing in the themes.

Quality Criteria

5.The findings relate multiple themes to each other.

Rate

2=Good

Evidence and/or Reasoning

The researchers analyze and report the interconnections among theme 4, theme 5 and other aspects in the discussion section. But the researchers have not conveyed the relationships among all five themes clearly in the text. However, the themes are interconnected and consistent with the overall case study research design.

Quality Criteria

6.The data analysis represents a good qualitative process.

Rate

2=Good

Evidence and/or Reasoning

a) Although the data analysis is an inductive and interpretive process, I think the process lacks of dynamic.

b) The results emerge from the data are credible and accurate.

Quality Criteria

7.The findings provide a good exploration of the central phenomenon.

Rate

3=Excellent

Evidence and/or Reasoning

The rigorous logic and detailed information of the findings make the findings perfect for answering the research questions.

Total Score = 17 (17-21 = High quality)

References

Carter, D., & Baghurst, T. (2014). The influence of servant leadership on restaurant employee engagement. Journal of Business Ethics, 124, 453-464.

Plano-Clark, V., & Creswell, J. (2015). Understanding research: A consumer’s guide (2nd ed.). Boston, MA: Pearson.

https://create.twu.ca/ldrs591-sp18/unit-6-learning-activities/

 

Article

The Influence of Servant Leadership on Restaurant Employee Engagement

Quality Criteria

1.The sampling strategy is appropriate and justified.

Rate

3=Excellent

Evidence and/or Reasoning

a) The researchers explain clearly in the method section about the reason of intentionally choosing Celebration Restaurant as the site.

b) Although the researchers do not explicitly offer the name of the purposeful sampling strategy, it is easy to conclude that the maximal variation sampling strategy is adopted in this study. Because the participants are selected based on several criteria (Carter & Baghurst, 2014).

c) The procedures are fully described and match the maximal variation sampling strategy.

d) The researchers provide strong reasons for selecting the maximal variation sampling strategy in the beginning of the method section.

Quality Criteria

2.The sample size is appropriate and justified.

Rate

3=Excellent

Evidence and/or Reasoning

a) There are 11 individuals participating the study, which means the sample size is sufficiently small so the researchers can capture rich detail.

b) The sample size is consistent with the case study research design.

c) A sound rationale is provided in the participants subsection to justify the sample size is appropriate.

Quality Criteria

3.The data types are appropriate.

Rate

3=Excellent

Evidence and/or Reasoning

The researchers collect different types of data including focus group data, documented data, direct and non-obtrusive observations, and focus group interviews with open-ended semi-structured questions. These types are appropriate for the case study research design.

Quality Criteria

4.The data are gathered using rigorous qualitative procedures.

Rate

3=Excellent

Evidence and/or Reasoning

a) The researchers gather information using open-ended forms to learn about participants’ perspectives and experiences.

b) The data are tape recorded for professional transcription, and the researchers take notes as able.

Quality Criteria

5.Data collection issues are handled ethically and thoughtfully.

Rate

3=Excellent

Evidence and/or Reasoning

a) The participants have read the consent and confidentiality statements and they have the opportunity to withdraw their participation as well as reserve the right to withdraw their participation once the focus groups are completed.

b) The training room is selected for the convenience of the participants and to maintain a level of comfort for them.

c) The participants are compensated for their time.

Quality Criteria

6.The selected participants are information rich.

Rate

3=Excellent

Evidence and/or Reasoning

a) The sites and participants clearly fit the study purpose.

b) Two focus groups have achieved data saturation.

Quality Criteria

7.The database provides extensive and credible information about the central phenomenon.

Rate

2=Good

Evidence and/or Reasoning

The study does not explicitly present the extensive database, but the multiple types of open-ended data collected in two-month period are clearly related to the study’s intent.

Total Score = 20 (17-21 = High quality)

References

Carter, D., & Baghurst, T. (2014). The influence of servant leadership on restaurant employee engagement. Journal of Business Ethics, 124, 453-464.

Plano-Clark, V., & Creswell, J. (2015). Understanding research: A consumer’s guide (2nd ed.). Boston, MA: Pearson.

https://create.twu.ca/ldrs591-sp18/unit-6-learning-activities/

Article

The Influence of Servant Leadership on Restaurant Employee Engagement

Quality Criteria

1.A research design guides the conduct of the qualitative study.

Rate

2=Good

Evidence and/or Reasoning

a) The researchers do not explicitly state the type of the research design, but they mention this article as a phenomenological study. Therefore, I believe this study adopts the case study research design.

b) The researchers correctly use several terms of a case study research design, such as themes. In addition, the research problem clearly calls for an in-depth exploration of understanding the influence of servant leadership on employee engagement from both a follower and leader perspective.

c) There is plenty of up-to-date literature cited in the study to support the research.

Quality Criteria

2.The choice of the research design is appropriate and justified.

Rate

3=Excellent

Evidence and/or Reasoning

a) The two research questions are clearly presented in the introduction section that aim to explore two cases: one is to explore the experience of servant leadership through lived-experience, the other one is to explore the experience and perception of employees.

b) The researchers illustrate the reason why the phenomenological study is used in this study right after the two research questions.

Quality Criteria

3.Good qualitative data collection procedures are used.

Rate

3=Excellent

Evidence and/or Reasoning

a) The procedures include collecting focus group data and documented data, direct and non-obtrusive observations, and interviews. The procedures are appropriate for the case study research design.

b) I think the procedures are used rigorously, although I have not finished reading chapter 10.

Quality Criteria

4.Good qualitative data analysis procedures are used.

Rate

3=Excellent

Evidence and/or Reasoning

a) The researchers use a seven-step process to analyze data and they state that this process has led to findings and themes based on the analyzed data.

b) The researchers particularly identify the five themes in a section. Although I am not familiar with the procedures of analyzing themes, I think they are rigorous.

Quality Criteria

5.Good qualitative results and interpretations are reported.

Rate

3=Excellent

Evidence and/or Reasoning

a) The results section includes a rich description of the focus group responses and a thorough presentation of the themes.

b) Although I am not familiar with the procedures of analyzing results, I think  they are rigorous.

Quality Criteria

6.The study used a rigorous research design.

Rate

2=Good

Evidence and/or Reasoning

Given the qualitative design is a case study research design, most of the elements of a case study design could be identified are depicted in a logical and coherent way.

Quality Criteria

7.The use of the qualitative research design addressed the study’s purpose.

Rate

3=Excellent

Evidence and/or Reasoning

The researchers analyze the findings from different aspects, and all the findings provide a rich exploration that answers the study’s research questions.

Total Score = 19 (17-21 = High quality)

References

Carter, D., & Baghurst, T. (2014). The influence of servant leadership on restaurant employee engagement. Journal of Business Ethics, 124, 453-464.

Plano-Clark, V., & Creswell, J. (2015). Understanding research: A consumer’s guide (2nd ed.). Boston, MA: Pearson.

 

Unit 6 Learning Activities

As a consumer of research reports, the most important things for me in the methods and results section of a high-quality quantitative research report are identifying the type of the quantitative research design, making sure the measures used to gather the data are good quality, and understanding the statistics.

According to the knowledge gained in this unit, I found an important thing for better understanding a research report from each chapter. The first one is to address the quantitative research design of the study. It is the foundation of understanding the analysis in the study. Because the data collection and the results analysis will correspond to the research design, it is crucial to make sure the direction is right before reading further to meet the following sections.

The second one is to determine whether the instruments are high quality. Before reading the chapter seven, I had no idea about the reliability and validity of the scores from the instruments. I just thought the scores were definitely reliable. The author presented a bathroom scale example to illustrate that the numbers obtained from the instruments could be inaccurate or unreliable. Then I realized that I had not considered the reliability and validity of the scores before because all the studies I had read were journal articles. They were all peer-reviewed and were reliable sources. The explanations of the Reliable and Valid have raised my awareness of the scores gathered from the instruments. Given that one day I may have a chance to conduct my study, the data and scores acting as the most important element have to be correct, accurate, and reliable.

The last one is understanding the statistics in the study. Basically, I used to quickly review the statistics in the result and method section because I did not understand the meaning of the symbols and numbers. Although I am still not very familiar with the definitions of some measures, such as the standard deviation and the effect size, I have gained a general framework of all the statistics, measures, and how they relate to each other, such as statistically significant and nonsignificant result. It is significant for me to analyze the statistics and measures by myself rather than just reading the report presented by the researcher. Furthermore, analyzing the statistics by myself is greatly helpful to build the sense of critical thinking during reading the articles in the future.

My question is: How do you evaluate the results of a quantitative study when the information of the statistics or instruments is not clear enough?

Reference

Plano-Clark, V., & Creswell, J. (2015). Understanding research: A consumer’s guide (2nd ed.). Boston, MA: Pearson.

Unit 5 Learning Activities

Article

The relationship between the servant leadership behaviors of immediate supervisors and follower’s perceptions of being empowered in the context of small business.

Quality Criteria

1. The data were rigorously scored and prepared.

Rate

2=Good

Evidence and/or Reasoning

a) Procedures to score the data in a consistent manner were used.

b) The researchers explained how data were prepared and checked the data for errors in the Result section.

c) The researchers did not provide the name of the quantitative statistical software program.

Quality Criteria

2. Good descriptive analyses were conducted.

Rate

1=Fair

Evidence and/or Reasoning

There is little information of descriptive statistics that could be identified.

Quality Criteria

3. Good hypothesis testing procedures were used.

Rate

3=Excellent

Evidence and/or Reasoning

a) The Pearson Product-Moment correlation coefficient (r) was mainly used to test the relationship between two variables in this study.

b) The researchers objectively applied the five steps of hypothesis testing, such as setting the alpha level, collecting data, calculating the statistic and p values, and making decisions.

c) The alpha level was set at .05, but the researchers did not explain the reason.

d) The value of the statistic (r) and the p value were stated clearly.

Quality Criteria

4. The results are comprehensive.

Rate

3=Excellent

Evidence and/or Reasoning

a) There is table in this study which reports the demographic characteristic (N) of the participants.

b) The results presented in the table contain the internal consistency of the instrument’s items and the identification of the six subscales found among the instrument’s items.

c) The results response to each of the study’s research questions.

d) The results are consistent with the overall research design.

Quality Criteria

5. The results include sufficient information.

Rate

3=Excellent

Evidence and/or Reasoning

a) The researchers discuss the results of six different facets including the p value and the alpha level respectively. All the p values are less than the predetermined alpha level (.05), which indicate statistically significant results.

b) The effect size is specificaly presented in the Result section.

c) The information included in the table and the text is clear, consistent, and accurate.

Quality Criteria

6. The data analysis represents a good quantitative process.

Rate

3=Excellent

Evidence and/or Reasoning

The data analysis is a objective and linear process, and the statistical results are found based on the gathered data.

Quality Criteria

7. The results provide a good explanation of the study’s purpose.

Rate

3=Excellent

Evidence and/or Reasoning

a) It is clear how the reported results address the study’s research questions. The authors have interpreted the results and thorough analyses in different aspects.

b) All the statistical tests are clearly related to the study’s purpose.

Total Score = 18 (17-21 = High quality)

References

Plano-Clark, V., & Creswell, J. (2015). Understanding research: A consumer’s guide (2nd ed.). Boston, MA: Pearson.

Van Winkle, B., Allen, S., DeVore, D., & Winston, B. (2014). The relationship between the servant leadership behaviors of immediate supervisors and follower’s perceptions of being empowered in the context of small business. Journal of Leadership Education, 13(3), 70-82.

Unit 5 Learning Activities

Article

The relationship between the servant leadership behaviors of immediate supervisors and follower’s perceptions of being empowered in the context of small business.

Quality Criteria

1. The sampling strategy is appropriate and justified.

Rate

2=Good

Evidence and/or Reasoning

a) The study uses a non-probability sampling strategy for explicitly stating that a combined purposive and snowball sampling method is used.

b) The authors do not provide the reasons for using this sampling strategy. However, as a correlational design, the sampling strategy is implicitly understandable.

Quality Criteria

2. The sample size is appropriate and justified.

Rate

3=Excellent

Evidence and/or Reasoning

a) As a correlational study, the participants are 116 individuals.

b) The sample size is much larger than the minimum size (30 participants) for a correlational design.

c) The authors have done their best to reduce the chance of sampling error by selecting as many possible participants as possible.

Quality Criteria

3. High quality instruments are used to gather data.

Rate

3=Excellent

Evidence and/or Reasoning

a) The authors use two instruments to collect the data: the Essential Servant Leadership Behaviors (ESLB) and the Conditions of Work Effectiveness Questionnaire II (CWEQ II).

b) The ESLB instrument measuring the independent variable is the behavioral observation checklists type of instruments; The CWEQ II instrument measuring the dependent variable is the attitudinal measures type of instruments.

c) The ESLB instrument contains 10 clear questions and returns a Cronbach alpha of .90 indicating a high internal reliability; the CWEQ II instrument consists of 19 questions divided across six subscales and returns Cronbach alphas of .77, .67, .86, .77, .62, and .76 respectively, which indicate a strong internal reliability and consistency.

d) The authors include many citations to the literature indicating that the instruments are previously developed and used for research purposes.

Quality Criteria

4. The data are gathered using ethical quantitative procedures.

Rate

3=Excellent

Evidence and/or Reasoning

a) The researchers obtain consent from participants because the individuals are willing to take the survey.

b) One group of the participants are adult business students recruited from a specific Californian college, and the permission is granted by the college.

Quality Criteria

5. The data are gathered using standardized quantitative procedures.

Rate

2=Good

Evidence and/or Reasoning

a) The two instruments include closed-ended questions that have preset response options so that all participants use the same standard set of options for their responses (definitely no, no, neutral, yes, and definitely yes).

b) There is no information of the training raters to ensure that all participants completed the instruments in similar conditions.

Quality Criteria

6. The study has a high level of internal validity.

Rate

0=Poor

Evidence and/or Reasoning

a) This study is not any type of experiments, it is a correlational one.

b) The procedures in this study is not to test whether the independent variable causes an effect in dependent variables

Quality Criteria

7. The study has a high level of external validity.

Rate

2=Good

Evidence and/or Reasoning

The researchers select a convenience sample, obtain a sample size that is large considering the type of design, use procedures to encourage as many participants as possible, and obtain a relatively high rate of response from participants (130 out of 156 response).

Total Score = 15 (11-16 = Adequate quality)

References

Plano-Clark, V., & Creswell, J. (2015). Understanding research: A consumer’s guide (2nd ed.). Boston, MA: Pearson.

Van Winkle, B., Allen, S., DeVore, D., & Winston, B. (2014). The relationship between the servant leadership behaviors of immediate supervisors and follower’s perceptions of being empowered in the context of small business. Journal of Leadership Education, 13(3), 70-82.

https://create.twu.ca/ldrs591-sp18/unit-5-learning-activities/

Article

The relationship between the servant leadership behaviors of immediate supervisors and follower’s perceptions of being empowered in the context of small business.

Quality Criteria

1.The choice of the research design is appropriate and justified.

Rate

3=Excellent

Evidence and/or Reasoning

a) The design perfectly fits the study’s intent of measuring the relationship between followers’ perceptions of the servant leadership of their immediate supervisor and the followers’ sense of empowerment in the context of small businesses.

b) The researchers focus on describing the relationship between two main variables (servant leadership behaviors of an immediate supervisor, perceived follower empowerment) in a correlational design.

c) The literature presented by the authors is more prescriptive and acts as strong convincing explanation for why the correlational design is provided.

Quality Criteria

2. Good quantitative procedures are used to select and assign participants.

Rate

3=Excellent

Evidence and/or Reasoning

The way that the participants are recruited totally fits the correlational design:

a) The authors recruit the participants who are conveniently available.

b) The participants are willing to take the survey or willing to take part in the study.

Quality Criteria

3. Good quantitative data collection procedures are used.

Rate

3=Excellent

Evidence and/or Reasoning

a) This study is approved by the institutional review board prior for the data collection.

b) The authors make a survey, and the survey link is sent to possible participants. The participants send the survey link to possible participants as well. Therefore, the study provides a wide range of responses for the variables of interest.

c) The authors receive 130 surveys, and 116 are usable.

d) There is no manipulating the experiences of the participants in this study.

Quality Criteria

4. Good quantitative data analysis procedures are used.

Rate

3=Excellent

Evidence and/or Reasoning

a) The quantitative survey used in this study contains two instruments: the Essential Servant Leadership Behaviors (ESLB) and the Conditions of Work Effectiveness Questionnaire II (CWEQ II).

b) The ESLB measures the independent variable, and the CWEQ II measures the dependent variable.

c) The Pearson Product-Moment (PPM) correlation coefficient is used to measure the relationship between the two variables and each subscale.

d) The authors use a table to present the correlation of each subscale with servant leadership behaviors. The table also displays information about the relationships among variables.

Quality Criteria

5. Good quantitative results and conclusions are reported.

Rate

3=Excellent

Evidence and/or Reasoning

a) Claims made are appropriate for the design.

b) The authors claim that the causality cannot be assumed.

c) The study examines that the servant leadership behaviors have a strong correlation with followers’ perceptions that they are empowered.

Quality Criteria

6. The study used a rigorous research design.

Rate

2=Good

Evidence and/or Reasoning

a) All elements of the study from problem to purpose to methods to results to conclusions fit the correlational design in a logical way.

b) The Data Collection section is not very concise and a little disordered. The authors have not specified the difference between possible participants and participants.

Quality Criteria

7. The use of the quantitative research design addressed the study’s purpose

Rate

3=Excellent

Evidence and/or Reasoning

a) The results and conclusions from the research design provide a rigorous explanation of the relationship of the variables that fulfills the study’s intent.

b) The analysis of the limitations of this study also addresses the deficiency of the correlational research design.

Total Score = 20 (17-21 = High quality)

References

Plano-Clark, V., & Creswell, J. (2015). Understanding research: A consumer’s guide (2nd ed.). Boston, MA: Pearson.

Van Winkle, B., Allen, S., DeVore, D., & Winston, B. (2014). The relationship between the servant leadership behaviors of immediate supervisors and follower’s perceptions of being empowered in the context of small business. Journal of Leadership Education, 13(3), 70-82.

Unit 5 Learning Activities