Explain the term "systematic inquiry" in terms of how it is presented to be impactful to the American Evaluation Association.

Question

Quizplus · Accepted Answer

The Answer is The core statement of Systematic Inquiry is that "evaluators conduct data-based inquiries that are thorough, methodical, and contextually relevant." In order to fulfill this principle, evaluators should, among other things, "adhere to the highest technical standards appropriate to the methods being used." Perhaps the most prominent one involves the commitment to "highest technical standards." Evaluators typically have much more training in research methodology than the stakeholders with whom they work. They know what designs are likely to produce the most defensible conclusions regarding a program's impact, what data-collection strategies hold the greatest promise for obtaining valid evidence in a given context, and, overall, what the chances are that a particular evaluation can answer the fundamental questions that stakeholders bring to the project. Of course, these stakeholders may possess local knowledge and insight that can inform discussion of all these issues. This challenge can be especially daunting when the advocacy comes from powerful stakeholders. In his survey of AEA members using case vignettes, it was found that the more power or influence a stakeholder group was perceived to have over the logistics of an evaluation, the more willing the AEA respondents were to modify the evaluation's design to accommodate the stakeholder's concerns. This result suggests that the impact of stakeholder input on evaluation design may be less related to the technical merit of that input than it is to the political clout of those offering it. We thus have a cautionary tale. In theory, engagement-oriented evaluation approaches may strive to honor the contributions of all relevant stakeholders. In practice, however, it is probable that not all stakeholders are treated equally, a dynamic of which evaluators might not even be fully aware.
The other major ethical consideration raised by Systematic Inquiry involves the assertion that evaluators "communicate methods and approaches accurately, and in sufficient detail, to allow others to understand, interpret and critique their work . . . [and] discuss in contextually appropriate ways the values, assumptions, theories, methods, results, and analyses that significantly affect the evaluator's interpretation of the findings." Stakeholders not only differ in their power, they also vary in their exposure to evaluation and to experiences (e.g., formal education) that can have implications for their ability to readily grasp the nuances of evaluation practice. These differences can place a significant communication burden on evaluators who are committed to substantively engaging a wide variety of stakeholders. Presentations may need to be modified for different audiences and include the preparation of multiple written reports of the same evaluation. The labor-intensiveness of these activities can tempt the evaluator to forego them, in the hope that a "one-size-fits-all" approach will suffice. This hope is risky, however, unless it derives from experience working with the stakeholders in question. In the absence of such experience, using a standardized approach can estrange rather than engage the very groups with which one is attempting to collaborate. Thus, in the preceding example, a report that was not responsive to the advisory board's modest level of methodological knowledge would likely be confusing and intimidating to these stakeholders, who in turn might be reluctant to ask questions of the evaluator that could lead to greater understanding. In these cases, the evaluator's "espoused theory" of collaboration is not consistent with his/her "theory in use," an inconsistency that has ethical significance in that it constitutes a failure to implement a core component of one's evaluation philosophy.

Explain the Term "Systematic Inquiry" in Terms of How It