Project Evaluation Plan

Internal Evaluator

Dr. Seth Matthew Fishman, Director of Academic and Curriculum Outcomes for the College of Liberal Arts and Sciences, and Assistant Professor of Education and Counseling will serve as the Internal Evaluator, collecting data and advising the leadership team on how to support the ongoing goals of the initiative. Dr. Fishman directs the assessment outcomes process for more than 20 academic departments and assists these departments in their program evaluation and accreditation reviews. Dr. Fishman teaches graduate social science courses in Counseling Research and Evaluation, and Educational Research Methods.  

External Evaluator

Dr. Mariko Chang, President of Mariko Chang Consulting, Inc., will serve as the project’s external evaluator. Dr. Chang is a former Associate Professor of Sociology at Harvard University with extensive experience conducting external evaluations of NSF-funded grants, particularly ADVANCE grants. She has also served on several ADVANCE review panels and site visit teams. 

Evaluation Process

Evaluators will abide by the effective and ethical practices and guidelines for program evaluation conveyed in the Joint Committee on Standards for Educational Evaluation’s (2010) Program Evaluation Standards and the American Evaluation Association’s (2004) Guiding Principles for Evaluators. 

The project team will utilize a multi-method approach to data collection to evaluate the implementation of project activities as well as project impact. The “Leadership During Times of Change” retreat will provide our initial study cohort. The team will conduct a pre-post survey of the participants about the knowledge, skills, and abilities they believe are necessary to serve as change leaders. For the workshops, the team will develop surveys, which include needs-assessment questions to provide formative feedback for the grant implementation team to make adjustments in future activities, while providing immediate feedback on meeting the objectives for each workshop/training to determine short-term impact. Having this baseline data will frame the plan to conduct focus groups and individual interviews with the participants throughout the project period.  

Part of the evaluation process will include the use of implementation audits in comparison with the baseline data. For example, the percentage of departments utilizing diversity statements in hiring, the number of search committees participating in diversity hiring trainings, the success rates of the departments that utilize designated funds to increase their URM diversity in faculty, the number of participants in the various training sessions offered, etc. will be tracked. These audits will provide internal data to enable the project team to monitor the effectiveness of the various activities and make improvements as necessary. This will also provide a set of metrics to continue to track after the grant ends to ensure long-term impact and sustainability. In addition, VISIBLE staff will gather data from the triennial HERI Faculty survey, which will be implemented campus-wide in Spring 2020 and Spring 2023. Villanova’s institutional effectiveness and research office (OPIR), had agreed to provide all relevant data from past and future HERI data and add additional questions related to STEM areas on the 2020 and 2023 surveys.  

External Evaluator: The external evaluator will conduct a formative and a summative evaluation of all project activities. A guiding evaluation question will be to examine the extent to which the project has reached its goals. The formative evaluation will provide immediate feedback to enable the project team to address challenges as they arise and monitor progress towards project goals and intended impacts. At the end of the grant period, the summative evaluation will assess the extent to which the project has achieved its goals and assess the overall impact of the project. At a minimum, Dr. Chang will evaluate quarterly and will provide an annual written evaluation report to the team. Adhering to the AEA (2004) 5 guiding principles of evaluation (systematic inquiry, competence, integrity/honesty, respect for people, and responsibilities for general and public welfare) the project evaluation will be guided by the following questions: 

  • Is the project being implemented effectively and according to the grant timeline? 
  • Are data being collected to provide baseline measures of desired outcomes and track progress toward project goals? Is there sufficient, multiple data methods, to provide a robust analysis? 
  • To what extent have the project results been disseminated (as part of the communication strategy plan) to a broader national audience through scholarly publications, conferences, and inter-institutional sharing?