In the rapidly evolving landscape of higher education, Generative AI (Gen AI) is emerging as a transformative technology with the potential to revolutionize teaching, learning, and institutional operations. However, as with any significant technological adoption, implementing Gen AI in colleges and universities comes with challenges, resource requirements, and potential unintended consequences.

As higher education institutions consider embracing Gen AI, it is crucial to carefully assess its value and impact. By understanding the costs, benefits, and risks associated with Gen AI, decision-makers can make informed choices that align with their institution’s mission, goals, and resources.

As I develop a proposed Gen AI capability maturity model and help lead our university’s AI efforts, a critical question on my mind is, “How do we measure the success of our Gen AI adoption and utilization in higher education?” This question is essential for understanding the value and impact of Gen AI investments and ensuring that these technologies effectively support institutional goals and student success.

Key Questions to Consider:

When evaluating the value of Gen AI in higher education, there are several key questions to consider:

  1. What specific problems or opportunities can Gen AI address in our institution?
  2.  How will adopting Gen AI contribute to our educational mission and student success?
  3.  What resources (financial, technological, human) are required to implement and maintain Gen AI systems?
  4.  What are the potential opportunity costs of investing in Gen AI compared to other institutional priorities?
  5.  How can we measure the effectiveness and impact of Gen AI on student learning outcomes and institutional performance?
  6.  What are the possible unintended negative consequences of Gen AI adoption, such as ethical concerns, privacy issues, or widening digital divides?
  7.  How can we mitigate the risks and challenges of Gen AI while maximizing its benefits?
  8.  How do we measure success in implementing Gen AI in higher education?

Measuring Success: Outputs and Outcomes

Measuring success is critical to assessing Gen AI’s value in higher education. One effective approach is to adopt the concept of outputs and outcomes. When seeking a potential framework for assessing success, one concept that comes to mind is ProSci’s definition of success, which includes project objectives and organizational benefits.

Project objectives focus on the project’s aims, how it will contribute to solving a problem or realizing an opportunity, what it will produce or enable, and how we will know when the objectives have been achieved. These objectives are closely tied to the outputs, which are the direct products or deliverables resulting from the implementation of Gen AI, such as AI-powered chatbots, personalized learning systems, or AI-assisted research tools.

On the other hand, organizational benefits represent what the institution gains from the project. They address the problem or opportunity that the project aims to solve or realize for the organization and define the benefits that will be achieved if the problem is solved or the opportunity is realized. These benefits are often linked to the outcomes: the broader impact and positive changes that the outputs bring to the institution, students, and stakeholders. Outcomes can be measured by establishing key performance indicators (KPIs) and regularly monitoring progress.

Examples of Outputs and Outcomes:

Here are some specific examples of how project objectives and organizational benefits relate to outputs and outcomes in the context of Gen AI and higher education:

Project Objectives and Outputs:

  • Objective: Develop an AI-powered chatbot to assist students with course-related queries and administrative tasks.
  •  Output: A fully functional chatbot integrated into the university’s website and learning management system.
  •  Objective: Implement an AI-driven personalized learning system that adapts to individual student needs.
  •  Output: An intelligent tutoring system that provides customized learning paths and resources based on student performance and preferences.
  •  Objective: Create an AI-assisted research tool to help faculty and students identify relevant literature and generate insights from large datasets.
  •  Output: An AI-powered research platform that integrates with academic databases and provides advanced search, analysis, and visualization capabilities.

Organizational Benefits and Outcomes:

  • Benefit: Improved student support and engagement
  •  Outcome: The reduced workload for faculty and administrative staff in addressing routine student queries led to increased student satisfaction and retention.
  •  Benefit: Personalized learning experiences
  •  Outcome: Improved student learning outcomes, higher course completion rates, and increased student motivation and engagement in their studies.
  •  Benefit: Enhanced research productivity and impact
  •  Outcome: Research processes were more efficient and effective, leading to higher-quality publications, grant success rates, and institutional reputation.

Measuring Achievement:

  • Chatbot objectives can be measured by the number of student queries successfully handled, reduced response time, and student satisfaction scores.
  •  Personalized learning system objectives can be evaluated through improvements in student grades, course completion rates, and student feedback on the adaptive learning experience’s effectiveness.
  •  AI-assisted research tool objectives can be assessed by the number of users, the volume and relevance of literature identified, the insights generated from data analysis, and user satisfaction with the platform.

Realizing Benefits:

  • Improved student support can be observed through surveys measuring student satisfaction, reduced dropout rates, and increased student engagement metrics.
  •  Personalized learning benefits can be realized through data analysis showing improved student performance, increased course completion rates, and higher student retention and graduation rates.
  •  Enhanced research productivity can be demonstrated by tracking the number and quality of publications, grant success rates, and the institution’s research ranking and reputation.

Steps to Measure Success:

To effectively measure the success of Gen AI initiatives, institutions should follow these steps:

  1. Define clear objectives and desired outcomes for each Gen AI project or initiative.
  2.  Identify specific outputs that contribute to achieving those outcomes.
  3.  Establish metrics and key performance indicators (KPIs) to track progress and measure the effectiveness of outputs and outcomes.
  4.  Regularly monitor and evaluate the performance of Gen AI systems against these metrics and KPIs.
  5.  Use data-driven insights to make informed decisions about Gen AI initiatives’ continuation, modification, or termination.

Tracking Progress with OKRs:

One practical framework for tracking progress in pursuing the intended outputs and outcomes is Objectives and Key Results (OKRs). OKRs are goal-setting and management tools that help organizations define and track objectives and outcomes. The framework consists of two main components:

  • Objectives: High-level, qualitative goals that are ambitious and align with the organization’s mission and strategy.
  •  Key Results: Specific, quantifiable measures demonstrating progress towards achieving the objectives. Key results should be measurable, time-bound, and realistically achievable.

By adopting the OKR framework, higher education institutions can effectively monitor the progress of their Gen AI initiatives and ensure that they are on track to deliver the desired outputs and outcomes. The benefits of using OKRs include clarity and alignment, focus and prioritization, measurability and accountability, and agility and adaptability.

Continuous Improvement:

Measuring success involves evaluating past performance and driving continuous improvement. By regularly assessing the impact of their Gen AI initiatives, institutions can identify areas for enhancement, refine their strategies, and explore new opportunities for innovation. This iterative approach ensures that Gen AI remains a dynamic and evolving tool that continues to deliver value to students, faculty, and the institution.

Social Justice and Ethical Considerations:

As we pursue the successful adoption and utilization of Gen AI in higher education, it is crucial to consider social justice and ensure that our efforts are rooted in AI’s ethical and responsible use. We must always remember that AI is a human-centered technology designed to serve the needs and well-being of students, faculty, and the broader community. This means:

  1. Ensuring equal access to Gen AI tools and resources, regardless of socioeconomic status, race, gender, or other demographic factors.
  2.  Addressing potential biases in AI algorithms and data sets to prevent the perpetuation of systemic inequalities.
  3.  Protecting student privacy and data security, and being transparent about how data is collected, used, and stored.
  4.  Fostering a culture of responsible AI use, where the limitations and potential risks of the technology are openly discussed and mitigated.
  5.  Engaging in ongoing dialogue with students, faculty, and other stakeholders to understand their needs, concerns, and perspectives on using Gen AI in education.

In human-centered AI, the POST framework, which stands for People, Objective, Strategy, and Technology, must be considered. Introduced by Charlene Li in the book Groundswell, this framework emphasizes the importance of putting people first when adopting new technologies, ensuring that the objectives align with the users’ needs and goals, developing strategies that support the effective implementation and use of the technology, and finally, selecting the appropriate technology to meet these objectives.

By applying the POST framework to adopting Gen AI in higher education, institutions can ensure that their AI initiatives are human-centered and designed to benefit students, faculty, and staff. This means:

  • People: Understanding the needs, preferences, and concerns of the various stakeholders involved in the educational process and designing AI systems that meet their requirements.
  • Objective: Aligning the goals of AI adoption with the overall mission and objectives of the institution, ensuring that the technology enhances teaching, learning, and student success.
  • Strategy: Developing comprehensive plans for the implementation, integration, and governance of AI systems, including training and support for users, data management, and ethical guidelines.
  •  Technology: Selecting the most appropriate AI tools and platforms based on the identified objectives and strategies and ensuring their compatibility with existing systems and infrastructure.

By prioritizing social justice and ethical considerations in our Gen AI initiatives, we can ensure that the technology benefits all members of our educational community and contributes to a more equitable and inclusive future for higher education.