Full Program Listing*
The ATP ECPS Summit will broadcast live on June 6th and 7th, 2023.
Monday | June 5, 2023
Pre-Conference Workshop: Learn more
Tuesday | June 6, 2023
11:00 AM - 12:15 PM EDT | The AI Revolution and Its Implications for Assessments
We are experiencing the rise of artificial intelligence (AI) at scale. Generative AIs for text, image, video and sound are everywhere. Sometimes their results are impressive, sometimes they are mediocre and repetitive, other times they are made up. Nevertheless they are powerful, global, and hype. What do these intelligent entities mean for assessment?
In this opening panel we are grounding our discussion in three areas of impact:
- Implications for the constructs to be assessed;
- Implications for assessment development; and
- Conceptual implications for working, teaching, assessing and learning alongside an intelligent tool.
In this symposium, several expert users of these technologies will share their results, experiences and insights for the future of assessment. This future is now!
Speakers: Jiangang Hao, ETS; Sue Lottridge, Cambium Assessment; Matthias von Davier, Boston College; Victoria Yaneva, National Board of Medical Examiners; Alina von Davier, Duolingo
12:15 PM - 12:30 PM EDT | Check out our Sponsors
12:30 PM - 1:15 PM EDT | AI in Education: The Industry Perspective
While for most of humanity the recent advances in artificial intelligence (AI) are surprising and overwhelming, several educational companies have been already building applications for the precursors of these tools, and now they are updating their products. At the same time, education providers are adjusting to support the responsible integration of such tools in teaching and learning at all levels in ways that the increasingly evident limitations and threats of these advanced AI tools do not outweigh the potential benefits. In this panel several industry experts will discuss their experiences with, and opportunities and challenges of these technologies.
Moderator: Steve Shapiro, Finetune, a Prometric Company
Speakers: Rebecca Hamer, International Baccalaureate; John Kleeman, Learnosity; Martyn Farrows, Soapbox Labs; Kishore Vakati, Pearson VUE
1:15 PM - 1:45 PM EDT | Check out our Sponsors
1:45 PM - 2:30 PM EDT | The Transformative Impact of Artificial Intelligence in Medical Education and Assessment
This podcast features two eminent academics in the field of artificial intelligence in medical education and assessment who will discuss their research and expand on how innovations can support personalized education and precision medicine. We will discuss where AI tools are most useful in designing medical school curricula, providing feedback to residents, and supporting better admissions processes through the medical education spectrum.
Moderator: Kimberly Swygert, National Board of Medical Examiners
Speaker: Dr. Marc Triola, NYU Langone School of Medicine; Dr. Schaye, NYU Langone School of Medicine
Wednesday | June 7, 2023
11:00 AM - 11:45 AM EDT | Learning and Assessment in the Age of AI
Artificial intelligence has made huge strides in the past months, bringing with it a lot of hype and a lot of fear. In order to get past both, we need to understand what current models do well and what they don't. Using new models should build on what we already know from learning and assessment research. Hear about ways this was attempted with the implementation of generative AI at Khan Academy and what has been learned so far with its use in an offering for learners and educators.
Speaker: Kristen DiCerbo, Khan Academy
11:45 AM - 12:00 PM EDT | Check out our Sponsors
12:00 PM - 1:00 PM EDT | Invited Sessions
Invited Session 1: Leveraging Response and Process Data to Detect External Assistance in Digital-First Language Assessments
The integrity of test results and score validity in assessments can be compromised when test takers maliciously seek external assistance. Traditional methods for detecting cheating in test-center-based assessments may not be directly applicable to digital-first assessments, which are delivered online and administer a unique set of items to each test taker.
This presentation outlines the approaches employed to detect various forms of external assistance in digital-first language assessments by leveraging both response and process data. To detect suspicious test takers who access external websites or braindumps during the assessment, we utilized natural language processing techniques to identify written or spoken responses that resemble public websites or historical test takers' responses. In detecting ghostwriting instances, where malicious test takers hire others to complete the assessment on their behalf, we employed keystroke dynamics analysis to identify potential impersonation. The combined use of response and process data enables a comprehensive approach to uncovering different forms of malicious behaviors, complementing each other to enhance detection capabilities. In addition to discussing the techniques themselves, we will address the considerations and challenges encountered when implementing these approaches in a production setting.
Speaker: Mancy Liao, Duolingo
Invited Session 2: Using Large Language Models to Generate Reading Assessment Passages
This talk discusses how automated item generation (AIG) can reduce the reliance on human subject experts in the production of high-quality items for computer-based assessments and individualized learning platforms. Although AIG has been used in test development for some time, the use of machine learning algorithms has introduced the potential to greatly improve the efficiency and effectiveness of the process. The approach presented in this talk utilizes OpenAI’s GPT-3, a transformer-based language model, to generate reading passages. The AI-generated passages are carefully crafted to have similar content and structure to a fourth-grade reading passage, using existing passages as prompts. Multiple passages are generated for each prompt, and the final passage is chosen based on its Lexile score agreement with the original passage. The selected passage then goes through a simple revision by a human editor to ensure its accuracy. The generated passages, along with the original ones, are evaluated for coherence, appropriateness for fourth-graders, and readability by human judges.
Speaker: Ummugual Bezirhan, Boston College
Invited Session 3: Exploration of the Utility of Person Misfit and Anomaly Measures in Developing Cheating Detector Using Machine Learning for Large-Scale Assessment
Machine learning methods have been explored for cheating detection in large-scale assessment programs. Both item response and response time (RT) data are more often analyzed to develop a detector. A few studies experimented with other data and data augmentation in cheating detection. This study explored the utility of both person-fit measures from psychometric analyses and outlier measures from anomaly detection methods for data augmentation in different machine learning algorithms for cheating detection. An empirical data set from a high-stake large-scale testing program was analyzed to demonstrate the application of the proposed method. The performance of the proposed method was compared with alternative approaches. It was found the proposed data augmentation approach effectively increased the cheating detection accuracy with the highest F1 score compared with the values reported in similar studies.
Speaker: Hong Jiao, University of Maryland, College Park (UMD) & Maryland Assessment Research Center
1:00 PM - 1:15 PM EDT | Check out our Sponsors
1:15 PM - 2:00 PM EDT | Applications of Computational Psychometrics
Handling complex data requires a new perspective of new methodologies to accommodate the complexities of the constructs being measured and embraces digital technology which has become an essential part of our daily communication. Computational psychometrics (CP) blends theory-based psychometrics and data-driven approaches from machine learning, artificial intelligence, natural language processing, speech processing and data science to help us provide a better theoretical and practical framework for improving our assessment/learning. In this session, experts will bring examples from the world of assessment/learning on how CP is changing our world.
Moderator: Ardeshir Geranpayeh
Speaker: Kate Knill, Cambridge University; Ada Woo, Ascend Learning; Alina von Davier, Duolingo English Test
2:00 PM - 2:15 PM EDT | Check out our Sponsors
2:15 PM - 2:45 PM EDT | How is AI Impacting Educational and Workforce Technologies? Better Question, What is the Cost?
ChatGPT and other AI technologies that impact edtech, assessment, and the education industry are taking the world by storm. There are many questions from legal, technology, and workflow, but the biggest question is what will it cost? How does this impact the business? Where is the money coming from and going?
Please join this interview-style session, where the most significant questions about how this affects the business and market will be answered. How do we see this shaping up the industry for the future?
Speaker: Jason Palmer, New Markets Venture Partners; Brodie Wise, ITS
2:45 PM - 3:00 PM EDT | Check out our Sponsors
3:00 PM - 4:00 PM EDT | Ethical Consideration of AI
In the era of AI and machine learning, how can we address ethical considerations of its use? OECD (2022) pointed out that AI development should be based on “human-centered values, including human rights, fundamental freedoms, equality, fairness, the rule of law, social justice, data protection and privacy, and consumer rights and commercial fairness.” In practice, how do we manage AI risks to bias and discrimination, and to privacy and data governance? How do we ensure AI transparency and trustworthiness so that people are aware of how AI is being used in making predictions and decisions? How do we ensure that those who develop and deploy AI systems are held accountable for any consequences that may arise? Please join our panel discussion on “Ethical consideration of AI.”
Moderator: Jinghua Liu, Pearson
Speaker: Marc Weinstein, Marc J. Weinstein PLLC; Jill Burstein, Duolingo; Charles Foster, Finetune Learning
* Please note: Schedule is tentative. Check back regularly for any updates. All times listed are EDT.
Register Now!