Full Program Listing*

The ATP ECPS Summit will broadcast live on June 7th and 8th, 2022.

Tuesday, June 7, 2022

11:00 AM - 12:00 PM EDT | Conference Welcome and Opening Keynote: Aligning Large-Scale Assessments with Classroom-Practices to Promote Lifelong Language Learning

The talk will outline a systemic approach to Learning Oriented Assessment (LOA) as an example of lifelong learning that can be used to inform the planning and implementation of educational reforms, with a particular focus on language learning and assessment. It recognises that language learning takes place inside and outside school classrooms and is a lifelong endeavour.

In this LOA approach, language professionals are encouraged to collaborate in building an ‘ecology of language learning’ in their own contexts, and to develop implementation plans that lead to more effective learning outcomes. A framework of reference levels and scales is central to this approach as it empowers all stakeholders to work together in setting learning objectives and raising standards over time based on internationally recognised levels of achievement. The fundamental principle is that assessment goals must be aligned to specific learning outcomes that are related to increasing levels of communicative language ability – e.g. the concept of a learning ladder aligned to an internationally recognised framework of reference for languages. When implemented effectively, this learning-oriented model offers teachers and learners a meaningful picture of progress and the capacity to measure learning outcomes consistently for formative and summative purposes. In terms of psychometrics and measurement, it is critical that the models we use recognise the complexity of the constructs of communicative language ability and how effective learning takes place over extended periods across the entire lifespan.

Dr. Saville will illustrate his talk by showing how assessment providers collaborate with policy makers in national educational contexts using this approach and suggest ways in which educational technology can open up opportunities for progress in this field.

Speaker: Dr. Nick Saville, Director of Thought Leadership, Cambridge University Press & Assessment

Moderator: Ardeshir Geranpayeh

12:00 PM - 12:15 PM EDT | Check out our Sponsors

12:15 PM - 1:00 PM EDT | Panel Discussion: AI and Lifelong Learning, and the Regulation of AI

This panel discussion will focus on AI and lifelong learning, including education and workforce, and the regulation on the use of AI. The use of AI is exponentially increasing in our day-to-day life: in education system, in workforce, in a variety of learning environment, etc. To guarantee improved learning outcomes for students, digital equity, access to meaningful learning environments and positive working environments, it is important to ensure that everything is done to support a responsible use of AI and address risks and challenges

Speakers: Beth Havinga, EdSafe AI Alliance; Alan Thiemann, ATP General Counsel; Russell Shilling, EVP for Research, Riiid Labs

Moderator: Jinghua Liu, Vice President, Global Assessment Service, Pearson VUE

1:00 PM - 1:30 PM EDT | Check out our Sponsors

1:30 PM - 2:10 PM EDT | Podcast: Conversation with Deborah Quazzo about the Future of Lifelong Learning

Deborah Quazzo is one of a few experts with the knowledge and experience to describe in detail the expected impact of advanced digital technologies on learning, workforce, and society.

Deborah is the Managing Partner of GSV Ventures, a venture fund investing in education and workforce skills technologies. She is also the co-founder of the ASU+GSV Summit, a highly successful annual event that brings together innovations and innovators from around the world. Deborah serves on numerous boards including Ascend Learning, Turnitin, and The Khan Academy Thought Leadership Council.

The role of disruptive innovation in the pre-K to gray market and the workplace is rapidly expanding. This ECPS Podcast explores with Deborah the ways transformative technologies are reinventing learning.

Speaker: Deborah Quazzo, Managing Partner, GSV Ventures

Moderator: William ‘G’ Harris, CEO, ATP

2:15 PM - 2:30 PM EDT | Check out our Sponsors

2:30 PM - 3:15 PM EDT | Sponsor Session: Quality Control and Proctoring in the Duolingo English Test

The Duolingo English Test ecosystem is a comprehensive and integrated set of frameworks that guide our multidisciplinary team of experts as they develop, deliver, and evaluate the test. In this session, Jill Burstein will provide an overview of the Duolingo's English Tests lanuage assessment ecosystem; then, Mancy Liao and Will Belzak will narrow in on two elements of the ecosystem; quality assurance and proctoring, respectively.

Speaker: Will Belzak, Assessment Scientist, Duolingo; Jill Burstein, Principal Assessment Scientist, Duolingo; Mancy Liao, Assessment Scientist, Duolingo

Moderator: Sophie Wodzak, Duolingo

Wednesday, June 8, 2022

11:00 AM - 11:45 AM EDT | Day 2 Keynote: The Future of Remote Proctoring

Online proctoring entered the education landscape to fulfill the need of test takers with alternative educational needs. It became apparent that by offering remote education, the expectation for learners to travel to their institution or a testing center for an assessment was unattainable for some and outdated for others. Over the last few years, online education has grown, requiring institutions to adopt remote proctoring to keep their programs operational amidst the sudden closure of their physical institutions. Students wanted the opportunity to continue their education without prolonging their graduation date or having a gap in their academic calendar. With so much growth, the industry faced challenges like never before, including scaling to meet demand and allowing for the customization of learning experience. With these challenges, the industry is growing, and remote proctoring is adapting to the varying needs of test takers everywhere. What began as an alternative and accomodation, has become a mainstream option for some learners for continuing their education, and is shifting to convenience. In offering convenience, increased accessibility and industry regulation are major trends that are affecting the way remote proctoring companies are expanding. The remote proctoring industry has come a long way as a trusted option for protecting the integrity of exams. Join Mike Olsen, the Founder and CEO of Proctorio, for this keynote address, as he discusses the latest trends in online proctoring, and challenges and strengths in the industry.

Speaker: Mike Olsen, Founder and CEO, Proctorio

Moderator: Alina von Davier, Chief of Assessment, Duolingo

11:45 AM - 11:55 AM EDT | Check out our Sponsors

11:55 AM - 1:05 PM EDT | Invited Sessions

Moderator: Jiangang Hao

I am a research director at Educational Testing Service and leading the Psychometric and Data Science Modeling group in the center of Learning and Assessment Foundation and Innovation within the division of Research and Measurement Science. My current research centers on collaborative problem solving, game-based assessment, educational data mining, learning analytics, AI, NLP, and automated scoring. I led several big research projects at ETS, including ETS collaborative science assessment prototype (ECSAP), ETS platform for collaborative assessment and learning (EPCAL), and ETS assessment data analytics solution (glassPy).

Invited Session 1: How Sequence Mining Can Help Us to Understand Failure on Interactive Tasks

Interactive tasks mirror dynamic, real-life environments, aiming at a more realistic assessment of what examinees know and can do. Incorrect responses to such tasks can occur for a variety of different reasons such as low proficiency levels, low metacognitive strategies, or motivational issues. This talk demonstrates how sequence mining techniques facilitate gaining insights into the different sources of failure on a task. Leveraging time-stamped action sequences from the Programme for the International Assessment of Adult Competencies 2012 assessment, we (a) investigate what distinguishes incorrect behavioral patterns from correct ones and (b) identify subgroups of examinees with similar incorrect behavioral patterns. We find incorrect behavioral patterns to be more heterogeneous than correct ones and identify multiple subgroups of incorrect behavioral patterns, which point toward different levels of effort and lack of different subskills needed for solving the task.

Speaker: Esther Ulitzsch, Research Associate, Leibniz Institute for Science and Mathematics Education in Kiel, Germany

Invited Session 2: Network Psychometrics with R

Recent years have seen a rise in the network approach to psychology: modeling human behavior as a complex system of interacting psychological, biological, and sociological components. This approach has since been utilized in various topics of psychological research, such as mental health, attitude formation and intelligence. From the network approach arose a methodological toolset aimed at estimating network structures from cross-sectional and longitudinal data - Network Psychometrics. This talk will present the first-ever textbook on Network Psychometrics with R, in which we introduce the network approach to psychology and the main pillars of network psychometrics.

Speaker: Adela Isvoranu, JSMF fellow/postdoctoral researcher in the Psychology Department of the University of Amsterdam

Invited Session 3: Enhancing Automated Scoring Using Pretrained Language Models

The field of natural language processing has undergone drastic changes with the introduction of large pretrained language models. These models effectively leverage large corpora of data and transfer learning to perform a range of downstream tasks using a process called fine-tuning. Many state-of-the-art results in automated essay scoring and automated short-answer scoring have been achieved in recent years using fine-tuned language models either on their own or in combination with other types of scoring engines. We present how these systems work in practice, some recent results we obtained on the Kaggle short-answer scoring set, and address some of the challenges in applying and deploying these models on a large scale.

Speaker: Dr. Christopher Ormerod, Principal Mathematician at Cambium Assessment Incorporated

Invited Session 4: How Machine Learning Supports Assessment and Learning in the NSF AI Institute for Student-AI Teaming

The NSF National AI Institute for Student-AI Teaming (iSAT; www.isat.ai) aims to (1) develop the theories, Artificial Intelligence (AI) technologies, and know-how for creating next-generation collaborative learning environments composed of diverse students, teachers, and AI; (2) grow a diverse workforce of future AI researchers and practitioners by engaging middle/high school students in innovative AI education through AI-enabled pedagogies; (3) serve as a national nexus point for empowering diverse stakeholders to engage in responsible co-design of student-AI collaborative technologies. I’ll discuss the vision, research agenda, and accomplishments of iSAT, focusing on artificial intelligence and machine learning for assessment and learning.

Speaker: Sidney D'Mello, PhD in Computer Science, Professor in the Institute of Cognitive Science and Department of Computer Science at the University of Colorado Boulder

1:05 PM - 1:15 PM EDT | Check out our Sponsors

1:15 PM - 2:15 PM EDT | Panel Discussion: Lifelong Learning and Assessment in Workforce Education - Healthcare Industry Case Studies

In recent years, we have seen a continued shortage of healthcare workers. An aging population, the recent pandemic, and greater patient acuity all contributed to the need for an ever-larger healthcare workforce. The demand is outpacing supply. According to the research and consultancy firm Mercer, the US will likely face a shortage of 446,300 home health aides, 95,000 nursing assistants, and 98,700 medical technicians by 2025. Can the education technology and testing industry be a part of the solution? How may we leverage assessment and lifelong learning to help current healthcare workers practice at their fullest scope? Can assessment play a role in increasing access to the healthcare profession for aspiring practitioners? In this session, healthcare practitioners and education experts will share their views and highlight challenges in the profession. They will discuss and propose solutions from the perspectives of practitioners.

Speakers: Miguel Paniagua, MD, FACP, FAAHPM, Associate Vice President, Assessment Operations & Medical Education, NBME; Beth Cusatis Phillips, PhD, RN, Strategic Nursing Advisor, ATI Nursing Education; Angie Kronlage, MA, Manager of Allied Health Educators, National Healthcareer Association

Moderator: Kimberly A. Swygert, Director, Test Development Innovations, NBME

2:00 PM - 2:15 PM EDT | Check out our Sponsors

2:15 PM - 2:45 PM EDT | Organizing EdTech for gains in Learning, Equity and Improvement

Innovative educational technologies based on AI, machine learning and data analytics continue to advance in their ability to collect and sort highly granular, and highly individualized, data on the ways that students learn, perform and engage in assessments. As is often the case, technology is developing faster than our ability to make effective and desired use of it — in this case, in formative assessment to support teaching and learning. Stakeholders in the EdTech community have begun to focus on that issue, as evidenced by the EdTech & Computational Psychometrics (ECPS) summit, "Building Quality Digital Assessments for Lifelong Learning." As this effort proceeds, it is critical that we ensure that these educational technologies are inclusive of all students so that they help to narrow rather than widen resource and opportunity gaps among student populations. Equitable deployment and effective use of educational technologies may best be achieved by ensuring that advanced technology platforms deliver positive gains across demographic population groups. But given the extraordinary amount of data being generated, organizing the effort through a unified governing structure, which will ensure the highest level of information sharing among stakeholders as well as policy and process standardization that is similar to what is underway in the digital credentialing sector. This presentation will discuss the importance of these aims.

Speaker: Michael Nettles, Senior Vice President and the Edmund W. Gordon Chair of Policy Evaluation and Research, ETS

2:45 PM - 3:00 PM EDT | Check out our Sponsors

3:00 PM - 4:00 PM EDT | Closing Panel Discussion: A Paradigm Shift in Assessment: The Remote Administration with Online Proctoring

The shift to digital assessments has led to an increase in online test administration. The pandemic accelerated the adoption of remote administration and proctoring for high stakes tests. This panel will focus on the re-evaluation of what is an accessible and secure test administration in this new paradigm, on the analysis of the successes and challenges of remote proctoring and administration, and on the discussion of specific issues, such as potential cheating, test takers’ experience, fairness, and relationships between technology, candidate experience, and test scores.

Speakers: Jarrod Morgan, Founder and Chief Strategy Officer, Meazure Learning; Mike Olsen, Proctorio; Mike Murphy, ProctorFree; Andre Kenji Horie, Duolingo

Moderator: Jim Holm, Examity

* Please note: Schedule is tentative. Check back regularly for any updates. All times listed are EDT.