11 Conclusion and Discussion
The Dynamic Learning Maps® (DLM®) Alternate Assessment System is based on the core belief that all students should have access to challenging, grade-level academic content. The DLM assessments provide students with the most significant cognitive disabilities the opportunity to demonstrate what they know and can do. It is designed to map students’ learning after a full year of instruction.
The DLM system completed its sixth operational administration year in 2019–2020. This technical manual update provides updated evidence from the 2019–2020 year. Due to the COVID-19 pandemic, only limited results were provided For a summary of results provided in 2019–2020, see Chapter 7 of this manual. that were appropriate and technically defensible. Because of this, the intended uses of the results were different than what would be intended under a normal administration year. Thus, the contents of this manual are meant to describe changes to the DLM Alternate Assessment System in 2019–2020 and the limited results that were provided, rather than contribute to the assessment validity argument to support the original intended uses of results. For evidence supporting these original intended uses, see the 2014–2015 Technical Manual—Year-End Model (Dynamic Learning Maps Consortium, 2016b) and the subsequent technical manual updates (Dynamic Learning Maps Consortium, 2017b, 2017c, 2018, 2019b). The contents of this manual are summarized in Table 11.1.
| Chapter | Contents |
|---|---|
| 1 | Provides an overview of information updated for the 2019–2020 year |
| 2 | Not updated for 2019–2020 |
| 3, 10 | Provides evidence collected during 2019–2020 of test content development and administration |
| 4 | Describes the new instruction and assessment planner and updates to the assessment blueprint |
| 5 | Describes the statistical model used to produce results based on student responses, along with a summary of item parameters |
| 6 | Not updated for 2019–2020 |
| 7, 8 | Describes results and analyses from the sixth operational administration |
| 9 | Provides additional studies from 2019–2020 focused on specific topics related to validity |
This chapter describes future research studies as part of ongoing and iterative processes of program responsiveness, validation, and evaluation.
11.1 Operational Assessment
As noted previously in this manual, 2019–2020 was the sixth year the DLM Alternate Assessment System was operational. The DLM Consortium is committed to continual improvement of assessments, teacher and student experiences, and technological delivery of the assessment system. Through formal research and evaluation as well as informal feedback, some improvements have already been implemented for 2020–2021. This section describes significant changes, if applicable, from the fifth to sixth year of operational administration, as well as examples of improvements to be made during subsequent years.
Overall, there were no significant changes to the learning map models, item-writing procedures, item flagging outcomes, or the modeling procedure used to calibrate and score assessments from previous years to 2019–2020.
There were significant changes to the assessment blueprint in 2019–2020. The number of overall Essential Elements (EEs) decreased, while preserving breadth of coverage, in order to assess remaining EEs with more items. The increase in items improves the confidence of reporting mastery classifications, which allowed for the fine-grained Learning Profile to be included in student score reports. Performance levels were not reported in 2019–2020 due to the impact of COVID-19; however, a standards adjustment process is planned to update the cut points to account for the adjusted blueprint.
Based on an ongoing effort to improve Kite® system functionality, several changes were implemented during 2019–2020. The new Instruction and Assessment Planner was introduced to assist teachers in the selection of EEs and linkage levels for their students during the optional fall instructionally embedded window. A new writing sample upload process was also created in the system to support teachers in uploading student writing samples associated with student records for evaluation of interrater agreement, rather than having local designees submit the samples outside the system. Additional updates to the writing sample upload process are planned for 2020–2021 to tie samples to specific testlets.
Studies planned for 2020–2021 to provide additional validity evidence are summarized in the following section.
11.1.1 Future Research
The continuous improvement process also leads to future directions for research to inform and improve the DLM Alternate Assessment System in 2020–2021 and beyond. The manual identifies some areas for further investigation.
In 2020–2021, the test development team will begin a review of existing Essential Element Concept Maps (EECMs) and supplemental item writing resources used to develop DLM testlets in English language arts (ELA) and mathematics. Using principles of evidence-centered design and Universal Design for Learning, teams will focus on revising the existing EECMs to incorporate additional descriptions of knowledge, skills, and understandings associated with the linkage levels for EEs, revise existing vocabulary suggestions, and integrate additional supplemental materials, as needed. While the structure and function of EECMs will remain the same, this revision will allow staff to improve the quality and utility of this tool.
Additionally, in the 2020–2021 year, ELA test developers will begin exploring the creation of prototypes for a new design for DLM writing testlets. Current testlets are delivered at two levels: emergent, which encompassess the Initial Precursor and Distal Precursor linkage levels, and conventional, which encompasses the Proximal Precursor, Target, and Successor linkage levels. The new design will include transitional level testlets that will encompass Distal Precursor and Proximal Precursor linkage levels. The prototypes will be used in future item tryouts to evaluate their quality before new writing testlets are developed and field tested in all grades.
Additional Kite enhancements will be implemented in future years, including updates based on teacher feedback to the Instruction and Assessment Planner used to create instructional plans and assign instructionally embedded testlets.
DLM staff members are planning several studies for spring 2021 to collect data from teachers in the DLM Consortium states. Additional updates to the writing sample collection process are planned for 2020–2021 to further streamline the upload process with the intention of expanding the total number of writing samples collected. The teacher survey will be repurposed to evaluate the impact of COVID-19 on instruction and assessment during the 2020–2021 year. However, some sections of the survey that have been included for multiple years (e.g., opportunity to learn) will continue to be included to examine any changes to longitudinal trends. State partners will continue to collaborate with additional data collection as needed.
Research is also underway to improve the model used to score DLM assessments. This includes the evaluation of a Bayesian estimation approach to improve on the current linkage-level scoring model and evaluation of item-level model misfit. Additional research is also planned to evaluate various methods for estimating reliability. The research agenda is guided by a modeling subcommittee of DLM Technical Advisory Committee (TAC) members.
Other ongoing operational research is also anticipated to grow as more data become available. All future studies will be guided by advice from the DLM TAC and the state partners, using established processes.