Data Collection After COVID-19: What Needs to Change?

Marianna Stepniak:

Hi, everyone. Welcome to today’s webinar. As folks are coming in, could you please add into the chat what your name is, where you’re coming from today, and your role or job title? We’d love to see who’s with us today. Thank you for being here with us. Folks who are coming in, please add into the chat your name, where you’re coming from, what state you’re in, and your job or role title. We will get started shortly. All right. I see some folks are adding in. Welcome. Thanks for adding in the chat. Again, your name, where you’re coming from, your job title. We’re glad to have you with us today. I’m going to get started, but please feel free to keep doing that as we move along.

Marianna Stepniak:

Welcome to our webinar today on Data Collection After COVID-19: What Needs To Change? Just before we get started, some webinar etiquette. I’m sure many of you have been to a million Zooms at this point, but recapping is always important. First of all, please use the chat box to engage with other participants. Many of you are doing that already. It’s either at the bottom or the top of your screen. You can just click it to open and type in it to chat. We’ll be doing a Q and A with our panelists at the end of the session today. Please use the Q and A box, which is right next to the chat box, and input questions there. It’s easier for us to pull the questions out of the Q and A box than it is out of the chat box. If you have any questions for our panelists, that’s the place to do it.

Marianna Stepniak:

All right. We have live captions turned on today. Just to give you directions on how to turn them off, if you don’t want them, or turn them back on if you accidentally turn them off, you’ll go down to the bottom of your screen and click on the live transcript or closed caption button. Once you click on that, select hide subtitle. If you want to turn them back on, do this again. Just select show subtitle instead. Thanks for all the folks who are joining us today. We’re just getting started with a few housekeeping notes.

Marianna Stepniak:

We have a number of folks who are supporting this webinar today. Three of the folks on our tech team behind the scenes are Nikevia Thomas. Nikevia is our senior specialist. She is on chat box support. We have Kathleen Pulupa, who is a communications associate, and Kathleen is our social media support. She’s monitoring the Facebook Live and live tweeting this session. We have Jessica Lim, who is our finance associate, and is on tech support today. Our moderators for today are myself … My name is Marianna Stepniak. I’m a content specialist at MAEC. And my colleague, Kasia … Kasia, you’re muted.

Kasia Razynska:

Hi, everyone. It’s a pleasure to be here. I’m the direction of evaluation and continuous improvement at MAEC. I’m going to talk a little bit about what this session is about. Over the last two years, with COVID-19, we have seen many of the metrics that we’re accustomed to disappear, as tests were not given, and many of the accountability systems that relied on those metrics have also not been collecting accurate data. So, as this school year starts, or is already in progress, and schools are reopening again, and we’re starting to collect data again, we wanted to open this conversation up to a wider community. How do we take advantage of this moment? How do we make sure that our data systems are more equitable, and still provide us the data that we need to make decisions that are best for the children our education systems are trying to serve?

Kasia Razynska:

That’s a huge question. We wanted to pull together brilliant minds from the field. So, today, we have with us a mixture of practitioners and researchers, who work both directly with education fields aligned with education, to help us think through how can we rethink education, and how can we use this opportunity post-COVID to make better decisions around data? Today, with us, we have Kecia Addison from Montgomery County Public Schools, and we have Dr. Kevin Gee and Jay Huguley. I will tell you more about them once we hear from them. They will give us short presentations, after which we will have a Q and A, where you’ll be able to ask us questions. I’m passing it back to you, Marianna.

Marianna Stepniak:

Thanks, Kasia. We are looking forward to those presentations. Just a couple of other notes before we get started. The focus of our conversation today is threefold. We’re working on identifying the ways in which data create and amplify inequities and how it can be used to support equitable outcomes. We’re going to describe data use and the misnomer of learning loss during COVID-19, as well as identifying possible new terms to describe this, and share strategies for school and district leaders to identify and collect data that meets student needs, as Kasia said.

Marianna Stepniak:

Before we hop in, we want to introduce ourselves as an organization. We are MAEC. We’re a champion of innovation, collaboration, and equity. This project is hosted under our Center for Education Equity, which we’ll hop into in a moment. So, as an organization, as an educational nonprofit, our mission is to promote excellence and equity in education to achieve social justice. We oversee many different projects, one of which is the Center for Education Equity, which is a federally-funded regional equity assistance center that we operate in partnership with WestEd and AIR. Our region extends from Maine all the way down to Kentucky and West Virginia, including Puerto Rico and the Virgin Islands. [inaudible 00:06:43], we offer a variety of services. We offer universal services, as we are today with this webinar. We also have publications and different blogs. Then we have targeted and intensive projects, which are working directly with our clients on whatever their exact needs are to provide that direct support to them.

Marianna Stepniak:

In our conversation today, we have done the introduction and objectives. We’re going to move into presentations with our presenters and panelists. Then we’ll move into a round table discussion and audience Q and A. So, again, if you have any questions throughout any of their presentations, please add them to the Q and A box on your screen. Then we will wrap up and talk about future webinars. I’m going to pass it over to Kasia to introduce our first speaker. Kasia?

Kasia Razynska:

Yes. It is with great pleasure that I introduce Kevin Gee. He is an associate professor and a chancellor fellow at the School of Education at UC Davis in California. His primary research agenda focuses on the nexus between health and education. He examines the role the schooling systems can play in influencing the health and the wellbeing of children. In addition, he investigates how school policies and programs can help promote the wellbeing and educational outcomes of children who face a broad array of adverse conditions and experiences, including school bullying, food insecurity, abuse, and neglect. Without further ado, I’m going to pass it over to Kevin Gee.

Dr. Kevin Gee:

Thanks so much, Kasia. Welcome, everyone, here today. You can go to the next slide. What I wanted to do today is to do some really high-level thinking about this question about what needs to change. I know change is scary, and we’ve been through multiple changes, but I think when we think about data, the things that we need to change, at the end of the day, are going to help us better serve the students that we all work with. What I wanted to go over today are just three broad ways I’m thinking about what needs to change with how we think about data, and has implications for what we collect.

Dr. Kevin Gee:

The first area that I want to talk about is we need to really think about collecting data on what we value versus valuing what we collect. What I’m going to do is I’m going to go through some examples of what that means. The second theme I want us to talk about today and think about is collecting data to reveal structural challenges versus individual deficits. Again, I’ll give you an example from some work that’s happening in California that’s trying to do that. Then finally, I’m going to wrap up my talk today by thinking about how we can use data and collect data to really uncover both lost opportunities to learn, but also newfound opportunities to learn, as well. When we talk about learning loss, I think that often pins the loss on students, where we need to pull back and say, “But what were some opportunities that were lost? But also, how were we engaging students in different ways during the pandemic? Where’s a pathway forward on how we use that data to inform how we reach especially the most vulnerable student groups?” Next slide.

Dr. Kevin Gee:

So, when we think about what is it that we value, oftentimes, we collect data because we’re told that is what we’re supposed to collect, and that’s what we’re supposed to value. Sometimes we hold these metrics in the data as this sacred thing. In the case of the pandemic, sometimes things were no longer seen as valuable because it was difficult to collect and it was messy to collect. I’ll share with you, I do some work in absenteeism. So, this example comes from the State of Connecticut. Rather than Connecticut pulling back and saying, “Well, we can’t collect absenteeism data anymore. It’s no longer valuable. It’s too hard. It’s too messy to collect,” what they did is they actually leaned into it and dug in, and figured out a way that they … what they valued within their own attendance data system.

Dr. Kevin Gee:

One of the things that they highly valued is we need to know what’s happening with kids by learning modality. I’ll share with you, I do some work with Attendance Works, which is a nationwide organization that keeps its eye on attendance and attendance data collection patterns. I would say over 30 states had some data collection process in place to handle attendance data. 19 states didn’t. So, I think there was this pullback on collecting attendance data. What I want to share with you is Connecticut really thought proactively and said, “How can we collect attendance data not only by modality?” So, they were collecting by in-person, hybrid, remote, but there was also a strong culture of data review. So, it’s not just about the data collection, but it’s what we do with the data. They set up virtual learning communities to review and share the data on attendance, and also to share some practices for implementing multi-tiered approaches to improve attendance data.

Dr. Kevin Gee:

Next slide, I just want to show you what they actually saw in their data, but how informative it was as they were sharing the data across different districts throughout the states. What they were able to do is disaggregate this data to look at the kind of inequities that were happening, in this case, racial, ethnic inequities in attendance patterns. They were also looking at patterns across grade levels. They were also able to really target what’s happening in certain grade levels with certain populations of students. As you’ll see here, what I just want to point out here are you can see some of the spikes in attendance rates, the chronic absence rates, actually, that were high and extremely disturbing, especially across all the different learning modalities. Again, collecting data on attendance, having data that allows them to look at patterns, to look at what’s happening, almost in a proactive way, help them really plan better. Again, what I’ll just add is Connecticut also collected data not retrospectively, but they were doing this in realtime. Next slide.

Dr. Kevin Gee:

So, the next area that I wanted to talk about is focusing on data that allows us to see structural challenges, as well as individual challenges, as well. Oftentimes, when we think about data, we want to describe problems and challenges, and that’s great, but also at times, we are collecting data on individual-level challenges, without thinking about what’s happening within the system. I wanted to share this example from some work that’s been done with my colleagues here at UC Davis. Sacramento has a Chronic Absence Learning Collaborative that brings together stakeholders to look at patterns of absence and data, and to try to think about ways we can better improve attendance. What they did as part of this project is to look at both attendance motivators and attendance challenges. I’ll show you how that plays out in the data. So, next slide.

Dr. Kevin Gee:

What they actually looked at is not just asking, “Okay, why aren’t kids showing up,” or assuming that kids aren’t showing up because they don’t value school. What I’ll share with you is sometimes I work with educators and school leaders, and sometimes they say, “Well, kids don’t show up because kids don’t value school, or parents don’t value school.” I often want to push on that and to say, “Well, how do you know that, number one? What is happening structurally?” What this collaborative did is to collect data on a couple of hundred kids in Sacramento City Unified, and they looked at both the push and pull factors for attendance. Right? Because it’s not just that kids aren’t able to come to school, but what is motivating kids to come to school, as well?

Dr. Kevin Gee:

What they were able to do is really parse out the reasons why students both wanted to come to school, but also what are the challenges? I think that’s a more nuanced way of thinking about attendance. You can see here, 81% of the kids who responded said, “I want to come to school because of people and relationships.” Right? Now, on the flip side, one of the main attendance challenges was student physical health. Okay? So, in allowing us to collect data on push and pull factors, but also look at the structural factors, that helped really shine a light on more than just these individual-level factors that I think we tend to think of, in terms of why kids show up to school or don’t show up to school, but look at the structure of the systems that support good attendance behaviors. So, next slide.

Dr. Kevin Gee:

I’ll also show you some of the more open-ended responses that students gave around absences. You’ll notice a lot of these are about structures and systems that are not necessarily about whether kids are interested in school or not. So, when kids were asked why they’re chronically absent, some of the barriers included being homeless. Sometimes they had grownups who weren’t able to help them to get to school. There was a lack of money for clothes and supplies, even an alarm clock. Sometimes there were family, cultural reasons. What this does is I think it peels back the layers on why kids show up to school and some of the structural challenges. Okay? That also can help us think about, “Well, what are the supports that are necessary to ensure that kids are getting the types of, for example, stability supports that kids may need to attend school?”

Dr. Kevin Gee:

Finally, next slide. I want us to think about this term learning loss, because I know we use this term, learning loss, a lot, but what if we pull back and think about learning loss from a structural lens? But also, what does this mean for the data that we collect? If you think about learning loss, I tend to reframe my thinking around this, and my thinking has evolved on this. I like to think of it as interrupted opportunities to learn. Right? If we think about learning as this cycle, that learning is always interrupted, it’s interrupted by small things that happen in our lives, but also large things. Again, we’ve never experienced such a large interruption. Learning is iterative. Right? So, I think of it as this loop. I think other educators, and some folks in this field, tend to think of learning sometimes as linear. Right? We have these kids. They’re inputs into the system. Then we have these outputs here. I would encourage us to think about learning as more of an iterative process.

Dr. Kevin Gee:

One thing to note is students may not have engaged in the kind of learning that our education system has set up and values. Right? On the flip side though, what were some of the opportunities that students could engage in. Right? What was some of the academic content in the curriculum that they did engage in that looked different because of this pandemic, and this ongoing pandemic now? Given that, I think we need to think about are there ways that data can really shed light on the new learning opportunities and experiences that students engaged in, that they would not have done so in more traditional in-person settings? Okay?

Dr. Kevin Gee:

I’ll share with you on this next and final slide I have with you, there’s a nice article in The New York Times about what students were saying about learning loss during the pandemic. I pulled this quote because I think it helps us reframe this notion of “learning loss,” and where was the actual loss? This student, her name is Julia, said something very thoughtful. She says here, “During the pandemic, I don’t feel as if I had experienced the threat of learning loss.” She says, “Well, in fact, I feel like I’ve learned more during the quarantine. This is because during my free time, I found ways to learn at my own pace and ways to practice what I’ve studied before.” She says, “This is the cause of learning loss.” Not being away from school, but rather, students have not found a way to learn and practice at their own pace.

Dr. Kevin Gee:

If we pull back, and think about what would data collection look like, number one, on what we value? Okay? If we value something like student independence, or students learning to take more autonomy over their learning, then how do we measure that? The second thing is how are we tapping into the data that allows us to get at the kind of learning experiences that students may have had but that were not recognized? Again, learning happens. Right? Learning continues. It’s not like learning just stopped. It’s the learning that, I think, are tied to these metrics that we’re sometimes bound to because of accountability systems. But if we expand our thinking about what learning is, what we value, looking at both structural challenges as well as individual challenges, I think that gives us a much more expansive and holistic view of the data that we need to drive decision making. From that, Kasia, I’ll pass it back to you. I’m looking forward to engaging in conversation and Q and A with everyone. Thanks.

Kasia Razynska:

Thank you so much, Kevin. I think that sets us really well for our next panelist, Dr. Kecia Addison, who is going to talk to us about what it looked like on the ground in one of the biggest districts in the country. So, Dr. Addison is the director of the Office of Shared Accountability for Montgomery County Public Schools here in Maryland. She has more than 20 years of progressive experience in research and evaluation, complemented by a demonstrated commitment to professional organizational experience. In her current role, as the director of the Office of Shared Accountability, she is devoted not only to the fair and equitable assessment of youth, but also to applied research and to examination of educational equity to lead to closing of the achievement and opportunities gaps among students and groups. Without further ado, I’m going to pass it to you, Kecia.

Dr. Kecia Addison:

Thank you, Kasia. Next slide, please. So, for a few minutes, I am going to share with you how, when our district reacted to the onset of the pandemic, and additionally, share how we moved from a state of reacting to proactive planning and intentional planning for data collection to inform the work in our district. So, as many of us who work in a school district remember this time very vividly, March 2020 was the time when, at least in the State of Maryland, it was announced by our state superintendent that effective March 16th, all school buildings will be closed due to the global pandemic. That initial closure was expected for one week, but we all know how that turned out.

Dr. Kecia Addison:

So, as a district, we worked on detailed plans and made adjustments to keep our school leaders informed and staff and families informed on things such as grading implications, testing windows, and even food distribution for our students and families. When the closure was announced, we were almost at the end of our third marking period. So, a decision was made to freeze grades while teams worked to develop guidance for how to grade students. Fast forward to further into the process of the remainder of that 2019-2020 school years, adjustments were made to grading practices. For the fourth marking period, grades of pass/fail were provided, which was a shift for us. We also reduced district assessments aligned to the curriculum. Additionally, state assessments were canceled.

Dr. Kecia Addison:

As a district, we have a locally developed accountability system that we call the Evidence of Learning Framework. As part of this framework, we use external measures like the NWEA MAP assessment. We also decided not to administer that assessment in the Spring. Essentially, we were trying to build a plane as we were flying it, but we kept that frequent communication with schools about our progress. At this time, we also began planning for Summer opportunities, because we recognized that with our need to immediately shift, and the adjustments made in the instructional day and instruction, there was a need to provide opportunities for students in the Summer. Next slide, please.

Dr. Kecia Addison:

In the Summer of 2020, we provided free virtual opportunities for students in elementary, middle, and high schools. As part of the Summer program for elementary and middle, we administered pre and post assessments. This was intentionally planned to assist us in not only knowing what support students need, needed, but also to provide information to evaluate the impact of our Summer opportunities. So, our Summer program served slightly more than 49,000 students at the elementary and middle school level. That number didn’t include the high school students, who also accessed and took courses. In addition to Summer programming, we worked centrally to plan for sharing with our board of education and the community the impact of the immediate school building closures on students.

Dr. Kecia Addison:

This plan included not only planning for Fall diagnostics, which was to include the NWEA MAP assessment, but also planning for end-of-year reporting through our Evidence of Learning Framework. So, given our need to shift to remote teaching and learning, adjustments to grading practices and the reduction of state and district assessments, we were not able to tell an end-of-year academic story for the 2019-2020 school year, as we typically did. With our plan to administer a diagnostic in the Fall, and as I stated previously, using the MAP assessment, there was a lot of learning for us as we still were in a state of school building closures. So, that assessment had to be given remotely. There were a few hiccups with the administration with technology from the vendor side, technology from the student and family side, but once we got through that, when it was time to present to our board, we presented three-year trend data to our board of education in late-October, as our preliminary lens of the impact on student performance.

Dr. Kecia Addison:

So, we also had to contend with those who did not believe in the results from virtual administration. So, we heard a lot of, “Oh, the data is invalid or reliable.” So, we were fortunate enough to have a great working relationship with our NWEA partners, who had recently pushed out research around the reliability and validity of a virtual or remote administration of that assessment. We also replicated that study in our district and found confirmatory analysis with regard to that, too. Next slide, please.

Dr. Kecia Addison:

I do want to take a few moments to share a little bit more about our Evidence of Learning Framework, which drives a lot of what we do. It was initiated during the 2016-2017 school year, and was the vision of our superintendent at the time, Dr. Jack Smith. He wanted a way for us to better support students and espouse five questions that guided this work and other work in the district. Are students learning? Are they learning enough? How do we know? If not, why not? What are we going to do about it? This Evidence of Learning Framework uses multiple measures. Classroom measures, such as report card grades, district measures, which are curriculum-aligned assessments, and external measures, such as the MAP and state assessments. It’s used to assess student learning and inform instructional decisions. So, it’s our multiple measures approach to identify what our students know and are able to do in both literacy and math, and allows for us to examine, identify measures within the three categories. Next slide, please.

Dr. Kecia Addison:

So, continuing with our Evidence of Learning Framework, we use this to report to our board at the end of the year. So, to connect to some of what Kevin stated in terms of what is the data that we value. Right? We value this data to help us tell the story of how our students perform, and we report on our readiness grade levels. So, our readiness grade levels of grades two, five, eight, and 11. These are the grade levels prior to the transition. So, if you see here, grade two ends that primary level. Grade five ends the intermediate level at the elementary school level. Grade eight ends middle school. 11th grade is when they’re about to transition into that high school, 12th grade, and almost ready for college and career.

Dr. Kecia Addison:

Fast forward again to the end of the 2020-2021 school year, we resumed grading practices that were in place prior to the onset of the pandemic. We administered district assessments. Although there was still not an end-of-year state assessment, we administered the MAP assessment in the Spring of 2021. So, this positioned us to share the performance of the students in literacy and mathematics, which we recently shared with our board of education this past September. Since this was the first time we could present again on this framework, we used pre-pandemic results from 2018-19 school year as a comparison. Next slide, please.

Dr. Kecia Addison:

So, here are our findings. We noted decreases for students in our readiness grade levels in both literacy and mathematics compared to 2018-2019, pre-pandemic performance, which wasn’t surprising to us. We expected that. We had been planning for that all along. With the decreases observed, our data revealed that the largest decreases were for our youngest learners, specifically our grade two students in the area of literacy. While there were decreases across both literacy and mathematics, the decreases were not as large in mathematics as they were for literacy. That was more surprising for us because we typically saw higher decreases, or we saw more challenges, in mathematics pre-pandemic.

Dr. Kecia Addison:

Then we had a few bright spots in our data, in our examination. We also disaggregate our data by different student groups. So, we noticed that for our Black or African American students not receiving free and reduced-priced meals, and for our Hispanic and Latino students not receiving free and reduced-price meals, and students with disabilities in grade 11, we saw a few increases. So, with that, I will turn it back over to you, Kasia.

Kasia Razynska:

Thank you so much, Kecia. I’m going to pass it over to our last panelist, Dr. Jay Huguley, who is the associate dean of diversity and equity and inclusion and an associate professor at the School of Social Work at the University of Pittsburgh. He is also a researcher who’s focused on school-based interventions that promote positive development and outcomes for students of color, particularly in urban settings. He’s the principal investigator for the Just Discipline Project and the co-principal investigator on both the Pittsburgh Parenting Project and the Pitt-Assisted Communities and Schools. I’m going to pass it over to you, Jay.

Dr. Jay Huguley:

Kasia, thank you. I want to thank the previous speakers, Dr. Gee and Dr. Addison, for what they shared. Deeply informative. I hope what I share will complement what they’ve already taught us today. I’m going to talk about our work here in Pittsburgh, which centers on behavioral and social-emotional outcomes for students within school climate and school behavior management frameworks. This comes out of our Just Discipline Project, which we work in partnership here with about five districts locally. It’s supported by The Heinz Endowments. This is Heinz like the ketchup. Yes? They are a major philanthropic body. Their generous support has gotten us started and made a lot of things possible here and in this work.

Dr. Jay Huguley:

We have examined the discipline practices in Greater Pittsburgh. We’ve looked at the racial disparities here. You probably are familiar with a lot of those patterns nationally. They tend to be pretty consistent. Here in Greater Pittsburgh, Black students are suspended more than seven times the rate of white students in the region. The rates are worse in suburban districts than they are in the urban districts in terms of disparities. We’ve piloted effective practices locally. I’ll say a bit about what we’ve done. We have efforts to influence policy and practice across the region, with important implications for data and data management.

Dr. Jay Huguley:

We have a multi-tier model, and I think what’s important for the work that we’re talking about today is we’re very interested in how students are experiencing school, what their school-based experience feels like for them, because we understand those linkages between school experience, relationships that they’re experiencing. Going back to Dr. Gee’s slide where he talked about those motivational factors, and relationship being the number one factor for many students, that’s everything. Right? So, if you work in schools, and you know that, it’s so important. We look at that, especially for students from historically oppressed groups, economically disadvantaged students. These things are critically important. We do multi-tier work, and we look especially at these students that are dealing with trauma, dealing with severe adversity at home, often come to school and face adversity in school settings, and how we can help them be high achievers, thinking about that in the context of COVID-19 era.

Dr. Jay Huguley:

We’ve had great results around reducing suspensions and having academic gains in where we’ve worked. Teachers are, most importantly, well, very importantly, happy with this work that focuses on climate, that focuses on relationships, because we provide them not only with mandates and things they have to do with staff in the building to do the work. So, they’re happy with that. When we looked at what happened last year, even as the onset of the pandemic in 1920, we’re having similar results before the pandemic hit. I think everything changed for everyone that March, as Dr. Addison mentioned.

Dr. Jay Huguley:

Mental health and behavioral health in schools. It’s important to recognize that for African American students in particular, even before the pandemic, we were seeing a rising mental health crisis. We’ve seen suicide attempts rising, depression rates that are rising faster than they are for other groups. We see Black youth less likely to receive treatment for these mental health needs, for a wide range of reasons, much of which stems from the long history of economic oppression in this country. People don’t have access. People don’t live in areas where they have the resources. Then you have to deal with stigma. There’s been a lot of racism and racial oppression in the mental health and health community. People have some feelings about those kinds of services.

Dr. Jay Huguley:

Conversely, 70% to 80% of students that receive mental health supports in this country received them through their schools. So, these supports are critically important. In the pandemic era, you’re talking about these supports, largely, not happening, or being very much reduced for students. There’s a group of students that for a year and a half have not been getting access to services they really need. This is in other areas, as well, that impact mental health, like nutrition. Schools do a lot of this work, and in many cases, it wasn’t able to happen. If you talk to school leaders, the ones we work with or the ones we talk to, they tend to feel under-resourced in terms of supporting student mental health. That’s just a problem already before COVID-19, and it’s all exacerbated by poverty.

Dr. Jay Huguley:

As students return from the pandemic, what are we starting to see? What are we thinking about? What are the implications for data collection? Well, this actually goes right along, I think, with what Dr. Addison was highlighting. Behaviorally, we are hearing about tougher challenges with younger students. Imagine a six year old or even a seven year old that they’re just now going into a school building for the first time, in some cases. Some students have aged out of their previous level. You can have a 16 year old that is just now starting high school that’s not been in the building before. Social relationships may need work. Schools are major socializing agents, and students have not been exposed to that and are not getting that opportunity. They’re getting it much later than they would have, for many. For students receiving IEP and mental health supports, they’ve missed special supports that they would have otherwise gotten.

Dr. Jay Huguley:

Ironically though, ironically though, and actually, Dr. Addison, your finding on the gains in achievement in certain Black and Latinx groups was interesting, because one of our researchers here found that students reported being less stressed, in some cases, having not been in school. We have to also know that school can be a stressor. It can be something that students don’t look forward to, or they don’t feel treated well there. They don’t feel like they have caring relationships there. So, you can’t take it for granted that being away has been harmful in terms of their mental health, but for students that are getting supports, that are getting help, and for young learners, we are seeing that there are challenges that need to be resolved.

Dr. Jay Huguley:

All of that in mind, when we think about data collection targets, I’ll break them into two separate categories. Pre and post assessments, things that we’ve heard about already today that we need to compare to maybe where we were before to assess what’s happened. Certainly, academic standing is key. Here in Pittsburgh, I think we had similar results to what we just heard. What we heard was that the academic gains in Pittsburgh were lower than they typically were during the pandemic year. Students didn’t go backwards, but they didn’t gain as much as they typically did. They gained about two-thirds what they typically did. Now, that doesn’t count students that weren’t able to be tested remotely. So, I think that’s a major factor. In fact, some grade levels, they don’t have data. They don’t trust the data because they didn’t get enough response rate. That’s key.

Dr. Jay Huguley:

Diagnostics, I think, on students’ self-perceptions of wellness, and how do they feel? How are they feeling right now? Simple diagnostic measures around whether they feel stressed, whether they feel like they have social support, how they feel affectively. This can be folded into school climate measures that you may already collect. And school climate data. How are people feeling about where they are, how their school feels? Compare this to where a building was two years ago and see if there are any major shifts that might be related to the pandemic. We’re, obviously, looking at behavioral outcomes and doing month-by-month comparisons. I mean, if you want to get granular, you can’t really look at behavioral data by year, because as school administrators will tell you, month-to-month is different. You have to compare October to October. Here, you have to compare May to May. You can’t compare May to August, I mean to November. You have to look at those outcomes and see what your trajectory looks like. We can’t forget about the teachers. How are they feeling in terms of their satisfaction and their needs?

Dr. Jay Huguley:

There are other assessments that are one-time assessments and a result of the pandemic. What are teachers feeling like is different now? What do they need now? Let’s talk about also qualitative data. Let’s talk about quickly gathering data from teacher assemblies. It doesn’t have to be large-scale data collection. How are teachers feeling? Teachers/staff surveys on mental health in the building, their needs themselves, and what they perceive students to need, and yes, their own self-care status. In conclusion, I think we have to think about the experiential component of the schools. We have to think about that from students and from teachers. We have to think about the students that were getting services and getting supports that they haven’t gotten now for many, many months, and really be watching and understanding where they are. Thank you.

Marianna Stepniak:

Thank you so much, Jay. At this point, we’re going to invite the other panelists to turn on their cameras, too, and we’re going to move into our question and answer. Again, audience members, if you have any questions for our panelists about any part of what they’ve just presented, or something that they haven’t talked about yet, please add your questions to the Q and A box at the bottom of your screen. Kasia is going to lead with the first question now.

Kasia Razynska:

Thank you so much. Some of you have started addressing this question, but we just wanted to get your opinions. If you are a district that’s coming back from the pandemic, you have a lot of competing decisions to make about where can you put your eggs, for lack of a better word. Where should you focus in the data collections as you’re bringing them back up? If you had to provide advice, where do you think would you get the most bang for your buck? What is the data that school districts should start with and should start focusing on that is probably going to be the most accurate, or give them the most actionable information, as systems are coming back up online, as testing is starting to happen, in the immediate period right now as schools are coming back in?

Dr. Kecia Addison:

I’ll start. It’s a tough question. Right? It really is a blend of what we all discussed. Right? When I think about what we’re doing as a district, yes, we need to know the academic aspect of students to help figure out how do we better support them, in terms of the opportunities they missed? I appreciate how Kevin rephrased that. I’m stealing that. Then also what Jay mentioned, too, because we have a heavy focus on the social-emotional wellbeing of our students, too, because we recognize during that isolation phase, for many of our students, they missed out on that social aspect. They are having to re-engage and learn how to socialize again. It’s creating that balance between the academic piece, as well as the social-emotional. It’s the whole child.

Dr. Kevin Gee:

Yeah. I agree 100%. To the extent to which I think it’s important for districts and schools and school leaders to have conversations about what they want to be collecting, and what is it they truly value. Kecia, I appreciated that pyramid diagram that you showed, and saying, “This is what we value.” That’s something that I think local districts and communities need to decide on, and to figure out a pathway forward on how to best collect data. I also just want to echo what Dr. Huguley said about data. We, oftentimes, think about data as numbers. There’s sometimes a false dichotomy between quantitative and qualitative. I think the more that we can also collect experiences of students and their stories … Because no one ever made a decision just based on a number. Right? I think they need a story to underline that, to really motivate the decision making.

Dr. Kevin Gee:

That said, one final thing is there also needs to be a capacity to generate the data, but also to interpret the data. I know, in some cases, districts, states, may be constrained with who’s available to actually take all of that data and make meaning of it, as well. More and more, I think if districts, if school leaders, could think about, “What is it that we want to do with this data? What do we want to show? How do we analyze it,” there’s a front-facing piece to the data, as well. How are we going to communicate this effectively to stakeholders, to our students, to our parents, so they can see what’s happening?

Dr. Jay Huguley:

Yeah. I agree with what my colleague said. Practically speaking, what, also, can you turn around quickly. Right? In Pittsburgh, they did give the state assessment at the start of the school year to assure that everyone got it, because not everyone got it in the Fall. I don’t think it’s come out yet, what the results are. So, classroom teachers can do some of their own assessments on an academic level to see what their kids need compared to where they typically are, and then the qualitative measures can be collected quickly. Not to say you can do a full qualitative analysis quickly. That’s definitely not true. But you can get a sense of where people are relatively quickly, and you can do a climate survey in a building in a short time and start to get a sense of where things are for people pretty quickly. We can’t let two or three months pass while we wait for analytics in this day and time.

Marianna Stepniak:

Thank you for all of your answers. I’m opening up the Q and A box, and I see a couple of questions here. Colleen asks, “How do you look at the intersection of data? Do you have specific protocols?” When I read that, I think about intersections within data in particular, as Kecia was saying, looking at specific student groups. So, with this question, how do you identify what those intersections are as you’re looking at data and keep that in mind?

Dr. Jay Huguley:

I’ll jump in because, I mean, in the behavioral space, we know that the intersectionality is critically important, and we know that, for example, Black boys are the highest, usually the highest, group in terms of receiving disciplinary infractions. We also know that Black girls experience the highest levels of disparities compared to white girls, in terms of how they’re perceived disciplinarily. That’s very important to think about. When we’re coming off a scenario where kids have been out of school, as Dr. Addison also noted, that they haven’t been in these social spaces, how are we perceiving them? How does our cultural continuity impact how we perceive them? Because you’ll see things like Black kids being under-diagnosed when they have legitimate mental health needs because people write it off as behavioral. You only write it off as behavioral if you assume that’s natural for that group. Right?

Dr. Jay Huguley:

So, that’s a racist perspective, unfortunately. We have to think about those intersections, and then we have to think about intersections with special education designations, because that population tends to be heavily disciplined. So, IEP, race, race, gender, and also LGBTQ students, where they can be identified, typically, also heavily, discipline, and we have to be alerted to that, as well.

Dr. Kecia Addison:

One of the things that we intentionally do in our district is look at race and poverty together. Right? So, when we analyze our data, we call them our focus groups, not from the lens of research and evaluation, but these are the groups that we pay attention to. So, when you saw my slides, I talked about our Black or African American students not receiving free and reduced-price meals. We have an intentional focus to look at those receiving free and reduced-price meals, which is a proxy for being impacted by poverty, and then those not receiving it. We look at it by racial ethnic groups, and examining our data through those lenses, because the assumption is when you see everyone, you look at it, “Oh, we’re looking great,” but then when you start dis-aggregating that data by these different groups and looking at the intersections between them, you can see a very different story. So, I would say that intentional planning around how do you want to examine your data is very important.

Dr. Kevin Gee:

Then I’ll add on top of an intersection, intersectional approaches, it’s also thinking about disaggregation, as well. Just to echo what Kecia had mentioned. In the State of California, we have a very diverse Asian American population of students, but oftentimes, that group gets clumped under Asian Americans, and we know Asian Americans have very different experiences, if we look at experiences of students whose backgrounds are they’re Hmong students, or Laotian, or Thai, relative to Chinese, Chinese American students, or Japanese American students. So, to do these thoughtful, intersectional pieces, how things cross with socioeconomic status, with disability status, with race ethnicity, we also need to drill down how data is being disaggregated in thoughtful ways to look at patterns.

Dr. Kevin Gee:

Again, we can’t be treating all foster youth … I do some work in this area, as well … as this monolithic group. So, the more that we can use data to uncover the layers, I think that’s going to be helpful in decision making.

Marianna Stepniak:

Yeah. We have one more question in the chat that we’re going to ask, and then Kasia will pose the final, and then we’ll wrap up. So, our last question in the chat box comes from Arnie, who says, “Schools just started, and NAEP has just released the results, the nation’s report card, which finds failing test scores even pre-COVID. How should schools be reacting and responding to this national snapshot?” Any reactions?

Dr. Jay Huguley:

I’ll jump in. Why not? Why not? Hey, it’s a great question. Thanks for it. I mean, I think it’s alarming. I mean, we want to pay attention to it. We can’t ignore it. NAEP, it’s not a high-stakes measure. Right? So, what that means is, one, people can ignore it, at least at local levels, and two … [inaudible 00:51:04] conversely, because it’s not high stakes. As far as tests go, it’s pretty accurate, because people aren’t really gaming it. You definitely have to pay attention to it, but I mean, it’s going to require a deep dive. This is the first I’ve heard this. It’s going to require a deep dive. It’s going to require disaggregated looks. It’s going to require regional looks. It’s going to take a lot to figure out what are the national trends? What are the things that seem to be happening nationally, where our pedagogy hasn’t caught up with the world, or we don’t know how to handle social media, or we don’t know how to handle devices, and kids are on devices and not sitting in their cubby and actually doing their work, like they might have been 30 years ago.

Dr. Jay Huguley:

Those kind of trends, we’re trying to figure out versus equity-related issues, regional approaches, policy-related approaches, how we’re managing poverty. We have a widening gap between economic haves and have-nots. In Pennsylvania, we have one of the widest school funding gaps in the country. So, it’s going to take some deep dives to figure out what the game plan is. I think the main thing right now is to really take it seriously, make sure people understand this is hard to respond to.

Dr. Kevin Gee:

What I’ll quickly add is, again, this is a nationwide snapshot, so I think in terms of what does this mean for local schools and school decision making, I think it’s understanding what your own achievement data is saying. Does it reflect what’s happening, knowing that schools may look different than other schools within the same districts, and districts look different than other districts within the same state, as well? That said, Jay was talking about the underlying why. Why is this happening? That’s that more difficult question. Setting up data, systems, to ask the why. Is it something about the curriculum? Is it something about school belongingness? Is it something about school climate? Is it something that’s beyond the school walls? Right? We know that so much impacts what happens in schools that’s outside of the sphere of influence of school systems. Is this a health issue? I think it’s going to take multiple data systems that are going to have to interact with each other to be able to answer the why we’re seeing this. That’s critical.

Dr. Kecia Addison:

Just one additional piece to add. I know I typically like to look at NAEP data when it comes out. Right? To what Kevin said, and also connecting to what Jay said, part of it is thinking from a local lens. Okay. Well, how are our data reflecting in this? When we look at our patterns on literacy and math, do we see similarities in that, or do we look different? We think about the norming group, as well, the systems who participated in the NAEP assessment. Do they look like our district? Do they not look like our district? Where do we see those differences? What can we learn from what comes from that assessment that can help us better instruct our students?

Kasia Razynska:

I’m going to ask a closing optimistically-phrased question. You have a magic wand, and you are able to reimagine how schools collect data and use data. What would that look like, from your perspective and from your lens?

Dr. Kevin Gee:

I’ll go ahead. From my live vantage point, I think having just a really robust data system that’s interlinked, as well. I know, for example, Minnesota has this really rich data system that links its data from the juvenile justice system and their foster care system and child welfare system with schools, because we know so much of what happens in schools is affected by what’s happening around and in kids’ lives. So, if we can have a system that interlinks data, that requires a lot of political will. Sometimes it takes a lot of just navigating the politics over data. Right? Data is very powerful. Data in itself, the numbers, are mutual, but how it’s used and who uses it, whose voices are amplified within that data is very, very political. If we can have a system that gets those issues out there and that puts out that we’re using data for the purposes of trying to reveal inequities and trying to address them, and having an interlinked data system. Also, just the people who can leverage that data system, as well. The people who can analyze it and present data, as well, I think is important.

Dr. Kecia Addison:

I was like, “Why can’t I answer this question? If I had a magic wand, what would I want?” I struggled. Right? But I agree 500% with what Kevin just stated, in terms of an interconnected system to allow you to answer any question that you need to answer. Right? When we think about from a behavioral side, from an academic side, from extracurricular side, from out-of-school time, anything that connects to a student, all of that data lives in the same place and allows us to be able to tell the story of what students are experiencing. That also connects and brings in what their voice is, too, around that. Yes. That would be perfect.

Dr. Jay Huguley:

I think I would echo the student voice piece. I think we underestimate how much they know about school and how much they know about what makes school effective. You know what? For better or worse, they know which teachers are strong teachers and which teachers are not. We may or may not want to listen to them, but they have that information. I think that is extra valuable. Then the power, the resources, to really use the data. I mean, some districts have offices that could do that work and translate it into actual practice, and it can be formative. It’s not just once a year. Because by the time you figure out what happened in the Spring, it’s really late to try to really make adjustments. What can happen on an ongoing basis? That translates back into the practice.

Dr. Jay Huguley:

It takes more than resources and relatable folks that can look at a monthly report and turn that around into something useful for teachers. That has accountability but also is tied to support. I think that’s the last point. Accountability has to be tied to support. Accountability without support is just going to make people hide, make people game the system. You need to have support if you’re going to have accountability. That’s a definite magic wand component.

Marianna Stepniak:

This conversation is only an hour today, but this … I mean, we could go for hours talking about the different systems necessary, how to make data actionable. I’m just jotting down notes. So many more questions I still have to ask, but we are out of time today. Thank you so much to our incredible panelists for your presentations and for this amazing Q and A. I’m going to share my screen with our survey QR code on it. We would love to hear from our audience members. What was good today? What could be even better next time? We really appreciate your feedback here. If you want to get in touch with us, it’s MAEC. We have Twitter, Facebook, LinkedIn, and a website. Please reach out to us. If you have any questions that weren’t answered today, you can ask them in the chat. Leave your email, we can get back to you. So, closing with thank you. Thank you to our panelists. Thanks to our tech team. Thank you to our audience for joining us today. This was wonderful. Thank you so much, everyone.

Dr. Jay Huguley:

Thank you.