Receptive & Expressive Mastery: Innovative ASL Testing via GoReact

PRESENTER

Cynthia Sanders

Cynthia Ann Sanders, D.A., is an associate professor at the National Technical Institute for the Deaf at Rochester Institute of Technology. She received her doctorate degree in Communication from the State University of New York at Albany. Her specialization focuses on curriculum design and pedagogical theories in the field of ASL teaching. Dr. Sanders is one of the authors of the sign language curriculum, ASL at Work. The curriculum was published in 2010 and has been adopted by American Sign Language programs at various colleges and universities.

TRANSCRIPT

Molly:

Hello everyone. My name is Molly. I’m here on the GoReact team. We’re so glad you joined us for the presentation today. Our presentation in this session is called Receptive and expressive Mastery, innovative A SL testing via GoReact. I want to make sure that you are all aware that if you have a question for the presenter, please put those questions in the q and a and you can also upvote them if it’s something that you would also like answered. We are thrilled to welcome Dr. Cynthia Sanders here today with us who works at N-T-I-D-R-I-T, and we’ll turn it over to you.

Cynthia Sanders:

Thank you, Molly, for that introduction. I am so excited to be here. This is my first time giving a virtual presentation, so this is new and exciting for me. I do want to call out the name Barry Haywood. You’ll see on the screen, unfortunately, he was not able to attend this session. He was supposed to be a co-presenter with me, but I do want to recognize his contribution to this presentation. So receptive and expressive mastery. How do we test that? How do we prove that someone has expressive and receptive mastery? We use innovative A SL testing via GoReact. So if you look at this quotation, the greatest challenge in creating a good test is ensuring that it measures not just knowledge but understanding.

So I think this quote is very powerful because it’s still true in our day. The way that we design our tests is with a goal in mind that you want the students to be able to incorporate the knowledge, but not just incorporate the knowledge, but also understand it. We don’t want them to just regurgitate information that they are memorizing from our classrooms. We want them to internalize that information, kind of have that light bulb moment, and then be able to prove that they have a total understanding. And to create that kind of test can sometimes be difficult. And this is a struggle for all instructors. We’re not perfect at it, but we always make changes and we always strive to design the best test we can. Next slide.

Today I want to talk about specifically test design. Your evaluations and your rubrics are a separate topic for another day. Those evaluations and rubrics have an important part of your instruction, but that’s not what I would like to talk about today. Maybe we can have a separate session another time, but today I just want to talk about test design, how to make your tests, how to create them in order to meet those outcomes. So in a SL teaching, our research has shown that for a SL levels one through four, A SL level one should be focused on receptive skills. They’re not quite to the expressive skills yet, but they focus on building that receptive ability in that level one course. Then once they hit level two, their expressive skills will start to equal or surpass their receptive skills. So in level two, they start to increase in those receptive or those expressive skills, excuse me. And so they’ll kind of match each other by level two. So when you’re designing a test, you have to design it in a way that incorporates the second language acquisition theories, how language is learned. So that’s an important part to remember. Next slide.

I wish that test design was easy. It truly is not. I’ve been teaching for 40 years, 40 years, and I still struggle with this, with designing my tests in a way that have the outcomes in mind, thinking about the different features, the different aspects, all of those things that are needed in order to test fluency. And understanding all of that in 40 years is still a challenge for me. And there are several challenges. So actful has some articles and research that share some of those things with us. And then there are some other articles about second language acquisition and things like that. And in all that research, I will address some of the things that were brought up.

So for conversational A SL, it’s difficult to measure authenticity. So remember in our ASL one courses, we haven’t gotten to the point where they’re able to converse. They’re just focused on their receptive skills. And so it’s really difficult to measure conversational skills. The second challenge is technology. 40 years ago when I started our technology was these huge screens and not even A VCR. These were massive boxes that we would put our movies in, and it was just a mess. But now technology has changed. And so we have this virtual technology like GoReact that has all of these abilities. But again, there are still some limits, right? There are some ways that we feel like we want to assess our students, but we have to figure out the best way to incorporate technology in that. So sometimes we have to figure out those ways to stretch those students, and we can do that using technology. So within GoReact, we try to develop activities that will stretch our students. We kind of push the boundaries of what GoReact can offer so that we can find those best ways for assessment. So technology is great, but it’s also limited.

Preparation for a test design is very time consuming. When you’re meeting with the teachers, you’re trying to design the standardized test, you have to make these videos, you edit the videos, you develop your rubrics, you set guidelines for your instruction. It’s very time consuming to create a test. And even if you have this great test, it’s still means that it’s extremely time consuming. Another example of a challenge is a large number of students in your course. So with your test design, you have to have the number of students you have in mind because that determines the length of time that you’re able to test these students. There’s a lot of questions to consider with your time and the students’ time.

We often also test our students in pairs, and sometimes that comes with a challenge because each student has diverse skills. Sometimes if you’re setting our students into pairs, sometimes a student with a limited A SL ability will kind of hurt the other person who has greater skills. When we’re assessing the quality of their A SL skills, having them in pairs is really a challenge. And then of course, we have to consider meeting the students’ learning needs. So when it comes to disabilities or accommodations, we have to make sure in our test design that those things are accessible and that they fit the student’s learning needs. And that can be a big challenge.

And then with test design as well, grading is very difficult. You can start to have kind of burnout with test fatigue. We’re expected to turn around these grades in a very quick timeframe, but for us to give valuable feedback, it takes time and it’s an exhausting process. Again, as I’ve said in my 40 years test design, the challenges with test design have not changed. These challenges are still here. So now I want to talk about A SL four courses. These are typically intro to interpreting courses, and in my many years, I’ve seen a receptive test, an expressive test.

So in that receptive test, typically you have a learning management system where you have my courses set up. So RIT, we use an LMS, and our teachers will sign a story, put it up on video, and then there’s a list of questions underneath that. Those questions could be typed out in written English. Some of them, the student’s responses can be signed. Sometimes you can put an image in there or select the correct image. So there are different ways of answering those receptive questions for the prompt. So the student also has the ability to rewatch these videos several times. Sometimes you set a limit to two to three times. I tend to set a limit of about two times because the stories can be kind of lengthy. So I tend to give them two opportunities to watch the video and then answer those questions. So that’s typically how I structure the receptive tests. For expressive tests, I tend to give the students a scenario and I give them instructions on this scenario. So the students can practice ahead of time and they go into GoReact, and they can practice as many times as they want to prepare themselves for this expressive exam. So they’ll hit start and then they sign their portion of this exam.

The challenge with that is the receptive test. It’s a struggle to gauge authenticity. Are we really testing them in an authentic way in the source or the target language? Sometimes with our receptive tests, we have an auto grader that grades those things, but an expressive test we have to go through and watch those videos and grade. So that becomes really challenging. It becomes very cumbersome for a SL level four.

So for a SL level four receptive and expressive test, sometimes we have students that they all get very good grades, they get all A’s. And I’ll meet with Barry, the other teacher, my co-teacher that I work with, and I’ll say, well, this is not meeting the standards. This is not right. This test is not measuring something correctly. There’s something missing here. And so Barry and I decided to do some experimenting, and we gave an additional test and we decided to put the receptive and expressive test in one. So we give an extemporaneous test, and I’ll explain what that means a little bit later. But we do the receptive and expressive test in one, and the students are given a prompt that they read, and then they will sign something inside GoReact. And we’ve noticed that our students are doing well, but that we’re still missing the target language in our assessments. So when we redesigned our test structure and we made some of those modifications, we noticed an improvement. But in our third round, we noticed some more leaps in improvements we were getting there. And I’ll explain that in the next slide. So go ahead, next slide. Okay, before I jump into this new test design that we were testing, I want you to just look at these two words, expressive and extemporaneous.

So I’ll explain the difference between the two. So for an expressive evaluation in an expressive exam, it means that the person is able to use the language. So during a test, they will be able to have practice time to sort of modify, refine their expressive response. And so they do a lot of practice and preparation before they produce the language. With an extemporaneous assessment, the students have to be able to use the language spontaneously. So we have to be able to test them and their understanding of the language without a lot of preparation time beforehand. So they have to be on the spot ready to give us an answer. So it’s a good real time measure of their ability to use the target language. So that’s the difference between the two. So Barry and I talked a lot about this, and then A-S-L-I-E department at NTID started to shift more to this extemporaneous method, and we focused less on the expressive exams.

We wanted to make sure that our interpreters that were graduating had this ability to use the language spontaneously. They had to build their language skills, their interpreting skills, their soft skills, all of these things. And it was important to be able to do those in a real world setting so that they could become professional interpreters and be ready for that field. So we talked with our department and realized that we needed a shift toward this extemporaneous way of testing, and I’ll show you what our test design looked like once we came to this conclusion. So we still do the receptive skills, but we do receptive and extemporaneous tests all in one. We do that at the same time. Next slide.

So in test design, it’s important to remember that you don’t just design one test and immediately hand it to your students. My rule of thumb is always create two. It’s important to have a practice test. And this design test, the practice test is so important because it helps students understand how to follow instructions, how to use GoReact, become comfortable with the process, how to internalize that information and prepare themselves for those tests. So there’s a lot of factors involved to help the students prepare for that test. So I think a practice exam is a must. I don’t think it’s negotiable. I think it’s one of the most important things you can do. So once you have given your students time with that practice test, when they know what to expect, they’re more prepared, they’re less anxious, and you can get a more authentic exam from them. Next slide. So Barry and I did a lot of research to figure out what the test should look like. So this is our outline. Please contact me if you’d like this. I’m happy to distribute this to you via email, so please just reach out to me. But this is our template of how we would design a test. So first we go into GoReact,

And we use a stimulus assignment. So that’s the format we choose for that assessment. So there usually is three screens. I close out the last one and just keep the two. So on the one screen, the students will be able to see the prompt, and then on the second screen is where we’ll see their video. I never have my students touch anything else on the screen during the test. They just have to watch the prompt. We give them some time to reflect and think about their answer and then respond. The reason that I encourage a pause and some for them to think is because in a SL level four, sometimes the content is very heavy. So giving the students about a minute to just think about their response first. We don’t give them time to practice. It’s not enough time to just run through the whole thing, but just enough time for them to formulate a response so that they don’t just jump into it without thinking about it first and giving them just amount of time, just that small amount of time gives them just enough to give an authentic response. So again, giving them about one minute of time to think before they formulate a response is very important. And then they give their response and then they close it out. And I’m able to see those videos for A SL four. Last fall, our students were very confused. They kind of struggled. So I gave two practice tests so that they could get used to using React, understanding the process, knowing what I was expecting of them. And then after that, they became experienced test takers and they were able to do fine. Next slide.

So this is my third exam, and I give directions so the students will know what to expect, what is required of them. And you’ll see that there are three questions here. Those three questions are not specific. We intentionally make them more vague. So once we go through a unit at the end of the unit, I will maybe pick out one thing and just keep it very broad for the students to answer. So sometimes that requires them to go back into the content practice, do a little bit of memorization, reread the content, and then I will show them an image of an animal. The students will read the information, and then they would describe and compare and contrast those two animals.

So for instance, they would see two pictures and they would have to describe the similarities. So they would go by topic, which the students have learned in class, how to do, and we apply that to the testing. The second question asks questions about the animal’s diet. So there’s three categories, and you’ll see this in a video shortly. You’ll see three categories, and you’ll see general questions, open-ended questions, so the students can study, absorb the information, and be ready for their responses. And then when it’s time, then the students are ready to share those expressive answers, and that’s how they prepare for the test. Next slide. So I have two examples for you. Let me pull up these videos.

Okay, so you’ll see the signed question here. Yeah, thank you. Go ahead and skip. Okay, question number two.

Perfect, thank you. So by this third test, the students are able to formulate these responses. They’re able to develop understanding. So what you’re learning, you’re internalizing, and then you’re able to express that information that you are now understanding. So Barry and I started this testing process, but we didn’t feel like it was complete, especially for our A SL level four students. They should be more proficient signers, they should be better with their expressive skills. So next slide. So we came up with this. We designed a fourth version of the test for a SL four students, and I think this is one of the better exams so far. There’s always, always room for improvement, but this is one of the better tests that we have formulated, and so that we give the test instructions that are very similar, and then we modify the test design, and I’ll show you how that works. The next slide, so this exam is focused on automobiles, rules for cars, information about car accidents, et cetera. That’s all part of one unit that we teach. Now, the logistics are the same for using GoReact. We use a stimulus assignment. So all of that is the same. We have just replaced some of the content. So

That’s how it looks a little bit different. Now, the goal of this

Was for Barry and I, let’s say we’re in a friend group, we’re chatting with our friends, and Barry starts talking about a car accident that’s happened. So I ask a friend of mine, so this is again a conversational dialogue that’s happening. So I would say, oh, I’ve heard this and this and this, and the would respond something. So we give them a scenario of an informal setting so that they learn to change that register for those more formal settings and more informal settings. A formal setting being a lecture or a classroom format. But in a more informal setting, it’s a common thing for people to talk about vehicles or a car accident or something like that. Things that have happened in their everyday life, these everyday conversations. So we try to design this test with that in mind. So Barry was talking about this car accident, and there were five parts that he was describing.

So he would start with the first part, and then I would respond with a question or a comment based on what he had said, and then the student would have a turn to respond, like we were having an actual conversation, and we noticed that this was a much more authentic way of assessing our students. It was much more effective. So you’ll see in these directions what I included in this assessment with the specific features that we needed to assess the students on to help them prepare. Again, that preparation is important beforehand. We always give a practice test to help them feel comfortable and ready for the exam. Next slide.

Okay, we’ll have two examples in this video as well. Okay. So you’ll see the question here, my response. Okay, you can pause it there. Yeah,

Let’s just pause it there for a second. You see this two minutes because we give the students two minutes to formulate their response. Sometimes the students worry about two minutes that it’s not enough time, but once they get a little bit of practice time, they tend to not go the full two minutes. It’s usually plenty of time. So two minutes seems to be a reasonable amount of time for the students. And I did check in with them to see if that was enough time, and I just explained that if they use less than that, then just take a few minutes for a breather and to prepare themselves for the assessment. Okay,

Now you can show this please.

Okay. So again, you’ll see this two minutes every time. So we describe the scenario, we ask the question and then give them some time. And we do this in five pieces. So we chunk the test into five sections and we go through. So Barry will sign a scenario and then I will ask a question. And the feedback from our students has been very positive. They feel like this is a real scenario, that they can really get an idea of what a SL looks like, feels like. And so we have some more experimenting to do with our A SL seven class. This group of students had me for a SL four as their instructor, and so that was when I gave separate expressive and receptive tests. So now these students are in level seven. They have me as an instructor again, and I warned them that the test was different.

They asked for a practice test because they were very concerned, and I said, of course, I would give them a practice test, but there’s more required of them because they’re at a higher level. So a fellow teacher of mine and I worked together, we developed this new exam, we gave a practice exam, and the students were very challenged. It was hard for them. They got some of the questions wrong. There were a lot of misunderstandings in the receptive part of the test, but overall, the students said they loved that ability to show their expressive and receptive skills, and they felt like that was a really authentic way to be assessed and to show their skills. So we are on track. We’re still developing and designing more of these tests. I think next year we’ll have some more of that down to bring some more authenticity to the classroom. That’s the goal. We want to make sure their interpreting skills are ready for when they enter the field. So next slide. Okay, we’re kind of right on the dot. So q and a?

Molly:

Yes. Does anyone have any questions they would like to bring up for Cynthia? If you have questions, you can go ahead and put them in the q and a. I don’t have the bottom of the screen. There’s a lot of processing happening. I’m sure there’s a lot of information that you shared. Very impressive. I am a teacher myself, and I do those separate expressive and receptive tests because combining them is a lot of work. But I think collaboration with your fellow teachers, it’s important to preserve that authenticity.

Cynthia Sanders:

And then also with technology evolving, it’s really a wonderful way for that to impact our test design as well. The better technology gets, the better our test design can be because we have more options available to us. So it only gets better from here.

Molly:

Well, I hope that you guys feel free to reach out to Dr. Sanders. She’s a great resource. And then of course, you can reach out to me for anything you need as well. Thank you so much, Dr. Sanders, for being here. This was a powerful presentation and we appreciate it. So we’ll go ahead and wrap up from here. We have four different sessions coming up next, so feel free to join the one you want and don’t forget to also enter to win AirPods. Thank you so much. See you in the next presentation. Thank you. Thank you for coming. Please contact me with any questions.