Assessment Considerations: A Simple Heuristic

In this episode I read and unpack my (2019) publication titled “Assessment Considerations: A Simple Heuristic,” which is intended to serve as a heuristic for creating or selecting an assessment.

  • Welcome back to another episode of the

    csk8 podcast my name is Jared O'Leary

    each week of this podcast is either an

    interview with a guest or multiple guest

    or a solo episode where I unpack some

    scholarship in relation to Computer

    Science Education in this week's episode

    I'm unpacking an article that I wrote

    called assessment considerations Colin a

    simple heuristic and if you hear that

    pitter patter in the background that is

    my dog Minnie who has decided to join

    this episode by running around so if you

    hear some squeaks or some pitter patter

    that's my dog all right here's the intro

    for this paper quote discussions on

    assessment and education often describe

    summative assessments as a product

    oriented assessment of learning

    formative assessments as a

    process-oriented assessment for Learning

    and ipset of Assessments as a

    self-reflective assessment as learning

    each of these types of assessment can

    serve different educational purposes so

    which assessment would work best for you

    and the students with whom you work this

    article presents a simple heuristic for

    creating or selecting an assessment end

    quote alright so this episode is going

    to build off of the discussion last week

    on assessment types so I talked about

    summative formative and implicitive

    assessment and so so now I'm going to

    provide a tool or a framework to kind of

    think through okay well how do I create

    or select some kind of an assessment but

    these are just some things to think

    through and if you want to actually read

    this as opposed to listen to me talk

    about it I'll include a direct link to

    this in the show notes which you can

    find at jaredollary.com or by clicking

    the link in the app that you're

    listening to the song just in the

    description while you're there you'll

    notice that this podcast is powered by

    boot up professional development which

    is the non-profit that I currently work

    for in 2022 but in 2023 I will be

    working at my website jaredelary.com so

    stay tuned for a lot of new free

    resources for gamers drummers and

    computer science Educators alright so

    here's from the article of six W's of

    assessment so the first section is why

    quote when creating or selecting an

    assessment start by asking why you are

    considering using an assessment your

    intended purpose for an assessment

    should serve as a guide for the

    remaining considerations as it can help

    determine if the assessment is intended

    to inform how you teach or facilitate

    provide feedback to students through

    automated peer teacher or

    self-reflective means or simply to a

    sign a grade for example if you're

    looking for an assessment that can

    produce a final grade for a class a

    summative assessment in the form of a

    project or test might serve such

    purposes however if you are looking for

    an assessment that can inform how you

    guide or facilitate each student during

    a project or process formative and

    ipsitive approaches might better serve

    your needs end quote Yeah so if you're

    working on some kind of like a coding

    project or CS project or whatever and

    you're like okay where do I want to

    start with an assessment before you

    figure out which one you're going to do

    you really need to think through like

    well why is it that you are going to do

    this kind of assessment what's the point

    of it like at the end of each one of the

    classes that I use to facilitate there

    are K8 coding classes and then like a

    middle school maker space I would have

    one half of the room go to the other

    half of the room and they would check

    out what they worked on their peers and

    then they would ask some questions and

    the reason why I wanted to do that was

    one I wanted them to be able to see what

    other people are doing to help get

    inspired to figure out oh that's

    something that I might do two I wanted

    students to get feedback from their

    peers through through in-process work to

    be able to ask questions with them like

    hey what do you think I should add here

    or what do you like about my project and

    then three I wanted students to be able

    to communicate to their peers what

    they're working on so they're practicing

    talking about their processes and what

    they're learning those were the goals

    and so the assessment that I came up

    with was having that peer-to-peer

    assessment self-reflection Etc but then

    I also had a different type of

    assessment where I was like okay I want

    students to be able to come in and kind

    of have some kind of direction for where

    they're going and to think longer term

    bigger picture about what is it they

    actually want to be able to do each week

    because they had three weeks in a row

    where they were in my class and I want

    them kind of pace out to like oh this

    week I want to be able to do this next

    week I want to be able to do that I

    think that's a very important skill to

    have so thinking through this I wanted

    students to be able to start planning

    out their creative processes because

    that's something that I found useful and

    so I thought of okay well what exactly

    do I want students to be able to do to

    help with that so that leads to the next

    section of this paper which is what

    quote after outlining a clear purpose of

    why consider what will be assessed when

    an assessment involves the creation of a

    product or expression it should focus on

    the Core Concepts practices and

    understandings evident within the

    creation rather than the creation itself

    in other words assessment criteria

    should focus on the Core Concepts

    practices and understandings associated

    with District standards or Community

    interests and needs rather than

    subjective or arbitrary criteria for an

    example an assessment of the Common Core

    Ela standard ccs.ela

    literacy.w.4.3 quote right narratives to

    develop real or imagined experiences or

    events using effective technique

    descriptive details and clear event

    sequences end quote should assess the

    development of real or imagined places

    through descriptive details and

    sequences of events rather than

    arbitrary criteria such as the number of

    characters in a narrative although not

    everything can or should be assessed

    narrowing the focus to the Core Concepts

    practices and understandings can provide

    Clarity of purpose while simultaneously

    opening up multiple Pathways for

    demonstrations of understanding and cool

    all right so let's let's tie this into

    computer science so in the former

    District that I was working in I was

    running our like CS group and we would

    talk about okay well how do we assess

    projects as students we're submitting so

    when we use icon Academy students would

    submit in like artwork or like

    animations using JavaScript and when we

    would pull up a project we would have

    not only what they created but also the

    code right underneath it so that way we

    could look at that and a lot of the

    times teachers will look at the end

    result and go oh this is really cool

    artwork or animation this looks great

    and so their immediate response was oh

    this student deserves an a because they

    made a wonderful picture but then we can

    actually look at the code you see oh

    they don't understand Loops they just

    like repeatedly wrote out the exact same

    sequence like five times in a row as

    opposed to just using a loop that

    repeated five times so one of the things

    that we tried to focus on in this was to

    not just look at the end result but also

    to look at the code sometimes there are

    happy accidents where it looked great

    but the code was just like not good or

    didn't really demonstrate understanding

    and something they're supposed to learn

    and so what we we often do is like just

    look at the code first analyze that and

    then look at what the end result was so

    the artwork or whatever so after you've

    thought through why you're doing doing

    something think through what exactly it

    is that you're looking at if you're just

    counting stuff like I mentioned last

    week like with number of costumes that a

    Sprite has what's the purpose of that

    does that demonstrate a specific CS

    understanding or concept or skill that

    you think is relevant if so then great

    use that in the rubric if it doesn't why

    is it in there maybe it can be removed

    all right so the next thing to consider

    is when quote knowing what is being

    assessed and the purpose behind the

    assessment can help determine when to

    implement an assessment for example when

    engaging in a multi-day project we might

    assess understandings prior to the start

    of the project shortly after beginning

    the project toward the middle or end of

    the project or well after the conclusion

    of the project in addition to

    considering when the assessment will

    occur also consider when the assessment

    will provide feedback for example an

    assessment that occurs in the middle of

    a project might not be useful if the

    feedback is made available after the

    conclusion of the project however such

    delayed timing between an assessment and

    feedback might be useful when the

    feedback is used as data that informs a

    self-reflection on how a student's

    understanding change over time end quote

    so I don't know about you but when I was

    in like K-12 and higher education

    classes I would frequently get really

    helpful feedback after I received a

    grade and there's really no opportunity

    to apply that feedback in any kind of

    meaningful way and for me that was

    frustrating because it was like I wish I

    knew this while I was writing it so that

    way I can go back and like add in the

    things that the teacher or Professor

    were asking for recommending that's why

    when I was teaching undergrad and

    graduate classes I I had it so that you

    either got like a hundred percent if

    everything was great and then I'd give

    you some feedback in the form usually of

    questions or like a 70 whereas like you

    turn it in on time but like ah it's not

    there's some things to work through or

    things to continue to improve and then

    if you decided to go back and continue

    to work on that you could resubmit it at

    any point and potentially then get all

    the way up to 100 and then a 50 was says

    well there are things to work on but you

    turned it in late and then a zero

    percent was you didn't turn in anything

    so there are only four grades you could

    get and one of the reasons why is

    because I wanted the opportunity for

    that ability to improve so when I gave

    feedback I wanted students to actually

    read the feedback it was usually in the

    form of questions or things to consider

    and I wanted to give them that

    opportunity to apply that feedback in

    some kind of a meaningful way rather

    than just get the grade and then move on

    so again if you got a 70 cool take a

    look at the feedback resubmit you could

    get 100 most students really loved this

    approach and then some students didn't

    like it they were used to just like

    getting a 91.4 because of the quality of

    their work and then not having to like

    redo stuff and so they just kind of

    accepted their fate but those were all

    types of feedback that I've described or

    after the completion of something what I

    really focused on like during class was

    assessments like formative assessments

    during the process constantly walking

    around the room constantly trying to

    figure out where students are at how

    many people to help them in that moment

    Etc now this kind of assessment was in

    person and then the previous assessment

    was usually asynchronous like via email

    or LMS or whatever which kind of gets

    into the next section which is where

    quote depending on when you plan on

    implementing an assessment you might

    consider where the assessment will take

    place if an assessment occurs outside of

    class this can allow for more in-class

    time to focus on Explorations and

    creative applications of understandings

    however when using out of class

    assessments consider whether an

    assessment requires access to physical

    or digital resources that are difficult

    to acquire outside of class for example

    chemistry equipment pianos platform

    specific software Etc and whether

    students should be encouraged to consult

    peers or external resources when

    completing an assessment end quote so

    the where side of things especially if

    you're in higher education or high

    school or there's like tendency to focus

    on like scantrons or tests and whatnot

    if you're doing something like that that

    might take an entire class period and

    let's say you do that every unit maybe

    every month imagine if you could get

    like a whole week or two worth of like

    class time back by not doing the

    assessment in class but if you did it

    outside of class asynchronously if

    assuming students had access to

    resources and whatnot that might make it

    so students can be more engaged in the

    class but to argue with myself it can

    also provide some pressures in terms of

    like more things for them to do outside

    of school which for some can be very

    difficult or demanding if you decide to

    do summative Assessments in particular

    outside of school where it's like a take

    home consider making an open book open

    resource open peer discussion that

    models what we do in real life I mean

    how many of you have like gone on stack

    Overflow and been like how do I do this

    thing and then you just copy and paste

    some code and change the parameters and

    function names Etc I've done that so

    many times I mean it's a meme within the

    programming community so to pretend like

    students can't do that or aren't going

    to be able to do that just doesn't

    really reflect what happens in real life

    in my opinion but another thing to

    consider not only like the why the what

    the win and the where is well who is

    going to do the assessment which leads

    to the next section of this paper quote

    although many discussions on assessments

    position and assessment as an act done

    by a teacher to a student a

    multi-perspectival approach to

    assessments might diversify feedback for

    example while teachers can and should

    assess students automated software can

    provide in the moment feedback on

    individual progress or understandings

    and students can assess themselves their

    peers or even their teachers encouraging

    both external and internal feedback

    distributes the assessment processes

    across people and software which can

    make it much easier to assess a large

    number of people end quote Yeah so even

    though I like generally don't like

    automated graders and things like that

    they can be very helpful if used as a

    tool for reflection which I've talked

    about in previous podcasts do a little

    go oh this thing rated my I don't know

    level of modularity as a four out of

    five or something like that okay well

    how can I improve my modularity or

    whatever or just being able to have

    peers work with each other can be really

    helpful not only the person being

    assessed but the person doing the

    assessment can learn from this process

    and then having it so that students

    assess you the educator or facilitator

    in the room can also be really helpful

    so each week I had students like fill

    out what their goals were for the week

    like I was mentioning earlier and then

    kind of like think through that but I

    also included a question in there that

    said how can I Jared help you with

    whatever your goal is for the week and

    good but sometimes a student would be

    like I could really use some resources

    on like how to do player controls and so

    then I'd go and find a student or a

    resource that would be able to help them

    with that but I also got some feedback

    online Kara's teaching some students

    wanted more structure like hey I could

    really use some more assistance with

    setting my own goals oh cool yeah I can

    help you out with that now I mentioned

    multiple assessment types already

    without necessarily labeling them and

    then talked about some last week but

    that leads into this next section titled

    which quote when determining which

    approach to use for an assessment

    consider each of the following types of

    assessment in relation to each of the

    aforementioned W's of assessment

    summative assessment of learning

    summative assessment is quote an

    assessment at the end of a unit scheme

    term or whatever which is designed to

    summarize student attainment into a mark

    grade or level end quote summative

    assessment is sometimes referred to as

    an assessment of learning because it is

    an assessment of intended learning

    outcomes demonstrated through

    performance examples of summative

    assessment include exams projects

    playing tests portfolios Etc formative

    assessment for learning performant

    assessment is quote an assessment which

    happens as work is being undertaken and

    is purposed with improving the work done

    by the student often undertaken in

    dialogic form in quote formative

    assessment is sometimes referred to as

    assessment for learning because it

    provides feedback which informs future

    learning examples of formative

    assessments include one-on-one

    facilitating and questioning daily

    journals entrants or exit tickets think

    pair share storyboarding Etc ipsative

    assessment as learning ipsative

    assessment is quote an assessment the

    student makes against their own prior

    performance so that they are measuring

    their personal progression against their

    own previous work and quote ipsive

    assessment is sometimes referred to as

    assessment as learning because it is an

    assessment where students are active

    participants in an assessment intended

    to produce learning an example of an

    ipset of assessment that can occur as a

    daily self-reflection or after the

    completion of a project is a reflection

    of what students learn while working on

    a project in comparison with their prior

    understandings EG when working on

    previous projects as well as setting a

    goal for continued learning an ipsitive

    assessment differs from student

    completed summative Assessments in that

    the focus of an ipsitive assessment is

    on reflecting on changes in

    understanding over time and setting

    goals for future learning rather than

    evaluating a completed product or

    expression end quote alright so I

    mentioned each of these three assessment

    types the summative formative ibps that

    is last week so I'll include a link to

    that in the show notes if you haven't

    happened to listen to it and I won't

    elaborate it on here but this gets into

    the very last section of this short

    article which is on combining assessment

    types quote each of the three assessment

    types mentioned above can serve

    different purposes that may address your

    why of an assessment however these

    assessment types are not mutually

    exclusive as each assessment type can

    work together to provide recurring

    feedback for different purposes for

    example in the K8 project-based classes

    I previously facilitated I heavily

    emphasize formative assessments to focus

    on in the moment feedback for projects

    that took weeks or years to complete

    however we also engaged in summative and

    ipsitive assessment as a completion of

    projects for setting weekly goals for

    providing peer-to-peer feedback at the

    end of each class and for providing

    feedback on how I might better assist

    the kids with whom I work throughout

    each of these assessment types I

    followed up with questions to further

    assess understandings as a correct

    answer apparent skill or completed

    product does not necessarily demonstrate

    understanding and abilities that are

    consistently replicable or transferable

    end quote that last point is super

    important like I mentioned with the

    coding teachers that I mentored in the

    previous District that I was working in

    even when a student like created a

    really cool product and their code was

    great you'd think of like what are some

    questions we could ask students to

    really assess whether or not they

    actually understand why it worked the

    way that it did sometimes it was a happy

    accident sometimes a Peter just wrote in

    the lines of code that worked and

    they're like I don't know what's going

    on but hey it's functioning now thank

    you so having this kind of combination

    of summative formative and ipsative is

    extremely important you can assess

    throughout the beginning the middle the

    end and well after the conclusion of

    some kind of a project or product or

    whatever so I guess the main purpose of

    this is to Think Through the different

    questions why what when where who which

    and how you are going to combine these

    different assessment types in the

    different classes that you're working on

    even though I really preferred formative

    I made sure that we still had ipsative

    and summative forms of assessment it's

    important to have this combo of them you

    might lean towards one over the others

    but it shouldn't be I only do summative

    or I only do formative they all have

    their purposes in places but at the end

    of these unpacking scholarship episodes

    I like to share some lingering questions

    or thoughts I kind of embedded some

    elaborations throughout and then because

    I wrote it I don't really have as many

    questions as I normally would one of the

    questions that I have for you is what

    questions or Frameworks do you use when

    considering creating or selecting an

    assessment another question that I have

    is what's missing from this discussion

    yeah it was helpful to have the six W's

    of assessment that's like an easy

    marketing way to kind of like frame

    things but what might have been useful

    for the heuristic that didn't have a w

    in it if you got some ideas feel free to

    let me know on social media you can find

    all my social medias at jaredleary.com

    as well as some links to other

    assessment resources and like 160 some

    odd episodes that are Solo episodes like

    this as well as some interviews with

    some amazing people talking about

    Computer Science Education research and

    practice thanks so much for listening to

    this episode stay tuned next week for

    another one until then I hope you're all

    staying safe and are having a wonderful

    week

Article

O’Leary, J. (2019. Assessment Considerations: A Simple Heuristic. www.JaredOLeary.com


Intro

“Discussions on assessment in education often describe summative assessments as a product-oriented assessment of learning, formative assessments as a process-oriented assessment for learning, and ipsative assessments as a self-reflective assessment as learning (Manitoba Education, 2006; Scott, 2012). Each of these types of assessment can serve different educational purposes, so which assessment would work best for you and the students with whom you work? This article presents a simple heuristic for creating or selecting an assessment.”


My One Sentence Summary

This article serves as a heuristic for creating or selecting an assessment.


Some Of My Lingering Questions/Thoughts

  • What questions or frameworks do you use when considering creating or selecting an assessment?

  • What’s missing from this discussion?


Resources/Links Relevant to This Episode



More Content