Mind the Gap: The Illusion of Skill Acquisition in Computational Thinking

In this episode I unpack Bao & Hosseini’s (2023) publication titled “Mind the gap: The illusion of skill acquisition in computational thinking,” which compares learning, perceptions of learning, and confidence among adult learners participating in interactive, video-based, and text-based learning.

  • quote while interactive platforms

    improve the perceived learning compared

    to other methods educational videos also

    provide a sense of confidence among

    Learners Although our findings do not

    show any impact on actual learning the

    gap between confidence in applying

    algorithmic techniques and perceived

    knowledge suggests that educational

    videos when used in isolation may be

    detrimental to learning by providing an

    illusion of confidence end quote that's

    from page 783 of the article titled mind

    the gap colon the illusion of skill

    acquisition and computational thinking

    which was written by yeding Bao and Hari

    hosseini apologies have mispronounced

    any names here's the abstract for this

    paper quote with the Advent of online

    educational platforms and the advances

    in pedagogical techniques self-directed

    learning has emerged as one of the most

    popular modes of learning distance

    education elevated by the kova 19

    pandemic involves methods of instruction

    through a variety of remote activities

    which only rely on educational videos

    for mastering in the absence of direct

    student engagement the asynchronous

    nature of remote activities May

    deteriorate the quality of education for

    Learners students often have an illusion

    of skill acquisition after watching

    videos which results in overestimation

    of abilities and skills we focus on the

    efficacy of skill acquisition through

    interactive Technologies and assess

    their impact on computational thinking

    in comparison with delivery through

    other traditional media EG videos and

    text in particular we investigate the

    relationship between actual learning

    perception of learning and learnness's

    confidence in adult Learners our results

    reveal intriguing observations about the

    role of interactivity and visualization

    and their implications on the

    pedagogical Design for self-directed

    Learning modules end quote Vada

    summarize this article into a single

    sentence I'd say that this article

    Compares learning perceptions of

    learning and confidence among adult

    Learners participating in interactive

    video based and text-based learning and

    as always in the show notes you can find

    the link to this article as well as over

    interviews with guests or multiple

    guests or these like solo episodes where

    I unpack implications of scholarship for

    computer science Educators in addition

    if you go to my website jaredelary.com

    there are hundreds if not thousands of

    free computer science education

    resources as well as a bunch of gaming

    and drumming stuff because I create

    content for a living now even though

    this article is talking about adult

    Learners I'm going to talk about what

    this looks like in K-12 classes as well

    as just computer science education

    classes in particular now if you don't

    know who I am I have a background

    working with all grades kindergarten

    through doctoral student in a variety of

    different contexts like music education

    and computer science education classes

    micv is linked on the website if you're

    interested in that kind of nerdiness now

    in the introduction of this article the

    authors talk about how there can be an

    illusion of skill acquisition for people

    who learn something by watching a video

    so in the introduction they talk about

    how there are some self-directed

    learning platforms like think of Khan

    Academy which has like a bunch of math

    stuff as well some Computer Science

    Education and web development courses

    and then they unpack what they mean by

    computational thinking which as I've

    talked about in other episodes is a very

    messy term that means many different

    things to many different people and here

    are the questions that kind of guide

    this particular study this is from page

    skill acquisition in teaching

    computational thinking through a variety

    of learning Technologies is there a gap

    between the perception of learning

    actual learning comprehension and

    learners's confidence end quote those

    are some really important questions to

    consider I've done some Publications and

    even some podcast episodes that kind of

    unpack self-efficacy of like teachers

    and Learners which isn't to say that

    those are two separate things but

    basically to say that between adults and

    kids but just because people gain

    confidence in being able to do something

    doesn't mean they're actually better at

    doing that thing now as an example of

    that because I have like a drum kit

    right behind me I could show you a bunch

    of videos that would teach you how to

    develop limb Independence to make it so

    that your two hands do two different

    things separate from your two feet and

    even your mouth you could have five

    weight Independence while playing on the

    drums and I could show you many

    different videos on how to develop that

    and it would make sense and you go oh

    that makes sense however actually

    putting into practice is a completely

    different thing you have to develop the

    myelination and the neural connections

    across your brain to actually be able to

    make it so that your limbs can basically

    interact in an interdependent way so

    there's a difference between

    understanding and skill or even

    confidence and understanding in an

    actual application of that understanding

    same thing applies for computer science

    so if we're talking about programming

    here I might be able to conceptually

    understand an idea or an algorithm or

    whatever but actually applying that and

    creating that on my own and putting it

    into use within a program that's a

    completely different skill set which one

    do you value or focus on in the classes

    that you work with so this paper is

    going to kind of explore that a little

    bit now if you're interested in section

    is on page 779 kind of gives an overview

    of learning comprehension perception of

    learning and confidence and interest in

    learning it gives like a summary of the

    paper itself so if you want a very quick

    read of that go ahead and take a look at

    that but this podcast will also kind of

    unpack each of those areas and then in

    section 1.2 they talk about some related

    work so you can find some studies that

    are related to this in some way alright

    so section two is on design and

    development so they kind of unpack well

    what was the interactive platform how

    did this compare with the educational

    videos and how did this compare with

    based instruction so all three were kind

    of given to different groups if you want

    to learn more about that check out

    section 2.1 and 2.2 and if you want to

    learn more about the experimental design

    I really appreciate the authors that

    actually gave a deep dive into here's

    exactly what our demographic survey

    looked like or here's what our

    assignment and workflow will look like

    Etc so that is all discussed on page 780

    in the experimental design as well as in

    going to nerd out on that in this

    episode but it is worth reading if

    you're interested in that so you can

    hopefully replicate this kind of study

    or design to see if you get similar or

    different findings with your different

    populations all right so let's get nerdy

    and talk about the results so this is

    starting on page 782 so 4.1 is on the

    perception of learning so in this

    section they kind of talk about the

    perception of learning as well as their

    confidence with learning here's a quote

    from page 782 quote while the Learners

    of the Interactive Group reported

    significantly higher score for perceived

    comprehension in contrast to video and

    text-based groups the participants in

    the video groups were as confident about

    their knowledge as those in the

    Interactive Group this finding suggests

    that educational videos may lead

    Learners to develop overconfidence about

    the subject matter in contrast

    interactivity provides an opportunity

    for Learners to get their hands dirty by

    manipulating the input and directly

    interacting with the algorithms which in

    turn aligns learners's perceived

    comprehension with confidence end quote

    now this is really important to consider

    so it's great that videos might be able

    to help like improve confidence with

    learning or understanding something but

    the interactivity actually made it so

    that they were able to apply their

    understanding in some way and kind of

    like maybe get some feedback in terms of

    what worked and what did not work

    depending on what kind of feedback you

    get from the platform again it's one

    thing to be able to understand something

    and it's a completely different thing to

    be able to actually apply that

    understanding consistently now what does

    this mean for educators so for myself

    what I really tried to focus on was

    maybe introducing a concept at the

    beginning of a class in the first like

    five minutes out of a 40 minute period

    at most and then the rest of the time

    was actually students being able to

    apply that on understanding in their own

    projects or explore it in different ways

    so if everybody happened to be working

    on the exact same thing which was a very

    rare circumstance in in the classes that

    I work with because everybody was able

    to pick a number of different projects

    with a number of different programming

    languages and platforms Etc and apply it

    in a meaningful way but in the few

    instances where students were all

    working on the exact same thing I would

    spend some time introducing an idea and

    then that would take up a very small

    percentage of the time the rest of the

    time would be spent applying that and me

    working with students one-on-one kind of

    coaching them or having their peers kind

    of Coach each other now if this had been

    flipped this would have made me look

    like 35 minutes of like video based

    lecture or me at the front of the room

    kind of talking about things in five

    minutes of application while that time

    spent learning from an expert whether

    it's from a video from a teacher or

    whatever may have like developed some

    sense of confidence of understanding

    that subject area the actual application

    of that understanding is a very small

    percentage of time so I've talked about

    in other podcasts if we think of like

    music or language or any kind of skill

    or understanding you need to be able to

    actually practice it to improve at it

    and apply it in different contexts to

    constantly look at a concept or practice

    from many different angles and many

    different perspectives to better

    understand how to use it and modify in

    different scenarios so for example if

    you're learning something like a for

    Loop if you only go and create one for

    Loop and don't understand how it works

    in other programs and other contexts

    other scenarios Etc you're going to have

    a very narrow understanding and not

    understand how to use it in many

    different scenarios or when to use it in

    comparison to like a while loop or

    something else how much time you

    actually spend exploring those different

    scenarios as opposed to just learning

    the concept itself will have a direct

    impact on how good you are at that it's

    the same thing with learning like a

    language or an instrument or like a

    sport or whatever you can't just watch

    videos of people teaching you how to do

    it you actually need to do it so a

    recommendation that I might give is that

    while students might have an increased

    sense of confidence worth learning

    something from a lecture or a video or

    whatever you need to actually really

    think through how much time we're

    actually spending applying that

    understanding because that will have an

    impact on learning which Speaking of

    four point 2 is on the actual learning

    comprehension now what's really

    interesting about this is that the

    authors found no statistically

    significant difference between learning

    comprehension among the Interactive

    Group and the video group so again even

    though they had an increase in

    confidence they did not have an increase

    in actual learning comprehension but

    what about with the interest in learning

    well 4.3 kind of talks about that very

    quickly now when the authors compared

    the text-based group with the video

    group there was an increase in interest

    among the video group however there was

    not a statistically significant

    difference between the video group and

    the Interactive Group if you have an

    option of being able to hand a textbook

    to a student or an interactive platform

    online or a video on how to do something

    in this particular study it appears that

    the textbook would be the least

    interesting to some adult Learners and

    maybe the most interesting would be the

    video or the interactive platform I

    would argue that kind of depends on what

    the content is that is being taught and

    who you are teaching it to or who is

    learning that content but in general I

    could totally see why video and

    interactive media would be more

    interesting or more engaging than just

    reading through text now under 4 0.4

    they talk about usability so this is a

    quote from page 782 that kind of talks

    about well how do these three groups

    compare quote First the interactive

    treatment required participants to

    proactively engage in the learning

    process by manipulating the input thus

    more effort is required in this mode

    compared to the video group second the

    educational videos contained audio in

    addition to the visual explanation we

    hypothesized that audio can have an

    important impact on the overall learning

    experience as expected those in the

    text-based group were less satisfied and

    found the instructions complicated and

    sometimes confusing end quote now if you

    look into like studies on learning

    theories and like cognitive load Theory

    Etc you'll find that like mixed media

    like something like a video that has

    like visuals and audio going on

    simultaneously might be more interactive

    and more interesting than something

    that's just text based but it really

    kind of depends on how it's presented

    like there are ways to create videos

    that maximize cognitive load and make it

    so that it's very confusing just like

    with text but as I've mentioned in many

    other episodes the more ways that you

    can kind of like take in

    multi-perspective approach or look at

    something from many different angles the

    better so with the text it's a little

    bit harder to do that you could do that

    with different stories different

    vignettes kind of explaining different

    applications of things or you could

    actually show that through a media like

    a video but again different contexts

    different scenarios different

    applications kind of depends on what

    students are learning and whether or not

    that would be more valuable than just

    having a conversation or reading

    something about it now in section five

    they talk about the positive

    correlations that they found in this so

    an example of a positive correlation

    would be the correlation between actual

    learning comprehension with the

    perception of learning so if students

    had a higher perception of learning they

    would likely have a higher learning

    comprehension and vice versa they also

    found a positive correlation between

    usability and learning so the more

    usable something is then the better it's

    going to be for people to learn that

    thing and then section 6 talks about

    limitations and Lessons Learned and then

    section 7 talks about some of the

    concluding marks if you're interested in

    reading that I recommend taking a look

    at those sections which again I linked

    to this paper in the show notes now at

    the end of these unpacking scholarship

    episodes I like to share some lingering

    questions and thoughts just to kind of

    share here are some things that I was

    thinking of that maybe you'd be

    interested in exploring as well so one

    of the questions that I have is how do

    you know when a student understands

    something so an example that I've given

    in other podcast episodes is when I used

    to teach in a K-8 District that I was

    working in it's kind of like the mentor

    for the coding teachers in there one of

    the conversations that we had one day

    was looking at different student

    projects and so this was using Khan

    Academy's interactive coding platform

    which made it so that students were

    using JavaScript to be able to create

    art and animation one of the projects

    had students creating pizza so it's like

    a visual depiction of pizza like on a

    plate or on a dinner table or whatever

    students could kind of be creative with

    this and so they're creating with

    different shapes to represent the

    different toppings on the pizza

    different slices or maybe a full pie of

    pizza whatever now one of the examples

    that we looked at it had this really

    detailed there's like really neat

    depiction of what pizza might look like

    with code in terms of like how it looked

    it was like a standout compared to every

    single one of the other projects like if

    this project was a 10 all the other

    projects were like a seven however when

    we actually looked at the code it was

    very clear just looking at the code that

    the student didn't understand Loops as

    well as didn't understand some of the

    shape commands Etc so in terms of the

    overall product it was phenomenal but in

    terms of like the demonstration of

    understanding it actually demonstrated

    they didn't understand what they were

    doing and they kind of happened to have

    some happy accidents that made a really

    interesting looking project that didn't

    actually demonstrate that they knew what

    they were doing so it's important for

    researchers and for educators to really

    think through how you're actually going

    to check for understanding are you just

    going to look at the end result are you

    going to look at the process that they

    took to get that end result are you

    gonna look at the final code are you

    going to actually ask them even if their

    code looks great and you look at it

    you're like there's no flaws here like

    this is very efficient and effective use

    of code it looks great but do they

    understand it if you sit down with a

    student and you say can you explain to

    me what this function does can they do

    that but to argue with myself does being

    able to demonstrate something tacitly

    have more or less value than be able to

    explain how to do that very thing now

    this is something that I talked about in

    episode 89 which is titled applications

    of affinity space characteristics in

    computer science education and I've

    related it to musicians so do you want

    to be able to have it so that a musician

    can play a solo or be able to explain

    how to play the solo which one are you

    valuing or assessing for same thing with

    coding do you want to see that students

    can be able to actually create the thing

    or finish an app or a program or

    whatever or be able to explain how to do

    that where along the Continuum between

    the application of understanding and

    explanation of understanding do you want

    them to lie are they going to be

    somewhere right in the middle where they

    can do both or is it going to be leaning

    towards one or the other depending on

    how you answer this is going to kind of

    depend on what kind of assessment you

    end up using which I talked about in

    many other assessment related episodes

    that I'll link to in the show now

    another question that I have is how

    important is it to various stakeholders

    that students understand Cs and then how

    does that compare with motivation for

    students to want to continue learning

    something oh I asked this because

    there's like different community members

    or even like corporations or like

    administrators or whatever they all have

    kind of different like weights given

    towards the value of computer science

    for students to be able to learn but

    what's not often discussed is the

    motivation to actually continue to learn

    something so are we just trying to get a

    future population of Voters to a base

    level understanding of what computer

    science is or are we trying to motivate

    people to actually want to continue to

    learn computer science after the colors

    is done so is the focus on your six to

    nine to 12 month course and just making

    sure that students learn the content or

    the standards or whatever within that

    particular period of time or is it to

    develop lifelong Learners or for an

    extended period of time as I've talked

    about in other episodes like on serious

    Leisure are we going to encourage

    students to mod as I did did when

    proposing to my wife through Minecraft

    and engage in other leisurely Pursuits

    of computer science and whatnot or are

    we just focusing on developing citizens

    who understand computer science or

    developing future programmers Etc

    depending on what you focus on whether

    it's the understanding application of

    understanding or motivation to continue

    to learn it it's going to kind of like

    determine what kind of an approach you

    use in your classroom so something

    important to consider when designing

    classes what is your focus on is your

    focus on understanding is your focus on

    application of understanding or is it

    focus on developing lifelong learning or

    is a combination of those in different

    ways so I had a conversation with

    somebody earlier this week that person

    was talking about how their kid was

    having to label the parts of a

    microscope in class and they're like

    cool that's great I'm glad you're able

    to do that but what did you see when you

    looked through the microscope and the

    kid's like well we actually didn't even

    get to touch the microscope we were just

    labeling the parts of it that to me is

    just focusing on understanding like a

    set of Standards not actually

    investigating exploring and applying

    those understandings to figure out well

    what does like a cell structure actually

    look like when I look through this how

    do I actually zoom in and zoom out with

    the different parts of the microscope

    when would I even use a microscope

    instead we're focusing so much on just

    like rote labeling and understanding

    rather than applying the understanding

    and exploring or engaging in an inquiry

    or whatever now that approach would have

    in my opinion a profoundly negative

    impact on learning it's going to make it

    so students look at science and go this

    is boring it's like what my math and

    like history classes were like we were

    learning facts we were learning how to

    solve problems but we didn't know when

    we'd use that we didn't know why these

    facts were important it was just

    memorize this date for the test and then

    forget it after that as CS Educators we

    really need to make sure that like we

    learn from the mistakes that you may

    have experienced as a student and go

    well that didn't work for me or my peers

    how can I take that failure for myself

    and for my peers and actually make it so

    the students that I work with don't have

    to suffer through that in the classes

    that are designed Etc but another

    question that I posed and this was like

    in episode 66 which was titled the role

    of deliberate practice in the

    acquisition of expert performance is how

    do you develop expertise over time so I

    was previously in a k6 school and a K-8

    school but I've also taught high school

    so at minimum I would see students for a

    few years even when I taught

    undergraduate and graduate courses like

    I'd see them over multiple years

    multiple courses Etc it was very rarely

    what I only see students for like a year

    or a semester when thinking about those

    kinds of classes like in the K-8 classes

    it was nine years that I could

    potentially see those students trying to

    think of how to make it so that I could

    develop expertise from kindergarten all

    the way to nine years later when they

    got into eighth grade what did I want

    them to be able to know and be able to

    understand and be able to explore or

    create or Express themselves Etc across

    that nine-year period of Time how you

    develop expertise over that time is very

    different than focusing on a single

    class where I'm only going to see these

    students for a single quarter semester

    year or whatever if you're only focusing

    on that and you're not thinking of that

    long-term development of expertise then

    we're going to in my opinion shortchange

    the students rather than focusing on

    lifelong learning of computer science we

    would instead focus on just learning

    facts figures standards Etc without

    actually thinking about how to apply it

    over the course of a lifetime because

    computer science is just constantly

    changing and evolving I mean think of

    like the developments in AI within the

    last year alone this is going to

    continue and likely accelerate over time

    if all we do is focus on the concepts

    and practices and skills and

    understandings that are important today

    we are going to fail the students who

    are going to be out of the schools 10

    years from now instead I think in my

    opinion we should focus on developing

    lifelong expertise and teaching students

    how to develop that expertise when an

    adult is no longer present to help guide

    them the ability to be able to learn how

    to learn is more important than whatever

    facts and figures you can learn in Cross

    Your K-12 tenure same thing with like

    undergrad and graduate classes the way

    that I designed every class that I ever

    worked with kindergarten all the way

    through graduate student was constantly

    trying to focus on not just what they

    needed to know in that moment but how to

    also learn that down the road when I was

    no longer with them in those classes so

    a challenge for you listener is that

    when you are designing your classes

    think of not only what you're going to

    do within that quarter semester or year

    or however long you're working with your

    students but think about how you're

    going to encourage them to continue to

    develop their learning and refine their

    understanding at the conclusion of

    whatever class you are designing if we

    can do that then perhaps we're going to

    have some lifelong Learners who are

    interested in computer science whether

    or not they even decide to pursue a

    career in it I hope this episode was an

    interesting study for you to look at I

    know it was for me to consider the

    different comparisons between like

    text-based interactive and video based

    lessons I personally think we need more

    of this in computer science education

    research and just education research in

    general if you enjoyed this episode

    please consider sharing it with somebody

    else or leaving a review on whatever app

    you're listening to this on there's over

    phenomenal guests on this podcast as

    well as many other episodes that explore

    different topics and you can find a link

    to that in whatever app you're listening

    to this on or just by going to

    jaredoleery.com stay tuned next week for

    another episode till then I hope you're

    all staying safe and are having a

    wonderful week

Article

Bao, Y. & Hosseini, H. (2023). Mind the Gap: The Illusion of Skill Acquisition in Computational Thinking. Proceedings of the 2023 ACM SIGCSE Technical Symposium on Computer Science Education - SIGCSE ’23, 778-784.


Abstract

“With the advent of online educational platforms and the advances in pedagogical technologies, self-directed learning has emerged as one of the most popular modes of learning. Distance education— elevated by the COVID-19 pandemic—involves methods of instruction through a variety of remote activities which often rely on educational videos for mastery. In the absence of direct student engagement, the asynchronous nature of remote activities may deteriorate the quality of education for learners. Students often have an illusion of skill acquisition after watching videos, which results in overestimation of abilities and skills. We focus on the efficacy of skill acquisition through interactive technologies and assess their impact on computational thinking in comparison with delivery through other traditional media (e.g. videos and texts). In particular, we investigate the relationship between actual learning, perception of learning, and learners’ confidence in adult learners. Our results reveal intriguing observations about the role of interactivity and visualization and their implications on the pedagogical design for self-directed learning modules.”


Author Keywords

Skill acquisition, interactive learning, computational thinking


My One Sentence Summary

This article compares learning, perceptions of learning, and confidence among adult learners participating in interactive, video-based, and text-based learning.


Some Of My Lingering Questions/Thoughts

  • How do you know when a student understands something?

  • How important is it to various stakeholders that students understand CS?

    • How does that compare with motivation for students to want to continue learning something?

  • How do you develop expertise over time?


Resources/Links Relevant to This Episode



More Content