Accessible CS Education through Evidence-based Programming Languages with Andreas Stefik
In this interview with Andreas Stefik, we discuss the importance of using evidence-based programming languages, problems with the lack of replication in CS education scholarship and academia in general, the importance of designing for accessibility and disabilities, lessons learned designing Quorum (an accessible programming language and platform), and much more.
-
Everyone I just want to give a very
quick disclaimer before we started the
episode that this episode was recorded
in January of 2020 now the reason why I
say that is because at one point in the
episode I mentioned doing weekly
Japanese lessons at my house and that
was accurate months ago but it is no
longer accurate doo doo Cove in 19 so
just FYI we are practicing social
distancing and all that anyways I hope
you really enjoy this episode and let's
get it started
welcome back to another episode of the
CSK8 podcast my name is Jared O'Leary in
this week's episode I'm interviewing
andreas tefik in our discussion we
talked about the importance of using
evidence-based programming languages in
particular we talked about it in
relation to designing for accessibility
for kids with disabilities we also
talked about some of the problems with
the lack of replication in CS education
scholarship as well as some of the
problems in academia in general in
addition we discuss an evidence-based
programming language that stefa helped
create that's called quorum as well as
many other topics relevant to CS
educators there are a good amount of
show notes that link to many of the
papers that we've discussed as well as
some other music related concepts that
we discussed at the beginning and
towards the end of the interview as well
as many resources that can help with
designing for accessibility in your cs
classroom you can find those show notes
by clicking the link in the app that
you're listening to this on or by
visiting jared O'Leary comm and we will
now begin this entertaining and
informative interview with an
introduction by steph ik my name is
andreas tefik almost everyone calls me
Steph ik I'm a computer scientist that
works at the University of Nevada Las
Vegas I do a lot of work related to two
major things one is investigating how
people invent things in software sort of
investigating programming languages but
I also do a considerable amount of work
related to how people with disabilities
do computer science I had meant a lot of
stuff in that direction can you tell me
the story of how you got into CS
education and research when I was an
undergraduate I was doing a degree in
music I've played in bands for years and
stuff I was never really very good at it
but I enjoyed it you know it was kind of
fun like hobby type of thing and I was
reading in textbooks at school on this
composer named Ben Johnston he did a lot
of music that's called microtonal it's
kind of like keys between the keys type
of stuff frequency bands not on the
piano that could be used as notes that
always sounds wacky when you say it
actually there's really good reasons for
it that's because some notes on the
piano would actually be more in tune and
others are actually less in tune and you
can figure that out using some pretty
simple math and then okay so as an
undergrad I moved to the East Coast to
work with this guy and when I was there
he was using a lot of music notations to
represent this kind of music inevitably
kind of complicated that were really
hard to understand and there was a
string quartet called the Keppler
quartet that wanted to play the music
but they couldn't figure out what it
would sound like based on the notation
so the problem was the notation was
really hard to understand they wanted to
play it but they really couldn't do it
so this guy Ben Johnston he sort of
asked me when I left the training is he
if I could figure out an alternative
that would make it easier for groups to
play as a notation so I started taking
classes in computer science thinking
maybe I could write a computer program
or something or do some kind of math or
some funny thing to make it easier and
it took a couple years and by the time I
figured it out I had a computer science
degree eventually though a number of
albums were produced on this and what my
graduate school I had already done all
this stuff related to notation to try to
make it easier to understand in addition
I've been doing music for a since
forever so I was trying to think of
projects related to who benefits from
sound based technologies in computer
science so I started talking to schools
for the blind and just talking to the
kids hey kid have you ever written a
program before do you want to try and
then kind of watching them and
inevitably watching them struggle
because doing programming when you're
blind using off-the-shelf technology can
be a pretty serious challenge and
everything kind of snowballed from there
in terms of my interest into computer
science education now it was the
notation that he was writing was it
aleatory so like chance based or was it
no this is a very specific note that i
want you to play but it's in between
what's typically the white and black
keys it's the second one so okay you
think you like John Cage's style
notations yeah it's more like graphical
where it's kind of like up to the
artists interpretation like how do I
play this thing
yeah actually funny if I were correctly
I think John Cage lived in Ben's attic I
think that's true
in any case no wasn't it wasn't
Elliott's work but he was influenced by
that kind of stuff so there are some
things that kind of sound chanting his
music but it's very specified so for
example if you had a particular note
combination you actually specified
mathematically the relationship between
the two notes so like d2a is exactly the
frequency of D multiplied by 3/2 right
and that's not a king on the keyboard
right but it is notation that would end
up with a certain set of symbols that
represent that frequency relationship
not the normal keyboard and it has all
these weird names like sin tonic commas
and stuff like that and it gets kind of
wacky but what's interesting about is it
actually explains when you're thinking
about choirs and stuff why fighters tend
to go flat and sharp over time because
the frequency bands on piano don't
actually match what would be in tune so
it kind of like is somewhat explanatory
sort of asking yeah and if this is too
abstract for people who don't have a
background in music I'll include some
resources in the show notes that kind of
explain the different types of tunings
and whatnot that you can find could you
elaborate on what you mentioned with
watching a blind student trying code
what did that look like in terms of your
AHA's and going oh this is not at all
how somebody who is able to see it uses
this program I had been primed with the
challenges related and notations I kind
of mentioned now it's totally different
kind of notation mm-hmm but nonetheless
that was clear in my head so I called
the Washington state school for the
blind and was like hey I want to do
something together and then eventually
after there's some grants and some stuff
like that I went we started observing
kids and so one of the very first things
we did we did some experiments where we
had kids that age try different
programming languages and literally just
try and like write a loop right it is
just simple little programs and because
they come out through screen readers
there's weird things that happen right
at least weird if you're not a screen
reader user which I usually explain
because most people aren't and so I'll
give an example you write a loop in C++
you can see it visually if you know the
code what sounds like is or left paren
int I equals 0 semicolon I less than 10
semicolon I plus plus right paren
imagine hearing that is a
fourteen-year-old kid three times faster
than what I just said and not knowing
what a single symbol meant right also
that's tricky and then if you have an
error in your code you're gonna have a
compiler error back now in in an IDE
development environment you might have
like a little visual display on the
bottom of the screen and your eyes
you're naturally skip certain things so
like microsoft visual studio at the time
you would have like all sorts of weird
error codes colons funny slashes and
stuff like that and when i visually look
at it with my eyes i skip stuff right
i'm not gonna read underscore underscore
c CDEC l colon colon four or five eight
five you know that stuff is just useless
right but if you're listening with a
screen reader guess what every character
gets read so you got a compiler error
that compiler might last a minute in
audio and the actual thing you did wrong
like oh you missed a semicolon is like
halfway through so it's real easy to
miss so we're sitting there watching
this and I was like man there's no way
that someone like maliciously designed
it this way I bet just no one cared and
even thought about for a moment whether
this would be the design for somebody
that's using it screaming oh that smells
like an opportunity and we kind of
jumped on it so how's your research in
accessible computing kind of informed
your own practices as an educator and a
scholar one of the first things that we
thought was when we're watching blind
students well babe this is probably just
a gladness issue right I skip over that
stuff I don't need it or maybe for left
her and all that kind of stuff it
doesn't really matter but it turns out
that we were suspicious that that was
probably wrong why is it the programmers
designers use the word for you know and
it's not just like C++ there's kids
languages to pencil code uses the word
for that's a block language it doesn't
really matter like lots of language
designers made decisions about how they
represent their code I wonder you know
what is the data well you know when C++
was made like what kind of evidence did
they use to make their decisions were
like you know pythons much newer
what is the evidence on various
decisions that they made in Python
what's the evidence on the whitespace
about the colons all the other funny
things and so I like kind of naively
thought well let's just go look it up
and gather all the data
designers been gathering and try to
figure out like what was the right
answer so that maybe we can make it a
little easier for these kids what I
wasn't expecting well is that there
wasn't any data and that which designers
have in fact emphatically been against
gathering data and I'm not kidding about
that for a long time and this is
actually this intuition that like this
sort of like it's not part of their
community gathering evidence is really
weird sounding so people that like
aren't used to this idea that scientific
communities also have their flaws and
imperfections this has actually been
formalized by a number of scholars one
of them is there's a guy named Adi Hani
kiona who is this Finnish scholar and he
actually analyzed effectively every
paper from like 1950 till 2012 this is
like peer-reviewed top conferences made
decisions about National Science
Foundation support all sorts of stuff
and it turns out between that entire 60
year time period there was 22
experiments and that is nuts absolutely
nuts like that just should not be true
when people are designing programming
languages there's two parts there's the
technical you got to make sure their
works and then people have to use the
damn thing that's a fact we all do it
and developers get paid an enormous
amount of money nationwide or worldwide
to use these things hundreds of billions
of dollars a year gets paid to devs so
tiny little impacts from productivity
impact billions of dollars national it's
a big deal and is that including even
just a quasi-experimental or yeah so
there's whenever you talk about
gathering data how you gather it depends
a lot on the context under which you're
gathering so for example the word to use
quasi-experimental often means that you
lack certain methodological controls in
an experiment but sometimes you have no
choice you actually have to do that for
example I can't give half of the country
cancer right that doesn't make sense I
don't think it passed IRB either yeah
like just plausible right just on a
practical level in education this is a
big deal too because even though it's
much less extreme I can't take kids and
randomly assign them to schools that are
at various levels of poverty I can't we
assign them to new parents or stuff like
that so there's always an element of in
the field you have to make certain
compromise
when you're conducting studies and
there's no choice about that even if you
really wanted to it would be impossible
to do otherwise on the other hand is
full experiments and there's surveys and
stuff like that too full experiments
where you do have custom controls those
have value to you just use them for
different kinds of things when you're
looking for evidence so we kind of do a
combination things and that's really
common
yeah and in education for people who
don't have a background in research it's
particularly difficult to do full
experimental studies one it tends to
make things decontextualize like you
might be going to a lab setting rather
than a social setting where kids are
learning but the quasi side of things
you have a lot of factors and variables
that you can't control for because
you're in a live setting where kids are
learning not just in the environment
that you are testing and yeah so I think
that the model that I like the most and
you may you may not like this particular
model I don't know there's a very famous
scholar named Austin Bradford Hill you
heard this guy mm-hm
you know that the phrase that smoking
causes cancer you know this yeah I mean
it's common commonly known not everybody
buys it I guess but the evidence is very
clear right
that's awesome Bradford Hill he was
basically the first caller to sort of
show pretty definitive evidence that
smoking not is correlated with cancer
that might be related to cancer but no
no it causes cancer right and the way
that that worked what's called Medical
randomized controlled trial now when
people say that word randomized
controlled trial they think it's a
custom experiment that just has very
careful controls and all sorts of stuff
like that but actually since Bradford
Hill experiments even in other
disciplines where you have these
constraints just like you do in computer
science education they use a combination
of what can we control and what can't we
and then how do we correlate with other
factors so I'll give you an example in
some of the original studies they would
do things like okay I know that I can't
give people the cancer and I know I
can't control their smoking habits so I
can't control for that but I can't
control for certain other kinds of
things and so they basically designed
these studies where they had control
groups and they did it for certain
things that they cared about that they
can manipulate and then there was other
things that they basically just tracked
anything related to it so that they
could correlate it in and rule out
alternative explanations so when
conduct studies although nothing's
perfect you have to just kind of do the
best you can we always try to follow the
Bradford Hill model where it's like
control what we can control and then
correlate in any other factor that we
can switch you least clues about
causality yeah and then I'm assuming
doing follow-up studies to kind of like
Rivera fie in different scenarios and
with different participants and whatnot
he'll upstairs replications want other
groups to do it in a different way you
want people to bury the experiments
science is complicated and you want all
that stuff one of the guests that I
spoke with had mentioned that that was
one of the biggest things that she
wished there was more of was people of
following up and doing studies and with
different sets of people and whatnot
just to kind of find out okay well we
thought this thing but now when we
applied it in this different context we
ended up with a different result and so
let's explore that and why in general
not a lot of replication going on and in
the field of education yeah and there's
a couple reasons why that is actually
and one of them is that especially in
computer science education it sounds
really weird to say but we actually have
no evidence standard now what does that
mean well actually it turns out there's
a formal thing called an evidence
standard it's not like a term I'm making
up it's a like a standard term evidence
standard and what it means is that when
you publish papers there's certain
protocols you follow for how you report
experiments now think about this if I'm
in the medical journals they actually
follow and called consort Co NS ort
consort is a medical standard that you
have to follow if you publish at certain
journals and it basically means that you
have a replication package you have
certain things you report you report
demographics you report how the study
was done that kind of stuff and the
reason that matters is because anybody
that wants to replicate it really needs
that information able to do it in
education there is a evidence standard
called a what works Clearinghouse it's
much more complicated and that's a whole
other topic of discussion but it's not
really the same and what do you talk
about computer science journals the
evidence center that uses nothing people
just kind of publish whatever they want
so if you want to replicate other
people's work sometimes it's almost
impossible because you really don't have
the information you need let alone like
a replication package and stuff like
that so it's really hot because we are
Journal
having kind of gotten on the bandwagon
yet for evidence standards and therefore
were really high
yeah and I've done some reviews for the
Department of Ed and then applied for
some grants that required what works
Clearinghouse and with a problem that we
were running into was there weren't any
studies related to CS education that we
could cite so what we ended up doing was
saying well here's some professional
development stuff that's in alignment
with what we're doing but it's in a
different content area so it's kind of
like this but not really I a hundred
percent understand exactly what you're
saying that's completely true because we
don't follow the standards then it is
what it is and it's tricky to computer
science education is also an
international community it's not just
Americans but the what works
Clearinghouse only covers Americans so
it's sort of funny like we almost in a
way want like an international evidence
standard that covers everybody cuz like
are the kids in Finland the same as
United States
maybe but like maybe not and in fact
there's some evidence to suggest that
that's not true there's a woman named
Felina Herman's that does really
fantastic computer science education
work I think she's in the Netherlands
and like they speak a different language
and the programming languages are in
English what are the challenges there
well they're not zero right they're
complicated you know but yeah that's not
gonna make it into the what works
Clearinghouse even if it's all work
right yeah hopefully in a couple of
years that'll change for now it's just
frustrating frustrating it's so much
easier to read studies that are in an
evidence standard like you look up study
on PubMed that follows concert you know
where he is like oh you want to look at
demographics look it's in the section
with that exact name every single time
there's no debate it's just simple yeah
for my dissertation I tried to make it
as replicable as possible so it was a
corpus assisted discourse analysis so I
looked at like 11 million words of data
and try to find patterns of discourse
and then what does this mean for
education so like in my methods chapter
in chapter 3 it was just all about okay
if somebody else wanted to be able to
redo this study what do I need to
include in there so there's very
conscious of that so hopefully people
could redo it if they wanted to oh
that's lovely ten years ago we had
replication packets but they were all
kind of like we didn't really know what
they should be and stuff nowadays
replication packets are download this on
get
and humans right right right
and so it's a lot at least theoretically
a lot easier to sort of rub lick the
studies now does anybody actually but
it's possible and that's the first step
yeah then there's also the issue of well
what counts in terms of publication and
for the institutions that you're at like
are you gonna get as many citations if
you're the one doing the follow-up study
or if you're doing the initial study
that people are then going to try and
replicate that's a good point I mean
like we obsess about these ridiculous
things like h-index and citation right I
mean they're not totally dumb like we do
care about it to an extent of like you
know you want your work to be used by
other people but it misses two
fundamental things like these kind of
weird metrics that were obsessed with an
academia number one is all the metrics
related to citation counts and paper
writing and stuff they usually emphasize
that you should be writing right like
your h-index how do you get your h-index
up you write more papers and then you
write papers in popular areas that's it
right that's the the most likely way to
do it so pick a big community and then
write lots of papers to that community
to increase your age index
however we forget that as computer
scientists we also can invent things and
that has value so for example we
invented this programming language
called quorum and it's used last year
but I think like eighty one thousand
people right now that's a metric but at
the same time like if I spend time
writing that application I'm actually
not spending time writing which might
look bad from other academics
perspective but honestly I don't care
because I think that has more tangible
impact on society than just obsessively
working on writing so it feels like we
need as a culture of academics you need
a better balance as to like how we think
about what we choose to do in our own
jobs so well and it rewards and kind of
punishes some approaches over others so
for instance survey really easy to send
that stuff out and to get data back but
doing a longitudinal ethnographic study
on a particular culture or set of people
or whatever that's going to take much
more time and it might be multiple years
until you can actually start publishing
on the actual findings from those kind
of studies one hundred percent the first
big study that we did
programming language design was in 2013
and it took my student and I for years
to put that study together now like
that's not what was me it took a long
time that's because we wanted to know
the answer right that mean that like I
could have been doing smaller studies or
split them up into like little mini
papers and like just so stupid it's just
a waste of time it's easier for the
research community if you have one
resource that's well vetted over a long
period of time and lots of little mini
pubs that get tossed out all the time
sort of I know I just feel like we waste
because we're so obsessed with these
metrics it also wastes our time because
we end up reading one the next increment
on the same paper which is like you know
a three percent improvement bomb is
algorithm or something that of like
waiting a couple years in getting a big
foundational advance so I'm curious to
what's something that you believed about
accessibility in particular when you
first began but you no longer believe
now that you've done some research in
that area when I first started working
in accessibility I was working only with
blind people now that's not true anymore
we work with deaf people by people
people with mobility impairments
learning disabilities all sorts of stuff
and I loved it I loved every minute of
it but when I first started out working
only with white people I didn't have
context for all these other groups yet
and so I sort of naively I think assumed
that when you add support for one group
for accessibility that that mostly
impacts that group but I know with
certainty that that is not true I'll
give you a few examples if you have
people that are blind or visually
impaired they use the keyboard very
heavily because you can't see the screen
you wouldn't use a mouse that makes no
sense right I mean some people that are
blind can also see that's complicated
but never mind that I even have a
student that is blind legally blind that
can drive but never mind depends on the
state and since they're using the
keyboard heavily because they're very
used to it if the keyboard shortcuts are
really nasty it will get very annoying
very fast so if you're blind it's
important to be able to remap keys to
sort of whatever is kind of convenient
to make sure they follow standard so
conflict with screenreader keys and so
like that however if you have a mobility
impairment you must be able to remap the
keys like you just have to otherwise you
really can't use the product in addition
you might think that if you have closed
captioning for a video
that that only impacts the death but
actually that's not true at all and it
impacts parents if you have closed
captioning and you want to turn the
sound off because your little kids are
asleep you might want to read or if
you're in a busy hotel or something at
an airport or whatever it is in addition
people that are blind often download
closed captions so they can read the
transcript it's our screen reader at
higher speed because waiting for the
video and something be very slow and
there might be visual things on screen
that like don't get shown except baby
and the captions or somewhere else
interesting yeah so a lot of the
technologies that you use that are
related to disabilities often get used
not just by that group but by other
disabilities groups that end up having
the same reliance but also like groups
that you wouldn't think of like a parent
that just has a screaming kid or like
you know someone in a wheelchair that
has a curb cut that they go up well
that's gonna use by a woman with a
stroller too that has a child so I think
that like even though I'd heard these
phrases like curb cuts before when I was
doing the accessibility research I think
that that message didn't really hit home
until later when I was sitting watching
different kinds of groups like watching
a kid with dyslexia use a screen reader
and you know things like that so what is
something that you wish more CS
educators kind of understood about
accessible computing there's really two
things that I often think about that
went with accessible computing for
computer science educators number one I
wish more people realized that it's
required by law like so for example if
you're developing a tool you're making
blocks or something like that right you
are putting yourself at legal liability
if you don't add accessibility support
in K through 12 that's never fun to say
that's like the not nice thing about
accessibility but actually there's laws
protecting people with disabilities
because they've been horribly
discriminated against historically like
for example if someone said I'm gonna
have a child with darker skin excluded
from this computer science curriculum
all of us would flip out right that's
just unacceptable
or if you said I don't think women
should be programmers we would say screw
you that's just completely unacceptable
that's just not not not not right but if
almost every single
language out there is inaccessible to
the blind we sort of throw our hands up
and say well it is what it is so number
one the not nice thing about
accessibility but I wish people knew
more it's required by law it is not
optional in K through 12 every child has
the legal right to do education as
States increasingly make computer
science mandatory people are gonna have
to cope with that but the second thing
that I think people that is much nicer
about accessibility is that like people
with disabilities often have
observations about how they invent
things that are super fascinating you
know like watch I would not have
realized that you can make progress on a
problem as complex as the programming
language Wars like how you know these
languages vary and whether they're easy
or hard has been a matter of debate for
a long time until I started watching
blind children I just want to notice so
I didn't think about it I'm so used to
you know these looping structures and
classes almost kind of stuff I wouldn't
thought about it so oftentimes when you
work with people with disabilities you
get the big advantage that you get to
observe things that many of your
colleagues in computer science education
would not have seen right you get to see
them like oh trying to program with
audio and seeing all the shortcuts they
press or you get to see someone with
deafness trying to interpret some really
bad captions in a funny way or looking
at the language differences people that
are deaf have in comparison to people
that weren't using ASL for most of my
life or watching someone with a mobility
impairment trying to get around some
particular feature that they're using
I mean it's fascinating and has a lot of
opportunities for making things easier
for everybody and like I feel lucky that
I get to kind of sit and observe these
people all the time that they kind of
allow me to do that because it gives me
a lot of fun insights that I get to come
back from playing it yeah I think that
really speaks to having diverse teams
and having diverse users whether it be
in beta testing or product users or
whatever I was at WWDC and was at 2016
and paraphrasing here one of the things
that they had announced at the
conference was that their approach was
focusing on designing for the margins so
that way you can include everybody
whereas if you design for the
average then you're excluding the
margins and whatnot and that really kind
of hit home for me or it's just like oh
yeah that makes a whole lot of sense and
they showcase it by having like blind
DJ's go up and perform and like things
like that just like little things that
are like okay this is really cool and
it's nice to see that one of the largest
companies in the world is really pushing
for making things more accessible for
people I think Apple has put in a lot
more work than a lot of groups and I
think it's gotten better over the years
nothing's perfect at any company and
even even with really good intentions
and stuff like that sometimes the
challenges can be hard to make something
accessible it's it's not like you flip a
switch it can be difficult right so
speaking of that what might you
recommend for CS educators who are kind
of designing either like curricula or
educational experiences what would you
have them consider to become more
accessible with the experiences or
curricula that they use yeah so there's
a few things number one is for the
curriculum itself when it comes to like
you know the text that you write and
stuff like that always at least do the
basics basically understand that web
browsers do a pretty good job for
accessibility they're really not that
bad but you have to not break it so like
I'll give you some examples of ways that
you could break it usually not
maliciously that are coming dorable that
I think of funny so my daughter right
now is at the time of this recording is
seven years old and she goes to a local
local school and at their local school
they have a calendar that like just kind
of says their events and stuff like that
I'm like it's clear they don't have an
expert putting their website together
which is normal it's a grade school
right right okay and this is so adorable
they actually have like it's like a I
don't know if it's like a spreadsheet 11
they like take a picture of it and then
they put up the picture that's it and
like as a computer scientist I look at
that I'm like oh dear II that's just not
that malicious they're just like trying
to put the schedule for everybody no you
don't do stuff like that like don't like
take pictures of text and just put it up
with an aunt or something like that's
just a really bad idea so use text
normally and that kind of thing and the
second thing is when you do have
pictures there's nothing wrong with that
but there's little tags you have to put
in or call alts that give you
descriptions
and those kind of things are the really
like the bare-bones basics of making
sure that stuff is accessible they don't
make your site accessible they don't
make your tools accessible or nothing
like that but it's like if you don't do
at least those things things are
probably broken for people I want people
to do more is to start making sure that
when you're writing grants and when
you're using tools make sure everything
in your tool chain is accessible and
just like if something in your chain
isn't accessible drop it get rid of it
use another technology that is
accessible because it is the law and
also because it's the right thing to do
you know and there are challenges there
like I think that we can make block
language is accessible or time I think
that will be solved I think that that's
not like a huge technical burden I think
that's okay and I also think that there
are technical challenges that people
have as they're doing stuff you know
like if you're making a curriculum
related to Android
well Android naturally has some nasty
accessibility stuff in it because the
Android accessibility architecture is
really not that it has issues right and
then vice versa like accessibility for
every platform it turns out is very
different so apples accessibility
architecture is 100% incompatible with
the Windows one which is 100%
incompatible with the Android one which
is 100% incompatible with the web so
cool great job so to a certain extent I
feel empathy for many curriculum
designers because they're like well I
really want to use platform X I don't
know about accessibility on that
platform and it's complicated and I need
help so the one other thing I would
mention if you are someone that does
curriculum and you don't have the
expertise to do it reach out to those of
us that do because we can totally help
you we have grants to help you and we
have like thoughts about it we can go
through it with you and talk to you
about it and go through some of that
stuff it can be complicated sometimes
depending on the platform but that's
reality of the nature of accessibility
today so speaking of platforms you had
mentioned quorum already I'm wondering
if you could describe what quorum is for
people who aren't familiar with it yeah
so quorum is a programming language so I
mentioned this issue of like evidence
with languages so my lab and I thought
well if you were to try to make a
programming product based on evidence
what would that even mean and what would
that be
so quorum has become a programming
language where we meticulously and
rigorously and sometimes even
obsessively go through every symbol
every character every combination of
semantics and whatever with human
studies to try to make it easier to
understand there's there's two pieces of
it one of which is the language itself
just like the raw text and all the
libraries and all that kind of stuff and
we evaluate that in different kinds of
ways to making things here but another
is very recently we've been working
toward visualization of the technology
oh this is really hard to make
accessible so it's taken us a couple
years to do it
for example are you familiar with
unity3d bourbonnais mm-hmm so for
anybody listening that hasn't unit III
is like a programming environment that
involves 2d and 3d graphics well how do
you make that accessible
well we've solved that and we have a
version that's totally blind accessible
and other disabilities that allows you
to do 2d and 3d gaming and all sorts of
cool stuff inside and porom that will
come out in July but in any case if you
want things like blocks or
visualizations on top of programming
languages to be accessible you need to
do that so when we eventually go in and
add that kind of support to quorum we
have to do the accessibility side first
which so it takes a little while but for
the language really what it is it's a
programming language where a group of
scholars not just me but people outside
of my research group haven't been
obsessively running studies on anything
you can think of to try to make the
language easier to understand if you
have loops in a normal language that
would be this for left paren int I
equals 0 semicolon type stuff in quorum
it would be repeat ten times and why is
it those specific words is because we
have survey data on the word choices we
have randomized control trials comparing
it to other programming languages and we
often compare those across groups
probably the most famous one is our
placebo study I don't know if you know
that one No so in 2013 we had this idea
for a study we're trying to think of
like how would you compare programming
languages and what would be the line
like you know for example in the medical
sciences they often use placebos which
are things like sugar pills and then we
compare against placebo to kind of give
like a comparison across a set of
medicines like if a placebo works in a
certain percentage of the time or works
is a rough word but you get mind
then you're comparing across but for
programming language what's a placebo
right so what we did was we took Orem we
ripped out all the symbols and then I
had my student mostly laughing rolling
electronic dice and randomly choosing
the symbols that go in the programming
language so like they're awful they're
absolutely horrific Liebeck does not
repeat it's like left slash so you got
like pound colon colon left brace for no
reason and then like a axon and students
just have to figure out that means
repeat 10 times right so very hard to
use and you might think cool we got a
baseline but that seems like
ridiculously unfair until we ran the
study it turns out Perl and Java don't
beat it really and when I say don't beat
don't be in statistics usually means two
things one is it it means it doesn't
find a statistically significant
difference between that in the group but
also crucially it means that even if
there is a difference and you can find
it at larger sample sizes it means that
it's inevitably pretty small the
question that is why on earth would that
be true and the hypothesis that we
generally have is that the word choices
suck yeah the word for doesn't mean
anything to a novice and a lot of these
little braces they have weird effects
that like we wouldn't have expected like
for example if you have the left brace
instead of a programming language like
usually it's match braces turns out and
offices get the left brace wrong more
often than the right one why I don't
know but thank you and so often times in
programming languages when we design
quorum we use that statistical data and
then we make directed decisions about
which things we leave in and out so for
example an if statement in quorum
doesn't have matching parentheses
because it's one of the most common
errors and this has been shown in a ton
of studies since but our data shudders
as well in 2013
Neil Brown's data in UK shows it Brent
Becker's data shows it pulled Denny's
data shows it I mean it's just the
general consensus from a variety of
programming language so that's true so
we leave it out if the left race is more
likely to be screwed up into something
like a function than the right the left
race goes away and it turns out that
works pretty well or like if you have an
equal sign like in programming languages
often it's double equal to imply
equality equivalent it
yeah and it turns out that if you have a
single equal it increases novices
abilities to use that in the context of
statement by 800% well yeah I know
that's what I said when we saw the data
too and in fact in the studies we then
going to use assignment and equality
test right next to each other to see if
we can screw up a novice and it doesn't
happen like novices can figure it out
based on context the point is like you
can use all that very strategically to
define programming languages where
you've really thought it through you've
gathered the data you've thought it
through done experiments you've
replicated those experiments your survey
data on the words and symbols and then
you put that all in so that people can
kind of use it a little bit more
effective and then every release we do
on quorum we work with outside scholars
that are doing different studies
sometimes to disprove this and sometimes
they're right in addition we keep doing
internal studies as well to make slow
incremental changes so that hopefully in
like 10 20 50 100 years programming
languages are much more condensed in
terms of how much garbage is it because
right now there's a lot of garbage right
I'm curious in the Placido study so I
proposed to my wife by modifying the
game minecraft which used Java so I've
used that but I haven't used Perl I'm
wondering if you found significant
differences between lower level versus
higher level languages compared to the
placebo right so we wouldn't up that we
didn't test that in that study but the
reason the most perform badly is
actually because they have a lot of
syntax in common okay the reason where
they tend to screw up is actually the
same across where they're the same and
in fact in languages that did slightly
better than placebo like Python did
slightly better it actually where Python
does badly is the same places where it
matches the syntax for Python and Perl
and the reason is they use the same
words and symbols right it just has a
colon at the end instead of a brace and
a match and stuff like that so the
reason why Python does slightly better
is actually accidental they still have a
lot of the cruft
from you know the 60s and 70s that
didn't get rid of but they have a couple
decisions that made a small difference
and in fact in those original studies
we're better than chance but it did
pretty badly too in a lot of ways and we
made a ton of changes since 2013 that
are based on that evidence from that
study and many others that's great
yeah in my coding classes the kids had
the opportunity to pick from several
different languages and one of the
things that I noticed is it was easier
for kids to go from a syntax heavy
language to a syntax light language like
JavaScript to Ruby but going the other
way around it was much more difficult
because they were used to this like
clear easy to understand click to right
didn't have to worry about semicolons
and then when you go to JavaScript it
was like wait I have to add in all this
other crud that I'm not used to so they
were not able to do as well with that
yeah I don't have evidence on transfer
of learning in both directions but that
might be true that there is different
effects going either way that wouldn't
surprise me it's interesting you
mentioned Ruby too because a lot of what
we found in those original studies as to
how to make poram better we actually
stole from Ruby because they had some
parts of the language where they did
really well like leaving off the
parentheses and stuff and that did
really well in those studies on the
other hand interestingly because the
whole field was non evidence base for
such a long time most languages make
certain mistakes that they probably
don't know they're making Ruby's a good
example ruby has dynamic typing so it's
JavaScript and it turns out dynamic
typing which is this idea that I don't
tell you what the type is I don't force
you to write it well it turns out that
in languages like Ruby because they have
dynamic typing that actually
significantly lowers your productivity
for people I think it's past years three
of experience level in college at the
beginning students that do have a
negative impact and it's like really
small but it goes away quickly and we
even know why it happens the reason is
because if you videotape programmers
basically what you observe is that when
it's dynamically typed if you want to
call a function you got to know what to
call it with the properties go with the
past so what you observe is developers
jump to the function try to figure out
what to pass to it and then they jump
back and guess what that takes time and
therefore productivity lost and to
hopefully clarify for people who aren't
familiar with it so like one example is
Khan Academy's JavaScript they use var
is a substitute for any kind of type of
data but instead of using something like
var you might say int for integer or dub
for double or say a string or whatever
to clarify in advance I'm going to pass
text or I'm going to pass an integer
that has decimal points or it doesn't
have decimal points etc yeah that's
exactly right
turns out that var type approach
actually lowers prickles productivity a
substantial amount that is absolutely
fascinating I think actually now that
effect of dynamic for second I think
it's actually the most well studied and
replicated in probably home computer
science off the top of my head I want to
say somewhere between ten and twenty
randomized controlled trials on that
topic now and they pretty consistently
show the same effect under different
context without code completion in and
out of an IDE under this condition that
condition is kind of funny like it used
to be we used to debate that in computer
science a lot I feel like that's a
solved problem now I mean there's more
to learn and there's probably contexts
under which you know so I have to put
mice and scientists head on for a minute
and say there are caveats and there are
but at the same time like if you want to
say the other direction now you got to
provide the data and show us the
evidence so we can see what conditions
you're testing I'm wondering going back
to quorum can you describe so I was on
the the webinar that you did a couple
weeks ago so I'm wondering if you could
describe in particular the positionality
of the camera that was particularly
fascinating to me how you had to think
of when looking at an object how close
or far away female you are from that
object with the distances and then even
possibly discuss how you sonna phi an
environment or a landscape by like kind
of panning through it and playing some
of the different sounds to get a picture
of exactly what is displayed on the
screen
yeah okay so just to provide anybody
that's reading some context if they
didn't hear that webinar and we'll have
a link to that at some point I'll put
that in the show notes whatever it goes
up in any case so it's right context of
people listening what we've invented is
we've been trying to figure out a way
that we can make it so that kids that
have disabilities but even including
blindness can create games that are
highly visual in nature and there's a
couple reasons why we want to do that I
always have to say this first just
because that sounds like why would you
do that of course a blind child's not
going to make a 3d game that doesn't
make any sense but actually there's got
the reasons and one is again in high
school if a kid is in a class with 3d
programming it's illegal to exclude them
if it's required that they take the
class you can't do that so that's number
one but number two is well like I just
got this hunch that if we can make that
accessible we're gonna learn all sorts
of weird neat interesting things but so
far what
built is we have an environment and
we've tricked it to be able to be usable
using technologies like screen readers
which talk to you and give you sounds
related to what's in an environment now
how that works on a technical level is
ridiculously complicated and unless you
want me to I won't go into it but the
short and skinny is that trying to make
animation loops all this kind of stuff
talk to screen readers is challenging it
took us years to figure out how to do it
in such a way that it would work
seamlessly so what we have now is that
we have like cameras on the screen that
that's are in a 3d environment there's
all sorts of 3d components like boxes or
spheres or things like that so the
question that I think you're getting at
jared is like how does that work in
terms of like what the user interface is
so like you know if I have a camera and
I'm looking at a 3d landscape well how
do i how do i describe that and i think
what we've learned so far in limited
user testing and by trying it ourselves
with screen readers because we haven't
finished our beta tests yet with actual
blind users or other disabilities is
that so far as far as we can tell the
most crucial thing that you want when
you're talking about making 3d things
accessible is actually not the gestalt
like trying to describe the whole scene
at once it's being able to know your
context which is classically called -
where am i problem in disabilities where
it's sort of like where do I exist in
this environment but in addition being
able to interact with the things that
you're working so I'll give you an
example if I'm like building a 3d model
of an ogre and I'm blind and I can't see
it I might want to know whether the ogre
which is just one model in a giant
landscape has the focus and I might want
to know that as a whole but once I'm on
the model I might want to know things
like how many parts does this ogre have
at a high level it might be the arms and
the legs and the head so I want to be
able to query that and start to break it
down and then I want to be able to
navigate those things so that I can get
information about what those are and
then I can start to drill down
mathematically or in some other way so
now that's not always easy like if you
had a castle and a head let's just say
ten million triangles you need to
basically figure out how to combine that
down into higher-level items so that the
and then allow drill downs so that you
can manipulate individual pieces how
exactly that ends up when the tool is
done but at least the first version of
that part is done by the summer will
kind of be a user testing thing as we
kind of try it with people but my
suspicion is that we're gonna have lots
of like drill down here and then go
there query the camera or query where
you are around you or what's around me
right now or how many things are in
front of me stuff like that is where I
suspect the directional go as we kind of
test it with people and they say Oh God
why'd you do with that which is what I
expect to happen the first time and you
mentioned animation I'm curious how you
would account for describing animation
when there are multiple parts going on
simultaneously
is it like different pitches or textures
and the voices or that's a great
question okay so the first thing to know
about animation with screenreader
technologies is that making them work is
like an insane challenge for reasons
that have nothing to do with the user
interface and I'll describe that very
briefly just to gonna give people that
sort of understanding notably if I have
in an environment like Windows Windows
uses something called the Windows
message pump to get information back and
forth to a computer program right and
all this is is like you presses the Mike
Mouse clinics and so but the Windows
message pump when you're working inside
of an animation library is tied to the
frame rate in certain ways depending on
how you pump it out so that for a second
I get a key press it goes on a frame
tada
normally for animation good that's
exactly how it should work but the
problem is screen reading technologies
don't work that way what they do is they
get a key press and then they go through
a dance with the application to try to
make sure that it's querying for
information a particular time so why
Bing if I press like down on the
keyboard and I'm in a text editor that
might actually send back a few hundred
on I mean hundred requests to the
application the gather like where's the
cursor what is it doing or there any
annotations here or there spelling
errors when I'm at or whatever it's
doing right and so what we learned very
quickly when trying to integrate an
animation
is that we have to do these like nasty
equations behind the scenes to sort of
force pumping in bizarre ways that don't
limit the framerate too much although
just a smidge and that's an unavoidable
but also allow it to be accessible so
that you can get information back and
forth the screener without delays
because if you had one request per
screaming of perkele and you have a 30
frames per second then one downer on the
keyboard all of a sudden you got 30
seconds until you've got your little
animation so once we saw that we then
realized that if you had multiple things
animating at once you're not gonna be
able to understand it all at the same
time so what we then do is we have a
system where you can provide information
to the screen reader at strategic times
and this is done in a couple ways
one of which is that there are certain
events that get fired as animations
trigger that provide information to our
screen reader architecture that then
selectively chooses whether or not sends
information to the system so that's
number one a good example of that would
be if you press a button you want it to
click and you want the screen reader to
know that and like that is technically
an animation with lots of frames doing
all sorts of stuff but the crucial
information you need is the click not
the fact that it's like you know
flashing or whatever it's doing and same
with 3d animation too like if I'm
rotating something I'm not gonna provide
screen reader information on every step
of that rotation but I might give it
only when it's done or maybe along the
way or stuff like that and those have to
be kind of tuned with user testing to
make sure we kind of get it right and it
feels good
normally when we use it it's pretty easy
to tell kind of when those should
trigger but then the last thing is
screen readers while they have good
mechanisms for connecting into things
like text editors and stuff they just
don't connect it all up to 3d engines so
there's some things that we have to
present to the user that don't follow a
standard Windows event infrastructure so
for those we have this little thing that
we call and notify and a notify is an
event that doesn't match to an event but
forces the screen reader to talk in
appropriate ways at appropriate times an
example of that might be when I'm using
a camera one of the things we found
pretty quickly is that if we try to
using a screen reader blind
I want to query it for all sorts of
information like where am I
what are my coordinates what
what's in front of me what's in my
frustum what's in my if i zoom out
what's far you know stuff like that
yeah so the way that we do that is we
use a series of accessibility notifies
they're not events really they give the
screen reader context about a certain
wearying condition and that tells us
creator oh I need to find some
information and then it goes from finds
it and you're good to go so it's really
tricky like it's not one thing right it
involves a series of technologies but
it's possible you just have to be kind
of judicious and you always have to test
it with people because it's really easy
to get it wrong or make it just not feel
good how do you recommend CS educators
use quorum in the classroom somebody
might be listening to this and go okay
awesome I have an option for children
who are blind or hard of sight but what
about all the other kids would they be
able to use it as well yeah so there's a
couple things going on in the quarren
project first is last year I think that
the number of users on our website for
forum was in the ballpark of 80,000
something like that which is a lot more
than students with blindness right like
it's not just blind kids that use this
stuff it's a lot of schools in general
use it I don't always know the ways they
use it they don't mail us what they're
doing in the classroom and stuff but I
would say the most common thing that I
see people doing online on analytics and
stuff is using it for computer science
principles because we have a mapping of
our curriculum to code orgs which they
kindly give us permission to do except
that all the tools and techniques for
accessible I will say that in the future
what I suspect people will use for is
will probably add a block visualization
mode on so if you're in a typical school
and you're wanting to do whatever you
feel like doing and you're used to that
kind of thing
you can just do whatever you want
normally but it's also accessible so
over time my suspicion is that quorum
will follow all the same visual
standards that some of those sort of fun
kids tools do but at the same time will
remain accessible so you can use it just
for whatever the last thing I'll mention
is teachers in the classroom that use
quorum use it for a lot of different
stuff that I wouldn't expect so like I
know there's schools in Maryland that
use it for Lego robot competitions
there's a lot of schools that use it for
game programming like two-year 3d games
under all sorts
there's people that use our digital
signals processing libraries to write
like little audio things there's people
that just make audio games that like
plant 3d sounds all over the environment
and then run around with them and stuff
there's things like that there's a group
of people in mostly Wisconsin that use
corn to connect to robotic telescopes
and then they send back images from
these telescopes and do analysis on them
something like that so lots of stuff
actually corns really powerful for doing
all sorts of like even non-tv style of
things
I just don't know all the details about
how everybody uses it but those are some
examples of what people do right now so
while you're talking about quorum and
like navigating a 3d environment I'm
curious have you looked into
accessibility issues in relation to AR
and VR so we've thought about it but we
haven't done anything with it yet it
turns out here's a way in our
environment to connect arv our libraries
up to our existing stack in fact it
turns out to be relatively easy to do
however we haven't like bought the
devices to test with it and I don't know
how the accessibility works on those
devices like for example does an oculus
have screen reader support
I'm guessing though but we haven't
looked into it so I think there's
actually a lot of opportunities there or
lots of people even if you can't see man
the things you can do if you can turn
your head and observe it's really
powerful but I haven't done much I
haven't done anything with that I think
there's a real opportunity in orem or
had yeah and I'm slowly learning
Japanese and so like each weekend
somebody comes over and she's a native
Japanese speaker and we kind of like she
helps me out with learning how to do
that and one of the things she showed me
I didn't realize in Google Translate you
could actually pull up your camera with
that and it'll translate an image like
you hold up a piece of paper or
something in Japanese and it'll on your
screen make it into English
oh I'm wondering if there are
applications of that in relation to
screen readers with an AR so if you
potentially had on some kind of a
headset that allowed you to basically
audibly translate the visual environment
as you're going through it I hadn't
really thought about that but that makes
complete sense I can imagine that being
highly possible that sounds like you
actually I think imagine that'd be
really
like maybe have a pair of glasses or
something that has a little camera on
them and as you're walking around as a
blind person you can see and it says
water closet if you're in Europe or
right now bathroom if you're in Japan or
something like that even though it might
be in a different language that doesn't
matter cuz it automatically translates
so I'm wondering if you could wave your
magic wand and instantly make CS in CS
education more accessible what kind of
changes would you make in terms of
accessibility there's really two that
immediately come to mind one is
accessibility infrastructures really
shouldn't be different across
programming languages and across
platforms and that particular issue
sounds like it's minor because we just
deal with this fact but the truth of the
matter is the programming language Wars
where every company makes its own
language with its own platforms and own
api's has caused serious damage to our
discipline and it's like triple e so for
accessibility because like you're going
to build accessibility support in the
Windows sorry it works
nowhere else because the code is
compatible and also with some
applications like all this just accept
well if I'm making a user interface in
Java it's different than Swift that
shouldn't be true either
the languages should be more
consolidated than that there should be
like an API that's across languages but
never mind that
but for accessibility it's even worse
because the accessibility
infrastructures usually are kind of
broken on each platform so like you go
out to use Microsoft's UI 8 providers
and it has just a million quirks because
it doesn't work in certain ways under
certain contacts like if you use
multi-threading in Jaws it breaks but
not with NVDA but if you go onto other
platforms like Android or Windows
I mean Apple you know is completely
different like literally you have to
throw your code away and start over and
that's a huge problem because those
accessibility infrastructures are
massively different and they're broken
in their own ways too so it's like you
need almost like a team of 20 to really
get accessibility support everywhere and
most teams just aren't that big and
that's just the accessibility part
before you even touch the problem you're
trying to work on so I wish that the
companies would get it together and make
a unified accessibility infrastructure
for all platforms and all languages so
that it could be more you
find everywhere and you ride it once and
everybody can back to it but that's
tight dreamy I mean like the companies
aren't doing that in any way right now
and in addition the companies do then
have a vested interest because like they
may if Swift makes apples stuff and they
make a custom language and they get
schools to use it then you boxed into
their platform right and so then
developers are more likely to build
stuff for their things that gets more
apps and so there's a commercial
interest personally I suspect long term
the government will have to step in to
sort of start to fix some of these
issues but how to do that without being
heavy handed and to make it actually
reasonable and equitable is a tricky
thing it's not it's not simple at all
yeah I'm a Windows Android kind of user
and taught Swift in my classes because
we had a Mac lab and I love the language
but I can't use it now because I no
longer have a Mac you know it's
unfortunate and that's just the platform
but like it's a legitimate issue and if
you have you know windows and you're
doing accessibility well you can't do
that immaculate I mean maybe there's a
way to get around some of it we have to
figure that I'm curious in your CS
classes that you teach do you use quorum
only as the language and platform or if
use others how do you consider the
various affordances and constraints when
you're picking a language or a platform
for your class oh that's a good question
so I always actually feel uncomfortable
pushing my own work on to my students
yeah I wouldn't say that it's unethical
like legally there's no mandate about it
but it just feels like you know I want
my students to feel able to explore I
might be able to do kind of more
interested in and that's okay
but in my classes I usually teach like
upper division stuff so they're
project-based students kind of get to
choose stuff so I almost always have a
student team that's at least willing to
try it and then you know not my mandate
but my choice and so one some of the
more popular ones are like building like
games that are accessible and stuff like
that but even then it's not always forum
so I I had a few years ago I had a team
try to make world of warcraft accessible
am i that is really hard they had like I
don't know 50 accounts banned by
Blizzard I'm trying to automate things
to make it accessible they were trying
to cheat right right right
blizzard didn't I see it that way
because they in the bone all
time that light will make like a
role-playing game trying to make it
accessible and they usually use quorum
there because there's libraries built in
to make it accessible so it's a lot here
then trying to do it in unity where it
would be a nightmare trying to make it
accessible my own personal balance which
other professors might have their own
own way is I never mandate it for
students I let them choose whatever they
want but oftentimes since students know
that I do this kind of stuff I do work
with some of them on quorum stuff and
that can sometimes lead some pretty fun
projects yeah I know it's a little weird
kind of asking people to use something
that you've created like having taught
classes or I'm like hey I wrote this
publication on this thing but here are
some other publications sort of related
to this topic again it's not like it's
unethical it's fine like there's nothing
wrong with it it's a legit product that
you can use and stuff right but it also
feels like you know am I just too close
to it am i pushing something you know
and maybe that's maybe that's wrong
maybe it's fine and I shouldn't even
think about it but I always second-guess
myself
maybe it's an imposter syndrome I don't
know maybe maybe I should but I like to
let students choose yeah I agree with
that sentiment having had some
professors that would push their books
on you and then you never end up using
it in your class yeah exactly right
stuff like that it just feels like it's
not necessarily like malicious it's just
you know thankfully those were
professors outside of my major they're
just like some of the general Studies
classes so I'm curious working in higher
education and research in particular is
a pretty demanding job in terms of like
some of the pressures in terms of output
for publications presentations other
works and whatnot so how do you kind of
take care of yourself or try and stave
off some of the burnout that comes with
jobs like these oh that's a good
question I don't need ever asking with
that before you're right it is
challenging is that kadai Mia you know
there's always these phrases like
publish or perish and they are true you
know if you don't publish you don't get
to keep your job at least until you get
tenure and then there's weirdly there's
a lot of protection but you know for me
I would actually say a better balance
after I got 10 years and before and
that's probably pretty typical but you
know in my own case I am careful about
my health number one I exercise
regularly i watch my calories and trying
to keep myself physically fit so that
that's important to me and I I know
have direct evidence that that helps but
I feel like it helps me mm-hmm that's
one thing but then number two I also
feel like academics should especially
once they have tenure when you're still
trying to keep your job you kind of
don't have as much choice you know
because you're trying to prove yourself
but once you do I also feel like
academics should take a step back and
realize that a lot of the academic
metrics that universities use are stupid
like for example if you wrote an
academic paper and you said I think we
should hire and fire academics based
purely on the number of papers that they
publish you would get a rejection letter
because that's stupid but in tenure
promotion guidelines
that's happen what we actually do in
practice right there's a lot of
universities they don't even read the
papers but their academics put in they
just say well you published six in this
time good job you get to keep your job
that just makes absolutely no sense it's
totally ludicrous and so I think part of
my own sanity is I've come to the
conclusion that I'm some of it I'm just
going to ignore and I will publish
enough that the admins will keep off my
back and stuff like that and I like to
publish anyway so I guess there's that
advantage but like I'm also that going
to spend a lot of time doing what I love
and to me that's programming and so I
spend a lot of my days like thinking of
how am I gonna hack together this thing
to make 3d work for a blind kid my
colleagues are supported but if they
didn't like it I would just I'm fine
with that
then give me a bad evaluation
fortunately I'm lucky and that we've
gotten awards and you know all sorts of
stuff and you know people are sort of
okay but at the same time like you know
some of it too is you have to kind of be
willing to speak out a little and remind
your colleagues that like some of these
academic metrics are really not
evidence-based they're just kind of like
you know we chose paper counts because
the admins can count them easily in
academia it's funny they always talk
about teaching research and service
right and it's funny with research we
have these absurd metrics for teaching
we often use teaching evaluations and
mine are very high but there's lots of
issues with them having gender bias that
I mean they're just not really good and
that's service like we don't take any
metrics and so therefore somehow people
just ignore service and like it's just
not taken seriously I'm like is
really weird that we evaluate academia
this way and it should probably change
if we want to keep our own sanity as one
academic I only have so much control in
such things anyway so for me it's like
health stuff and then kind of ignoring
some of them that makes sense
and what do you wish there is more
research on that could inform your own
practices well I mean this is my own
bias talking but I really want like
impact of programs I mean look this
affects more than 300 billion dollars a
year in wages in the US alone it's a
huge deal for students it's a
monstrosity
and then in k-12 computer science think
about this right now we've got these
things like computer science principle
right which i think is generally
actually a pretty well thought through
standard however because there's no
language attached to it and computer
scientists cannot seem to agree that for
is a bad word choice having 34 years
we're in this weird situation where you
might go to one school you're learning a
notation about programming the school
write over use this one different and
the problem is that's absurd I'm like in
math they don't do that if you have an
integral we use the integral symbol I
mean there are alternative notations if
you want to use the plus symbol in math
everybody agrees but the freaking plus
symbol means science we don't not even
the Plus do people agree on what that
means in C++ it means something
different in Java and that's stupid we
need to get rid of that so like you know
in my magic wand waving way but I would
want I want a lot more human factors
research on that because I think if we
can sort a lot of that stuff out and
rule out a lot of these bad decisions
that language designers may have made we
can start to construct products that are
based on sound evidence and then get
them adopted slowly over time
get rid of some of these languages like
you know like maybe like Perl that
aren't very well constructed in terms of
their human factors impact slowly let
them die and let languages that are
using evidence that are slowly making
improvements over time
let those slowly get adopted there's one
reason why I'm quorum we focus on
adoption as well that's why when we
first started we had four kids and then
now it's like more than 80,000 I don't
want it to grow too fast and I don't
want it to be
only option out there because arm still
has a lot of areas where we need more
data to say the least especially right
now we're redoing our compiler error
messages because there's good strong dad
we need to but in any case I think that
would probably make a big difference
because this language horse problem has
been systemic huge problems for our
discipline and we don't really talk
about it that much just kind of except
that it's this big nasty mess yeah and
in the k-12 in particular in the younger
grades it is based more on how shiny it
looks and the marketing that's behind it
rather than the evidence that supports
the learning hopefully that's involved
with whatever platform is being used or
language yes right like I mean you know
to a certain extent we all want students
to be engaged and so like I'm all for
like making little animations and doing
this kind of stuff that's all good I get
it I mean like I don't want to just have
kids use a console that's pretty damn
boring but on the other hand too like
keynote the data does matter and even if
like that stuff is shiny you know the
fact that there's a bazillion different
products and the kids are all learning
different things you think that I don't
know in my magical dream world which you
know maybe in like 10 lifetimes from now
what I would really like to see is that
there's one standardised syntax for
languages that is like a bare minimum
and then any language that out there and
this super evidence-based there's hard
core data people have tried to refute
the evidence countless times a lot a lot
and then any language that exists they
can innovate on top of that standard but
the that standard for basic things like
looping like everybody uses repeat ten
times because it just makes damn common
sense right and then the standard
libraries are all built on the common
part not the experimental part thereby
making it so we can get to closer
standards over time over like a 50-year
century so that maybe we can get rid of
some of these problems before the next
three or four generations come by cuz
like the promise is they don't do
anything kids three generations from now
are gonna have the same damn problem for
the same reason it's just not right it's
gonna screw up generations of kids if
you don't fix it so I don't know maybe
I'm being pedantic but that's five you
know I like it I mean in general like
multi-perspective all approaches with
most things in life but we need to have
those perspectives built off of some
kind of valid research that supports the
claims that are typically marketed
with it yeah exactly well and in other
disciplines too to my brain it feels
like there's sort of a core and then
there's an experimental part where we
tweak like for example generally in
schools teachers aren't gonna change the
meaning of the plus symbol right because
like that's just part of our standard we
just thought we accept that that symbol
means one thing or at least you know at
least didn't eight the twelve setting if
you want to make some innovative
mathematics you can use your own crazy
symbols but if that becomes standard
where everybody's learning about it then
the textbook is trying to get some kind
of normal notation for it we try to kind
of boil it down so everybody's doing we
need that computer science to like if
we're talking about looping we should
try to let use the same symbols and
terminology so that any kid no matter
what school they're at and start to like
you know communicate with each other it
just feels like it'll help society you
know like I mean think about this all
right any evidence of this but I wonder
if you were to take the number of hours
work from building the standard
libraries that do the same thing in
every programming language that exists
lists a race games all this good stuff
how much money have we blown down the
drain by rebuilding the same ceremony I
mean it's gotta be one of the most
massive duplications of effort in human
history to do that and like we don't
talk about that in computer science
we're just like tripod foam that's great
whatever language you want to want to
pick in the irony of that in relation to
CS in particular is not loss of me in
the fact that it's like oh why don't I
just find somebody else's code that will
do this automate this thing that I make
my life a whole lot easier
exactly I've been there - absolutely
it's just the nature of the beast cuz we
still live in the society I mean we
can't change it immediate looking at
your google scholar profile there's a
lot of publications on there and in your
CV I'm curious if there's in particular
that you really wish more people read or
kind of understood the study on that
placebo study that's probably the one
that's the most one done I don't know if
I have one in particular where I'm like
this is the one thou shalt know I feel
like the the broader theme of the work
that we've been doing is to try to get
at these broader like language design
type issues and I wish that people had
not necessarily one of the publications
but what I wish
you know these text books for like
principles of programming
and we talk about like they often
they're like they make a crap ton of
claims like this thing is better than
nothing and stuff like that I wish those
would be rewritten and like thrown in
the trash and then we would make a new
generation of those text books where
instead of just talking about like this
is what scoping is and then also here's
the impact on people that are working
when they use these things here's the
impact on students like for example when
students leave a class on principles of
programming shouldn't they actually know
that differences in programming
languages in fact people very strongly I
mean like that isn't that a fundamental
principle of programming languages all
we really do is just say here's a
smattering of languages that exist and
like you can try them all and then
that's it somehow they're not better or
worse they're just different
right which is crap because it's not
true at all or like this other common
phrase you hear in classes like that is
you just have to choose the right tool
for the job
right and like that's just such a
ridiculous nonsensical statement yes if
you're on Apple you only have so many
choices for what languages you use but
that doesn't make it the right tool it
means the only one that you can use
because of either license agreements or
something else it's not quite the same
so I mean to me which these principles
of programming books would be rewritten
completely and that inside of them would
contain the smattering of evidence
across a lot of scholars on what we know
now well so so that when students leave
school they actually have some
background in like what's going on in
the the myriad of research groups that
are finding these particular effects in
the last 10 years or so yeah so even
though the last few years have been
working full-time in CS education like
I've mentioned outside of this interview
that all my degrees are music education
so I'm technically coming from from an
outside perspective and in music
education and other areas within
education there's a lot more critical
discussion on the discourse that is
going on in those fields and I think
that is something that's kind of like
lacking in CS education in particular is
we need to have more conversations about
some of the things that you're talking
about that no these are not equally as
good they're better or worse in certain
ways and others and we need to kind of
talk about those things I think so too I
don't know what the actual debates are
music education can I ask what are they
yeah so the biggest debate right now is
the debate about whether large ensembles
anachronistic so yes they provide some
awesome opportunities for people to
create music but what opportunities are
there for people to continue that music
making once they finish their k12 tenure
oh man isn't that the truth that's
interesting so that is like one of the
biggest things and there's a huge divide
among the field where people are like no
I'm a hundred percent for a large
ensembles and other people are like no
we need to just burn that down and just
start with like popular music making
today's like deejaying and rapping or
whatever and do only that and then
there's like people who are like well
they both have their places and so
that's a large one that's pretty
interesting especially given that you
and your experience might be different
but when I was doing my music singular
degree in just a lot a lot of what you
do learn it does feel really old right
there's a thing called counterpoint and
it has to do with like when notes are
sorted together and how they're kind of
linked up and stuff like that what that
ends up sounding but when you if you
take a class an actual counterpoint the
actual theories are from the scanning
Fuchs if I recall if Fuchs Ian
counterpoint the cotton is from like the
haven't been updated you know 400 500
years right 400 I forget exactly and so
it is funny that the music education did
feel like it needed more evidence to
sort of update itself as well that's a
whole other lifetime of scholarly work I
suppose I had run into a lot of conflict
through my degree in that the last
course I ended up taking was a Russian
music theory course for my PhD and it
had absolutely nothing to do with
anything I was interested in scholarship
wise it had no sway on anything I was
doing but I had to do it because it was
a music degree technically in a music
school and the electronic studio courses
that I took would not count as music
theory so I took a class that was about
creating music software through max/msp
which is a graphical programming
language and that would not count as
music theory and then even though it was
a music theory course housed on the
music theory department so like they
even have that divide where oh if it's
anything electronic sorry that doesn't
count as real music theory doing air
quotes there it you need to do this
other thing that is
completely irrelevant to whatever it is
you're doing that is crazy because you
know so I used to program and I don't
even know if it's still around anywhere
a programming language called super
collider yeah yeah yeah so it was it's
like kind of a text-based version of
Maximus P I think max is a lot easier to
use frankly but nonetheless different
letters what we were kind of messing
with and it is very theoretical right
like it's you need to know some math you
need to know how all these pieces link
together there's these weird Jen's and
make up I mean like that they wouldn't
count that as serious just crazy yeah I
felt the same way that's why I kept
pushing off that course because it was
like maybe if I petition it they'll
eventually count these as an actual
music theory course but they wouldn't so
that was the last thing I had to do that
is so funny it does just feel like that
it's like that's legitimately
theoretical in nature but clearly not I
guess do you have any questions for
myself or to the field at large nothing
immediately comes to mind I will say
though for people they can that I should
try to make yourself accessible it's
good for your students there's a lot of
people that have disabilities challenges
or whatever and it does people good it's
a good way to not only feel good but
like it does good it's a good thing it's
for the betterment of society so if you
can give it a shot where might people go
to connect with you and the
organizations you work with they can
email me at my normal University email
which is online if you're interested in
corner stuff there's a quorum mailing
list and that's linked on the quorum
website which is at quorum language comm
if you're interested in accessibility
the name of our grants that we do we
call it access yes for all and that's
Google Bull for sure but I can get you I
can toss you a link before for some of
these things so people can get up on the
site or whatever access you use for all
has a number of various failing lists
for different kinds of interests that
people have their relatively low traffic
on purpose so we don't bombard people if
you want to get even more involved we
have regular meetings for developers
related to accessibility for people that
part of want more or just want to learn
more like for example I think you
actually went to our most recent webinar
which was on 3d technologies that are
yeah and I think I became aware of that
webinar because of being on the
newsletter in the mailing list rather oh
and there's others too that we have on
that too I think we had Stephanie Moody
does also work in Texas on accessibility
and I think we had someone from Google
that was doing some cool accessibility
stuff lots of people like that and with
that that concludes this week's episode
of the cska podcast I hope you enjoyed
listening to this episode as much as I
enjoyed interviewing Steph ik just as a
friendly reminder I'd really appreciate
it if you would share this with somebody
else or simply write a review on
whatever platform you're listening to
this on up so that people find it
speaking of finding stuff friendly
reminder you can find me show notes in
the description that you listen to this
on or by visiting jared O'Leary com stay
tuned next week for another unpacking
scholarship episode and two weeks from
now you will get another interview until
then I hope you all have a wonderful
week
Guest Bio
Andreas Stefik is an associate professor of computer science at the University of Nevada, Las Vegas. For the last decade, he has been creating technologies that make it easier for people, including those with disabilities, to write computer software. He helped establish the first national educational infrastructure for blind or visually impaired students to learn computer science and invented the first evidence-based programming language, Quorum. The design of Quorum is created from data derived through methodologies similar to those used in the medical community. He has been a principal investigator on 5 NSF-funded grants, many of which related to accessible graphics and computer science education. Finally, he was honored with the 2016 White House Champions of Change award and the Expanding CS Opportunities award from Code.org and the Computer Science Teachers Association.
Resources/Links Relevant to This Episode
Other podcast episodes that were mentioned or are relevant to this episode
Accessibility and Inclusion in CS Education with Maya Israel
In this interview with Maya Israel, we discuss Maya’s shift from special education into CS education, the importance of universal design for learning (UDL) in CS classes, understanding the spectrum of accessibility for students with a diverse set of needs, the current status of accessibility and inclusion within the field of CS education, problematize deficit framings of students with disabilities, and so much more.
Exploring (Dis)Ability and Connecting with the Arts with Jesse Rathgeber
In this interview with Jesse Rathgeber, we discuss what educators should know about (dis)ability culture and research, person-first language vs identity-first language, suggestions for combating ableism through anti-ableist practices, how the arts and CS can come together and learn from each other (great for sharing with arts educators who might be interested in CS), and much more.
How to Get Started with Computer Science Education
In this episode I provide a framework for how districts and educators can get started with computer science education for free.
Planning K-8 Computer Science through the UDL Framework
In this episode I unpack Israel, Lash, Bergeron, and Ray’s publication titled “Planning K-8 computer science through the UDL framework,” which discusses the potential for using Universal Design for Learning (UDL) in CS classes.
Should I Say “Disabled People” or “People with Disabilities”?
In this episode I unpack Sharif, McCall, and Bolante’s (2022) publication titled “Should I say “disabled people” or “people with disabilities”? Language preferences of disabled people between identity- and person-first language,” which summarizes findings from a survey on participant preferences for language around disability and an analysis on language in conference abstracts.
Music-related resources relevant to our introductory discussion
Learn more about aleatoric music
Here’s an example of what aleatoric sheet music can look like
Listen to an example of what the example above sounds like
FYI this piece depicts the fire bombing of Dresden, Germany, so it’s an intense piece of music that has resulted in many tears by audience members and performers alike (including myself, as I have performed this piece multiple times).
Listen to what equal temperament vs just intonation sounds like
Music-related links that Stefik has been involved with
Andreas Stefik, Melissa Stefik, Mark Curtiss: Ammmp: An Automatic Translation System for Musical Languages. Computer Music Journal, 31(4), 2007.
Learn more about MAX, the programming language we discuss at the end
Scholarship discussed in this episode
Accessibility-related resources relevant to CS educators
Connect with Stefik and learn more about the projects his working on by visiting his website
Find other CS educators and resources by using the #CSK8 hashtag on Twitter