You're testing them wrong: Whiteboard coding interviews are 'anti-women psychological stress examinations'
- Reference: 1594796052
- News link: https://www.theregister.co.uk/2020/07/15/it_hiring_whiteboard/
- Source link:
But recent research suggests that whiteboard technical tests – so daunting to job seekers that there are [1]books on how to deal with them – often fail to assess technical skill, according to new research. Instead, they're all about pressure.
In [2]a paper [PDF] to be presented later this year at the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, researchers from North Carolina State University (NCSU) and Microsoft in the US argue that whiteboard sessions test for stage fright rather than, y'know, coding competency.
The title of the paper hints at its conclusion: "Does Stress Impact Technical Interview Performance?" NCSU authors Mahnaz Behroozi, Shivani Shirolkar, and Chris Parnin, with Titus Barik from Microsoft, say it most certainly does.
"Through a happy accident, the software industry has seemingly reinvented a crude yet effective instrument for reliably introducing stress in subjects, which typically manifests as performance anxiety," the paper explains.
When Facebook says you're not a good 'culture fit', it means you're not White or Asian enough – complaint [3]READ MORE
"A technical interview has an uncanny resemblance to the [4]Trier Social Stress Test , a procedure used for decades by psychologists and is the best known 'gold standard' procedure for the sole purpose of reliably inducing stress."
As a consequence, whiteboard interviews may fail to assess coder competency. Rather, the researchers argue, they measure how well job candidates handle anxiety.
Using 48 graduate and undergraduate students with programming experience, the researchers conducted a randomized controlled trial to compare the traditional technical interview (done while being watched) with a private session evaluation (done without being observed). The experiment was designed to measure cognitive load and stress through the collection of eye tracking metrics, specifically fixation duration and pupil dilation.
The researchers found that stress hinders interview performance, with participants in the traditional technical interview exhibiting higher cognitive load, lower scores, and higher stress levels. In essence, social anxiety took otherwise qualified job candidates out of the running because of the circumstances of the interview.
Further flaws
What's more, whiteboard technical interviews appear to favor men over women.
"We also observed that no women successfully solved the problem in the public setting, whereas all women solved it correctly in the private setting," the paper says.
In a phone interview with The Register , Christopher Parnin, assistant professor at NC State University and one of the paper's co-authors, said he doesn't have a conclusive reason why this might be the case. He said there's some support in academic literature to indicate the women have more performance anxiety than men, but he stressed that's a gross oversimplification because men experience performance anxiety too.
For Parnin, the problem is whiteboard tests themselves. "It all comes down to the fact that the test is designed to make almost anyone fail," he said. "You're basically having to interview tons of people just to find those who can pass it."
Parnin took issue with the way the industry has dealt with the difficulty of its tests. Rather than coming up with a fair way to evaluate software engineers, companies like Google advise at least 40 practice sessions - a time commitment that's not an option for everyone. This amounts to stress inoculation training and it does help people pass whiteboard tests, he said, but it doesn't make the tests an effective skill assessment tool.
As an alternative, the paper points to the way devops biz Honeycomb (Hound Technology) – overseen by a female CEO, CTO, CMO and VP of engineering – approaches hiring. The company provides interview questions in advance so it's not a Trier Social Stress Test.
As the company [5]explains on its website, its goal is to avoid surprises. "The research is clear: unknowns cause anxiety, and people don’t perform well when they’re anxious," the company says.
"The big picture is to provide more accessible alternatives," said Parnin. "There are a lot of ways to test for the same thing without putting all this pressure on people." ®
Get our [6]Tech Resources
[1] https://www.amazon.com/Cracking-Coding-Interview-Programming-Questions/dp/0984782850/
[2] http://chrisparnin.me/pdf/stress_FSE_20.pdf
[3] https://www.theregister.com/2020/07/06/facebook_racial_discrimination_complaint/
[4] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5314443/#
[5] https://www.honeycomb.io/blog/observations-on-the-enterprise-of-hiring/
[6] https://whitepapers.theregister.com/
I had one interview about... 8 years ago? 3 managers sat around a conference table while I stood at the front with a paper flipchart and a chunky board pen. They read out programming scenarios and I had to write the code to solve them.
Paper was definitely not an ideal medium, being impossible to erase or change anything I'd written when I realised I needed to insert a line between two I'd already written.
"... and working under stressful conditions will be required."
Yes, but "Kobayashi Maru" tests should be performed exactly with this in mind, not to assess people skills in a given field.
When I give tests to people I usually leave them alone - just maybe removing all electronic aids to solve them - I don't really need much StackOverflow search capabilities, although even using SO correctly instead of blindly copying the first answer is a skill itself.
Interactive questions are of a different kind.
"the test is designed to make almost anyone fail"
While it's not a bad thing in itself to see if a candidate can hold up under pressure, from my personal experience as a candidate (numerous times) and as an interviewer (several times), I can vouch for the fact that the mere interview procedure is a stress test for everyone, so I fail to see the need to add to that.
Especially with a procedure that is apparently made to disqualify a candidate. If I'm looking for a programmer, I want to find one, not spend my time disqualifying the possibles. I wouldn't have done a whiteboard test anyway. I would bring a printed part of code and ask the candidate what is wrong. I find that a very good test procedure ; you can quickly see if the person knows what they're talking about or if they flounder totally.
Oh, and there will be something wrong in the code. I'm not doing psychological tests, I want to see if the person can find the mistake.
Re: "the test is designed to make almost anyone fail"
Of course there's something wrong in the code. There's no such thing as a perfect program.
But I assume you're looking for debugging technique more than programming language skills... so start with "where's the input validation?", progressing to "what if someone has set this parameter to zero?", then hitting the language issues "surely this print command should be a printf?" and "still allocating memory manually?"
Re: "the test is designed to make almost anyone fail"
When I use these "what's wrong with this code?" tests with interview candidates, I'm looking for people who have language knowledge, understanding of particular idioms, debugging skills and can spot typical pitfalls. I've found I can glean a lot more information this way than from asking questions.
Sure, maybe I'm asking the wrong questions, but this way you get to see the candidates' thought processes, and gain objective metrics to evaluate performance.
but this can't be true
I've been told that women are equal to men, and can do anything a man can do. So how could two equally competent software engineers pass or fail a test entirely because of their gender?
Whiteboard coding tests are nonsense. Getting someone to draw a diagram on the board - a design diagram, a toolchain flow, anything really - is reasonable but only a muppet would expect a software engineer to write code on a whiteboard.
But they're not sexist. Stop pretending they are. Especially stop reporting that they are without providing data like 'how many men actually passed' which would tell us whether this headline grabbing misreporting is based on a rounding error.
Re: but this can't be true
Equal does not mean identical.
I don't know if it is true that all women have more trouble than men in coping with a whiteboard test, but after 30 years of marriage to the same woman, I can vouch for the fact that my wife does find it particularly difficult to deal with cold calls and other marketing types. She just can't brush them them off. She can't build up the nerve to cut the call short and hang up.
I don't have that kind of problem.
Re: but this can't be true
Er, equal does mean identical. Its either equal or its 'sort-of-like-that'.
As we all have experienced, some interviews and techniques are good and some are rubbish. Its a thing these days that these experiences are seen automatically through a lens of anti-women or anti-BAME.
Does an interview identify the characteristics needed for the role? If no, then it prejudices against all potentially good candidates. Its not a women thing, its not a BAME thing, its a rubbish process thing.
Re: but this can't be true
Women are about the same[1]on average in their ability to come up with the correct answer as men, but the process by which they come up with the correct answer is not necessarily the same.
[1] Actually slightly better on average than men, but men are less likely to be average, so a group with the best people will have more men, and a group with the most useless idiots will be mostly men.
If you go back to the early years of computing when many of the fundamental things we now take for granted were invented, most programmers were women.
Re: but this can't be true
Still now, I have noticed that a lot of COBOL and AS/400 programmers I know are women.
Because I think there were high school course of accountant programmer, with lesson on book keping, touch typing and COBOL. Being a book keeper is seen as a 'normal' job for a girl.
On the other hand a technical engineering high school had courses of electronics, calculus,PASCAL, C and assembler. Technical engineering was ok for boys.
Damore's Caution
Careful now. You can [1]get fired for suggesting men and women are different .
[1] https://en.wikipedia.org/wiki/Google%27s_Ideological_Echo_Chamber
Whiteboard coding - never done it, never ask anyone to do it
We don't write code on a whiteboard during our day to day, so why ask them in an interview?
I know that they are popular with Goomicropplebook but I don't think they are effective. Much better to get someone to bring some of their code with them and get them to talk through it; explain how it works, what they would improve it, ask them how a feature might be implemented. That's more like the real world rather than this nonsense about whiteboard coding.
As others have suggested, I like to have paper or a whiteboard at hand so that the candidate can explain things by drawing. That's not the same as writing up on a board with a pen.
The whole whiteboard coding smacks of quasi-faux-academia. "Look how cutting edge we are: algorithms on a whiteboard; just like real science". Yes, I did that in academia but there tended to be two or more collaborating on a problem, not 3 judging another for a job role. And it was never much code, lots of diagrams, some maths and doodles.
Re: Whiteboard coding - never done it, never ask anyone to do it
Exactly this. Whiteboards are a terrible way to wite code, but they can be geat collaborative tool for exploring ideas, getting your point across visually, and just generally working shit out. If you want to use a whiteboard in interviews, do it for the right reasons.
Re: Whiteboard coding - never done it, never ask anyone to do it
Isn't this yet another lurch by Google into shit interviewing technique - that they then arrogantly tell everyone else about how great it is?
Remember ten years ago when the cool interviewer was asking questions like "how many golf balls can you fit into a school bus" - because Google were doing it. None of them knew why it worked, and then it turned out that it didn't and Google abandoned it for being no better at telling you anything about people than random.
Probably a good test for my job actually, in technical sales. Because it's a test of how well you can bullshit under pressure while sounding technically knowledgeable about something you are utterly ignorant of. But personally I hate winging it, and am perfectly happy to say, "I don't know, but can find out for you." Which in my experience often boosts your credibility.
I concur
A few years ago I flew around the world to attend a wide range of job interviews, including for companies like Apple and Microsoft. I found the 'on-site' aspect to be rather stressful already, with the logistics of train, plane and local transport. Even without jetlag coming into play, finding oneself the next day in front of a whiteboard running through the same 'implement memcopy' question again and again with every question from the person watching you seemingly aimed at raising your anxiety.
When I put those experiences next to me actually working for a big company (major German car manufacturer) and attending meetings with heads of departments about the project which I was leading... then I must confess to not feeling anxious about these meetings or phone calls at all. Because you know what you are talking about, what has to be done, and how to communicate it to others.
In comparison those job interviews were pure Hell and led me to instead decide to just freelance things. Stressful as freelancing may be, it also means that you are picked based on your work and (apparent) skills, not through some inane 'job interview'. I don't think I could ever put myself through another one of those interviews.
Psychologists
We often moan about research Psychologists ( and I was a Psychology graduate working within the field) stating the bleeding obvious. But sometimes the bleeding obvious just needs to be stated.
The ultimate chalkboard moment
From Hidden Figures - one of my favourite films - a truly exceptional woman delivering under exceptional pressure:
https://www.youtube.com/watch?v=g7J_RrBcchQ
Enjoy!
Statistics
I'm not a professional statistician but...
There were only 48 test subjects, 22 'control' who did the test in private (4 of whom identified as female), 26 were subjected to the stress test (6 women).
Those numbers don't sound too good to start with... but there's more:
Of the 22 private tests, 13 of the bottom 14 were male. Isn't that as significant as 13 of the top 14 stress tests being male? So if stress testing favours males, then does non-stress testing favour women? Clearly not, because only stress tests favouring men gets mentioned...
Crucially they did not test the same subjects in both environments, so I suspect that the results were more related to individual ability than any inherently sexual bias. The study just didn't have enough candidates (particularly women) to be statistically significant. I suspect that the study group was also likely to be biased because it was taken from current students with a knowledge of java and algorithmic representation who volunteered to be tested.
By the way, I don't think whiteboard coding is a sensible interview test. If you're going to make an interviewee use a whiteboard then make it interactive, see how they react to suggestions, modify solutions according to additional input, etc. You know, like how programmers actually use whiteboards...
While it's nice to think all jobs will be happy-go-lucky, relaxing jaunts through life, there will be times when the shit hits the fan and working under stressful conditions will be required. Possibly more often than not in some industries depending on the experience level of the employee.
Knowing someone can think clearly and make rational decisions during these times can be a useful thing to know.
That said...
Who are these people who actually do this? Over the last 25+ years I don't recall ever having had to write actual code on a whiteboard during an interview. I've been asked to describe solutions to problems out loud (and refine those solutions as the goalposts are moved), and been asked to describe what specific code fragments do, but I don't think I've ever written code on a wall. That's not how people write code.
What I have had to do in the past (and what I've asked someone to do for a technical interview tomorrow, as it happens) is write a solution to a problem before the interview and present and discuss that code in person. Code reviews are (should be) a regular part of the job. (Are you listening, Boeing?)