Interviews seek to estimate long-term job performance. Since they cannot measure this directly, they measure attributes that correlate with performance. If you interfere with these measurements (e.g. through fraud), you break those correlations. This doesn't mean the interviews are measuring the wrong thing, or that people who cheat on the interviews will meet the performance bar in the actual job.

If someone uses AI to defeat a CAPTCHA, does that mean the CAPTCHA was poorly designed because "the bot is just as valid a user as any human"? Of course not - the CAPTCHA's entire purpose was to establish that correlation between "can solve this" and "is human," and breaking that correlation through automated tools defeats its purpose entirely.

First off, yes, actually that's exactly what it means. If bots can solve CAPTCHAS then the CAPTCHAS are poorly designed, that's the whole point.

I don't think this sort of CAPTCHA type step should have a significant place in a good interview process in the first place, if it is important to you to put your applicants through this sort of gauntlet you're not treating them with the dignity they deserve.

If you're just looking to fill cogs in your machine, sure, but in that case you're just as likely to get kids who studied a bunch of interview questions but also don't know shit. I'd just as soon have someone who's good at ChatGPT.

Let me put it a different, perhaps simpler way:

Let's agree on the premise that good human engineers are much better hires than good AI users. If your hiring process fails to distinguish between good human engineers and good AI users that's evidence that your process is not producing good data.

In the exact same way that if your CAPTCHA is getting solved by bots it's a bad CAPTCHA.

Right, so the problem with the interview process isn't that it tests the wrong skills, but that it fails to prevent the secret use of AI such that a human can appear to possess those skills when they do not, in fact, possess them.

It seems there are at least two different perspectives:

A) If a human can use AI to pass a remote interview that they would otherwise fail, then the questions are unsuitable for use in interviews, whether remote or in person.

B) If a human can use AI to pass a remote interview that they would fail if it were face to face, then the questions may be reasonable, but the safeguards for remote interviews are insufficient.

I disagree with A.

The types of questions that an AI gets right are still bad questions to ask in 2024, regardless of being remote or in person.

If you're allowing your programmers to use AI on the job you're not getting particularly useful information if unaided someone fails a question that they would answer correctly with AI assistance. Sure, being able to answer the question in person may still correlate with being a better applicant, but that doesn't mean it's a good question.

Interviews take time, you should make good use of the time by asking questions that tell you a lot. Questions easily answered by AI (this also goes for questions easily answered by studying interview questions by the way) are not good questions. Measure the things that matter.

   (this also goes for questions easily answered by studying interview questions by the way

I agree, but it’s easier said than done.

> they measure attributes that correlate with performance. If you interfere with these measurements (e.g. through fraud), you break those correlations

Not necessarily... You just get the people that are technically competent enough and are self-corruptable enough to pass the test. There is still a correlation with performance but it is a weaker signal and you are more likely to miss out on people with equal ability but higher morals.

Unfortunately real life problems involve a lot of context and introducing that to interviews is hard. It either requires a long conversation where someone might lose track, or involve writing a large ish code base and getting them to work on that (which is a lot of work, and once again they don’t have context).

In my experience if a problem simple enough to be completed in an hour, it’s simple enough for AI.

IMO if your interview question has a right answer that's like, a basic early screening tool to see if the person is paying the slightest bit of attention. The sorts of questions that really give you useful insight into who someone is as an engineer are the ones that don't have a right answer, and can't be reasonably solved in an interview. If your question has an achievable correct answer you're clipping the gamut of info you can glean from it.

I recently had a question where I was just given a bunch of tuples and told "let me know anything you can glean about this data in two hours". I think you learn much more by seeing how someone fails at a complex nebulous task than whether someone can implement a doubly linked list from memory or whatever.

Real life programming is like 80% failing at complex nebulous tasks, picking yourself up and trying again (by volume). Interviews should simulate that. If AI helps you, so be it.

I work in infra. I don’t want an incident to be the time that I find out my coworkers don’t actually know how anything works, and are frantically posting questions to ChatGPT.

That said, I also don’t know how to adequately test, other than to carefully watch people’s eyes during questions; even that isn’t foolproof.

Some of the best tests come from taking something mundane and twisting it just a little off the beaten path in a way that LLMs cannot handle. The usual example is the river crossing puzzle which has a standard format and solution but can be modified to make that solution incorrect. For code, an example would be parsing log lines to compile insights but switching it up so that the timestamp means something other than the time the log was collected, etc.

You can completely change the complexity of a bog standard leetcode question with the slightest of modifications, and I’m certain no extant LLM will catch such a nuance if your problem is a very common one.

A fun exercise is to ask chatGPT what the river crossing goat puzzle is and then say “do the same puzzle but with two goats and a cabbage please”. It can’t get it right even with multiple corrections. Might stop working at some point now that I posted this though!

"tell me about potential issues while doing X", then dig into answers? We may get there one day with LLMs, but currently it would be extremely hard to keep the conversation context and process the longer answers while talking to someone. The voice recognition quality and generating speed would be an issue too if you wanted to monitor the whole conversation.

Currently the cheating mostly aims for coding tasks because they're pretty simple to do and self-contained. Much less for helping with a good dialog.

Interview performance used to be correlated to job aptitude, but no longer.

I find this all rather intriguing because to me it says more about the employers/interviewers than the applicants.

Over the years I've interviewed many people for technical positions and there are many ways of getting to the core of a prospective employee's competence.

Of course there are questions about training, past experience, qualifications and those about the work you expect them to perform but other matters also enter the equation and they're almost as important (the person's demeanor, confidence, etc.). By letting the person do most of the talking—explaining things about themselves and their work—you can learn a great deal.

Stating the obvious, interviewers ought to be on top of the job they're hiring for but from this story and from other anecdotal info I know that's not always so. In my case as head of the department I was pretty much up to speed on most aspects of the work, so it was clear to me when an applicant was spinning me BS. If they knew more than I did then they almost certainly got the job (life's easier when you've smart employees, for starters one can ask them questions instead of vice versa).

What's that leading up to? Well, when interviewing it was obvious to me (and to the other interviewers) within just the first few minutes whether the person was worth considering. It's uncanny how good one can become at doing this (and my record speaks for itself, I was pretty pleased with those employees I hired).

Let me illustrate with an example: some decades ago I had an applicant apply for an electronics job. He spoke only Chinese and a few broken words in English and none of us on the interview panel spoke Chinese. BTW, he turned up with a Chinese/English dictionary in hand.

I dispelled with many of the questions I usually asked and showed him a somewhat complex circuit diagram with many subsections that performed very different functions (if I showed this at all then it was usually towards the end of an interview).

Within seconds of looking at the diagram he progressed logically through the processes from beginning to the end (note the processes/stages were not drawn in linear progression and consisted of many branches). He did this mainly by pointing.

It was immediately obvious to both my independent technical assessor and me that this guy knew his subject very well. The only point was a minor objection from the nontechnical employment officer who questioned if we'd manage to communicate adequately with someone who didn't speak English. I didn't see that as a significant issue.

He turned out to be an excellent employee who knew his work very well. And within six months his English was good enough not to be an issue.

In summary, if AI is a problem now then it will be more so in the future and that needs to be tackled. That said, I find it hard to accept that an experienced interviewer can't quickly cut to the core even when an interviewee puts up smoke screens and distracting noise.

Like, Popper's falsifiability it doesn't take much to knock a BS or inconsistent argument off its pedestal when one has enough data, refutability usually works well.

That said, one must always keep in mind that just because an applicant can get some things wrong it doesn't mean that he/she isn't the best person for the job. All that tells me is that I have to probe a little deeper.

Edit: asking awkward or trick questions usually ought to be avoided. An interviewee is usually already under considerable stress and can easily be flummoxed when asked them (even if they know the answers they can bind up and not spell them out).

Usually, it's immediately obvious when an interviewee doesn't know an answer to a question or is struggling with it because of stress—body language alone usually makes that abundantly clear. It's then best to gently segue to another question. If the question is really relevant to the job then one can return to it at the end of the interview when the person is more at ease. By then, it's pretty clear to the panel whether the applicant is in the running or not. If not, then do not return to the question.