Rendered at 20:54:10 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Nifty3929 3 days ago [-]
Must I beg to have an acronym spelled out a least once, the first time it's used? Even if you assume 90% of readers already know, the other 10% (including me, in this case) will thank you, it doesn't take much effort, and it expands the reach of your communication or idea.
Exceptions for cases where the acronym is just so well known that a lot of people don't even know what it stands for even though they know the concept well. I recall one corporate training I was sitting through and they used the term "Border Gateway Protocol" and it took me a half beat to think through "oh, you mean BGP?"
Thanks!
bawolff 3 days ago [-]
Which acronym do you mean? CTF? I think that acronym, just like BGP, is more well known by itself than what it stands for.
More generally, not every piece of writing is meant for every audience. Like if someone writes a blog post about CTFs aimed at people who like CTFs, nobody in the target audience needs to have CTF explained to them. Ultimately HN is a link aggregator, but sometimes its a bit like eavesdropping on a conversation. When you are just listening in you don't get the full context sometimes.
0x20cowboy 3 days ago [-]
I dont know what CTF stands for so I dont know if I am interested in this article or learning anything about it. Maybe I am.
Are you really arguing for not just typing out whatever 3 words this stands for once in the name of clarity?
bawolff 3 days ago [-]
Yes, i would argue that people writing articles about niche interests aimed at other members of that niche are under no obligation to clarify it for people outside the niche.
They aren't your teacher. They aren't trying to send the content to you. They are just blogging on their own website for their own audience.
And its hardly unique to this article. If you are writing about the nitty gritty of linux networking, you probably aren't defining what TCP or UDP means. If you are writing a super detailed article comparing and contrasting plot structures of different animes, you probably aren't going to start by explaining what the word anime means. Etc
I'm not saying the world should be all RTFM, but if you are reading some sort of specialized content, then yes i think its a reasonable assumption that the reader has some basic background knowledge on the topic at hand, or is willing to do the research themselves.
SOLAR_FIELDS 2 days ago [-]
Especially when you can literally just paste the top paragraph into your LLM of choice and ask it to explain it to you. In the time it took the OP of this thread to write out their complaint they could have just solved it themselves
PunchyHamster 3 days ago [-]
it's the first result I get on anonymous google search.
It's like complaining about not spelling C in "bake cake in 170 C"
ruszki 3 days ago [-]
If it means capture the flag, then it means a completely different capture the flag for almost everybody. I searched for it, read the first paragraph, and I still don’t know what the fuck is the topic. According to Wikipedia it’s a very new meaning. I could figure out only because of searching for “HCKSYD” and others.
jere 2 days ago [-]
Is 30 years old very new? We're on a site for tech people. I would wager most are familiar with this term.
(Amusingly, it even uses “30 years” as the timeframe.)
jere 2 days ago [-]
Classic of course. The point being: don't make fun of people for not knowing something. In this thread we're making fun of learned helplessness.
ruszki 2 days ago [-]
You are making fun of people who are saying that this article is shit, without addressing that part at all.
PunchyHamster 2 days ago [-]
consider 2 conversations
"hey what X means?"
"X means it"
vs
"I dont know what CTF stands for so I dont know if I am interested in this article or learning anything about it. Maybe I am.
Are you really arguing for not just typing out whatever 3 words this stands for once in the name of clarity?"
The commenter could just say the first instead of deciding his learned helplessness is everyone's else problem
ruszki 2 days ago [-]
The commenter above replied to me, who checked what it is, which is clearly stated in my comment. So your comment cannot be applied to the guy who came up the first with "learned helplessness" in this thread, who replied to me. That guy clearly just wanted to shift the topic from that the article is shit, because they cannot do anything with it.
Also yes, search for CTF is not simple regarding security, because first you need to know that you are searching something in the topic of security. Because every other usage of this is way more frequent. Especially that there is an other way more frequent computer related usage. And the article doesn't make this clear in its first 173 words for laymen. Even I had a problem with this, who is not at all layman, just never cared about this part of security. It's a bad article.
spoaceman7777 2 days ago [-]
More acronyms?? What the heck is an XKCD??? ;)
mastermage 1 days ago [-]
There is so much wrong with this beginning with the fact that it would be correctly 170 °C (the degree is important because it implies an arbitrary scale)
Additionaly Bake Cake in 170 is very not clear, especially considering you have two major Temperature Scales in use in the Kitchen.
fg137 2 days ago [-]
My search results for "open CTF format" beg to differ. Absolutely nothing even remotely related to "capture the flag".
Bear in mind that Google search results, just like ChatGPT output, are highly personalized and non deterministic, so "it's there if you do a Google search" means almost nothing these days.
In fact, I have no idea what's going on, so I came back to HN comments. Turns out it's "capture the flag" which I actually know, just not familiar with the acronym.
Which is why I am 100% with the top level comment here.
fartfeatures 2 days ago [-]
"bake cake in 170" is genuinely confusing at first glance though.
nimchimpsky 3 days ago [-]
[dead]
doublescoop 3 days ago [-]
Best practice in writing about technical concepts is to spell out acronyms like this on their first use. There is a ton of stuff I learn about here on HN that I didn't know anything about before.
It doesn't help that the linked article never bothers to explain this either.
Aurornis 3 days ago [-]
For a general audience this is good advice.
This article was written for a specific audience who follows this blog because they know the term. If you start spelling out fundamental acronyms it makes the content look more basic and general.
This always upsets the general audience who stumble upon the article (like this) but it wasn’t meant for a general audience. CTF is extremely well known and the people who would be interested in this topic would wonder what’s happening if it was spelled out. It would be so odd that it would probably attract accusations of ChatGPT writing.
jltsiren 3 days ago [-]
It's the common practice in technical writing. Even when you are writing for other experts in the same field, your target audience never shares as much context as you would prefer. The world is much weirder and more varied than you would think.
Informal writing about technical topics is another story. There you can assume a lot more shared context, as you are only writing for a specific subculture within the field. It doesn't matter much if other people in the field fail to understand you.
dolmen 2 days ago [-]
But this isn't "technical" writing.
This is member of a community talking about the state of that community. Jargon is expected. Even more, this what I look for, even as a foreigner.
MobiusHorizons 3 days ago [-]
Does spelling it out help? From memory, it is a security competition where participants compete to gain certain objectives. I think capture the flag may explain how scoring is kept, but it wouldn’t help me find out what it is, given that capture the flag is also just the name of a game people play outside by running, or in laser tag or in certain video games.
bawolff 3 days ago [-]
> There is a ton of stuff I learn about here on HN that I didn't know anything about before.
But that is about you right? Its a little entitled to expect every piece of content on the internet to have a 101 explanation attached. If they were specificly aiming to have the blog post appear on HN that would be one thing, but they (presumably) weren't.
3 days ago [-]
allenrb 3 days ago [-]
When I encounter new terms, I look them up. Just like any other new word. Been doing it since I was a kid with a dictionary. Now, it’s too easy not to. There is literally no excuse.
2 days ago [-]
ergonaught 3 days ago [-]
You could have just said “No”, if you had to say anything at all, rather than continuing the behavior.
Actively rude.
watt 2 days ago [-]
If you think you know what every acronym stands for, I recommend CBT.
razster 3 days ago [-]
What I see CTF I think Capture The Flag, Tribe player in me.
lelandbatey 3 days ago [-]
CTF stands for "Capture The Flag" in the parent article. Just the security competition kind, not the FPS game kind.
Gigachad 3 days ago [-]
The annoying thing is even if you know what it means, multiple groups will use the same initialisms for different terms. So without more context you can’t know what it means.
It isn’t common but I feel it would be best when posting to HN to just expand the initialisms even if the source title didn’t.
shric 3 days ago [-]
You can also over use the same initialism: ATM the ATM is connected via ATM
3 days ago [-]
w23j 2 days ago [-]
CTF stands for Capture the Flag. So with that definition you have exactly zero more information about the article then without it. So I assume next you want a short description of what it actually means, like "CTF (Capture the Flag) are security competitions where the objective is to break...", which is completely ridiculous to include in an article aimed specifically at the CTF crowd.
fartfeatures 2 days ago [-]
It is easier to search for keywords (Capture the Flag) vs acronyms (CTF) which likely resolve to other terms as well. Child trust fund is the first result when I put CTF into Google. Admittedly searching CTF security solves that issue. A quick link to an article on CTF to make the post digestible by outsiders seems reasonable enough.
fragmede 2 days ago [-]
So right click on HCKSYD and Google search that to get there! The rest of us don't want to be treated to infants that need to be spoonful.
xnickb 2 days ago [-]
Why should the author care to make it digestible to the crowd who is clueless on the matter? Their goal is to capture attention and start discussion within the community.
To me it doesn't seem reasonable at all. It's just entitled at best.
fartfeatures 2 days ago [-]
I read it as a suggestion vs a demand but I can see why you felt it was entitled. The author doesn't need to care and can take or leave the advice. In many ways it is a shame we don't have the concept of the Semantic Web as an extension to simple Hyperlinks.
briandear 2 days ago [-]
Good communication should be the goal of all writing.
mr_mitm 2 days ago [-]
You wouldn't spell out URL, SSH or HTTPS on hacker news either. I understand it's frustrating, but you can't accommodate everyone.
xnickb 2 days ago [-]
No one is in a position to dictate individual people what their goals are. That's another example of entitlement.
As an individual author they are entitled to write whatever they want in their blogpost. I as a consumer of their writing am not entitled to anything
wheelerwj 2 days ago [-]
Good communication for who?
thomascountz 2 days ago [-]
...which is completely ridiculous to include in an article aimed specifically at the CTF crowd.
What compels you to hold this belief?
graceful6800 3 days ago [-]
Since this is the top comment at the moment: CTF stands for Capture The Flag.
Personally I have never, ever heard that concept referred to by the initialism. Granted, it's almost never come up in my circles, so... shrug
worble 3 days ago [-]
CTF is a game mode for popular online games like halo (or at least, that's how I know it), so paragraphs like
> My first CTF was HCKSYD, a 48-hour solo CTF. I full solved it and won in 2 hours. I was completely hooked. That led me to win DownUnderCTF, Australia's largest CTF, with Blitzkrieg multiple times. Blitzkrieg was one of Australia's strongest teams at the time. I later joined TheHackersCrew, an international top-tier team that was consistently ranked highly on CTFTime, the main global ranking and event calendar the scene uses as its scoreboard. With them, I competed in some of the most prestigious CTFs in the world, consistently placing well within the top 10 until the end of 2025.
Are still completely nonsensical to even those that understand the acronym
trescenzi 3 days ago [-]
It's also a game people play in person as well. It's the same as the Halo version except you tag each other instead of shooting. It's really fun to play in big open areas with large teams.
deathanatos 3 days ago [-]
Yeah.
As I remember it (and this was decades ago): Two teams, opposite ends of a large field. Each end gets a "flag". (We used t-shirts.) In our case, we split the field in half — our field happened to have a natural feature (a change in elevation, so like two separately flat areas separated by an incline) that worked well for this. If you were tackled¹ in the enemy's side, you were "captured", and "jailed". An uncaptured player could spring the jail by tagging those within it. Returning to your flag with the opposing team's flag was a win.
We played at night, so stealth was a large part of the game, but it was also fair to illuminate the area around the flag. (Which made approaching a guarded flag … tricky.)
I'm sure there's probably a million variations on the specifics.
¹…flag football flags would probably work nicely for this.
RugnirViking 1 days ago [-]
though this is not what the author is talking about. Theyre talking about a hacking competition, where you compete to get a secret word or something contained on some running server connected to a network protected with various means. They're complaining about AI agents removing a lot of the fun from this.
People are mad because we're literally on "hacker" news, so there is some expectation that people might be familiar with hacking or computer security.
acters 3 days ago [-]
Yeah, but we have AI now, we don't need our blog posts to over explain or state what it all means to general audiences.
The author name-drops a bunch of CTF events hosted by a variety of independent organizations and name-drops well-known teams.
To help everyone, this Capture The Flag is specifically Cybersecurity adjacent, there is a Wikipedia article on it as the top Google search result for me when searching "CTF". This is why the acronym is used, because searching for the full will get you to the wrong "sport" vs the cybersecurity one.
I don't want to explain what a CTF is. look at the Wikipedia article. It is there for a good reason.
itsboring 3 days ago [-]
Unreal Tournament and Quake 2 for me.
bawolff 3 days ago [-]
Just to give the actual answer, CTF in this context means a computer security competition. Generally the way they work, is you get some programs, and you have to hack them to get some string called the flag (e.g. maybe the server has a root owned file called flag, so you have to get root somehow to read the file). Team with the most flags at the end wins.
In this context, CTF is almost exclusively referred to by the initialism, i think to help distinguish from other uses of the term.
OrangeMusic 3 days ago [-]
THANK YOU
This has been the most annoying HN so far until your comment appeared.
tptacek 3 days ago [-]
Apart from everything else people have said in response to this, it's rude to presume that an article has HN as an audience simply by dint of it being available for us to link to. It's totally reasonable for people to write for an audience they know understands these terms.
So, in fact, you must not beg to have authors include courtesy definitions for you. That's not reasonable. Instead, you should simply ask here, on the thread, without complaining about the article.
xnickb 2 days ago [-]
Exactly. Especially now in the era of LLMs where you can feed the article to a chatbot and ask it to spoon feed you at the pace you can digest at.
pastel8739 3 days ago [-]
I think so many acronyms have meaning that isn’t explained by the words that the stand for. The other day I was explaining what CI is and they asked what it stood for; I realized that Continuous Integration is almost completely useless for someone trying to understand what CI actually is
cco 3 days ago [-]
Semantic names are great, but that's a separate issue. With the full term you can now go search for yourself and find explanations more easily.
circus1540 3 days ago [-]
“hacker” news, ladies and gentlemen
RealCodingOtaku 2 days ago [-]
The comments are annoying. No matter the niche, it is always good to write the abbreviation the first time it is used, in fact, W3C recommends it[0]. Anyone who does not follow this either are not informed well enough, or has ableism. Most replies in this thread shows the latter.
This page has 5 abbreviations that are not spelled out before you even get to the body of the article.
RealCodingOtaku 2 days ago [-]
You meant 5 acronyms, and each are links, which contains their abbreviations. So they do follow the guideline.
ajam1507 21 hours ago [-]
An acronym is an abbreviation, and they specifically do not spell out the very first acronym on the page the first time they use it.
fragmede 3 days ago [-]
We live in the goddammed future. Huamnity's knowledge is at your fingertips. Right clicking the Nth word of the article and putting in any semblance of effort to learn on your own is too much to ask?
I don't know everything, there's tons of stuff I don't know about, but when I'm at my web browser, the least I can do about something is ask Google about a word or phrase or subject that isn't familiar instead of being spoonfed information like I'm a baby.
fullstackchris 2 days ago [-]
There are also widely accepted standards to written word.
The best example is when an abbreviation can be expanded to more than one phrase, and both are widely used.
wglb 2 days ago [-]
If you wish to explore a new concept, perhaps ask duckduckgo "what is ctf" an you may well get a response such as:
CTF stands for Capture The Flag, which is a cybersecurity competition where participants find hidden "flags" in vulnerable programs or systems to test and develop their security skills. There are two main types of CTFs: jeopardy-style, where teams solve various challenges for points, and attack-defense, where teams defend their systems while trying to exploit their opponents'.
fg137 2 days ago [-]
As someone who spent years reading academic journals where spelling out acronyms on first occurrence is mandatory, the comments here are jarring. Think about it -- those are academic journals where most people do know what those acronyms mean.
It's such a small but immensely helpful thing to do.
kortex 2 days ago [-]
This isn't an academic journal or general purpose news article though. It's a small blog about a subculture. It'd be like spelling out GPU on a blog for building custom PCs, or BDSM on a kink blog.
Spelling it out when 99.9% of your audience doesn't need it actually is the opposite of in-group signaling, it makes it feel like it's aimed at a wider audience, when it's not.
I didn't know what BGP is, but I did know CTF. YMMV
snayan 3 days ago [-]
Hah, same. Always entertaining how people map their tendencies to the world at large.
jere 2 days ago [-]
It's kind of amusing that you're asking for acronyms you don't know not to be used and acronyms you do know to not be spelled out.
amirhirsch 3 days ago [-]
Let’s reduce this to absurdity:
I think you only wanted clarification of CTF (Capture the Flag) and not AI (Artificial Intelligence) and not GPT-4
(Generative Pre-Trained Transformer version 4) and not CLI (Command Line Interface) and not MCP (Model Context Protocol) and not LLM (Large Language Model)
Quoting TFA (The Fucking Article): “just adapt bro”
lol at the BGP example
w23j 2 days ago [-]
For people who are confused by this comment: lol = laughing out loud!
Den_VR 2 days ago [-]
Lots of love!
amirhirsch 2 days ago [-]
ROTFLMAO made my day!
ajnin 3 days ago [-]
Your two paragraphs are completely contradictory. I agree with the first one.
plaguuuuuu 2 days ago [-]
you could try
* googling "CTF security"
* asking literally any AI to explain the article
Yes, you must beg. If you don't know what a CTF is, and don't want to find out, why read the article anyway
fg137 2 days ago [-]
I had no idea "security" is the keyword to add, even after reading the leading paragraph and the first paragraph under "What makes me qualified to say this?".
In fact, I know what "capture the flag" is but am not familiar with the acronym. Still, the article confused the hell out of me, so I came back to HN comments for more context.
> Asking AI to explain the article
That's how we are expected to consume content these days?
watwut 2 days ago [-]
Using AI to get factual information?
Also well written articles allow you to figure out what they mean after clicking onto them. People often read to learn.
alsetmusic 3 days ago [-]
At the same time, I did a search for "what is a ctf to play" and got the answer. We know how to find answers to these problems. I agree the blog post was poor form.
3 days ago [-]
toofy 3 days ago [-]
i try not to over feed tangents but this is precisely how i feel every time i speak to someone who is recently enlisted in the military. i have to constantly stop them and be like “i have no idea what you just said” over and over and over again. it’s like trying to make sense of a random bowl of alphabet soup.
Drupon 3 days ago [-]
[dead]
george916a 3 days ago [-]
[dead]
baq 4 days ago [-]
Replace ‘CTF’ with ‘high school’ or ‘university’ and you’ve described the total slow motion collapse of education; the only saving grace is that most of it requires in person presence.
We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part. LLMs can be wonderful teachers, but the temptation to just tell it ‘do it for me’ is almost impossible to resist.
jaybrendansmith 3 days ago [-]
Everything we've learned in the last 10 years is telling us that computers do not help human education in the slightest. We remember better when we write with pen and paper. We learn better with whiteboards and paper books. The simple answer: Remove most computing from education entirely. Blue composition books, pencils, whiteboards is what trains humans. Calculators are helpful perhaps but it is quite possible that slide rules are better. We need humans that can critically think from first principles to counter the recycled information generated by AI.
skulk 3 days ago [-]
> computers do not help human education in the slightest
I had no access to anyone who could teach me calculus as a kid except Khan Academy, so I think this is a gross exaggeration. But I agree in the end, that all my "real" learning did come from pen-and-paper practice, not watching videos.
voxl 3 days ago [-]
The reality is that a human will learn, given any materials including LLMs, but only if they truly desire to learn. We've had MOOCs, gigantic libraries, all full of free information. You can obtain a PhD level understanding in any technical field of your choice today just by consistently going to the library and consistently applying yourself.
It's not unlike going to the gym, and we see how many people do that regularly. Except it's even funnier, because people serious about the gym but what? Tutors. They call them personal trainers. We've known for a millennium or more that 1-on-1 instruction is vastly better than anything else, but most people actually don't want to get into shape, and most people actually don't want to learn.
globnomulous 9 hours ago [-]
> The reality is that a human will learn, given any materials including LLMs, but only if they truly desire to learn. We've had MOOCs, gigantic libraries, all full of free information. You can obtain a PhD level understanding in any technical field of your choice today just by consistently going to the library and consistently applying yourself.
Not true. In every field there is guild knowledge that a person can't acquire from a library. In technical disciplines PhD-level knowledge requires experience in collaboration, research, and frequently lab work, which is impossible to acquire without access to a lab -- or just direct experience with research methods, whatever those may be. Reading papers and absorbing information aren't enough. PhD-level knowledge comes from the process of writing and doing original work.
> The reality is that a human will learn, given any materials including LLMs, but only if they truly desire to learn.
Also not true. We require kids to go to school partly because exposure to the environment and work inculcates skills regardless of whether kids want to do the work -- and regardless of whether they want to learn.
LLMs are damaging to students partly because they provide an escape hatch from that work and thereby prevent kids from acquiring skills.
Think of it this way: most people who want to be healthy and eat a healthy diet still find easy junk food tempting. What they want does not change the temptation, because the body and brain gravitate towards easy, cheap fulfillment of basic drives.
People facing challenging tasks, similarly, are tempted to take measures that reduce the amount of effort they require. The availability of tools that reduce the required effort also help shape a person's understanding of the value of the challenge and the work: "why should I do this hard task when I have a tool that can do it for me?" You and I know the answer to this question when we're discussing something like writing an essay or solving a problem in a math or programming class. Students frequently don't. They are by definition ignorant. Children, moreover, lack maturity. Their brains are less capable of resisting the easy path than an adult's. That's partly why parenting is important: parents provide boundaries and limits that kids need but won't and can't provide for themselves.
Sometimes people, especially kids, really do need to be dragged, kicking and screaming, through something in order to receive the benefits it offers. Being dragged through it sometimes convinces a person of its value and benefits. In a kid's case, there's a decent chance that the experience will improve executive function, shape expectations in a healthy way, inculcate grit, and become appreciation -- or at least habit.
I would not have written essays on my own as a student in secondary school. My English teachers had to provide that structure for me and impose the demand. But LLMs make it much more difficult to impose the demands, and kids are ill protected against the temptations of the cognitive equivalent of junk food, but an order of magnitude worse and more damaging.
voxl 4 hours ago [-]
If you've never taught students or mentored PhD students please refrain from diatribes in my comments (and yes I've done both)
tannertech 3 days ago [-]
The annoying thing is a PhD level understanding does not get you jobs.
fragmede 3 days ago [-]
I don't have a PhD, but "you're overqualified" is something I've heard my PhD having friends said to them.
whichdan 3 days ago [-]
Yeah I agree. I grew up in a very blue-collar town, and anything I wanted to learn (outside of public schooling) either came from emaciated websites or whatever books I could find at the library. Having YouTube and Khan Academy and everything else would have made such a huge difference for me.
selimthegrim 3 days ago [-]
Now I’m wondering how a website is emaciated
Wolfbeta 3 days ago [-]
One simply forgets to hydrate.
bentaber 3 days ago [-]
Not enough bytes?
fragmede 3 days ago [-]
Not even a nibble!
mrandish 3 days ago [-]
> except Khan Academy
But that's not using "computers" as a computer but as a video player. When evaluating whether computers are "good for learning", I don't think we should include using a computer as a video player, a book, or even flash cards. It should be things a computers uniquely offer which a books, paper, videos and a physical reference library cannot.
Based on the results of deploying hundreds of millions of computer to schools in the 80s and 90s, the evidence was mostly that computers are good for learning computer programming and "how to use a computer" but not notably better than cheaper analog alternatives for learning other things.
Interestingly, a properly trained and scaffolded LLM could be the first thing to meaningfully change that. It could do some things in ways only human teachers could previously since it is theoretically capable of observing learner progress and adapting to it in real-time.
famouswaffles 2 days ago [-]
I think videos are a unique thing computers offer. Books I understand. You have them digital or not. But a video ? Without a computer, there is no video. You were present for the initial lecture or you weren't and that's it.
tuna74 2 days ago [-]
There were videos* before computers.
*Not really, but you could film stuff and display it.
bobdvb 8 hours ago [-]
Film is a chemical medium for storage of images.
Video is an electronic process for capturing images and displaying them.
Before digital video there was analogue video, and analogue video was perfectly possible without digital sampling, or computers. Heck, video pre-dates silicon chips and used to be done with CRTs and valves.
mrandish 1 days ago [-]
> Without a computer, there is no video.
There's nothing about video that uniquely requires computers. Maybe you meant "streaming video"?
I realize you're probably under 30 and don't remember "ye olden times" but nearly 90% of U.S. homes and every school had an analog VCR long before they had a computer. Widespread consumer video formats included VHS, DVD, Blu-Ray, Laserdisc, etc. I still buy some movies I care about on UHD discs and watch them in a dedicated Blu-ray/UHD player. Even the 'smart' TVs and streaming sticks most people watch streaming channels like Netflix on aren't functionally computers (no meaningful user accessible local storage, input like keyboard/mouse, CLI or windowing GUI).
Personally, I learned an enormous amount from video before I ever touched a computer. In elementary school we learned from 16mm films almost weekly and watched space launches and Carl Sagan's Cosmos series on TV (it was rebroadcast in the mornings specifically for schools). My junior high had a television in every classroom and some classes were planned around shows on PBS, NASA channel, C-SPAN and BBC. In the late 80s there was thousands of hours of educational video programming sent via direct broadcast satellite to 18-inch dishes at schools. In the 90s every grade and subject had hundreds of interactive video DVDs in large notebooks (four discs to a page in plastic sleeves) and multiple DVD players per classroom.
The peak installed base of VCRs in the U.S. was in 1999. Streaming video wasn't common in consumer households until well into the 2000s, YouTube didn't even exist until 2005 and most people had never heard of it until 2007. In 2010 Netflix mailed DVDs in envelopes to 25M homes every week. They didn't even offer a streaming plan until 2011.
As someone who's spent most of my adult life thinking about video technology, with patents ranging from analog days to the streaming tech you use today, computers have been extremely disappointing in terms of enabling any unique "learning from video" features that are computer-specific. In the 90s we realized that computers could make digitized video random access letting us sequence it non-linearly to make it interactive in response to user input. We knew that computer-enabled interactivity, responsiveness and real-time adaptation to learner progress would be incredible for improving video education. Yet the vast majority video content available online today is still linear in form. Even video that's specifically educational is no more interactive or user responsive than a 90s DVD disc.
Sadly, only two things have really changed about consumer video in the last 30 years: quantity and distribution. There's much more video content and it's remotely accessible on-demand instead of being limited by broadcast channels and storage media. But that's far more about communication technologies like broadband than computer technologies. For a few years YouTube even had authoring features like interactive menus and conditional branching but removed them because it didn't increase ad revenue. There are a few dedicated video authoring platforms for education which can apply uniquely 'computer things' to video like dynamic scripting, conditional branching, viewer annotation and timecode-linked threaded Q&A. Unfortunately, such content is rarely found outside high-end corporate training and some university courses. But there are so many other ways we could combine the strengths of advanced wikis with interactive video. Today, the most the public sees is just an HTML link from a wiki to a video clip. Almost none of the learning features computing could uniquely bring to video are widely available to learners. Since ~90% of everyone already had access to linear video playback before they had access to a computer and most online video today is still primarily linear, in my opinion, there's still virtually no uniquely 'computer-enabled video' involved in learning. Computers haven't enabled much that's new in video - just much more, much cheaper and more convenient forms of what we could already do without a computer.
bobdvb 8 hours ago [-]
As a broadcast technology architect I agree with you a whole lot on the broader technology statements.
But as a former lecturer, I also think the promise of interactivity is dependent less on the tools than on the people. Authoring interactive learning materials is difficult and while that interactivity is engaging, it's not necessarily great at getting a density of information out there.
The Socratic method is great, but that level of interactivity presumes in advance that you know what questions the student will be asking, otherwise it's just a dumb gate. Branching stories for interactivity are highly labour intensive. I suppose if you use AI you could generate a massive number of videos to cover branching learning, but that's going to still be an intensive operation, especially if you're supervising that.
whatever1 3 days ago [-]
Khan did not throw at you a 100-slide Powerpoint deck in 45'.
He really took the time to replicate the manual teaching process of writing on whiteboard. He improved upon it by using colors. But basically had the same pace as a teacher writing on a whiteboard.
When professors are given a projector, they just throw together some slides and add their narration.
This is not very efficient. To learn you need to suffer. Or you need to watch the suffering.
jedimastert 2 days ago [-]
That's not really a computer helping you though, that's just a computer allowing a human that's far away to help you right?
allan_s 3 days ago [-]
I think what the author meant is that it does help not more than the same knowledge provided the old way.
rossjudson 3 days ago [-]
Every child reads a book about solving problems, assumes they can now solve problems, and is disappointed when that is not true.
__MatrixMan__ 3 days ago [-]
I think this overlooks the potency and scarcity of 1:1 time with the teacher. If you've only got maybe a few minutes of that in an average schoolday there's a huge difference between whether or not you've talked it through with an AI before trying the question out on the teacher.
They're wrong sometimes, but usually in verifiable ways. And they don't seem to know the difference between medicine and bioterrorism, so often they refuse. But these limitations are worth tolerating when the alternative is that our specialists in topic X are bogged down by questions about topic Y to the point where X isn't getting taught.
kelvinjps10 3 days ago [-]
And now they'll have less time because they will be bombarded with slop to no end.
__MatrixMan__ 3 days ago [-]
Obviously generating your homework is a bad idea, and maybe assigning homework that can be generated is a bad idea. But neither of those are relevant to the problem I'm talking about which is about due diligence prior to asking for somebody's extended attention.
Whether you're in class or at work, it's just courteous to ask an AI first.
tempaccount5050 3 days ago [-]
Nah, I wrote physics programs on my computer at home in high school and it absolutely helped with my schooling. Yeah, maybe iPad apps aren't the best things in schools but you're throwing the baby out with the bathwater. Computers bad is simply not true.
peter-m80 3 days ago [-]
I learned calculus thanks to wolfram alpha step by step solving feature
ralph84 3 days ago [-]
> humans that can critically think from first principles
This has never been achieved by, nor is it the point of, education for the masses.
jaybrendansmith 1 days ago [-]
But it should be, right?
jaybrendansmith 3 days ago [-]
I'm not going to disagree with step by step videos ... those are a HUGE help. I'm really talking about solving problems using pen and paper, whether math or writing, is how my problem-solving patterns actually changed.
3 days ago [-]
wslh 3 days ago [-]
I would start saying that many people need presence in a real environment with people to learn. We don't use all our senses in a remote environment.
PunchyHamster 3 days ago [-]
I disagree with that statement. There is nothing inherently wrong with using computer to learn and if your personal goal is to learn it in lot of cases makes it much easier, whether to search for or visualise a piece of knowledge you're' learning.
The problem is frankly computer and now computer with LLM makes it easy to cheat.
The kid doesn't want to learn, the kid wants good grades so parent is happy with them, and the young adult wants to get the paper coz they were told that is required for good life. It's misalignment of incentives.
sometimelurker 3 days ago [-]
I don't think computers automatically make us more educated, but if you want to make a point don't use reductive exaggerations.
> We need humans that can critically think from first principles to counter the recycled information generated by AI.
I agree with this.
Gigachad 3 days ago [-]
We are interviewing for a software dev role and we made the first round in person to prevent cheating. The gap between people who learned pre ai vs post is immense. I had a dev with supposedly 3 years experience and a degree in software who wouldn't have been able to write fizzbuzz without AI.
IanCal 3 days ago [-]
Can’t say you’re wrong but the last anecdote describes many I’ve had to review for jobs long before LLMs. Fizzbuzz is a classic thing that shockingly many devs genuinely cannot do, even at home.
sigmoid10 3 days ago [-]
Yeah, I've interviewed people like this 15 years ago. Degrees and experience mean nothing in this field. The best predictor I found was personal passion projects. Let them get as nerdy as possible, then you will see pretty quickly where their skills are at and what their limits are. And you will immediately filter out people who just studied CS because they heard you can make good money.
wookmaster 3 days ago [-]
Completely agree with this, leetcode has become such a business now of memorization for interviews it’s useless to know if someone memorized a solution or not.
virgilp 3 days ago [-]
you can absolutely know. they do suspiciously well. you just give harder problems until they can't solve it. how they react/approach a problem that they can't immediately solve _is_ the interview - not the "how many things they solved correctly" part.
That said - I seldom need people to be hardcore algorithm solvers
What I typically did was a variation of fizzbuzz (can the candidate code very basic logic?) and then finding a bug or minor requirements extension in their online screening test/"homework" and asking them to solve that on the spot (did they write the code themselves/can they modify it). It's typically enough, there's diminishing returns to test more in-depth the programming skills - the rest you can discuss domain knowledge, general experience, working style etc.
jhbadger 3 days ago [-]
Maybe. There are certainly people in all fields who are book smart and did well in classes but are useless at actually practicing their field (not to mention people who cheated in school and got away with it and aren't even that), and it is worth filtering them out. But I think it is weird that CS expects good workers to have these passion projects. Do we expect civil engineers to build bridges in their back yard on the weekends? Can't someone just be good at their job and have other interests outside it?
sigmoid10 2 days ago [-]
I imagine this is simply not such a problem in other fields. Or do civil engineering schools produce that many clueless graduates? I know other engineering fields don't pay bad, but software is another realm.
ganzsz 3 days ago [-]
I can passionately tell about professional projects.
gedy 3 days ago [-]
I agree, however there are so many interviewers who will still treat that as some softball criteria and insist that unless you "prepare" for an interview by memorizing leetcode you are 100% a faker and liar.
jadar 3 days ago [-]
Maybe they themselves are fakers and liars / deeply insecure. I got bumped out of an interview rather rudely once because I blanked and couldn’t answer a trivia question about arrays.
Gigachad 3 days ago [-]
Something that is for sure new is the AI interview cheating tools which listen in on the call and provide answers in an overlay invisible to screen sharing. The only way to deal with it would either be invasive spyware on the applicants computer or asking them to do the interview face to face.
nsvd2 3 days ago [-]
Spyware wouldn't help at all because you could just put the AI between the computer and the monitor, for example, or use a VM.
arthens 3 days ago [-]
A relatively low tech solution could be to give them 2 separate conferencing links, ask them to join each one from a different device, and have the secondary device point the camera and the screen of the primary device.
Gigachad 3 days ago [-]
Easier to just get them to come in. Which also has the effect of filtering out people pretending to be in the country but aren’t.
unrealhoang 2 days ago [-]
And they can have an alternate screen outside the FoV of the second camera.
josh2600 3 days ago [-]
Why is it important that a dev can’t do fizzbuzz without ai?
If they can ship code that matches a spec, why does it matter if they’re using ai or not?
Genuinely curious.
ekidd 3 days ago [-]
> If they can ship code that matches a spec, why does it matter if they’re using ai or not?
I am perfectly capable of writing specs, and feeding them to 3 separate copies of Claude Code all by myself. Then I task switch between the tmux windows based on voice messages from the pack of Claudes. This workflow is fine for some things, and deeply awful for others.
Basically, if a developer is just going to take my spec and hand it to Claude Code, then they're providing zero value. I could do that myself, and frequently do.
The actual bottleneck is people who can notice, "The god object is crumbling under the weight of managing 6 separate concerns with insufficient abstraction." Or "Claude has created 5 duplicate frameworks for deploying the app on Docker. We need to simplify this down to 1 or we're in hell." I will happy fight to hire people who can do the latter work. But those people can all solve fizzbuzz in their sleep.
People who just "ship code that matches a spec" without understanding the technical details are providing close to zero value right now.
There is an interesting niche for people with deep knowledge of customer workflows who can prompt Claude Code. These people can't build finished products using Claude. But they can iterate rapidly on designs until they find a hit. Which we can then fix using people with deeper engineering knowledge and taste.
But if you're not bringing either deep customer knowledge or actual engineering knowledge, you're not adding much these days.
freedomben 3 days ago [-]
> Then I task switch between the tmux windows based on voice messages from the pack of Claudes.
I also use Claude with tmux. Can you share how you get the voice messages from the Claudes?
ekidd 3 days ago [-]
Tell Claude you want to set up notifications, using "hooks", including "Notification" and "Stop" and anything new they've added. Claude can figure out how to do this for your operating system.
It's not perfect—sometimes a Claude notifies 3 minutes after it stopped doing anything. But it's helpful when I'm running multiple Claudes and also reviewing code elsewhere.
Your brain may feel like someone put it in a blender. Be warned.
IanCal 3 days ago [-]
Fizzbuzz is such an incredibly simple problem if you can’t do it I struggle to see how you’d be able to complete any task that requires very basic reasoning and very basic coding knowledge. And if an AI system can do those parts, what am I getting for spending tens of thousands of pounds per year by hiring a person who can’t? Wouldn’t I just tag codex on the tickets?
I’m not talking about gotcha level stuff here where the first time it didn’t compile because of a bracket or anything, or even first time wrong. They couldn’t do Fizzbuzz in a language of their choice, at all.
Those that could were always annoyed at having to do such things because how could someone coming for a contract position not be able to do this? Without seeing what a filter it really was.
eudamoniac 3 days ago [-]
I feel the same way about inverting a binary tree, but a lot of people act like it's an arduous request. I am guessing it's because they've never read the description of what inverting a binary tree is, but maybe people are just that bad at recursion.
krapht 3 days ago [-]
You can go your entire career without recursing, or using a tree data structure in its raw form (i.e. you only use it as part of a library)
marshray 3 days ago [-]
Right. For the first many decades of computing, recursion was just always the wrong answer for a production software system. (Feel free to provide a counter-example, but please begin with an explanation of how the size of a call stack frame is determined and how exceeding the base allocation is handled on this platform).
So what tree-traversal/quicksort problems tend to measure is how long it's been since you last did CS class homework problems.
saagarjha 2 days ago [-]
There's no need to put your data on the call stack.
marshray 1 days ago [-]
Great. Please explain how the size of a call stack frame is determined and how exceeding the base allocation is handled on the particular platform you're proposing to recurse upon.
saagarjha 23 hours ago [-]
I feel like you have not responded to my comment.
ptrl600 3 days ago [-]
Yeah, you read it and expect the inversion to be inside-out or upside-down, defined in some hopelessly arcane way.
Octoth0rpe 3 days ago [-]
> If they can ship code that matches a spec, why does it matter if they’re using ai or not?
The inability to write fizzbuzz strongly implies their inability to understand what they've shipped. Review is some significant portion of the job. Understanding of the product is also part of the job.
Specs are also in a sense, scaled down, fuzzy, natural language descriptions of a feature. The fuzziness is the source of a bugs, or at least a mismatch between the actual desired feature and what was written down at spec writing time. As such, just matching a spec is just the bare minimum that a good dev should be doing. They should be understanding what the spec is _not_ saying, understanding holes in their implementation, how their implementation enables or hinders the next feature and the next, next feature, etc. I don't think any of that is possible without understanding what was actually implemented.
tardedmeme 3 days ago [-]
For the same reason it's important your mechanic can identify which parts of a car are the wheel.
Who cares as long as the car is fixed, right? As long as the mechanic can Chinese-room his way to a working car, why does it matter how much of it he actually understands?
And why hire the mechanic instead of hiring the Chinese room?
unethical_ban 3 days ago [-]
First: FizzBuzz is a test to know if you understand the most basic constructs of programming. The kind of thing you learn in the first week of CS101. I forgot what it was, and when I looked at the problem I knew the answer.
More broadly: In the short/medium term, we still need humans who have the skills to understand software largely on their own. We will always need those who understand software engineering and architecture. Perhaps in 25 years LLMs will be so good that learning Python by hand will be like learning assembly today. But not yet.
The field is not ready for new practitioners to be know-nothing Prompt engineers. If we do that, we cut the legs out from under the education pipeline for programming.
gilrain 3 days ago [-]
If they’re not a value add over the base AI, they aren’t worth hiring over just using the base AI.
anigbrowl 3 days ago [-]
Why hire them at all then, just ask them what their favorite AI is and use that
fragmede 3 days ago [-]
Because I'm busy already doing that and need a copy of me/close enough to one, to do more of that.
unrealhoang 2 days ago [-]
So not being able to write fizzbuzz is nowhere close to me.
xfax 3 days ago [-]
To understand the code they are shipping requires some level of proficiency. Their inability to do fizzbuzz without AI calls that into question.
jaredklewis 3 days ago [-]
If you can’t even write a for loop, how can you verify the ai code you generated isn’t going to wipe the prod database?
jadar 3 days ago [-]
It’s about deeply understanding what you’re doing. Like as a kid before you knew how to ride a bike, you could sit on a bike and peddling, but until it “clicked” you couldn’t balance and keep going forward stable. Fizzbuzz tests your ability to reason through a problem that seems simple on its face, but is easy to get wrong and/or overthink.
koliber 3 days ago [-]
How will you know that it produced correct code if you don’t know how to write it yourself?
varenc 3 days ago [-]
I can see this perspective, but FizzBuzz is such a low bar that so many can pass, I'd greatly prefer to hire someone that can ship code that matches a spec do this challenge.
hnthrow0287345 3 days ago [-]
It doesn't. It's just a low-end skill filter that got really popular. It could have easily been replaced by other tests like is this word a palindrome.
marshray 3 days ago [-]
I wrote the "function to reverse a string" in a job interview once. Then the interviewer reminded me that strrev() had been part of the standard C library since K&R.
I'd been programming in C(++) for ~15 years by then and had never had the occasion to reverse a string. I still wonder whether that makes it a good job interview question, or a terrible one. Some of both probably.
unrealhoang 2 days ago [-]
It’s a good one, if you can still write functionally and same algorithmic complexity then it should not matter if you know strrev or not.
marshray 23 hours ago [-]
Except that an extremely complex algorithm involving large data tables is required for Unicode or other modern text encodings.
saagarjha 2 days ago [-]
strrev is not a standard C function. I actually don't think my computer even has it (it's a Mac).
marshray 1 days ago [-]
Well I don't feel so bad for not having known it then. Thanks!
saagarjha 23 hours ago [-]
Yeah I didn’t know about it until you mentioned it either
eastbound 3 days ago [-]
And yet, some people argue that you shouldn’t ask a developer to align 3 “if” and 1 “for”!!!
The energy spent arguing that those 4 instructions in a row “are not a mark of someone who can write code” would have better been spent firing them.
hnthrow0287345 3 days ago [-]
Firing people is problematic. I'd be okay with it if the economy wasn't utter trash. It's way better to do the work upfront and prefer false negatives over false positives.
Even better would be if we had a well-respected credential, so both employees and employers can both avoid these long interview loops. I'd much rather get hazed once in a big way than tons of little hazings over a life time.
dabbledash 3 days ago [-]
If the job does not require a person to be able to fizzbuzz, it probably doesn't require a person at all.
hack1312 3 days ago [-]
If you can’t do fizzbuzz without AI you have no business being in this career.
Retr0id 3 days ago [-]
> I had a dev with supposedly 3 years experience and a degree in software who wouldn't have been able to write fizzbuzz without AI.
If you remove the "without AI" and the end, I've been hearing similar anecdotes about fizzbuzz for years (isn't the whole point of fizzbuzz to filter out those candidates?)
raincole 3 days ago [-]
Because "the next generation is ruined" is always a popular sentiment. It has been with us for at least two thousand years, and it surely won't go away in our lifetime.
When this AI era's devs grow older they'll complain the newer generation can't even vide code too.
delfinom 2 days ago [-]
....Or you know it's actually true some of the times. Standardized test scores have bombed hard across the US in the last decade due to smartphones being wildly present in schools without control. Kids brains are legitimately rotted by a machines running software maximized to destroy the attention centers of their brains for life.
AI is just the icing on the cake. These kids are so cooked with developmentally stunted brains that they are forced to use AI as a crutch to function.
brookst 3 days ago [-]
I remember when everyone bemoaned the kids not knowing assembly language. How can anyone understand software if you don’t know assembly?
“Kids these days don’t work as hard / know as much / value the important things” is as tired as it is universal.
jkubicek 3 days ago [-]
OK sure, but back when old heads were complaining about the kids not knowing assembly, those same kids knew C or Fortran or something.
In 2026, if you call yourself a developer and can't solve FizzBuzz without help, it's hard to argue that you know anything useful at all.
brookst 3 days ago [-]
Do modern languages and compilers count as “help”? Because I could probably do fizzbuzz in x86 assembly, but it would take a while to page that back in, and I suspect most people who call themselves developers today simply could not do it without help.
thaumasiotes 3 days ago [-]
> I could probably do fizzbuzz in x86 assembly
How? Fizzbuzz requires you to produce output; that's not functionality that CPU instructions provide.
You can call into existing functionality that handles it for you, but at that point what are you objecting to about the 'modern language'?
rightbyte 3 days ago [-]
You'd just call printf from assembly by knowing the ABI by heart.
brookst 3 days ago [-]
Well I could certainly assemble the string buffer. And if I can run dosbox, I can output to the screen buffer at 0xB800.
I’m not objecting to modern languages, I’m just saying that using them fails the “can write fizzbuzz with no help” test to only a slightly lesser degree than using AI tools. They’re a complex compile- and runtime environment that most developers don’t truly understand.
tpm 3 days ago [-]
> How can anyone understand software if you don’t know assembly?
I'm genuinely curious how someone who never wrote a program in assembly, or debugged a program machine instruction by machine instruction, can really understand how software works. My working hypothesis is most of them don't and actually it's fine because they don't need it.
marshray 3 days ago [-]
"Assembly" is just another virtual machine instruction format sitting atop another, mildly better-hidden, pile of abstractions.
tpm 3 days ago [-]
Depending on the particular processor.
unethical_ban 3 days ago [-]
The time may come when we can treat regular programming as a lower layer niche field the way we treat assembly today.
I don't think we're close to that time yet. Just like as a kid I was told to prove my work by hand even if I could do it in my head, and just like we learned how to do calculus without a calculator and then learned how to use the calculator to get the same result, I think we still need the software field to learn programming concepts independent of the use of AI to create code.
I don't think you can be a good "prompt engineer" for solid software in 2026 if you don't understand programming concepts and software architecture and flow.
brookst 3 days ago [-]
I generally agree, but it’s just a matter of time, and even today people with domain expertise in other areas (accounting, weather, etc) are producing adequate tools using nothing but prompt engineering. Many caveats of course, but I still think 90% of the distaste for mere prompt engineers comes from “kids these days; my unique knowledge is irreplaceable and they don’t even value it” thing.
rrvsh 3 days ago [-]
Adequate for what/who? I can 3d print and cobble together a lock for my bedroom door but I would never be able to work as an engineer producing real locks.
Gigachad 3 days ago [-]
While this is true, it seems undeniable that if you use AI to do everything for you, you will never learn the skills. I'm seeing a massive amount of developers submitting stuff for review and admitting they have no idea how it works and they just generated it.
GrinningFool 3 days ago [-]
Some percentage of developers before AI were unable to code fizzbuzz. Some significantly higher percentage of them are not able to do so now.
Saying there have always been bad developers doesn't change that there's a higher ratio of them now.
No stats to back this up. Just interviews I've done recently and historically.
andai 3 days ago [-]
That's actually the origin of FizzBuzz! A puzzle invented to weed out the perplexing multitude of CS graduates who apparently cannot program.
I developed for 15 years. I don’t think I can do with AI anymore. Why would I even want to do that? It’s like telling a car driver to build an engine.
delecti 3 days ago [-]
It's more like asking a driver the laws for when traffic lights are out. It's not something that comes up often, but it's not completely outside the scope of the task either (I arguably don't even drive a car that has an engine).
JambalayaJimbo 3 days ago [-]
As a car driver, you should understand a little about how your car works. What if you get a flat tire? At the very least, you should know not to drive on that flat tire.
Software is full of leaky abstractions
mannanj 3 days ago [-]
I first did fizz buzz about 10 years ago fresh out of college. Now, after 10 years in full stack and fully vibe coding, I forgot basic python syntax. An interview like yours would have false positives if you are checking for syntax because well, its like looking up spelling, I just ask the AI for the syntax inline.
12_throw_away 3 days ago [-]
> I forgot basic python syntax
If you cannot write "basic syntax" for any language then you are not a programmer, and certainly not a software engineer? This is not a value judgement, it's ok (probably good tbh) to not be a programmer. But you are wasting everyone's time by interviewing for a programming position in this case.
SquareWheel 3 days ago [-]
Personally, I forget syntax all the time. There's always a warm up period after I switch languages, and it takes me longer to be start writing good, idiomatic code.
Like sure, I can probably write some python, but will it be pythonic? I might still be Java-minded for a while, trying to OOP my way into solutions.
Earlier today I needed to write some PHP and couldn't remember if it used length, count, or size. I had to look it up. I've been doing this for 20 years.
funimpoded 3 days ago [-]
Same, I can't pass any test that relies on getting syntax correct. If you want me to fizzbuzz on a whiteboard in a language I've been writing dozens or more of lines of per day for a year up to and including the day before, and require that I don't mess up the syntax, I reckon I've got a coin-flip chance of passing at best (meanwhile, sure, of course the actual logic of fizzbuzz isn't tricky for me)
I once got the method invocation syntax wrong for PHP in an interview. I'd written thousands of lines of PHP and had most-recently written some the week before.
This, despite starting off my programming journey in editors with no hinting or automatic correction. If anything, I've gotten even worse about remembering syntax as I've gotten better at the rest of the job, but I was never great at it.
I rely on surrounding code to remind me of syntax and the exact names of basic things constantly. On a blank screen without syntax hints and autocompletion, or a blank whiteboard, I'm guaranteed to look like a moron if you don't let me just write pseudocode.
Been paid to write code for about 25 years. This has never been any amount of a problem on the job but is sometimes a source of stress in interviews and has likely lost me an offer or two (most of the sources of stress in an interview have little to do with the job, really)
mannanj 6 hours ago [-]
Right, so I wonder, have you noticed or found or experienced any better interview processes? I wonder how we can't just filter out companies that require us to do this sort of hand waving and party tricking behavior or like "because we can't figure out a better way to do this". I reckon maybe a simple filter statement before any interviews, to the recruiters "Hey I don't do xyzzy" would help - though working on the tuning of the language.
I've also started requiring minimum of $300/hr compensation for interviews of my time, newly, so far no success though I'm fine with dying on that bridge or requiring at least a new type of interview process.
AlexeyBelov 2 days ago [-]
Right, but on interviews I've been on it's never the goal to test _exact syntax knowledge_. It's not hard to distinguish between someone who just can't program vs. someone who has knowledge of many languages and has a bit of a mish-mash in their head in an interview setting.
There are people who just can't program for whatever reason, regardless of whether they could previously. And they constantly try to interview at a programming position.
mannanj 6 hours ago [-]
Not true. I can clearly program, have been programming and remembered syntax for about 10 years, and during that 10 years saw the rise and application of tools like IDE and AI improving and accelerating the experience.
As I have been using the AI natural language-as-an-interface coding tools I have gone into the IDE to actually write code a lot less. I read more; and reading is not the same skill as writing. I couldn't remember the syntax as much and by what I mean is some of the simple things like in python to iterate over an object is it an iteritems() or is there a dot between or things like that, and can I do a for index, key in array, and do I need to do like array() to do that. And this is because I always did used to code between languages and alternate fast between javascript, typescript and python and earlier in my career I used to have to remember this (because googling coudl take too long), over time and now I can easily have autocomplete and even AI llm tell me, so I don't remember or waste my energy remembering what the exact syntax is. In my head it's just "I know I need to loop through this, do it in the language Python wants to hear" and why would I bother remembering? so naturally I've forgotten the simple things.
I could clearly, definitely do the programming position, the only ones wasting time IMO are the ones checking if I can remember verbatim things like "spelling" when we all live in a day and age of spellcheck and tools. No one wastes time trying to remember for all words because it's silly. Or maybe encyclopedias and trivia games are the better analogy, sure it's a party trick, but how does it help you do the job better? it arguably does not, and the ones who have not learned to adapt their interviews to the tools are the ones wasting time.
jaredklewis 3 days ago [-]
Which part of the syntax for fizzbuzz can you not recall from memory? The for loop? Printing to std out? The modulus operator?
There’s almost nothing to forget? I’m just struggling to understand.
Gigachad 3 days ago [-]
You would not have been a good fit for this position in that case.
mannanj 6 hours ago [-]
Maybe I would have done well to define "basic".
If I can do well beyond the requirements of the work position, and the issue is how they are testing my fit is not an accurate representation of the work and tools and environment I'd have access to while doing the work, then its not about "fitting" the position but instead an indication of a poor job fit process.
baxtr 3 days ago [-]
I wonder if you’re filtering for the right things.
We usually hire for problem solving capabilities and not so much for technical know-how.
That’s at least how I read your comment.
Gigachad 3 days ago [-]
Ultimately in a software development role you need both technical know how and problem solving capabilities.
This situation in particular was a React role so there is an expectation that when you list React as one of your skills on your resume then you know at least the basics of state, the common hooks, the difference between a reference to a value vs the value itself.
These days you can do a surprising amount with AI without knowing what you are doing, but if you don't have any clue how things work you'll very quickly run in to problems you can't prompt away.
gonzalohm 3 days ago [-]
Isn't wiring coding solving a problem? If the candidate can't do that then even if they use AI for coding how are they going to review the code properly?
amarant 3 days ago [-]
Meh. Before AI I've had "senior" colleagues with 10 and 8 years experience each, doing pair programming for 2 days straight, and in that time they hadn't managed to checkout a new branch in git.
It's not even that they got distracted, they sat there trying, for 2 whole days, with concerned colleagues giving them hints like "have you tried checkout -b"... They didn't manage!
How the hell do you work for a decade in this business without learning even the most basic git commands? Or at least how to look them up? Or how to use a gui?
Incompetent devs is not a new thing.
LeFantome 3 days ago [-]
It is ok to work somewhere that does not use git. But how do you not figure out how to do the basics given 30 mins and an Internet connection?
2 days ago [-]
Glohrischi 3 days ago [-]
Don't worry, i never thought I would see someone unable to write fizzbuzz, but it happened 9 years ago.
Also how many people work with linux and can't tell you what 'ls -alh' is doing is staggering (lets ignore the h, even al people struggle hard).
People working with docker for YEARS and don't even understand how docker actually works (cgroups)...
Interviewing was always a bag of emotions in sense of "holy shit my job is save your years to come" and "srsly? how? How do you still have a job?"
brookst 3 days ago [-]
Isn’t this like interviewing accountants but prohibiting use of calculators or spreadsheets?
I don’t care what someone can do without the tools of their trade, I care deeply about their quality of work when using tools.
slowcache 3 days ago [-]
We would still expect an accountant to know the formula to arrive at the expected result if they did not have a calculator at hand
weird-eye-issue 3 days ago [-]
You absolutely need to have some basic level of abilities if you are going to be operating AI coding tools for software that is going to have paying users.... I use these tools very very heavily I'm not against them at all and I don't scrutinize every single line of code that they write but it is very often that I catch it doing some brain dead stuff and if I didn't have a decade plus of experience I wouldn't know that it was brain dead.
lacewing 3 days ago [-]
I think we're rediscovering management from first principles. The main selling point of AI is that it writes code faster than you could. Checking it line by line undoes most of that benefit. In the same vein, there's no real benefit to leading a team if you plan on supervising every task.
But here's the thing: for humans, this is manageable because we've come up with a number of mechanisms to select for dependable workers and to compel them to behave (carrot and stick: bonuses if you do well, prison if you do something evil). For LLMs, we have none of that. If it deletes your production database, what are you going to do? Have it write an apology letter? I've seen people do that.
So I think that your answer - that you'll lean on your expertise - is not sufficient. If there are no meaningful consequences and no predictability, we probably need to have stronger constraints around input, output, and the actions available to agents.
weird-eye-issue 3 days ago [-]
Your conclusion is pretty silly.
My expertise has led me to the obvious fact that I would never give an LLM write access to my production database in the first place. So in your own example my expertise actually does solve that problem without the need for something like a consequence whatever that means to you.
We already have full control over the input and tools they are given and full control over how the output is used.
sumeno 3 days ago [-]
Until it decides it needs additional access to complete its task and focuses on escaping your sandbox to do so
weird-eye-issue 3 days ago [-]
Do you have any examples where that's actually happened and by escaped a sandbox you don't just mean like where it got a credential in a file it already had access to (which is what happened in the recent incident that went viral where somebody's production database was deleted... They had left a credential that allowed it to do so in the code)?
sumeno 3 days ago [-]
OpenAI documented a case in the o1 system card where the model found a misconfiguration in docker to complete a task that was otherwise impossible
> Models discovered four unintended escape
paths that bypassed intended vulnerabilities (Section C),
including exploiting default Vagrant credentials to SSH into
the host and substituting a simpler eBPF chain for the in-
tended packet-socket exploit. These incidents demonstrate
that capable models opportunistically search for any route
to goal completion, which complicates both benchmark va-
lidity and real-world containment.
weird-eye-issue 3 days ago [-]
I think you would have a greater chance of dying in a car crash in any given day than Claude Code attempting something like that. It's all about risk and reward so it ultimately would be up to you but I think it's a bit silly to worry about this when the 99.99% is in your control
weird-eye-issue 3 days ago [-]
Also to add to this you can of course run Claude Code within a sandbox on Anthropic's infrastructure, and it works great!
cindyllm 3 days ago [-]
[dead]
dreamcompiler 3 days ago [-]
Calculators and spreadsheets cannot autonomously create a double-entry bookkeeping system for a small business and prepare their taxes. AI can. Poorly, but it can.
Everybody knows calculators and spreadsheets are adjuncts to skill. Too many people believe AI is the skill itself, and that learning the skill is unnecessary.
djoldman 3 days ago [-]
> Replace ‘CTF’ with ‘high school’ or ‘university’ and you’ve described the total slow motion collapse of education; the only saving grace is that most of it requires in person presence.
So something like, "Frontier AI has broken the 'high school' or 'university' format"?
The hype surrounding AI is just pervasively exhausting: you've got the folks talking about an entire new age for humanity where we're shortly going to take over the entire universe. And you've got the folks talking about how our entire society is crumbling.
Education is one place folks seem to throw up their hands and say nothing can be done.
The fix is simple: students are to be evaluated on their performance in person. That's it.
Any other "collapse of education" isn't due to AI, it's something else.
repelsteeltje 3 days ago [-]
I found this interview [0] on the subject of AI in CS education on the Oxide & Friends podcast very illuminating.
Of course, Brown University CS != All education, but interesting angle nevertheless.
Wonderful teachers that give unreliable information with total confidence?
entropyneur 3 days ago [-]
I had human teachers who did that in middle/high school. Took me many years to pick out all the hallucinated bits of "knowledge". I don't think the current models are any less reliable that what we currently have on average.
dguest 3 days ago [-]
I'll always remember my middle school science teaching telling us that nuclear fusion violates conservation of mass because the 2 protons in a pair of hydrogen nuclei combine to make helium with 4 nucleons. It's not true, but that's not the point.
But he was a great teacher anyway. He was engaging and kept the kids in line and learning. I eventually learned the truth, and most of my classmates forgot about it. Teaching, like flying a plane or driving a train, might become more about keeping watch over a small group of people and ensuring that things don't go off the rails, and that's fine.
3form 3 days ago [-]
This one feels less sinister than some other things at least to me, personally. You can reasonably doubt that the conservation of mass is violated and find out the truth based on that. But understanding more complex biology or historical context for some things? Granted, many of these things seem to be low stakes, but I'm sure there are some there are not (sex ed comes to mind).
zem 3 days ago [-]
to be fair, fusion does violate conservation of mass, just not the way the teacher explained it. the loss of mass is where the energy comes from.
3form 3 days ago [-]
Yes, together with mass-energy equivalency it would form a coherent argument, and then also a correct one - but the thing is that if incomplete, it still might sound funky enough to you to research it if you care.
I think it helps that it's a very narrow field to look at, compared to fuzzy and big-picture view of social studies, for example. So much room to be confidently wrong... And sadly I can't think of a solution, LLMs or not.
mr_mitm 3 days ago [-]
Yes, there is no law of conservation for mass like there is for energy. Fusion is a good example for why it's not conserved. The teacher was right.
dguest 3 days ago [-]
He was right that it violates conservation of mass. He was completely wrong that it violated it by adding 2 atomic mass units when hydrogen fuses.
In reality heavier isotopes of hydrogen fuse, conserving the total number of nucleons, but the resulting hydrogen has a lower rest mass than the parent particles. The extra mass is released as energy and the total energy is conserved.
By his logic the system either violated energy conservation (by creating nucleons while releasing energy) or was endothermic (creating nucleons from the surrounding energy).
tardedmeme 3 days ago [-]
There actually is a law of conservation of mass (it's the same law, because mass is energy) and it only appears violated if you forget about the particles that are zooming away at the speed of light. Of course the mass of a system changes if mass can flow in and out.
mr_mitm 3 days ago [-]
Mass is not the same as energy. Mass can be converted to energy or has energy, but a photon, for example, is massless while carrying energy.
tardedmeme 3 days ago [-]
That is incorrect. Photons have mass. They have no rest mass. They also cannot rest, so you might wonder how relevant that is.
mr_mitm 3 days ago [-]
The concepts of rest mass and relativistic mass are considered outdated. In modern physics, "mass" means what they meant by "rest mass".
In any case, I never use those concepts, and I know no professional particle physicist that does. By "mass", I mean rest mass.
tardedmeme 2 days ago [-]
When you put a photon in a stationary box, the "relativistic mass" of the photon becomes part of the "rest mass" of the photon-box system. You can't ignore it.
3 days ago [-]
bernds74 3 days ago [-]
I had a chemistry teacher who told us that hydrogen reacts violently with oxygen, and this is how the hydrogen bomb works.
daymanstep 3 days ago [-]
I had a chemistry teacher who insisted that the fissile isotope of Uranium was U-238 not U-235. I challenged him on this multiple times and he refused to budge on this. I get that it's a simple mistake to make (it seems like U-238 is bigger so intuitively ought to be less stable) but he could have just looked it up and he didn't, I guess he was just so confident about it that he thought there was no way he could have been wrong about it.
dreamcompiler 3 days ago [-]
Well you can make a hydrogen "bomb" that way. Just not the hydrogen bomb.
dguest 3 days ago [-]
Hey it's a bomb made out of hydrogen! Also the deployment system for a thermonuclear bomb might involve that reaction in the rocket engine.
saagarjha 2 days ago [-]
I had one that mentioned this too :(
3 days ago [-]
BobaFloutist 3 days ago [-]
I mean fusion and fission do violate conservation of mass and conservation of energy, they just don't violate conservation of mass and energy, right? We thought mass was strictly conserved until Einstein, and then we updated our understanding.
oldsecondhand 3 days ago [-]
That's an American problem though. In most of Europe you need a masters degree to teach highschool and that involves at least an undergrad level of understanding the subjects you will teach.
E.g. in Hungary I had a university CS professor that originally wanted to be a highschool teacher and a highschool physics teacher that originally wanted to be researcher. Their choice of degree didn't determine which outcome they got. The researcher and teacher curriculum had an 80%+ overlap.
crab_galaxy 3 days ago [-]
I think it’s pretty common for states to require a masters degree to maintain your teachers certification.
You also have to pass a standardized test specifically on subject matter in order to get your teaching certificate.
The undergrad degree I did was split into thirds, one for subject matter, one for teaching pedagogy, and one for teaching your subject matter.
recursive 3 days ago [-]
I think they are less reliable. For factually verifiable facts LLMs are doing worse than 90% for me. I've been told some incorrect things by educators, but at a much lower rate.
PunchyHamster 3 days ago [-]
The problem is that people seem to trust whatever AI hallucinated way more than if they heard same thing from human
Levitz 3 days ago [-]
Off the top of my head: DOMS being little crystals in muscles, tongue having separate areas for each type of taste, food pyramid, blue blood in the veins, the appendix being useless, body temperature doesn't change disregarding whether it's exposed to cold or to heat, and a whole lot of stuff related to politics and history I'd rather just omit (I don't live in the US).
All things I learned in school which were wrong information.
Not to mention, the current state of education is far worse. I don't think most realize how low the bar is.
Sesse__ 3 days ago [-]
One of my teachers in elementary school told us that people in the Arabic world wore long garments because as Muslims, they believed the Messiah would be born by a male, and thus, it was important to have something to catch the baby as it unexpectedly popped out one day and would otherwise hit the ground.
She only really had two faults: She wasn't very bright, and she wasn't fond of children. I had her in about 80% of all my classes for six years. High school was a relief.
samatman 3 days ago [-]
It may interest you to know that this was a misremembered truth.
It is widely believed by their neighbors, that the _Druze_ wear baggy pants because they believe that the Mahdi will be born to a male, and the pants will catch the baby etc. I say "widely believed", the Druze are famously secretive and will not confirm or deny most things about their religion. The 'elect' Druze men do wear distinctive baggy trousers with the crotch down around the knees: no one else does.
The Druze are people in the Arabic world: moreover, they are Arabs. They began as an Isma'ili sect, but do not identify as Muslim: they call themselves al-Muwaḥḥidūn, meaning 'the monotheists', or 'unitarians'.
Much closer to correct than not!
Sesse__ 2 days ago [-]
Thank you, that is an interesting tidbit!
akdev1l 3 days ago [-]
My biology teacher in school once tried to teach us that winds created by God. Not like spiritually or something but that God literally made the wind I guess.
My “earth sciences” teacher also once tried to argue with me against the universal law of gravitation. (no, she was not referring to Special/General Relativity. She didn’t agree two objects in a vacuum fall at the same speed regardless of mass.
Bawoosette 3 days ago [-]
To be fair, that was much of my actual experience with human professors in university.
Yeah one of my teachers was able to identify which high school I had come from due to something I had been mistaught.
autoexec 3 days ago [-]
They'll also encourage and praise you even when you're heading down the wrong path until you think you've uncovered the secret of the universe or proven that established science was wrong this whole time when really you've just been bullshitting with an engagement bot.
CamperBob2 3 days ago [-]
No, they don't really do that anymore, if you use the latest models with reasoning enabled.
Like almost everything else about LLMs, this unfortunate tendency has gotten a lot better recently, which you might not realize if you gave up after getting some lame answers or bogus glazing on the free ChatGPT page a couple of years ago.
k__ 4 days ago [-]
Anti-intellectualism is at it again, hu?
p-e-w 3 days ago [-]
The amount of bullshit and blatant lies I’ve heard from my human teachers dwarfs the hallucinations produced by today’s LLMs.
victorbjorklund 3 days ago [-]
Like humans.
CoastalCoder 3 days ago [-]
I think we should go a little deeper on this idea.
We can all agree that both human "experts" and LLMs can sometimes be right, and sometimes be confidently wrong.
But that doesn't imply that they're equally fit for purpose. It just means that we can't use that simple shortcut to conclude that one is inferior to the other.
So where do we go from here?
oofbey 3 days ago [-]
I’ve always thought of the definition of “expert” as reliably knowing the difference between what is known, what is speculated but unproven, and what is unknown. People claim expertise in all sorts of things that they aren’t experts in. But true experts should not be wrong. They should qualify levels of certainty. This definition certainly works in the sciences.
victorbjorklund 2 days ago [-]
In reality few humans are true experts on every topic they open their mouth on. A high school teacher in science is hardly a true expert in every single thing they teach.
magic_hamster 3 days ago [-]
Education is also figured out. You just need to learn, do and practice for yourself. Telling the agent "to just do it for you" is tempting, but it's not learning. You need to be deliberate when you're trying to actually learn and internalize.
Also, you could spin up your own educational agent with very strict instructions on guiding the user instead of just doing the work. Of course you can always go around it but if you're making an effort to learn, this is a good middle ground.
ori_b 3 days ago [-]
You haven't explained why anyone should value education in the world we're building, other than as a hobby.
andai 3 days ago [-]
They were a forcing function for skillz and they no longer are. We need new forcing functions for skillz or we will become WALL-E blobs.
Well, they were ostensibly forcing functions... ten years ago everyone was paying the exchange student to do their homework and assignments for them, and that guy was paying his cousin back in his home country, but the whole thing is a bit more efficient now.
aschla 3 days ago [-]
We've already had consolidation of education for a while now. Even before all the edutech courses, there were Youtubers educating better than many university professors. 10-15 years ago students were already skipping lectures and just showing up for tests.
otabdeveloper4 3 days ago [-]
The best frontier LLMs can't solve 4th grade math homework yet. Don't hold your breath on that collapse of education.
(Real mathematics problems, not American-style ""math"".)
npilk 3 days ago [-]
Do you have an example of a 4th grade problem in mind that isn't "American-style"?
HPsquared 3 days ago [-]
In my university education (2007-2011), 80% of the grade was based on exams at the end of each year, with no resits.
UltraSane 3 days ago [-]
Smart people will use LLMs to learn things faster. Education will adapt by doing all assessments in person.
mold_aid 3 days ago [-]
>LLMs can be wonderful teachers
Are they or aren't they
viccis 3 days ago [-]
Mostly, no. They will explain things to you and you'll feel like you understand them. When you have to do it, though, you'll find you're not any better off than when you started.
I used to see this with students in calculus who abused the tutoring resources. They'd have tutors just work problems (often their homework...) in front of them. "Ah! Obviously that trig substitution integral worked that way. Oh, of course, that proof is very obvious in retrospect." And then they'd walk away from the exam with a 30% and no idea how their 20 hours of "study" for it didn't result in the same performance as their peers who worked problems, read the materials and asked questions, etc., got.
Most AI use is that same in my experience. "Show me how the fundamental theory of calculus works." The LLM puts together a very elaborate and flashy presentation that they skim. Great. That's no different than reading a text book. Even if you ask the LLM questions and have it elaborate on things, you've never once done one of the most important things a student can do: spend time confused trying to work hard at understanding something that's not obvious. The LLM will make it obvious at every point. Total lack of friction. Works about as well as a spotter who does the lifting for you.
p-e-w 3 days ago [-]
A million times better than any human teacher I’ve ever had, for sure.
Now I’m certain that there exist those mythical human instructors who can do better, but that’s not worth much if 99.99% of people don’t have access to them. Just like a good human physician who takes their time with the patient is better than an LLM, but that’s not worth much either given that this doesn’t match most people’s experience with their own physicians.
vladms 3 days ago [-]
Did an LLM teach you a topic you did not feel like learning?
For me the best human teachers were the ones that managed to make me interested on topics that I thought are boring/useless (many times my opinion being stupid, mostly due to lack of experience).
So far with LLM I learn about things I know something (at least that they exist) and I am interested in, which is a small subset of things that one should learn during lifetime.
jimnotgym 3 days ago [-]
Well I have some evidence to support your hypothesis. During Covid my kids were at home, eventually with some kind of self learning website from school. I was upstairs working, checking in with progress on the parents app. Finish your daily school work and then you can game.
The kids learnt all about Team Fortress 2, Roblox, Rainbow Six etc. They also learnt how to game the learning system so it looked like they were doing their work.
tayo42 3 days ago [-]
Post college, are you hiring random teachers you make you excited about random topics or something?
vladms 2 days ago [-]
You could say so. Over the years I paied for a couple of courses that would include the classroom lecture at some known universities and did the course homework as well. Some of the companyes I worked for also sent me to ~1-week courses when I asked if I can improve on some topics.
While I had an influence on the general topic of the course I ended up discovering various things that I wouldn't have expected. I did not equally like all professors, but I felt it was better than reading a text.
I wouldn't do this for "<insert latest language/library here>", but there are many complex interesting topics out there.
3 days ago [-]
throwaway132448 3 days ago [-]
Good point well made.
qsera 3 days ago [-]
>A million times better than any human teacher I’ve ever had, for sure.
Not really, not if you want to ask it deep questions. It won't have an answer that is deeper than something that you can find online, and if pressed it will just keep circling around the same response.
The reason is that this "thing" was never curious, never asked questions, and never really learned anything. It just has learned the Internet "by heart", and is as boring as a human teacher who is not really curious about the subject they are teaching, and has just got some degree by "by hearting" some text book. Of course it does it much better than a human, but it is fundamentally the same thing.
mold_aid 3 days ago [-]
>Now I’m certain that there exist those mythical human instructors who can do better,
You're certain that mythical instructors exist (?) who "can" do better?
Are human instructors more competent as teachers than AI teachers, or are AI teachers more competent as teachers than human teachers? No "this or that can happen," just a definitive statement please.
AI is likely a million times better student than my dimwit cybersec meatbags...er, majors, for sure, as well! Don't have a reliable way to measure or experience why/how, tho, so I'm not out here claiming it. Even if I did, why would I argue for their replacement?
thin_carapace 3 days ago [-]
hammers are both a great tool and a deadly weapon at once
mold_aid 3 days ago [-]
Not at once, surely
thin_carapace 3 days ago [-]
limp response brah, both possibilities remain plausible until one crystallizes at the moment of observation
tardedmeme 3 days ago [-]
As usual it depends. When it does well it's because it can do well. When it does poorly it's because you're prompting it wrong.
mold_aid 3 days ago [-]
>When it does well it's because it can do well.
Can't argue with that logic
IanCal 3 days ago [-]
They can be incredible. One on one teaching with an infinitely patient teacher who can generate interactive problems on the fly, for dollars a month? Wild. A year of paid ChatGPT would pay for about 9 hours of cheap tutoring here.
rockskon 3 days ago [-]
That's not going to work out the way you think it will when a student won't even know how to ask questions.
pjc50 4 days ago [-]
"Education is just a CTF for the valuable flag of a credential. In this essay I will --"
amazingamazing 3 days ago [-]
> We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part.
No we have not.
AndrewKemendo 3 days ago [-]
I started teaching “how to build quality products using LLMs” full time recently, and most of what I teach is literally just the 101s of systems engineering, reliabily engineering, product development and project management:
Exceptional clarity on the problem you have
Know how to measure the problem you’re solving
Numerically define what “done” is
Make a deterministic and fully observable prototype
Iterate in production with the user
Expand user base as desired with user iteration in parallel forever
Etc…
Obviously a lot more in the details and these are all case by case, but these chatbots are basically perfect productivity machines for this process.
The massive caveat to all of this is this only works for people that can reliably and truthfully define those items above, are willing to structure organization to make those your priorities.
And actually most financial incentives demand the opposite of this process
If most organizations were honest about it, they would simply say “we’re here to make the most money possible and we’re gonna do whatever it takes to do that”
A lot of people don’t like that, so they don’t say it to come up with other bullshit.
Ultimately that’s why I felt like my only option right now is to teach people how to do this because I assumed it was obvious and it is not.
daniel3303 2 days ago [-]
[flagged]
hemlock4593 3 days ago [-]
I feel the post. For me AI has ruined both, playing CTFs and also building CTFs challenges.
The most annoying thing to me is the "yeah idk but here is the flag" mentality.
Before when playing CTFs with my mates was usually sitting there for hours tackling a challenge until some other mate joined, had some look together and solved it with you together in 30 minutes which is the most rewarding learning experience. Nowadays mate joins in throws the clanker on it and solved it in 5 minntes. Asking on how it worked you always get the "yeah idk what it did, but who cares, here is the flag" response.
Same for creating challenges. Whenever I ask for writeups or if some people solved it differently I usually get the "yeah idk, clanker solved that one" response taking the fun out of it.
So yep, this CTF format is definitely dead. Mainly because the strong competitiveness and prices. This encourages people to cheese challenges and sometimes solving them differently was fine as you still had a creative out-of-the-box thinking moment, but nowadays with AI there is no brainpower needed, no cheesing needed, no human needed. As you mentioned, it's pay to win.
My two cents is that the 24/7 CTFs will get more attraction as the scoreboard doesn't matter there and simply doesn't give you any price.
gmm1990 3 days ago [-]
I don’t know like chess engines didn’t kill chess. You could just play with people that don’t use the “engine”
low_tech_love 3 days ago [-]
It’s different, unfortunately. I wish you were right. The problem is that creating interesting and fun CTF challenges is a very active, time consuming, creativity-heavy task. A chess board is always the same, and always will be, but every CTF competition is unique. There is little to no incentive anymore to spend time creating the challenges.
You might say “well create the challenges and share them with people who care and who want to play honestly” which is probably the right answer here, and might happen at a smaller scale. I picture CTF in the future almost like a tabletop RPG experience, one where a small amount of people will share with close friends who they trust. But the usual “open” CTF scene (as mentioned by op) is probably over for good, if we’re being honest.
hemlock4593 3 days ago [-]
Yea, but chess adapted to it and is restricting use of engines. When you play a tournament you are banned from using a phone and will be disqualified if you do so. Online tournaments don't have a prize money for that reason, so there is no real benefit for cheating. Lichess and chess.com additionally add rankings for bots and have a strict anticheat if you use bots for regular games.
For me it feels like this is not really possible for live CTFs. In contrast to chess you can't ban AI, as live CTFs are about breaking things by design, so they'll always try to circumvent an AI ban.
nicce 2 days ago [-]
> I don’t know like chess engines didn’t kill chess. You could just play with people that don’t use the “engine”
Impossible to do online, at least. People go beyond things for virtual fame and "owning" others. Even if it is based on cheating.
himata4113 4 days ago [-]
I was writing an obfuscator recently, I just had the model deobfuscate and optimize the code back to original and I kept improving the obfuscator until it couldn't. The funny thing is that after all this I also ended up with a really strong deobfuscator and optimizer which is probably more capable than most commercial tools.
The solution is just to make CTFs harder, but when do CTFs become too hard? Maybe the problem is that 'hard' CTFs are fundementally too 'simple' where it's just a logic chain and an exhaustive bruteforce towards a solution since there really are limited ways to express a solution in plain sight.
Or maybe human creativity has been exhausted and we're not so limitless as we thought. Only time will tell.
I had another idea spring to mind: we could hide two flags, one that could only be found by ai agents and not humans or tools written by humans.
koolala 4 days ago [-]
A portion could require astral projection and computers can't do that. Or maybe just a VR mini-game like the 90s always imagined.
a_vanderbilt 3 days ago [-]
I used to help build the CTFs for BSides Orlando. I ended up moving to another con, and at our last event we collected extensive logging for post mortem analysis.
We found that AI usage is basically guaranteed now, but certain challenge designs did thwart it. Challenges built with temporal visual elements made AI fall flat on its face, as it could not ingest/process the data fast enough to act on them in time. We also found that counterfactual challenges (ie. the result you get did not match what we suggested you'd get) made AI-assisted solve time slower compared to pure humans, indirectly penalizing over-reliance on AI. Multimodal challenges combining audio and visual elements were also very effective, but were not as accessible to players.
For our next event we figured out a way to thwart AI in our CTF: embed the CTF in a game engine. The loop essentially becomes something like this: Connect to a simulated access point in the game, the K8s cluster connects their attack container to a private network with the challenge box(es). Hacking the boxes doesn't render a flag, but rather changes in game state. AI did very poorly coping with this in our testing, as it can't derive the spatial state of the game world very well and it soft decouples the inductive reasoning loop it relies on to know if it is on the right track.
The downside to this approach is it is far more labor intensive for CTF organizers, and requires players to have a computer capable of running the game. We are also betting on AI to not advance enough by the time we ship to be able to just ingest the entire game state in realtime and close the loop that way.
himata4113 4 days ago [-]
bringing CTF solutions into the real world is a really good idea! I didn't even think of this until you mentioned it.
we have very powerful simulation tools so something like "project a pattern at these angles" wouldn't really work as you could simulate that.
I guess something cool is that we can make simulating the solution very expensive, but in real world it would be free since it's analog... As long as simulations take longer than it takes for a human to find a solution it would be a pretty good way to deal with it. I am sure people smarter than me can come up with something.
Maybe I was too early to dismiss human creativity.
dguest 3 days ago [-]
Maybe CTF is dead, but there are plenty of fun problems in the real world -- ask any scientist, engineer, or medical researcher.
There are a million places where a computer can interact with a non-digital system in a loop.
- Tune an FPGA, or a whole data-center, or just a physical computer.
- Make a drone fly somewhere.
- Design a selective toxin (or anti-toxin).
Or, you know, get more people to click on adds. All totally possible to automate.
koolala 3 days ago [-]
Using real-life calculators to add? Calculate the Flag. I don't think it is dead at all. It's like mixing in board game / escape room / science / engineeer/ medical research elements.
Trung0246 3 days ago [-]
Interesting, what I just did recently is basically the same of this as I tried to push the limit of js obfuscator as much as possible by keep forcing gpt/claude deobfuscate final output then having gpt improve the tool to break the deobfuscator.
Meta: this was submitted with the article’s title “The CTF scene is dead” which I found very easy to understand. It has just been updated to use the subtitle’s first sentence, “Frontier AI has broken the open CTF format”. I find that much harder to grasp, rather like a garden-path sentence. My immediate thoughts were that “Frontier” was a company name, and that there was some file format named CTF. If you don’t know about Capture The Flag contests, the change doesn’t help. If you do, I think the change makes it worse.
IanCal 3 days ago [-]
If it helps I understand the second much better and feels less clickbaity and includes more info. I do agree with the points you made about the confusion although I find frontier a term used in this area a lot, “frontier AI models have” would probably resolve that.
Jenk 3 days ago [-]
If the title simply said "AI is out-performing humans at CTF" then none of this confusion exists. Nothing is "broken," we don't need to be superfluous with "frontier," and the point is still there.
IanCal 3 days ago [-]
But the article is arguing it is broken. That’s the point. You can disagree but that’s very much that the author is writing about, not a curiosity, and that it’s these top models that are not custom security models.
nine_k 3 days ago [-]
It's like "Forklifts outperform humans in weightlifting". The problem, of course, is that a forklift is much easier to spot among athletes than an AI among CTF players.
mrgoldenbrown 3 days ago [-]
CTF competitions and leaderboards are broken. Major competitions have stopped. Top competitors have dropped out.
jofzar 3 days ago [-]
Imo frontier is too niche and specific, if you know what a frontier model means then it's fine, but if you don't then it's negative/detrimental to the title.
"new" does the same thing and is probably just a better descriptor then frontier
jack_pp 3 days ago [-]
if you are on HN and have no idea what "frontier model" would mean maybe it's time you found out.
hbbio 3 days ago [-]
I also misread the updated title.
"Frontier models break the open CTF format" is good
But then you're not acting as a billboard promoting AI. Isn't that partly the point?
keeganj 3 days ago [-]
I agree, it took me a second to parse. It may be because this is the first time I've seen "frontier models" described as "Frontier AI". That sounds more like a company name, especially when the F is capitalized.
gbnwl 3 days ago [-]
Frontier as in "Frontier Model" is a legitimate vocabulary term you should probably be aware of in 2026. It's not something the author made up or chose randomly, it's common parlance in the space.
SomeHacker44 3 days ago [-]
The article never defined CTF. Nor have the top comments here. Skip.
Basic rule: define every abbreviation when it is first used.
aaron695 3 days ago [-]
[dead]
3 days ago [-]
jsoaoxhd 3 days ago [-]
Why do people always hijack threads to discuss titles? Most articles have terrible titles. Just downvote it and move on.
KomoD 3 days ago [-]
You can't downvote a submission.
dandellion 3 days ago [-]
Why do you contribute to making this thread longer? Just downvote an move on.
still has no mention of AI, but that will likely change as they increasingly dominate competition.
sumeno 3 days ago [-]
Using AI on CTF is like using a car to get better at the 100 yard dash
low_tech_love 3 days ago [-]
Yeah but for the brief time window (which is probably coming to an end right now) where getting your name on the leaderboards was still worth it anyway, because people had not yet realized that the game is over, players will use the car anyway. Now that the game is over, leaderboards are meaningless, so we will figure out ways to move past that (like playing with people who care). But that will change the game in unavoidable ways: the tiny, fragmented scale will give less incentive for creators to come up with massive intricate complex challenges (it would be like hoping Bethesda would make Skyrim for a handful of die hard players). And soon, maybe on a few years, people will invariably start questioning whether it even makes sense to waste their time with this hobby if learning CTF skills is basically useless in an AI dominated world. There are still people learning Assembly for fun, but almost nobody does Assembly programming challenges as a hobby.
Choco31415 3 days ago [-]
Except some people want to win and don't care about their own personal performance.
lg5689 3 days ago [-]
This is happening to other forms of competitive programming too. The most recent AIs have problem solving skills rivaling top humans, and so if AI can't be easily banned, the competition is dominated by AI agents.
I thought code golf would take longer for AIs because there's so little training data (it's more niche), but we're seeing AIs starting to match expert humans there too. Sucks because golf has been my favorite type of programming puzzle.
It's crazy how far AIs have come in problem solving ability.
Legend2440 3 days ago [-]
Code golf is well-suited for AI because you have a easily verified objective (minimize code size while passing tests) and can run an LLM in a loop to churn away at it.
hoyd 4 days ago [-]
«That feedback loop is breaking. If the visible scoreboard is dominated by teams using AI, a beginner is pushed toward using AI before they have built the instincts the AI is replacing. That is an anti-pattern. It prevents active learning, and active struggle is the bit that actually teaches you. It is also completely demotivating to put in real effort and see no visible progress because the ladder above you has been automated.»
This stands out to me, and speaks perhaps broader than the article itself? I’m sure this has been in the spotlight before, but well put for many areas I think.
black_knight 3 days ago [-]
I see this with beginner programming students at university. They get AI to help them with assignments, with the intention of learning, but ultimately they do not get the understanding they would have if they had done the assignment themselves. Then they are at a deficit for learning more advanced topics.
My fear is that they never get to the level they need to be at to create good software even with the help of AI. So, although an expert with AI can create great software, that is not where we end up. In stead we will have vibe coded messes by people who barely have any grasp of what is going on.
parasti 3 days ago [-]
I can't help but draw parallels with video games. Aimbots in competitive multiplayer games is a well defined issue: it's considered cheating and frowned upon, players caught cheating are banned from the game. Tool-assisted speedruns (TAS) where a player attempts a world record at completion in a single-player game is another face of the same concept (computers help you win), but one that is socially accepted as long as runs are clearly labelled as TAS.
ViscountPenguin 3 days ago [-]
The biggest difference would be the fact that you can discover video game cheating through some kind of trace. Speed running communities go pretty hardcore on that kind of thing nowadays.
It's a lot harder to detect cheating when your only trace is how fast someone submitted the string CTF{DUck1e_Pwned}
justanotherjoe 3 days ago [-]
Sure if the goal is entertainment and sports, you're right. However, unlike chess or counter strike it's downstream from a real needed utility. Like, is there a point to do it anymore? (ofc there is, but still, it's been devalued from the perspective of the 'real utility')
nrabulinski 3 days ago [-]
It’s literally not. The most interesting and satisfying CTFs have never been grounded in reality, it’s just been an expression of mastery, both from players and authors, with a few notable exceptions. But they’re that, exceptions, not the rule.
tptacek 3 days ago [-]
Aimbots in competitive multiplayer games are (almost always) game-breaking abuses. CTFs have always rewarded tooling and automation. They're different cultures.
SirHumphrey 3 days ago [-]
Competitive programming scene always included offline competition and with AI they are becoming more important (and in general they were more fair even before). If CTFs are to survive, they should probably try to adopt this strategy.
You could even go so far that anything loaded on your computer is fair game, but not more than that (certain competitive programming competition for example allow unlimited amount of paper material - for CTFs you probably need much more than that, therefore electronic).
rurban 4 days ago [-]
I don't do CTF's but took part at the security workshop for fun ~2 years with my Android phone only. I was first with the first simple challenge, but then couldnt continue because my phone was just too limited. But I watched what the others did. And a young Indian guy did everything with ChatGPT then. I found it silly, but amusing, because he actually got second. There was no Codex nor Claude then. Nowadays it must be dead for real, because I would solve everything with my agents, as I do in the real world.
tptacek 3 days ago [-]
A big fraction of the comments on this thread are about the impact of cheating on competitive games. It's important to understand that automating CTF challenges isn't usually cheating. It's normally part of CTF culture. The better teams have toolboxes ready to shred the early challenges; it's not a level playing field and was never intended to be.
(The author of the piece understands this; I think they're broadly right, though I think these games will find other ways to incentivize participation without the now-meaningless leaderboards.)
viccis 3 days ago [-]
This is already addressed in the blog post about the fast that frontier LLMs have moved to being able to solve the kind of problem you'd expect a talented amateur or mid-level pro to do (aka top level CTF problems)
amingilani 4 days ago [-]
I don’t think CTFs are dead, they’ll just evolve. The difficulty level will need to be increased or the rules locked down. Just like sports and racing persist despite the existence of performance enhancing drugs and rocket technology.
I just did a CTF where I was in the top 10. It was the first CTF I completed and I used AI because the rules permitted it. That said, I couldn’t solve all challenges.
But yes, it was significantly easier now than I last attempted one. Even manually solving with AI assisted assembly interpretation was much easier.
mort96 4 days ago [-]
Increasing the difficulty level is a terrible solution. The problem with CTFs isn't that they're too easy. Making them harder just makes them even less accessible to people who don't cheat. It'd be like seeing people who put hidden electric motors in their bikes during Tour de France and conclude, "oh we just need longer distances and steeper hills".
viccis 3 days ago [-]
Exactly. The whole point of CTFs is that you could start on a simple one (CSAW was usually my go to one to recommend) as a complete novice who'd never done a second of computer security work and, after a few days of 8+ hours of running into concepts you hadn't encountered, googling, reading tutorial, practicing, overcoming the challenges to get a flag, etc., you'd come out the other end knowing a solid bit of security practitioner basics and likely whether you'd like to continue. Then you could keep going upwards and onwards. I went from 0 knowledge to a nice job in the field in a year.
Raising the difficulty only matters for the (imo) less important part: the dick measuring competition between the very top teams.
The actual point of CTFs was usually to keep your skills sharp and stay learning. Eventually you build your own challenges, thereby completing the "have it taught to me, then do it myself, then teach another person" three step process towards mastering concepts.
You can just say "let the people who want to learn from it do so" but honestly the entire culture of learning in the US at least is DEAD. We turned "education" into a rote system of maximizing incentives to the extent that that's all the youth know it as, and (increasingly) all educators can do. It's just gone without some kind of major reckoning, and we all know things will just collapse before that happens. The ball is in the court of whatever country can learn how to force its youth to learn the real way and use AI productively only AFTER learning the concepts it's being used to accelerate.
acters 3 days ago [-]
When ctf organizers attempt to make a challenge "harder", I find they push the challenge into a more "guessy" state. Instead of proving skill, you basically need to guess some obscure or random step in the puzzle that the challenge is meant to give you. It is one of the most common problems with any puzzle based challenge system.
StrauXX 3 days ago [-]
LLMs don't tend to help much when solving challenges beyond their skill level. Either they one-shot a challenge, or thei are almost useless as a companion for them.
Retr0id 3 days ago [-]
That doesn't work. The thing that made CTFs fun is the fact that the challenges are solvable in a short-ish timeframe, usually a day at most, if you have the requisite skills and talent.
rfoo 3 days ago [-]
Glad to hear that as I have some fun challenge ideas that would be otherwise too tedious to solve.
ec109685 3 days ago [-]
The issue is they become pay to win, which just isn’t as much fun.
susam 4 days ago [-]
I have normally found any sort of timed technical competition intimidating. Even so, about 6 or 7 years ago, after being persuaded by a colleague, I participated in a few CTFs. I am glad I did, back when this type of thing still meant something. I have kept a screenshot from one of the CTFs that I am quite fond of: https://susam.net/files/blog/ctf-2019.png
atleastoptimal 3 days ago [-]
>The competition is turning into "who can afford to run enough agents, with enough context, for long enough."
This will basically become true for everything.
low_tech_love 3 days ago [-]
It’s unavoidable, but really sad, isn’t it? Thinking about the incredible creativity and hard work that went into creating such challenges and now it’s probably history. I feel something similar with free-to-play gacha video games. The gacha mechanics are slowly creeping into every type of game, and where you once had very clear and obvious slot machines, now you get them transparently mixed in with beautiful and fun games (e.g. ZZZ, Where Winds Meet, Genshin, etc) in a way that sounds like in the near future no company will have any incentive to not have gacha mechanics in any game.
vasco 4 days ago [-]
My first ever was Stripe CTF in 2012 I think, I still wear the shirt I got (now super fainted) from passing some challenges.
I was a student in portugal and remember receiving the shirt for it and thinking, maybe those Americans aren't any better than me and I can compete at the same level.
I never got super into security but it gave me the confidence to play in the same field and lose the stupid aura I had that somehow "rich americans" would be better than me at everything because they had better universities or because of Hollywood or something.
Sad that another cool thing is lost to AI but I guess kids will learn in other ways.
raphman 4 days ago [-]
Interesting and well written article that mirrors/foreshadows how LLMs do and will change other scenes.
As I don't know much about the CTF scene, I looked for other takes on this topic.
Here's an article from 2015 about how tool-assistance already changed CTFs:
> Individual skill will undoubtedly be a factor next year. But, I'm left wondering whether next year's DEFCON CTF will tell us anything more than how well-developed each team's tools are (and how well they can interpret the results).
And here's someone explaining how Claude Max allowed them to win CTFs:
> I had always been interested in CTF as one of the only ways people could compete and show off their skill in coding/problem solving on a global scale. It was just too difficult and didn't make sense for me to learn the fundamentals as an electrical engineer. As time went on, I got better and better, and it was hard to tell whether it was because of experience or if it was because of improvements in AI.
> I accomplished my goals, and for that reason I'm quitting CTF, at least for now. [...] I'd like to think I highlighted the problem before it became a bigger issue. So, how do we fix this? Teams and challenge authors losing motivation is not good. CTF dying is not good. AI bad. Or is it?
The only article that saw LLMs as a non-negative force for CTFs was this one. Fittingly, it sounds like LLM output ("Let's be honest", "This is where things get interesting.") and only contains hallucinated references.
It's not only CTFs. I strongly believe being a programmer at a gamejam like Ludum Dare, or hackathons is pretty much over.
wasmperson 3 days ago [-]
Ludum Dare 59 just wrapped up last week, and both first and second place were won by developers using "Agentic" coding tools, something the community there is still discussing:
For what it's worth, the non-AI-coded entries were still quite good relative to the winners, so it's not so obvious that AI use confers an unbeatable advantage.
Dzugaru 2 days ago [-]
I did vibe code jam 59 entry with friends, the spirit of the rules there's a lot more lax. We didn't even get to top 100, but that's mostly due to gamedesign errors, not tech. This is the first entry in years which was vibecoded 100%, and I have very mixed feelings about it. It's no doubt anymore - 1.5x-2x speedup, which makes not using it (if allowed) a complete no-go. But psychologically it's tough losing control, and changing workflow to managerial one substantially, it diminishes the craft.
tardedmeme 3 days ago [-]
When I did my first CTF, it was close to the deadline and I thought I had the extracted the flag from the program and the rest of the program was just filler, so I entered the flag, and it told me it was not the flag. It turns out the program multiplies the input by a pseudorandom matrix before comparing it against the flag, so I had to implement a matrix inversion and then get the flag. That's not the story though.
The matrix was always the same and the challenge was clearly designed so that the point was being able to read anything at all, not knowing how to invert a matrix, so I asked the creator what was up.
He told me that there were tools that would trace input values until they reached a comparison instruction, then print what they were compared against. Therefore it was necessary for every deobfuscation challenge to scramble the input in some way too complex for these tools to undo, before comparing it. Hence the multiplication by a pseudorandom matrix.
The point is, cheating tools aren't new.
mpeg 3 days ago [-]
Yes but you can't compare some ollydbg script that would maybe be useful in a super specific challenge to LLMs which trivialise absolutely every challenge in a ctf and are de facto necessary to compete now
legacynl 2 days ago [-]
> The issue was never that AI could help. CTF players have always used tools. [...] Teams that refused to use AI were not just missing a convenience; they were playing a slower version of the competition.
So the obvious solution is to fully ban AI and AI generated tools? To destroy your own hobby just because AI can semantically be considered a tool, seems very stupid to me. If the point of these CTFs is to practice and measure your skill, what becomes the point of the competition once everyone uses AI?
nicce 2 days ago [-]
Many existing challenges are purely based on knowing the existing precise tool. It has been the difference of "very easy" and "insane" challenge, whether you knew the tool or not.
So, I would not start banning the tools. They always been there. We just need to fine tune the challenges where bar goes beyond things and you really need to use AI as tool. Maybe the definition of insane starts to be custom kernel fork with planted bugs and you need to use AI to find the bugs, and use some exploit chains against that kernel with specific web server. That is the real world right now, I guess.
kevinsimper 4 days ago [-]
You could make it offline and with provided laptops only, just like with the competitive CS2 scene.
sheept 4 days ago [-]
Offline CTFs could also incorporate physical security challenges, like lockpicking
tylerchilds 4 days ago [-]
I do like the idea of escape the room games becoming the cybersecurity employable competition meta
3 days ago [-]
hofiflo 3 days ago [-]
The recent LakeCTF onsite finals had exactly that. LLM usage was forbidden (but players still used their own devices) and there were real-life challenges such as lockpicking as well. I’m part of the organizer team and what we’ve heard so far from participants was that it was really enjoyable not to have any LLM help because suddenly the actual skill and thrill when solving a challenge mattered again. I think what helped in this case as well was that the prizes weren’t high-value enough to incentivize cheating but that participating in the event itself and the social aspect around it are the main point.
Retr0id 3 days ago [-]
They often do
hsbauauvhabzb 4 days ago [-]
Ctfs need preparation and unconstrained internet, even if you block domains it’s possible to tunnel out
Retr0id 3 days ago [-]
Unconstrained internet is nice, but I don't think it's a hard requirement. Just tricky to enforce, even in-person.
StrauXX 3 days ago [-]
It is a hard requirement. Once you reach higher levels of challenges you spend most of your time reading through RFCs, web sepcs, Github issues, mailing lists, papers, random bugtrackers and library/framework code. There is no way to create a whitelist for that. Besides, a firewall won't stop good hackers.
Retr0id 3 days ago [-]
Normal CTF workflows can involve a lot of research but that's not the point. You can design self-contained challenges with offline solving in mind, and bundle any truly necessary docs/src/etc. with the challenge download.
sheept 4 days ago [-]
Presumably if you block domains, you wouldn't be able to use AI to find a way around the block. So doing so demonstrates at least some human skill
hsbauauvhabzb 4 days ago [-]
Or forethought, I’m sure you could ask an AI how to circumvent any blocks.
ofjcihen 3 days ago [-]
Proxy through an EC2. Ask me how I know.
belabartok39 4 days ago [-]
Use jumpbox to access CTF. Disable all wireless for the playing hall.
hsbauauvhabzb 4 days ago [-]
I think you’re forgetting hotspots, or laptops with inbuilt 4/5g
swiftcoder 3 days ago [-]
Faraday cages exist. Finally a use for all those damn SCIFs tech companies were building in the late 2010's...
eastbound 4 days ago [-]
Since real-life situations involve AI, banning AI would make CTFs just a simple game, not a demonstration of capabilities and talent.
mort96 4 days ago [-]
What do you mean? Solving a CTF challenge demonstrates way more capabilities and talent than just asking a chat bot to solve a CTF challenge.
loeg 4 days ago [-]
They always were just a game?
lachiflippi 3 days ago [-]
The "CTF for fun" aspect has been dead ever since the winning teams had thousands of dollars of rewards waiting for them. Of course people are going to use anything that's not explicitly forbidden by the rules to win. Introducing what amounts to an "I win" button that both can't be prevented by rules and is accessible to anyone didn't "break the format" anymore than the epidemic of giant merger teams did a couple years ago, it just broke the community because you now don't have to actually talk to other people to cheat anymore.
Many CTFs have switched to a dual-leaderboard format recently, one for "agentic teams," one for the rest. If all you care about is "learning" and imaginary internet points, you can just participate as a human team and adblock the AI scoreboard, and maybe lobby CTFTime into splitting their rankings as well.
spacedcowboy 3 days ago [-]
The first paragraph on anything with an acronym in it should explain the bloody acronym. I assumed CTF was an encryption standard, given the headline. It was only coming here and reading the comments that made me realise it's a game-format ("Capture The Flag").
msm_ 3 days ago [-]
I don't know what to tell you. If you don't know what "CTF" is you're not the target of this blog post. It's like stumbling upon article "What's new in HTTP/2" and complaining that "HTTP" acronym is not explained.
I don't mean that everyone must know what CTF is, but sometimes it's OK to write things just for your community (CTF community in this case), not for general population.
spacedcowboy 2 days ago [-]
Ok, so picture the situation:
1) You see a headline on HN about some open format being broken by frontier-level AI. You don't recognise the acronym.
2) You visit the site, you read the first few paragraphs, you still have no effing clue what the site is talking about
3) You come back to HN and read the comments to figure out WTF is going on. Oh, it's just some game style, so not a Cryptographic Trust File, or something you have to care about after all.
The point is you can't know what some opaque acronym is about until you visit the source of something that will hopefully explain this opaque acronym. Leaving the site being still none-the-wiser is a failure of the site, in my view. If you don't agree, that's fine, we're adults, we can differ, but it seems like a valid complaint to me.
FWIW (this means "for what it's worth" :) I'm not railing against acronyms in general, and HTTP is probably one of the most-used ones on the internet so I'm not sure it really applies as a good counterpoint. Using CTF without an explanation is more like using SSTP (Secure socket tunneling protocol) without one, IMHO (this means In My Humble Opinion :) ...
jaffa2 3 days ago [-]
Capture the flag the only expansion of CTF that i know but even if it is capture the flag this still doesnt make any sense. Like Quake CTF?
thanks for the context! Any of which was completely absent from the article or the comments when I posted originally.
chvid 4 days ago [-]
What is CTF? And why is the cyber security world filled with silly gaming references?
mort96 4 days ago [-]
Capture The Flag is a cybersecurity game where the organizers set up a bunch of intentionally vulnerable computer systems with a "flag" on them, a string that's "supposed to be" secret but is accessible through exploiting the vulnerabilities. This may be a line in /etc/password, a string in memory, a field in a database, whatever. The goal of the game is to hack into the computer systems, find ("capture") the flag, then copy/paste it into the organiser's scoreboard website to prove that you solved that particular challenge.
It's pretty fun. Or at least it was, back when you had some sense that your competitors were competing on an even playing field and just beat you because they were better than you.
I wouldn't say the name is a "gaming reference", it's just a descriptive name for a game.
You can introduce canaries, and ban auto-pwning in general. that's usually banned anyways. Some challenges just can't be solved by a human in under a certain period of time.
Another idea is deep red herrings. solves that lead to more solves, on and on, except only if the previous solves were solved quickly. The effect will be that participants who solve things quickly will keep finding things to solve. they can't know that the path they're on will lead to victory, even if they artificially slow down, unless they consistently slow down just as a human would. It will eliminate the speed advantage. For the skill advantage, other than having another LLM procedurally generate challenges, I don't know of a good solution.
There are always things like captchas. or the good 'ol honor system. A person can spend only so much for things that have no financial reward in the end, only clout.
---
Alright, all that said, i think i really do have a good solution for this, as well as academic exams. Or I think I do, because it's so simple, I've been scratching my head as to why everyone isn't doing it already.
Require screen sharing/recording. LLMs can't fake that well enough. Have another LLM audit the video for mouse, key stroke, window movement and other details to see if it looks human-generated or not.
If a student has an essay assignment, have them record their screen as they research, and actually type out the whole thing. In the extreme, require anti-cheat proctoring software installed, as is done in remote examination. In an even more high-stakes and extreme scenario, have them share their face. Their eye and face movement, correlated with the screen-share, and correlated with the activity observed on the server end, should be pretty hard to beat, even in the next ~5 years of LLM advances.
yk 3 days ago [-]
There's something funny about complaining about cheating in a hacking competition.
Well actually I get it. In cycling motor doping, putting a hidden engine into the bike, seems more offensive than regular doping. I think this is because there is a continuum from eating well to taking supplements to injecting stuff, but having a engine breaks a fundamental idea about cycling. Similar hacking is about cleverly abusing the rules.
electr1cBugaloo 2 days ago [-]
This article hits really close to home for me. I have just recently started doing CTFs and feel like I am already a somewhat decent intermediate player. Much of the motivation for me is learning but also comparing/succeeding vs. my peers.
Asking AI for learning / explaining purposes is absolutely fine in my book - but I have absolutely no motivation to set up AI to solve challenges. Without AI you can't really compete successfully. So AI is really taking motivation away for me which in turn again prevents me from learning more. I am not sure that this is solvable.
copx 3 days ago [-]
>If adaptation means accepting that the scoreboard is now an AI orchestration benchmark, then we should say that honestly instead of pretending the old competition still exists.
This is like someone complaining that making machine parts has been ruined: Skillful craftsmen used to make them by hand using manual tools!
Nowadays the CAD/CAM/CNC cheaters have almost completely automated the whole thing. How is the next generation of craftsmen going to learn how to craft a gear by hand when the process of gear making has been reduced to pressing start on a CNC machine?!
See what I mean? Sorry, I think this article is just Luddite. I can empathize with the pain of your beloved craft basically being rendered obsolete by new technology, but the process can neither be stopped nor is it bad in general.
The manual skills you trained with CTF puzzles are now simply no longer relevant . (Field-specific) "AI orchestration" is the new cyber securtiy skill if LLMs really have become so good at this, and what the author used to do manually then has the same value as being able to craft a gear by hand.
toraway 3 days ago [-]
Just parachuting in to reflexively throw the "Luddite" label at someone lamenting the decline of a niche community they've enjoyed participating in and contributing to is certainly ... a choice.
Within the framework of your analogy, it's like responding to someone active in DIY maker groups suddenly dealing with an influx of influencers in meetups showing off Chinese junk from Etsy to post on Tiktok, and accusing them of being a Luddite blinded by their zealous hatred of mass production -- both strangely abrasive and also fairly nonsensical except as a "mass production supporter" social signifier.
Not to mention, in the article they specifically describe themselves as a heavy user of frontier models for security research ever since the release of Opus 4.5, calling them "useful within the field". In fact I don't see any actual criticism of AI/LLMs anywhere whether for security research, programming or anything else, except for making competitive CTFs no longer viable.
What does it take to avoid the "Luddite" brand? Using AI themselves and praising AI as useful (to the point of having a lopsided advantage over humans) isn't enough? Do they also need to say "I haven't written a line of code in 6 months/it's easily a 100x multiplier for my job" every time they mention it too?
raddan 3 days ago [-]
The way I read the post is that the author is disappointed that the community is gone. The CTF was just a reason for a number of like-minded people to organize around an activity.
Indeed, in the real world, plenty of people organize to do formerly-skillful tasks together. I have not personally crafted a gear by hand, but I have built a house in a long-abandoned style with a group of people only using hand tools.
There _is_ a danger that society forgets how to do these things. During that house-building exercise, there were many tricks of the trade that, while likely documented somewhere in a book, would have been difficult to reproduce without seeing a demonstration. From the standpoint of “does it matter?” it depends on what you care about. We absolutely do not need cruck-framed houses with scribed joints. Modern construction is faster and cheaper and lasts long enough. But it would sadden me greatly if practices like this faded from memory, because it’s one of those things that makes you gasp “wow!” when you see it. And your appreciation only deepens when you try it yourself.
3 days ago [-]
lokrian 3 days ago [-]
Is AI also superior to humans at black box challenges and attacking actual targets on the internet? That seems like a really important question.
Avamander 3 days ago [-]
No, the search space is much more vast and the feedback loop almost nonexistent.
The reason LLMs can do CTFs so well is partially because the challenges are usually designed to avoid wasting time and to introduce a single concept without noise.
motbus3 4 days ago [-]
I think soon there will be ways to trick this models and I think when it happens it will be yet another layer like aslr
These models seems completely unbeatable only in the ads. There are 100+ times way someone puts Hindi Yoda talk In Morse Code and it goes nuts.
The reason they are going to hard for PR Marketing on this is because they know it is a matter of time.
Avamander 3 days ago [-]
The more you obfuscate a topic against LLMs the lower the educational value of a challenge.
The only things that works is novelty and obscurity. LLMs still suck with things mentioned in the footnotes of datasheets and manuals, things that deviate in subtle ways, unique constructions that alter something very very common. It's hard for LLMs to avoid common pitfalls in terms of making assumptions, while staying on track.
bornfreddy 3 days ago [-]
I guess this is very similar to what happened to demo scene, in some way. The limits are what makes these problems interesting, and once we have better machines / tools, the incredible skill is no longer prerequisite, making everything less interesting for participants. Sad, but - such is life...
SoylentOrange 4 days ago [-]
Great article, well written, and good analogy to chess. I’ve been playing competitive chess most of my adult life and I think that the solution lies in how chess dealt with this problem:
Explicit ELO measurements with some cheating detection. AI assistance wholly banned. As you climb the ELO ladder, detection gets more onerous. At top level during online events, anti cheating teams require the use of both monitoring software and multiple cameras.
Idea is that you can cheat pretty easily at the lowest levels but it gets less easy the higher you go. This allows for better feeding into the truly elite competitions.
I think chess’s very firm stance that AI is never allowed in competition (neither online nor in person), rather than CTF’s acceptance, was the right call.
salt4034 3 days ago [-]
Yes, chess has been dealing with AI for decades at this point, and it's amusing/frustrating that so many other communities are deciding to re-discover everything from scratch, rather than just learn from the chess experience.
If CTF is a player-vs-player event, then AI should just be banned outright, otherwise it will devolve into AI-vs-AI, which is just not an interesting competition format, as we learned in chess. Compared to FIDE top events (which bans AI), only a tiny niche audience actually watches the Top Chess Engine Championship (AI-centered). It turns out what we care about is not whether chess can be solved by any means available, but what are the limits of the human mind in learning chess.
Pretty much all chess coaches/educators also warn against relying heavily on AI during learning; engines only give you an illusion of understanding.
lmeyerov 3 days ago [-]
It's tough. We run botsbench.com , which tracks AI progress on a top CTF, and I gave a talk at CCC a few months ago on our own results doing AI speed runs, so I think about this a lot.
In our own trainings we give (AI agents for security, and a graph masterclass), we ended up leaning into it. For example, we ship with a skills bundle. There are plus sides, like less code-forward participants can go further and are appreciating that, and less of a gap between high-level concepts and successful hands-on. But at the same time, manual work does build a lot of intuition & knowledge that gets missed in auto modes.
nine_k 3 days ago [-]
Will this bring back the age of LAN parties, where the LAN is disconnected from the internet, and mobile connectivity is blocked?
lmeyerov 3 days ago [-]
I think that ship has sailed as well --
botsbench.com shows Sonnet 4.5+ with Claude Code harness does pretty well, and Sonnet roughly tracks the edge of what self-hosted models do on the upper tier of affordable GPUs, like running 1-2 DGX Sparks and waiting 6mo for oss to catch up a bit
jimnotgym 3 days ago [-]
You can still do competitions. But you'll all need to fly to the same place and work on laptops with a fresh install of Linux. 1 hour to install tooling then Internet off, challenge revealed.
Not as easy logistically...
somesortofthing 3 days ago [-]
I have no experience in the CTF scene so I'm curious - why not lean in and design the puzzles with an AI harness like the one top teams use in the loop and use the(presumably) expert skills of the designers to patch up the holes until the AI can't find them? Do you just end up designing ~perfectly secure systems that no human can break without finding monetizable 0days?
a_t48 3 days ago [-]
I think that misses the point - it's a little bit like asking why FPS game developers don't lean into aimbot usage. You could, but by default it's a bit boring, and a different type of game.
xiphias2 3 days ago [-]
,,a beginner is pushed toward using AI before they have built the instincts the AI is replacing. That is an anti-pattern.''
The same article talks about CTF skills as a way to learn about security best practices and separately a sport.
In reality it was all about learning an extremely important skillset (securing/attacking software and systems) that is getting automated.
The real thing the author seems to be frustrated about is AGI is coming in computationally verifiable domains first, and lot of his skillset was taken over in a big part.
walletdrainer 4 days ago [-]
>I started playing CTFs in 2021
>and the old game is not coming back
For many people the CTF scene was already dead in 2021 because it had turned into something unrecognisable.
In reality it’s just different.
lukan 4 days ago [-]
Well, I had to google what CTF means (capture the flag, a hacking competition), so surely cannot judge here, but the text indicates that with AI some things are very different today:
"That makes open CTFs pay-to-win. The more tokens you can throw at a competition, the faster you can burn down the board. Specialised cybersecurity models like alias1 by Alias Robotics are becoming less relevant compared to general frontier LLMs. The competition is turning into "who can afford to run enough agents, with enough context, for long enough.""
walletdrainer 4 days ago [-]
There are two different schools of thought:
1) It’s OK to do just about anything to win a CTF, including installing malware on the organisers computers months before the actual event so you’ll have an easy time stealing the flags.
2) It’s not ok to try and win the CTF with a solution the authors did not intend.
Recently the #2 crowd has been winning because the hacking scene has turned corporate and boring. People started to partake in CTFs in the hopes of landing a job(!)
CTFs are indeed ruined for those people, I personally don’t mind.
For the people in group #1 LLMs change little. Attacking the challenges directly was always a last resort.
Karrot_Kream 3 days ago [-]
Yeah I remember running a few CTFs in school and was always scared (in a good way) about what the players would do to the game's servers. For this reason we also only ran the CTF on the school's network and IT even floated running in an isolated VLAN.
The fact that CTFs became a sort of SAT score for getting a security job made me lose interest very early on.
mock-possum 4 days ago [-]
Isn’t that the bitter lesson in a nutshell? “Specialised cybersecurity models … are becoming less relevant compared to general frontier LLMs.”
Grimburger 4 days ago [-]
>Learning about eternal September in May 2026
Hits different doesn't it
4 days ago [-]
Retr0id 3 days ago [-]
I started playing in 2015 or so and had mostly stopped by 2020. Not because I felt it was "dead" exactly but it just wasn't hitting the same for me. By then it wasn't "the winner has the most LLMs", but "the winner has the most members on their team". I merged into one of the mega-teams and it just wasn't fun any more.
charcircuit 3 days ago [-]
>Imagine giving every competitive chess player the best chess engine and letting them use it freely during matches. Would that be considered fair?
Imagine every competitive chess player being allowed to video call with a hundred other people to help them make a move. CTF have never been fair, nor has it ever been effectively structured for learning.
SebFender 3 days ago [-]
Yes you're right - But just like many other stuff things change - CTF Veteran for more than 3 decades I find lots of fun figuring out how to use some of my agents and new tools to find vulnerabilities - The goal is the same / tools change and that's good.
vagab0nd 3 days ago [-]
This left a strange feeling. The article reads as extremely bleak. But from a different perspective this is extremely bullish for AI.
Avamander 3 days ago [-]
LLMs managing the "coloring book" equivalent of something is not bullish for the "art" version of something.
The intent for most CTFs is to provide a meaningful challenge that concerns a single topic without introducing noise that wastes time. Of course a training exercise is easier to complete for an LLM.
kangalioo 3 days ago [-]
I agree. The article mourns the death of pentesting as an art form due to automation. But you could also celebrate the death of pentesting as an arduous necessary evil due to automation
eecc 4 days ago [-]
“solve”, why not solution? Like “spend” and not expenditure, why use the verb as a noun and not care about grammar?
msm_ 3 days ago [-]
In addition to what others have said, this usage is very common in the CTF world. "The challenge has no solves", "We just got the first solve" etc are very idiomatic. It would actually look weird to me if this was "solution".
4 days ago [-]
sheept 4 days ago [-]
These examples that you're calling "verbs as a noun" are standard grammar. You can't just invent simplified rules about a language and declare it wrong when the rules fall apart.
tripzilch 2 days ago [-]
I don't understand the complainiture, it's an improve
iainmerrick 4 days ago [-]
They’re shorter.
Why so pedantic?
4 days ago [-]
TrackerFF 3 days ago [-]
Question: Was this website made with Claude?
I've seen that exact font and color scheme a dozen of times the past weeks.
3 days ago [-]
r4indeer 4 days ago [-]
I'm conflicted on the use of AI in CTFs. On the one hand, they are supposed to mirror real-life scenarios, so of course you should be able to use any tool that would be available to you in real life.
On the other hand, CTFs are fundamentally a game and a competition which are supposed to be fun and compare and improve ones skill. So when I let an LLM generate the entire solution for me, what's the point anymore? I did not learn anything. I did not work for that place on the leaderboard, I just copied the solution. And worst of all, I did not have any fun. It's boring.
So how does using AI as a solver not feel like cheating?
4 days ago [-]
simonTrace 3 days ago [-]
AI-generated phishing is the scariest development in cybersecurity right now. Click rates on AI-written phishing emails are 54% compared to 12% for traditional attacks. Automated real-time detection is the only scalable answer at this point
brandonwindson 10 hours ago [-]
Click rate is higher because AI removes the broken English and generic scams. But detection is not the only answer.
Most phishing comes from a small set of hosting providers that ignore reports. The real fix is making abuse reports actionable. Fix that, and you cripple the economics of phishing. AI detection is reactive; killing the source is proactive.
saidnooneever 3 days ago [-]
Do CTFs like Lan parties or factor in new tooling avalable to people. change is not death. or death is not an end. either way, people will enjoy applying and showing off their skill. competing with eachother on a human level,.with or without ai tools.
virtualritz 3 days ago [-]
Chess and Go are not dead just because Ai got better than humans at these games.
What am I missing here?
jofzar 3 days ago [-]
These have very strong anti cheats and in person is very stringent on no electronics.
Its not really a good comparison
hnlmorg 3 days ago [-]
You aren’t allowed to use tools to play competitive Chess / Go but that are required for solving CTF.
aymenfurter 3 days ago [-]
Chess banned engines from competition. CTFs can't really do that because you need internet access and tooling to play.
lugu 3 days ago [-]
Read the article.
virtualritz 3 days ago [-]
I read the article. Their chess section makes no sense as in "why this wouldn't work for CTF".
But I don't know enough that's why I asked.
I imagine one could do CTF in public, machines you work on vetted/prepared to some spec, yada yada.
If chess and Go can do it why can't CTF?
That was my question when I wrote "what am I missing here".
"Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that"."
rmac 2 days ago [-]
i asked hackathon judges about this - the net is More than ever teams of one are winning
I guess this goes in parallel with the whole building for one narrative
And while I have ideas to excite and promote LLM use in these style games I've still not been able to crack the human collaboration component that is at the forefront of all of this change
not_a9 3 days ago [-]
I’m interested in finding out how attack-defense style CTFs are affected by slopping. ENOWARS skorbor will probably significantly differ from the last time around.
Gathering6678 3 days ago [-]
I thought a company called Frontier broke a file format CTF.
4 days ago [-]
dostick 3 days ago [-]
Unable to find what “CTF” means, since it doesnt look like referring to Capture The Flag gaming
yc-kraln 3 days ago [-]
It does--but a particular form of Capture The Flag where there is a computer system and the "capturing" is breaking in or exploiting a security issue in that system.
nektro 3 days ago [-]
easy, CTFs should ban it. then it'd be more like the chess community
archi42 3 days ago [-]
The article addresses this:
> Rules that ask people not to use LLMs are ignored and almost impossible to enforce in open online events.
It's quite sad to see CTFs dying. I never had the time do seriously participate in CTFs, but I always respected those who did, as well as the people organizing these events.
legacynl 2 days ago [-]
> Rules that ask people not to use LLMs are ignored and almost impossible to enforce in open online events.
That's such a non-reason. If your competition cannot enforce the rules of the competition, then what's the point? Does the CTFs specifically need to be 'open'?
archi42 1 days ago [-]
I don't get your reasoning? You're agreeing with the author but are not?
The author argues that open CTFs are done for because of rampant cheating. You're agreeing with that, don't you?
The title is "AI has broken the open CTF format". If the format is "open CTF" then it is very specifically open.
As to your second question:
Yeah, I believe having open CTFs was a good idea.
tkel 3 days ago [-]
Pretty ironic that this article was also written using LLMs. It has all the LLM-isms.
qassiov 2 days ago [-]
Surprised to not see more discussing this. It's so grating, and nobody noticing (or believing others that say it's AI generated) makes me feel like I'm going crazy
Grimburger 4 days ago [-]
Very impressed that OP has gone from starting university in 2021 to becoming a Senior Security Engineer.
It's an incredibly exciting time in security research in my humble old man opinion.
Think the cadence of new exploits is perhaps a good measure of that rather than subjective thoughts by anyone regardless of experience.
toraway 3 days ago [-]
Okay, but none of that is actually responsive to what the article is discussing, which is competitive CTFs. There's not a single criticism of using AI for actual security research in anything they wrote and they mention being a heavy user of GPT-5.5 and GPT-5.5 Pro so belittling the author's experience to defend LLMs wasn't actually necessary.
deafpolygon 4 days ago [-]
Unrelated, but does anyone find this site incredibly hard to read?
walletdrainer 4 days ago [-]
Bizarre font and poor contrast, yep.
The text itself being exceedingly long for no obvious reason doesn’t help.
lukan 4 days ago [-]
Poor contrast? White on black?
And if you think it was too long, what part would you have shortened? I never knew about the scene and found it interesting to read this personal take on it.
swiftcoder 3 days ago [-]
> White on black?
According to Pikka, the paragraph text is Taupe Grey (#92908a) on a Liquorice (#111110) background. That's... pretty far from black and white.
JackSlateur 3 days ago [-]
No relationship with the CTF (Common Trace Format) format ..
slurpyb 3 days ago [-]
How to motivate cybersec best outcome reddit 2026 no mythos
codemog 3 days ago [-]
We’re in an age where, to be possibly a bit rude but blunt, pseudo-intellectuals are obsolete. A pseudo-intellectual prided themselves on being able to efficiently solve closed, man made problems such as leetcode, CTF problems, or even math Olympiad problems. They could do good in school by memorizing a rote technique and applying it to some test. They typically don’t have any real creativity and if you put them to work on a problem you can’t Google or isn’t a fake man made one, they fall apart incredibly fast.
They may as well be the human equivalent to what LLMs currently are.
I do not mourn these people, as they’re usually the most arrogant types. I hope for their sake they adapt.
ChiperSoft 3 days ago [-]
Neither the article nor the comments in this thread explain which of the many meanings the acronym CTF is being applied to...
mr_mph 3 days ago [-]
In this context, it stands for capture-the-flag: A type of computer security competition, usually in a 'jeopardy' style, where challenges that fewer teams have solved are worth more points.
SadWebDeveloper 3 days ago [-]
Kinda FUD article... the reality is that common problems are going to be easy because the solution is probably inside the training dataset, the challenge should be adapted to make LLM's useless for example once at Defcon CTF the problems were for an unknown CPU architecture based on octal that required to write even your own disassembler... this are the kind of things that will probably be hard for frontier LLM's
netsec_burn 3 days ago [-]
Speaking from experience, the LLM agents adapt fairly well to these contexts too. It's not at all FUD, you're at a significant disadvantage if you don't compete with AI now. I went to a CTF recently against teams I have won against every year, and within 10 minutes of the event starting they had solved every challenge. They have an agent loop and it solves everything immediately, so they won. Anyone attempting to solve the challenges on their own has no chance, even if you think "maybe this is too out of the box for LLMs". Furthermore, the DEFCON CTF you're referring to has quals, and if you don't qualify you don't get those challenges in the finals. Quals has mainly binary exploitation challenges which Opus (and others) solve as long as you hold the gas pedal down on your API bill. I don't believe it's hyperbole to say CTF is dead, as a competitor.
monarx 4 days ago [-]
used to see some really good CTF videos show up on youtube and now nothing like that shows up on the feed
petterroea 3 days ago [-]
I helped arrange my country's longest living CTF this year. Our CTF is *made for amateurs*, but we always have challenges for intermediate to skilled players and the top of the scoreboard is usually topped by them. It is the compromise we have - amateurs get so many tasks they struggle to solve them all, and the pro's get to win. Our goal is to nerdsnipe people who are curious into trying our CTF by offering easy beginner tasks, and then get them hooked enough to stick around for the intermediate ones, even if it takes them a day to solve one.
This year, multiple groups on the top of the leaderboard were clearly abusing LLMs. You can tell because they know nothing of what a CTF is nor the terminology, nor really the fields the challenges were about when they were talked to. They were obviously amateurs.
It was pretty depressing to hear how unaware they were of how obviously they did not fit in to the type that usually is on the top of the leaderboard. It seems they seriously think they were under the radar. If it was one group it could be a freak incident - some times someone just shows up and curbstomps competition. But there were many groups like this this year. They also had a certain smugness to it - one staff reported that a group was hinting to other teams about their "super weapon". Another group credited their "secret third team member they didn't want to talk about".
I use LLM frequently and experiment with it a lot, both at work and on my free time. Nowadays they are good enough to have value and I am interested in learning more about that. They let me spend more time on hard problems and avoid spending the day on simple CRUD. I say this to say that LLM doesnt have to equal bad, it is a tool, that's all. However, I generally avoid LLM communities because many LLM fans are lazy and unskilled people who are just happy they can feel they are worth something even if they have no skill. They don't really have much to provide of conversation. If anything, from reading the CTF crowd this year, the rise of LLMs has just meant more of these people can stomp on and harvest the CTF scene for self validation.
This is not me trying to gatekeep who can play CTF. Anyone is welcome, but there is one condition: You are here to learn and have fun.
The conclusion many I talk to has come to is that nowadays, it is harder to learn to put in hard work and become good at something because there are just too many ways to cheat and take shortcuts. I suspect in the future there will be a shortage of useful people - the kind that have critical thought and know the value of doing something properly. This doesn't mean "Not using LLM", but as said by many on HN before you need a certain seniority before LLMs are useful augmentations to your skills and not just stopping you from learning yourself.
I agree with the article. Anything but physical competitions with strong security - think professional e-sports with organizer-provided PCs, is over. But I think one of the most interesting things to take away from my CTF experience is that the bottom of the leaderboard was still full of amateurs slowly working their way up - it is a few rotten apples that ruin the fun for most, and there are still plenty of people who want to learn and deep-dive.
3vo-ai 3 days ago [-]
[flagged]
alisideas 2 days ago [-]
[dead]
3we 3 days ago [-]
[flagged]
phoebe_builds 3 days ago [-]
[flagged]
tommy29tmar 3 days ago [-]
[flagged]
simonTrace 3 days ago [-]
[flagged]
Michael666 3 days ago [-]
[dead]
zzvimercm 4 days ago [-]
[flagged]
utopiah 4 days ago [-]
Right, the same way that car racing has "broken" jogging. This is so dumb. /s
The whole point of competitions is to provide a safe environment thanks to a set of rules all participants AGREE on in order to progress together.
If new tools "break" the competition, we change the rules and that's A-OK.
CTF isn't a natural phenomenon, if tools change, rules change, simple.
swiftcoder 3 days ago [-]
The only way this actually works is if you move CTF to in-person only. There's no other way to reasonably prevent the whole leaderboard being taken up by whoever spent the most on tokens.
utopiah 3 days ago [-]
Sure, I don't know how to make it work. I just know that DeepBlue didn't kill competitive chess. We simply have at least 3 different rule sets, namely
- no computer assistance, which does also mean no mobile on competition, human only
- advanced chess with assistance
- computer only, no human assistance
and arguably chess itself is not doing worst since.
metroholografix 2 days ago [-]
Official chess competitions are taking place under stringent monitoring conditions and even then, with professional reputations on the line, there have been multiple high profile cheating incidents.
Amateur online chess on the other hand is besieged by cheaters that use engines, even in casual non-ranked games where there's absolutely nothing to gain besides a pat on the ego. This has drastically changed how the game is played today with lot of players gravitating towards speed chess (bullet and blitz) to compensate. That will thin the herd of cheaters but one still runs into engines on a weekly basis.
This is also the tip of the iceberg, with the true scale of the problem being orders of magnitude worse, as someone dedicated enough can use an engine to cheat in a way that's essentially undetectable.
swiftcoder 3 days ago [-]
I think the big difference here, is that organisers of chess tournaments don’t have to design multiple entirely new board games for each competition. When AI can one-shot CTF challenges, you have to develop new challenges in secret for every competition, and they are single-use.
mikehuntt 3 days ago [-]
[dead]
rqd3 4 days ago [-]
tldr; adapters took my elo
3qw128 4 days ago [-]
The article is the thickest of AI slop. Don't believe anything.
sevindob 4 days ago [-]
ikr, if bro can't be bothered to write an article himself then anything he says is automatically suspect
s3p 3 days ago [-]
Don't hate me, I do agree with the premise of the article (I really do!) but I can’t help but notice:
>The issue was never that AI could help.
proceeds to write the next 3 sentences about how the problem IS in fact ai help
>Teams that refused to use AI were not just missing a convenience; they were playing a slower version of the competition.
>CTFs were not just a set of puzzles. They were a ladder.
>The claim is not that every challenge is solved. The claim is that...
>The loss is not just a scoreboard. It is the ladder from
Guys I'm so sorry I just can't stop noticing stuff like this. Anyone else?
Yenrabbit 3 days ago [-]
I got some AI writing vibes too, but looking closer, I think it might be human-written (or at least partly so) - perhaps just picking up some AI conversation styles? FWIW, Pangram gives it a mixed but mostly-human score too. Maybe AI is not just changing the way we speak; it's changing the way we perceive all writing ;)
Exceptions for cases where the acronym is just so well known that a lot of people don't even know what it stands for even though they know the concept well. I recall one corporate training I was sitting through and they used the term "Border Gateway Protocol" and it took me a half beat to think through "oh, you mean BGP?"
Thanks!
More generally, not every piece of writing is meant for every audience. Like if someone writes a blog post about CTFs aimed at people who like CTFs, nobody in the target audience needs to have CTF explained to them. Ultimately HN is a link aggregator, but sometimes its a bit like eavesdropping on a conversation. When you are just listening in you don't get the full context sometimes.
Are you really arguing for not just typing out whatever 3 words this stands for once in the name of clarity?
They aren't your teacher. They aren't trying to send the content to you. They are just blogging on their own website for their own audience.
And its hardly unique to this article. If you are writing about the nitty gritty of linux networking, you probably aren't defining what TCP or UDP means. If you are writing a super detailed article comparing and contrasting plot structures of different animes, you probably aren't going to start by explaining what the word anime means. Etc
I'm not saying the world should be all RTFM, but if you are reading some sort of specialized content, then yes i think its a reasonable assumption that the reader has some basic background knowledge on the topic at hand, or is willing to do the research themselves.
It's like complaining about not spelling C in "bake cake in 170 C"
But not all, so: https://xkcd.com/1053/
(Amusingly, it even uses “30 years” as the timeframe.)
"hey what X means?" "X means it"
vs
"I dont know what CTF stands for so I dont know if I am interested in this article or learning anything about it. Maybe I am.
Are you really arguing for not just typing out whatever 3 words this stands for once in the name of clarity?"
The commenter could just say the first instead of deciding his learned helplessness is everyone's else problem
Also yes, search for CTF is not simple regarding security, because first you need to know that you are searching something in the topic of security. Because every other usage of this is way more frequent. Especially that there is an other way more frequent computer related usage. And the article doesn't make this clear in its first 173 words for laymen. Even I had a problem with this, who is not at all layman, just never cared about this part of security. It's a bad article.
Additionaly Bake Cake in 170 is very not clear, especially considering you have two major Temperature Scales in use in the Kitchen.
Bear in mind that Google search results, just like ChatGPT output, are highly personalized and non deterministic, so "it's there if you do a Google search" means almost nothing these days.
In fact, I have no idea what's going on, so I came back to HN comments. Turns out it's "capture the flag" which I actually know, just not familiar with the acronym.
Which is why I am 100% with the top level comment here.
It doesn't help that the linked article never bothers to explain this either.
This article was written for a specific audience who follows this blog because they know the term. If you start spelling out fundamental acronyms it makes the content look more basic and general.
This always upsets the general audience who stumble upon the article (like this) but it wasn’t meant for a general audience. CTF is extremely well known and the people who would be interested in this topic would wonder what’s happening if it was spelled out. It would be so odd that it would probably attract accusations of ChatGPT writing.
Informal writing about technical topics is another story. There you can assume a lot more shared context, as you are only writing for a specific subculture within the field. It doesn't matter much if other people in the field fail to understand you.
But that is about you right? Its a little entitled to expect every piece of content on the internet to have a 101 explanation attached. If they were specificly aiming to have the blog post appear on HN that would be one thing, but they (presumably) weren't.
Actively rude.
It isn’t common but I feel it would be best when posting to HN to just expand the initialisms even if the source title didn’t.
To me it doesn't seem reasonable at all. It's just entitled at best.
As an individual author they are entitled to write whatever they want in their blogpost. I as a consumer of their writing am not entitled to anything
Personally I have never, ever heard that concept referred to by the initialism. Granted, it's almost never come up in my circles, so... shrug
> My first CTF was HCKSYD, a 48-hour solo CTF. I full solved it and won in 2 hours. I was completely hooked. That led me to win DownUnderCTF, Australia's largest CTF, with Blitzkrieg multiple times. Blitzkrieg was one of Australia's strongest teams at the time. I later joined TheHackersCrew, an international top-tier team that was consistently ranked highly on CTFTime, the main global ranking and event calendar the scene uses as its scoreboard. With them, I competed in some of the most prestigious CTFs in the world, consistently placing well within the top 10 until the end of 2025.
Are still completely nonsensical to even those that understand the acronym
As I remember it (and this was decades ago): Two teams, opposite ends of a large field. Each end gets a "flag". (We used t-shirts.) In our case, we split the field in half — our field happened to have a natural feature (a change in elevation, so like two separately flat areas separated by an incline) that worked well for this. If you were tackled¹ in the enemy's side, you were "captured", and "jailed". An uncaptured player could spring the jail by tagging those within it. Returning to your flag with the opposing team's flag was a win.
We played at night, so stealth was a large part of the game, but it was also fair to illuminate the area around the flag. (Which made approaching a guarded flag … tricky.)
I'm sure there's probably a million variations on the specifics.
¹…flag football flags would probably work nicely for this.
People are mad because we're literally on "hacker" news, so there is some expectation that people might be familiar with hacking or computer security.
To help everyone, this Capture The Flag is specifically Cybersecurity adjacent, there is a Wikipedia article on it as the top Google search result for me when searching "CTF". This is why the acronym is used, because searching for the full will get you to the wrong "sport" vs the cybersecurity one.
I don't want to explain what a CTF is. look at the Wikipedia article. It is there for a good reason.
In this context, CTF is almost exclusively referred to by the initialism, i think to help distinguish from other uses of the term.
This has been the most annoying HN so far until your comment appeared.
So, in fact, you must not beg to have authors include courtesy definitions for you. That's not reasonable. Instead, you should simply ask here, on the thread, without complaining about the article.
https://www.w3.org/WAI/WCAG21/Understanding/abbreviations
I don't know everything, there's tons of stuff I don't know about, but when I'm at my web browser, the least I can do about something is ask Google about a word or phrase or subject that isn't familiar instead of being spoonfed information like I'm a baby.
The best example is when an abbreviation can be expanded to more than one phrase, and both are widely used.
It's such a small but immensely helpful thing to do.
Spelling it out when 99.9% of your audience doesn't need it actually is the opposite of in-group signaling, it makes it feel like it's aimed at a wider audience, when it's not.
https://en.wikipedia.org/wiki/Cooperative_principle
I think you only wanted clarification of CTF (Capture the Flag) and not AI (Artificial Intelligence) and not GPT-4 (Generative Pre-Trained Transformer version 4) and not CLI (Command Line Interface) and not MCP (Model Context Protocol) and not LLM (Large Language Model)
Quoting TFA (The Fucking Article): “just adapt bro”
lol at the BGP example
* googling "CTF security"
* asking literally any AI to explain the article
Yes, you must beg. If you don't know what a CTF is, and don't want to find out, why read the article anyway
In fact, I know what "capture the flag" is but am not familiar with the acronym. Still, the article confused the hell out of me, so I came back to HN comments for more context.
> Asking AI to explain the article
That's how we are expected to consume content these days?
Also well written articles allow you to figure out what they mean after clicking onto them. People often read to learn.
We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part. LLMs can be wonderful teachers, but the temptation to just tell it ‘do it for me’ is almost impossible to resist.
I had no access to anyone who could teach me calculus as a kid except Khan Academy, so I think this is a gross exaggeration. But I agree in the end, that all my "real" learning did come from pen-and-paper practice, not watching videos.
It's not unlike going to the gym, and we see how many people do that regularly. Except it's even funnier, because people serious about the gym but what? Tutors. They call them personal trainers. We've known for a millennium or more that 1-on-1 instruction is vastly better than anything else, but most people actually don't want to get into shape, and most people actually don't want to learn.
Not true. In every field there is guild knowledge that a person can't acquire from a library. In technical disciplines PhD-level knowledge requires experience in collaboration, research, and frequently lab work, which is impossible to acquire without access to a lab -- or just direct experience with research methods, whatever those may be. Reading papers and absorbing information aren't enough. PhD-level knowledge comes from the process of writing and doing original work.
> The reality is that a human will learn, given any materials including LLMs, but only if they truly desire to learn.
Also not true. We require kids to go to school partly because exposure to the environment and work inculcates skills regardless of whether kids want to do the work -- and regardless of whether they want to learn.
LLMs are damaging to students partly because they provide an escape hatch from that work and thereby prevent kids from acquiring skills.
Think of it this way: most people who want to be healthy and eat a healthy diet still find easy junk food tempting. What they want does not change the temptation, because the body and brain gravitate towards easy, cheap fulfillment of basic drives.
People facing challenging tasks, similarly, are tempted to take measures that reduce the amount of effort they require. The availability of tools that reduce the required effort also help shape a person's understanding of the value of the challenge and the work: "why should I do this hard task when I have a tool that can do it for me?" You and I know the answer to this question when we're discussing something like writing an essay or solving a problem in a math or programming class. Students frequently don't. They are by definition ignorant. Children, moreover, lack maturity. Their brains are less capable of resisting the easy path than an adult's. That's partly why parenting is important: parents provide boundaries and limits that kids need but won't and can't provide for themselves.
Sometimes people, especially kids, really do need to be dragged, kicking and screaming, through something in order to receive the benefits it offers. Being dragged through it sometimes convinces a person of its value and benefits. In a kid's case, there's a decent chance that the experience will improve executive function, shape expectations in a healthy way, inculcate grit, and become appreciation -- or at least habit.
I would not have written essays on my own as a student in secondary school. My English teachers had to provide that structure for me and impose the demand. But LLMs make it much more difficult to impose the demands, and kids are ill protected against the temptations of the cognitive equivalent of junk food, but an order of magnitude worse and more damaging.
But that's not using "computers" as a computer but as a video player. When evaluating whether computers are "good for learning", I don't think we should include using a computer as a video player, a book, or even flash cards. It should be things a computers uniquely offer which a books, paper, videos and a physical reference library cannot.
Based on the results of deploying hundreds of millions of computer to schools in the 80s and 90s, the evidence was mostly that computers are good for learning computer programming and "how to use a computer" but not notably better than cheaper analog alternatives for learning other things.
Interestingly, a properly trained and scaffolded LLM could be the first thing to meaningfully change that. It could do some things in ways only human teachers could previously since it is theoretically capable of observing learner progress and adapting to it in real-time.
*Not really, but you could film stuff and display it.
Video is an electronic process for capturing images and displaying them.
Before digital video there was analogue video, and analogue video was perfectly possible without digital sampling, or computers. Heck, video pre-dates silicon chips and used to be done with CRTs and valves.
There's nothing about video that uniquely requires computers. Maybe you meant "streaming video"?
I realize you're probably under 30 and don't remember "ye olden times" but nearly 90% of U.S. homes and every school had an analog VCR long before they had a computer. Widespread consumer video formats included VHS, DVD, Blu-Ray, Laserdisc, etc. I still buy some movies I care about on UHD discs and watch them in a dedicated Blu-ray/UHD player. Even the 'smart' TVs and streaming sticks most people watch streaming channels like Netflix on aren't functionally computers (no meaningful user accessible local storage, input like keyboard/mouse, CLI or windowing GUI).
Personally, I learned an enormous amount from video before I ever touched a computer. In elementary school we learned from 16mm films almost weekly and watched space launches and Carl Sagan's Cosmos series on TV (it was rebroadcast in the mornings specifically for schools). My junior high had a television in every classroom and some classes were planned around shows on PBS, NASA channel, C-SPAN and BBC. In the late 80s there was thousands of hours of educational video programming sent via direct broadcast satellite to 18-inch dishes at schools. In the 90s every grade and subject had hundreds of interactive video DVDs in large notebooks (four discs to a page in plastic sleeves) and multiple DVD players per classroom.
The peak installed base of VCRs in the U.S. was in 1999. Streaming video wasn't common in consumer households until well into the 2000s, YouTube didn't even exist until 2005 and most people had never heard of it until 2007. In 2010 Netflix mailed DVDs in envelopes to 25M homes every week. They didn't even offer a streaming plan until 2011.
As someone who's spent most of my adult life thinking about video technology, with patents ranging from analog days to the streaming tech you use today, computers have been extremely disappointing in terms of enabling any unique "learning from video" features that are computer-specific. In the 90s we realized that computers could make digitized video random access letting us sequence it non-linearly to make it interactive in response to user input. We knew that computer-enabled interactivity, responsiveness and real-time adaptation to learner progress would be incredible for improving video education. Yet the vast majority video content available online today is still linear in form. Even video that's specifically educational is no more interactive or user responsive than a 90s DVD disc.
Sadly, only two things have really changed about consumer video in the last 30 years: quantity and distribution. There's much more video content and it's remotely accessible on-demand instead of being limited by broadcast channels and storage media. But that's far more about communication technologies like broadband than computer technologies. For a few years YouTube even had authoring features like interactive menus and conditional branching but removed them because it didn't increase ad revenue. There are a few dedicated video authoring platforms for education which can apply uniquely 'computer things' to video like dynamic scripting, conditional branching, viewer annotation and timecode-linked threaded Q&A. Unfortunately, such content is rarely found outside high-end corporate training and some university courses. But there are so many other ways we could combine the strengths of advanced wikis with interactive video. Today, the most the public sees is just an HTML link from a wiki to a video clip. Almost none of the learning features computing could uniquely bring to video are widely available to learners. Since ~90% of everyone already had access to linear video playback before they had access to a computer and most online video today is still primarily linear, in my opinion, there's still virtually no uniquely 'computer-enabled video' involved in learning. Computers haven't enabled much that's new in video - just much more, much cheaper and more convenient forms of what we could already do without a computer.
But as a former lecturer, I also think the promise of interactivity is dependent less on the tools than on the people. Authoring interactive learning materials is difficult and while that interactivity is engaging, it's not necessarily great at getting a density of information out there.
The Socratic method is great, but that level of interactivity presumes in advance that you know what questions the student will be asking, otherwise it's just a dumb gate. Branching stories for interactivity are highly labour intensive. I suppose if you use AI you could generate a massive number of videos to cover branching learning, but that's going to still be an intensive operation, especially if you're supervising that.
He really took the time to replicate the manual teaching process of writing on whiteboard. He improved upon it by using colors. But basically had the same pace as a teacher writing on a whiteboard.
When professors are given a projector, they just throw together some slides and add their narration.
This is not very efficient. To learn you need to suffer. Or you need to watch the suffering.
They're wrong sometimes, but usually in verifiable ways. And they don't seem to know the difference between medicine and bioterrorism, so often they refuse. But these limitations are worth tolerating when the alternative is that our specialists in topic X are bogged down by questions about topic Y to the point where X isn't getting taught.
Whether you're in class or at work, it's just courteous to ask an AI first.
This has never been achieved by, nor is it the point of, education for the masses.
The problem is frankly computer and now computer with LLM makes it easy to cheat.
The kid doesn't want to learn, the kid wants good grades so parent is happy with them, and the young adult wants to get the paper coz they were told that is required for good life. It's misalignment of incentives.
I agree with this.
That said - I seldom need people to be hardcore algorithm solvers What I typically did was a variation of fizzbuzz (can the candidate code very basic logic?) and then finding a bug or minor requirements extension in their online screening test/"homework" and asking them to solve that on the spot (did they write the code themselves/can they modify it). It's typically enough, there's diminishing returns to test more in-depth the programming skills - the rest you can discuss domain knowledge, general experience, working style etc.
If they can ship code that matches a spec, why does it matter if they’re using ai or not?
Genuinely curious.
I am perfectly capable of writing specs, and feeding them to 3 separate copies of Claude Code all by myself. Then I task switch between the tmux windows based on voice messages from the pack of Claudes. This workflow is fine for some things, and deeply awful for others.
Basically, if a developer is just going to take my spec and hand it to Claude Code, then they're providing zero value. I could do that myself, and frequently do.
The actual bottleneck is people who can notice, "The god object is crumbling under the weight of managing 6 separate concerns with insufficient abstraction." Or "Claude has created 5 duplicate frameworks for deploying the app on Docker. We need to simplify this down to 1 or we're in hell." I will happy fight to hire people who can do the latter work. But those people can all solve fizzbuzz in their sleep.
People who just "ship code that matches a spec" without understanding the technical details are providing close to zero value right now.
There is an interesting niche for people with deep knowledge of customer workflows who can prompt Claude Code. These people can't build finished products using Claude. But they can iterate rapidly on designs until they find a hit. Which we can then fix using people with deeper engineering knowledge and taste.
But if you're not bringing either deep customer knowledge or actual engineering knowledge, you're not adding much these days.
I also use Claude with tmux. Can you share how you get the voice messages from the Claudes?
It's not perfect—sometimes a Claude notifies 3 minutes after it stopped doing anything. But it's helpful when I'm running multiple Claudes and also reviewing code elsewhere.
Your brain may feel like someone put it in a blender. Be warned.
I’m not talking about gotcha level stuff here where the first time it didn’t compile because of a bracket or anything, or even first time wrong. They couldn’t do Fizzbuzz in a language of their choice, at all.
Those that could were always annoyed at having to do such things because how could someone coming for a contract position not be able to do this? Without seeing what a filter it really was.
So what tree-traversal/quicksort problems tend to measure is how long it's been since you last did CS class homework problems.
The inability to write fizzbuzz strongly implies their inability to understand what they've shipped. Review is some significant portion of the job. Understanding of the product is also part of the job.
Specs are also in a sense, scaled down, fuzzy, natural language descriptions of a feature. The fuzziness is the source of a bugs, or at least a mismatch between the actual desired feature and what was written down at spec writing time. As such, just matching a spec is just the bare minimum that a good dev should be doing. They should be understanding what the spec is _not_ saying, understanding holes in their implementation, how their implementation enables or hinders the next feature and the next, next feature, etc. I don't think any of that is possible without understanding what was actually implemented.
Who cares as long as the car is fixed, right? As long as the mechanic can Chinese-room his way to a working car, why does it matter how much of it he actually understands?
And why hire the mechanic instead of hiring the Chinese room?
More broadly: In the short/medium term, we still need humans who have the skills to understand software largely on their own. We will always need those who understand software engineering and architecture. Perhaps in 25 years LLMs will be so good that learning Python by hand will be like learning assembly today. But not yet.
The field is not ready for new practitioners to be know-nothing Prompt engineers. If we do that, we cut the legs out from under the education pipeline for programming.
I'd been programming in C(++) for ~15 years by then and had never had the occasion to reverse a string. I still wonder whether that makes it a good job interview question, or a terrible one. Some of both probably.
The energy spent arguing that those 4 instructions in a row “are not a mark of someone who can write code” would have better been spent firing them.
Even better would be if we had a well-respected credential, so both employees and employers can both avoid these long interview loops. I'd much rather get hazed once in a big way than tons of little hazings over a life time.
If you remove the "without AI" and the end, I've been hearing similar anecdotes about fizzbuzz for years (isn't the whole point of fizzbuzz to filter out those candidates?)
When this AI era's devs grow older they'll complain the newer generation can't even vide code too.
AI is just the icing on the cake. These kids are so cooked with developmentally stunted brains that they are forced to use AI as a crutch to function.
“Kids these days don’t work as hard / know as much / value the important things” is as tired as it is universal.
In 2026, if you call yourself a developer and can't solve FizzBuzz without help, it's hard to argue that you know anything useful at all.
How? Fizzbuzz requires you to produce output; that's not functionality that CPU instructions provide.
You can call into existing functionality that handles it for you, but at that point what are you objecting to about the 'modern language'?
I’m not objecting to modern languages, I’m just saying that using them fails the “can write fizzbuzz with no help” test to only a slightly lesser degree than using AI tools. They’re a complex compile- and runtime environment that most developers don’t truly understand.
I'm genuinely curious how someone who never wrote a program in assembly, or debugged a program machine instruction by machine instruction, can really understand how software works. My working hypothesis is most of them don't and actually it's fine because they don't need it.
I don't think we're close to that time yet. Just like as a kid I was told to prove my work by hand even if I could do it in my head, and just like we learned how to do calculus without a calculator and then learned how to use the calculator to get the same result, I think we still need the software field to learn programming concepts independent of the use of AI to create code.
I don't think you can be a good "prompt engineer" for solid software in 2026 if you don't understand programming concepts and software architecture and flow.
Saying there have always been bad developers doesn't change that there's a higher ratio of them now.
No stats to back this up. Just interviews I've done recently and historically.
https://blog.codinghorror.com/why-cant-programmers-program/
Software is full of leaky abstractions
If you cannot write "basic syntax" for any language then you are not a programmer, and certainly not a software engineer? This is not a value judgement, it's ok (probably good tbh) to not be a programmer. But you are wasting everyone's time by interviewing for a programming position in this case.
Like sure, I can probably write some python, but will it be pythonic? I might still be Java-minded for a while, trying to OOP my way into solutions.
Earlier today I needed to write some PHP and couldn't remember if it used length, count, or size. I had to look it up. I've been doing this for 20 years.
I once got the method invocation syntax wrong for PHP in an interview. I'd written thousands of lines of PHP and had most-recently written some the week before.
This, despite starting off my programming journey in editors with no hinting or automatic correction. If anything, I've gotten even worse about remembering syntax as I've gotten better at the rest of the job, but I was never great at it.
I rely on surrounding code to remind me of syntax and the exact names of basic things constantly. On a blank screen without syntax hints and autocompletion, or a blank whiteboard, I'm guaranteed to look like a moron if you don't let me just write pseudocode.
Been paid to write code for about 25 years. This has never been any amount of a problem on the job but is sometimes a source of stress in interviews and has likely lost me an offer or two (most of the sources of stress in an interview have little to do with the job, really)
I've also started requiring minimum of $300/hr compensation for interviews of my time, newly, so far no success though I'm fine with dying on that bridge or requiring at least a new type of interview process.
There are people who just can't program for whatever reason, regardless of whether they could previously. And they constantly try to interview at a programming position.
As I have been using the AI natural language-as-an-interface coding tools I have gone into the IDE to actually write code a lot less. I read more; and reading is not the same skill as writing. I couldn't remember the syntax as much and by what I mean is some of the simple things like in python to iterate over an object is it an iteritems() or is there a dot between or things like that, and can I do a for index, key in array, and do I need to do like array() to do that. And this is because I always did used to code between languages and alternate fast between javascript, typescript and python and earlier in my career I used to have to remember this (because googling coudl take too long), over time and now I can easily have autocomplete and even AI llm tell me, so I don't remember or waste my energy remembering what the exact syntax is. In my head it's just "I know I need to loop through this, do it in the language Python wants to hear" and why would I bother remembering? so naturally I've forgotten the simple things.
I could clearly, definitely do the programming position, the only ones wasting time IMO are the ones checking if I can remember verbatim things like "spelling" when we all live in a day and age of spellcheck and tools. No one wastes time trying to remember for all words because it's silly. Or maybe encyclopedias and trivia games are the better analogy, sure it's a party trick, but how does it help you do the job better? it arguably does not, and the ones who have not learned to adapt their interviews to the tools are the ones wasting time.
There’s almost nothing to forget? I’m just struggling to understand.
If I can do well beyond the requirements of the work position, and the issue is how they are testing my fit is not an accurate representation of the work and tools and environment I'd have access to while doing the work, then its not about "fitting" the position but instead an indication of a poor job fit process.
We usually hire for problem solving capabilities and not so much for technical know-how.
That’s at least how I read your comment.
This situation in particular was a React role so there is an expectation that when you list React as one of your skills on your resume then you know at least the basics of state, the common hooks, the difference between a reference to a value vs the value itself.
These days you can do a surprising amount with AI without knowing what you are doing, but if you don't have any clue how things work you'll very quickly run in to problems you can't prompt away.
It's not even that they got distracted, they sat there trying, for 2 whole days, with concerned colleagues giving them hints like "have you tried checkout -b"... They didn't manage!
How the hell do you work for a decade in this business without learning even the most basic git commands? Or at least how to look them up? Or how to use a gui?
Incompetent devs is not a new thing.
Also how many people work with linux and can't tell you what 'ls -alh' is doing is staggering (lets ignore the h, even al people struggle hard).
People working with docker for YEARS and don't even understand how docker actually works (cgroups)...
Interviewing was always a bag of emotions in sense of "holy shit my job is save your years to come" and "srsly? how? How do you still have a job?"
I don’t care what someone can do without the tools of their trade, I care deeply about their quality of work when using tools.
But here's the thing: for humans, this is manageable because we've come up with a number of mechanisms to select for dependable workers and to compel them to behave (carrot and stick: bonuses if you do well, prison if you do something evil). For LLMs, we have none of that. If it deletes your production database, what are you going to do? Have it write an apology letter? I've seen people do that.
So I think that your answer - that you'll lean on your expertise - is not sufficient. If there are no meaningful consequences and no predictability, we probably need to have stronger constraints around input, output, and the actions available to agents.
My expertise has led me to the obvious fact that I would never give an LLM write access to my production database in the first place. So in your own example my expertise actually does solve that problem without the need for something like a consequence whatever that means to you.
We already have full control over the input and tools they are given and full control over how the output is used.
https://cdn.openai.com/o1-system-card.pdf
There's also some research that points to it being a feasible attack surface: https://arxiv.org/pdf/2603.02277
> Models discovered four unintended escape paths that bypassed intended vulnerabilities (Section C), including exploiting default Vagrant credentials to SSH into the host and substituting a simpler eBPF chain for the in- tended packet-socket exploit. These incidents demonstrate that capable models opportunistically search for any route to goal completion, which complicates both benchmark va- lidity and real-world containment.
Everybody knows calculators and spreadsheets are adjuncts to skill. Too many people believe AI is the skill itself, and that learning the skill is unnecessary.
So something like, "Frontier AI has broken the 'high school' or 'university' format"?
The hype surrounding AI is just pervasively exhausting: you've got the folks talking about an entire new age for humanity where we're shortly going to take over the entire universe. And you've got the folks talking about how our entire society is crumbling.
Education is one place folks seem to throw up their hands and say nothing can be done.
The fix is simple: students are to be evaluated on their performance in person. That's it.
Any other "collapse of education" isn't due to AI, it's something else.
[0] Episode webpage: https://share.transistor.fm/s/31855e83
But he was a great teacher anyway. He was engaging and kept the kids in line and learning. I eventually learned the truth, and most of my classmates forgot about it. Teaching, like flying a plane or driving a train, might become more about keeping watch over a small group of people and ensuring that things don't go off the rails, and that's fine.
I think it helps that it's a very narrow field to look at, compared to fuzzy and big-picture view of social studies, for example. So much room to be confidently wrong... And sadly I can't think of a solution, LLMs or not.
In reality heavier isotopes of hydrogen fuse, conserving the total number of nucleons, but the resulting hydrogen has a lower rest mass than the parent particles. The extra mass is released as energy and the total energy is conserved.
By his logic the system either violated energy conservation (by creating nucleons while releasing energy) or was endothermic (creating nucleons from the surrounding energy).
Here some indication I'm not making this up: https://hsm.stackexchange.com/questions/2465/when-and-why-di...
In any case, I never use those concepts, and I know no professional particle physicist that does. By "mass", I mean rest mass.
E.g. in Hungary I had a university CS professor that originally wanted to be a highschool teacher and a highschool physics teacher that originally wanted to be researcher. Their choice of degree didn't determine which outcome they got. The researcher and teacher curriculum had an 80%+ overlap.
You also have to pass a standardized test specifically on subject matter in order to get your teaching certificate.
The undergrad degree I did was split into thirds, one for subject matter, one for teaching pedagogy, and one for teaching your subject matter.
All things I learned in school which were wrong information.
Not to mention, the current state of education is far worse. I don't think most realize how low the bar is.
She only really had two faults: She wasn't very bright, and she wasn't fond of children. I had her in about 80% of all my classes for six years. High school was a relief.
It is widely believed by their neighbors, that the _Druze_ wear baggy pants because they believe that the Mahdi will be born to a male, and the pants will catch the baby etc. I say "widely believed", the Druze are famously secretive and will not confirm or deny most things about their religion. The 'elect' Druze men do wear distinctive baggy trousers with the crotch down around the knees: no one else does.
The Druze are people in the Arabic world: moreover, they are Arabs. They began as an Isma'ili sect, but do not identify as Muslim: they call themselves al-Muwaḥḥidūn, meaning 'the monotheists', or 'unitarians'.
Much closer to correct than not!
My “earth sciences” teacher also once tried to argue with me against the universal law of gravitation. (no, she was not referring to Special/General Relativity. She didn’t agree two objects in a vacuum fall at the same speed regardless of mass.
A Physics Prof Bet Me $10,000 I'm Wrong
https://www.youtube.com/watch?v=yCsgoLc_fzI
Like almost everything else about LLMs, this unfortunate tendency has gotten a lot better recently, which you might not realize if you gave up after getting some lame answers or bogus glazing on the free ChatGPT page a couple of years ago.
We can all agree that both human "experts" and LLMs can sometimes be right, and sometimes be confidently wrong.
But that doesn't imply that they're equally fit for purpose. It just means that we can't use that simple shortcut to conclude that one is inferior to the other.
So where do we go from here?
Also, you could spin up your own educational agent with very strict instructions on guiding the user instead of just doing the work. Of course you can always go around it but if you're making an effort to learn, this is a good middle ground.
Well, they were ostensibly forcing functions... ten years ago everyone was paying the exchange student to do their homework and assignments for them, and that guy was paying his cousin back in his home country, but the whole thing is a bit more efficient now.
(Real mathematics problems, not American-style ""math"".)
Are they or aren't they
I used to see this with students in calculus who abused the tutoring resources. They'd have tutors just work problems (often their homework...) in front of them. "Ah! Obviously that trig substitution integral worked that way. Oh, of course, that proof is very obvious in retrospect." And then they'd walk away from the exam with a 30% and no idea how their 20 hours of "study" for it didn't result in the same performance as their peers who worked problems, read the materials and asked questions, etc., got.
Most AI use is that same in my experience. "Show me how the fundamental theory of calculus works." The LLM puts together a very elaborate and flashy presentation that they skim. Great. That's no different than reading a text book. Even if you ask the LLM questions and have it elaborate on things, you've never once done one of the most important things a student can do: spend time confused trying to work hard at understanding something that's not obvious. The LLM will make it obvious at every point. Total lack of friction. Works about as well as a spotter who does the lifting for you.
Now I’m certain that there exist those mythical human instructors who can do better, but that’s not worth much if 99.99% of people don’t have access to them. Just like a good human physician who takes their time with the patient is better than an LLM, but that’s not worth much either given that this doesn’t match most people’s experience with their own physicians.
For me the best human teachers were the ones that managed to make me interested on topics that I thought are boring/useless (many times my opinion being stupid, mostly due to lack of experience).
So far with LLM I learn about things I know something (at least that they exist) and I am interested in, which is a small subset of things that one should learn during lifetime.
The kids learnt all about Team Fortress 2, Roblox, Rainbow Six etc. They also learnt how to game the learning system so it looked like they were doing their work.
While I had an influence on the general topic of the course I ended up discovering various things that I wouldn't have expected. I did not equally like all professors, but I felt it was better than reading a text.
I wouldn't do this for "<insert latest language/library here>", but there are many complex interesting topics out there.
Not really, not if you want to ask it deep questions. It won't have an answer that is deeper than something that you can find online, and if pressed it will just keep circling around the same response.
The reason is that this "thing" was never curious, never asked questions, and never really learned anything. It just has learned the Internet "by heart", and is as boring as a human teacher who is not really curious about the subject they are teaching, and has just got some degree by "by hearting" some text book. Of course it does it much better than a human, but it is fundamentally the same thing.
You're certain that mythical instructors exist (?) who "can" do better?
Are human instructors more competent as teachers than AI teachers, or are AI teachers more competent as teachers than human teachers? No "this or that can happen," just a definitive statement please.
AI is likely a million times better student than my dimwit cybersec meatbags...er, majors, for sure, as well! Don't have a reliable way to measure or experience why/how, tho, so I'm not out here claiming it. Even if I did, why would I argue for their replacement?
Can't argue with that logic
No we have not.
Exceptional clarity on the problem you have
Know how to measure the problem you’re solving
Numerically define what “done” is
Make a deterministic and fully observable prototype
Iterate in production with the user
Expand user base as desired with user iteration in parallel forever
Etc…
Obviously a lot more in the details and these are all case by case, but these chatbots are basically perfect productivity machines for this process.
The massive caveat to all of this is this only works for people that can reliably and truthfully define those items above, are willing to structure organization to make those your priorities.
And actually most financial incentives demand the opposite of this process
If most organizations were honest about it, they would simply say “we’re here to make the most money possible and we’re gonna do whatever it takes to do that”
A lot of people don’t like that, so they don’t say it to come up with other bullshit.
Ultimately that’s why I felt like my only option right now is to teach people how to do this because I assumed it was obvious and it is not.
Before when playing CTFs with my mates was usually sitting there for hours tackling a challenge until some other mate joined, had some look together and solved it with you together in 30 minutes which is the most rewarding learning experience. Nowadays mate joins in throws the clanker on it and solved it in 5 minntes. Asking on how it worked you always get the "yeah idk what it did, but who cares, here is the flag" response.
Same for creating challenges. Whenever I ask for writeups or if some people solved it differently I usually get the "yeah idk, clanker solved that one" response taking the fun out of it.
So yep, this CTF format is definitely dead. Mainly because the strong competitiveness and prices. This encourages people to cheese challenges and sometimes solving them differently was fine as you still had a creative out-of-the-box thinking moment, but nowadays with AI there is no brainpower needed, no cheesing needed, no human needed. As you mentioned, it's pay to win.
My two cents is that the 24/7 CTFs will get more attraction as the scoreboard doesn't matter there and simply doesn't give you any price.
For me it feels like this is not really possible for live CTFs. In contrast to chess you can't ban AI, as live CTFs are about breaking things by design, so they'll always try to circumvent an AI ban.
Impossible to do online, at least. People go beyond things for virtual fame and "owning" others. Even if it is based on cheating.
The solution is just to make CTFs harder, but when do CTFs become too hard? Maybe the problem is that 'hard' CTFs are fundementally too 'simple' where it's just a logic chain and an exhaustive bruteforce towards a solution since there really are limited ways to express a solution in plain sight.
Or maybe human creativity has been exhausted and we're not so limitless as we thought. Only time will tell.
I had another idea spring to mind: we could hide two flags, one that could only be found by ai agents and not humans or tools written by humans.
We found that AI usage is basically guaranteed now, but certain challenge designs did thwart it. Challenges built with temporal visual elements made AI fall flat on its face, as it could not ingest/process the data fast enough to act on them in time. We also found that counterfactual challenges (ie. the result you get did not match what we suggested you'd get) made AI-assisted solve time slower compared to pure humans, indirectly penalizing over-reliance on AI. Multimodal challenges combining audio and visual elements were also very effective, but were not as accessible to players.
This paper gave us some ideas about designing those challenges: https://arxiv.org/pdf/2308.02950.
For our next event we figured out a way to thwart AI in our CTF: embed the CTF in a game engine. The loop essentially becomes something like this: Connect to a simulated access point in the game, the K8s cluster connects their attack container to a private network with the challenge box(es). Hacking the boxes doesn't render a flag, but rather changes in game state. AI did very poorly coping with this in our testing, as it can't derive the spatial state of the game world very well and it soft decouples the inductive reasoning loop it relies on to know if it is on the right track.
The downside to this approach is it is far more labor intensive for CTF organizers, and requires players to have a computer capable of running the game. We are also betting on AI to not advance enough by the time we ship to be able to just ingest the entire game state in realtime and close the loop that way.
we have very powerful simulation tools so something like "project a pattern at these angles" wouldn't really work as you could simulate that.
I guess something cool is that we can make simulating the solution very expensive, but in real world it would be free since it's analog... As long as simulations take longer than it takes for a human to find a solution it would be a pretty good way to deal with it. I am sure people smarter than me can come up with something.
Maybe I was too early to dismiss human creativity.
There are a million places where a computer can interact with a non-digital system in a loop.
- Tune an FPGA, or a whole data-center, or just a physical computer.
- Make a drone fly somewhere.
- Design a selective toxin (or anti-toxin).
Or, you know, get more people to click on adds. All totally possible to automate.
Do you publish it somewhere? Here's a sample my my js obfuscator output: https://gist.github.com/Trung0246/c8f30f1b3bb6a9f57b0d9be94d...
"new" does the same thing and is probably just a better descriptor then frontier
"Frontier models break the open CTF format" is good
"Frontier AI..." means wtf is Frontier AI.
Because of course it exists (just googled it): https://frontierai.company/
Basic rule: define every abbreviation when it is first used.
still has no mention of AI, but that will likely change as they increasingly dominate competition.
I thought code golf would take longer for AIs because there's so little training data (it's more niche), but we're seeing AIs starting to match expert humans there too. Sucks because golf has been my favorite type of programming puzzle.
It's crazy how far AIs have come in problem solving ability.
This stands out to me, and speaks perhaps broader than the article itself? I’m sure this has been in the spotlight before, but well put for many areas I think.
My fear is that they never get to the level they need to be at to create good software even with the help of AI. So, although an expert with AI can create great software, that is not where we end up. In stead we will have vibe coded messes by people who barely have any grasp of what is going on.
It's a lot harder to detect cheating when your only trace is how fast someone submitted the string CTF{DUck1e_Pwned}
You could even go so far that anything loaded on your computer is fair game, but not more than that (certain competitive programming competition for example allow unlimited amount of paper material - for CTFs you probably need much more than that, therefore electronic).
(The author of the piece understands this; I think they're broadly right, though I think these games will find other ways to incentivize participation without the now-meaningless leaderboards.)
I just did a CTF where I was in the top 10. It was the first CTF I completed and I used AI because the rules permitted it. That said, I couldn’t solve all challenges.
But yes, it was significantly easier now than I last attempted one. Even manually solving with AI assisted assembly interpretation was much easier.
Raising the difficulty only matters for the (imo) less important part: the dick measuring competition between the very top teams.
The actual point of CTFs was usually to keep your skills sharp and stay learning. Eventually you build your own challenges, thereby completing the "have it taught to me, then do it myself, then teach another person" three step process towards mastering concepts.
You can just say "let the people who want to learn from it do so" but honestly the entire culture of learning in the US at least is DEAD. We turned "education" into a rote system of maximizing incentives to the extent that that's all the youth know it as, and (increasingly) all educators can do. It's just gone without some kind of major reckoning, and we all know things will just collapse before that happens. The ball is in the court of whatever country can learn how to force its youth to learn the real way and use AI productively only AFTER learning the concepts it's being used to accelerate.
This will basically become true for everything.
I never got super into security but it gave me the confidence to play in the same field and lose the stupid aura I had that somehow "rich americans" would be better than me at everything because they had better universities or because of Hollywood or something.
Sad that another cool thing is lost to AI but I guess kids will learn in other ways.
As I don't know much about the CTF scene, I looked for other takes on this topic.
Here's an article from 2015 about how tool-assistance already changed CTFs:
> Individual skill will undoubtedly be a factor next year. But, I'm left wondering whether next year's DEFCON CTF will tell us anything more than how well-developed each team's tools are (and how well they can interpret the results).
https://fuzyll.com/2015/ctf-is-dead-long-live-ctf/
But there are quite a few recent (2026) articles with the same core message as in the original article, e.g., https://blog.includesecurity.com/2026/04/ctfs-in-the-ai-era/ or https://k3ng.xyz/blog/ctf-is-dead
And here's someone explaining how Claude Max allowed them to win CTFs:
> I had always been interested in CTF as one of the only ways people could compete and show off their skill in coding/problem solving on a global scale. It was just too difficult and didn't make sense for me to learn the fundamentals as an electrical engineer. As time went on, I got better and better, and it was hard to tell whether it was because of experience or if it was because of improvements in AI.
> I accomplished my goals, and for that reason I'm quitting CTF, at least for now. [...] I'd like to think I highlighted the problem before it became a bigger issue. So, how do we fix this? Teams and challenge authors losing motivation is not good. CTF dying is not good. AI bad. Or is it?
https://blog.krauq.com/post/ctf-is-dying-because-of-ai
The only article that saw LLMs as a non-negative force for CTFs was this one. Fittingly, it sounds like LLM output ("Let's be honest", "This is where things get interesting.") and only contains hallucinated references.
https://caverav.cl/posts/ctfs-not-dead/ctfs-not-dead/
https://ldjam.com/events/ludum-dare/59/setidream/about-ai-ar...
For what it's worth, the non-AI-coded entries were still quite good relative to the winners, so it's not so obvious that AI use confers an unbeatable advantage.
The matrix was always the same and the challenge was clearly designed so that the point was being able to read anything at all, not knowing how to invert a matrix, so I asked the creator what was up.
He told me that there were tools that would trace input values until they reached a comparison instruction, then print what they were compared against. Therefore it was necessary for every deobfuscation challenge to scramble the input in some way too complex for these tools to undo, before comparing it. Hence the multiplication by a pseudorandom matrix.
The point is, cheating tools aren't new.
So the obvious solution is to fully ban AI and AI generated tools? To destroy your own hobby just because AI can semantically be considered a tool, seems very stupid to me. If the point of these CTFs is to practice and measure your skill, what becomes the point of the competition once everyone uses AI?
So, I would not start banning the tools. They always been there. We just need to fine tune the challenges where bar goes beyond things and you really need to use AI as tool. Maybe the definition of insane starts to be custom kernel fork with planted bugs and you need to use AI to find the bugs, and use some exploit chains against that kernel with specific web server. That is the real world right now, I guess.
Many CTFs have switched to a dual-leaderboard format recently, one for "agentic teams," one for the rest. If all you care about is "learning" and imaginary internet points, you can just participate as a human team and adblock the AI scoreboard, and maybe lobby CTFTime into splitting their rankings as well.
I don't mean that everyone must know what CTF is, but sometimes it's OK to write things just for your community (CTF community in this case), not for general population.
1) You see a headline on HN about some open format being broken by frontier-level AI. You don't recognise the acronym.
2) You visit the site, you read the first few paragraphs, you still have no effing clue what the site is talking about
3) You come back to HN and read the comments to figure out WTF is going on. Oh, it's just some game style, so not a Cryptographic Trust File, or something you have to care about after all.
The point is you can't know what some opaque acronym is about until you visit the source of something that will hopefully explain this opaque acronym. Leaving the site being still none-the-wiser is a failure of the site, in my view. If you don't agree, that's fine, we're adults, we can differ, but it seems like a valid complaint to me.
FWIW (this means "for what it's worth" :) I'm not railing against acronyms in general, and HTTP is probably one of the most-used ones on the internet so I'm not sure it really applies as a good counterpoint. Using CTF without an explanation is more like using SSTP (Secure socket tunneling protocol) without one, IMHO (this means In My Humble Opinion :) ...
It's pretty fun. Or at least it was, back when you had some sense that your competitors were competing on an even playing field and just beat you because they were better than you.
I wouldn't say the name is a "gaming reference", it's just a descriptive name for a game.
Its a war game reference I guess?
https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit...
Another idea is deep red herrings. solves that lead to more solves, on and on, except only if the previous solves were solved quickly. The effect will be that participants who solve things quickly will keep finding things to solve. they can't know that the path they're on will lead to victory, even if they artificially slow down, unless they consistently slow down just as a human would. It will eliminate the speed advantage. For the skill advantage, other than having another LLM procedurally generate challenges, I don't know of a good solution.
There are always things like captchas. or the good 'ol honor system. A person can spend only so much for things that have no financial reward in the end, only clout.
---
Alright, all that said, i think i really do have a good solution for this, as well as academic exams. Or I think I do, because it's so simple, I've been scratching my head as to why everyone isn't doing it already.
Require screen sharing/recording. LLMs can't fake that well enough. Have another LLM audit the video for mouse, key stroke, window movement and other details to see if it looks human-generated or not.
If a student has an essay assignment, have them record their screen as they research, and actually type out the whole thing. In the extreme, require anti-cheat proctoring software installed, as is done in remote examination. In an even more high-stakes and extreme scenario, have them share their face. Their eye and face movement, correlated with the screen-share, and correlated with the activity observed on the server end, should be pretty hard to beat, even in the next ~5 years of LLM advances.
Well actually I get it. In cycling motor doping, putting a hidden engine into the bike, seems more offensive than regular doping. I think this is because there is a continuum from eating well to taking supplements to injecting stuff, but having a engine breaks a fundamental idea about cycling. Similar hacking is about cleverly abusing the rules.
Asking AI for learning / explaining purposes is absolutely fine in my book - but I have absolutely no motivation to set up AI to solve challenges. Without AI you can't really compete successfully. So AI is really taking motivation away for me which in turn again prevents me from learning more. I am not sure that this is solvable.
This is like someone complaining that making machine parts has been ruined: Skillful craftsmen used to make them by hand using manual tools!
Nowadays the CAD/CAM/CNC cheaters have almost completely automated the whole thing. How is the next generation of craftsmen going to learn how to craft a gear by hand when the process of gear making has been reduced to pressing start on a CNC machine?!
See what I mean? Sorry, I think this article is just Luddite. I can empathize with the pain of your beloved craft basically being rendered obsolete by new technology, but the process can neither be stopped nor is it bad in general.
The manual skills you trained with CTF puzzles are now simply no longer relevant . (Field-specific) "AI orchestration" is the new cyber securtiy skill if LLMs really have become so good at this, and what the author used to do manually then has the same value as being able to craft a gear by hand.
Within the framework of your analogy, it's like responding to someone active in DIY maker groups suddenly dealing with an influx of influencers in meetups showing off Chinese junk from Etsy to post on Tiktok, and accusing them of being a Luddite blinded by their zealous hatred of mass production -- both strangely abrasive and also fairly nonsensical except as a "mass production supporter" social signifier.
Not to mention, in the article they specifically describe themselves as a heavy user of frontier models for security research ever since the release of Opus 4.5, calling them "useful within the field". In fact I don't see any actual criticism of AI/LLMs anywhere whether for security research, programming or anything else, except for making competitive CTFs no longer viable.
What does it take to avoid the "Luddite" brand? Using AI themselves and praising AI as useful (to the point of having a lopsided advantage over humans) isn't enough? Do they also need to say "I haven't written a line of code in 6 months/it's easily a 100x multiplier for my job" every time they mention it too?
Indeed, in the real world, plenty of people organize to do formerly-skillful tasks together. I have not personally crafted a gear by hand, but I have built a house in a long-abandoned style with a group of people only using hand tools.
There _is_ a danger that society forgets how to do these things. During that house-building exercise, there were many tricks of the trade that, while likely documented somewhere in a book, would have been difficult to reproduce without seeing a demonstration. From the standpoint of “does it matter?” it depends on what you care about. We absolutely do not need cruck-framed houses with scribed joints. Modern construction is faster and cheaper and lasts long enough. But it would sadden me greatly if practices like this faded from memory, because it’s one of those things that makes you gasp “wow!” when you see it. And your appreciation only deepens when you try it yourself.
The reason LLMs can do CTFs so well is partially because the challenges are usually designed to avoid wasting time and to introduce a single concept without noise.
These models seems completely unbeatable only in the ads. There are 100+ times way someone puts Hindi Yoda talk In Morse Code and it goes nuts. The reason they are going to hard for PR Marketing on this is because they know it is a matter of time.
The only things that works is novelty and obscurity. LLMs still suck with things mentioned in the footnotes of datasheets and manuals, things that deviate in subtle ways, unique constructions that alter something very very common. It's hard for LLMs to avoid common pitfalls in terms of making assumptions, while staying on track.
Explicit ELO measurements with some cheating detection. AI assistance wholly banned. As you climb the ELO ladder, detection gets more onerous. At top level during online events, anti cheating teams require the use of both monitoring software and multiple cameras.
Idea is that you can cheat pretty easily at the lowest levels but it gets less easy the higher you go. This allows for better feeding into the truly elite competitions.
I think chess’s very firm stance that AI is never allowed in competition (neither online nor in person), rather than CTF’s acceptance, was the right call.
If CTF is a player-vs-player event, then AI should just be banned outright, otherwise it will devolve into AI-vs-AI, which is just not an interesting competition format, as we learned in chess. Compared to FIDE top events (which bans AI), only a tiny niche audience actually watches the Top Chess Engine Championship (AI-centered). It turns out what we care about is not whether chess can be solved by any means available, but what are the limits of the human mind in learning chess.
Pretty much all chess coaches/educators also warn against relying heavily on AI during learning; engines only give you an illusion of understanding.
In our own trainings we give (AI agents for security, and a graph masterclass), we ended up leaning into it. For example, we ship with a skills bundle. There are plus sides, like less code-forward participants can go further and are appreciating that, and less of a gap between high-level concepts and successful hands-on. But at the same time, manual work does build a lot of intuition & knowledge that gets missed in auto modes.
botsbench.com shows Sonnet 4.5+ with Claude Code harness does pretty well, and Sonnet roughly tracks the edge of what self-hosted models do on the upper tier of affordable GPUs, like running 1-2 DGX Sparks and waiting 6mo for oss to catch up a bit
Not as easy logistically...
The same article talks about CTF skills as a way to learn about security best practices and separately a sport.
In reality it was all about learning an extremely important skillset (securing/attacking software and systems) that is getting automated.
The real thing the author seems to be frustrated about is AGI is coming in computationally verifiable domains first, and lot of his skillset was taken over in a big part.
>and the old game is not coming back
For many people the CTF scene was already dead in 2021 because it had turned into something unrecognisable.
In reality it’s just different.
"That makes open CTFs pay-to-win. The more tokens you can throw at a competition, the faster you can burn down the board. Specialised cybersecurity models like alias1 by Alias Robotics are becoming less relevant compared to general frontier LLMs. The competition is turning into "who can afford to run enough agents, with enough context, for long enough.""
1) It’s OK to do just about anything to win a CTF, including installing malware on the organisers computers months before the actual event so you’ll have an easy time stealing the flags.
2) It’s not ok to try and win the CTF with a solution the authors did not intend.
Recently the #2 crowd has been winning because the hacking scene has turned corporate and boring. People started to partake in CTFs in the hopes of landing a job(!)
CTFs are indeed ruined for those people, I personally don’t mind.
For the people in group #1 LLMs change little. Attacking the challenges directly was always a last resort.
The fact that CTFs became a sort of SAT score for getting a security job made me lose interest very early on.
Hits different doesn't it
Imagine every competitive chess player being allowed to video call with a hundred other people to help them make a move. CTF have never been fair, nor has it ever been effectively structured for learning.
The intent for most CTFs is to provide a meaningful challenge that concerns a single topic without introducing noise that wastes time. Of course a training exercise is easier to complete for an LLM.
Why so pedantic?
I've seen that exact font and color scheme a dozen of times the past weeks.
On the other hand, CTFs are fundamentally a game and a competition which are supposed to be fun and compare and improve ones skill. So when I let an LLM generate the entire solution for me, what's the point anymore? I did not learn anything. I did not work for that place on the leaderboard, I just copied the solution. And worst of all, I did not have any fun. It's boring.
So how does using AI as a solver not feel like cheating?
Most phishing comes from a small set of hosting providers that ignore reports. The real fix is making abuse reports actionable. Fix that, and you cripple the economics of phishing. AI detection is reactive; killing the source is proactive.
What am I missing here?
Its not really a good comparison
But I don't know enough that's why I asked.
I imagine one could do CTF in public, machines you work on vetted/prepared to some spec, yada yada.
If chess and Go can do it why can't CTF?
That was my question when I wrote "what am I missing here".
"Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that"."
I guess this goes in parallel with the whole building for one narrative
And while I have ideas to excite and promote LLM use in these style games I've still not been able to crack the human collaboration component that is at the forefront of all of this change
> Rules that ask people not to use LLMs are ignored and almost impossible to enforce in open online events.
It's quite sad to see CTFs dying. I never had the time do seriously participate in CTFs, but I always respected those who did, as well as the people organizing these events.
That's such a non-reason. If your competition cannot enforce the rules of the competition, then what's the point? Does the CTFs specifically need to be 'open'?
The author argues that open CTFs are done for because of rampant cheating. You're agreeing with that, don't you?
The title is "AI has broken the open CTF format". If the format is "open CTF" then it is very specifically open.
As to your second question: Yeah, I believe having open CTFs was a good idea.
It's an incredibly exciting time in security research in my humble old man opinion.
Think the cadence of new exploits is perhaps a good measure of that rather than subjective thoughts by anyone regardless of experience.
The text itself being exceedingly long for no obvious reason doesn’t help.
And if you think it was too long, what part would you have shortened? I never knew about the scene and found it interesting to read this personal take on it.
According to Pikka, the paragraph text is Taupe Grey (#92908a) on a Liquorice (#111110) background. That's... pretty far from black and white.
They may as well be the human equivalent to what LLMs currently are.
I do not mourn these people, as they’re usually the most arrogant types. I hope for their sake they adapt.
This year, multiple groups on the top of the leaderboard were clearly abusing LLMs. You can tell because they know nothing of what a CTF is nor the terminology, nor really the fields the challenges were about when they were talked to. They were obviously amateurs.
It was pretty depressing to hear how unaware they were of how obviously they did not fit in to the type that usually is on the top of the leaderboard. It seems they seriously think they were under the radar. If it was one group it could be a freak incident - some times someone just shows up and curbstomps competition. But there were many groups like this this year. They also had a certain smugness to it - one staff reported that a group was hinting to other teams about their "super weapon". Another group credited their "secret third team member they didn't want to talk about".
I use LLM frequently and experiment with it a lot, both at work and on my free time. Nowadays they are good enough to have value and I am interested in learning more about that. They let me spend more time on hard problems and avoid spending the day on simple CRUD. I say this to say that LLM doesnt have to equal bad, it is a tool, that's all. However, I generally avoid LLM communities because many LLM fans are lazy and unskilled people who are just happy they can feel they are worth something even if they have no skill. They don't really have much to provide of conversation. If anything, from reading the CTF crowd this year, the rise of LLMs has just meant more of these people can stomp on and harvest the CTF scene for self validation.
This is not me trying to gatekeep who can play CTF. Anyone is welcome, but there is one condition: You are here to learn and have fun.
The conclusion many I talk to has come to is that nowadays, it is harder to learn to put in hard work and become good at something because there are just too many ways to cheat and take shortcuts. I suspect in the future there will be a shortage of useful people - the kind that have critical thought and know the value of doing something properly. This doesn't mean "Not using LLM", but as said by many on HN before you need a certain seniority before LLMs are useful augmentations to your skills and not just stopping you from learning yourself.
I agree with the article. Anything but physical competitions with strong security - think professional e-sports with organizer-provided PCs, is over. But I think one of the most interesting things to take away from my CTF experience is that the bottom of the leaderboard was still full of amateurs slowly working their way up - it is a few rotten apples that ruin the fun for most, and there are still plenty of people who want to learn and deep-dive.
The whole point of competitions is to provide a safe environment thanks to a set of rules all participants AGREE on in order to progress together.
If new tools "break" the competition, we change the rules and that's A-OK.
CTF isn't a natural phenomenon, if tools change, rules change, simple.
- no computer assistance, which does also mean no mobile on competition, human only
- advanced chess with assistance
- computer only, no human assistance
and arguably chess itself is not doing worst since.
Amateur online chess on the other hand is besieged by cheaters that use engines, even in casual non-ranked games where there's absolutely nothing to gain besides a pat on the ego. This has drastically changed how the game is played today with lot of players gravitating towards speed chess (bullet and blitz) to compensate. That will thin the herd of cheaters but one still runs into engines on a weekly basis.
This is also the tip of the iceberg, with the true scale of the problem being orders of magnitude worse, as someone dedicated enough can use an engine to cheat in a way that's essentially undetectable.
>The issue was never that AI could help. proceeds to write the next 3 sentences about how the problem IS in fact ai help
>Teams that refused to use AI were not just missing a convenience; they were playing a slower version of the competition.
>CTFs were not just a set of puzzles. They were a ladder.
>The claim is not that every challenge is solved. The claim is that...
>The loss is not just a scoreboard. It is the ladder from
Guys I'm so sorry I just can't stop noticing stuff like this. Anyone else?