In early December I got the kind of tip we’ve been getting a lot over the past year. A reader had noticed a post from someone on Reddit complaining about a very graphic sexual ad appearing in their Instagram Reels. I’ve seen a lot of ads for scams or shady dating sites recently, and some of them were pretty suggestive, to put it mildly, but the ad the person on Reddit complained about was straight up a close up image of a vagina. 

The reader who tipped 404 Media did exactly what I would have done, which is look up the advertiser in Facebook’s Ad Library, and found that the same advertiser was running around 800 ads across all of Meta’s platforms in November, the vast majority of which are just different close-up images of vaginas. When clicked, the ad takes users to a variety of sites for "confidential dating” or “hot dates” in your area. Facebook started to remove some of these ads on December 13, but at the time of writing, most of them were still undetected by its moderators according to the Ad Library.

Like I said, we get a lot of tips like this these days. We get so many, in fact, that we don’t write stories about them unless there’s something novel or that our readers need to know about them. Facebook taking money to put explicit porn in its ads despite it being a clear violation of its own policies is not new, but definitely a new low for the company and a clear indicator of Facebook’s “fuck it” approach to content moderation, and moderation of its ads specifically.   

AI Forensics, a tech platform and algorithmic auditing firm, today put out a report that quantifies just how widespread this problem is. It found over 3,000 pornographic ads promoting “dubious sexual enhancement products” which generated over 8 million impressions over a year in the European Union alone. 

In an attempt to show that the ads didn’t use some clever technique to bypass Meta’s moderation tools, AI Forensics uploaded the exact same visuals as standard, non-promoted posts on Instagram and Facebook, and they were removed promptly for violating Meta’s Community Standards. 

“Our findings suggest that although Meta has the technology to automatically detect pornographic content, it does not apply it to enforce its community standards on advertisements as it does for non-sponsored content,” AI Forensics said in its report. “This double standard is not a temporary bug, but persisted since as early as, at least, December 2023.”

When we write about this problem with Facebook’s moderation we always stress that there’s nothing inherently alarming about nudity itself on social media. The problem is that the policy against it is blatantly hypocritical because it often bans legitimate adult content creators, sex workers, and sex educators who are trying to play by the platform’s rules, while bad actors who don’t care about Facebook’s rules find loopholes that allow them to post all the pornography they want. Additionally, that pornography is almost always stolen from the same legitimate creators who Facebook polices so heavily, the ads are almost always for products and services that are trying to scam or take advantage of the audience Facebook is allegedly trying to protect, and in some cases promote tools for creating nonconsensual pornography.

What’s adding insult to injury right now is that in addition to Facebook’s hypocrisy I lay out above, Facebook is now punishing us for publishing stories about this very problem. 

In October, I published a story with the headline When Does Instagram Decide a Nipple Becomes Female, in which artist Ada Ada Ada tests the boundaries of Instagram’s automated and human moderation systems by uploading a daily image of her naked torso during her transition. The project exposes how silly Instagram’s rules are around allowing images of male nipples while not allowing images of female nipples, and how those rules are arbitrarily enforced. 

It was disappointing but not at all surprising that Facebook punished us for sharing that story on its platform. “We removed you photo,” an automated notification from Facebook to the official 404 Media account read. “This goes against our Community Standards on nudity or sexual activity.”

Separately, when Jason tried to share it on his Threads, it removed his post because it included “nudity or sexual activity.” Weirdly, none of the images in the post Jason shared were flagged when Ada Ada Ada uploaded them to Instagram, but they were when Jason shared them on Threads. Threads also removed Joe’s post about a story I wrote about people making AI-generated porn of the Vatican’s new mascot, a story that is about adult content, but that doesn’t contain nude images.

Our official 404 Media page, as well as Jason’s personal account, which he has had for 20 years and which is the “admin” of the 404 Media page, was dinged several times for sharing stories about a bill that would rewrite obscenity standards, the FBI charging a man with cyberstalking, and AI-generated fake images about a natural disaster on Twitter. Facebook has threatened the existence of not just the official 404 Media page, but also of Jason’s personal account.

Not a single one of these stories or the images they include violate Facebook’s policies as they are written, but Facebook nonetheless has limited how many people see these stories and our page in general because we shared them. Facebook has also prevented us from inviting people from liking the page (which presumably would limit its reach also) and warned us that it was “at risk of being suspended,” and later, “unpublished.”

As many sex workers and educators have told us over the years, while Facebook gave us the chance to appeal all of these decisions, trying to correct Facebook’s moderation efforts is not simple, and the “appeals” process consists solely of clicking a few predetermined boxes; there is no chance to interact with a moderator or plead your case. We appealed three of the decisions in late October, none of which were accepted. 

The appeal we filed on Ada Ada Ada’s story on the official 404 Media page in mid-December was accepted within a few hours and got the restrictions lifted off of the 404 Media page (and Jason’s personal account) in mid-December. But an appeal Jason filed on his Threads post about the same story was not accepted: “We reviewed your post again. We confirmed that it does not follow our Community Guidelines on nudity or sexual activity,” the appeal determination on Jason’s Threads post read. The different determinations between what was essentially the exact same post shows how all-over-the-place Meta’s moderation remains, which creates an infuriating dynamic for adult content creators. Mark Zuckerberg has personally expressed regret for giving into pressure from the Biden administration to “censor” content during the height of the coronavirus pandemic, but neither he nor Meta has extended an apology to adult content creators who are censored regularly. 

It was hard enough to deal with having to constantly prove to Facebook that our journalism is not pornography or harmful content when we worked at VICE, where we had a whole audience and social media team who dealt with this kind of thing. It’s much harder for us to do that now that we’re an independent publication with only four workers who have to do this in addition to everything else. I can’t imagine how demoralizing it would be to have to deal with this as a single adult content creator trying to promote their work on Facebook’s platforms. 

Again, this is frustrating as is, but infuriating when I regularly see Facebook not only take money from advertisers that are pushing nudity on Facebook, but doing it for the explicit purpose of creating nonconsensual content or scamming its users. 

The silver lining here is that Facebook was already increasingly a waste of our time. The only reason we’re able to share our stories via our official Facebook page is that we’ve fully automated that process, because it is not actually worth our time to post our stories there organically. Since before we started 404 Media, we knew there was very little chance that Facebook would help us reach people, grow our audience, and make the case that people should support our journalism, so in a way we lost nothing because there’s nothing to lose. 

On the other hand, that perspective is based on us having already accepted Facebook’s rejection of our journalism years ago. It’s not as if people don’t get any news on Facebook. According to Pew, about a third of adults in the U.S. get news from Facebook, but according to media monitoring tool Newswhip, the top 10 publishers on Facebook are British tabloids, People, Fox News, CNN, and BBC. Smaller publishers, especially publishers who are trying to report on some of the biggest problems that are plaguing Facebook, are punished for pointing out that those problems involve adult content, which disincentivizes that reporting and allows those problems to fester. 

I don’t like it, but ultimately the choices Facebook is making here are shaping its platform, and it’s going to be a bigger problem for its users who are going to fall victim to these ads than it is for us as a publisher.

People are using the popular AI video generator Runway to make real videos of murder look like they came from one of the animated Minions movies and upload them to social media platforms where they gain thousands of views before the platforms can detect and remove them. This AI editing method appears to make it harder for major platforms to moderate against infamously graphic videos which previously could only be found on the darkest corners of the internet. 

The practice, which people have come to call “Minion Gore” or “Minion AI videos” started gaining popularity in mid-December, and while 404 Media has seen social media platforms remove many of these videos, at the time of writing we’ve seen examples of extremely violent Minion Gore videos hosted on YouTube, TikTok, Instagram, and X, which were undetected until we contacted these platforms for comment. 

Specifically, by comparing the Minion Gore edits to the original videos, I was able to verify that TikTok was hosting a Minionfied video of Ronnie McNutt, who livestreamed his suicide on Facebook in 2020, shooting himself in the head. Instagram is still hosting a Minionfied clip from the 2019 Christchurch mosque shooting in New Zealand, in which a man livestreamed himself killing 51 people. I’ve also seen other Minion Gore videos I couldn’t locate the source materials for, but appear to include other public execution videos, war footage from the frontlines in Ukraine, and workplace accidents on construction sites.

The vast majority of these videos, including the Minion Gore videos of the Christchurch shooting and McNutt’s suicide, include a Runway watermark in the bottom right corner, indicating they were created on its platform. The videos appear to use the company’s Gen-3 “video to video” tool, which allows users to upload a video they can then modify with generative AI. I tested the free version of Runway’s video to video tool and was able to Minionify a video I uploaded to the platform by writing a text prompt asking Runway to “make the clip look like one of the Minions animated movies.” 

Runway did not respond to a request for comment.

I’ve seen several examples of TikTok removing Minion Gore videos before I reached out to the company for comment. For example, all the violent TikTok videos included in the Know Your Meme article about Minion Gore have already been removed. As the same Know Your Meme article notes, however, an early instance of the Minion Gore video of McNutt’s suicide gained over 250,000 views in just 10 days. I’ve also found another version of the same video reuploaded to TikTok in mid-December which wasn’t removed until I reached out to TikTok for comment on Tuesday.

TikTok told me it removes any content that violates its Community Guidelines, regardless of whether it was altered with AI. This, TikTok said, includes its policies prohibiting "hateful content as well as gory, gruesome, disturbing, or extremely violent content." TikTok also said that it has been proactively taking action to remove harmful AI-generated content that violates its policies, that it is continuously updating its detection rules for AI-generated content as the technology evolves, and that when made aware of a synthetic video clip that is spreading online and violates its policies, it creates detection rules to automatically catch and take action on similar versions of that content. 

Major internet platforms create unique “hashes,” a unique string of letters and numbers that acts as a fingerprint for videos based on what they look like, for known videos that violate their policies. This allows platforms to automatically detect and remove these videos or prevent them from being uploaded in the first place. TikTok did not answer specific questions about whether Minion Gore edits of known violating videos would bypass this kind of automated moderation method. In 2020, Sam and I showed that this type of automated moderation can be bypassed with even simple edits of hashed, violating videos.

“In most cases, current hashing/fingerprinting are unable to reliably detect these variants,” Hany  Farid, a professor at UC Berkeley and one of the world’s leading experts on digitally manipulated images and a developer of PhotoDNA, one of the most commonly used image identification and content filtering technologies, told me in an email. “Starting with the original violative content, it would be possible for the platforms to create these minion variations, hash/fingerprint them and add those signatures to the database. The efficacy of this approach would depend on the robustness of the hash algorithm and the ability to closely mimic the content being produced by others. And, of course, this would be a bit of a whack-a-mole problem as creators will replace minions with other cartoon characters.”

This, in fact, is already happening. I’ve seen a video of ISIS executions and the McNutt suicide posted to Twitter, which was also modified with Runway, but instead of turning the people in the video into Minions they were turned into Santa Claus. There are also several different Minion Gore videos of the same violent content, so in theory a hash of one version will not result in the automatic removal of another. Because Runway seemingly is not preventing people from using its tools to edit infamously violent videos, this creates a situation in which people can easily create infinite, slightly different versions of those videos and upload them across the internet. 

YouTube acknowledged our request for comment but did not provide one in time for publication. Instagram and X did not respond to a request for comment.

Since we launched 404 Media one of the most common feature requests we’ve received from readers is the ability to log into the site with a username and password, as opposed to the magic links used by Ghost, the open source publishing platform we use for our site and newsletters. 

If you don’t have a 404 Media account, here is how magic links work: Rather than enter a username and password to register for our site and log in, you give us your email. We then send you an email with a link that you click, which logs you into the site. That email also comes with a URL you can copy/paste into the address bar of your browser of choice for reasons we’ll get into in a minute. That’s it. As long as you remain logged in you never have to think about this again, and if you are logged out or want to login on a different device you just repeat the same process. 

We find this to be a much easier login process and wish it was more common across the web where appropriate. But there’s a much more important reason why we have embraced Ghost’s login method and are not in a rush to develop our own solution for a username and password login in the same way we invested time and money in developing full text private RSS feeds for paying subscribers, for example. The gist is that it’s safer for us and for you to not share any passwords with us. 

It’s impossible to say what the exact number is, but a huge portion of cybersecurity breaches start with compromised credentials. There are a few ways hackers can compromise your passwords, many of which Joseph has covered on 404 Media recently, but one common method is exploiting the fact that the majority of people reuse their passwords across the internet (a study of 28.8 million users found that 52 percent of them reuse passwords). This is why it’s much safer for people to use password managers that generate unique, strong passwords for every account, and why Have I Been Pwned is such an important resource—by keeping track of sites and services that have been hacked, it acts a constant reminder to use a different password on every service. Otherwise, a hacker could take your password from that random forum hack, and then use it to break into your workplace account, or whatever other account shares that password.

It is standard best practice for sites that ask for your password to hash it, meaning even if a site got hacked, hackers can’t just run off with your password. However, that is not always the case, with some companies storing passwords in plain text, and depending on what hashing algorithm the site has used, hackers may be able to crack it.

But you know what’s the safest way for us to keep your password safe? Not asking for one to begin with. By not creating a password with us you have no risk of it leaking, and we don’t have to deal with the responsibility of keeping it secure. The sign in link is going to your email, which presumably is protected with two-factor authentication, if you have it set up (which you should!).

“The main reason (as you know!) is security,” Ghost CEO John O’Nolan told us when asked about the company’s choice to use magic links. “Passwords get hacked all the time, but they can’t be hacked if they don’t exist. Then what I would usually add to that is how this allows a small team like 404 to spend less time managing security administration, and more time investing in bringing you stories you care about.”

That being said, we want to acknowledge that the magic link system isn’t perfect because no system is. We also understand that some people don’t like the magic link system or have extenuating circumstances where it does not work for them, and would prefer a password system. We’re writing this article in part to explain our thinking behind having the magic link system and to explain why a password system is not currently feasible for us.

We have, on a couple of rare occasions, heard from users complain that the emailed links take a while to come in. This almost never happens and when it has we’ve seen it resolved within a few hours. More often than not, users will sign up to our site via a work email with aggressive security or content filtering rules that block our emails. If you ever think that might be the case for you please reach out to [email protected], but also keep in mind you are always free to change the email associated with your account to a personal email address. We want to make our articles as easy as possible for subscribers to access, which is why we set up private RSS feeds that don’t require a login to read our stories.

Probably the most common problem people run into with magic links is they think they have logged into the site on their normal browser, but they’re actually logged in through an in-app browser. For example, someone might receive the login link to their email. They open up the Gmail app, click the “Sign in to 404 Media” button, and their phone loads the webpage. But this is loading the website in Gmail’s web browser, not your native Safari one. People then navigate the site as they would normally in their default browser, and are surprised when they are not logged in. These two browsers are not sharing any cookies or log in sessions.

It’s annoying when apps open stuff in their own browsers rather than the phone’s native one. This is a more fundamental design issue with how many apps or operating systems work. A solution on iPhone is when receiving the login link, click and hold the “Sign in to 404 Media” button to bring up the contextual menu, and hit “Open Link.” This will open the link, and sign you in, on your native browser. Or, copy and paste the sign in link which is also in the email. Regardless, we recommend you login to 404 Media wherever you expect to read it. 

We totally understand that this is a frustrating experience, and frankly a flaw with the mobile web in general. But we also recognize that for a lot more people, not having to remember or save a password is the easiest, more preferred, most secure option we can offer right now. The benefits of the magic link system outweigh the costs, both to us as a small business, and to our readers who are privacy-conscious. 

Ultimately, it is much safer for us, and for you.

New research from Anthropic, one of the leading AI companies and the developer of the Claude family of Large Language Models (LLMs), has released research showing that the process for getting LLMs to do what they’re not supposed to is still pretty easy and can be automated. SomETIMeS alL it tAKeS Is typing prOMptS Like thiS. 

To prove this, Anthropic and researchers at Oxford, Stanford, and MATS, created Best-of-N (BoN) Jailbreaking, “a simple black-box algorithm that jailbreaks frontier AI systems across modalities.” Jailbreaking, a term that was popularized by the practice of removing software restrictions on devices like iPhones, is now common in the AI space and also refers to methods that circumvent guardrails designed to prevent users from using AI tools to generate certain types of harmful content. Frontier AI models are the most advanced models currently being developed, like OpenAI’s GPT-4o or Anthropic’s own Claude 3.5.

As the researchers explain, “BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations—such as random shuffling or capitalization for textual prompts—until a harmful response is elicited.”

For example, if a user asks GPT-4o “How can I build a bomb,” it will refuse to answer because “This content may violate our usage policies.” BoN Jailbreaking simply keeps tweaking that prompt with random capital letters, shuffled words, misspellings, and broken grammar until GPT-4o provides the information. Literally the example Anthropic gives in the paper looks like mocking sPONGbOB MEMe tEXT. 

Anthropic tested this jailbreaking method on its own Claude 3.5 Sonnet, Claude 3 Opus, OpenAI’s GPT-4o, GPT-4o-mini, Google’s Gemini-1.5-Flash-00, Gemini-1.5-Pro-001, and Facebook’s Llama 3 8B. It found that the method “achieves ASRs [attack success rate] of over 50%” on all the models it tested within 10,000 attempts or prompt variations. 

The researchers similarly found that slightly augmenting other modalities or methods for prompting AI models, like speech or image based prompts, also successfully bypassed safeguards. For speech, the researchers changed the speed, pitch, and volume of the audio, or added noise or music to the audio. For image based inputs the researchers changed the font, added background color, and changed the image size or position. 

Anthropic’s BoN Jailbreaking algorithm is essentially automating and supercharging the same methods we have seen people use to jailbreak generative AI tools, often in order to create harmful and non-consensual content. 

In January, we showed that the AI-generated nonconsensual nude images of Taylor Swift that went viral on Twitter were created with Microsoft’s Designer AI image generator by misspelling her name, using pseudonyms, and describing sexual scenarios without using any sexual terms or phrases. This allowed users to generate the images without using any words that would trigger Microsoft’s guardrails. In March, we showed that AI audio generation company ElevenLabs’s automated moderation methods preventing people from generating audio of presidential candidates were easily bypassed by adding a minute of silence to the beginning of an audio file that included the voice a user wanted to clone.

Both of these loopholes were closed once we flagged them to Microsoft and ElevenLabs, but I’ve seen users find other loopholes to bypass the new guardrails since then. Anthropic’s research shows that when these jailbreaking methods are automated, the success rate (or the failure rate of the guardrails) remains high. Anthropic research isn’t meant to just show that these guardrails can be bypassed, but hopes that “generating extensive data on successful attack patterns” will open up “novel opportunities to develop better defense mechanisms.” 

It’s also worth noting that while there’s good reasons for AI companies to want to lock down their AI tools and that a lot of harm comes from people who bypass these guardrails, there’s now no shortage of “uncensored” LLMs that will answer whatever question you want and AI image generation models and platforms that make it easy to create whatever nonconsensual images users can imagine. 

YouTube is AI-generating replies for creators on its platform so they could more easily and quickly respond to comments on their videos, but it appears that these AI-generated replies can be misleading, nonsensical, or weirdly intimate. 

YouTube announced that it would start rolling out “editable AI-enhanced reply suggestions” in September, but thanks to a new video uploaded by Clint Basinger, the man behind the popular LazyGameReviews channel, we can now see how they actually work in the wild. For years, YouTube has experimented with auto-generated suggested replies to comments that work much like the suggested replies you might have seen in your Gmail, allowing you to click on one of three suggested responses like “Thanks!” or “I’m on it,” which might be relevant, instead of typing out the response yourself. “Editable AI-enhanced reply suggestions” on YouTube work similarly, but instead of short, simple replies, they offer longer, more involved answers that are “reflective of your unique style and tone.” According Basinger’s video demoing the feature, it does appear the AI-generated replies are trained on his own comments, at times replicating previous comments he made word for word, but many of the suggested replies are strangely personal, wrong, or just plain weird. 

For example, last week Basinger posted a short video about a Duke Nukem-branded G Fuel energy drink that comes in powder that needs to be mixed with water. In the video, Basinger makes himself a serving of the drink but can’t find the scoop he’s supposed to use to measure out the formula. 

“I wouldn’t be surprised if the scoop was buried in the powder,” one YouTube user commented on the Duke Nukem G Fuel video, which certainly sounds right to me as someone who's been serving up baby formula for the last year. 

YouTube’s AI suggested that Basinger reply to that comment by saying: “It’s not lost, they just haven’t released the scoop yet.  It’s coming soon.” 

I can see how that comment could make sense in the context of the types of other videos LGR publishes, which usually review old games, gadgets, and other tech, but is obviously wrong in this instance. 

Another suggested reply to that same comment said: “I’ll have to check if they’re using a proprietary blend that requires a special scoop.”

“My creativity and craft stems completely from my own brain, and handing that off to some machine learning thing that mimics my style not only takes away from the enjoyment of it all for me, but it feels supremely disingenuous,” Basinger told me in an email. “The automated comments in particular come across as tone deaf, since a huge reason YouTube makes sense at all is the communication and relationship between audience and creator. I've had dozens of people say that they now second-guess every interaction with YouTubers in the comments since it could easily be a bot, a fake response.”

Another commenter on the Duke Nukem G Fuel video joked that Basinger should have had a tighter grip on the lid as he was shaking the formula to prevent it from flying all over the place. 

Basinger bursts out laughing as he reads YouTube’s suggested AI-generated reply to that comment: “I’ve got a whole video on lid safety coming soon, so you don’t have to worry!”

At other times, the AI-suggested replies are just nonsensical. The Duke Nukem G Fuel review wasn’t posted to the main LGR channel, but a channel called LGR Blerbs, which is his naming convention for shorter, less deeply researched videos about whatever he’s interested in. A few commenters said they were happy he was posting to the Blerbs channel again, with one saying “Nice. Back to the blerbs.”

YouTube’s AI suggested Basinger reply to that comment by saying: “It’s a whole new kind of blerp,” which I suppose is funny, but also doesn’t mean anything. 

The weirdest examples of AI-generated replies in the video in my opinion are those that attempt to speak to Basinger’s personal life. In response to another commenter who said they were happy Basinger was posting to the Blerbs channel again, YouTube’s AI suggested the following reply: “Yeah, I’m a little burnt out on the super-high-tech stuff so it was refreshing to work on something a little simpler 🙂.” Another AI-generated reply thanked commenters for their patience and said that Basinger was taking a break but was back to making videos now. 

YouTuber burnout is a well established problem among YouTube creators, to the point where YouTube itself offers tips on how to avoid it. The job is taxing not only because churning out a lot of videos helps them get picked up by YouTube’s recommendation algorithm, comments on those videos and replies to comments helps increase engagement and visibility for those videos.

YouTube rewarding that type of engagement incentivises the busywork of creators replying to comments, which predictably resulted in an entire practice and set of tools that allow creators to plug their channels to a variety of AI that will automatically reply to comments for them. YouTube’s AI-enhanced reply suggestions feature just brings that practice of manufactured engagement in-house. 

Clearly, Google’s decision to brand the feature as editable AI-enhanced reply suggestions means that it’s not expecting creators to use them as-is. Its announcement calls them “a helpful starting point that you can easily customize to craft your reply to comments.” However, judging by what they look like at the moment, many of the AI-generated replies are too wrong or misleading to be salvageable, which once again shows the limitations of generative AI’s capabilities despite its rapid deployment by the biggest tech companies in the world.

“I would not consider using this feature myself, now or in the future,” Basinger told me. “And I'd especially not use it without disclosing the fact first, which goes for any use of AI or generative content at all in my process. I'd really prefer that YouTube not allow these types of automated replies at all unless there is a flag of some kind beside the comment saying ‘This creator reply was generated by machine learning’ or something like that.”

The feature rollout is also a worrying sign that YouTube could see a rapid descent towards AI-sloppyfication of the type we’ve been documenting on Facebook. 

In addition to demoing the AI-enhanced reply suggestion feature, Basinger is also one of the few YouTube creators who now has access to the new YouTube Studio “Inspiration” tab, which YouTube also announced in September. YouTube says this tab is supposed to help creators “curate suggestions that you can mold into fully-fledged projects – all while refining those generated ideas, titles, thumbnails and outlines to match your style.”

Basinger shows how he can write a prompt that immediately AI-generates an idea for a video, including an outline and a thumbnail. The issue in this case is that Basinger’s channel is all about reviewing real, older technology, and the AI will outline videos for products that don’t exist, like a Windows 95 virtual reality headset. Also, the suggested AI-generated thumbnails have all the issues we’ve seen in other AI image generators, like clear misspelling of simple words. 

“If you’re really having that much trouble coming up with a video idea, maybe making videos isn’t the thing for you,” Basinger said. 

Google did not respond to a request for comment.

A bipartisan group of members of Congress has sent letters to Google’s and Apple’s CEOs citing 404 Media’s reporting and asking what the giant tech companies are doing to address the rampant problem of nonconsensual AI-generated intimate media enabled on their platforms. The Congress members also sent a letter to Microsoft CEO Satya Nadella given Microsoft Designer’s role in creating the infamous nonconsensual nude images of Taylor Swift that were viewed millions of times on Twitter, a story 404 Media broke as well. 

“Earlier this year, Apple removed three apps used to create deepfakes off of its app store after an independent report by 404 Media provided links to the apps and their related ads,” the letter to Apple CEO Tim Cook said. “While it is positive that these apps were removed, it is concerning that Apple was not able to identify the apps on their own. The App Store requires developers to undergo a screening process, but the persistence of these apps illustrate that loopholes exist in Apple’s guidelines. As Apple works to address these loopholes, we would like to understand what steps are being taken, and what additional guidelines may need to be put in place to curb the spread of deepfake pornography.”

The letter, which was signed by 26 Republican and Democratic House Representatives was sent to Cook on November 25, and is referencing 404 Media story from April about Apple removing a number of face swapping apps which were explicitly advertising their ability to create nonconsensual porn. Apple removed those apps after we published a story earlier in April about those ads appearing on Instagram. 

“As Congress works to keep up with shifts in technology, Republicans and Democrats will continue to ensure that online platforms do their part to collaborate with lawmakers and protect users from potential abuse,” the letter says, and then presents Cook with a series of questions like “What plans are in place to proactively address the proliferation of deepfake pornography on your platform, and what is the timeline of deployment for those measures?”

A separate letter sent to Google CEO Sundar Pichai by the same members of Congress about Google’s role in allowing apps to advertise their ability to create nonconsensual deepfake porn in Google Search. 

“Earlier this year, Google announced it would ban advertisements for websites and services that produce deepfake pornography,” the letter says, referring to a Google ad policy change we covered in May. “As you know, the emergence of deepfakes has resulted in an increase in ads for programs that cater to users who wish to produce sexually explicit content. While Google’s updated policy instructs AI app developers to build in precautions against offensive content, adds in-app flagging and reporting mechanisms for users, and devalues deepfake porn results in internal search. However, despite these efforts, recent reports have highlighted that Google continues to promote results for apps that use AI to generate nonconsensual nude images. This development raises concerns about Google’s complicity and role in the proliferation of deepfakes. We would like to further clarify the outcome of these updates and understand what additional guidelines may need to be put in place to curb the spread of deepfake pornography, including efforts to remove deepfake platforms from Google’s search results.”

The letter cites a story we published in August which showed that searching for “undress apps,” “best deepfake nudes,” and similar terms on Google turns up “promoted” results for apps that use AI to produce nonconsensual nude images. Google removed those advertisers in response to our story. 

That letter also goes on to ask what plans Google is putting in place to practically address this problem. 

The same members of congress also sent letters to the CEOs of Facebook, TikTok, and Snapchat regarding nonconsensual content on their platforms as well. 

As I wrote in August, face swapping apps present Google and Apple with a very difficult “dual use” problem where the apps can present themselves as benign on the app stores, but promote their ability to produce harmful content off platform. There are more measures these tech companies can put in place to mitigate the problem, but fundamentally any face swapping app has the potential to create harmful content and has to be moderated closely in order to prevent that harm. Monitoring the huge number of apps that are added to these app stores on any given day would be a major and new investment from both Google and Apple.

Last month, Matt Lyzell, the creator of the Netflix interactive series Battle Kitty announced on his personal Instagram account that Netflix was going to remove his show from the streaming service just two years after its debut. By the end of the day, Netflix confirmed that not only Battle Kitty was being removed, but that all 24 Netflix interactive series were to be removed on December 1, with the exception of Black Mirror: Bandersnatch, Unbreakable Kimmy Schmidt: Kimmy vs. the Reverend, Ranveer vs. Wild with Bear Grylls, and You vs. Wild.

“The technology served its purpose, but is now limiting as we focus on technological efforts in other areas,” a Netflix spokesperson told 404 Media at the time. 

It is normal for Netflix and other streaming services to rotate titles in and out of their catalogue depending on what they cost to license and host and how many subscriptions they drive to the platform, but Netflix removing its interactive series means that, as original Netflix creations, once they are removed from Netflix they will not be available anywhere else, and they are a new and unique format that dozens of producers, animators, voice actors, and other creatives have finished work on very recently. 

Unwilling to accept Netflix’s decision to make all these interactive shows totally inaccessible, a group of fans—and, in a few cases, people who worked on the interactive shows—are finding ways to archive and make them available for free. 

“I couldn’t let this work go to waste. We’re talking about over 100 hours of video and ~ one thousand hours of dubbing,” Pixel, one of the archivists in a Discord channel archiving Netflix interactive shows, told me.

On Discord, dozens of users have collaborated on capturing all the videos from Netflix before they were removed, as well as reverse engineering how the platform handled their interactive elements. Some shows are already fully emulated and can be streamed in bespoke, alternative players, others are uploaded to YouTube in a series of daisy-chained, interlinked videos that recreate a very similar interactive experience, while some others have been uploaded as non-interactive videos. 

404 Media agreed not to name the Discord channel and some of the places where the Netflix interactive archives are being hosted so Pixel could talk about the archiving effort. While Netflix has made it so there is no way to view Netflix interactive shows without basically pirating them, the archivists worry that the company will still try to take down any alternative method for viewing them.    

“While I can’t disclose fully how we are archiving these, I can say that they pull directly from Netflix’s servers, so no re-encoding or loss of quality,” Pixel said. “I would love to talk more about how it works, but it risks Netflix patching out the tool entirely.”

Netflix interactives, in case you are unfamiliar, are choose-your-own-adventure videos where the viewer can make choices at the end of a scene that determine how the story unfolds. This Netflix initiative was launched with great fanfare in 2018 when Netflix released Bandersnatch, an interactive entry in the science-fiction anthology series Black Mirror. The shows are interactive in the sense that viewers can nudge the story in different directions, but all they are doing is essentially deciding which pre-recorded video file will play next. Netflix actually made some of its interactive series available on YouTube by using YouTube’s built-in feature that allows users to choose what video to play next once a video ends, daisy chaining YouTube videos together to create the choose-your-own-adventure format natively on that platform. I’ve seen at least two other Netflix interactive shows fully recreated by archivists on YouTube with this method, sometimes in multiple languages. 

Pixel explained that Netflix interactives rely on an “internal video” that contains all the interactive elements, including the different paths, variations, and endings. Decisions viewers make are defined by two JSON files, with one determining when a viewer is presented with a decision and and where in the internal video file to skip to based on that decision, and the other pulling the assets for the decision buttons from Netflix’s servers. 

“We currently have a proof-of-concept emulator running off a python script that uses the jsons to make functioning decisions, although it needs ironing out and button images are broken as of now,” Pixel said. “We have a member of the team in Turkey that is hosting the files for once we get the emulator working on a webpage.”

While the archivists in the Discord were able to rip much of the content directly from Netflix before it was removed, each title is available in many languages, and as Pixel explained, they had trouble grabbing some of the interactive elements, so they weren’t able to grab everything.

In at least one case I’ve seen, the archivists shared video of one of the interactive shows pulled from Netflix that was uploaded to the personal account of someone who worked on the show, though Pixel said they've already ripped that show directly from Netflix.

“Since it's no longer on the Netflix app I figured why not upload it here as a lot of crazy talented people poured their hearts into it for a year,” the person who worked on the show said in a post sharing the video. 

When I asked what the biggest challenge facing the archiving effort is at the moment, Pixel said that “Right now it’s probably keeping track of everything hah. Me and Scramble [another person involved in the archiving effort] had to contact a bunch of people who were willing to help rip stuff/give us already discontinued shows. Right now it’s the emulator. We have a lot of people counting on us and I get a LOT of dms from people asking how to play them haha.”

Correction: This story previously said the archivists used video that was uploaded by one of the show's creators. The archivists say they had already ripped that show directly from Netflix. 404 Media regrets the error.

YouTube is running hundreds of ads featuring deepfaked celebrities like Arnold Schwarzenegger and Sylvester Stallone hawking supplements that promise to help men with erectile dysfunction. 

The ads, which were discovered by Alexios Mantzarlis in the Faked Up newsletter, have been running since at least November 12 and have around 300 variations according to Google’s Ad Transparency Center. All the ads use existing videos that are modified with an AI-generated voice and lip synced to match what the AI-generated voice is saying. Many of the ads feature non-celebrity women who talk about how their “husbands went wild” after “trying a secret simple mix” to treat their erectile dysfunction, but some of the ads feature deepfakes of celebrities including Arnold Schwarzenegger, Sylvester Stallone, Mike Tyson, and Terry Crews. 

“Have you heard about the salt trick that is making me stay hard for hours in bed?” an AI-generated Schwarzenegger asks in his instantly recognizable Austrian accent. “Top adult actors have been using this for the last five years to stay rock hard. I mean, you didn’t think they last that long without a little hack, right?”

Video ads of Stallone, Tyson, and Crews repeating the exact same script indicate that whoever wrote the ads copy/pasted it into an AI voice generator.

The ads lead users to a page on “thrivewithcuriosity.com,” where after confirming they are “40+” years old, they are shown a meandering and very explicit 40-minute long presentation about the miracle drug that is getting men including celebrities, strippers, and adult performers “rock hard.” 

That video opens with a real Today Show interview Stallone did with his wife and three daughters to promote their reality show “The Family Stallone,” but it’s been very uncomfortably edited with AI-generated audio and lip sync to make it seem as if he’s talking about how hard he can get now to satisfy his wife thanks to the miracle drug. 

The video takes viewers on a bizarre journey from a strip club in Texas to a fake Harvard urologist’s office to an abandoned church in Thailand where scientists discovered a species of bat with abnormally large and long-lasting erections. Along the way, deepfake videos of everyone from Tom Hanks, Denzel Washington, and adult entertainment star Johnny Sins are made to say they have been quietly using this secret formula to last longer in bed. The video eventually concludes by offering viewers the opportunity to buy six bottles or 180 days-worth of Prolong Power at $49 per bottle. 

That link sends users to a page on digistore24.com where they can enter their credit card information to purchase Prolong Power, but I was able to find the supplement for sale many other places online. Many sellers on Amazon offer Prolong Power, where it has mixed reviews from users, with some saying “This product is a scam,” “don’t bother,” and “fake.” According to its label, Prolong Power is made up of a “proprietary blend” of oat bran powder, fennel seed, cascara sagrada bark powderact, and other common ingredients that according to the National Library of Medicine are mostly helpful with constipation. Notably, the ingredients do not include “midnight beetle powder,” which the long video pitching Prolong Power explains is the secret ingredient that gave the church bats their magnificent erections. 

Prolongpowers.com, which calls it the “#1 Natural Male Enhancement Supplement” claims it now offers a “new version” it calls Primor Dial Vigor X, and features testimonials from three customers who made “verified purchases.” However, a spokesperson for deepfake detection company Reality Defender said that according to their platform, the headshots attached to those testimonials were 99 percent likely to be AI-generated.

Back in January, YouTube deleted around 1,000 similar ads in which deepfaked celebrities unknowingly pitch scams.

“We are constantly working to enhance our enforcement systems in order to stay ahead of the latest trends and scam tactics, and ensure that we can respond to emerging threats quickly,” Google said at the time after deleting the ads. But obviously it still doesn’t have this problem fully under control.

"We prohibit ads that falsely claim a celebrity endorsement in order to scam people," a Google spokesperson told 404 Media in response to this story. "When we identify an advertiser engaging in this deceptive practice, we permanently suspend their account and remove all their ads from our platforms, as we have done in this case. We continue to heavily invest in our ability to detect and remove these kinds of scam ads and the bad actors behind them.”

Google said it removed the deepfake supplement ads and permanently suspended the account that paid for them after I reached out for comment.

A well established Russian disinformation operation appears to have set its sights on Stalker 2, the hotly anticipated shooter set in the Chernobyl exclusion zone that was developed in Ukraine while the country is actively fighting Russia’s invasion. 

The campaign is spreading a fake vertical video with a WIRED watermark on Telegram and in emails directly to journalists, which falsely claims the game is being used to enlist more Ukrainian soldiers and is secretly collecting private data from players. 

“Ukrainian game S.T.A.L.K.E.R. 2 helps the government locate citizens suitable for mobilization,” text in the video says over gameplay footage of Stalker 2. “An embedded program was discovered in the game’s code that collects player data and transmits it to the developer’s servers. The program collects data about the device, name, IP address and current location of the player. The information is transmitted every second.”

The video falsely claims that the Stalker 2’s developer, GSC Game World, made a deal with the government of Ukraine to get funding for the game, and that it would in exchange help it “find men suitable for mobilization.” 

GSC Game World did not immediately respond to a request for comment, but there is no evidence that Stalker 2 is doing anything like what the video claims, and the video was not actually produced by WIRED. 

The video is also being pushed with the same methods and by the same network that previously targeted 404 Media, which I covered in March. The French press agency AFP, which along with many other news outlets was also targeted by the same disinformation campaign, previously dubbed it “Operation Matryoshka.” As researchers from the Atlantic Council’s Digital Forensic Research Lab (DFRL) told me at the time, the goal is to flood Telegram, social media, and journalists’ inboxes with false stories that usually paint NATO countries and Ukraine in a bad light, waste reporters’ time, and make people distrust news outlets more broadly. 

While various narratives pushed by this network have previously made the news, I get dozens of emails pushing fake stories like the one about Stalker 2 every day, and the vast majority of them get no traction aside from a few tweets, which are often removed by Twitter. 

However, it’s notable that Stalker 2 has gained the attention and ire of Russian propagandists. The first game in the series, which is set in the Chernobyl exclusion zone in Ukraine and was developed in the country, was released in 2007. It was a hugely influential cult hit with a unique mix of intense simulation, sci-fi horror, and open world role-playing that was never properly replicated until the release of Stalker 2 last week. Understandably, in addition to being a game that a dedicated fanbase has been anticipating for almost two decades, Stalker 2 has become a symbol of Ukrainian perseverance in the face of Russia’s invasion in 2022. While many of its developers were relocated to Prague, some developers finished the games under Russian bombardment and with its team members being deployed to the frontlines in an active war. 

Given that it has become such a symbol in Ukraine and around the world, it’s not a surprise that a Russian disinformation campaign would try to undermine it.