Civitai founder champions open source, downplays AI deepfake porn

Justin Maier, the Mormon-raised, Boise, Idaho-based founder of open-source AI platform Civitai, has had a wild year — and a rough few months. 

His company was founded a year ago to support a community discovering, creating and sharing models and image-generated content based on the popular text-to-image generator Stable Diffusion. Since then, it has exploded from a four-person startup and less than 100,000 users to a 15-person company with $5 million in funding from VC firm Andreessen Horowitz, growing rapidly to 10 million unique visitors each month and millions of uploaded images and models. 

At the same time, he has recently been dealt two serious blows: There was his daughter’s Type 1 diabetes diagnosis and treatment, which came the same week as Civitai’s funding round. There was also months of critical coverage by independent tech journalism site 404 Media, which has published several stories about Civitai accusing the company of creating an  “AI porn marketplace,” profiting “from nonconsensual AI porn”; introducing bounties for deepfakes of real people; and generating images that ‘could be categorized as child pornography.’” 

But Maier, a graduate of Brigham Young University whose X profile describes him as a “father, husband, and developer” who is “trying to become less wrong and making mistakes along the way,” believes the 404 Media reports mischaracterize Civitai’s primary user base and use cases.

He told VentureBeat in an exclusive interview that it is “challenging and sad to be…thrown into this mess.” He calls Civitai “a small company doing our best to get access to more people that generally are using this for good.” 

Maier further said Civitai has ”worked really hard to ensure that we’re keeping things safe, but this space is moving so quickly, and interest is growing so quickly, that we have to move and change and adapt on a daily basis,” adding that half of the company’s team is dedicated to content moderation. 

Maier has found himself at the center of the debate around the merits of open-source generative AI that continues to play out around the internet and among regulators. Civitai can be seen as an example of both the promise of the technology — creating thriving new communities — and the downsides, allowing objectionable content to be created at a scale greater than before and difficult for even motivated platform owners and administrators who are opposed to it to reign in.

Civitai is a platform for LoRA model enthusiasts 

The vast majority of Civitai users, Maier explained, are simply LoRA model enthusiasts — LoRA models are small, fine-tuned models trained on specific characters or styles — looking to express themselves through AI art generation for everything from fan fiction and anime characters to photorealism and even fashion. 

“When we launched just a year ago, we had 50 models, which was like all that had been made for the last three months,” he said. “And now on a daily basis, we get 500 models.” 

While 404 Media’s accusations are disturbing, Maier emphasized that they are often misleading, using figures from June 2023, for example, when Civitai’s image-generating feature was still in internal testing (the company said it launched in September). 

Contrary to those figures showing 60% of content on Civitai as NSFW (Not Safe for Work) — a figure derived from 50,000 images — today users on Civitai generate 3 million images daily, and the company says “less than 20% of the posted content is what we would consider ‘PG-13’ or above.”

“It makes me really sad to be dragged through the mud for something we’re actively working to prevent and doing our best to solve,” said Maier.

He also pointed to a new safety center on its website and policies such as Three Strikes and Zero Tolerance for inappropriate content. Maier said the center was introduced to make it easier for users to find the policies, but that it was not introduced in response to 404’s reporting — and that the policies predated the reports.

Among the policies listed in Civitai’s safety center are a ban on “all photorealistic pictures of minors” as well as “all sexual depictions of minors.” The policy says Civitai uses Amazon Rekognition to automatically detect and flag content that violates these policies. “We have a 0 strike policy for violations involving minors,” the policy FAQ states. “Offending content will be removed, and the uploader will be banned from the platform.”

Civitai started as a ‘passion project’ for generative AI hobbyists

Civitai started as “a passion project,” Maier said. After a friend introduced him to Midjourney in August 2022, Midjourney’s limitations — in speed and styles — led him to become active in the Stable Diffusion community. 

“I started to see people sharing models intended to do specific styles — they figured out how to put themselves into a model and so I made a model for each of my family members,” he said. People began to share their models on sites like Reddit and Discord, and Maier said he felt there should be a place to make it easier to browse the models. 

Civitai launched in November and by January, the site had 100,000 users. “It’s just been a whirlwind since then,” he said. “By March, we hit a million users.” 

Maier said that Civitai has lowered the barrier to entry for open-source generative AI. “There’s consumer-focused tools like Midjourney, and enterprise-focused tools like Hugging Face — we’ve struck that fine balance of hobbyists that want to dive a little deeper without having to figure out all of the innards of machine learning,” he said. 

NSFW content has always been an issue

Even before he launched the site, Maier said he was aware of people using Stable Diffusion for NSFW content.  “We basically had to prepare in advance for the things we had already seen,” he said. “We wanted to give people a lot of control over what they could and couldn’t see.” 

When asked why he didn’t simply reject NSFW content on the site entirely — as other popular image generators such as Midjourney and OpenAI’s DALL-E 3 do — he said “We could have prevented that stuff from being posted, but I felt like it would put us at risk of hampering the development of the community too early,” adding that “we’re kind of at the center of open source AI development around images.” 

For example, he explained that he saw what was happening in terms of LoRA models being developed based on Stable Diffusion specifically to do human anatomy better for pornographic purposes. 

But he pointed to the New Testament’s Parable of the Weeds as an explanation — the parable, related by Jesus in the Book of Matthew, describes how servants eager to pull up weeds were warned that in doing so they would also root out the wheat, so they were told to let both grow together until the harvest.

“People that are there to make these NSFW things are creating and pushing for these models in ways that kind of transcend that use case,” Maier said. “It’s been valuable to have the community even if they’re making things that I’m not interested in, or that I prefer not to have on the site.” 

As people in the community tried to improve anatomical concepts in Stable Diffusion, he explained — training models on better faces, eyes, hands or yes, even penises — the result was models that were better at doing things like human faces, or anime, which were then merged for even more improvement, he explained. “This is an open source community of hobbyists who have pushed the technology forward, perhaps even further than Stability [the company behind Stable Diffusion], this company that had hundreds of millions of dollars for the tech.” 

When asked if by pushing the technology forward Civitai also enables the potential for deep fakes or pornography, Maier said that one challenge is that anatomical concepts can overlap.“ If we didn’t capture penises, what else is going to be affected by that?” he said. “How the weights affect each other with this stuff is that by not properly capturing penises means that fingers look funny now.” 

He pointed out that after the original Stable Diffusion was released, Stability AI received backlash for having trained on things including nudity. “They actually went back and trained again from the ground up, removing essentially tons and tons of content from their data set,” he said. “The end result was this model that had been trained on high-resolution images, but could not render good looking people.” 

Accusations of ‘bounties’ for deepfakes

404 Media’s recent coverage of Civitai also included accusations of ‘bounties’ for deepfakes of real people. According to a Civitai representative, ‘bounties’ allow users to post listings for desired services such as AI model creations, for which other users can submit their entries. For example, someone might post a bounty for a model that creates photorealistic images of Tom Cruise. Bounty submissions are private, the company said, and can only be seen by the poster. 

“Bounties are something that we originally thought of in December [2022],” said Maier. “People were basically contacting each other on Discord or Patreon and were like, ‘I’d like to have this thing and I’ll send you a tip’ — so we called them ‘bounties’ because it seemed like an awesome opportunity for people that looking to make a name for themselves to see what people wanted.” 

If a poster wants to share a bounty model publicly on Civitai, they have to post at least three sample images alongside the model, which “are bound to the same content moderation filters and review as all other content posted on Civitai prior to being approved,” said the Civitai representative, including the company’s Real People Policy —which says that “Portraying real people in any mature or suggestive context is strictly prohibited.” All content uploaded to Civitai is scanned and tagged by AI systems to identify what is in the image/video and what resources were used, the representative said: “If it is detected that a real person resource was used and any suggestive/mature content labels, it’s reviewed by a human moderator before it is visible on the site.” 

According to Civitai, the company “also encourages and incentivizes community reporting of inappropriate content — much like the Civitai user base has rallied around the rollout of bounties, they are also strongly motivated to help keep the site a safe and positive environment for its users. Both completing bounties and reporting violating content are incentivized by Civitai’s on-site currency, Buzz (similar to Reddit Karma points).” 

Civitai cannot control how models are used once downloaded or moved 

However, there is a catch — as an open-source AI platform, Civitai cannot control how the models shared on their site are used once downloaded or moved to another platform. So someone could download a Tom Cruise-generating model acquired on Civitai, install it onto a generator with fewer moderation filters, and use the model to create NSFW content. “However, this type of content cannot be created or posted while utilizing the Civitai platform,” said the representative.

Maier said there is recourse for those who want images with their likeness removed, or artists who want images with their styles removed, but said that people rarely reach out to do so. For example, the 404 Media coverage mentioned an Instagram influencer, Michele Alves, with a bounty on Civitai, who said: “I don’t know what measures I could take since the internet seems like a place out of control. The only thing I think about is how it could affect me mentally because this is beyond hurtful.”

Maier said the company did remove the Bounty but never actually heard from Alves.  “We saw that she was concerned about it, we don’t want people to feel like they don’t have any recourse here,” he said. “We try to make it as obvious as possible that people can request to have these things removed.” He added that the company is currently working on a way “for people to essentially come and claim their likeness, to own who they are when it’s generated by AI.” 

In December, he continued, Civitai also added the ability for artists to say “Hey, I think this uses my images and its training data, and then make a request to have us reach out to the creator of the resource and say we’d like to have this removed. We’ve gone through that process maybe five or six times now.” Maeir said, “It’s pretty rare for an artist to actually reach out.”  

AI development is only accelerating

AI development is only accelerating, said Maier, who explained that Civitai — which until June was a team of four — and other companies are “moving quickly as they can” to make sure that policies are developed to keep up. “And it’s not just companies,” he said. “I was in a meeting last week with the governor of Utah, Spencer Cox, talking about how we can have a light touch to ensure that the space continues to develop and that the public is safe.” 

When asked about the impact of Civitai on his two daughters, he explained that one of his daughters loves to draw. “She wants to be an artist, so from time to time I take drawings she’s made — she loves drawing zombies — and she’ll work with me on AI generations that turn it into something that looks really real,” he said.

Maier also hopes for other Civitai impact on his daughter: This month, Civitai is doing a holiday charity drive for the Juvenile Diabetes Research Foundation. 

“I’d love to be able to get more money for the JDRF so that I can work on making things a little bit better for my daughter,” he said, “because it’s been a rough few months.” 

The bottom line, he added, is that “we do care deeply about making sure that our platform is safe.” Civitai’s goal, he said, is to make AI more accessible to more people. “But that doesn’t come without challenges and it doesn’t come without difficulty to try and make it so that people can use this in so many different ways,” he said. “So we try on a daily basis to keep things on the rails. For a small company like ours, it’s a challenge, but we’re doing the best we can.”