Concerns Grow over AI-Generated Abusive Content among Children

Children using AI to generate pornographic imagery and mock their school friends is a growing concern in the region. Authorities say they have few such cases, but NGOs warn the phenomenon is widespread and must be addressed in law.

In February, Serbia’s public broadcaster reported that an unspecified number of 13-year-old boys were suspected of using AI apps to create nude images of their female classmates and teachers at a Belgrade school and then shared them in chat groups. Parents and teachers were outraged, but, according to a BIRN investigation, the case is just one among many across the region.

A simple Google search turns up dozens of free apps offering users the ability to ‘undress any girl for free’ or ‘remove clothes simply and fast’.

Aimojo.pro, which analyses AI tools, has identified more than 30 undress AI tools, many of them free to use. The apps offer customisable options, including age, body type, and personalisation of AI companions, enabling the creation of AI girlfriends and explicit content via deep learning algorithms to digitally remove clothing. Some are free if used within the chat app Telegram, which is frequently used for the sharing of inappropriate content in secret channels.

According to BIRN’s findings, only a small fraction of such cases get reported; experts and authorities expect AI to increase the creation of pornographic content. Some end up in court as the cause of private lawsuits for crimes such as misuse of data or identity theft, but experts say laws must catch up, and fast.

“It would be valuable if the state would investigate this phenomenon and, accordingly, develop new policies or revise existing policies and, if necessary, amend laws to clearly identify this criminal offence within the system for protection children from violence and sexual abuse,” said Radovan Cicvaric, manager of the Children’s Rights Centre in Uzice, western Serbia.

Serbia, Montenegro

Since July last year, high-tech crime investigators of the Serbian police say they have identified several cases in which AI has been used to create explicit content; in November, one individual was arrested on suspicion of creating content based on images of six female persons, including one minor. The individual was charged with sexual harassment, stalking, and threatening security.

Illustration: Ben Wicks/Unsplash.com

In Montenegro, the NGO ‘Parents’, which works to promote the rights of parents and children said it had received several reports of manipulated, sexually-explicit content involving minors over the past year.

“The latest case concerns school peers who created nude photos of one girl and the material was shared among them,” the NGO told BIRN.

‘Parents’ said that the victim’s parents had contacted local authorities. Public prosecutors in the municipality where the incident occurred told BIRN a case was opened at the beginning of the year against four minors suspected of the unauthorised distribution of another minor’s private content.

The Montenegrin police high-tech crime department said it had recently detected the use of AI-generated fake content but did not specify whether any included elements of child pornography.

The police cited cases of ‘deepfakes’, in which images, audio or videos are digitally altered with AI to ‘fake’ someone’s appearance; according to a study published in February by World Vision Romania, 10 per cent of teenagers in Romania have been the target of a deepfake.

Around 40 per cent of respondents, most of whom were secondary school children from rural areas, said they had been the victim of online abuse, many of them by adults who sent them private messages containing sexual content.

It was not specified whether the deepfakes reported were sexually explicit in nature.

Iva Erakovic, executive director of Friends of Children of Serbia, said image-based sexual abuse in general remains unrecognised as a criminal offence.

“This situation requires the victim to pursue a private lawsuit, which will likely result in further abuse, including the use of artificial intelligence,” Erakovic told BIRN.

Romania

While the use of AI tools to modify images is legal per se, when it comes to children, any intent to create child pornography should be considered a criminal act, experts have told BIRN.

Most cases reported in the region, however, are prosecuted only as breaches of privacy, identity theft, and misuse of data.

Without legislation specifically addressing such cases, however, most end up as private lawsuits.

In September 2023, judges at Giurgiu Tribunal, just south of Bucharest, ruled in the first and the only publicly known case in which a schoolgirl used artificial intelligence to produce a deep fake explicit video of a school friend. The case was a civil proceeding launched in mid-2021 by the parents of the victim, a 12-year-old boy, against the parents of the girl.

According to the plaintiff’s lawyer, the girl superimposed the 12 year-old’s face on a pornographic video, showed it to their friends in school and posted it to YouTube. The girl’s parents were ordered to pay 1,000 euros in compensation; the victim’s parents were ordered to pay the same amount after he insulted the girls for posting the video on YouTube.

Police in Romania told BIRN they had not registered any specific cases of child pornography involving deepfakes.

Legislation outlawing deepfakes came before Romanian MPs in April 2023 but was removed from consideration in February this year and sent back to the committee stage following criticism from civil society about inconsistencies in the bill.

Illustration: Gaelle Marcel/Unsplash.com

North Macedonia, Albania

In North Macedonia, six minors in the south-central town of Kavadarci were accused of downloading the private photos of their female classmates, altering them and sharing them with friends.

One parent told the MIA state news agency that they “made photomontages in which the heads from photos of girls taken from Facebook were edited with the app onto explicit photos of naked bodies”.

In February, prosecutors in the town said they had launched a case based on the crime of ‘misuse of personal data’.

The police’s Sector for Computer Crime, said it did not have any cases of identity abuse via AI.

In Albania, iSigurt.al, which deals with cases of the cyber-abuse against children and young people, said it had started to receive reports of AI-generated content containing child sexual abuse.

“Having as our primary mission the protection of children and youth from every form of online abuse, we are not only reporting these cases to the respective national and international departments, but we are also conducting international training to be more up to date with this new important phenomenon that is a concern in the online sphere,” iSigurt.al psychologist Borjana Dine told BIRN.

In Kosovo, Petrit Tahiri, programme manager at the Kosovo Education Centre, said it is essential that new legal frameworks and regulations are developed to address the unique issues arising from AI.

“Legal systems must evolve to keep pace with technological advancement and to protect everyone’s rights, privacy and security,” Tahiri told BIRN.

In Bosnia, police in the mainly Bosniak and Croat Federation entity and the Brcko district said they had not yet registered any such cases. The Safer Internet Centre, a non-profit leading an umbrella group of child protection organisations and in charge of reporting and monitoring inappropriate content online, said the same, but added that it expects to do so. Police in the predominantly Serb-populated Republika Srpska entity did not respond to a request for comment.

It remains unclear how such cases would be processed in Bosnia given its outdated legal framework and disparities between the various levels of governance.

“Bosnia and Herzegovina does not have a state-level law regulating these things,” said Amela Fazlic, manager of the Safer Internet Centre.

According to Fazlic, the Republika Srpska has one article in the Criminal Code concerning child exploitation via IT technology, while the Federation and Brcko have laws that are not harmonised between Bosnian entities or between the 10 cantons in Bosnia’s Federation entity .

In the event of such cases, the suspects would be processed under the Criminal Code of the entity or Brcko District, depending on where the crime took place, but which currently only regulates child pornography. “None of the laws regulates a child’s exploitation without explicit contact and direct creation of pornographic material, grooming, or inappropriate communication,” said Fazlic.

Illustration: Vitolda Klein/Unsplash.com

Prevention and education

The experts who spoke to BIRN said prevention requires the involvement of parents, teachers, and other professionals working with children.

Child digital literacy and responsibility should become a priority both at school and in the home, they said.

“Informing parents and the population is considered very important widely for internet security,” Tahiri said via email.

“Parents do not have enough information about internet safety and therefore are unable to inform their children. Therefore, at the school level, information sessions for parents should be prepared. There are movies/videos of very good awareness-raising briefs for parents and the community to use for raising their awareness about such topics.”

Erakovic also expressed concern at the lack of education concerning data privacy, protection on social media, potential abuses, and unmonitored internet usage by children, often disregarding age requirements.

“It is necessary to assist parents and teachers in navigating these situations better, with an emphasis on preventive activities,” said Erakovic.

“Parents, in turn, should maintain an open and trusting relationship with their children, as it is crucial for a child to have someone to immediately report such an incident to and not succumb to pressure and threats. The fact that known cases worldwide have ended fatally, such as with suicide, should not be ignored.”

BIRN’s Digital Rights Monitors contributed reporting: Djurdja Radulovic (Montenegro), Aida Trepanic and Azem Kurtic (Bosnia and Herzegovina), Bojan Stojkovski and Goce Trpkovski (North Macedonia), Adina Florea (Romania), Ákos Keller-Alánt (Hungary), Tijana Uzelac (Serbia), Nensi Bogdani (Albania), Flaka Fazlija (Kosovo).