How do you feel about some of the ingame chat moderation being done by an AI?

I don't know a whole lot about AI, but from my limited experience with it and from what I have read about it, it's pretty much become the norm for the big social media sites to use AI/machine learning to moderate their content. We've reached a point where the AI can speak pretty much any language, and understand context fairly well. Facebook, Tinder, Linkedin, Instagram and more use it.

Those are huge platforms, so it's obviously impossible to hire enough humans to do the job. And with potentially a million or more Ashes players, that's a lot of chat being generated every second, so I can see a potential for AI moderation in the game as well. I have zero clue how much it would cost Intrepid to implement it, but for the sake of the argument, let's say they can do it.

But is it a good idea? And if so, how much power do you want to give the AI? Should it be able to ban people? For example, if it identifies one of those annoying goldsellers spamming, using their usual tricks of using weird symbols and spaces to circumvent simple word filters, should it just straight up ban the account? Perhaps only mute and flag for moderation by an active GM?

What about guild or group chat, and even /whispers? Obviously nothing is private in the game, including whispers. Assume it's all being logged anyway, even if only for a short period. Do you want the AI to moderate whispers, so if someone starts calling you all kinds of slurs or threaten or wish you bodily harm, the AI catches it before it even reaches you and blocks the message and flags the offender for the GM team to review.

There are some obvious downsides, like the AI not exactly being infallible, but I can see some upsides as well.

Thoughts?

Find more posts tagged with

Comments

Liniker

Absolutely No, terrible, terrible idea, we need human beings to make decisions, we have more than enough examples of how bad it is when you have automated bans in video games

Azherae

It would be mostly fine, but the adaptations of toxicity would be quick, and more importantly, 'meant to create false positives'.

Humans will adapt their 'methods of being toxic' faster than most language programs can keep up because humans can easily substitute terms. It's not that the AI wouldn't detect the new terms after a while, it's that people find ways to change the context to make 'regular conversation' cause things to flag.

Definitely not impossible, but I personally feel that the input-to-effectiveness ratio might be too low, for now. A full custom solution might be worth implementing, but this would almost certainly lead to 'English Only'.

And of course, some people are 'naturally foulmouthed or lewd' in specific ways that their friends or allies are okay with, and while that would probably be 'okay' to just 'ignore chat between people on friends lists', that also has its issues.

MybroViajero

I think AoC should have a system of honor and veracity (something like the honor system of Lol), players with honor 10 (from honor 5 the moderators/devs grant the honor) could make reports and the moderators could check more in those reports, that way more field would be covered.

In Rust, server moderators review more reports from people who have previously had confirmed reports.

Ludullu

I think players should just be given word filter and word report tools. You see a word you dislike? You report it (which automatically filters it on your end). Enough reports of the same word would filter it for most people (excluding anyone who checked an option of "don't filter words"). And GMs could get flags of people who had their words filtered.

In other words, I just don't think that AI would be able to properly control the chat.

Dolyem

No thanks, AI isn't great with context as far as I can tell. Leave it to people with the least biases to moderate communication.

Kilion

I don't want anyone living there Orwellian wet dreams out even in a freaking game. There is no need to have anyone read and oversee every bit of information going back and forth between players. We are not criminals, we are not children. If some dispute happens, sure ask a mod and if necessary they can dig up the conversation log.

No need for a piece AI to sniff out every conversation like everyone is under permanent suspicion to do something bad. Adding to that: Intrepid would have to buy that piece of tech from someone else. You can bet your behind on that AI processing all the data and funnel it back to someone else.

AI surveillance puts all players under suspicion for no reason and is a risk to privacy.

daveywavey

Ashes will have in-game GMs. If all the AI does is flag potential breaches up to the in-game GMs, then that's fine.

Jhoren

Azherae wrote: »

It would be mostly fine, but the adaptations of toxicity would be quick, and more importantly, 'meant to create false positives'.

Humans will adapt their 'methods of being toxic' faster than most language programs can keep up because humans can easily substitute terms. It's not that the AI wouldn't detect the new terms after a while, it's that people find ways to change the context to make 'regular conversation' cause things to flag.

In general terms, sure. But for Ashes? People are gambling with their accounts when experimenting with substitutes and workarounds in this case. Accounts aren't that easy to replace once you have a levelled up character.

daveywavey wrote: »

Ashes will have in-game GMs. If all the AI does is flag potential breaches up to the in-game GMs, then that's fine.

I think this would be an ok solution.

NishUK

Dolyem wrote: »

No thanks, AI isn't great with context as far as I can tell. Leave it to people with the least biases to moderate communication.

Even Elon is struggling to do that atm with who he's currently hired but at least he has the excuse of he is managing more than 1 thing. This company is based in California so I have little faith that they can find or even bother to utilize smart people for the job that possess very little bias and they will bend the words "unsafe" and "toxic" for all their worth and vague meanings.

Dolyem

NishUK wrote: »

Dolyem wrote: »

No thanks, AI isn't great with context as far as I can tell. Leave it to people with the least biases to moderate communication.

Even Elon is struggling to do that atm with who he's currently hired but at least he has the excuse of he is managing more than 1 thing. This company is based in California so I have little faith that they can find or even bother to utilize smart people for the job that possess very little bias and they will bend the words "unsafe" and "toxic" for all their worth and vague meanings.

You're not wrong. I'd make it a remote job that can be done online honestly. And screen people with psychological tests to vet them.

maouw

It would probs be most effective as an assistive tool to mods/community managers - so that it's always a human making the final decision, but they don't have to read every word ever typed by the community hahaha.

You could also reduce the "human adaptation" delta by publicly marking posts that the AI has picked up, so the community just focuses on reporting things it DOESN'T pick up.

---

I think we could learn a thing or two from the Twitch moderation community - they basically specialize in live moderation of big data (there is no way mods can read every message in a twitch chat).

That said, I know some moderation teams use what's basically equivalent to a social credit system - keeping a hidden score for every user in chat, so mods can quickly identify repeat offenders.

I dunno if I like that. But I can't deny it works.

Songcaller

Dolyem wrote: »

NishUK wrote: »

Dolyem wrote: »

No thanks, AI isn't great with context as far as I can tell. Leave it to people with the least biases to moderate communication.

Even Elon is struggling to do that atm with who he's currently hired but at least he has the excuse of he is managing more than 1 thing. This company is based in California so I have little faith that they can find or even bother to utilize smart people for the job that possess very little bias and they will bend the words "unsafe" and "toxic" for all their worth and vague meanings.

You're not wrong. I'd make it a remote job that can be done online honestly. And screen people with psychological tests to vet them.

I disagree with psychological tests to be able to participate in a form of relaxation.

Dolyem

Neurath wrote: »

Dolyem wrote: »

NishUK wrote: »

Dolyem wrote: »

No thanks, AI isn't great with context as far as I can tell. Leave it to people with the least biases to moderate communication.

Even Elon is struggling to do that atm with who he's currently hired but at least he has the excuse of he is managing more than 1 thing. This company is based in California so I have little faith that they can find or even bother to utilize smart people for the job that possess very little bias and they will bend the words "unsafe" and "toxic" for all their worth and vague meanings.

You're not wrong. I'd make it a remote job that can be done online honestly. And screen people with psychological tests to vet them.

I disagree with psychological tests to be able to participate in a form of relaxation.

I mean for a moderation job

Songcaller

Oh okay no worries.

maouw

I like this thread.

Diamaht

Kilion wrote: »

I don't want anyone living there Orwellian wet dreams out even in a freaking game.

This. No thanks.

It always breaks down the same way.

Q: "Well OK, but who decides?"
A" "Well, me of course."

Dygz

Terrified!!!

maouw

Diamaht wrote: »

Kilion wrote: »

I don't want anyone living there Orwellian wet dreams out even in a freaking game.

This. No thanks.

It always breaks down the same way.

Q: "Well OK, but who decides?"
A" "Well, me of course."

I mean, I agree this is 100% an issue right now, but I feel like it has a really simple solution:

Public logs of moderation decisions.

My theory is that if you just leave everything in the light and give it fresh air, then it's much harder for mould to grow. Maybe I'm too naive?

Diamaht

maouw wrote: »

Diamaht wrote: »

Kilion wrote: »

I don't want anyone living there Orwellian wet dreams out even in a freaking game.

This. No thanks.

It always breaks down the same way.

Q: "Well OK, but who decides?"
A" "Well, me of course."

I mean, I agree this is 100% an issue right now, but I feel like it has a really simple solution:

Public logs of moderation decisions.

My theory is that if you just leave everything in the light and give it fresh air, then it's much harder for mould to grow. Maybe I'm too naive?

Its not a bad idea really.

The issue is that an auto moderation function is the opposite of fresh and free. It's biased based on the person creating the software. Its an instance where the few decide for the many, and it eliminates context. It's not as large of an issue in a video game chat log as it is applied to wider society, but the same concerns apply.

You wouldn't be able to blame Intrepid for filtering out certain types of content/speech, however an AI implies that software is making decisions and (more importantly) taking actions against players based on its own, human created and flawed algorithm.

If I'm facing in game discipline from Intrepid, I want to hear from Intrepid in some capacity, not from software.

Edit: Also, as for public logs, it's a slippery slope there too. It can be a bit of a public flogging (and no end to Youtube content) if the nasty things people say to each other are aired out for all to see. It would also present the entire community in that light.

Azherae

Diamaht wrote: »

maouw wrote: »

Diamaht wrote: »

Kilion wrote: »

I don't want anyone living there Orwellian wet dreams out even in a freaking game.

This. No thanks.

It always breaks down the same way.

Q: "Well OK, but who decides?"
A" "Well, me of course."

I mean, I agree this is 100% an issue right now, but I feel like it has a really simple solution:

Public logs of moderation decisions.

My theory is that if you just leave everything in the light and give it fresh air, then it's much harder for mould to grow. Maybe I'm too naive?

Its not a bad idea really.

The issue is that an auto moderation function is the opposite of fresh and free. It's biased based on the person creating the software. Its an instance where the few decide for the many, and it eliminates context. It's not as large of an issue in a video game chat log as it is applied to wider society, but the same concerns apply.

You wouldn't be able to blame Intrepid for filtering out certain types of content/speech, however an AI implies that software is making decisions and (more importantly) taking actions against players based on its own, human created and flawed algorithm.

If I'm facing in game discipline from Intrepid, I want to hear from Intrepid in some capacity, not from software.

I don't know how Amazon does it, but for the ones I've poked at or worked with/on, this isn't how it works.

The 'AI' parses two things, 'sentiment' and 'intent'. What it lacks most of the time is 'context'.

So it can tell you 'at X time on Y date Z player used a clearly racial slur in a chat'. And good ones can tell you 'whether the tone of the conversation in the surrounding lines was positive or negative or neutral' based on some other types of 'emphatic' language.

It SHOULDN'T then be able to ban or probably even mute the person. But it COULD be set to the point where if a person SAID something and another person reported it, the flag that would come through would be more like 'Confirmed usage of racial slur in negative sentiment at X time', probably with the required snippet from chat logs included.

A human can glance over that easily.

Similarly, the AI could issue warnings under the same conditions, because a warning is a useful point to tell a player 'hey you're about to go over the line maybe, and you are now being watched more'. They could then complain that the algorithm is badly tuned and explain the thing they said. There's no punishment then, it could be outright ignored.

An AI to tell people 'hey you might be close to breaking the rules here, tone it down' would not be a problem, and I consider this part of chat moderation.

Child Item