Edit: I've edited this idea to only apply to something like siege equipement players can carry, like bombs, mines, (Arcane Engineering?) etc that would work like this: The more targets they hit, the bigger boom they make. Smaller groups can take more advantage of this mechanic better than larger groups would.
Zergs and Deathballs where players are just clumped up in one big ball, steamrolling enemies/objectives is a common gripe people have with a lot of pvp in many games. Obviously numerical superiority is a factor in the success of a fight (in addition to tactics, coordination, organization, etc). But just as it works in "real" battle, overly concentrating your forces in a small area can risk those forces being "nuked" all at once.
In the case of an MMO that wants to feature large player driven battles however, for the purpose of balance and gameplay, ideally we would want to find a way that breaks up these zergs and potentially discourages or "punishes" mingless zerging tactics but at the same time, not lending those zergs the same tools to just turn around and use against smaller, possibly more organized groups and just bringing you back to square one.
So what about this (ignore the percentages, theyre just a rough example): AoE's (whether ranged or melee) scale UP in damage when exceeding a certain amount of players hit. Example: 1-8 targets = 100% normal ability damage /// 8-16 targets = 125% ability damage /// 16-32 targets = 150% damage, so on and so forth.
What this would do ultimately is allow for better organized. smaller groups to concentrate their attacks and punish mindless zergs despite being a smaller force. On the flip side, the only way for large raids to AVOID getting their groups nuked like this would be to actively coordinate them and split them up into better organized groups, hence literally acting as a way to "break up" the usual "deathball" we often see in pvp.
And if they cant? Great! They die