you mean scaling UP the more players are hit?
UP
Theres no reason to scale down
fights unbalanced in numbers
Picking bad fights is not reason to increases aoe damage neither.
Do you mind creating an argument?
I agree that some figths should not be fought but it is fun to e.g. run away 3 fights to fight one?
Making AOE damage an impressive tool to fight against larger groups based on the enemy's numbers could encourage people to run in less numbers.
Also, a lot of players in other games out there use "stack and spam AOE" to just kill everything without any additional efforts or downsides. With a system like this, you could make AOE usage against small groups something more irrelevant, encouraging large groups to fight individually while keep a constant care about positioning since the enemy attacks will be stronger if you stack too much.
scaling up could work for something like a plague - where the contagion is accumulative, and the more people around you who are affected by it, the faster is accumulates on you.
Also worth noting: if you boost the AoE meta, you also boost the assassin meta because everyone spreads out.
This system would have to work for every AOE and yes, this would boost assassin's play but do not forget that even people having to spread out and having downsides on AOE usage against lonely assassins, they are more in numbers and can still focus fire easily on him.