Greetings, glorious testers!
Check out Alpha Two Announcements here to see the latest news on Alpha Two.
Check out general Announcements here to see the latest news on Ashes of Creation & Intrepid Studios.
To get the quickest updates regarding Alpha Two, connect your Discord and Intrepid accounts here.
Check out Alpha Two Announcements here to see the latest news on Alpha Two.
Check out general Announcements here to see the latest news on Ashes of Creation & Intrepid Studios.
To get the quickest updates regarding Alpha Two, connect your Discord and Intrepid accounts here.
Preparing for the masses, the crashes, and the Anti-cheat system.
Acarith
Member, Alpha Two, Early Alpha Two
Hello everyone.
My main experience with MMOs is ESO. If you’ve followed the story of its main PvP content, Cyrodiil, you’ll know it’s a tragic story of a lost vision.
I really like the vision of AoC. As many say it’s probably the most inspiring MMO coming out. Its melodic open world PvP is a dream many have wished for. I believe in it’s ambitions. Where there’s a will there’s a way. But ESOs problems arise from so many dark corners it’s hard for me to start this post.
Let me explain what Cyrodiil is. It’s ESO massive tri alliance pvp arena. The estimates of the initial size of it’s server cap were 600v600v600 players per server. This was a gamemode that had keeps (little mini castles) and sieges that could be initiated at any moment.
Sounds fun? Well the whole thing now is in a sorry state of disrepair. Some say it can’t be fixed. Some say it's better abandoned. As a result of bad decision making it was abandoned, leaving it’s player base in disarray. Then carma struck and now the whole game is starting to act like Cyrodiil. It was made apparent with a near failed launch of their newest DLC, things over there aren’t right.
However plenty of lessons for AoC to learn for free can come out of that game.
One excuse that is always made is that ESO runs on an old outdated engine from 2007. There is they say that the licensing doesn't allow them to update the engine, but that sounds heavily like a management issue. Because how hard is it to get a new license?
One thing AoC can learn from that, is to maybe export textures and resources to Universal file formats that then can be added to future versions of the engine as plug-ins. Or at least starting meshes and templates so you're not left rebuilding the whole game if you want to upgrade the engine in say 2027. I’m no animator or graphic artist, but where my field of reference comes from, I know you can download 3D models from off the internet to be used in various programs. Is there a similar template approach that could be created and stored for the company’s artwork? A redundant back-up.
I said all that, but that’s not the big issue with ESO. One big issue with ESO, is that no one knows what the big issue with ESO is. Theories are all over the place. A problem is it doesn't have only one problem.
A big management mistake they made was not tackling problems with Cyrodiil. They ignored them. They let them stack up, releasing patch after patch of mistakes and extra glitches. One thing AoC can do is make a nominated team of skilled individuals from different departments that makes a list, looks for problems, and fixes them. It’s important this team is diverse so they can pinpoint where and which department the problem arries, maybe it is a cross department issue. They don’t need to have one specialist of the team do the job, but have the ability to send the problem back to the right department.
At ESO I really don’t know what they have going on but it doesn’t work. Unlike Intrepid, their company is ran by business types, who I think hear the intake of problems and look for the cheapest solution. Like for all its current server issues they’re trying to have the combat team fix it. Like they’re trying to have the people who just play with and change damage values fix server load… It’s a mess.
Now getting to the title of this thread.
If you ask a lot of veteran ESO players they’ll tell you performance has been on the decline in ESO.
A big problem is when they moved certain client combat values and interactions server side so they couldn’t be modified by modders and hackers to give themselves the edge in combat. Like people were flying, had unlimited ultimates(a super move that usually has like a minute +cooldown,) it was a circus. The bigger circus came after that as ESOs solution to it was a curse. Player positioning became a problem. They’d reduced the server population. Abilities wouldn’t fire correctly or be timed right. Performance took a big hit. And things got worse as the game progressed for reasons I don’t fully know, and I doubt anyone fully knows. But there are plenty of ideas why.
The big one is they kept adding input checks server side, so that nothing the client said was trusted. Something they obsess over despite all the negative consequences and alternatives. Intrepid probably has smart innovative people that can come up with their own solutions. Word of advice don’t copy ESO on this one.
One possible solution to this is to partner with or develop a cloud based gaming service and host the game on your partnered or owned gaming service. This gives you control over what can and can’t be installed on the cloud systems so you don’t have to worry about mods. I threw this idea out to ESO too and pointed them towards a company called Shadow. I’m not sure if they took interest, but might as well make it industry practice as it solves a lot of problems as I will continue to point out.
The way the concept works is you hardwire the server and hosting cloud PCs to a LAN. This eliminates pretty much all the latency in the games calculations. Network latency won’t cause problems where players get hit with abilities all at once, instead they may get small screen hiccups they can adjust for in gameplay.
Another problem hosting on high end cloud PCs will fix is client discrepancies and rendering issues. A problem ESO had, even though they have the minimum hardware specs, is that clients couldn’t handle the combat. ESO Devs still ignore certain quality of life problems with this issue like consecutive loadscreens, but the game used to crash constantly. It’s to the point Xbox and PS4 in all their gaming dedicated resources and splendor were considered toasters because they could hardly handle the game. On a regular PC the amount of system resources the game takes up is ridiculous. The sad part is even with a top of the line PC you still lag like the “toasters” did back in the day because it’s gotten that bad. ESO Devs choose to ignore this hardware issue, because players can try to improve it. Some credit they did eventually lower crashing, by lowering Fx and some stuff with the server. However, some including myself, point to their Anti-crash solutions as another source of ESO’s compounding server lag.
Lastly there’s the masses problem, tragedy of commons and so forth. Some say AoC is ready for it with an open beta. But what about the excessive client character and ability rendering that made ESO crash at every launch it’s ever had. Can the servers handle consecutive ability spams from so many players in a cluster. I hear the dungeon raid sizes are going to be huge. Will all the graphic aesthetics be too much for the average client?
Cloud based solutions again can be tapped to control this problem as you’ll know exactly what are on systems and what they can handle. A lot of problems some gamers have with games is even though their PC’s to spec, their Anti-virus taking up the whole CPU. If the game doesn't run doesn’t matter what, developers still get blamed for it.
I think I’m going to wrap this one up. Thank you all for reading my concerns.
My main experience with MMOs is ESO. If you’ve followed the story of its main PvP content, Cyrodiil, you’ll know it’s a tragic story of a lost vision.
I really like the vision of AoC. As many say it’s probably the most inspiring MMO coming out. Its melodic open world PvP is a dream many have wished for. I believe in it’s ambitions. Where there’s a will there’s a way. But ESOs problems arise from so many dark corners it’s hard for me to start this post.
Let me explain what Cyrodiil is. It’s ESO massive tri alliance pvp arena. The estimates of the initial size of it’s server cap were 600v600v600 players per server. This was a gamemode that had keeps (little mini castles) and sieges that could be initiated at any moment.
Sounds fun? Well the whole thing now is in a sorry state of disrepair. Some say it can’t be fixed. Some say it's better abandoned. As a result of bad decision making it was abandoned, leaving it’s player base in disarray. Then carma struck and now the whole game is starting to act like Cyrodiil. It was made apparent with a near failed launch of their newest DLC, things over there aren’t right.
However plenty of lessons for AoC to learn for free can come out of that game.
One excuse that is always made is that ESO runs on an old outdated engine from 2007. There is they say that the licensing doesn't allow them to update the engine, but that sounds heavily like a management issue. Because how hard is it to get a new license?
One thing AoC can learn from that, is to maybe export textures and resources to Universal file formats that then can be added to future versions of the engine as plug-ins. Or at least starting meshes and templates so you're not left rebuilding the whole game if you want to upgrade the engine in say 2027. I’m no animator or graphic artist, but where my field of reference comes from, I know you can download 3D models from off the internet to be used in various programs. Is there a similar template approach that could be created and stored for the company’s artwork? A redundant back-up.
I said all that, but that’s not the big issue with ESO. One big issue with ESO, is that no one knows what the big issue with ESO is. Theories are all over the place. A problem is it doesn't have only one problem.
A big management mistake they made was not tackling problems with Cyrodiil. They ignored them. They let them stack up, releasing patch after patch of mistakes and extra glitches. One thing AoC can do is make a nominated team of skilled individuals from different departments that makes a list, looks for problems, and fixes them. It’s important this team is diverse so they can pinpoint where and which department the problem arries, maybe it is a cross department issue. They don’t need to have one specialist of the team do the job, but have the ability to send the problem back to the right department.
At ESO I really don’t know what they have going on but it doesn’t work. Unlike Intrepid, their company is ran by business types, who I think hear the intake of problems and look for the cheapest solution. Like for all its current server issues they’re trying to have the combat team fix it. Like they’re trying to have the people who just play with and change damage values fix server load… It’s a mess.
Now getting to the title of this thread.
If you ask a lot of veteran ESO players they’ll tell you performance has been on the decline in ESO.
A big problem is when they moved certain client combat values and interactions server side so they couldn’t be modified by modders and hackers to give themselves the edge in combat. Like people were flying, had unlimited ultimates(a super move that usually has like a minute +cooldown,) it was a circus. The bigger circus came after that as ESOs solution to it was a curse. Player positioning became a problem. They’d reduced the server population. Abilities wouldn’t fire correctly or be timed right. Performance took a big hit. And things got worse as the game progressed for reasons I don’t fully know, and I doubt anyone fully knows. But there are plenty of ideas why.
The big one is they kept adding input checks server side, so that nothing the client said was trusted. Something they obsess over despite all the negative consequences and alternatives. Intrepid probably has smart innovative people that can come up with their own solutions. Word of advice don’t copy ESO on this one.
One possible solution to this is to partner with or develop a cloud based gaming service and host the game on your partnered or owned gaming service. This gives you control over what can and can’t be installed on the cloud systems so you don’t have to worry about mods. I threw this idea out to ESO too and pointed them towards a company called Shadow. I’m not sure if they took interest, but might as well make it industry practice as it solves a lot of problems as I will continue to point out.
The way the concept works is you hardwire the server and hosting cloud PCs to a LAN. This eliminates pretty much all the latency in the games calculations. Network latency won’t cause problems where players get hit with abilities all at once, instead they may get small screen hiccups they can adjust for in gameplay.
Another problem hosting on high end cloud PCs will fix is client discrepancies and rendering issues. A problem ESO had, even though they have the minimum hardware specs, is that clients couldn’t handle the combat. ESO Devs still ignore certain quality of life problems with this issue like consecutive loadscreens, but the game used to crash constantly. It’s to the point Xbox and PS4 in all their gaming dedicated resources and splendor were considered toasters because they could hardly handle the game. On a regular PC the amount of system resources the game takes up is ridiculous. The sad part is even with a top of the line PC you still lag like the “toasters” did back in the day because it’s gotten that bad. ESO Devs choose to ignore this hardware issue, because players can try to improve it. Some credit they did eventually lower crashing, by lowering Fx and some stuff with the server. However, some including myself, point to their Anti-crash solutions as another source of ESO’s compounding server lag.
Lastly there’s the masses problem, tragedy of commons and so forth. Some say AoC is ready for it with an open beta. But what about the excessive client character and ability rendering that made ESO crash at every launch it’s ever had. Can the servers handle consecutive ability spams from so many players in a cluster. I hear the dungeon raid sizes are going to be huge. Will all the graphic aesthetics be too much for the average client?
Cloud based solutions again can be tapped to control this problem as you’ll know exactly what are on systems and what they can handle. A lot of problems some gamers have with games is even though their PC’s to spec, their Anti-virus taking up the whole CPU. If the game doesn't run doesn’t matter what, developers still get blamed for it.
I think I’m going to wrap this one up. Thank you all for reading my concerns.
A guy who came from ESO.
2
Comments
Except they already tested this out and are very confident on 250v250 battles, with thousands of NPCs to boot. They're even trying to push it to 500 vs 500. So I'm sure that Intrepid knows what they are doing.
As for cloud based solutions, I'm not really knowledgeable on the matter. I know what cloud gaming services are though, so if you're referring to that, I'm not really sure if that would be a good idea as it would mean that Intrepid would have to pay the cloud service company with whatever they make off of the sub fee. So basically they will either get less money to develop the game, or they will have to increase the sub fee, both of which can be harmful to the game.
I never had a chance to play ESO but I didn't hear very much about it while managing a gamestop so I assumed it had issues.
Thing is, live servers are different when dealing with remote connections. Chances are they are going to run into a problem with excessive messaging that can fill or corrupt the server memory depending on how it's designed.
What can happen is if a server or network service slows down for a fraction of a second several thousand to a million client requests can stack up overloading and spiking the server. This typically causes combat to slow down and freeze in a game for like 5secs, but can also disconnect people from the server.
One thing that came to the realization of ESO developers is that they can't test lag in a simulated environment. So now they've opted to conduct live testing.
As far as cloud based solutions go, I do expect it to be fiscally inconvenient, but how far do you want to enhance the vision. I hear Intrepid struggles with Net Code. Shadow has the user net code built. As a matter, I picked up from a video that Intrepid actually needed the my.com partnership for various reasons along these lines. What was their slice of the deal and can things simply be readjusted?
In business you can hire or partner. Looking at the movie industry there's plenty of studio partnerships for less complex projects then gaming. Star Wars from off the wiki was a 275 million dollar project, but it made billions. Who said there's a limit at two production companies. Maybe someone can do some clever talking and get shadow to buy my.com out of the deal. Getting the cloud companies interest in partnerships would more than likely keep the cost of the subscription down.
Hiring and contracting is possible but may stack a few dollars on the monthly sub fee. A deal can be struck at a low cost though.
They'll be doing castle siege testing next year. We will have to wait for that to see conclusive evidence on whether they can or cannot handle 250v250 battles.
I'm the same most of my MMO experience is with ESO. And everything you said sums up all my frustration with ESO and ZoS.
You nailed all the reasons why I just cant stand it anymore and why AoC spoke to me so well even from just its conceptual state.
I like you was there during the 600/600/600 days and the blatant cheating and meteor ping pong spamming days of 2014.
And your concerns are entirely valid. I think you'll be happy to know steven and the team are already working on the exact issues you raised with regards to server performance, destructible keeps (which in AoC will all be cyrodill essentially) and several hundred character animations and movements have been the extreme focus for the past few months that ties directly why they released a test cycle BR phase to see if their combat and server code structure was working correctly.
They identified a few problems that occured exactly with ESO with rendering and delays related to foundation code and while it delayed their progress, its very nice to hear they are working on the core design philosophies in tangent with their mechanical implementations.
Unlike with most beta/alpha phases (and this is what ESO did) is that they use alpha to build the core structure but dont actually stress test the systems to see if it's working correctly with the numbers in design.
Another issue (that you mentioned directly) is that ESO made a huge mistake in how they handled cheat engine issues by moving all their calculations to their main servers from their client servers. This not only did not stop cheating, it increased the strain on the server and codes to the limit because their engine wasnt designed with that in mind with all the other parts of the game. This was a major reason we saw continued decline in performance each patch, but wasnt the only reason either.
I'm glad you made this post as it is a great review for the coders/developers at Intrepid to read over and see if they can compare any of it with some of their own internal designs.
@StevenSharif
-Most of bgs functions ok (better than cyro at least)
-The quest designs were all well done, even the small side quests that all had unique dialouge and npc animations, as well as felt fun to read what was actually you were doing in regards to area. Vvardenfell really nailed this part.
-The dungeons were well designed and mechanics usually good.
- The art team and the people designing/coding the world are A1 without a doubt.
- Voice actors were amazing.
They had good combat up until this year with AoE's placing CD's on everything and implementing changes to unique classes and skills that dont work with how the games combat and mechanics work in full tangent. Again another source of massive frustration that I hope wont become an issue here with AoC.
However I have greater faith here because they are sticking with a more traditional design philosophy that some classes will do better against others and
vice versa. But the difference wont be unattainable since the 8 classes here will be able to have 64 morphs to their main class from one of the other 8 classes.
They say its 64 classes, but thats not true really. Its 8 classes with 64 other class skill "morphs". Think like esos 3 treees per class but they all have skill morphs. The other class are your skill morphs instead of a magic/stamina varient of each
IS has devs from a game that uses thousands of players in PVP and they are very confident that they can 250 vs. 250 (it was discussed in one of the recent monthly videos. I don't remember the name of the game). I don't see a reason not to give them some time and look at criticizing their 250 vs 250 once they show us what they've got. Castle siege testing (250 vs. 250) was going to be before Alpha one, but has been moved to after Alpha 1 due to widespread player requests to see a product.
IS is no longer affiliated with my.com. Source: https://forums.ashesofcreation.com/discussion/46125/it-is-done-they-broke-their-chains-we-are-free#latest
Planetside devs yes. They hold the record for the largest pvp battles ever in an area that didnt lag out to hell.
Not only them but blizzard devs too who have the smoothest movements for combat (as far as lag and skill delay goes)
I dont know if they have any eso devs, I wouldnt personally without extensive screening to see what part they specialized in, but I openly admit eso did alot of things right with the main part of the world.
PvP cyrodill though has always been a mess though. Even when it was working good there were many bugs
Well the my.com split is excellent timing for this thread. I refer to Shadow because they're probably a more modest company. I don't follow them personally though.
They do have a page for developers: https://developers.shadow.tech/
If anyone reaches out to them for the Cloud based solutions don't expect them to know what your talking about. I haven't pitched it to them. A developer would have to do it. There's a section provided on the linked page to fill out a description. If anyone needs anymore information I'll be around.
But think about the trinity of credit names Intrepid, Shadow, and Unreal.
Not to say they are the only company that can do it though.