Really enjoyed the stream. It's nice to see the underlying tech and how it influences the way we play. Also helps to boost hope that it's not going to be a laggy disconnect-fest. Thanks!
[SERVER MESHING & REPLICATION LAYER BOTTLENECKS COMPARED WITH OTHER GAMES]
There have been other games in the past and in development that have had some sort of server meshing type tech. i.e. Dual Universe, Atlas and namely Star Citizen (Still in Development).
I would like to know from an Intrepid Developer's standpoint, how was the tested Replication Layer (which was found to be a needless bottleneck) compare to Star Citizen's upcoming implementation of a Replication Layer?
Of course, scale, engine and focus is different between both games. However, it seems there's also enough similarity to consider the pros/cons of a Replication Layer for each game.
I think it would be interesting to compare the two systems, in particular the Replication Layer and services so as to understand challenges in both systems.
Can any Intrepid Developer / Network Guru pipe in to help elaborate or even offer some theory and speculation(with disclaimer) on this topic further?
Star Citizen's replication layer is a suite of scalable microservices that handles inter-server replication. Basically an instance of the microservice suite handles replication between a specific number of servers and can interact with other instances of the replication layer when needing to replicate data to servers outside that instance's managed servers (like when traveling from one star system to another for example).
They will also have the ability for large ships (and theoretically even small, single seat ships but that will likely never actually happen) to be their own servers so they need to be able handle authority for a server that will be dynamically changing what other servers it shares boundaries with so being able to pass that between replication layer instances instead of updating which servers it shares boundaries with directly is likely more efficient. This isn't something Ashes needs so it's probably just unnecessary added complexity in their case to have a dedicated replication layer
Right, but it will be interesting to see how SC avoids any bottleneck (if any) with their replication system.
Static server meshing will come in first, but eventually the goal is Dynamic server meshing.
I'm a bit concerned about what type of redundancy will be built in for the Replication Layer in SC, incase replication itself goes down.
But I'll stop here. This isn't an SC forum. Just hoping to learn the pros/cons of both Ashes and SC's implementation of this tech.
If replication somehow goes down entirely in SC (it should never completely go down as it's a distributed suite of microservices) then the affected clients will temporarily freeze until new instances of all of the services are started. If a specific microservice crashes then it will handle that almost exactly the same way that AoC will handle one of their microservices crashing, that specific system just won't work for the affected players until a new instance is started but the rest of the game will be unaffected.
As for redundancy, they have a scalable graph database that handles all of the persistence for the game that keeps a real time image of the game world and their replication layer services and a separate "long term" database that is written to regularly, which I'm assuming is backed up regularly as well. The chances of everything crashing are slim to none but in case the replication layer crashes, the graph database will have an almost current state to restore to once the new replication services are started, and if both of those somehow crash then there is still going to be a recent world state to restore from their long term database after the graph DB and replication layer services are restarted.
The main con for SC's approach is the overall complexity stemming from needing to be able to not only scale both out and up but to also dynamically "move" the location that a server is responsible for between different server boundaries. The main pro is that with how decoupled all of their services are designed to be it should be very easy and quick to recover from most failures with little risk of losing any data.
AoC should technically have marginally less latency since the servers are communicating directly and the main con I can see for AoC's approach, outside of potential performance issues caused by Unreal not being designed for this, is that since the servers handle replication directly then if something goes wrong with replication (data corruption, dropped packets, etc.) then the servers that are involved with that specific replication task could run into a deadlock or crash if those cases aren't handled properly or if there's some unusual edge case that was missed (and yes, there will be missed edge cases in something as complex as this). And also even though they rewrote a lot of Unreal's replication graph code and made it multithreaded, there will be performance issues after going live that they couldn't foresee with internal tests, even stress testing with bots.
I loved the deep dive into the nitty gritty aspects. The team members were very well spoken and clearly know their project well.
Elder Scrolls Online does a similar method for netcode especially in open world PvP. One major issue they encountered is Damage over Time skills. When thousands of players are applying 100s of DoTs across a choke point you start getting astronomical damage calculations going on.
They've tried to bandaid this by changing most DoTs to calculate and "hit" every 2 seconds instead of every second.
What I'd suggest exploring to mitigate this issue from the start is every DoT effect added to a player (even from the same skill) have a damage tick rate that isn't uniform.
One way you could do this is have a varied assignment style.
DoT 1 applied every second at 1x damage
DoT 2 applied every 0.3 seconds at 0.3x damage
DoT 3 applies every 0.7 seconds at 0.7x damage
This could very minimally impact damage and you'd want someone mathematically inclined to find rates that don't overlap often or at all.
This DoT assignment style would break up damage calculations across different ticks and alleviate the load of 100s of players applying 100s of DoT instances each.
1 DoT calculation is basically free processing-wise. 100 players applying 2 or 3 AoE DoTs to 100s of players becomes insane (100 x 2 DoTs x 100 players in a chokepoint = 20,000 damage calculations per tick)
Although expensive I'm sure, I am curious that with the current system scaling horizontally within the distributed system which overarching server maintains the final authoritative state? Is it a microservice we didn't see in the presentation?
If you have several authoritative servers meshed together, who determines the final authority? I was thinking a higher level authority server that maybe only writes every 15 min or so to manage the persistent state of everything without large performance overhead.
Pretty sure there isn't one authoritative server, what they mean when they talk about authority is that every server is authoritative for its own allocated map/grid/data area.
And the servers retrieving replication data / proxies from it would be the servers bordering that server in the grid (perhaps some data even gets replicated beyond bordering servers for map changes that can be seen from very far away; but that's probably not necessary, because that doesn't need to be represented with active actors.)
There is no hierarchy of authority, just a binary "authority or replication" over everything that's being calculated and represented. All data a server is replicating, it has no authority over, it just replicates it from the sole authoritative server that does control it, so the replicating server can take over the responsibility of passing that data on to its own users that it's responsible for, and the server with authority over that zone doesn't have to connect to players looking/interacting into it from its borders.
If players fire arrows across borders, they probably stay in the authority of the original server, but whether they hit something across the border gets decided by the server with authority over the location it lands in.
More definitively, players passing through from one server to the other belong to the authority of their original server as long as they're within its borders, and start out as proxies in the other server, but start to be controlled authoritatively by the other server once they've passed into its grid section. [See in the section of the VOD where Hunter explains promotion.]
That much all seemed pretty definitive, perhaps with a few exceptions in the details, like buffer space before a player really gets committed to the server they stepped into, in order to avoid repeated back-and-forth.
Apologies if this is a silly question: in the video, the servers are shown in a grid, and a decent amount of time is spent discussing how to reduce artifacts amongst neighboring servers. Yet perplexingly, a perfectly aligned square grid was shown for the server "tiling geometry". This choice wasn't discussed in the video, and intuitively it doesn't make sense. There are many tilings with fewer tiles at any vertex, including one where you restrict yourself to squares; see attached example.
Why would you intentionally create points where 4 servers have to communicate proxy information to one another when you can just as easily tile the game grid with servers that never come to more than 3 at a point? Is there a technical reason for this? What about servers of different sizes for that matter?
This almost looks like its considered during the "Dynamic Gridding" portion of the video, but I don't see any reason why you couldn't do Dynamic Gridding with the "brick-layering" geometry shown above.
If you really wanted to, you could create long thin servers at different game "latitudes" that span the entire globe, reducing the number of adjacent servers at any point to 2. (But that involves different server areas which may not be supportable given the game's size)
During the Spatial Grid Bucket, the number of servers is directly referenced as the reason for a performance boost, using 3 adjacent squares looking towards a corner. Again, if you glide half of the servers (as above), you would always be using 2 adjacent squares.
1
MorashtakMember, Braver of Worlds, Kickstarter, Alpha One
[*] How do you feel about the Server Meshing Technology Preview?
[*] What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
[*] Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
[*] Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
[/list]
The preview was well planned out and the presentation made it clear what to expect when all the pieces come together.
Neither excited or concerned at this time - Will reserve detailed comments until we see it implemented and working properly.
Have only played one other game with similar system - Star Citizen. There is is nice to know that there will not be large server loads in my little slice of the galaxy even if there are numerous large fleet battles constantly erupting on the other side of the planet - the server will only send my client the data it needs to keep my gaming experience as smooth as possible. Expecting the same from IS' server tech.
0
ShadowVenMember, Braver of Worlds, Kickstarter, Alpha One
I really enjoyed this Livestream.
The team did an exceptional job on presenting it as clear as possible.
I find it really impressive that Intrepid has this quality of innovation in its ranks. Great job overall and I look forward on the continue working of it to get to that point of optimization target for 500v500!
But if it's not ppt like presentation will be better.
I prefer to see tech demo in game that going in real time even it's not working perfectly, what I want to know is what you aim for and how it looks right now like before after comparison and what the hard parts are, just share what you can share with players.
We’d like your feedback on the Server Meshing Technology Preview shown during the June 2024 Development Update Livestream.
To help guide this conversation, here are a few thought starters you can choose from:
How do you feel about the Server Meshing Technology Preview?
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
Please don’t feel limited by the thought starters above. Feel free to share anything you’d like about Ashes of Creation’s Server Meshing Technology Preview shown during the June Development Update
We’ll be compiling a report for the design team on Friday, July 19, 2024, so please try to get your feedback into this thread by then!
Everyone here at Intrepid Studios looks forward to reading all the feedback you have to share![/quote
Im a nerd so it was interesting, could have been glossed over in 5 minutes (which is not to in any way underplay the amazing job) and shown us more of the game which all of us want to see.
Amazing work. I'm always down to hear more about the technical marvels that are required for MMOs to function nicely. I've still got a couple questions about dynamic gridding, like how does a server split and pass along it's authority seamlessly to the new servers? Do they have to duplicate all the state/processes and get running in lock-step first, like a baton-pass in a relay race?
In any case, knowing how much work it takes to even get 100 people on screen at once makes it a lot easier to understand and forgive little bugs or network hitches.
The presentation was great and easy to understand. Looking forward to seeing what kind of player densities the server meshing tech can handle in Alpha 2.
Star Citizen's live build isn't using server meshing so nobody only playing during those free weekends has ever experienced it. They've been doing short tests over the last couple of months but not during free trial periods. The tests have been pretty stable and performance has been better in general than their live servers.
Sounds like Patrick68794 is a SC dev trying to promote his system....
Anyway, as a non-tech guy (Construction) I very much enjoyed the presentation, and deep dive into how this works, and how/why it has been an issue in the past.
All I knew was...zoning...it was just a thing. Your running along the road, then your frozen mid-stride, then continue on
There was a lot of concern this deep dive would turn off guys like me, but honestly it was cool seeing the behind the scenes, and how they are working on preventing lag/bottlenecking.
All good, lots of love...except to haters > Patrick/SC clone
Star Citizen's live build isn't using server meshing so nobody only playing during those free weekends has ever experienced it. They've been doing short tests over the last couple of months but not during free trial periods. The tests have been pretty stable and performance has been better in general than their live servers.
Sounds like Patrick68794 is a SC dev trying to promote his system....
Anyway, as a non-tech guy (Construction) I very much enjoyed the presentation, and deep dive into how this works, and how/why it has been an issue in the past.
All I knew was...zoning...it was just a thing. Your running along the road, then your frozen mid-stride, then continue on
There was a lot of concern this deep dive would turn off guys like me, but honestly it was cool seeing the behind the scenes, and how they are working on preventing lag/bottlenecking.
All good, lots of love...except to haters > Patrick/SC clone
Later
Lol so pointing out that server meshing isn't in SC yet makes me an "SC clone" and means I somehow hate AoC I guess? What a dumb conclusion to jump to based on that post, though I guess maybe I shouldn't be surprised
If you actually understood what you were reading you'd see that if anything, my posts are just showing confirmation that this kind of tech can work in general and that they're an endorsement for Intrepid's claims. I know understanding what you're reading before replying is tough for some people but give it a shot next time
If you actually understood what you were reading you'd see that if anything, my posts are just showing confirmation that this kind of tech can work in general and that they're an endorsement for Intrepid's claims. I know understanding what you're reading before replying is tough for some people but give it a shot next time
Your posts try to say that Intrepid's can work...but not as good as SC. You are all about promoting SC in all your trash talk. Maybe you should go play with your little friends in a corner and stop trying to talk down to ppl that call you out.
You are clearly a little troll, and to say otherwise is to belittle even your little brain.
Next....
If you actually understood what you were reading you'd see that if anything, my posts are just showing confirmation that this kind of tech can work in general and that they're an endorsement for Intrepid's claims. I know understanding what you're reading before replying is tough for some people but give it a shot next time
Your posts try to say that Intrepid's can work...but not as good as SC. You are all about promoting SC in all your trash talk. Maybe you should go play with your little friends in a corner and stop trying to talk down to ppl that call you out.
You are clearly a little troll, and to say otherwise is to belittle even your little brain.
Next....
They do not. Learn to understand what you're reading before making a fool of yourself. Not once have I said or implied that the meshing tech in SC is "better" nor have I "promoted SC" in any of my posts lol all I've done is point out the differences and even provided a few potential pros and cons that stand out to me as being likely (unlike you, I do have experience and knowledge in this area). Neither is better than the other from a technological standpoint and there's no way of knowing if either approach is actually going to work better than the other. They're both groundbreaking new tech and they will both have their teething problems, same as any other cutting edge technology. Now if you could stop acting like a child and pretending to be a keyboard warrior, why not discuss the actual showcase and the technology?
How do you feel about the Server Meshing Technology Preview?
This could be the single most advancing technology for Unreal Engine which if done absolutely flawlessly, no one will ever know how hard this was, what it took, only that we can enjoy the game at scale we've not been able to do in prior games. This could change how any sandbox/mmo style games are built infrastructurally. Hats off to the engineers and developers who I'm sure spent countless hours with "it can't be done" staring them in the face.
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I appreciate it as I'm in tech and understand the amount of effort that engineers and developers spend. This is one of those moments where you all just made the most introverted people who probably don't "toot their own horn" feel honored and proud of the work they put into this feature/infrastructure. Seriously, this is very impressive and you all should continue do these style updates.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I do think people might try to overload the tech, and organize massive "blitz" runs, only because you all showed how the system will add more servers and sub-divide zones, I'm sure just because, people are going to test out at scale, the system.
While you all showed this off and it's really awesome, what happens if 100, 1000, 10,000, 100,000, 1,000,000 people run into one area. Honestly major towns, hubs, in-game are going to be this, will this all still handle that load?
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
I don't love server shards/instance, this is just a pain to transfer around to find your friend who's in the same place. This at times can really suck if you have to constantly manage instances, transfer around, etc.
FF14 probably has the best transferring from not only same "worlds" but also across data centers, these take about 5-10 seconds between worlds, and about 20-30 seconds between data centers. I don't love this - but at least it's doable.
How do you feel about the Server Meshing Technology Preview?
This could be the single most advancing technology for Unreal Engine which if done absolutely flawlessly, no one will ever know how hard this was, what it took, only that we can enjoy the game at scale we've not been able to do in prior games. This could change how any sandbox/mmo style games are built infrastructurally. Hats off to the engineers and developers who I'm sure spent countless hours with "it can't be done" staring them in the face.
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I appreciate it as I'm in tech and understand the amount of effort that engineers and developers spend. This is one of those moments where you all just made the most introverted people who probably don't "toot their own horn" feel honored and proud of the work they put into this feature/infrastructure. Seriously, this is very impressive and you all should continue do these style updates.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I do think people might try to overload the tech, and organize massive "blitz" runs, only because you all showed how the system will add more servers and sub-divide zones, I'm sure just because, people are going to test out at scale, the system.
While you all showed this off and it's really awesome, what happens if 100, 1000, 10,000, 100,000, 1,000,000 people run into one area. Honestly major towns, hubs, in-game are going to be this, will this all still handle that load?
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
I don't love server shards/instance, this is just a pain to transfer around to find your friend who's in the same place. This at times can really suck if you have to constantly manage instances, transfer around, etc.
FF14 probably has the best transferring from not only same "worlds" but also across data centers, these take about 5-10 seconds between worlds, and about 20-30 seconds between data centers. I don't love this - but at least it's doable.
I wholeheartedly agree with your first paragraph, I would love to see more studios that are making groundbreaking improvements to existing off the shelf tech allow that to be integrated into future commercial iterations so that it can be built upon. Lumen is likely the result of The Coalition doing this with their signed distance field solution for ray traced shadows (this is the exact same approach Lumen uses by default for global illumination and shadows) in Gears 5 that was so efficient it ran on the original Xbox One. The Coalition worked directly with Epic on UE5 and on some of the initial tech demos showing off Lumen and other new tech like Nanite.
Multithreading the replication graph would be huge for all of the developers that are trying to build complex online games in Unreal. If anything makes it back to Epic from AoC's development I think it should be this. I can also understand Intrepid wanting to keep the actual meshing tech they've developed in-house and proprietary though as it gives them a unique edge compared to other traditional MMOs, but also shows other studios that it is a viable solution. One interesting thing I noticed is that Intrepid spent a good amount of time talking about how they came up with a solution to make actors in the world "dormant" when it came to the replication graph update but this functionality already exists in Unreal (and is actively used already in games) so I'm curious if Intrepid built their own approach that works differently.
As of now at least they're not actually increasing player counts for a "realm" (equivalent to FFXIV's worlds) so there will still be an upper bound (I believe around 10k) on the number of clients that an entire mesh will need to handle. I'm pretty sure they already had server meshing in mind when they initially announced the player count per realm honestly. What I'm curious to see is how they'll handle that at the client level because no current PC is going to be able to handle rendering 10k other player controlled characters with the level of fidelity that is on display in AoC with the level of performance needed for action combat.
0
persianMember, Braver of Worlds, Kickstarter, Alpha One
Insanely exciting stuff for the future of Ashes and MMORPGs in general, looking forward to further progress updates and news on this!
The technology sounds good and the livestream was great, however I think what really counts will be the launch-day. I've seen and played so many MMOs in the last decades and noticed that many people leave a game quite quickly these days if server stability issues are showing. Especially in a MMO where you join to play with your friends/guilds, this is impacting the mood quite a lot.
So in that regard I just hope that the developers of Ashes have counted that in and are not surprised (like all the other MMOs) if their servers do not work the way they intend even after x amount of Alphas and Betas. Hope precautions are on, so we'll see. I always hope the best until the worst happen ^^
How do you feel about the Server Meshing Technology Preview?
Well put together
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
The tech side of MMO's is interesting as it lets us get a feeling if we're getting an instance based mmo or maybe small population "channels" type of mmo etc.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I'm concerned that the tech might not work as well when we introduced latency, disconnects, chat spamming, skill spamming etc. to the mix.
Firstly, I'm incredibly impressed with the strides Intrepid Studios is making with this technology. The preview demonstrated a sophisticated approach to server networking that promises to revolutionize the MMO experience in Ashes of Creation. The seamless integration of multiple servers to create a unified, dynamic world is a game-changer. It not only enhances immersion but also opens up vast possibilities for gameplay, community interaction, and world events.
The presentation itself was enlightening, providing valuable insights into the complexities of server infrastructure and networking. It's great to see the transparency in sharing these behind-the-scenes details, as it reinforces trust and excitement among the community.
Looking ahead, I'm particularly excited about the potential for large-scale battles and events with this technology. The ability to seamlessly transition between different server instances without disruptions is something I've been eagerly anticipating in MMOs. This could truly redefine what's possible in terms of player-driven narratives and emergent gameplay.
In terms of similar systems in other games, I've seen attempts at similar technologies, but none have been as ambitious or promising as what I've seen from Intrepid Studios. The attention to detail and commitment to making server meshing a cornerstone of the next-generation MMO experience is commendable.
Overall, I believe that integrating Server Meshing Technology into Ashes of Creation is not just a step forward, but a leap towards defining the future of MMOs. I look forward to seeing how this technology evolves and enhances our adventures in Verra.
Thank you for pushing the boundaries and striving for excellence. I'm excited to be a part of this journey with you all.
The presentation was very informative. I was impressed by the effort made to educate those of us with less knowledge of the machinery under the hood in the game. I encourage more of this in the future. It can only help us as we provide feedback during upcoming testing phases. It will also make future updates on game status easier with a more informed group of player/testers.
We’d like your feedback on the Server Meshing Technology Preview shown during the June 2024 Development Update Livestream.
To help guide this conversation, here are a few thought starters you can choose from:
How do you feel about the Server Meshing Technology Preview?
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
Please don’t feel limited by the thought starters above. Feel free to share anything you’d like about Ashes of Creation’s Server Meshing Technology Preview shown during the June Development Update
We’ll be compiling a report for the design team on Friday, July 19, 2024, so please try to get your feedback into this thread by then!
Everyone here at Intrepid Studios looks forward to reading all the feedback you have to share!
I feel like I'm in a "show me" phase. I want to believe everything Intrepid is saying is true, that they've cracked the code on how to truly have a seamless, open world experience where we don't experience crashes, I just feel like in order to 100% believe it, I'm going to have to get my hands on A2 and actually get into Verra to fully believe it.
I'll be honest, I didn't care about alot of it. To me, it's Intrepid's job to make the vision a reality, so if they create something, I do my best to trust that it works. Again though, not fully believing it until A2.
I'm curious how instances factor into all of this, specifically the minimal raid instances.
Nope, I've really only ever played SWTOR seriously, and that game is a joke compared to AoC.
I loved the stream, it was technical but not too technical. It all sounds very exciting. Considering it was a presentation it is easy to forget that we only hear one side of things so its hard to know how accurate all the information actually are as we have to take your word for it.
My biggest concern was that the mini services sound a lot like a miniature, split up and virtual version, of the distributor that was criticized so heavily at the beginning of the stream. I would like to know why this is so different.
As the images shown looked great and were accurate they also portrait it a lot in aoc favor.
Before it was a bunch of servers connected to the distributor which in turn was connected to the WAN.
Now we have a bunch of servers connected to the WAN and also connected to a bunch of distributors (one for each service) which in turn are connected to the other servers (LAN).
I get the advantages we get out of this but there is still the one point failure in a fully interconnected system.
If on my server the crafting system is down, it's due to one distributor server not running. Why can the distributing server of my neighbor game server not temporary take over this task?
Isn't that one of the advantages we have from all the servers being interconnected?
It would probably be slower on both servers now, but at least crafting is still working.
It's all virtual anyway, I assume, so why not?
0
MissionCreepMember, Braver of Worlds, Kickstarter, Alpha One
I've watched every single one of these going back 4 years or whatever, and honestly I turned this one off.
This Power Point presentation gave me "This meeting could have been an email" vibes. I know that what gets people excited to buy or play a game is gameplay, core mechanics, etc. Of course people will be happy if you crack the server meshing thing and I do appreciate that it's a bear to figure out and all the work that's gone into it.
I just don't need to know how the sausage is made.
Concern, in other games players would change their network speed. This cause issues with active play. I.e. combat for example. Player is in front of you then disappeared, then behind you. The system was trying to sync experience on the client and this caused issues such as the example.
How are you protecting against these types of exploiters? Or maybe not how but are you testing for it?
Thanks
Comments
As for redundancy, they have a scalable graph database that handles all of the persistence for the game that keeps a real time image of the game world and their replication layer services and a separate "long term" database that is written to regularly, which I'm assuming is backed up regularly as well. The chances of everything crashing are slim to none but in case the replication layer crashes, the graph database will have an almost current state to restore to once the new replication services are started, and if both of those somehow crash then there is still going to be a recent world state to restore from their long term database after the graph DB and replication layer services are restarted.
The main con for SC's approach is the overall complexity stemming from needing to be able to not only scale both out and up but to also dynamically "move" the location that a server is responsible for between different server boundaries. The main pro is that with how decoupled all of their services are designed to be it should be very easy and quick to recover from most failures with little risk of losing any data.
AoC should technically have marginally less latency since the servers are communicating directly and the main con I can see for AoC's approach, outside of potential performance issues caused by Unreal not being designed for this, is that since the servers handle replication directly then if something goes wrong with replication (data corruption, dropped packets, etc.) then the servers that are involved with that specific replication task could run into a deadlock or crash if those cases aren't handled properly or if there's some unusual edge case that was missed (and yes, there will be missed edge cases in something as complex as this). And also even though they rewrote a lot of Unreal's replication graph code and made it multithreaded, there will be performance issues after going live that they couldn't foresee with internal tests, even stress testing with bots.
Elder Scrolls Online does a similar method for netcode especially in open world PvP. One major issue they encountered is Damage over Time skills. When thousands of players are applying 100s of DoTs across a choke point you start getting astronomical damage calculations going on.
They've tried to bandaid this by changing most DoTs to calculate and "hit" every 2 seconds instead of every second.
What I'd suggest exploring to mitigate this issue from the start is every DoT effect added to a player (even from the same skill) have a damage tick rate that isn't uniform.
One way you could do this is have a varied assignment style.
DoT 1 applied every second at 1x damage
DoT 2 applied every 0.3 seconds at 0.3x damage
DoT 3 applies every 0.7 seconds at 0.7x damage
This could very minimally impact damage and you'd want someone mathematically inclined to find rates that don't overlap often or at all.
This DoT assignment style would break up damage calculations across different ticks and alleviate the load of 100s of players applying 100s of DoT instances each.
1 DoT calculation is basically free processing-wise. 100 players applying 2 or 3 AoE DoTs to 100s of players becomes insane (100 x 2 DoTs x 100 players in a chokepoint = 20,000 damage calculations per tick)
nice
Why would you intentionally create points where 4 servers have to communicate proxy information to one another when you can just as easily tile the game grid with servers that never come to more than 3 at a point? Is there a technical reason for this? What about servers of different sizes for that matter?
This almost looks like its considered during the "Dynamic Gridding" portion of the video, but I don't see any reason why you couldn't do Dynamic Gridding with the "brick-layering" geometry shown above.
If you really wanted to, you could create long thin servers at different game "latitudes" that span the entire globe, reducing the number of adjacent servers at any point to 2. (But that involves different server areas which may not be supportable given the game's size)
During the Spatial Grid Bucket, the number of servers is directly referenced as the reason for a performance boost, using 3 adjacent squares looking towards a corner. Again, if you glide half of the servers (as above), you would always be using 2 adjacent squares.
The preview was well planned out and the presentation made it clear what to expect when all the pieces come together.
Neither excited or concerned at this time - Will reserve detailed comments until we see it implemented and working properly.
Have only played one other game with similar system - Star Citizen. There is is nice to know that there will not be large server loads in my little slice of the galaxy even if there are numerous large fleet battles constantly erupting on the other side of the planet - the server will only send my client the data it needs to keep my gaming experience as smooth as possible. Expecting the same from IS' server tech.
The team did an exceptional job on presenting it as clear as possible.
I find it really impressive that Intrepid has this quality of innovation in its ranks. Great job overall and I look forward on the continue working of it to get to that point of optimization target for 500v500!
Keep up the outstanding work Intrepid =^_^=
But if it's not ppt like presentation will be better.
I prefer to see tech demo in game that going in real time even it's not working perfectly, what I want to know is what you aim for and how it looks right now like before after comparison and what the hard parts are, just share what you can share with players.
I couldn't comment one way or another about what the best server solution would be, I'm just thankful you guys are willing to show us stuff like this.
It's an interesting and refreshing change of pace every once in a while.
In any case, knowing how much work it takes to even get 100 people on screen at once makes it a lot easier to understand and forgive little bugs or network hitches.
Sounds like Patrick68794 is a SC dev trying to promote his system....
Anyway, as a non-tech guy (Construction) I very much enjoyed the presentation, and deep dive into how this works, and how/why it has been an issue in the past.
All I knew was...zoning...it was just a thing. Your running along the road, then your frozen mid-stride, then continue on
There was a lot of concern this deep dive would turn off guys like me, but honestly it was cool seeing the behind the scenes, and how they are working on preventing lag/bottlenecking.
All good, lots of love...except to haters > Patrick/SC clone
Later
Lol so pointing out that server meshing isn't in SC yet makes me an "SC clone" and means I somehow hate AoC I guess? What a dumb conclusion to jump to based on that post, though I guess maybe I shouldn't be surprised
If you actually understood what you were reading you'd see that if anything, my posts are just showing confirmation that this kind of tech can work in general and that they're an endorsement for Intrepid's claims. I know understanding what you're reading before replying is tough for some people but give it a shot next time
Your posts try to say that Intrepid's can work...but not as good as SC. You are all about promoting SC in all your trash talk. Maybe you should go play with your little friends in a corner and stop trying to talk down to ppl that call you out.
You are clearly a little troll, and to say otherwise is to belittle even your little brain.
Next....
They do not. Learn to understand what you're reading before making a fool of yourself. Not once have I said or implied that the meshing tech in SC is "better" nor have I "promoted SC" in any of my posts lol all I've done is point out the differences and even provided a few potential pros and cons that stand out to me as being likely (unlike you, I do have experience and knowledge in this area). Neither is better than the other from a technological standpoint and there's no way of knowing if either approach is actually going to work better than the other. They're both groundbreaking new tech and they will both have their teething problems, same as any other cutting edge technology. Now if you could stop acting like a child and pretending to be a keyboard warrior, why not discuss the actual showcase and the technology?
This could be the single most advancing technology for Unreal Engine which if done absolutely flawlessly, no one will ever know how hard this was, what it took, only that we can enjoy the game at scale we've not been able to do in prior games. This could change how any sandbox/mmo style games are built infrastructurally. Hats off to the engineers and developers who I'm sure spent countless hours with "it can't be done" staring them in the face.
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I appreciate it as I'm in tech and understand the amount of effort that engineers and developers spend. This is one of those moments where you all just made the most introverted people who probably don't "toot their own horn" feel honored and proud of the work they put into this feature/infrastructure. Seriously, this is very impressive and you all should continue do these style updates.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I do think people might try to overload the tech, and organize massive "blitz" runs, only because you all showed how the system will add more servers and sub-divide zones, I'm sure just because, people are going to test out at scale, the system.
While you all showed this off and it's really awesome, what happens if 100, 1000, 10,000, 100,000, 1,000,000 people run into one area. Honestly major towns, hubs, in-game are going to be this, will this all still handle that load?
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
I don't love server shards/instance, this is just a pain to transfer around to find your friend who's in the same place. This at times can really suck if you have to constantly manage instances, transfer around, etc.
FF14 probably has the best transferring from not only same "worlds" but also across data centers, these take about 5-10 seconds between worlds, and about 20-30 seconds between data centers. I don't love this - but at least it's doable.
I wholeheartedly agree with your first paragraph, I would love to see more studios that are making groundbreaking improvements to existing off the shelf tech allow that to be integrated into future commercial iterations so that it can be built upon. Lumen is likely the result of The Coalition doing this with their signed distance field solution for ray traced shadows (this is the exact same approach Lumen uses by default for global illumination and shadows) in Gears 5 that was so efficient it ran on the original Xbox One. The Coalition worked directly with Epic on UE5 and on some of the initial tech demos showing off Lumen and other new tech like Nanite.
Multithreading the replication graph would be huge for all of the developers that are trying to build complex online games in Unreal. If anything makes it back to Epic from AoC's development I think it should be this. I can also understand Intrepid wanting to keep the actual meshing tech they've developed in-house and proprietary though as it gives them a unique edge compared to other traditional MMOs, but also shows other studios that it is a viable solution. One interesting thing I noticed is that Intrepid spent a good amount of time talking about how they came up with a solution to make actors in the world "dormant" when it came to the replication graph update but this functionality already exists in Unreal (and is actively used already in games) so I'm curious if Intrepid built their own approach that works differently.
As of now at least they're not actually increasing player counts for a "realm" (equivalent to FFXIV's worlds) so there will still be an upper bound (I believe around 10k) on the number of clients that an entire mesh will need to handle. I'm pretty sure they already had server meshing in mind when they initially announced the player count per realm honestly. What I'm curious to see is how they'll handle that at the client level because no current PC is going to be able to handle rendering 10k other player controlled characters with the level of fidelity that is on display in AoC with the level of performance needed for action combat.
PvX OCE/SEA Guild
Discord https://discord.gg/ea9NwHhMTc
https://www.amnestyguild.net/jointheguild
So in that regard I just hope that the developers of Ashes have counted that in and are not surprised (like all the other MMOs) if their servers do not work the way they intend even after x amount of Alphas and Betas. Hope precautions are on, so we'll see.
Good luck!
The tech side of MMO's is interesting as it lets us get a feeling if we're getting an instance based mmo or maybe small population "channels" type of mmo etc.
I'm concerned that the tech might not work as well when we introduced latency, disconnects, chat spamming, skill spamming etc. to the mix.
The presentation itself was enlightening, providing valuable insights into the complexities of server infrastructure and networking. It's great to see the transparency in sharing these behind-the-scenes details, as it reinforces trust and excitement among the community.
Looking ahead, I'm particularly excited about the potential for large-scale battles and events with this technology. The ability to seamlessly transition between different server instances without disruptions is something I've been eagerly anticipating in MMOs. This could truly redefine what's possible in terms of player-driven narratives and emergent gameplay.
In terms of similar systems in other games, I've seen attempts at similar technologies, but none have been as ambitious or promising as what I've seen from Intrepid Studios. The attention to detail and commitment to making server meshing a cornerstone of the next-generation MMO experience is commendable.
Overall, I believe that integrating Server Meshing Technology into Ashes of Creation is not just a step forward, but a leap towards defining the future of MMOs. I look forward to seeing how this technology evolves and enhances our adventures in Verra.
Thank you for pushing the boundaries and striving for excellence. I'm excited to be a part of this journey with you all.
I feel like I'm in a "show me" phase. I want to believe everything Intrepid is saying is true, that they've cracked the code on how to truly have a seamless, open world experience where we don't experience crashes, I just feel like in order to 100% believe it, I'm going to have to get my hands on A2 and actually get into Verra to fully believe it.
I'll be honest, I didn't care about alot of it. To me, it's Intrepid's job to make the vision a reality, so if they create something, I do my best to trust that it works. Again though, not fully believing it until A2.
I'm curious how instances factor into all of this, specifically the minimal raid instances.
Nope, I've really only ever played SWTOR seriously, and that game is a joke compared to AoC.
My biggest concern was that the mini services sound a lot like a miniature, split up and virtual version, of the distributor that was criticized so heavily at the beginning of the stream. I would like to know why this is so different.
As the images shown looked great and were accurate they also portrait it a lot in aoc favor.
Before it was a bunch of servers connected to the distributor which in turn was connected to the WAN.
Now we have a bunch of servers connected to the WAN and also connected to a bunch of distributors (one for each service) which in turn are connected to the other servers (LAN).
I get the advantages we get out of this but there is still the one point failure in a fully interconnected system.
If on my server the crafting system is down, it's due to one distributor server not running. Why can the distributing server of my neighbor game server not temporary take over this task?
Isn't that one of the advantages we have from all the servers being interconnected?
It would probably be slower on both servers now, but at least crafting is still working.
It's all virtual anyway, I assume, so why not?
This Power Point presentation gave me "This meeting could have been an email" vibes. I know that what gets people excited to buy or play a game is gameplay, core mechanics, etc. Of course people will be happy if you crack the server meshing thing and I do appreciate that it's a bear to figure out and all the work that's gone into it.
I just don't need to know how the sausage is made.
How are you protecting against these types of exploiters? Or maybe not how but are you testing for it?
Thanks