What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I know this is easy to say shouting from the sidelines, but I always thought something like dynamic gridding and inter-server replication would be the obvious solution to make server meshing work and allow MMOs to avoid layers, without failing when unusual situations happen, like a thousand players walking into a city.
"Just" make the servers pass players from one server to another when they enter different parts of the world, define those parts by server load needs rather than purely by map location, and add a lot of redundancy that's covered by multiple servers, so players never experience connectivity issues when they transition from one server to another.
Obviousy the reduncancy is where the technology needs to be really good, so you neither risk connectivity failures, nor allow mismatched data to occur (always have only one single server authoritatively dictate the changes for each grid section, the rest only copy), nor copy everything with so much redundancy that the additional servers just do all the work 5 times over, and you get barely any capacity increase from server meshing in the first place.
Which is all to say, without figuring out the precise solutions, "my" idea isn't worth anything.
But I'm just surprised there haven't been more studios that committed to these methods. Considering how rich some of these studios are, it's kinda crazy that they felt the need to rely on layers, when it kills so much of the natural community interaction, and community is what defines MMOs in the first place...
So I guess thank you for actually doing the work to create the logical, thorough solution, rather than going the path of least resistance. It's what the world needs lot more of.
I guess I do also have some questions/suggestions.
1) In the explanation of microservices (which I'm sure are an important tool either way) the example given is something like players trading across the map or something.
Would this really be so difficult to connect without microservices?
Hopefully, player inventory and gold are always saved in persistent databases, not just on the current server the player is on, right?
2) With the explanation of performance increases through feats like multithreading, i the performance still struggles, have you considered turning off certain game features when more than 1000 players are in a condensed area? Refreshing certain data less often, and assuming certain defaults?
Simplifying cosmetics and character information down to a tenth of the detail (perhaps let the client retain the information they already have, so players notice the difference less)? Simplifying ability casts by representing fewer augments for the player (but still getting correct battle calculations on the server itself?)
I'm not sure how much any of this would even be useful with the engine network code already avoiding unnecessary engine queries, but perhaps that's something to look into, if the limits of what's possible are reached; just as a fallback to make it through the most extreme individual server load scenarios.
3) When you talk about promotion, you list a loooot of components that every actor has to have transferred. Why does there need to be something like an "interactive" component on an actor that crosses a server? Shouldn't that component be intrinsic to all actors of the type [caravan] or [mount] (or whatever the actor with that component happens to be) and not have to be data that is communicated by the servers, but simply be assumed to be an attribute of the asset?
Has anyone else noticed the tendency for users with animal icons to swoop in and comment on the OP without acknowledging any of the discussion that has been going on so far?
How do you feel about the Server Meshing Technology Preview?
These are very ambitious as you described. Would love such things to work but promising and showcasing everything will only increase the amount of hype and expectations. Seeing such pioneering technologies also made me think how bad can things get during or after launch. But I am also happy that someone is still pushing the boundaries because we would still be playing tap target and topdown 2D MMOs if nobody attempted any of these. It is also a smart business move to patent and sell/rent these services via IntrepidNet if it all works out in the end.
As a gamer though, what I'm so happy to hear was more about being able to monitor every node and player states so that you can more actively participate and even counter the botting issue that plague mmos. I dont know if that is possible to solve bots just by monitoring some stats BUT that sounds much more controllable and profitable than having to hire hundreds of Game Masters. (Even though I love Game Masters)
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
As a game dev who loves multiplayer games (especially mmos), It was quite advanced even for me that I had to rewatch entire thing and pause. It is such knowledge I coveted from multiple GDCs and tried to imitate myself for my game jam attempts using some commercial multiplayer libraries. Honestly, one of the reasons I sought early access to AoC was just to get to see behind the scenes for information like these hoping it might help me become a better game designer or developer. I didn't expect to see them made public like this.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
The graphs and visuals helped a ton for me to understand and visualize what I heard so it was a huge plus. I was concerned that it would somehow break the game flow or create exploitable interactions between servers while players are moving across the map but visibly seeing ingame footage of the world borders/colliders helped visualize and you also underlined something which I recall as "conditions" to migrate/promote between the two servers while moving which are probably checked to avoid such stuff from happening.
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
I have not seen anything similar but I played many games that desired such systems such as Ultima Online, Lineage 2 and Black Desert Online because of the open world PvP and the karma/corruption systems they implemented. I am a huge fan of entire playerbase being on one realm instead of seperate channels/instances that are swappable at will. Such dynamic systems sound really much more fun presuming these instances/servers the realm consists of are not manually switchable and much more performant and happening all in the background without players noticing. If it is as good as it looks on the videos, it is almost indistinguishable.
P.S. I want to mention that as a person who have been in production and publishing of a mere few video games, I have always been trying to recall and make research about game mechanics from games that I have played to try and recreate (and maybe even improve upon) them but almost most companies don't care to share and/or don't even want to share their knowledge because they want to make more profit off it before such technologies become mainstream. I have only seen something similar to these in some GDCs (Rocket League's networking systems come to mind) and Path of Exile 2's new shader technology reveals recently. I love the upfront and public approach to game development making consumers also become more aware of difficulties of developing games and helping other game devs also see behind the curtain. This also feels like a good "investment" for future game developers to not struggle over the same hurdles people struggle with even if it is not entire codebase but visual depiction or explanation showing that it is possible and how it is achieved.
Sorry for the long post, thanks for reading.
Feel free to correct me if I'm mistaken at any part.
I play to win, not just for fun
Because I enjoy giving it all I got.
truenoirMember, Braver of Worlds, Kickstarter, Alpha One
How do you feel about the Server Meshing Technology Preview? It shows something that's next generation. Previously from most mmorpg's designs since most of the server meshing was more related to the client like you guys said it opens the door for exploits. Examples of this are like world of warcraft you could send commands to the client in order to force it to enter areas it wasn't supposed to traverse being able to mesh into textures and go into areas that normally would be restricted by the client giving that access also made it possible to dupe hack on wow because you could use a script function to to kick yourself from game before the character server had time to save the state of the inventory causing auction duplication and of course speed hacks where also possible since you could tell the client to move faster then movement speed. We could look at a newer game like Tera Online which had an open world but cryengine had the same issues of hacking such as client modification like lowering water level so you could fly across the sky or raising the water level to swim in the sky. They used anti-hack system to try to ban some of the botting and hacking but as long as people traveled 200% speed which is what most ships could travel at it was still possible and teleporting places was also an issue. The point I'm making is the client should never have had such control the more control the client has the more exploits become possible. So seeing ashes move towards a serverside system really shows a level of commitment and expertise that most game developers don't put the effort into. And I can tell that the system your building will likely not have the exploit problems that many games have been faced with in the past. Of course botting will always be an issue since bots often use the packet system sent from the server in order to get the processes completed. Even still the exploiting does more damage to the server.
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking? Since I've worked in development of games before I know the level of effort that normally goes in. But I believe that you guys have gone above the bar of what most server networking and expertise that's out there. The systems you came up with don't match anything else on the market.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview? The only thing that I'm really concerned with the meshing system is related to items. Most people don't understand the mechanics of what caused the dupe in world of warcraft because the development team buried the fact there was a dupe in world of warcraft from the beginning up till cataclysm when it was finally fixed. It was related to the auction house and saving character data's state. The server meshing aspect and transferring data from one server to the other comes with a risk. And the risk is server delay as well as state changes. I don't know the specifics of how the server will handle the data and maybe you already have safeguards in place so that it's not an issue. But if a character manages to find one of these borders whether it's through lag or some other means could they use it to duplicate items by either dropping them just before they make a transition from one realm to another or by creating a delay within the server by repeating a process of entering and exiting a realm enough times to cause a delay. Of course, I do understand that the likelihood of finding a mesh area in itself would take some ingenuity in the first place but finding a dupe hack exploit through a script function command and the auction house in wow in itself requires a lot of thought process in itself. Hackers usually find a way if there is one. It could require just packet sniffing to look for a server transfer or something to that effect and then knowing that it's a grid boundary. I bring this up because I'm really looking forward to ashes and the last thing I want to see is it get ruined by a hack or exploit.
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain! Never seen a company take the time to bother with most of these systems. The only thing somewhat similar was really Ragnarok Online had script functions for events and such which would keep track of running events and mail service and other features but many of those were temporary it wasn't controlled by a separate server. And issues could arise if function tags were not properly removed. We had problems in the past where events ended up using some of the previous tags we had for events causing players to be stuck. Honestly, you can't compare ashes with any game I've worked with even older ones from a programming point of view. Ashes is doing something unique that no one else thought of. It's a pioneer of new technology. It reminds me of Ragnarok Online when we found we could use defpattern script as a listening mechanism so that interaction with NPCs could be listened and responded to by having NPCs talk to the player base. It was something that no one ever thought of or knew was possible. The bottom line is if someone is creative enough they can make something amazing and that's what Intrepid has done.
I'm a sucker for infinite information. I'm also thrilled to see that the consideration for expansion of testers is coming down the pipeline.
When it comes to the stream in particular, I couldn't be more happy. I've seen grid loaded games in the past, and they've shown some promise but have often eventually fell on their face. I'm guessing we will see a boat load of bugs associated with it. Weird things like getting hit by a pushing spell as you're riding a vehicle of some kind and contacting a mob in the middle of a server transition. Hopefully we can put in more than enough work and the testing parameters are clear enough to the players during Alpha 2 to ensure we can cover those bases.
Which leads to my final point! Speaking about heavy amounts of detail and providing a massive amount of information to the masses on the minute details in presentation for makes me think of many ideas. Often the team will remind the viewers and followers of the project to get involved, visit the forums, visit the socials, leave feedback, apply for jobs, etc. Although most of the involvement the everyday follower of the project requires surface level understanding, I would love to see a presentation that can walk everyone through all the various ways they can get involved.
Also when approaching Alpha 2 itself, I would love to see some tutorial videos on how to actually get involved. Downloading the launcher, getting in game, submitting reports, how to act as a bug tester yourself. Once again these things are evidently relatively straight forward for most of our lovely Phoenix companions, but the more people we can welcome in with additional Accessibility features... the better.
Thanks for the consideration! Fantastic stream and have a fantastic day!
This all went over my head but the parts i did understand sound AMAZING!! great job to everyone who has worked on this and can't wait to try it out in beta or launch
I think this presentation has me more excited to play the game than any other. I’m incredibly impressed with what the networking team accomplished here. It feels like the architecture is a living and breathing world itself. That’s a very cool relationship to have to the world as a player.
I cannot wait to test this and see it in action. I spend a lot of time networking now on the ops team for another online game company. Just coming off a long night from it and happened to see this preview on YouTube. Gotta say the timing was just right, I was enthralled.
Any more content like this would be awesome. I love listening to teams communicate in real time. At some point down the line, maybe after a catastrophe or two, I’d love a video about how you maintain IntrepidNet and respond to outages.
Thanks for all the hard work, I can’t wait to play your game.
P.S.
IntrepidNet? Give it a cool name and let’s include this shit in the lore. The servers are basically the fabric of reality in Verra.
I will start by saying that everything was easy to understand and well explained from my perspective.
Now, I’m not sure if you answer stuff here, but I have a couple of doubts born from concern.
What’s the maximum number of players + proxies in the minimum area you can support?
When talking about server resources consumed, how lightweight are proxies compared to actual player characters?
The reason I have these doubts is that based on your explanation, if you keep dynamically dividing servers, at some point the proxies from the surrounding serves will take more server resources than the actual players, to a point making servers smaller in area becomes useless. This also means that this system has a cap and only one point of improvement that I can see, that is proxies’ weight compared to player characters’.
In other words, can you have … let’s say, 100 players in a 100 square meters area(1 server ) surrounded by equally populated servers?
I was blown away by the recent livestream! The depth of detail and clarity made complex topics easy to understand, which I truly appreciated. It's inspiring to see the team at Intrepid Studios pushing the boundaries and striving for full server functionality by Alpha 2 release. The ambition to go beyond what other developers have done is evident and commendable.
To answer the questions:
How do you feel about the Server Meshing Technology Preview?
The preview was impressive, showcasing the innovative approach to server technology. It’s exciting to see how this will enhance the game experience.
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
The presentation was well-structured and informative, striking a perfect balance between detail and simplicity. It’s fascinating to get an inside look at the technology powering the game.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I'm excited about the potential for seamless large-scale battles without the typical MMO lag. However, I share some concerns about potential bugs and exploits, but I’m confident that Alpha 2 testing will address these issues.
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
This level of server technology is relatively unprecedented. While other games like Star Citizen have attempted similar feats, it seems Intrepid Studios is on track to set new standards.
Keep up the fantastic work, Intrepid! Looking forward to seeing this tech in action.
I started working specifically on the idea of Dynamic Gridding as I'm a Comp Sci student that's very new to networking and software development so it's cool to see it's a known attempted solution to high player load. It's a massive undertaking with high risk high reward and I hope you guys pull it off!
Although expensive I'm sure, I am curious that with the current system scaling horizontally within the distributed system which overarching server maintains the final authoritative state? Is it a microservice we didn't see in the presentation?
If you have several authoritative servers meshed together, who determines the final authority? I was thinking a higher level authority server that maybe only writes every 15 min or so to manage the persistent state of everything without large performance overhead.
If you guys have achieved multithreading in replication, what is the most optimal number of cores each server should have?
Although expensive I'm sure, I am curious that with the current system scaling horizontally within the distributed system which overarching server maintains the final authoritative state? Is it a microservice we didn't see in the presentation?
If you have several authoritative servers meshed together, who determines the final authority? I was thinking a higher level authority server that maybe only writes every 15 min or so to manage the persistent state of everything without large performance overhead.
Pretty sure there isn't one authoritative server, what they mean when they talk about authority is that every server is authoritative for its own allocated map/grid/data area.
And the servers retrieving replication data / proxies from it would be the servers bordering that server in the grid (perhaps some data even gets replicated beyond bordering servers for map changes that can be seen from very far away; but that's probably not necessary, because that doesn't need to be represented with active actors.)
There is no hierarchy of authority, just a binary "authority or replication" over everything that's being calculated and represented. All data a server is replicating, it has no authority over, it just replicates it from the sole authoritative server that does control it, so the replicating server can take over the responsibility of passing that data on to its own users that it's responsible for, and the server with authority over that zone doesn't have to connect to players looking/interacting into it from its borders.
If players fire arrows across borders, they probably stay in the authority of the original server, but whether they hit something across the border gets decided by the server with authority over the location it lands in.
More definitively, players passing through from one server to the other belong to the authority of their original server as long as they're within its borders, and start out as proxies in the other server, but start to be controlled authoritatively by the other server once they've passed into its grid section. [See in the section of the VOD where Hunter explains promotion.]
That much all seemed pretty definitive, perhaps with a few exceptions in the details, like buffer space before a player really gets committed to the server they stepped into, in order to avoid repeated back-and-forth.
EDIT TO ADD:
4) Perhaps the most fun one:
Do you plan on inventing a separate term for the "server" world players create their characters on, now that it's hosted on multiple meshed actual servers?
Has anyone else noticed the tendency for users with animal icons to swoop in and comment on the OP without acknowledging any of the discussion that has been going on so far?
I have often wondered why nobody has attempted utilizing multiple servers working together and/or a system that only renders the part of the game world you're currently at seemlessly, but I suppose the technology wasn't there and nobody has dared to take on the challenge. I've only ever seen Star Citizen attempt this and while it is groundbreaking tech, it has it's flaws. For example in SC, if you fire a rocket into another Server sector, it will shift slightly once it crosses over and does not fire in a straight line. Obviously this is uncharted waters for online games and is still being tuned, but I'm glad you are attempting to create something that could revolutionize the online game industry. I can definitely see the potential, but I will remain skeptical until I see it for myself in game, stable and with good performance. The good news is you were able to make it use multithreading, which should be scalable. It would be interesting to see this paired with the latest Servers that have hundreds of cores and threads and what kind of performance it could bring. Keep up the good work.
We’d like your feedback on the Server Meshing Technology Preview shown during the June 2024 Development Update Livestream.
To help guide this conversation, here are a few thought starters you can choose from:
How do you feel about the Server Meshing Technology Preview?
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
Please don’t feel limited by the thought starters above. Feel free to share anything you’d like about Ashes of Creation’s Server Meshing Technology Preview shown during the June Development Update
We’ll be compiling a report for the design team on Friday, July 19, 2024, so please try to get your feedback into this thread by then!
Everyone here at Intrepid Studios looks forward to reading all the feedback you have to share!
How do you feel about the Server Meshing Technology Preview?
I really enjoyed it, I like finding out how things work and it was interesting to hear some inside knowledge of the industry.
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I thought the presentation was great and the vocabulary used was perfect for easing us into it. Great high level demo. I didn't really want it to end and wanted to learn even more, but time constraints happen.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
My only question - what happens if I just happen to plonk myself across the border? Say I'm 50% in server A and 50% in server B, which server would have the authoritative me? Would it flick between the two constantly? Do I exist purely as a proxy? Or even worse, if I'm on a corner of 4 servers?
Nothing else to add. Can we get some more techy demo stuff? It was surprisingly interesting!
How do you feel about the Server Meshing Technology Preview?
This is quite possibly one of the most exciting things about AoC. There are lots of things to like but this is going to push the limits on what we've seen today. Amazing tech preview
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I think for me personally it's a little different to most, there will have been a lot of Developers like myself watching but I can only speak for myself here. As I was able to understand and follow along very well, there were obviously a lot of things in there that we reiterated to me for things that I would have come across in the past, so personally I wanted to get straight down to the nitty gritty. That said, I'm happy with the level of detail this had as it means hopefully the non-developers were able to follow along and understand/enjoy this too
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I think multithreading the replication graph. What you guys achieved here is insanely impressive on top of what was already impressive, to hear about this really lit a fire inside of me.
I had one question around the replication time. In an earlier slide, you showed an average replication time of 150ms, and later mentioned a 94% reduction. Does this mean the new replication time is around 10ms? Are we safe to assume this?
It was a showcase that while very technical, gave us a lot of confidence in the team overall. Great showcase.
My only hope is that since it's a new technology, there are no loopholes that might facilitate duping of mats/coins/etc. And if there are, I hope these bugs are found early in the Alpha 2 and stamped out. Duping killed New World for a lot of people.
How do you feel about the Server Meshing Technology Preview?
This is quite possibly one of the most exciting things about AoC. There are lots of things to like but this is going to push the limits on what we've seen today. Amazing tech preview
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I think for me personally it's a little different to most, there will have been a lot of Developers like myself watching but I can only speak for myself here. As I was able to understand and follow along very well, there were obviously a lot of things in there that we reiterated to me for things that I would have come across in the past, so personally I wanted to get straight down to the nitty gritty. That said, I'm happy with the level of detail this had as it means hopefully the non-developers were able to follow along and understand/enjoy this too
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I think multithreading the replication graph. What you guys achieved here is insanely impressive on top of what was already impressive, to hear about this really lit a fire inside of me.
I had one question around the replication time. In an earlier slide, you showed an average replication time of 150ms, and later mentioned a 94% reduction. Does this mean the new replication time is around 10ms? Are we safe to assume this?
We’d like your feedback on the Server Meshing Technology Preview shown during the June 2024 Development Update Livestream.
I will ask straight up because this is exactly the technology that holds back Star Citizen which is a millions-dollar project, definitely bigger than you are. I understand the scale is different most likely, however, the principle of this method must be the same.
How did you achieve this? Is this your own in-house piece of technology, or was it bought/rented/whatever from CGI? I do not believe this is your work if it works as announced in the video preview. Sorry, but this is far too advanced in comparison to what we have seen from you to this day. So rather you are not that transparent, or this simply can't be your own work.
So rather you are not that transparent, or this simply can't be your own work.
Arent those both the same thing?
I feel like the way they explained it clearly shows the differences between what SC had and what they're building. The concept of the way they're transferring data between servers, skipping out a singular replication server probably has been done before (Someone mentioned to me that banks have been using this method for a while) but probably not likely that we can see has it been done in a game.
We’d like your feedback on the Server Meshing Technology Preview shown during the June 2024 Development Update Livestream.
I will ask straight up because this is exactly the technology that holds back Star Citizen which is a millions-dollar project, definitely bigger than you are. I understand the scale is different most likely, however, the principle of this method must be the same.
How did you achieve this? Is this your own in-house piece of technology, or was it bought/rented/whatever from CGI? I do not believe this is your work if it works as announced in the video preview. Sorry, but this is far too advanced in comparison to what we have seen from you to this day. So rather you are not that transparent, or this simply can't be your own work.
They have developed this all in-house. There is zero reason to believe that they haven't. There is zero chance this is cribbed from CIG's implementation either as the actual design is fundamentally different.
So rather you are not that transparent, or this simply can't be your own work.
Arent those both the same thing?
I feel like the way they explained it clearly shows the differences between what SC had and what they're building. The concept of the way they're transferring data between servers, skipping out a singular replication server probably has been done before (Someone mentioned to me that banks have been using this method for a while) but probably not likely that we can see has it been done in a game.
CIG is technically doing the same thing in SC (there isn't a single replication server, they're just using "replication layer" as a blanket term for their microservices suite for stuff like guilds, mail service, etc.) but they have a different method for inter-server replication. Instead of a server handling replication between the servers it shares boundaries with directly they have a distributed set of microservices dedicated to handling replication in addition to the other microservices they have for handling other game functions.
This was a really impressive showing of raw talent from both a server architecture and netcode perspective. I don't think many folks are going to fully appreciate the amount of work that goes into something like this despite it literally being what makes the M in MMO possible; something it seems nearly everyone has forgotten about due to how hard this tech is to get right. Regardless, I'm very curious to get more information about something interesting Anton said with regard to dynamic meshing:
"Since I joined the company I've been wanting to implement this." - Anton
Anton has been with the company for 3 years; despite this, Ashes has claimed a server population of 8-10k concurrent players per shard since at least 2018. If dynamic server meshing can actually be implemented successfully, it seems like the concurrent player population could be effectively unlimited removing the need to have multiple shards in a single region e.g. 10 North American shards should the concurrent player count exceed 100,000 players in North America.
So the big question is this: If dynamic meshing can be implemented, is the cap on a shard still 8-10k players?
I was pleasantly surprised with this stream, wasn't expecting much and really enjoyed what was shown, I do think it might have extended a bit too much but overall it was a good presentation.
As for what was shown on stream, I don't have much to comment, but I do have a concern, I believe there 100% Should be at least one or two NDA A2 tests with all the A2 backers, to make sure things wont break when the NDA is lifted
also, reopening A2 sales BEFORE A2 started is a big, big mistake, as a content creator myself, it is very hard for me to reasonably explain this to people, it does send a very negative message of intrepid studios making it look like a FOMO tactic, we that are closely following the development know the reasons, but from the outside perspective this is very bad - please consider at least reopening these sales a few weeks or a month after A2 has started
and also, I have a MAJOR piece of feedback to share - this is coming from someone who willingly spent thousands of dollars in cosmetics to support AoC so I have no bias or personal interest while saying this:
Please, Do Not make the A2 access at a lower price than what people paid for their pre-order packs, it should be at least 250$ for the access, most people didn't pay that thinking about the cosmetics, game time etc, they paid it purely for A2 access, I have people in my community that are still to this day finishing paying their A2 packs installments, they sold stuff, worked, saved money, did everything they could to buy their A2 packs, imagine if you now sell the access for cheaper...
I cant stress this enough, it will be an absolute shitshow similar to Apoc days if Intrepid not only reopen sales before A2 started, but also open it for cheaper, I hope the CMs can bring this message to the team because across multiple communities this is the feedback I am getting, and keep in mind, that as a Guild Leader this feedback piece I'm sharing goes against my own interest, as I personally would be happy to have a lot of new members participating and the cheaper the access is the better, but I have the responsibility to let the developers know the overall sentiment on this matter.
SolaxoMember, Leader of Men, Kickstarter, Alpha One
I am so excited to see the advancements you all have made with this technology. The video was fantastic, and informative. The future is looking brighter!
Comments
"Just" make the servers pass players from one server to another when they enter different parts of the world, define those parts by server load needs rather than purely by map location, and add a lot of redundancy that's covered by multiple servers, so players never experience connectivity issues when they transition from one server to another.
Obviousy the reduncancy is where the technology needs to be really good, so you neither risk connectivity failures, nor allow mismatched data to occur (always have only one single server authoritatively dictate the changes for each grid section, the rest only copy), nor copy everything with so much redundancy that the additional servers just do all the work 5 times over, and you get barely any capacity increase from server meshing in the first place.
Which is all to say, without figuring out the precise solutions, "my" idea isn't worth anything.
But I'm just surprised there haven't been more studios that committed to these methods. Considering how rich some of these studios are, it's kinda crazy that they felt the need to rely on layers, when it kills so much of the natural community interaction, and community is what defines MMOs in the first place...
So I guess thank you for actually doing the work to create the logical, thorough solution, rather than going the path of least resistance. It's what the world needs lot more of.
I guess I do also have some questions/suggestions.
1) In the explanation of microservices (which I'm sure are an important tool either way) the example given is something like players trading across the map or something.
Would this really be so difficult to connect without microservices?
Hopefully, player inventory and gold are always saved in persistent databases, not just on the current server the player is on, right?
2) With the explanation of performance increases through feats like multithreading, i the performance still struggles, have you considered turning off certain game features when more than 1000 players are in a condensed area? Refreshing certain data less often, and assuming certain defaults?
Simplifying cosmetics and character information down to a tenth of the detail (perhaps let the client retain the information they already have, so players notice the difference less)? Simplifying ability casts by representing fewer augments for the player (but still getting correct battle calculations on the server itself?)
I'm not sure how much any of this would even be useful with the engine network code already avoiding unnecessary engine queries, but perhaps that's something to look into, if the limits of what's possible are reached; just as a fallback to make it through the most extreme individual server load scenarios.
3) When you talk about promotion, you list a loooot of components that every actor has to have transferred. Why does there need to be something like an "interactive" component on an actor that crosses a server? Shouldn't that component be intrinsic to all actors of the type [caravan] or [mount] (or whatever the actor with that component happens to be) and not have to be data that is communicated by the servers, but simply be assumed to be an attribute of the asset?
These are very ambitious as you described. Would love such things to work but promising and showcasing everything will only increase the amount of hype and expectations. Seeing such pioneering technologies also made me think how bad can things get during or after launch. But I am also happy that someone is still pushing the boundaries because we would still be playing tap target and topdown 2D MMOs if nobody attempted any of these. It is also a smart business move to patent and sell/rent these services via IntrepidNet if it all works out in the end.
As a gamer though, what I'm so happy to hear was more about being able to monitor every node and player states so that you can more actively participate and even counter the botting issue that plague mmos. I dont know if that is possible to solve bots just by monitoring some stats BUT that sounds much more controllable and profitable than having to hire hundreds of Game Masters. (Even though I love Game Masters)
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
As a game dev who loves multiplayer games (especially mmos), It was quite advanced even for me that I had to rewatch entire thing and pause. It is such knowledge I coveted from multiple GDCs and tried to imitate myself for my game jam attempts using some commercial multiplayer libraries. Honestly, one of the reasons I sought early access to AoC was just to get to see behind the scenes for information like these hoping it might help me become a better game designer or developer. I didn't expect to see them made public like this.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
The graphs and visuals helped a ton for me to understand and visualize what I heard so it was a huge plus. I was concerned that it would somehow break the game flow or create exploitable interactions between servers while players are moving across the map but visibly seeing ingame footage of the world borders/colliders helped visualize and you also underlined something which I recall as "conditions" to migrate/promote between the two servers while moving which are probably checked to avoid such stuff from happening.
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
I have not seen anything similar but I played many games that desired such systems such as Ultima Online, Lineage 2 and Black Desert Online because of the open world PvP and the karma/corruption systems they implemented. I am a huge fan of entire playerbase being on one realm instead of seperate channels/instances that are swappable at will. Such dynamic systems sound really much more fun presuming these instances/servers the realm consists of are not manually switchable and much more performant and happening all in the background without players noticing. If it is as good as it looks on the videos, it is almost indistinguishable.
P.S. I want to mention that as a person who have been in production and publishing of a mere few video games, I have always been trying to recall and make research about game mechanics from games that I have played to try and recreate (and maybe even improve upon) them but almost most companies don't care to share and/or don't even want to share their knowledge because they want to make more profit off it before such technologies become mainstream. I have only seen something similar to these in some GDCs (Rocket League's networking systems come to mind) and Path of Exile 2's new shader technology reveals recently. I love the upfront and public approach to game development making consumers also become more aware of difficulties of developing games and helping other game devs also see behind the curtain. This also feels like a good "investment" for future game developers to not struggle over the same hurdles people struggle with even if it is not entire codebase but visual depiction or explanation showing that it is possible and how it is achieved.
Sorry for the long post, thanks for reading.
Feel free to correct me if I'm mistaken at any part.
I play to win, not just for fun
Because I enjoy giving it all I got.
Steam Profile
Twitch Channel
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking? Since I've worked in development of games before I know the level of effort that normally goes in. But I believe that you guys have gone above the bar of what most server networking and expertise that's out there. The systems you came up with don't match anything else on the market.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview? The only thing that I'm really concerned with the meshing system is related to items. Most people don't understand the mechanics of what caused the dupe in world of warcraft because the development team buried the fact there was a dupe in world of warcraft from the beginning up till cataclysm when it was finally fixed. It was related to the auction house and saving character data's state. The server meshing aspect and transferring data from one server to the other comes with a risk. And the risk is server delay as well as state changes. I don't know the specifics of how the server will handle the data and maybe you already have safeguards in place so that it's not an issue. But if a character manages to find one of these borders whether it's through lag or some other means could they use it to duplicate items by either dropping them just before they make a transition from one realm to another or by creating a delay within the server by repeating a process of entering and exiting a realm enough times to cause a delay. Of course, I do understand that the likelihood of finding a mesh area in itself would take some ingenuity in the first place but finding a dupe hack exploit through a script function command and the auction house in wow in itself requires a lot of thought process in itself. Hackers usually find a way if there is one. It could require just packet sniffing to look for a server transfer or something to that effect and then knowing that it's a grid boundary. I bring this up because I'm really looking forward to ashes and the last thing I want to see is it get ruined by a hack or exploit.
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain! Never seen a company take the time to bother with most of these systems. The only thing somewhat similar was really Ragnarok Online had script functions for events and such which would keep track of running events and mail service and other features but many of those were temporary it wasn't controlled by a separate server. And issues could arise if function tags were not properly removed. We had problems in the past where events ended up using some of the previous tags we had for events causing players to be stuck. Honestly, you can't compare ashes with any game I've worked with even older ones from a programming point of view. Ashes is doing something unique that no one else thought of. It's a pioneer of new technology. It reminds me of Ragnarok Online when we found we could use defpattern script as a listening mechanism so that interaction with NPCs could be listened and responded to by having NPCs talk to the player base. It was something that no one ever thought of or knew was possible. The bottom line is if someone is creative enough they can make something amazing and that's what Intrepid has done.
College Lessons
When it comes to the stream in particular, I couldn't be more happy. I've seen grid loaded games in the past, and they've shown some promise but have often eventually fell on their face. I'm guessing we will see a boat load of bugs associated with it. Weird things like getting hit by a pushing spell as you're riding a vehicle of some kind and contacting a mob in the middle of a server transition. Hopefully we can put in more than enough work and the testing parameters are clear enough to the players during Alpha 2 to ensure we can cover those bases.
Which leads to my final point! Speaking about heavy amounts of detail and providing a massive amount of information to the masses on the minute details in presentation for makes me think of many ideas. Often the team will remind the viewers and followers of the project to get involved, visit the forums, visit the socials, leave feedback, apply for jobs, etc. Although most of the involvement the everyday follower of the project requires surface level understanding, I would love to see a presentation that can walk everyone through all the various ways they can get involved.
Also when approaching Alpha 2 itself, I would love to see some tutorial videos on how to actually get involved. Downloading the launcher, getting in game, submitting reports, how to act as a bug tester yourself. Once again these things are evidently relatively straight forward for most of our lovely Phoenix companions, but the more people we can welcome in with additional Accessibility features... the better.
Thanks for the consideration! Fantastic stream and have a fantastic day!
I think this presentation has me more excited to play the game than any other. I’m incredibly impressed with what the networking team accomplished here. It feels like the architecture is a living and breathing world itself. That’s a very cool relationship to have to the world as a player.
I cannot wait to test this and see it in action. I spend a lot of time networking now on the ops team for another online game company. Just coming off a long night from it and happened to see this preview on YouTube. Gotta say the timing was just right, I was enthralled.
Any more content like this would be awesome. I love listening to teams communicate in real time. At some point down the line, maybe after a catastrophe or two, I’d love a video about how you maintain IntrepidNet and respond to outages.
Thanks for all the hard work, I can’t wait to play your game.
P.S.
IntrepidNet? Give it a cool name and let’s include this shit in the lore. The servers are basically the fabric of reality in Verra.
Now, I’m not sure if you answer stuff here, but I have a couple of doubts born from concern.
What’s the maximum number of players + proxies in the minimum area you can support?
When talking about server resources consumed, how lightweight are proxies compared to actual player characters?
The reason I have these doubts is that based on your explanation, if you keep dynamically dividing servers, at some point the proxies from the surrounding serves will take more server resources than the actual players, to a point making servers smaller in area becomes useless. This also means that this system has a cap and only one point of improvement that I can see, that is proxies’ weight compared to player characters’.
In other words, can you have … let’s say, 100 players in a 100 square meters area(1 server ) surrounded by equally populated servers?
To answer the questions:
How do you feel about the Server Meshing Technology Preview?
The preview was impressive, showcasing the innovative approach to server technology. It’s exciting to see how this will enhance the game experience.
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
The presentation was well-structured and informative, striking a perfect balance between detail and simplicity. It’s fascinating to get an inside look at the technology powering the game.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I'm excited about the potential for seamless large-scale battles without the typical MMO lag. However, I share some concerns about potential bugs and exploits, but I’m confident that Alpha 2 testing will address these issues.
Are there similar systems you’ve seen in other games that you like or dislike? If so, please explain!
This level of server technology is relatively unprecedented. While other games like Star Citizen have attempted similar feats, it seems Intrepid Studios is on track to set new standards.
Keep up the fantastic work, Intrepid! Looking forward to seeing this tech in action.
I started working specifically on the idea of Dynamic Gridding as I'm a Comp Sci student that's very new to networking and software development so it's cool to see it's a known attempted solution to high player load. It's a massive undertaking with high risk high reward and I hope you guys pull it off!
Although expensive I'm sure, I am curious that with the current system scaling horizontally within the distributed system which overarching server maintains the final authoritative state? Is it a microservice we didn't see in the presentation?
If you have several authoritative servers meshed together, who determines the final authority? I was thinking a higher level authority server that maybe only writes every 15 min or so to manage the persistent state of everything without large performance overhead.
If you guys have achieved multithreading in replication, what is the most optimal number of cores each server should have?
When will we be able to buy ALPHA 2 keys ?
And the servers retrieving replication data / proxies from it would be the servers bordering that server in the grid (perhaps some data even gets replicated beyond bordering servers for map changes that can be seen from very far away; but that's probably not necessary, because that doesn't need to be represented with active actors.)
There is no hierarchy of authority, just a binary "authority or replication" over everything that's being calculated and represented. All data a server is replicating, it has no authority over, it just replicates it from the sole authoritative server that does control it, so the replicating server can take over the responsibility of passing that data on to its own users that it's responsible for, and the server with authority over that zone doesn't have to connect to players looking/interacting into it from its borders.
If players fire arrows across borders, they probably stay in the authority of the original server, but whether they hit something across the border gets decided by the server with authority over the location it lands in.
More definitively, players passing through from one server to the other belong to the authority of their original server as long as they're within its borders, and start out as proxies in the other server, but start to be controlled authoritatively by the other server once they've passed into its grid section. [See in the section of the VOD where Hunter explains promotion.]
That much all seemed pretty definitive, perhaps with a few exceptions in the details, like buffer space before a player really gets committed to the server they stepped into, in order to avoid repeated back-and-forth.
EDIT TO ADD:
4) Perhaps the most fun one:
Do you plan on inventing a separate term for the "server" world players create their characters on, now that it's hosted on multiple meshed actual servers?
top
I really enjoyed it, I like finding out how things work and it was interesting to hear some inside knowledge of the industry.
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I thought the presentation was great and the vocabulary used was perfect for easing us into it. Great high level demo. I didn't really want it to end and wanted to learn even more, but time constraints happen.
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
My only question - what happens if I just happen to plonk myself across the border? Say I'm 50% in server A and 50% in server B, which server would have the authoritative me? Would it flick between the two constantly? Do I exist purely as a proxy? Or even worse, if I'm on a corner of 4 servers?
Nothing else to add. Can we get some more techy demo stuff? It was surprisingly interesting!
This is quite possibly one of the most exciting things about AoC. There are lots of things to like but this is going to push the limits on what we've seen today. Amazing tech preview
What did you think about the presentation during this development update, and hearing more about what goes on behind the scenes of server networking?
I think for me personally it's a little different to most, there will have been a lot of Developers like myself watching but I can only speak for myself here. As I was able to understand and follow along very well, there were obviously a lot of things in there that we reiterated to me for things that I would have come across in the past, so personally I wanted to get straight down to the nitty gritty. That said, I'm happy with the level of detail this had as it means hopefully the non-developers were able to follow along and understand/enjoy this too
Is there anything in particular you’re excited or concerned about regarding what was shown with the Server Meshing Technology Preview?
I think multithreading the replication graph. What you guys achieved here is insanely impressive on top of what was already impressive, to hear about this really lit a fire inside of me.
I had one question around the replication time. In an earlier slide, you showed an average replication time of 150ms, and later mentioned a 94% reduction. Does this mean the new replication time is around 10ms? Are we safe to assume this?
My only hope is that since it's a new technology, there are no loopholes that might facilitate duping of mats/coins/etc. And if there are, I hope these bugs are found early in the Alpha 2 and stamped out. Duping killed New World for a lot of people.
i think its around 8~ ms so yea
I will ask straight up because this is exactly the technology that holds back Star Citizen which is a millions-dollar project, definitely bigger than you are. I understand the scale is different most likely, however, the principle of this method must be the same.
How did you achieve this? Is this your own in-house piece of technology, or was it bought/rented/whatever from CGI? I do not believe this is your work if it works as announced in the video preview. Sorry, but this is far too advanced in comparison to what we have seen from you to this day. So rather you are not that transparent, or this simply can't be your own work.
My only concerns are bugs/exploits but thats why testing exists!
Arent those both the same thing?
I feel like the way they explained it clearly shows the differences between what SC had and what they're building. The concept of the way they're transferring data between servers, skipping out a singular replication server probably has been done before (Someone mentioned to me that banks have been using this method for a while) but probably not likely that we can see has it been done in a game.
They have developed this all in-house. There is zero reason to believe that they haven't. There is zero chance this is cribbed from CIG's implementation either as the actual design is fundamentally different.
CIG is technically doing the same thing in SC (there isn't a single replication server, they're just using "replication layer" as a blanket term for their microservices suite for stuff like guilds, mail service, etc.) but they have a different method for inter-server replication. Instead of a server handling replication between the servers it shares boundaries with directly they have a distributed set of microservices dedicated to handling replication in addition to the other microservices they have for handling other game functions.
"Since I joined the company I've been wanting to implement this." - Anton
Anton has been with the company for 3 years; despite this, Ashes has claimed a server population of 8-10k concurrent players per shard since at least 2018. If dynamic server meshing can actually be implemented successfully, it seems like the concurrent player population could be effectively unlimited removing the need to have multiple shards in a single region e.g. 10 North American shards should the concurrent player count exceed 100,000 players in North America.
So the big question is this: If dynamic meshing can be implemented, is the cap on a shard still 8-10k players?
Edit: Steven answered the question:
As for what was shown on stream, I don't have much to comment, but I do have a concern, I believe there 100% Should be at least one or two NDA A2 tests with all the A2 backers, to make sure things wont break when the NDA is lifted
also, reopening A2 sales BEFORE A2 started is a big, big mistake, as a content creator myself, it is very hard for me to reasonably explain this to people, it does send a very negative message of intrepid studios making it look like a FOMO tactic, we that are closely following the development know the reasons, but from the outside perspective this is very bad - please consider at least reopening these sales a few weeks or a month after A2 has started
and also, I have a MAJOR piece of feedback to share - this is coming from someone who willingly spent thousands of dollars in cosmetics to support AoC so I have no bias or personal interest while saying this:
Please, Do Not make the A2 access at a lower price than what people paid for their pre-order packs, it should be at least 250$ for the access, most people didn't pay that thinking about the cosmetics, game time etc, they paid it purely for A2 access, I have people in my community that are still to this day finishing paying their A2 packs installments, they sold stuff, worked, saved money, did everything they could to buy their A2 packs, imagine if you now sell the access for cheaper...
I cant stress this enough, it will be an absolute shitshow similar to Apoc days if Intrepid not only reopen sales before A2 started, but also open it for cheaper, I hope the CMs can bring this message to the team because across multiple communities this is the feedback I am getting, and keep in mind, that as a Guild Leader this feedback piece I'm sharing goes against my own interest, as I personally would be happy to have a lot of new members participating and the cheaper the access is the better, but I have the responsibility to let the developers know the overall sentiment on this matter.