PDA

View Full Version : On Distributed Servers (Uru Live) (long)



bfioca
11-22-2003, 10:34 PM
Hi all,
I've been reading this forum for a few days now (read: lurking), and I figured I'd try my hand at shedding some light on the issue of maybe why the Online version isn't quite ready yet.

Here's some background first. I was a beta tester, so I've seen Live. I'm also a software developer who works in the field of internet distributed systems. The company I work for, in fact, has recently been experiencing issues not unlike (presumably) those of Ubi/Cyan on their live servers.

The main issue is this: real-time distributed systems are *hard*. There are many reasons for this, the main one being the nature of the internet - firing packets off, some getting lost/dropped, resends, out-of-order communications, etc. The internet was never designed with real time systems in mind. In fact, much of the work going on with the next generation internet (IPv6) has to do with this issue.

Another major issue is intra-server concurrency. When you have hundreds of people on a server, all in contention for the same resources, there are many, many possibilities for failure, including deadlock issues, out-of-synch problems, the list is pretty large.

Compounding the complexity of planning for and solving these issues is the fact that for a system such as an online game, there are no load-testing programs out there you can use to test for bugs. You are basically forced to test in the production environment, since there is almost no way of simulating a real-world load in the QA cycle. No matter how much design and testing you do, until you get your software out there being hammered on you won't really know how it behaves under stress.

Now you may be saying "yeah, but that's why they had the beta test," and I agree with you to some extent. From what I saw of the beta, they were constantly making improvements, but the one thing that never really got much better was the lag caused by too many people. Lag is caused when the servers get bogged down with trying to keep track of everyone/everything and then having to send all of that information to everyone connected. There are ways to optimize this (You may have seen other MMORPGS deal with this problem by having the client assume a person continues doing what he's doing until it hears otherwise - causing the avatars to skip around, or float when they jump, etc.) But I'm sure Cyan doesn't want to compromize the look of the environent or the smoothness of the experience at all, which leads to the need for *entirely new ways of thingking* on this subject to fix these problems.

In short, these guys are really breaking into new territory here. What they're attempting to do from a distributed computing/real-time systems standpoint has never been done before. I imagine they are working extremely hard to make this game outrageously immersive and interactive in ways no-one has ever seen before, and I understand how obscenely difficult it is do accomplish what they're trying to do.

What most likely happened is when Cyan said that they needed more time to perfect the online game, Ubi pushed them into releasing the Prime version to get the cash flow up to a point to enable them to really produce the game they wanted to make. I'm all for this, as the alternative solution would have been to compeltely compromise the fundamental beauty and complexity of the online game.

Thanks for reading, I hope I've maybe cleared some of this up. I'm hoping I get my invite on monday. Cyan/Ubi - Please consider this an application for a Dev position (resume available upon request)

Brian Fioca
brian@fioca.com

bfioca
11-22-2003, 10:34 PM
Hi all,
I've been reading this forum for a few days now (read: lurking), and I figured I'd try my hand at shedding some light on the issue of maybe why the Online version isn't quite ready yet.

Here's some background first. I was a beta tester, so I've seen Live. I'm also a software developer who works in the field of internet distributed systems. The company I work for, in fact, has recently been experiencing issues not unlike (presumably) those of Ubi/Cyan on their live servers.

The main issue is this: real-time distributed systems are *hard*. There are many reasons for this, the main one being the nature of the internet - firing packets off, some getting lost/dropped, resends, out-of-order communications, etc. The internet was never designed with real time systems in mind. In fact, much of the work going on with the next generation internet (IPv6) has to do with this issue.

Another major issue is intra-server concurrency. When you have hundreds of people on a server, all in contention for the same resources, there are many, many possibilities for failure, including deadlock issues, out-of-synch problems, the list is pretty large.

Compounding the complexity of planning for and solving these issues is the fact that for a system such as an online game, there are no load-testing programs out there you can use to test for bugs. You are basically forced to test in the production environment, since there is almost no way of simulating a real-world load in the QA cycle. No matter how much design and testing you do, until you get your software out there being hammered on you won't really know how it behaves under stress.

Now you may be saying "yeah, but that's why they had the beta test," and I agree with you to some extent. From what I saw of the beta, they were constantly making improvements, but the one thing that never really got much better was the lag caused by too many people. Lag is caused when the servers get bogged down with trying to keep track of everyone/everything and then having to send all of that information to everyone connected. There are ways to optimize this (You may have seen other MMORPGS deal with this problem by having the client assume a person continues doing what he's doing until it hears otherwise - causing the avatars to skip around, or float when they jump, etc.) But I'm sure Cyan doesn't want to compromize the look of the environent or the smoothness of the experience at all, which leads to the need for *entirely new ways of thingking* on this subject to fix these problems.

In short, these guys are really breaking into new territory here. What they're attempting to do from a distributed computing/real-time systems standpoint has never been done before. I imagine they are working extremely hard to make this game outrageously immersive and interactive in ways no-one has ever seen before, and I understand how obscenely difficult it is do accomplish what they're trying to do.

What most likely happened is when Cyan said that they needed more time to perfect the online game, Ubi pushed them into releasing the Prime version to get the cash flow up to a point to enable them to really produce the game they wanted to make. I'm all for this, as the alternative solution would have been to compeltely compromise the fundamental beauty and complexity of the online game.

Thanks for reading, I hope I've maybe cleared some of this up. I'm hoping I get my invite on monday. Cyan/Ubi - Please consider this an application for a Dev position (resume available upon request)

Brian Fioca
brian@fioca.com

graham_614
11-22-2003, 10:39 PM
*grin*

Impressive - you managed to give us a simple and accurate analysis of a fairly technical system in plain english...

WHILE you pimped your services as a developer http://ubbxforums.ubi.com/infopop/emoticons/icon_wink.gif

From the peanut gallery - we salute you sir http://ubbxforums.ubi.com/infopop/emoticons/icon_smile.gif

Free Urulive!

Kwartha
11-22-2003, 10:58 PM
Great post. I, too have had a fair bit of experience researching and building distributed systems. MMORPG players often fail to grasp the very real differences between a 16 player Quake server and a persistant world spanning a number of servers tracking thousands of players and millions of objects. The technical issues are enormously complex, and developers like Cyan are breaking new ground everytime they work on this scale of game.

It's very easy to insult the devs or the publisher and assume that they're too greedy or stupid to give you what you wanted. It's much harder to ask "What could be the real reason a hard working company like Cyan would keep this game out of the hands of their loyal fans?" You have to account for an almost uncountable number of technical issues which Cyan had no real means of predicting they would encounter. Distributed systems are notoriously hard to debug, since there's usually no way to replicate the circumstances that cause bugs or crashes. Cyan has a ton of expertise at their disposal and (as we saw in the beta) they know how to put together a stable, working online game. We just have to give them more time to work out the issues.

By the sounds of it, Cyan has put all their hopes for the future into Uru. If they can't get Live to work, they could all be out of jobs, so I would say they've got more reasons to want Live to work than us fans do. In all likelihood the technical issues that have kept people out of the game are having very real ramifications for the people who work for Cyan.

And in case anyone from Cyan is reading: My resume is also available upon request. As another fan once wrote, I would gladly accept a position mopping the floors of your offices. I think I would be more valuable writing network code, but if that job's taken I'd understand.

bfioca
11-22-2003, 10:58 PM
Thanks, graham...

Don't get me wrong though, it does suck that Live isn't ready yet... especially since they had that long beta and all... But I, for one, know what it's like working nights and weekends trying to nail down a really obscure bug that only happens randomly when 1500 or so people are logged in.

The funny thing is, usually it doesn't really matter how many people you have looking at it. Sometimes it just takes time...

maztec
11-22-2003, 10:59 PM
I have IPv6, I wish Cyan would implement it on their servers too. That'd rock!

Great post bfioca -- and kudos for graham's reaction, I do believe you are the first to elicit such a response http://ubbxforums.ubi.com/infopop/emoticons/icon_wink.gif

Whereas my help with tech issues would only earn me a job with Ubi telephone support. What a waste http://ubbxforums.ubi.com/infopop/emoticons/icon_wink.gif

Kwartha
11-22-2003, 11:05 PM
<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by maztec:
I have IPv6, I wish Cyan would implement it on their servers too. That'd rock!<HR></BLOCKQUOTE>
Sadly IPv6 is going to require that most of the routers and switches on the internet get upgrades as well. That's something that's really out of Cyan's hands, even if they had it available on their servers.

Pelke
11-22-2003, 11:48 PM
It's nice to see a positive post for a change. I like the fact that your post addresses the way things are, not just the way everyone would like them.

I suppose some people feel cheated since their "out-of-box" experience has not been perfect, so far. That is understandable. However, I see this game as a large experiment -- a journey into unkown territory. Not everything goes smoothly along the way. But, it is the experience of being part of something new that is exciting to me. We will help shape this new technology. We will be able to look back a few years from now and say "yeah, I was a part of that when it was just getting off the ground".

Enough rambling. Thanks for a clear, concise post minus the whining. It's refreshing!

Pelke
Austin, TX

maztec
11-23-2003, 12:28 AM
Kawartha, yup. This is why the US Govt and military are starting to adopt the new standards. Hopefully that will speed up everyone elses adoption.

Ahh! Cute Korean kid, must adopt!