This book is a work in progress, comments are welcome to: johno(at)johno(dot)se
Back to index...This document is a discussion of the different multiplayer-enabled application architectures that Massive Entertainment has employed over the years, including thoughts on new directions.
All of our past projects have used a Client - Server model, in that one of the nodes in a game acted as an authoritative host and performed the application logic, while the other nodes handled input and visualization. This model also allowed us to create dedicated server applications, which could be run at centralized locations on powerful hardware, thereby easing the load on user machines by alleviating the need for them to host games themselves.
Due to the nature of network latency, messages cannot be sent or received instantly. This makes asynchronous event-driven architectures a necessity for most real-time multiplayer games, in order to preserve the illusion that things are happening in real-time. Given this premise, a number of different architectures are possible.
Ground Control used the concept of a Message to facilitate communication between the Client and Server, which was basically a glorified data buffer. Message objects exposed interfaces for reading and writing basic datatypes, like ints, floats, and strings. Messages existed in two flavors; WriteMessage, used for writing outgoing data, and ReadMessage, used to read incoming data. In this architecture, everything was designed with the "lowest common denominator" in mind, namely that in the remote case, where the Clients and Server existed in seperate address spaces, information would need to be transmitted in a byte-stream format (as that is what Winsock allowed us to send across the network).
Beyond this concept of a Message there existed the concept of a Communicator, which could send and receive Message objects. This again was designed with the "more complex" remote case in mind. In the remote case, the Communicator used was a RemoteCommunicator, which could send messages across the network using Winsock. In the local case, where the Client and the Server modules existed and executed in the same address space, a LocalCommunicator was used, which simply mimicked the way that Winsock worked, asynchronously passing data around in the local address space instead of sending it across the network.
With the Communicator abstraction in place, the application could be constructed around the concept of Message objects as the communication medium. For example, when a Client wanted to affect the logic of the Server, input was sampled and written as an "event" to a WriteMessage, and this WriteMessage was then passed to the Communicator object that was currently in use (which as mentioned could be a local or remote version). What happened beyond that point was opaque to the Client. For the Server to receive this "event", it had to ask it's own Communicator object to "receive" ReadMessage objects, which the Server would then parse in gargantuan switch statements (switching on event type), extracting the parameters passed by the Client, and then affect logic internally by setting state or calling internal methods.
While this system worked, the level att which communication took place was extremely low. Almost every aspect of the Client and Server were exposed to the concept of a Message, and the fact that the entire architecture was designed for the remote case was evident throughout. Indeed, attempting to implement a "WinSock simulator", as the LocalCommunicators were, was also quite complex, and in the end the differences between the local and remote cases were impossible to completely isolate from the Client and Server.
Additionally, a clear redundancy was evident in this architecture, as all conceptual aspects of the Server also existed in a secondary form on the Client. For example, the Server had SV_Entity, while the Client had CL_Entity. Both were conceptually the same thing, but were in different formats, so almost no re-use was possible.
During Ground Control II, an alternative architecture was explored. In this architecture, the concepts of Client and Server mapped to that of Player and Game, respectively. The main goal of this architecture was to completely hide the existence of the network, and allow the main application logic and visualisation to be isolated from such details. All networking was external to these basic concepts, which removed the need to create any type of "network simulation"; the networking aspects of the application could be implemented in a straightforward manner.
These goals were achieved through the use of proxies, as below:
IPlayer implementations use the IGame interface, while IGame implementations use the IPlayer interface. In this way, proxies could be constructed that existed in the same address space as the objects that used them, allowing the programmers to proceed as though the local case was all that existed. With this architecture it was indeed possible to defer implementation of the remote case until much later, with full confidence in that it would indeed work fine when implemented. Also, any kinds of optimizations that were required by the remote implementation, including things like positional and rotational interpolation, became implementation details of the remote case, and did not affect the local case at all.
Instead of using Message objects as the communication medium, the IPlayer and IGame interfaces were able to express high level events in terms of pure virtual functions. This was fortunate, as these interfaces more clearly expressed the application in terms very close to the problem domain.
However, some problems did still remain, the largest being that the replication of concepts still existed in Player and Game in the same way that they did in the Ground Control architecture in Client and Server (Game had EXG_Units, Player had EXP_Units). This problem was severely exacerbated in Ground Control II due to the fact that there were several implementations of IPlayer that required their own representations of these concepts.
Back to index...