You certainly already know Google Earth, a system that enables you to see satellite photographs of any location on earth. It proposes you both 2D representation and 3D.
This service is accessible on desktop, but unfortunately, nothing is available for your mobile phone.
Mobile Earth is a technology dedicated to the mobile wireless industry: it’s purpose is to provide mobile industry with an innovative technology that enables users to play in a real time 3D environment; the graphics are true-life representation of cities and landscapes that shows off "infinite" unmatched level of details.
I started the project during late April this year, with the objective of providing a streaming technology capable of displaying interactive realtime 3D with unmatched quality for hi-performance 3.5G wireless networks: HSDPA.
A handset user will play a game — and later will use Geographic Information System-based services — that displays very high quality backgrounds based on real world satellite photographs of earth.
The idea behind is to have a server that renders the world in realtime 3D at the location of the avatar of the player. The world is rendered with a very large z-far value — typically more than one hundred kilometers, hence the sensation of extremely large persistent world.
Every single frame generated by the engine is compressed on-the-fly to feed an upstream video.
On the other end, an handset is connected to the server via a TCP connection over a HSDPA radio network, itself connected to the Internet. HSDPA is the 3.5 G network capable of 1Mbps downstream bandwidth. HSDPA deployment are currently starting up in France.
The video is then streamed down the handset, and the video decodes the stream and display the background at the phone resolution — typically QVGA, 320×240 pixels.
As a result, the player sees an incredibly hi-quality graphics.
The player can moves his avatar up, down and right left while moving. This information is send up to the server. Upon receipt, the game engine interprets the player's move and places the camera accordingly, such that the upstream video is altered.
As a result, the player feels that he can freely moves in the background and perform interactions.
Playing such a game requires very high precision in order to see building, houses and even cars on high way for example.
Partnering with IGN, I was provided with a set of very high resolution satellite photographs: four pixels per square meter. At this resolution, we can clearly see cars, but also parasols on sunny on beaches!
Unfortunately, the over side of the coin is the very large size of the related files. For example, the size required to store all photographs for one department is 60 Giga bytes.
I will then need 6 terra bytes to store satellite photographs for France — equivalent to 1257 DVDs!
I started off with a LOD-based system for the management of the world.
I set up a server that processes data by implementing a data-flow from the IGN photographs and altitude images to the engine database. 60 Gb is necessary to hold "Alpes Maritimes" original data.
— Altitude: Altitude information is contained in a 38Mb 8-bit indexed TIFF image where the index indicates the level of altitude with a resolution of 50 meters; I slices the image into many 2048×2048 pixels square with a down-sampling algorithm.
— Photographs: the satellite photographs are converted from 2000×2000 pixels RGB TIFF images into 2048×2048 pixels RGB jpeg, from which I generate a set of 10 mipmap images. At the moment, this takes 20+ minutes to be generated on a P4 2GHz, for the small portion I use for the game.
The rendering engine for the moment uses only a small subset of the department information. The LOD system command the tessellation on-demand in order to minimize the required bandwidth.
Each frame is captured from the frame buffer and compressed as jpeg files on the disk, for a second process in charge of the streaming.
The set of generated images are compressed on-the-fly and send down the handset by using a PacketVideo server.
This project has to prove the real performances of HSDPA technology that offers a theoretical bandwidth of 1Mbps! Meanwhile, the application is scalable and will also support UMTS networks.
The client needs to playback video in a background layer while displaying interactive assets on top of it. It therefore requires a correct handling of alpha blending or punch through at least.
Oddly enough, there is no solution for doing this, at the time of writing; this is why we turned to a different approach, and chose to use a flash player where video playback is supported; the chosen solution is the Bluestreak client.
This solution raised two important issues that impact us: firstly, rendering vector-based graphics is less efficient than simple bitmap, and secondly, the result is less realistic, which deserves the goal of the project to render real-life images.
Whatever, we go for it.
The game displays a space craft that must pass thru 3D big "donuts" bonuses laid out along the chained-cubic spline path.
The player can "freely" move within the virtual tunnel. But in the first version, he cannot go anywhere in the world.
In this stage, the player will be totally free to go anywhere he wants. He shall be able to explore the whole background.
Player's inputs will be uploaded in realtime to the server thru a TCP connection. Upon receipt, the engine will take the necessary action to perform what was required by the player — including moving the camera — hence generating new video frames.
The big issue here is that we have 6 seconds of buffered video on the client, which causes us a 6 seconds latency. For this reason, the client anticipate the background move by applying the 3D transformation on the spacecraft itself with the background still in the previous position.
When frames begins to downstream, the client start to compensate the move of the background by canceling anticipated transformations already applied to the aircraft. For this purpose, I have embedded 3D transformation information as meta data within the graphics frame themselves.
But this is for the next stages… be patient and enjoy the first images and video on the right…
I have recently added the support for the collada file format: this enable me to have better detailed obejects in the game. In addition, I can now get any object published on Google 3D Wharehouse.
Here is a movie of the technology update captured straight from the engine: I was piloting the jet, chasing a Boeing B747 regular flight.
The first movie I captured on June 26th 2006 straight from the engine, featuring a jet hovering of Nice next to the mediterranean sea coast.
Below are some screenshots I like the most. currently I did not apply textures on jets.
Satellite photographs are processed in order to fit LOD requirements.
In 2004 I had designed and created the http-based multiplayer system that enables mobile phone owners to play peer-to-peer games thru a central node such as NxN game-play is possible.
I choose to address only HTTP connection because at the time, it was the only connection available to ALL phone models.
*In 2005, I proposed to the company to push the envelope one step further, and I've designed and created a second generation system that will enable both PC players and mobile phone owners to play altogether on the same MMO; that's what we call now "web-to-wireless" gaming
Market segmentation is probably one of the most challenging issue to address when talking about wireless gaming development.
Though more than 800 millions brand-new handsets have been sold during 2005 — probably 1 billion will in 2006 — the installed based still need to be addressed in order to maximize the
Another issue of importance is certainly the notion of scalability: this is the term to define the difference nature and performance between many different handset models. Some handset have small screen, others larger ones, some has some 3D capability, some support TCP connection others don't, etc. It is then important to categorize our game feature in order to know what features are required by the set of targeted handsets.
Keeping that in mind, I added three goals for the project:
* Cross visual representation: Enable both 3D and 2D game play of the same game: as most of the mobile handset models only supports 2D display, it was a MUST to support those in addition to 3D capable models.
* Cross Network, cross operators: Enable 3G networks but also GPRS. Despite their very different performances, we should be able to capture as many as possible model handsets. We also want that users play altogether regardless of their operator.
* Cross Platform: Enable both personal computers (internet) and mobile phones (radio network) to play altogether, seamlessly.
The game implements a catch-the-flag game play. As a player, you will be provided with a car, appearing somewhere in the world. Your goal is to catch the flag — a cup in the final game — and keep it as long as possible,
avoiding any chasing cars trying to steal it.
PC users — take PC as a general term as we support any platform running Java™ — will benefit from realtime 3D graphics as long as 3D capable mobile such as SonyEricsson JP-6 phones for instance.
Non 3D capable phones will implement the 2D view of the same game.
3D players will benefit from 3D perspective view of the scene in a third-view camera, while 2D players will benefit from a orthogonal, top-view camera.
The world database is designed in such way that it must provide all the information in an efficient way to all type of clients being 2D or 3D for PC and 3D for mobiles.
MMO world are very large and cannot be held in memory as a whole: for this reason, it must support streaming, that is required parts are being downloaded from servers on-demand.
The world is made of a grid of square modules: Each module is tiled-based in the x, z plane — 64×64 quads —, the altitude y is taken from a height field image.
As the world is potentially very large, a tessellation system generate primitives on-demand,
The collision data-structure is similar to the one for the geometry — with a LOD system — and are sub-sampled on-demand, JIT.
Obviously, a cache system assists the tessellation and sub-sampling processes in order to maximize memory cache system and minimize computation.
otherwise I use
As you may already know, I have always used extreme programming to develop my softwares. And once again, I defined a series of micro milestones throughout the development stage.
This way, every week I can show off or demo the technology to the numerous visitor I daily have at my desk! In addition, fitting my own weekly objective, I can demonstrate new features or improvement visually every week.
I started off with existing assets, my Mario™ database, as place holders. They are 2D sprite projected on a transparent cube, as you can see on the first screenshot on the right.
For the background I firstly used a zero-height field which makes my ground looks like a pure plane. But of course, it's made of 4096 quads in order to keep with the real speed performance.
Each square is mapped with a tile taken from a relatively "large" tile map, each quad can have any rotation or flipping operation.
Based on the experience I had for MMO design — and mainly on the previous system — I decided to develop a brand-new protocol that would have the following features:
* high performance: we need to handle hundreds to thousands simultaneous connected people
* high extensibility
* built-in diagnostic system: the system must enable any human or software administrator to detect and prevent drop of performances — due to overload — and future failure such as a hub or bridge shutdown.
* built-in auto-optimization and auto-fix system: the system must auto-regulate bandwidth usage and have fallback systems in order to overcome any failure automatically
Additionally, the system must nicely handle software cycles as follows:
* automatic software update, without requiring service shutdown,
* game data deployment, still without requiring any service cut.
All these requirements have lead to the creation of both:
* the SLLTP protocol: data packet binary protocol with built-in routing functionality. It is well suited for wireless network as long as internet or lan connections.
* a non-concentrated system architecture where each server module has it's own independency, with a software buses that enable communication paths to be created on-demand.
Server modules have built-in hot-upgradability functionality: if an upgrade is available, a module can upgrade itself in a snap, without requiring the infrastructure to stop. Once up and running again, it connects to the bus and resumes.
The SLLTP protocol enables for administration communication between module. A set of control messages can go through the MMO in order to demand things to be done to a set of others module or a given target.
This is the base for diagnostic: administrators can connect from one point to another one, including game clients running on an handset. You can for instance require a module to exports its internal terminal to your screen...
This human communication goes thru the chat console from a game client or a server module front-end.
A large set of commands is available:
* whosthere, whois, shutdown,
* ping, traceroute, uptime,
* export console, export graphics,
* spawnbot, etc.
When a client or a server module tries to connect to the infrastructure, it is first routed to authentication servers, in order to determine rights and access.
Once it is granted, the system requires the load balancing server to chose an IP and a port to an available machine that could take the connection: the IP and port is returned to the client or server module along with a temporarily session ID.
Upon reception, the module disconnects and reconnects to the given IP and port and authenticate itself with the session ID before the session expires.
This solves most of the load balancing issue.
One of the biggest issue in MMO is to synchronize event over the network in a manner that it is unique for all clients and server modules.
In my case, I wanted any player to steal the flag that an opponent could hold.
Unfortunately, instant notification does not exist other a network because the time it takes to transports a piece of information — the latency — is non null. As a consequence detecting a collision in a module at time t are not likely to be detected at the exact same time in an other module that is supposed to handle the exact same situation.
A consequence of that could be a terrible switching of the flag between the two cars during the short period of time where they collide each other.
I implemented a notion of hierarchy of message that can be sorted by any module attached to the bus. This solves most of the difficulties.
Having hundreds of connections at the same time yields thousands of messages throughout the buses in the infrastructure.
Bandwidth usage becomes very high and we observe network jam. Server modules — especially the hub - are then saturated, yielding continuously increasing latency.
The phenomenon yields to overloaded CPUs and message saturation, blocking the whole infrastructure.
In order to limit that disastrous effect, I've implemented three main solutions:
* Buses and modules continuously control the effective bandwidth they have access to. When performances get down, they limit outgoing message emission
* The SLLTP protocol has a built-in support for "importance of message" which enable the infrastructure to reorder message delivery
* Hubs implement a saturation mechanism: when a hub detects that the bus is going to overload, it decides to trash less important messages in order to de-saturate connections. The rate of trashing depends on the actual realtime jam situation.
The final wireless 2D version is using two sets of graphics that you can choose from the starting menu. 2D graphics have been designed from real 3D mesh that we pre rendered in 3DS Max. The resulting images are cleaned on a pixel by pixel manually.
Regarding the perception of quality the game gave to players, I noticed that lags incurred by dramatic varying network conditions was really damaging the perception of fluidity.
For this reason I have created a set of funny icons that I've asked my artist to realize. Each icon symbolize the network conditions, a system very comparable to the weather condition iconography.
Should the network condition change at any moment, a sliding icon replaces the former one to indicate the new conditions, as shown below where you can see the "sunny face" which denotes the "good, or no problem" state.
I also innovated in the field of car control. Most car games — if not all — choose a relative control system, where you are place "in" the car: you must press the left arrow to go on the left, regardless of the actual orientation of the car on the screen. Maintaining the left key pressed makes you draw a circle.
Rather, I've programmed an absolute system, where if you press left, you go on the left of the screen. Unlike the previous system, maintaining the left key pressed makes you draw a line in fine.
Clearly, this system yelds superior playability, and moreover, is better adapted to casual players because it is natural.
For this specific part of the project I partner-shipped with NVidia in order to have early access to their forthcoming GoForce 4500/4800 3D graphics hardware accelerator.
In June 2005, I received an electronic board featuring a Motorola phone running under a linux kernel and featuring a GoForce 4500 chip and equipped with a QVGA color screen.
The product has to be connected to a PC running a 7.3 red hat in order to have access to network and disks.
An ARM C compiler has to be used to produce OpengGL ES 1.0 C code; unfortunately, at that time no VM with JSR-184 was available. Few weeks later, NVidia sent me a C upgrade implementing OpenGL ES 1.1 and giving access to shaders which I experimented as well.
I created a stripped down port of my J2se code, and in a couple of days, I had 32 cars running in the world. I could experiment the hardware and benchmark real-world performances.
By the end of the year, I received a pre-version of the W800i which is equipped with the same GoForce 4500 chip.
Unlike the board, the handset is running a Java Virtual Machine and implements JSR-184.
After many experiments, I was convinced that many facts was dramatically impacting the performances I had on the board with straight C:
* First, Java is a source of performance losses. Ease of use, flexibility and its "program once, run anywhere" motto has a price.
* Second, most of the phones have poor memory access throughput, and this mobile — though comparing well against others existing handset — do not avoid this penalty. Note that this handset implements JP-6, the SonyEricsson Java Platform. By the time of writing — June 24th 2006 — SonyEricsson just released the new JP-7 platform which dramatically improve this area.
On April 2006 a Barcelona, MobileScope/In-Fusio demonstrated Project X on its booth; people were invited to play a catch-the-flag persistent game with either their mobile phones or the PCs.
Note that I used my favorite set of video game heroes: mario all stars! All thee place-holders will be replaced as soon as possible by final artworks.
h3. Network statistics
Implementing networking support requires you tune bandwidth usage and minimize latency issue. Realtime statistics help you a lot in this area.
I developed a dead reckoning system in order to smooth path prediction for other player's avatar.
Here you can see some path trails.
During 3D model support development, I used some visual augmented reality to improve debugging and fine tuning.
As you can see, I've developed a set of embedded tools that continuously profile both networking performances and topology.
The circle — where the HUB is at the center — shows the distance from which the client is in term of roundtrip (in milliseconds).
A couple of shaders have been added for handling special rendering. A new warp replaced our former place-holder.
For the ground I've added a texture splattering system for giving details without consuming too much memory.
The world has been updated with “real” 3D altitude for the ground. Water have been integrated, with a nice light-reflection system.
My favorite yellow duck has been integrated as well, with the programming of a physics engine (floating, falling, popping for the duck).
Cars now benefit from a real physics capable of : collisions (car/ground & car/objects), car jumping over hills, skid and lose of control.
In order to make it even more fun, jumps are really expressive and the car shakes its "nose" when you try to climb big slopes.
The addition of the final artworks — that I wanted to be very rough and non-realistic — gave the final touch to this demonstrator of the ProjectX technology and service.
As a first attempt full 3D to support MMO connectivity, I recycled a Mario Kart™-like engine that I once wrote for the J2me platform.
I've added the Connectivity set of classes and I could see the numerous avatars wandering in the world…
The 2D versions derives its code from the desktop version for connectivity. The rest of the game is written from scratch, using MIDP2.
This version features a top view camera, closer to the final version.
For the purpose of rapidly demo this version, I used Midtown Madness car sprites as placeholders.
I have received a devkit from NVidia which is a Motorola linux-based mobile phone prototype. It embeds a GoForce 4500 chips capable of 1.5Mtri/s.
I've developed from scratch an opengl 3D rendering engine in C to benchmark actual usable performance. It rocks!
I've put 32 cars, 4K tri each. Plus the background (3D plane of 4096 quads). It sustains 30fps.
Of course, the final game has to be written in java j2me and must support JSR-184.
Making games on mobile is somewhat diverting: on one hand you have a very limited platform to work on — which is extremely demanding as you lack almost everything: memory space, speed performance — and on another you have to make people play to your game despite all these limitations.
You find yourself being back in time, while programming `80s systems: ZX-81, TO7, etc…
But things have changed: playing gorgeous titles on PlayStation and Xbox is common. How could people play '80s titles nowadays?
Trying to answer this question leads you to a less common question: what makes a mobile phone the unique play station when compared to others?
Connectivity! In space — you connect to others players — and in time — you "wear" your phone almost 24/7.
Connecting players in 2004 was something challenging. You want people play altogether, regardless of their actual handset technology — java, Brew, Symbian, etc. — and also regardless their operator.
Most of the available system widely supports HTTP connections. Sockets were not always available: for example with java, only MIDP2 defines a TCP support, but unfortunately, implementation was optional.
The system had to use HTTP support then. Unfortunately, performances of HTTP connections are not really impressive. Though HTTP is an applicative protocol that lies on top of TCP/IP, it only defines a peer-to-peer connections.
Peer-to-peer basically means that for connecting from one end to the other end, intermediate routers or servers can relay the connection to the next "hop" by choosing what type of HTTP connection is the best suited for it.
All these "hops" are time consuming. Each per creating a new socket to the next hop, and potentially check and rewrite HTTP headers.
As a result, roundtrips performance drops dramatically when compared to direct TCP connection.
Another issue of importance is certainly the radio to internet part of the connection.
On the operator side, you don't really have access to the infrastructure. The only thing you know is that your connection is routed by a special equipment called an APN. It is the connection to the Internet, think of it as an HTTP proxy.
Behind it, on the operator side, IPs are local. When on the internet, you basically have the operator's public IP.
Some operators block packets that would get to "unknown" server IP, and in that case, you have to deal with the operator in order it lets you pass through the APN, down to your server.
When HTTP connections arrives to your front server, IPs are usually not a mean to authenticate the player. At most, you can know from what operator he comes from, if you recorded all possible APN IPs somewhere in a database.
Some APN, gently add an HTTP header extension which tells what IMEI number the mobile has; but you shouldn't count on it, as this is the exception.
You have to rely on a custom, different mechanism of your own, such as a login/password system or a smarter thing.
HTTP connection usually gives you an average roundtrip on GPRS networks of about 2400ms. Yes, you read it right: 2 and a half second to send a small message and get a response from the server.
If you tell me you can't do any game with such a latency, well my answer is: you're wrong.
Of course, you cannot design a racing car game, where you expect realtime collision detection over the network. 2400ms of roundtrip is of oder of magnitude to make such a game.
But if you design a game that do not offers high sensibility to latency you really can make funny games.
For example, we designed a multi-player version of the Microsoft/Rare Mr. Pants license.
Basically, Mr Pants is an action puzzle game where you have to make rectangular blocks with the pieces you get form the game by rotating them appropriately in a limited time.
Every time you complete a block, the surface it takes is cleared and you collect bonuses and points.
Adapting the game to the above-mentioned constraints, the team designed a game play that wouldn't suffer from very high latency.
First, the screen of the opponent is also displayed on the player's screen in the top right corner, in order for you to look at his progression.
Second, when you complete a block, the game send a malus to the opponent, making its time limit to shrink a little bit, thus increasing the match pressure.
This way, it does not that matter wether the malus is coming "in-time" or not. What it counts is that it arrives!
As a result, the multiplayer version is really funny, and maintains the interest over time.
It’s Mr Pants features the famous Microsoft and Rare’s license.
The mobile version is derived from the Rare's Nintendo GB Advance It's Mr Pants sku. The multiplayer version is only available from In-Fusio for most of the handset types.
"It's Mr Pants" is © and ™ or ® by Microsoft Corp. and Rare Limited and published by THQ — The mobile version is designed, developped and published by In-Fusio SA.
In-fusio created the business of the downloadable apps on mobile phones back in 2001 with the ExEn technology: it embedded a very compact Java VM from Shlumberger plus a thin native API to provide maximum performances. This was before J2ME even existed. EXeN was deployed on more than 200 mobile phone models and brands.
In 2003, with the emergence of the first smartphones, I was in charge of creating a realtime 3D engine API and native code to embed into the EGE, the successor of ExEn.
This was the first time the engine ran on an actual handset: this was on Thursday January 29th 2004 à 09:32am.
Jouez mobile technologies engagées : Après des débuts timides, le jeu vidéo sur mobile est en plein boom : "En un an, nous sommes passés de 0 à plus de 200 000 jeux téléchargés par mois", s'enthousiasme Anne-Laure Descleves, responsable communication chez Gameloft, un éditeur de jeux sur portables. Même son de cloche chez In-Fusio, le pionnier du secteur, qui affirme compter un nouveau joueur toutes les cinq secondes. Un succès dû à la multiplication des téléphones Java. On désigne ainsi les appareils capables d'exécuter des applications programmées avec J2ME (Java 2 Micro Editionl. une version allégée du célèbre langage de Sun Microsystems. Cette techno change tout car elle permet de concevoir des jeux dignes d'une console de poche, ne pesant pas plus de 150 Ko et téléchargeables en moins de deux minutes sur les réseaux GPRS. Ainsi. Gameloft propose, dans son catalogue, qui compte une trentaine de titres, des adaptations de jeux pour consoles et PC comme Splinter Cell, Prince of Persia ou Rayman 3. Si les jeux sur mobiles se doivent d'être simples et faciles d'accès, des applications plus évoluées sont déjà disponibles. Sur Splinter Cell, par exemple, il existe des modes experts qui permettent de pratiquer l'infiltration plutôt que le tir. Et sur le jeu de course automobile IF Racing, le dernier titre d'In- Fusio, on peut courir contre son "ghost", une voiture fantôme indiquant l'ancienne position du joueur, et l'envoyer à un ami pour qu'il puisse comparer ses performances. La prochaine étape sera le passage à la troisième dimension. Mais avant d'en arriver là, il faut que les API (interfaces pour la programmation d'application) de jeux Java pour mobiles, baptisées MIDP (Mobile Information Device Profile), évoluent suffisamment et que les constructeurs se mettent d'accord sur un standard. "Pour l'instant, chacun y va de son implémentation propriétaire", se plaint Yann Mondon, le directeur de la communication d'In-Fusio. Voilà pourquoi cette société met en avant sa technologie ExEn, une machine Java optimisée pour le jeu, dont la prochaine mouture, prévue pour la fin de l'année, proposera un moteur 3D dédié, des fonctions sonores et multi-joueurs avancées ainsi que des animations Flash. Si Alcatel, Panasonic, Siemens, Sagem, Philips et Trium ont intégré ExEn sur leurs appareils, d'autres constructeurs comme Nokia ou Motorola y sont rétifs. Mais, à terme, tout ce petit monde devrait utiliser les mêmes technos et la 3D se généralisera. Quant aux jeux multijoueurs, ils s'apprêtent à débarquer : Gameloft met la dernière touche à une nouvelle version de Splinter Cell qui permet de jouer à plusieurs en mode coopératif. Elle sera disponible à Noël sur la N-Gage de Nokia, un hybride de téléphone et de console de jeu, dotée d'une connexion Bluetooth. A.M.
Gameloft, l’éditeur de jeux sur mobiles, prépare une version 3D du jeu XIII. inspiré de la célèbre bande dessinée, pour la fin de l'année.