Adding object class method in Python at run-time

Posted on March 20, 2010 by Wolverine

Just a quick note about Python. I’ve recently tried to find answer for adding methods dynamically to Python objects created at run-time. I’ve seen solutions that use import new and then new.instancemethod, but it seems this module is deprecated and isn’t going to make it in Python 3.0. However the new solution seems fairly simple and elegant (this is what I really like in Python). So basically you use types module and then create a method with types.MethodType. Here’s quick example how you can use it.

>>> import types
>>>
>>> class Foo(object):
...    def __init__(self):
...        self.x = 10

Now if you want to add method dynamically at run-time, you create a function out of class context.

>>> def show(cls):
...     print cls.x

You should create a first parameter as cls, instead of self, because the bound-method will be using cls attribute as object instance designator. You can use whatever attributes for a function you need, simply remembering that first parameter is always an instance. After you do this, to assign our new function as an object method, we create our object and then assign the function to object as it’s bound-method like this:

>>> bar = Foo()
>>> bar.show = types.MethodType(show, bar, Foo)

What types.MethodType does is basically take parameters: “function to bind”, “object instance”, “object class”. It creates a bound method which we must assign to our instance with instance_name.method_name. And voila! We can now use our new method with object created at run-time:

>>> bar.show()
10

That’s it. Really simple and elegant.

Posted in Coding, Programming, Python | Leave a comment

Game compiling headaches under Linux

Posted on January 23, 2010 by Wolverine

I’ve been using Linux for almost thirteen years now. I’m a fan of this system and I can really say I’ve seen it’s evolution from mostly developmental state to polished system that can be used on daily basis for the power user. If you’re like me and you use your computer mostly for work, especially in areas of programming and network administration you know this operating system is a powerful tool generating more productivity if you know how to manage it well. This needs substantial amount of learning and learning curve for Linux is steep. But in the end you are able to do your work and daily routine tasks quicker.

Linux is not an ideal system and there’s certainly a room for improvement, especially in usability and user experience. Hardware incompatibilities are not such a big problem like they were few years ago. Software is abundant and you can find it for almost all, but the most specialized tasks. Like with every other operating system there are pros and cons of using Linux, but this is not the scope of this article. Perhaps I will write something about this in the future. Like most other *nix systems, Linux was built with networking in mind. This is the area, where I can honestly say that most of the time it outperforms other OS. Internet communication is smooth and despite some minor problems you can do whatever you want on the Internet and with computer networks.

There’s however one area where Linux is not so good. If you like playing computer games, Linux is definitely on the dark side when compared to Windows or even consoles. There are some commercial games ported to this OS of course and there is software like Cedega or Wine to emulate some games written for Windows, so they can run in Linux, but there are some problems with it. However if playing is not your primary focus for using computer and you’re just like me that from time to time you just want to relax and waste some time with rather simple games and games from the past, Linux offers you quite a good choice.

There are many open source games out there on the Internet. Various Linux distributions have them included out-of-the-box. There are many really funny and engaging games just for free, ready to be downloaded from the Internet and installed on your Linux system. Many sites like for example Happy Penguin offer a database of thousands of free games for Linux. If you’re using popular distribution like Ubuntu, Mandriva or Redhat, there’s good chance some of those games are already in the online repositories of those distributions. If not, there’s also a good chance you can download packages for your distro from the home page of the game.

Unfortunately, some games that are in development or their authors that don’t have time for creating packages for various distributions are only giving you game sources. And this can really give you a hard time if you are not a power user or at least you know how the compilation works and should be used. It’s true not only for games, but for software in general. The problem with software compilation on various platforms and architectures, under different Linux distros could be detrimental for your mental health if you are not expert user or even a programmer. This article is hopefully going to ease your pains, but it cannot bring relief to all headaches caused by compilation (games and software in general).

So. You’ve just come across an open source game you really, really want to play. You’ve checked online package repositories for your Linux distribution and found nothing or some really old version. Then you’ve checked the website of the game, but there are no packages matching your distro. There’s only this “source” thing for download. And it has this creepy .tar.gz or .tar.bz2 extension you really don’t know what to do with. But you give it a shot. So you download this .tar.gz/.tar.bz2 file and in the meantime you’ve learned it’s a compressed TAR archive file. What you should do now is to put your archive file in some directory like for example /home/user/games.

And now what? Well. Because it’s an archive you must unpack it. If you know how to do it just skip this paragraph. Unpacking is easy. There are some programs like Ark in the KDE that will offer you a graphical interface for unpacking this archive. Just remember to unpack it within subdirectory. If you prefer console (which most of the time is much faster for doing things like unpacking and compilation cannot be done without console usage) you can go to the directory where your archive file is (e.g. /home/user/games) and then issue command like this (if your archive ends with .tar.bz2):

tar xvjf archive_file.tar.bz2 .

or if your archive file ends with .tar.gz:

tar xvzf archive_file.tar.gz .

Remember that you shouldn’t omit space and the dot “.” at the end of this command. I won’t explain what it means, because you can google it or just read the manual. If you are really curious it means to extract archive file, creating directories, showing files extracted using specific decompression routines (GZip or BZip2). This should unpack the archive to the directory that’s usually the same as package name, e.g. superbgame-1.0.3.

When you have your source files extracted from archive you should enter the subdirectory with extracted files. This is where the compilation process occurs. So finally we can begin discussing what’s it all about.

First of all you should look for text files like README and INSTALL. You should read them, because most of the time they convey very important information regarding software compilation and installation process. They can specify prerequisites and requirements for compilation. You should read it carefully, because there are example commands and things you should check before you try to compile your newly downloaded software. If you stick to it usually compilation process will be smooth for you.

So let’s stop here for a moment. What is this compilation anyway? Compilation is simply a translation of programmer written code in so called programming language to native machine code. Of course it’s more complicated than this, but you shouldn’t bother. Native machine code allows your computer to read it’s instruction sets, effectively running a program when it’s executed. That’s all you need to know for now.

What are requirements for compilation? Mostly a compiler, build tools and so called development libraries. A compiler is a program which translates source code written in programming language to machine code. Depending on programming language the game is written in, there are various incompatible compilers. What it means? You cannot compile the game (or any software) with compiler written for another programming language. It’s like if you want the book translated from Chinese to English, using Spanish translator. You get my point. It won’t work. Compilers can also (and most of the time are) be incompatible even if they were written for exactly the same programming language. It can be explained as getting book translated from Cantonese to English, using Catalan translator. Of course most of the code will be translated correctly, but some words could be translated inappropriately. And unlike with book translation, if the whole source code isn’t translated correctly, it cannot be compiled and therefore used.

So let’s get back to our compilers. Most games for Linux are written in two programming languages, that are very similar (in fact sometimes they are treated as one) – C and C++. There are of course some games written in other languages like Java, Python and various others, but those are not in the scope of this article. We assume that most of the time games for Linux are written in C/C++.

Most widely used compiler on Linux based distributions is so called GCC. It’s an abbreviation for GNU C Compiler. This compiler is a set of tools that allow translation of various programming languages to machine code, but for clarity’s sake let’s assume it’s for C and C++ only (with exception that compiling C++ with GCC needs additional steps, but I will get to it later). Most of the time GCC is installed with your Linux distribution from the start. But you have to find out for yourself if it’s installed on your system, usually through software package management program. If it’s not you’ll have to install it and compiler’s dependencies (additional software). You can quickly check if it’s installed by issuing this command in console:

gcc -v

or

cc -v

And it should give you output like this at the end (among other information):

gcc version 4.4.1 (GCC)

If you get “command not found” error it probably means you have to install the compiler. This output is important. It tells you which version of compiler you have installed. Check out requirements stated in README or INSTALL files in the subdirectory where you unpacked sources of your game. If those files doesn’t state which version to use, most of the time it means you can use the latest version or even don’t bother about compiler version at all.

Remember. This will only give you information that you have GCC compiler installed. If the game needs C++ you’ll have to make additional check if it’s installed too. You can do this in a similar manner as with standard C compiler:

g++ -v

or

gpp -v

And it should give you similar or exactly the same output as standard C compiler. If you get “command not found”, you’ll have to check if it’s installed. Usually the package is named “gcc-c++” or “gcc-cpp”. If you’re sure you have this package installed and commands above are still giving you “command not found” errors it probably means, that your distribution is using “gcc” command for compilation of both C and C++. So you should be fine.

Ok. So we’ve discussed compiler. Of course there are other compilers like for example Dev-Cpp, but most games are written with GCC, so I’m not bothering to explain them. If you really want to compile something with other compiler you’ll have to ask for help elsewhere. Most of the time tips that I’m talking about could be applied to other compilers.

The second most important thing needed for compilation is so called build system or build tools. Those names are interchangeable (to some degree). What is a build system? Simply speaking it’s a set of various tools used for automating build process. Build process is a complete compilation process that encompasses all steps within compilation. Compilation consists of various steps like preprocessing, compiling and linking. There are various more or less popular build systems. The most commonly used in Linux is so called automake (which includes packages called m4, make, and various others). This is the most popular build tools set. From my experience 80% of all games and software projects uses automake for building final software package.

It consists of few steps, usually:

./configure

make

make install

Usually those steps are described extensively in README or INSTALL file. Issuing those commands in a build directory i.e. the directory source files reside, most of the time will work and give you compiled and installed program. Automake is by far most popular build system for games, from what I have seen.

There are of course other build systems like Scons for example. Scons needs Python to be installed on the system and it’s build process is somewhat different. It’s gaining popularity, so I’m mentioning it here. Building with Scons or any other build tools set will not be covered here. I recommend you to read installation instructions for your software. It should explain steps needed to compile a program with those tools. If not I recommend reading manual for those build tools.

Let’s focus on first command. The “configure” command. This command checks system for dependencies needed to build your software. It usually takes few parameters which are explained when you issue:

./configure –help

This will give you list of all available options. Some software can be configured to use extended capabilities or even make it optimized (software runs faster), but that depends on installed development libraries in the system. Let’s stop here for a moment.

What are those development libraries? Development libraries are “archives” (libraries) of routines used by your software. They can be used by many programs at the same time and are shared between programs. If program needs for example manipulation of JPEG pictures it would use routines from library that supports JPEG reading, manipulation and operations on JPEG pictures. Libraries are usually a set of functions that perform specific tasks.

Imagine for example a company. There’s a boss (that’s our program) and there are workers responsible for specific tasks (that’s our libraries). Worker can perform only a specific task (like for example financial accounting, carpentry or using a machine). That’s what libraries do. They do specific work for their boss or manager. Manager can be thought of as a library that uses other lower level libraries. When you run a company, for example building tables, the boss is responsible for running the company. He has various managers, who have responsibility for clients, finances, production and logistics. Those managers are libraries that our software (the boss) utilizes and which in turn are utilizing lower level libraries (workers) responsible for accounting, wood delivery, carpentry, warehouse operations and delivery to clients. Workers (low level libraries) depend on managers (medium level libraries) which in turn can be controlled directly by the boss (our software) or other managers (medium level libraries, so called middleware).

Software is a hierarchical structure just like the company. Almost all software uses some kind of libraries. Games especially depend on libraries that perform low-level graphic showing, audio management, input controlling and many other tasks. That’s why they need libraries someone else has written, because if someone would like to write a game only with his own code he’d probably do nothing else through his life, but write this one game. There’s no point. Programmers are specializing in some areas and expert in graphics will do graphics better than the person who has absolutely no knowledge about it. Libraries are sets of functions that can be utilized in various software without even knowing how exactly the specific task is performed. All that counts is expected output.

When compiling the code, various parts of the software are using so called “functions” from development libraries. That’s why almost all, besides the least complicated applications, depend on some libraries. Required libraries are mostly specified in README or INSTALL files I have mentioned. They can have various versions and hardware architectures they are built (compiled) for.

Most games in Linux depend on libraries like SDL (Simple DirectMedia Layer), Allegro, OpenGL (Open Graphics Library), OpenAL (Open Audio Library) and various other. There are many libraries responsible for graphics rendering, audio mixing and playback, input controlling, network routines, mathematical calculations, user interface drawing, system and hardware management and others. Usually those required libraries are specified.

Requirements are checked for availability when you run “configure” program from build system. It will tell you if something is missing by generating output like: “cannot find library libSDL_image” or something like this. This means you have to install the library so it will be available for build system. However. You may already have this library installed and configuration step of compilation is still not working. This means that you have only compiled library installed. Unfortunately software requires for compilation so called “header files”. Those header files are specifying what functions each library has. Compiler needs to know which library exposes specific function (routine). Header files are doing just that. They are exposing interfaces to the library functions, specifying exactly what should be given to function by the software (function input) and what should be expected in return (function output). Compiler knows that specific library exists in the system only by scanning available header files.

In most currently available Linux distributions those header files for a library are separate packages from the compiled library itself. Most of the packages within Linux distributions have additional package for each library ending with -devel or -dev suffix. When trying to compile your game (or any other piece of software) you have to make sure those development header files are installed alongside the library your software uses.

When those requirements are met you should be able to compile your game smoothly. However specific games can use either newer or older versions of library which are incompatible with application you want to get compiled. Sometimes when compilation fails because of unmatched major version of library, you can try to symlink the newer/older version. Most of the time it will work. For example you have /usr/lib/libSDL.so.1.3 and your game requires /usr/lib/libSDL.so.1.2. You can try to issue:

ln -s /usr/lib/libSDL.so.1.3 /usr/lib/libSDL.so.1.2

Sometimes it will work and sometimes it won’t but after all it’s worth a try. Sometimes you’ll have to compile older version of library by yourself. Remember you shouldn’t add –prefix=/usr option to configure command when you do this. Use another prefix like for example /usr/local/name_of_library_and_version. This will ensure stability for your system and it won’t allow to accidentally overwrite the version of the library you’re using system-wide (and probably many applications in your system depend on this version of library).

The –prefix option for configure command tells where the software should be installed. Most of the time into PREFIX/bin goes executable files, PREFIX/lib goes library files and to PREFIX/share goes data files required by the software (but sometimes data files can go into PREFIX/lib, depends on programmers fantasy). If you want to install your software system-wide you should always use configure –prefix=/usr.

That’s all for now. The next note will explain things further and I will explain more advanced concepts and also try to show you some tips and tricks regarding the compilation process of the software. I hope you enjoyed reading this article. Feedback is appreciated.

Posted in Programming | Leave a comment

Rabbyt and GTK GLExt example

Posted on September 28, 2008 by Wolverine

To clarify my previous post a little I have made a simple example on how to use Rabbyt library with GTK GL Extension. Hope this helps. Cheers all…

# /usr/bin/env python
#
# Rabbyt GTK GLext test
#

__author__ = 'Karol "Wolverine" Tomala'
__email__ = 'karol (at) karoltomala (dot) com'

# Imports
import sys

import pygtk
pygtk.require('2.0')
import gtk
import gtk.gtkgl
import gobject

import rabbyt

# Globals
angle = 0.0
sprite = None
zfactor = 0.005
zoom = 1.0

# Drawing function
def draw(glarea, event):
    # Get surface and context
    glcontext = glarea.get_gl_context()
    gldrawable = glarea.get_gl_drawable()

    # Start opengl context
    if not gldrawable.gl_begin(glcontext): return

    # Actual drawing
    global sprite, angle, zoom

    # Clear screen
    rabbyt.clear((0.0, 0.0, 0.0))

    # Render sprite
    if sprite is not None:
	sprite.rot = angle
	sprite.scale = zoom
	sprite.render()

    # Flush screen
    gldrawable.swap_buffers()

    # End opengl context
    gldrawable.gl_end()

    return True

# Resizing function
def reshape(glarea, event):
    # Get surface and context
    glcontext = glarea.get_gl_context()
    gldrawable = glarea.get_gl_drawable()

    # Start opengl context
    if not gldrawable.gl_begin(glcontext): return

    # Get widget dimensions
    x, y, width, height = glarea.get_allocation()

    # Reset rabbyt viewport
    rabbyt.set_viewport((width, height))
    rabbyt.set_default_attribs()

    # End opengl context
    gldrawable.gl_end()

    return True

# Initialization function
def init(glarea):
    # Get surface and context
    glcontext = glarea.get_gl_context()
    gldrawable = glarea.get_gl_drawable()

    # Start opengl context
    if not gldrawable.gl_begin(glcontext): return

    # Get widget dimensions
    x, y, width, height = glarea.get_allocation()

    # Reset rabbyt viewport
    rabbyt.set_viewport((width, height))
    rabbyt.set_default_attribs()    

    # Get sprite variable
    global sprite

    # Load sprite
    sprite = rabbyt.Sprite('sprite.png')

    # End opengl context
    gldrawable.gl_end()

    return True

# Idle function
def idle(glarea):
    # Get vars
    global angle, zoom, zfactor

    # Update angle
    angle += 1.0
    if angle > 359:
	angle = 0.0

    # Update zoom
    if zoom > 10 or zoom < 1:
	zfactor = -zfactor
    zoom += zfactor

    # Needed for synchronous updates
    glarea.window.invalidate_rect(glarea.allocation, False)
    glarea.window.process_updates(False)

    return True

# Map events function
def map(glarea, event):
    # Add idle event
    gobject.idle_add(idle, glarea)
    return True

# Create opengl configuration
try:
    # Try creating rgb, double buffering and depth test modes for opengl
    glconfig = gtk.gdkgl.Config(mode = (gtk.gdkgl.MODE_RGB |
					gtk.gdkgl.MODE_DOUBLE |
					gtk.gdkgl.MODE_DEPTH))
except:
    # Failed, so quit
    sys.exit(0)

# Create window
win = gtk.Window()
win.set_title('Rabbyt GTK test')

if sys.platform != 'win32':
    win.set_resize_mode(gtk.RESIZE_IMMEDIATE)
win.set_reallocate_redraws(True)

win.connect('destroy', gtk.main_quit)

# Create vertical packing box
vbox = gtk.VBox()
win.add(vbox)
vbox.show()

# Create our glarea widget
glarea = gtk.gtkgl.DrawingArea(glconfig)
glarea.set_size_request(300, 300)

glarea.connect_after('realize', init)
glarea.connect('configure_event', reshape)
glarea.connect('expose_event', draw)
glarea.connect('map_event', map)

vbox.pack_start(glarea)
glarea.show()

# Create button quit
btn = gtk.Button('Quit')
btn.connect('clicked', gtk.main_quit)

vbox.pack_start(btn, expand = False)
btn.show()

# Show window
win.show()

# We're finished, enter main gtk loop
gtk.main()

# Go get some pizza :)
Posted in Coding, Game Programming, Python | Tagged , , , , | Leave a comment

Python, Rabbyt and GTK GLExt

Posted on September 26, 2008 by Wolverine

It’s been a while since I wrote something actually useful to anybody here. I’ve decided to take down my personal posts, they are of no interest to anyone except me. Life sometimes sucks for me big time, but I’m moving along… So. Enough of this crap.

Recently, due to almost completing my work project I had more spare time to shift focus onto game programming once again. Since it’s always been my dream to create a game I’d love to play, at the beginning of this year I decided to give it a go and try writing game engine and then based upon it, develop a 4X game called Galaxy Lords.

So for the last few days in my spare time I moved back to work on the SpaceX Engine. I have designed brief structure for the engine, much thinking and reading bits of information regarding game engines spread around the net. Then I moved to diagramming in UML parts of the engine. Since I had another work projects for the last few months, I have gained few more experience points in regard to Python programming. Two days ago I have designed engine scene graph components. The structure of rendered scene and much inclusions of the object hierarchy gave a clear view that I would need few editors, especially for textures, sprites, animations among other things. Since the internal scenegraph is hierarchical, it would be an excellent choice for a descriptive file format if I went with XML. Of course I could edit those things by hand, but honestly for animations and effects it would be too much wasting time. Hand editing of XML animation description files would be simply too much overhead, except for very simple things. So I have decided to write a full-blown editor for SpaceX engine.

Since I’m using Python 2.5, 64bit Linux and the engine relies heavily on OpenGL via Rabbyt library I evaluated few possibilities for OpenGL canvas widgets from different GUI toolkits available. I had few things in mind. The toolkit must be multiplatform, must be relatively stable, be open and free of charge and have widgets for OpenGL context. After evaluating few possibilities like wxPython, PyQT4, Fox Library I ended up with GTK2.

However GTK2 does not have an OpenGL canvas element out of the box. It used to have one called GLArea, but it’s deprecated for a long time now. I needed a widget that could be easily integrated with my existing SpaceX code to leverage it’s power, not needing to rewrite things especially for canvas widget. After all the widget should have OpenGL context and nothing else.

Rabbyt by default has support for PyGame and PyGlet libraries. They are fine when you are developing a fullscreen app like game, but for editor you ultimately need a very different thing. PyGame and PyGlet allows to create only window that has OpenGL context and they don’t allow embedding this window as widget in some kind of GUI toolkit. Last but not least, even if it was possible most editors need multiple OpenGL contexts to render the object from different perspectives.

So I’ve came across GTK GLExt which stands for GTK GL Extension. This extension allows you to create a canvas object with OpenGL context. I has also a nice Python wrapper. So I’ve began investigating possibility for Rabbyt compatibility with this extension. Suprisingly, browsing Rabbyt’s code it does not rely on PyGame or PyGlet for anything except texture loading. It’s almost pure OpenGL. So I’ve began examining the examples for PyGtkGlExt. After few minutes fiddling with it I had success with displaying Rabbyt through GTK! Actually yes, it’s pretty simple and very obvious.

I’ll try to explain how I did it.

PyGTKGLExt can be initialized in two different ways. I’m not sure if it matters or simply the second function is an alias to the first. You start by importing necessary things and setting all up.

import pygtk
pygtk.require('2.0')
import gtk
import gtk.gtkgl
import rabbyt
size = (200, 200) # Size in pixels

Additionally you can import OpenGL. Then you have to create a GL context configuration. This can be done like this:

try:
    glconfig = gtk.gdkgl.Config(mode = (gtk.gdkgl.MODE_RGB | gtk.gdkgl.MODE_DOUBLE | gtk.gdkgl.MODE_DEPTH))
except gtk.gdkgl.NoMatches:
    glconfig = gtk.gdkgl.Config(mode = (gtk.gdkgl.MODE_RGB | gtk.gdkgl.MODE_DEPTH))

If gtk.gdkgl.NoMatches error is raised it means that requested mode is not available, so we try without double buffering. For more information on the supported modes refer to GTK documentation. This is the GL context configuration which is then passed to widget.

After you have your imports you have to create a widget. It can be done like this:

glarea = gtk.gtkgl.DrawingArea(glconfig)
glarea.set_size_request(size[0], size[1])

Or like this:

glarea = gtk.DrawingArea()
glarea.set_size_request(size[0], size[1])
gtk.gtkgl.widget_set_gl_capability(glarea, glconfig)

This would create widget and initialize its OpenGL context capabilities. Ok. So we have our widget initialized. What’s next? Well we actually need to connect some events (or signals using PyQT terminology) to our newly initialized widgets. There are four events we are mostly interested in. They are as follows:

  • realize – used for widget initialization
  • configure_event – used when widget is resized
  • expose_event – used for actual drawing
  • map_event – used for mapping (widget is idle)

Ok. So now we know what those events are for. So what we do now? Actually we connect these events to functions. This is how it’s done:

glarea.connect_after('realize', init)
glarea.connect('configure_event', reshape)
glarea.connect('expose_event', draw)
glarea.connect('map_event', map)

After we did that we need to write actual functions. This is where our rabbyt integration occurs. Let’s start with init function.

def init(glarea):
    # Get OpenGL context
    glcontext = gtk.gtkgl.widget_get_gl_context(glarea)
    # Get actual OpenGL surface
    gldrawable = gtk.gtkgl.widget_get_gl_drawable(glarea)
    
    if not gldrawable.gl_begin(glcontext): return
    
    # Actual OpenGL initialization code goes here
    
    gldrawable.gl_end()
    
    return True

So this is our new init function. Actually it’s not used here at all, but can be used for setting some specific OpenGL attributes. In fact each of our functions will use the construct such as above. gldrawable.gl_begin(glcontext) allows to connect widget surface to OpenGL context. If you have two different OpenGL widgets I think it’s not a problem, because context switching will occur, but I’m not really sure if you wouldn’t need additional OpenGL context locking/realising mechanisms. I’m not also sure if it would be thread safe. But it’s up to you to find out.

Let’s get back to our functions. We create a reshape function. This is how it should look like:

def reshape(glarea, event):
    # Get OpenGL context
    glcontext = gtk.gtkgl.widget_get_gl_context(glarea)
    # Get actual OpenGL surface
    gldrawable = gtk.gtkgl.widget_get_gl_drawable(glarea)
    
    if not gldrawable.gl_begin(glcontext): return
    
    # Get viewport size
    x, y, width, height = glarea.get_allocation()
    
    # This is where Rabbyt initialization goes
    rabbyt.set_viewport((width, height))
    rabbyt.set_default_attributes()
    
    gldrawable.gl_end()
    
    return True

And that’s it really. Note that rabbyt.set_default_attributes() resets OpenGL state. Now we have setup our OpenGL context to work with Rabbyt. Widget may be resized and Rabbyt will adjust itself to new situation. So. How the actual drawing function looks like?

def draw(glarea, event):
    # Get OpenGL context
    glcontext = gtk.gtkgl.widget_get_gl_context(glarea)
    # Get actual OpenGL surface
    gldrawable = gtk.gtkgl.widget_get_gl_drawable(glarea)

    if not gldrawable.gl_begin(glcontext): return

    # Clear screen
    rabbyt.clear((0.0, 0.0, 0.0))

    # ACTUAL DRAWING GOES HERE!!!
    # ... code goes here ...

    if gldrawable.is_double_buffered():
        gldrawable.swap_buffers()
    else:
        # Remember this needs 'from OpenGL.GL import *'
        glFlush()

    gldrawable.gl_end()

    return True

So that’s it. Remember that if you actually want to draw something you need to place it above “if gldrawable.is_double_buffered():“. The last function we need is our map function. Actually there is also another function. So here it goes:

# First we start with importing gobject
import gobject

def map(glarea, event):
    gobject.idle_add(idle, glarea)

def idle(glarea):
    # ACTUAL COMPUTATION GOES HERE
    # ... code goes here ...

    # Invalidate window
    glarea.window.invalidate_rect(glarea.allocation, False)
    # Update window synchronously
    glarea.window.process_updates(False)

    return True

You update your computations (like for example calculating sprites rotation and so on) in idle() function. And that’s it!

That’s all you need to start your Python Rabbyt application to use GTK GL Extension. Fairly simple right? If you have any ideas on improvement or you noticed some quirks please feel free to post your views in comments. This would be greatly appreciated. Hope you found this tutorial useful. See you next time.

Posted in Coding, Galaxy Lords, Game Programming, Python, SpaceX Engine | Tagged , , , , , , , , | 2 Comments

Coding, Galaxy Lords and MOO2 Graphics Viewer

Posted on May 1, 2008 by Wolverine

Recently I’ve been doing some pretty hard coding. As you may have already seen in the previous post I’m currently the developer of social-networking site for relatively young company. This is my primary project currently, one that I do for living. I won’t speak of any details for now, but from the technical side it provided me with an opportunity to enhance my Python coding skills and I must admit, that I’m much better at it then I ever was before in any other language. I’ve learned much about web application frameworks and technologies for Python. Despite it being a business project, I’ve really got into the guts of Python. However one has to make some breaks from continuous work on one programming project and sometimes you have to take a break. But what better way to spend some time than spending it over just-for-fun programming projects?

Some of you maybe lost hope regarding my old Master Of Orion 2 Graphics Viewer/Editor and my 4X game Galaxy Lords. Well. If you were patient enough and you’re still looking at this blog sometimes I may have some nice news for your eyes. Last month I had some time and will to work on both of them. I’ve decided to try and finish as much as I can in the little spare time I have left. But let’s get to the point.

We’ll start with my game project – Galaxy Lords.  I’ve done some startup work at planning and writing basics of what is going to be, a full-blown game engine. First I started with creating a directory layout. The engine core consists of few basic elements:

  • Audio
  • Game
  • Graphics
  • Input
  • Math
  • Misc
  • Network
  • Physics
  • Scene
  • System

Those elements should be self-explanatory. They are basic subsystems of the game engine. Each subsystem consists of further classes, objects and functions related to it. But it doesn’t end only at directory and engine layout creation. What I did are also prototypes of functions and classes within those subsystems. Some things like various managers are already implemented. There is very core of the engine finished (like State, Progress, Event managers) and some mathematical functions like Vector operations. The library used for displaying graphics for this engine is choosen. It will be the excellent Rabbyt library created by Matthew Marshall. Networking support will be done probably with Twisted networking library and both audio and physics subsystems are yet to be determined. I have some ideas for this engine, but my knowledge regarding game engines is still somewhat vague, so I won’t get into details. If I manage to bring this engine to state of usability sometime I’ll certainly publish it for free.  But in my opinion it’s still some months off.

The second thing is my Master Of Orion 2 Graphics Viewer/Editor. Because my Python knowledge significantly improved over last few months, and after Siron published my descriptions of MOO2 gfx formats, I decided that I have to provide at least usable tool for modding graphics in MOO2. I’ve taken a look at previous code and decided that I should start from scratch. Tonight I’ve written from scratch almost seventy percent of previous viewer code. This time I had few things in mind when coding. First was to create clearly structured and easy to read code with multitude of comments for anyone who would like to learn how to read and write graphics in MOO2. Second was to write universal, reusable classes so they could be incorporated in another people’s projects. They should have obvious interfaces, be well commented and easy to use. Another thing was: “no more dirty hacks”. This time classes and functions are written until they are finished and polished, as well as retaining a wealth of functionality. This also allowed me to optimize the speed of those routines. Performance improvements are quite drastic, however they are nowhere perfect (especially considering pure C/C++ functions that could be reimplemented to do the same). However their speed is very good for activities like loading/saving, extracting and also for simple manipulations.

State of this project as of now is that I have a working Color, Palette (with fancy operations like mixing and blending) classes, there is fully functioning LBX archive extractor, compressor (though compressing LBX needs two more functions to work) and there is a Graphics class that has fully functional loading of graphics and animations and also have some functions for saving graphics (however there’s still some more work to do on saving). Classes and functions are very well commented, well structured and should be fairly usable for other programmers. They also have extensive error handling and checking. I tried to optimize them for speed whenever I could at the expense of some memory requirements. However overall reimplementation should result in more consistent package that can be reused.

Things that are still left are some functions to enable saving graphics and Converter to editable graphic formats like GIMP palettes and GIF/PNG files for graphics. There are some functions left that would also allow dumping graphics data in various raw formats like CSV, arrays, text files and other things that may be useful for future programmers. Graphical User Interface is also lacking because I have set my goal for providing useful base classes library and well tested command-line tool that works first, before I’m going to do anything with user interface and further improving usability for normal user. Hopefully I can finish this tool soon, demonstrate it’s ability in modding MOO2, write some tutorials and documentation so users and programmers alike could start messing with this tool, providing enough feedback for stable base. If that would be the case I’ll consider writing more user-friendly tool directed at general modding community.

I don’t have anything to show you currently, but I must admit that I’m quite proud of myself for reimplementing this project in such a short time and breathing new life to it. Perhaps if it was not for Siron’s post that somehow put pressure on me, this project would be still dead. However I expect to devote some time to it as my free time allows, so expect some more news on the topic in the near future.

Posted in Coding, Galaxy Lords, Master Of Orion 2 | 7 Comments

Description of MOO2 graphic formats part 2

Posted on March 16, 2008 by Wolverine

It’s been a really long time. Many things changed, some usual and some not so usual twists and turns in my life caused abstinence from writing here. I had time to think about everything and I’ve finally decided to passionately dive into information technology as I used to and screw everything else. These are two things I’m really good at. But enough of this.

I’m not really good at keeping promises, but this time I’ve decided that I have to do this. It’s been a long time since I’ve written part one describing MOO2 graphic formats. If you want this initial description of the formats check “Description of MOO2 graphic formats”. In that post I have described how to retrieve graphic header information and external palettes. This time I’m going to show you how to read internal and mixed palettes, and how to retrieve each animation frame. Some of you skilled hackers out there probably figured it out, but still there could be some of you who want to know. Here we go.

Just as a quick reminder. The header of the graphic file consists of: two bytes width, two bytes height, two always null bytes, two bytes with frame count, two bytes of delay between frames, and two bytes of bit-flags. Then we have (frame_count + 1) * 4 bytes for each frame offset. For example if we had byte indication of three frames we would need to read sixteen bytes. Each four bytes within those sixteen would be a frame offset. Mind you that the last offset is pointing to a special frame-like identifier designating end of file (EOF). Let’s stop here for a moment. After you have read the header you must identify if Internal Palette flag has been set. If so the next offset immediately after frame position offsets is the beginning of internal palette. Internal palette has it’s own pseudo header. It consists of four bytes. Two bytes representing color shift of the internal palette in relation to standard palette. And two bytes representing number of colors. REMEMBER that all bytes should be read as LOW-ENDIAN! For more info check part 1 of this tutorial. After reading four bytes of internal palette header, there are actual color definitions.

Those internal palette color definitions are sequences of DAC RGB. Just to remind you what it means: each color is represented as four bytes, where each byte consists of a value from range of 0 to 63 in decimal. Each of these values must be then multiplied by four. The first byte represents alpha channel (which is really a pseudo alpha, it seems that it only has a value of either 00 or 01), the second is RED, third is GREEN and fourth is BLUE. For more information refer to Part 1 describing MOO2 graphic formats.

To read the whole internal palette you must read four bytes multiplied by number of colors from internal palette header. Then if you’d probably want to convert those colors to RGB, so remember to multiply color values by four. This will give you internal palette in RGB. Let’s stop for a moment.

It’s really rare in MOO2 to find images with internal palette that spans entire 256 color space. Mostly internal palletes will have color count of 64 or less. What it really means? If internal palette is less than entire color space it means that the internal palette is really a mixed one. That’s where the color shift from internal palette header comes in. Mixed palette is really internal pallete “mixed” with external palette. It means that internal palette substitutes number of colors within external palette. The color shift is basically the starting color of this substitution.

Let’s suppose we have graphic file with internal palette. We read from internal palette header that color shift is 24 and color number is 64. Then we read color values from internal palette, convert them to RGB and voila. We have read our internal palette. What’s next? To create mixed type palette we have to choose one of the external palettes. Which one? That’s a tough question really. There’s no indication whatsoever which one should we choose. They are probably hardcoded within MOO2 executable.You’ll have to experiment by yourself. Suppose we chose the first external palette available. What’s next? Well… It’s really simple. The external palette always has 256 colors beginning with 0 and ending at 255. If we have color shift at 24 we start substituting color values of external palette at 24 (not 23 as some of you may think, because palette starts at 0!) and all the way to 88, because we have 64 colors within internal palette. It’s really that simple.

There’s also another type of palettes. I have found them to be within 1×1 pixel size images. These are not really images, but a placeholders for internal palettes. Some of the SHIPS.LBX_XXX files have them. These special files are used exactly as mixed palette, but they apply not only to one image, but to many graphic files following this special “image”. If you used my viewer you could see that after encountering such special image, the following files are images of the ships for each “nation”. It’s not really nation, but if you look closely in MOO2, ships are chosen not by race, but by the color banner. And that’s it. Palette of this special image applies to each graphic representing ship until the next special image with internal palette is encountered.

Ok. This should be enough for you in getting familiar with palettes. So now we’ll get into reading actual graphics (frames).

If you want to read each frame you have to position your reading cursor (fp.seek() in C) to offset stored in memory. (Frame offsets reading was described in Part 1 of this tutorial). At this seek position we read two bytes for frame beginning indicator. It’s always 1. If it’s not, you should not continue to read the frame and skip to next offset. If you had frame beginning indicator the next thing is frame y indent (relative vertical position). This is the number of vertical pixels you should render frame relative to screen (or rendering context) top position. After reading those bytes we must stop here again for a while.

Each frame has multiple data sequences. I call them lines or sequences for short. In MOO2 graphics not needed bytes are not stored. This way of saving graphics also allowed transparency effects. If there’s no pixel it means it’s transparent. Transparency is achieved simply by storing relative indents (x and y positions). Each sequence (or color line) has it’s own pseudo header.

Sequence begins with two bytes representing number of pixels in sequence. It’s also a special indicator, but I’ll explain that later. For now you have to remember that the following applies only if this pixel number is greater than 0! Next we read two bytes for x indent (x relative position). This is the amount of pixels that should be skipped relative to previous sequence x position when rendering. If we had x indent of value 12 and our last sequence ended at 24th pixel on screen we should start painting this new sequence from 36th pixel. I hope this is clear enough.

Then we have to read pixel data. Pixel data is simply one byte multiplied by the number of pixels. So you have to read number of pixels. Pixel data is 8bit value (1 byte) which represents pixel color number within the palette. If you had value of 46 you use 46th color from current palette in memory when rendering. Simple isn’t it?  BUT! You have to remember one thing. If number of pixels was odd value, YOU HAVE TO READ ADDITIONAL BYTE! This byte is then discarded, but you should always remember doing it. It’s probably linked to internal image loading mechanism used in MOO2.

So, let’s get back to our pixel number in sequence. I told you that the above applies only if pixel number in sequence is greater than zero. But what if pixel number IS ZERO? Well. It’s a special indication of something fancy going on (not really 😉 ). If you encountered 0 in pixel number of sequence, you have to read additional two bytes. Those two bytes are y indent of the sequence. It means that you have to start painting next sequence in new relative y position, but resetting x indent to ZERO! It’s like a number of newlines + carriage return, except for graphics. Sequence that has y indent DOESN’T HAVE ANY PIXEL DATA. But that should be obvious.

However. There’s also one small glitch. If y indent equals 1000 it means we have encountered END OF FRAME. When encountering this we should save the frame data in memory and proceed to the next frame offset (if any).

That’s all. I hope you found this post useful. If you have any comments or questions please post them. Also please forgive me for not showing  any pseudo code or  hex data, but I think this should be pretty obvious. If it’s not download: moo2gfxview-008tar.gz unpack it and refer to file graphics.py, function __getframes. I know it’s in Python, but you should be able to understand it, because file reading in Python is almost the same as in C or C++.

Hope you liked it. If you have any comments or questions feel free to post them.

Posted in Coding, Formats, Hacking, Master Of Orion 2 | Leave a comment

Writing engine for Galaxy Lords

Posted on August 19, 2007 by Wolverine

Hello everybody. I’m back from my holidays and almost instantaneously went back to coding. As you may know from my last post I’ve began coding simple 4X game with a working title Galaxy Lords. I just wanted to tell you how things are progressing. Well… I must say quite well… I’m working on the game engine core and the progress is neat. I’ve written several classes dealing with State, Event and Process Management, Textures, Animations, Screen functions and began to write Animation Sequencer and Scene management system. Engine is supposed to be very simple, yet quite powerful. There’s still yet to be developed input and sound system. However I have to say that there’s no way to find any decent sound system libraries for Python and I’m quite fond of it. I may end up using standard SDL mixer system for sound, but I wanted to use something more powerful like for example OpenAL bindings for Python. However it turned out that PyOpenAL is not developed anymore. So I have checked PySonic library which is supposed to be a binding for FMOD sound system in Python, but it’s also not developed anymore and quite frankly it doesn’t compile on my system at all, despite my best efforts to compile it. SDL mixer isn’t very bad, but it doesn’t support directional sound, no special effects for sound and the mixing system is very limited. I’m just disappointed. There’s no decent sound system libraries for Python at all. Anyway the engine core is at least 75% complete. Besides the input and sound system and scene management it’s almost complete. The engine uses Rabbyt, PyGame (SDL bindings for Python) and PIL (Python Imaging Library).

As for the game itself I have few things in my mind about how some things should be done. I have played many 4X space games like GalCiv 2, Sword of The Stars, Space Empires V, Ascendancy, MOO2, Imperium Galactica, Thousand Parsec and Alpha Centauri. I must say that still the best 4X space empire game is MOO2 with Space Empires V at the close second place. There’s also very simple 4X game for mobile phones named Rise of Antares by WarDreams, quite similar to simpler MOO2, which is quite cool when you don’t have computer near you. Recently I’ve been playing sword of the stars, but I must say. It’s quite cool when you start the game, but later it’s getting veeery boring, and despite the fact each race has it’s own unique ways of transportation and each game-run gives you quite different technology tree it’s simply boring. GUI is not very obvious, you feel like the game is playing with you, not you’re playing with the game. Technology tree is boring and options to develop your own starships is quite limited. Also weapon development progress is not noticeable, options for armor and shields are limited. There’s no way to develop your colonies and the game is quite strange. I’ll tell you about my game in a spiral galaxy, playing humans. After almost hundred turns I’ve finally began to explore the galaxy, but then I’ve encountered the Hivers. Despite the fact that I’ve began in one arm of the spiral galaxy that was connected to main cluster of the galaxy via node lines, after conquering Hiver colonies in my arm, the node-lines connecting my galaxy arm to main cluster where gone. I have asked myself – WTF? Needless to say it’s quite frustrating that you have absolutely no control over development of your colonies other than infrastructure and terraforming. This is bad. There’s no circle of climates and each “star system” consists only of one planet. That’s bad. It’s even not realistic, it’s great oversimplification.

But let’s get back to Galaxy Lords and what I’m planning to do. Well… MOO2 GUI wasn’t bad at all despite the small glitches and later micromanagement issues. I find Colonies screen one of the best aspects of that GUI. You knew how many people are scientists, workers, farmers… You could control what to develop first on your new colonies. Each planet had it’s own capabilities like climate, resources and size. It was quite obvious. You had few indicators to keep an eye on. You could run your empire with ease, despite the fact later in the game you had to keep a lot of micromanagement stuff under control. I want to go the MOO2 way in overall look and feel to it. However I don’t want to go with the technology tree introduced in MOO2. Some of my ideas are to keep focus on what our current science knows and predicts. I would like to construct some of the technologies based upon our current knowledge and what may be possible in future. Some of the most promising aspects of our science currently is fusion, cold fusion (LENR/CANR), genetic modifications, various aspects of mind control, intelligent management systems, A.I., development of information technology and many others. However I have an idea that some technologies should not only bear positive effects with them, but also negative ones. Think about it this way. Suppose humans in the game will develop cybernetic interfaces for computer networks or implants allowing them to better control populations, giving it better scientific or industrial output. However this will have downside like for example population feeling controlled and the morale being lower. Or for example development of virtual reality systems that will make population happy, but some of the population quite rebellious, because of the fact that population will be obedient to the will of those in power. I’m planning to create something like loyalty system among empire population. Colonies farther of governing bodies will be more susceptible to revolt and start independence moves. In 4X games the cultural influence factor is greatly dismissed, but it can really create very interesting strategic and tactical opportunities. What I disliked the most in MOO2 was Diplomatic and Espionage system. I don’t know if I’ll be able to develop sophisticated systems for diplomacy and/or espionage in Galaxy Lords, however I have few ideas how should it be done. Well… In my opinion diplomatic system in MOO2 was not obvious. You should have a way to determine if your actions are going to succeed to some degree… You could use spies to mark fleet positions of the enemy and gather some data regarding their colonies. There should be spy satellites that will constantly monitor the movements of the enemy fleets. Also in real $X strategy games you should be able to take advantage of asteroid fields, cloaking devices, nebulas and other objects that occlude your presence within the enemy systems. Your spies should try to steal technologies you have marked for stealing. Your spies when developed sufficiently should try to change orders of colonies and fleets. They should really influence the enemy empire given sufficient funding. Each population unit in your empire should have it’s own profile. There should be people wholove your politics and those who don’t.

I’m not sure how much of this I could implement, however I’ll try to put into Galaxy Lords as much as I can without sacrificing the overall goal of relatively simple and fun to play game. I know that I want to develop technology tree that focuses on things that are real. However I’m planning that each race should have independent technology tree. This could really make the game more interesting. Imagine trying to exchange technologies not available to you, with a technology not available to other races. Simply because of different thoughts each race has. Or maybe even exchange of technologies giving you something else than the race that exchanges the technology with you. I know that I want to focus the technology trees depending on each race abilities. And not only this, but this could be the most significant achievement of Galaxy Lords.

Well, there’s much to say what I could do, but ideas are very simple to come up with. It’s hard to get them into the game. So these are my ideas I’d love to see in my game, but still I’m quite realistic person and want to achieve quite playable game, that will be fun for anyone playing it, despite the fact it may be quite simple.

I’ll keep you informed how things are progressing.

Posted in Galaxy Lords | 1 Comment

Quickie status update

Posted on July 23, 2007 by Wolverine

Welcome again. Almost two months without any update, you should all blame me. Unfortunately I’ve been busy coding, writing, struggling with clients, work and everyday crap we all know too good about. I know I promised describing MOO2 graphics formats, but it won’t happen this time. Next time maybe. After somewhat off-topic last post I decided not to do this again, however I’m not going to abandon such rants, however shorter, from time to time. 😉 But I promise it won’t be anytime soon.

My Content Management System for websites stalled at version 0.0.86, but when I’ve taken a look at it I decided not to continue it further. It wasn’t meant to mix dynamic javascript with PHP5 and because of some requests from my client I had to introduce complex hacks to the code. That broke down universality principle I had in mind writing this software. I thought, ok after I finish it I’ll be able to cleanup the code and align it again with this principle. But it turned out that if I had to sacrifice time for code cleanup I’d rather take pieces of good code and rewrite the rest from scratch. This will take much less time, simplify the system and make it truly universal. After almost two years of writing this CMS I have learned few things that could be used for improved development process and make it much better from the programmers and users perspective. I don’t want to go into details, but now I have much better understanding of how should I create various managers mostly the plugin subsystem. Also some of the complex process handling within the system could be simplified by a factor of few times with introduction of truly Model-View-Controller programming paradigm, despite the fact of non-looping nature of the system; introduction of fully XML’ized structure for layouts, articles, configuration and plugin description files; and perhaps even introduction of simple home-brewed application server in Python or C++ that would make PHP5-only code dumb thin-client running on top of application server middleware. Of course this would mean writing from scratch large portions of the code, but speaking in RPG terminology… I’ve gained few experience points along the way. After all I’m still learning good programming techniques and methods. I expect another half year to create something useful, but I hope it’ll be worth it.

Besides the fact of coding my CMS system, I’ve been in contact with few programmers for quite some time and I must say that experience and idea sharing is taking it’s toll. I’ve had few small side-projects that I have completed in the meantime. But nothing worth mentioning. However my dream was to create a game to learn, to make something creative and to have just pure fun. For a few years I’ve been thinking about this, but being lazy and in my opinion quite inexperienced I’ve never completed any game projects. But the time has come that I’ve decided to seriously give it a try and write a simple game. Due to lucky coincidence, I’ve found really cool graphics library named Rabbyt for Python. It is an accelerated 2D graphics library that uses OpenGL and PyGame for rendering, but it’s so easy to use and powerful, despite the fact it’s version is 0.0.4 only, that I’ve decided to give it a try. And because I’m literally in love with Python programming language, yesterday I’ve began prototyping the engine for simple 4X game. 😀 Oh yes. I love the genre and why not mix pure programming fun with gaining new experience points. After all writing games is the best way to do it.

The game has a working title Galaxy Lords (I know it’s not very cool and fancy, but I couldn’t come up with anything better for the time being). It will be 2D game with somewhat MOO2-ish look and feel to it, but not so complicated, because I don’t think I’ll be able to embark on such large-scale project now. However I want the game to be fun and have things every 4X game has. That means colonizing space, waging wars and perhaps simple diplomatic actions, managing resources, researching new technologies and things like that. The game would be turn based. However I don’t want to make a copy of existing 4X space-themed games, but rather create something new and unique. Ideas are welcome. 🙂 I think that after finishing the game I’ll release it under Open Source license (probably GPL) and the game content under Creative Commons Attribution Non-Commercial Non-Derivative License. The game will be written in Python with extensive usage of PyGame, Rabbyt and Twisted if I decide to do multiplayer. I’m currently planning game engine. My for it goals are: simple, relatively fast, easily extensible and using common formats like XML, PNG, JPG and OGG Vorbis. I want to write few tools for this engine like Animation and Sprite Editor, Animation Sequencer for creating intros, cut-scenes, dynamic GUI and menus and in-game actors animations. perhaps GUI editor and some other tools for rapid content creation for the engine.

Unfortunately I’m going on my holidays tomorrow and won’t be available for some time, but I’ll have time to plan the engine and the game. After coming back I’m going to give this project full time for fun and gaining experience. As for now, I’d like to show you what I was able to do in just under one day. Excuse me for bad quality of the videos, but the capture program was capturing at 3 FPS and youtube converted it to 25 FPS so the video looks very fast. XVidCap is not really meant for capturing OpenGL. Anyway. Enjoy.

http://www.youtube.com/watch?v=q0MjMEEKXzI

http://www.youtube.com/watch?v=_xpQaX06TpE

http://tinypic.com/view.php?pic=67zr5no

Posted in Coding, Galaxy Lords, General | Leave a comment

Description of MOO2 graphic formats

Posted on April 24, 2007 by Wolverine

Some time has passed since I’ve written something here. Some people have downloaded my viewer, however I know Python is not obvious to everybody so I’ve decided to publish information regarding graphic formats of Master Of Orion 2. I don’t have time to work on this viewer and it was always just a proof of concept that graphic modding of MOO2 can be done. I’m not saying no to further development of the viewer for MOO2, however my priorities have shifted once again and development of the viewer/converter halted for now.

First we’ll start with palette files. There are three types of palettes in MOO2. External, internal and mixed. Palette format in MOO2 is in DAC RGB which means that color is represented as four bytes, where each byte consists of a value from range of 0 to 63 in decimal. Each of these values must be then multiplied by four. The first byte represents alpha channel (which is really a pseudo alpha, it seems that it only has a value of either 00 or 01), the second is RED, third is GREEN and fourth is BLUE. Let’s take a look at one of the so called External palette files in this case FONTS.LBX_001 which is the second file where external palettes reside. Take a look at this hexadecimal representation of the beginning of the file:

0000:0000 01 00 00 00 01 00 03 00 01 04 04 06 01 06 06 08
0000:0010 01 09 09 0a 01 0b 0b 0d 01 0d 0d 0f 01 0f 0f 11
0000:0020 01 12 12 14 01 14 14 16 01 16 16 18 01 18 18 1a
0000:0030 01 1b 1b 1d 01 1d 1d 1f 01 1f 1f 21 01 21 21 23
0000:0040 01 22 22 24 01 24 24 26 01 26 26 28 01 27 27 2
0000:0050 01 29 29 2b 01 2b 2b 2d 01 2d 2d 2f 01 2f 2f 31
0000:0060 01 31 31 33 01 33 33 35 01 35 35 37 01 37 37 39
0000:0070 01 38 38 3a 01 39 39 3b 01 3b 3b 3d 01 3c 3c 3e
0000:0080 01 00 03 04 01 00 05 07 01 03 0a 0d 01 0a 0f 13
0000:0090 01 0e 12 16 01 13 16 1a 01 16 19 1e 01 1a 1d 21
0000:00a0 01 1e 21 25 01 22 25 29 01 27 29 2d 01 2c 2e 31

The first column is a hex offset and the second is the hex representation of byte values. I have set to bold every odd numbered color bytes and to normal an even numbered color bytes. Each four bytes represents one color in the format mentioned above. Within the external palette files (FONTS.LBX_001 to FONTS.LBX_013 and IFONTS.LBX_001 to IFONTS.LBX_004) palettes start from zero byte to 1023. Each external palette has 256 colors counted from 0 to 255. If you’d like to read one color you have to read four bytes. For example we have color represented by 01 0a 0f 13. Let’s assume we have read those four bytes and we want to convert them to RGB. The first byte with value 01 is alpha channel. However it only has 00 or 01 values. It probably acts as a boolean: either we have this color (value 01) or not (value 00). But I’m not sure. Perhaps it acts as a special modifier in MOO2 (transparency or rewritable color perhaps?). We can safely ignore it. The next three bytes represents DAC RGB values. It means that each intensity value must be multiplied by four. Take a look:

A R G B = 01 0a 0f 13 = 01 10 15 19

The range of values for R G B in DAC RGB is from 0 to 63 (hex 0x3f). If you want to get 24bit color representation you have to multiply each DAC RGB value by four:

red = dacred * 4; green = dacgreen * 4; blue = dacblue * 4;

If you want to convert 24bit RGB back to DAC RGB you have to do the reverse operation including rounding to integer. Let’s assume we have 24bit RGB color with values (35,127,193) which in hex is (0x23,0x7f,0xc1). You do it like this:

r = 35; dac_r = int(r / 4); g=127; dac_g = int(g / 4); b=193; dac_b = int(b / 4);

Remember! You have to round it to integer, otherwise sometimes you’ll get a floating point value. The external palette files reside within LBX archives FONTS.LBX and IFONTS.LBX. Those archives must be unpacked before they can be used as palette files. After unpacking FONTS.LBX you have to remember that FONTS.LBX_000 IS NOT PALETTE FILE! It’s only a font file. Palette files start from FONTS.LBX_001 (the second file in archive). Palettes reside within first 1024 bytes (256 * 4 bytes). However as you’ll see those files are not 1024 bytes long, but few times longer. That’s because there are also mouse cursors stored behind the palette. They’re visible in some files in a good hexeditor if you properly adjust the viewing columns. However I wasn’t able to figure out how they can be retrieved. I think that’s all regarding external palettes.

The internal palettes are stored within the graphic files. They have the same DAC RGB format as external palettes, however they are stored differently. But before I’ll give you description on how to read internal palettes I have to give you information regarding the graphic format itself.

REMEMBER! All multibyte values in MOO2 formats are stored as LITTLE ENDIAN! That means we have integer (2 bytes) and long (4 bytes) bytes representing the value stored in reverse fashion. Each atomic element size is 8bit which means it’s one byte long. For clarification let’s assume we have an integer with value of 52304 (remember that unsigned integers allow values in range of 0-65535 in that case). In hexadecimal this value is represented as 0xCC50. That means when we want to store this value we have to divide it to two bytes, in this case 0xCC and 0x05. So the LEAST SIGNIFICANT BYTE (in this case 0x05) is saved FIRST! It means that we save this value as “05 CC” not CC 05! The same applies to long. Let’s say we have long integer with value of 492062186 (hex 0x1D5445EA). In this case we have four 8 bit (one byte) atomic elements: “1D 54 45 EA”. The least significant byte is 0xEA. If we want to save it in little endian we have to do this in reverse order. The result would be: “EA 45 54 1D” and that’s what we save. The same applies except in reverse order when we want to read little-endian multibyte value. In this case if we have “7F 03 B9 0C” we reassemble this value reading the last byte first. The result must be 0x0CB9037F which is 213451647 in decimal. Hope that’s clear enough.

So now I can give you the information on how graphics are saved in MOO2. As our example we take the first unpacked file from BLDG0.LBX archive the BLDG0.LBX_000. This LBX archive stores graphics for the buildings visible on colony screen. If you want to properly view the buildings you have to use FONTS.LBX_002 external palette file. This graphic file we have taken shows the Alien Management Center building at center bottom of the colony screen. I will show you how the beginning of the file is seen in hexeditor.

0000:0000 80 02 e0 01 00 00 01 00 00 00 00 08 14 00 00 00
0000:0010 1c 21 00 00 01 00 55 01 01 00 0e 01 55 00 00 00
0000:0020 01 00 01 00 0e 01 55 00 00 00 01 00 01 00 0e 01
0000:0030 55 00 00 00 01 00 01 00 0e 01 55 00 00 00 01 00
0000:0040 01 00 0e 01 55 00 00 00 01 00 01 00 0e 01 55 00
0000:0050 00 00 01 00 01 00 0e 01 55 00 01 00 0c 00 55 00
0000:0060 00 00 01 00 01 00 0e 01 56 00 01 00 0c 00 55 00
0000:0070 00 00 01 00 01 00 0e 01 56 00 01 00 0c 00 55 00
0000:0080 00 00 01 00 01 00 0e 01 56 00 01 00 0c 00 55 00
0000:0090 00 00 01 00 01 00 0e 01 57 00 01 00 0c 00 55 00
0000:00a0 00 00 01 00 01 00 0e 01 57 00 01 00 0c 00 55 00
0000:00b0 00 00 01 00 01 00 0e 01 58 00 01 00 0c 00 55 00
0000:00c0 00 00 01 00 01 00 0e 01 58 00 01 00 02 00 55 00
0000:00d0 01 00 09 00 56 00 00 00 01 00 02 00 0d 01 55 d5
0000:00e0 02 00 02 00 55 55 01 00 08 00 56 00 00 00 01 00
0000:00f0 02 00 0d 01 d3 b8 01 00 02 00 55 00 01 00 09 00

This dump from hexeditor similar to the previous one. Left italic – offsets, right – byte values. I have colored some bytes and backgrounded sections of the file for clarity. The brownish background under the bytes shows the header of the graphic file. I’ll explain the header of the file now.

First two bytes marked as red are “80 02”. This is the width of the image. As you remember what I have told you about the multibyte values in MOO2 (little-endian) this value is in fact 0x0280. I wouldn’t remind you about this again so keep in mind that all multibyte values are LITTLE ENDIAN. In this case we have 640. The value represents width in pixels.

The next two bytes (green) is the height of the image. In this case it’s 480 so we have size of the image 640×480 pixels. The next two bytes are always zero. Two bytes marked in blue is number of frames (in this case one). Next two bytes marked in yellow is the frame delay i.e. delay between each frame is shown. It’s used in animations. In this case we have only one frame so the delay is 0, but if we had multi-frame graphic (animation) the delay would be set. The last two bytes in the header marked as cyan represents image flags. We have to stop here for a moment and explain those flags.

The flags is a two byte value also stored as little endian. Contrary to previous multibyte values this doesn’t represent decimal value, but a binary value. Each flag is represented as one bit within the value. Take a look:

0000 0000 0000 0000
..JI FB.N .... ....

This is the binary representation of the two byte flags. The first line shows bits and the line below corresponding flags. The dot in the second line means that setting this bit either doesn’t do anything (is not used) or the behaviour of the flag is unknown. I have given those flags unique letters and they are as follows:

  • J – Junction – I don’t know really what it’s for. Grig de Griz has pointed out this flag in his format description so I’m including it here. (TAlchemist pointed out that: “Junction flag means that you add each frame to the frame before it making a composted image. and start from the begining on the first frame. That is how only the first frame of the big animations is large and all other frames just include the pixels that changed.”)
  • I – Internal palette – Tells if the image has internal palette
  • F – Functional color – Tells if the image has functional color. This color is used for effects like transparency or shading.
  • B – Fill background – Tells the game if it should clear the area on which the image is going to be drawn.
  • N – No compression – Tells if the image uses compression. For me it’s behaviour is unknown. It’s probably a left-over from MOO1 graphic formats which are almost identical to MOO2.

Back to our header. We have the value “00 08” in our header for flags. This means the value is 0x0800 in hex. Let’s see what flags reside here. After converting this value to binary we get 0000 1000 0000 0000. Let’s compare this new value to our line describing the flags:

0000 1000 0000 0000
..JI FB.N .... ....
Voila! We now know that the image has functional color flag. If we had for example:

0011 1001 0000 0000
..JI FB.N .... ....

We would have junction, internal palette, functional color and no compress flags set. There are really only two flags that are significant for us. The more significant is INTERNAL PALETTE and the less significant is FUNCTIONAL COLOR. The first one would be covered later in greater detail. The second is really needed for properly plotting graphics and I wouldn’t give it too much attention, because I haven’t completely figured out how it works. All I can say for now regarding FUNCTIONAL COLOR is that it’s used in our BLDG0.LBX_000 for drawing shadows. If you manage to draw this image you’ll see that the shadow is represented by dark violet color, but within the game it’s giving a proper shadow on the colony background screen.

Let’s get back to our header. I’ll refresh your memory here:

0000:0000 80 02 e0 01 00 00 01 00 00 00 00 08 14 00 00 00
0000:0010 1c 21 00 00 01 00 55 01 01 00 0e 01 55 00 00 00

We have already covered all the bytes marked in colors up to cyan marked flags. What are those last eight bytes that are in the header? Those are four byte frame offsets. This needs greater explanation. Frame is the proper graphic that is displayed on the screen. It’s an animation frame. Since we have only one frame image (still image, not animation) here there are only two frame offsets, namely: “14 00 00 00” and “1c 21 00 00”. Those offsets say where within the file each frame starts. However I have said we have only one frame image here so why the hell we have two frame offsets you might ask. Well. It’s because we always have frame_number + 1 frames. The last frame isn’t de facto a frame. It’s a special frame-like identifier that tells that the end of file (EOF) has been reached. It’s probably connected to format reading specific used in MOO2. How to get frame offsets then? Here’s a little pseudo-code to show you how:

offset_array = array(); // Create result array of offsets
for (i=0; i<frame_number+1; i++) // Go through each frame offset
{
bytes = fp.read(4); // Read four bytes representing the offset
bytes = reverse_bytes(bytes); // Convert little endian to proper byte representation
offset = str_to_long(bytes); // Convert string to long number
offset_array.add(offset); // Add offset to offsets array
}

If you now want to read frames you just set internal file pointer (fp in the pseudo-code) to the offset of each frame. In C++ you can do this with: fp.seek(offset); and that should do.

That is all for now. I’ll explain frame reading and internal and mixed palettes in the next post. For now you can still download my viewer and browse the code or you can read somewhat outdated previous posts regarding the formats. For now the most complete source for getting information regarding the formats if still source code of my viewer. You can download it from the previous post and I encourage you to do so. Python is easy language, I have learned it in a day or so if you’re a programmer you’ll be able to decipher my code quite quickly. Greets to all and I’ll try to write the second part of the tutorial ASAP.

Posted in Coding, Formats, Hacking, Master Of Orion 2 | 4 Comments

MOO2 mooding ;) and first public release of the Viewer

Posted on February 21, 2007 by Wolverine

After some time I’ve finished all essential functions in my MOO2 graphics viewer. All critical classes are almost finished (the only thing that is left in underlying functionality is palette merging in palette class, shouldn’t be hard to do). The GUI is still in the state as it was before, but all essential classes are finished and this is the reason I’m releasing the code to the public. It’s licensed under GPL version 2 (see www.fsf.org website for details regarding the license) so you can do almost anything you want with the code as long as you stay in compliance with the license. The Viewer and associated programs could be used as a technology demonstrator with some patience and little changes in the code. To run it you will need Python (for Windows possibly from http://www.activestate.com/products/activepython/) version 2.4 at least and Python Imaging Library (could be obtained from http://www.pythonware.com/products/pil/). This piece of software was developed under Mandriva Linux 2007 so it will probably need slight changes within the code to run properly on other operating systems, but Python is fairly portable and I have used struct.pack/struct.unpack functions to stay in compliance with other systems/architectures. As I don’t have MacOS X I’m not sure about big endian architectures. struct.pack/unpack functions have little endianess specified, but I’m not sure about byte sizes of integers and longs. It certainly runs under Linux and intel x86 architecture.

Before you get overhyped TOO MUCH you have to be warned. This release is intended PRIMARILY FOR CODERS!!! Power users should be able to make use of this software, but it’s NOT beginners friendly right now. I’ve decided to release the code, because I won’t have time to code this app for some time and I know there are some people subscribing to my feed eagerly waiting to get some information regarding the graphic formats used in MOO2. The source is extensively commented so you should be able to make use of it and learn few bits from it. The code is not optimised, I know that few things could be speeded up, but I don’t have time for it right now. It should be treated as Master Of Orion 2 graphic formats reading/saving demonstrator.
There are few basic classes within the code: Graphic, Palette (needs internal structure rewrite to be more efficient), LBX and Converter module. There are also some GUI specific classes and few support programs that I won’t describe here, because they are still work in progress and are only included so you could see how to use those four basic classes. With some skill you should be able to utilize this app for basic MOO2 moding, because I have included MOO2GFX to PNG and PNG to MOO2GFX converter. The app is slow, I know about this, but it’s not optimised as I have said. Most of the drawing and reading/writing is done as pixel by pixel so it is THAT SLOW. Despite this fact it is fairly usable for advanced users and you can do with it things like this:

Captain Jean Luc Picard of the starship Enterprise :D Engage! On desert planet Seche...

Take notice that I haven’t implemented internal palette merging and I didn’t have time to write functions that will take proper merged palettes so you could import files with those palettes. That’s why there are some glitches. It still needs further research on which palettes are used for what. If you’ll be able to run this program please send me your findings regarding palettes so I could include them in this application when I’ll be able to finish the GUI. You can experiment freely, as you can see in the screenshots MOO2 accepts every graphic file that was converted from PNG to MOO2 graphics and then packed with my LBX compressor. Palette class has exportgimp() function to export palettes as GIMP (www.gimp.org) palettes so you can convert PNG to indexed palette and then use convert.py class to create a graphic file. After this you can use packer.py (needs changing filenames inside) or use lbx.py class to create new archive from specified directory. Of course there’s still some work to do, but this is just a demonstration of what will be possible in near future. So. All you brave programmers and adventurers download this program now!!! Happy MOOding… 🙂

Posted in Coding, Master Of Orion 2 | Leave a comment