home pictures webcam about me various eu2
inferis.org

eu2 » TBL Files » lightmap.tbl

 
The lightmap.tbl files (actually, lightmap1.tbl, lightmap2.tbl, lightmap3.tbl) represent the map data you actually see in EU2. It does not contain sprites (for example, the trees, city, harbour, ... icons), but only the image of the provinces themselves.

There are 5 lightmap files in the map directory, but lightmap4.tbl and lightmap5.tbl are unused. The data contained in them is not conform to the data in the three other files, so they are probably leftovers from early development.

File Info
Location » (eu2 root)\map\lightmap1.tbl
Size » 11,749,936 bytes (11.2 MB)

Location » (eu2 root)\map\lightmap2.tbl
Size » 4,059,884 bytes (3.87 MB)

Location » (eu2 root)\map\lightmap3.tbl
Size » 1,689,938 bytes (1.61 MB)

Zoom levels
and map files
The standard EU2 mapsize is 18944x7296 pixels. This is for the standard zoom level (largest zoom), and that info is contained in lightmap1.tbl.
If you zoom out one time, the size gets divided by two, resulting in a 9472x3648 map. That info is contained in lightmap2.tbl.
If you zoom out again, the size gets divided by four, resulting in a 4736x1824 map. That info is contained in lightmap3.tbl.

The map is also divided into 32x32 blocks. Each block compresses part of the map. This is unregardless how many provinces are "present" in that block. This is at least one, but it can be as much as 16 or more. That also means that each block has to know what pixel belongs to what province. More on that later.

Basically, we have the following table:

File Zoom Level Zoom factor
(2^level)
Size
(pixels)
Size
(blocks)
Total blocks
lightmap1.tbl 1 1 18944x7296 592x228 134,976
lightmap1.tbl 2 2 9472x3648 296x114 33,744
lightmap1.tbl 3 4 4736x1824 148x57 8,436

The zoom factor basically defines the ratio between a displayed pixel and the number of pixels it represents on the standard zoom.

File structure
and Offsets
The file is divided into two parts: offsets and blocks:

The number of offsets differs from file to file, because each file contains a different blocks (as seen in the table above).
The offsets are encoded as 3 byte integers. This allows for 24 bits of info, and that also means that the total datablock size (all blocks together) cannot be larger then 16.777.216 bytes. This is not much of a problem as the largest file is 11 MB, including offset data.
As the other files, the offsets are stored little-endian: the least significant byte is stored first.
Each block has one offset. The offsets are specified relative to the start of the data block, in other words, offset 0 means the first byte after the offset block (including the total size byte, see below).

After the offsets comes a 4 byte integer, which denotes the total size of the datablock. You could also see this as the (totalblocks+1)th offset, padded with an additional 0.

And then come the datablocks. Each block is of variable size.

Datablocks
It took me quite a while to find this one out, to be fair. If it wasn't for Johan, I guess I wouldn't have found out anyway, as it's quite simple, but hard to decipher. Especially the quadtree part was daunting to find out.


Each block has the same layout (see right).
I'll discuss each part separately.
      

Province ID List
The first part of a block is the province id list. This is list of two-byte entries containing the province ids of the provinces (or parts of them) that are visible in this block.

Each entry in the list consists of a terminator bit and a 15-bit province id. The terminator bit is always zero, except for the last entry in the list. In other words, the bit signals the end of the list.

However, if the province id exceeds the maximum province value (1615), the lower 15 bits are formatted differently. In this case, the lower part is divided in 4 parts: index1, river adjacency, index2 and color.

  • The most important of these is index1. It denotes an offset in id table list as specified in the block (for example, index1 = 0 points to the first entry in the list). This index is stored as "offset + 4". Why, is unclear to me.
  • The index2 is "neighbour" pointer, relative to the province specified by index1. This pointer is used by the engine to find a second province, neighbouring the first province (how this is done, is out of scope for this document). The second id could be a river provinces (rivers are provinces too). If that is the case, the "river adjacency" value smaller than 15 (in other words, not all 4 bits are set). That value then points to another index for a "neighbouring" province, again based from the provinces specified by index1. In other words, you "jump over" the river to the next "real" province. As a final result, either the province in index1 or the "jumped" province id is placed into the id table after processing a special entry.
    To make this more clear, this is the code used in the lib I use to decompress a block:
    // C#
    for ( int i=0; i<idCount; ++i ) {
    	if ( idTable[i] <= Province.MaxID ) continue;
    
    	ushort id = idTable[((idTable[i]>>9) & 63)-4];
    	if( id > Province.MaxID )
    		id = install.Provinces.TerraIncognita.ID;
    	else {
    		int river = (idTable[i]>>5) & 15;
    		if ( river != InvalidAdjacency ) id = install.Provinces[id].GetNeighbor( river ).ID;
    
    		idTable[i] = id;
    	}
    }
  • The color field is not used, as far as I can tell.

    The processing of the special entries is done after the full list is read in.

    Quadtree nodes
    This part contains a tree representing the map image data. It is a list of bits specifying a quadtree structure, which is maximum 5 levels deep (not coincidentally, as 32 = 2^5).

    The algorithm is actually very simple. It starts off at block level (level = 5, block size 32 pixels). It checks the first bit (and moves the bit pointer to the next one). If it's a 1, the block is divided into 4 smaller blocks, each half the size of the current block (thus 16 pixels). Recursively, the subblocks are processed too (in a bottomright » bottomleft » topright » topleft manner) by doing the same process on each of them. The level is decreased by 1 (as we're one level deeper). This process stops at level 1 (NOT level 0), which means that we're a 2-pixel resolution. At that level, 4 leafs are created (without a bit in the tree stream).
    When the bit read is a 0, processing of the (sub)tree stops, and 1 leaf is registered.

    In pseudo code, I'd describe it like this:

    Proc ReadTree( level )
    	bit = readNodeBit()
    	moveToNextNodeBit()
    
    	If bit is set Then
    		If level > 1 Then
    			addBranch()
    
    			ReadTree( level - 1 )
    			ReadTree( level - 1 )
    			ReadTree( level - 1 )
    			ReadTree( level - 1 )
    		Else
    			add4Leaves()
    		End If
    	Else
    		addLeaf()
    	End If
    End Proc
    
    Starting it off would be with a call to ReadTree( 5 ).

    This just reads in the tree. In effect, you have a tree structure with leafs at the bottom. The values in the parts below are ordered in a breath-first traversal (go as deep as you can in the tree, and start reading there, moving "from left to right" as you go).

    Using this date to draw it to a surface is another matter. I'll do a seperate document about that!

    Leaf Ownership entries
    For each leaf in the quadtree, this part contains an ownership index into the id list at the beginning of the block. This is to identify each leaf with a province, which is necessary because the engine needs to know how to color each part of the map (for example, the different terrain types give each province a different color). Since the map data is just greyscale values (see the next part), the coloring must be applicable on a per province base.
    Depending on the number of entries in the ID list, this part is formatted differently. As said before, each entry in this part is actually an index into the ID list, and it does not contain actual province values.

    1 province » If there is only one province in the id list, this part is empty. The engine knows there's only one entry and can automatically fill the ownership of each leaf with that province id.

    2 provinces » For two provinces in the list, this part is encoded as 1-bit values, one for each leaf. That means there are (leafcount/8) bytes in this part, in this case.

    3 or 4 provinces » When there are 3 or 4 provinces in the list, this part is encoded as 2-bit values, one for each leaf. That means there are (leafcount/4) bytes in this part, in this case.

    5 to 16 provinces » In this case, this part is encoded as 4-bit values, one for each leaf. This means there are (leafcount/2) bytes in this part.

    more than 16 provinces » When there are more than 16 provinces in the id list, each entry in this list occupies one bytes, and there will be as many bytes in this part as there are leafs. This also means the upper limit of provinces for a block is 256.

    Leaf Color Entries
    This is the simplest part of the whole block.
    For each leaf in the quadtree, this part contains a 6-bit color value. These are encoded in sets of 4 entries, totaling 3 bytes each: 4x6 bits = 24 bits = 3x8 bits = 3 bytes.

    The color values stored are greyscale values: they are actually offsets into a table generated from the colorscales.csv file, which is used to color the map accordingly. This allows the enging to use the same map data for the different views, by combining "lighting" values (hence the name, lightmap) with coloring.
    The greyscale values are 6-bit, this means there are 64 variations of each color possible. This also explains the "63" index used in the colorscales.csv.

    Some observations
    The leaf count will always be a multiple of 4.
    The upper limit of provinces is 256 (due to encoding in the leaf ownership part).

  • Usability
    These files contain the map data to visualise the EU2 map.