Humble Plugin request: File operations [solved]

[abstauber]

I hope this is right place to ask: I'd like to speed up the file operations in my tile engine. And since now the engine itself is fast again, this calls for bigger maps . Unfortunately that also increases loading times dramatically, so I wonder if anyone could help me out and put these functions into native C code.

Code: ags

int noloopcheck Readword(this File*) {  
  char c1 = this.ReadRawChar();
  char c2 = this.ReadRawChar();
  return c2*256+c1;
}

int noloopcheck ReadDoubleword(this File*) {
  int i1 = this.Readword();
  int i2 = this.Readword();
  // cut 16th bit due to int being signed
  i2 = i2%16;
  return i2*65536+i1;
}

String noloopcheck BitString(int i, int bits) {
  int p = bits - 1,are;
  String s="";
  int q;
  while (p > -1) {
    q = FloatToInt(Maths.RaiseToPower(2.0, IntToFloat(p)), eRoundNearest);
   are = i/q;
    i = i -are*q;
    s = String.Format("%d%s",are, s);
    p--;
  }
  return s;
}

int noloopcheck BitStringToInt(String s) {
  int l = s.Length,are;
  int i;
  String ss;
  int add = 1;
  while (i < l) {
    ss = s.Substring(i, 1);
    if (ss == "1")are =are + add; 
    i++;
    add = add * 2;
  }
  returnare;
}

There's also a second thing I'd die to see as a plugin and that's a way to save data structures (whole arrays and/or structs) to a file. Currently I serialize everything into giant strings and save those. Of course it takes forever to finish


Imagine the game turns out to be great if it weren' for the loading times...?  That would ruin your fun too! See.. it's a win/win situation  
Seriously, if anyone could lend a me helping hand, that would be totally great!
Eternal gratitude assured  

[Wyz]

Yes that would definitely increase speed.I've been thinking of writing something similar some time ago. Well I can help you out with that I guess.

Edit:

Code: ags

#include <stdio.h>
#include <stdlib.h>

inline short Readword(FILE *fp)
{
	short word = 0;
	fread(&word, sizeof (word), 1, fp);
	return (word);
}

inline long ReadDoubleword(FILE *fp)
{
	long dword = 0;
	fread(&dword, sizeof (dword), 1, fp);
	return (dword);
}

// make sure that str has at least length bits+1
inline void BitString(char *str, long flags, long bits)
{
	str += bits;
	str[1] = 0;
	
	while (bits--)
	{
		*str-- = '0' + (flags & 1);
		flags >>= 1;
	}
}

inline long BitStringToInt(const char *str)
{
	return strtol(str, NULL, 2);
}

Btw a tip for future projects,look into the << and >> operators when you're doing bit operations, AGS will support them and they are a hell of a lot faster then using floating point conversions.

[Kweepa]

You must be doing something seriously wrong.
Why would you need to convert back and forth to strings?
Can we see a bit more of the code?

[Khris]

This is code originally by me, in no way optimized, that's used to read ProMotion's tilemap files.
I was using strings as an easy way to read certain bits off double words.

I've only used it for quite small tilemaps (20x15) and there wasn't any noticeable delay so I never bothered to make it faster.

Afaik, there's no straightforward way to read raw bit data other than to repeatedly call File.ReadRawChar() so that's what I used.

[abstauber]

Sorry, I should have mentioned it right away, that these functions were done by Khris.

@Wyz: Yaay, thanks for accepting the challenge
Could that future plugin maybe also have write functions? Pretty please? </puppy dog eyes>

PS: Is it my browser or the board converting "r - -" (without spaces) into the word "are"?

[Khris]

It's the board; unfortunately, it tries to turn "AOLSpeak" like "r_u_2_cool" into English even inside code tags.

[Kweepa]

Ah, I see. I didn't realize it was an external format.
How about preprocessing ("cooking") the levels (with another AGS program) and writing them out in AGS format?
Or in your game, when starting a level, check if there's a preprocessed version and if not cook the level.
Unfortunately AGS doesn't have a File.DateTime function, so you'll have to delete the old cooked file manually in this case. You might be able to hook up a batch file from ProMotion's export?

Cooking the levels obviates the need for non-portable plugins.

[abstauber]

But the speed issue would still be present, right?

For example in order to travel back and forth between levels I save all data structs to files using Monkey's stacks module. That is so painfully slow (especially loading back the stacks into structs) that I'd rather have a plugin for that.

Okay, I'd ruin the fun for ScummVM, but for the key players (Windows, x86 Linux, Mac OS) I could simply provide custom builds.

Btw. couldn't both techniques be combined? Something like: plugin present -> use turbo, plugin missing -> long loading times, but playable.

[Calin Leafshade]

If speed is a factor *dont* use the stack module.

That module has huge overheads for fairly simple things which is fine when you just want a stack of a few characters but for something like a tile engine you really should use arrays.

Also Steve is suggesting that you 'cook' the maps which means that you process and resave them in a format which AGS is able to read much more quickly (\r\n delimited ints for instance).

If you do both these things then loading times for maps should be almost instant

[Kweepa]

Yes, again, I think you must be doing something wrong. Why would you need to use stacks? Show us the code!

[abstauber]

Back then I already knew that nobody would take a look 
http://www.adventuregamestudio.co.uk/yabb/index.php?topic=40922.0

Just kidding Here are the functions:
Painfully slow map loading: TENG::load_map_files
Stacks are used in teng.asc: TENG::save_room and TENG::load_room


edit: Forgot to answer Calin:
I still don't get how it would affect the speed in a good way. As far as I know AGS natively supports writing ints and strings. So for doubles and floats I'd still have to use strings and convert them in AGS script, right?

[Wyz]

@Wyz: Yaay, thanks for accepting the challenge
Could that future plugin maybe also have write functions? Pretty please? </puppy dog eyes>

At this point I have not a very clear idea about what said plug-in should be capable of. My first idea was to make a dynamic map plugin to store and access of very large data sets efficiently. The main problem I ran into is how to store structs. Well I don't know if you are still interested in such a plugin, if so tell me what operations it should be capable of.

[abstauber]

Oh, it seems that I've been a bit confusing here. Let me explain:

I've a map format (default GBA, no promotion specialty), which looks like this:

Position(bytes)	Type	Description
$00	DoubleWord	Number of tiles in a row(width)
$04	DoubleWord	Number of tile rows (height)
$08	Word	width * height Word values where each value is divided into Bit sections  0..9 tile index 10 flag if the tile must be displayer mirrored horizontally 11 flag if the tile must be displayer mirrored vertically 12..15 unused

In order to process that file Khris was kind enough to provide the functions. PM could also export .txt files, but processing these took even longer.

The next step is that a level consists of 5 maps (layers), all in the same format and the same dimensions. Layer 1 + 2 = background and foreground tiles
Layer 3 = Items, Objects, Platforms and Waypoints
Layer 4 = Player, enemies and other characters
Layer 5 = hotspots

So for a level to load, parsing all these files is needed and takes a lot of time. Even more when doing it with AGS script And of course this data is stored in arrays, not in a stack.
Steve is definitely right and cooking Layers 3 to 5 will speed things up (and I think, I'll do that too). Still the big part is loading the tiles.


Now the second thing I'm trying to achieve has nothing to do with map loading itself. Once the player advances to the next level, I'm saving Items, Object and Platforms to stacks and reload them once the level is being entered again. That way the level design can be non-linear and enemies don't respawn. Same thing for items and other objects.

These are pretty small arrays stored into stacks and I already shrunk the stack separator, but it still takes some time to save and load these. In combination with the already slow map loading, it feels like forever on a slower CPU


That's how things are working at the moments. In order to optimize things without having to write my own level editor, I thought switching from Khris' helper functions to compiled plugin functions could help.


@Wyz
Readword
Writeword
ReadDoubleword
WriteDoubleword
BitString
BitStringToInt

Those functions would be more than enough. The super-duper solution could of course be a complete map loader, but this is too much to ask for. Having those functions mentioned above would already be awesome²

[monkey0506]

I've actually learned that the ASCII standard actually supplies some non-displayable characters specifically for the purposes of data delimitation. I think I used these in the version of the List module that I released, which also has a vastly optimized algorithm for getting and setting data versus the existing Stack module. I've been meaning to update both but never really had much indication they were getting used.

In any case, I know that's not really the point of the thread, but honestly the Stack module's data access was pretty lazy on my part.

[Kweepa]

Here's a few low-level comments that might help a bit:

Why use ReadDoubleword and WriteDoubleword? Just use ReadRawInt and WriteRawInt instead - they are native. That'll be quicker than a plugin function. Ok, that won't help much.

You can replace BitString and BitStringToInt functions with & and >> operators for much increased efficiency.
For example:

        s = BitString(info, 16);
        tile[index].tileno[k] = BitStringToInt(s.Substring(0, 10));
        if (tile[index].tileno[k] != 0) 
        {
          if (BitStringToInt(s.Substring(10, 1))) tile[index].mirrored_x[k] = true;
          if (BitStringToInt(s.Substring(11, 1))) tile[index].mirrored_y[k] = true;   

Can be replaced by:

        tile[index].tileno[k] = info & 1023;
        if (tile[index].tileno[k] != 0) 
        {
          if ((info & 1024) != 0) tile[index].mirrored_x[k] = true;
          if ((info & 2048) != 0) tile[index].mirrored_y[k] = true;

Loops could be a little more efficient:

  int i, j = 0;
  while ( j < num_tiles_y )
  {
    i = 0;    
    while ( i < num_tiles_x )
    {
      int index = j * num_tiles_x + i;
      ...
      i++;
    }    
    j++;
  }    

Is a little quicker like so:

  int i, j = 0, index = 0;
  while ( j < num_tiles_y )
  {
    i = 0;
    while ( i < num_tiles_x )
    {
      ...
      i++;
      index++;
    }    
    j++;
  }    

As for the stack usage, you only use floats 3 times per "person". You'd be much better off just writing the floats as strings to the file and then reading them as strings and converting to floats.

  String floatString = String.Format("%f", people[n].x);
  fhandle.WriteString(floatString);

  ...
  String floatString = fhandle.ReadStringBack();
  people[n].x = String.AsFloat();

Then you can ditch the stacks entirely.

[EDIT] I ran a quick test on the version of PlaTENG you linked to earlier. With BitStrings, it took 1.5 seconds to load the first map. I just replaced the BitStrings with the & operator, and the load time dropped to around 0.1 of a second. That's probably all you need to do.

[EDIT 2] While running the tests, I found that if I run this code

  int q = 0;
  while (q < 30)
  {
    TENG.load_map_files("level1",16, 16, 145, 70, 0,0);
    TENG.clean_up();
    q++;
  }

It crashes loading waypoints when q is between 20 and 30. I fixed it by adding wp_index = 0; into clean_up.

[Khris]

I guess this is the main culprit:

Layer 3 = Items, Objects, Platforms and Waypoints
Layer 4 = Player, enemies and other characters
Layer 5 = hotspots

Three additional layers are read in their entirety while 95% of each doesn't contain any information anyway.

I'd use a map editor that loads the two tile layers, then allows to add all the extra stuff. This information is then saved using regular File methods or even as generated script supposed to be pasted into the actual game.

[Wyz]

Actually I agree with Steve, those operations would increase speed very much.

I was halfway through building a map loader plugin while I wondered how much it would increase speed. All it would do is load a map (simple file operation) and store it in the memory and allow access to it. It would still return a word value and I guess that the speed difference between storing it with a plugin, or in a array in AGS is not that much. Well I not sure  so I'll continue my work so you can put it to the test. Do introduce the changes Steve suggested.

[abstauber]

I've now implemented the suggested changes and here are the results:

No changes: 15.2s loading time
Replaced BitStringToInt with "magic" ( info & 1023;) and applied it to layers 1+2: 8.2s
Replaced BitString completeley: 1.3s
Replaced BitString completeley and replaced ReadDoubleWord with ReadRawInt: 0.2s

I'm speechless and have updated my avatar


Thanks a ton, all of you! I'd never thought that map loading could be that fast. Otherwise I've never asked for a plugin. And apologies to Wyz that I've wasted your time... sorry pal. I'll now go and learn about bitwise operations.


..yaay!

[Wyz]

OK, well I should have read more carefully what you were trying to do, then I should have tested it myself. But oh well I was only just starting so not much time was wasted. Great it's solved now!