MODULE: List v0.3 BETA - Specialized dynamically expanding data storage

Started by monkey0506, Thu 05/08/2010 10:34:59

Previous topic - Next topic

monkey0506

It all started with the concept by HeirOfNorton that with the virtually infinite data storage the String type offered we could serialize integers into a dynamically (and automatically) sized container. I expanded that idea to floating point decimals and strings of text with the VectorClass module. Next came the Stack module which allowed many more types including Character, GUI, Hotspot, Object, etc. as well as custom data types and other Stack objects.

So what is the List module? The List module actually goes back to the specialized types that the VectorClass module used, but allows a couple of very important differences. The List module will make it possible to serialize certain pointer types previously not considered candidates, namely DynamicSprite and Overlay. Also, there will be* a generic List type comparable to the StackData type used in the Stack module which will allow you to pass specialized lists as function parameters, and store them as members of custom structs.

Anyway, I'll update this as I continue to develop it. Thanks for any feedback.

Download v0.3 BETA

To-do:
-Add ListOverlay for Overlay functions.
-Add ListViewFrame for ViewFrame functions.
-Add generic List type for including lists in custom structs and use as function parameters.
-Add conversion function for List type (in UNSUPPORTED functions)
-Optimize ListDynamicSprite and ListOverlay functions.
-Optimize PushArray and PopArray for all types.
-Add ability to push/pop null pointers for lists of all pointer types.

9 August 2010

Uploaded v0.3 BETA.

Optimized most functions for faster data access, and added eListEnforceRoomSpecificPointers to make room-specific pointers such as Hotspot, Object, and Region return null if Popped from a different Room than they were Pushed into the list in. Added constant List_VERSIONSTRING for use with functions such as Display.

6 August 2010

Uplodaded v0.2 BETA with the following changes: Added ListFloat, ListCharacter, ListInventoryItem, ListGUI, ListGUIControl, ListButton, ListInvWindow, ListLabel, ListListBox, ListSlider, ListTextBox, ListHotspot, ListObject, and ListRegion types for working with their respective data types.

I also added a to-do list.

5 August 2010

v0.1 BETA of the module is very preliminary. The reason I'm releasing it is primarily to request further testing of the DynamicSprite functions. I am taking some complex steps to be able to provide serialization of the DynamicSprites. I have done testing, but I know that being the author of the script I can sometimes overlook things. The module presently includes functionality for integers and Strings in addition to the DynamicSprite functions. The DynamicSprite methods don't include the FIle functions that the integer and String methods do, and I'm still experiencing some issues with getting the PopArray function to work for the DynamicSprites. The former likely won't be included because it would require the DynamicSprites themselves to be saved to disk.

*v0.3 BETA of the module does not yet implement the generic List type, or support Overrlay functions.

Monsieur OUXX

where's SCR_NO_VALUE defined? I can't find it. Well, who cares?

EDIT:
From what I saw, you basically mix 2 technologies :
- For ints and Strings, you use the good old stack module, that relies on Strings.
- For other data types, it's similar in concept to my module for lists of unmanaged structs: You have a unique, "static" array of pointers (in my case, it was an array of structs) that's stored inside the module, and the scripter uses fancy indices to items in this array.

...Correct me if I'm mistaken!



The difference lies here :
- Your code is very elegant (use of "this", and the use of a hashing system looks sexy), doesn't need to to be adapted to the use, and is super simple to use. Your system uses strong types.
- My code has to be duplicated but allows the handling of as many lists as wanted. Also, an item in a list can actually be another list. My system uses weak types, which makes it more flexible in some circumstances, but the scripter has to be careful.


How to chose?
- Your system covers 90% of uses and is ready to use.
- My system is required by perverts when they want some sort of homogenous "pointers" system, where the "pointers" can refer to any object of any type in any list (similar to scripts such as Javascript, etc.)
 

monkey0506

It's a secret, undocumented, built-in AGS value that probably isn't intended for general consumption. I don't know if it's been changed since AGS 2.72, but in that version it was defined as 31998. Depending on the implementation you could potentially need higher values than that for some of the parameters..so it would probably make more sense to add an enumerated value and just set SCR_NO_VALUE as the default. That way it could be easily changed if needed.

Since you're examining my code you might also be interested in my use of the unsupported indexer technology. I only make that available if a macro is defined, which is not defined by default. There's technically no advantage from a speed or overhead point of view, but it looks cooler. :=

Also, I've decided to PM you some information taken from AGS 2.72, which should still be relevant to your interests.

Monsieur OUXX

Quote from: monkey_05_06 on Thu 05/08/2010 11:10:59
It's a secret, undocumented, built-in AGS value. In that version it was defined as 31998.

Haha, mine is defined as -32000. :-)))
Give 2 choices to 2 humans, they will never agree! :-D

I forgot to tell you : WELCOME BACK!
 

monkey0506

First of all thanks on the welcoming. :)

As for int and String (as well as the other types which were implemented in the Stack module, once implemented) the storage method is the same as that used by the Stack module. The difference here being that the end-user does not have to perform the conversion of the data. It is my supposition that the user will generally want to keep only a single type of data within a single collection (counting complex types as a single type). If they do want to store varied types of data, they can always use the Stack module. :P

Certain pointer types cannot be generically serialized. Unlike Character, GUI, Hotspot, Object, etc. which are globally accessible from the respective associated arrays, types like DynamicSprite and Overlay (my two main points of focus for pointer-types at the moment) do not have any such arrays. Further, DynamicSprites and Overlays require persistent pointers or the objects they point to will be automatically cleaned up by AGS's garbage collection. For these types I am using a combination of the String-based serialization of the VectorClass and Stack modules and a managed linked list.

To explain (technical mumbo jumbo warning!), what gets stored in the Data member of a ListDynamicSprite is a unique 128-bit String generated at the time that the sprite is pushed into the list. This gets stored in the same way as a String gets stored into a ListString. There is a global DynamicSprite[] and ListString which are used in addition to the user's ListDynamicSprite. The DynamicSprite[] holds the actual DynamicSprite*s necessary to keep the sprites from being deleted (even if the end-user does not keep a pointer to the sprite after pushing it into the list). I haven't fully tested it yet, but the DynamicSprite[] should not store any DynamicSprite more than once.

The global ListString that gets used stores a CSV list of all the unique IDs referencing a specific sprite. The indices in the ListString directly correlate to those of the DynamicSprite[]. If the user pushes a DynamicSprite into the list that is already stored in the array then the value will simply be appended to the appropriate entry in the ListString. Otherwise both will be dynamically expanded, the sprite being added to the array and the unique ID added to the ListString.

Popping data just reverses the storage process by looking up the stored unique ID and returning the appropriate DynamicSprite from the array.

I hope that makes sense to you. It is nearly 5 in the morning here. ;)

Anyway, in addition to just adding specialized, "strong-typed" lists, one of the purposes of this module is to allow the serialization of these pointer types. Adding them to the Stack module would be very messy due to the delayed conversion process, whereby a user could potentially store the StackData long after the associated data had been destroyed.

There's a couple more points before I go that I wanted to address..you say that your code "allows the handling of as many lists as wanted" but that's not really a difference since you can create as many instances of the various List types as you want (within physical memory limitations of course). And as for your consideration that weak-types may be beneficial, there is still the Stack module of course. Also, I plan on implementing the generic List type, which will be enforced by the associated functions to keep the strong-type, but I plan on implementing a conversion function.

Not that it's a contest or anything, I just wanted to address the points that you raised. ;)

Monsieur OUXX

Quote from: monkey_05_06 on Thu 05/08/2010 12:08:04
that's not really a difference since you can create as many instances of the various List types as you want

Can you provide a snippet of how you create two lists of sprites?
Do you mean that you create two lists of Strings where you'll store the Uid's of the DynamicSprite*'s ?

In that case, yes, the result is similar. I'm working with int indices instead of String uid's. My goal is to have direct access to every single array or list. There's no conversion at any time.

The reason why I describe the similarities and differences is indeed not because of some sort of competition. It's because, so far, those paradigms were not addressed by any module. It's slowly starting :-) YEY!

Anyway. PEOPLE, YOU MAY NOT START SCRIPTING IF YOU'RE NOT USING MONKEY'S MODULE. Unless you like re-inventing the wheel, or suffering when managing your arrays.

I demand that this module is added to the "Modules" section  :)

 

monkey0506

Hah. Well there is a significant reason behind the version numbering and the "BETA" status label. This module isn't yet fit for general consumption, but I appreciate the support of course.

As for a code snippet, it's pretty complicated but it looks something like this (Edit: I thought here you were asking for a code snippet explaining more about the internal workings of the module, not the end-user scripting..see my post below for that instead.):

Edit: [removed confusing internal code explanation]

I tried to work up some pseudo-code, but it got overly complicated. :P

Monsieur OUXX

By the way, I've seen that you're looking for testers, I'll test the module if I can. It's always the main obstacle to release something clean.


I didn't quite get your example : why did you use an array of DynamicSprite* when there's already one in the module??? I thought only the UIDs would be needed by the scripter.
 

monkey0506

I misread what you were asking for before. Sorry.

The end-user doesn't need to worry about any of what I posted there. That's the beauty of what the module is doing. What I posted is what's taking place internally. Here's an example of the (actual) code that the end-user would use:

Code: ags
// global script
ListDynamicSprite Sprites;

// inside some function
DynamicSprite *sprite = DynamicSprite.CreateFromExistingSprite(42);
Sprites.Push(sprite);
// at the end of the function the local pointer 'sprite' is released
// this is okay because the Sprites list will serialize the pointer internally

// elsewhere
DynamicSprite *sprite = Sprites.Pop();


For two different lists you would simply do:

Code: ags
ListDynamicSprite ListOne;
ListDynamicSprite ListTwo;


And use them each the same as the above example.

You can't create a dynamic array of custom data types, but it would be possible also to create a static array:

Code: ags
ListDynamicSprite SpriteLists[5];


Edit: Also, if you don't need to use the pointer at the time that you push it into the list, you don't even have to store it beforehand:

Code: ags
ListDynamicSprite Sprites;
Sprites.Push(DynamicSprite.Create(30, 45));

Monsieur OUXX

Yey, that's cool!

I'll have to re-read the source, I didn't spot how that was done (I'll find out by myself, no need to confuse the thread's readers) ;)
 

monkey0506

Mostly this is a lot of copypasta. In fact, I haven't even tested the new functions because that's how awesome I am.

SMF spam blocked by CleanTalk