Suggestion: Color coded anti-aliased masks

[subspark]

I think room masks 'walk-behinds' could be greatly improved in the following ways:

* Ability to import 24bit Bitmap into mask editor.
* Colors are picked up and automatically assigned baselines and induvidual channels.
* Mask is premultiplied so that the black is transparent with no black halo or artifacts.

For example:


Note: For some reason masks currently need to have a vertical resolution identical to the game resolution.
For games with the LucasArts GUI or any GUI for that matter the room graphics may not extend to the bottom of the screen.

Such restrictions increase file size and and force unessicary resolutions onto masks beyond that of their graphical counterpart.

I beleive that with this technique we are not just killing two birds with one stone; it's a bird massacare and a life saver for those finnecky artists like myself who take pride in their pixel perfect walk-behinds.

Cheers.

[strazer]

* Ability to import 24bit Bitmap into mask editor

Yeah, even if not all colors are used 24bit format should be supported, if it's not alreasy. If you're talking about aplha-blending, see below.

* Colors are picked up and automatically assigned baselines and induvidual channels.

I think that would be too much work for too little gain. How often do you have that many walkbehinds anyway? And in most cases the baselines would probably need tweaking anyway, so you can just as well put them there in the first place.

For some reason masks currently need to have a vertical resolution identical to the game resolution.
For games with the LucasArts GUI or any GUI for that matter the room graphics may not extend to the bottom of the screen.

Why is the filesize of a mask important? It is only used at design-time, it doesn't affect the compiled game at all.

* Mask is premultiplied so that the black is transparent with no black halo or artifacts.

As to why alpha-blended walkbehinds are a problem, the this thread.

[subspark]

Why is the filesize of a mask important? It is only used at design-time, it doesn't affect the compiled game at all.

It's not. But the error message when you import a mask the same resolution as your background eg. 320x150 is inconvenient and unessicary.

As to why alpha-blended walkbehinds are a problem, the this thread.

Read it already. Many have said that you need two images for an alpha blended mask. First the color pass then an alpha.
You really only need one image; like the one I provided above. It essentially IS the alpha only that the colors determine exactly what channel each walkbehind is assigned to.

And in most cases the baselines would probably need tweaking anyway.

Not nessicasrily. The idea is to read the image from the bottom up, catching the first pixel of each color. Yes, the tool does suggest a change in mask format or the feature would be worthless. But thankfully that is up to the artist.

I think that would be too much work for too little gain. How often do you have that many walkbehinds anyway?

I have many walk behinds as our game is full of interesting places with complex hiearchy and design. The game runs at 32 bit color depth. As intruiging, practical and intuitive this is I fear you may be right. There arne't many games I am aware of that make full use of perspective and complex occluding however I beleive it is better to have the tool and not need it than need the tool and not have it.

As AGS advances so do the games. In fact many games transparently advance AGS. Isn't this the whole idea of being part of the developer community to share ideas particularly those that future proof AGS for more complex games like ours? Please don't get me wrong in a certain light I agree with you Strazer but is it not worth testing and ultimately adding to CJ's to do list?
It's not the tool thats worth worrying about it's the principle and how often it will be applied.

It is my strong opinion that such a tool will prove invaluable beyond a reasonable doubt.
Would anyone agree?

I appreciate your perspective on this folks. It's worth exploring at the very least.

[strazer]

Why is the filesize of a mask important? It is only used at design-time, it doesn't affect the compiled game at all.

It's not. But the error message when you import a mask the same resolution as your background eg. 320x150 is inconvenient and unessicary.

True, I'm sure that's doable.

As to why alpha-blended walkbehinds are a problem, the this thread.

Read it already. Many have said that you need two images for an alpha blended mask. First the color pass then an alpha.
You really only need one image; like the one I provided above. It essentially IS the alpha only that the colors determine exactly what channel each walkbehind is assigned to.

I have to admit I don't completely understand the problem, but Steve seemed pretty adamant about it. Maybe someone else can weigh in.

Isn't this the whole idea of being part of the developer community to share ideas particularly those that future proof AGS for more complex games like ours? Please don't get me wrong in a certain light I agree with you Strazer but is it not worth testing and ultimately adding to CJ's to do list?
It's not the tool thats worth worrying about it's the principle and how often it will be applied.

It's a matter of opinion. I think there a more important things for CJ to work on, but ultimately it's up to him to decide. So let's see what he has to say when he comes back (whenever, from wherever... ).

[subspark]

Agreed.
Although I would have thought the AGS majority's opinion would also be considered seeing we are the ones making the games here.
Not about to call a vote but do any AGS users have an educated opinion? Beware it is for the technical minded.Ã, 

It's up to you in the end CJ!
No problem.

[scotch]

There'd have to be some interface for saying where you wanted the mask to go on the screen... I think just making your masks full screen in your paint program is easier (you should probably paint them on a layer above your background anyway...), and there aren't really any drawbacks in size, they are rle compressed, which means that blank areas compress to almost nothing.

As for the problem with walkbehinds, while I'd love intermediate transparency levels for walkbehinds if they'd work, there is a problem that you probably aren't picturing, as pointed out in the thread Strazer linked to. Ã, To get the correct pixel colour for the alpha blended area between the character and the walkbehind you need three layers, the background without the walkbehind, the character, and the alpha channeled walkbehind. Ã, The reason for this I'm not sure I can explain properly, because I'm bad at that, but perhaps I can show you with an image:

What you want is this presumably, character blended onto background, walkbehind blended onto both of them.


What you'd get with your method of having a layer saying how much to make the character transparent as it walks around is a halo where the character overlaps the attempted semi transparent area, something like


This was just a quick example I made in photoshop, it can actually look worse than that.  It certainly isn't an improvement over no alpha channel.  So unfortunately there simply isn't enough information in your setup for AGS to do the alpha blending between walkbehind and character correctly.  The only way to achieve the effect would be to force people to draw their backgrounds in alpha channeled layers.

[subspark]

Not at all. You just read the mask as a premultiplied image. The transparent areas are black in my example so, of course, you would set the premultiply color to black.Ã, It's really not that daunting. Better explained in these examples:
http://www.gimp.org/docs/plug-in/appendix-alpha.html
http://www.td-grafik.de/ext/xfrog/alpha/index.html
Trust me I have done a lot of compositing for film and TV I know exactly how this tool will work or I wouldn't have posted it.

Cheers for your example though. Nice palm!

EDIT:
9 Logical steps for AGS import:

1. Check if file is is greater than 4bit. (NO ALPHA!) Just a straight image with at least 256 colors.
2. Read picture from bottom up.
3. Store picture in memory.
4. Remove black from image using techniques described in the latter of the above urls.
5. From stored picture in memory, find the first non black pixel (so ignore 0,0,0) in vertical direction for each shade of color. (Shades of red, shades of Green, Blue, yellow etc.) (Masks regions can not overlap and must be againt a constant color such as black) Note that in my example the base of each color region is a straight jump from 0,0,0 to the most bright of that color.
7. Set baseline on row of first pixels depending on color shade.
8. When all colors have been detected, store color data into memory.
9. Use aquired color data to seperate colored mask regions into different channels and move the baselines to their corrosponding channels.

[Privateer Puddin']

Eh, i'm pretty sure scotch (and steve in the last post, hopefully he'll see this)



On the left, the black is behind the green and infront of the red. This is how we want it to behave and how it does currently if you have alpha objects on a background.

The problem with having it the other way is that you can't seperate it and the character can't just go behind the green. It'll have both the green and red acting on it as far as i can see.

¬¬

[Snarky]

Pre-multiplied alpha wouldn't work here, since you don't have separate textures for the walkbehind object and the background (which is the whole problem in the first place), and since the background color varies from pixel to pixel.

If your background was always red and your foreground was always blue, you could say that in a purple pixel along the edge of the foreground, the red component must come from the background and the blue from the foreground. Then if a yellow object moved behind it, you could mix the blue with the yellow and get green, correctly. However, when both the foreground and the background can be any color, you don't know how they're each contributing to the blended color, and you can't filter out the background component.

[subspark]

I'm sorry Snarky but either I'm not with you or you have not fully understood the technique I am describing.

1. Yes you DO have two seperate images. All you need is two. One is the room background image, the second, the mask image.
2. The colors are easily distinguishable from each other because you are choosing from a series of shades. So as long as your colors values are proportionate to each other as they darken in shade then there shouldn't be a problem.

Or were you replying to Privateer Pudding's post, which I also didn't fully understand.

You could essentially do it all with photoshop layers but that wouldnt be as fun to program.
Such steps would include:
1. Read PSD or another format that supports layers.
2. Read picture from bottom up assuming that the first photoshop layer is the background color to become transparent.
3. Store picture in memory.
4. Remove black from image. (AGS treats the layers as premultiplied based on the color constant of the first photoshop layer. In my case, pure black RGB:0,0,0.
5. From stored picture in memory, and reading the file from bottom up, AGS finds the first non black pixel in each layer (so ignore 0,0,0). (Masks regions in this case CAN overlap due to the seperate layers in photoshop) Note that in my example the base of each color region is a straight jump from 0,0,0 to the most bright color. Just gives you better indication in Photoshop of where your basline will end up.
6. AGS sets a baseline on first row of non-black pixels in each layer.
7. AGS loads each detected color region from the photoshop layers and assigns them into a new channel, one layer at a time moving each color region into the next channel.

My method detects all the colors in a single color coded antialiased mask.

Instead of using layers to determine A. the color to premultiply to and B. To distinguish the shades of color from any other, you just need a bitmap. You import the bitmap into AGS.

1. AGS displays a color picker asking you which color is transparent and premultiplied.
2. AGS displays another color picker for you to grab the brightest pixel of each color (basically clicking in the middle of your mask and not the antialiased border)
3. AGS stores the opaque color constant and each shade is then assigned a level of transparency based on the color values chosen.
4. As long as your mask color regions are not overlapping each other in any way and are surrounded by a constant color such as black and are made up of a constant color themselves AGS then assumes that as each detected shade of one of the selected color constants, the shade's color values are proportionate to that of the color constant of that mask region.

For example if I paint a mask with a soft brush in photoshop, using a color such as (RGB:255,216,0) then the value of the darker shades of that color will be proportionate to the brightest value. So AGS knows that the brightest color is (RGB:255,216,0)Ã,  and any darker shades that are detected can be distinguished from lets say a red mask region in the same bitmap because you have specified that the values, in this case (red and green), do not subtract more than one another as they get closer to the premultiply color (in my case black).

So an example of a correct darker shade that would be distinguished from a darker shade of another color would be (RGB:239,202,0) and again darker (more transparent) (RGB:216,183,0). See the pattern? As long as AGS remembers the three color values of the brightest pixel of a given color region, the proportion of those three RGB values in conjunction with another should allow sufficient distinguishability.

In my case I use the fully saturated colors for my mask regions. So one of the color values is always 0. This isn't crucial and plays no part in destroying the relationship of the RGB value's proportions.

This is the brainchild I have been cooking up for a whole day now! Phew!

Now if there was only a n easy way to figure out how to achieve all this with overlapping color regions. Perhaps AGS could compensate between all the color constants.

Tell me some of you are following me here.

Maybe CJ could throw us a bone here. He is the most proven technical mind of all of us and knows AGS better than any.
You just have to get your heads around the logic involved.

As much fun this would be for any whole-hearted programmer I would personally opt for the photoshop layers approach. It's make things easier for AGS as the tool would seem to write itself in a manner of speech.

[Snarky]

It sounds like you still don't get the issue. The problem isn't providing gradations in the walkbehind mask to provide partial transparency. That bit is easy. The problem is filtering out the background component of those partly transparent pixels.

Pre-multiplied alpha is only able to solve this problem if the background is a known, flat color.

When you're drawing the screen, you're essentially doing it in layers. What you want is:

- foreground (walkbehind part of background image)
-object/character
- background

However, in reality there's no foreground. You're just not drawing the objects in that part of the screen:

-object/character
-foreground+background

So for the partly transparent bits of a soft foreground, say a 50% transparent pixel, you expect:

- foreground: 50%
- object/character: 50%
- background: 0%

But what you actually get is:

- object/character: 50%
- foreground+ background: 50%

And since the foreground and background have already been mixed at 50%, this is the same as:

- foreground: 25%
- object/character: 50%
- background: 25%

The result is halos, bits of the background "shining through" your character, and other nasty artifacts.

[Gilbert]

I didn't have time to read all this, but I just couldn't understand, is there a problem with using sprites (objects, characters, etc.) with AA (alpha-blended) edges for this? This is more straight forward and we don't need to deal with the troubles in making AA masks working.

[Gregjazz]

Now wait a second... if the problem that is being described is true, we should already be having those problems when two characters using alpha-blended sprites are overlapping. For example, when an alpha-blended character walks partially in front of another alpha-blended character. Then you have essentially the same setup of layers as you would with an alpha-blended foreground...

But the issue is that walkbehinds are NOT drawn like sprites. They are simply pixels of the screen that have orders not to draw sprite pixels, leave the background untouched. Apparently this is the reason why you have that background bleed-through problem, correct?

[Gilbert]

One problem I see is that a background mask is a single bitmap, in which it's very complicated to deal with overlaping edges.

[Snarky]

Now wait a second... if the problem that is being described is true, we should already be having those problems when two characters using alpha-blended sprites are overlapping. For example, when an alpha-blended character walks partially in front of another alpha-blended character. Then you have essentially the same setup of layers as you would with an alpha-blended foreground...

But the issue is that walkbehinds are NOT drawn like sprites. They are simply pixels of the screen that have orders not to draw sprite pixels, leave the background untouched. Apparently this is the reason why you have that background bleed-through problem, correct?

Right. I'm not sure whether you're responding to me, but we're saying the same thing. The problem is that we're trying to represent three layers, but in reality the "foreground" layer and the "background" layer are part of the same image. That means that when bits of the foreground are partially transparent, the only thing you have is the mix of the foreground with the background, not each separately. So when you're trying to draw the foreground on top of the object (transparently), you're actually drawing both the foreground and the background.

Maybe this clarifies the problem:


Background


Walkbehind mask


This is what we want...


...but this is what we get
(Notice that the policeman on the left is transparent, and that the cactus on the right has a halo)

[Gregjazz]

Very well explained, Snarky!

So it's almost looking like a better route would be to use objects rather than walkbehinds... but they're slower.

[Gilbert]

Well anyway, making the mask idea possible or making such a mask would be like nightmare, since there're multiple walkable areas, so far trying to develop it to work for a single area is hard already, making it possible for more areas would only make it worse.

[subspark]

Very well explained, Snarky!

I would have to disagree. With all due respect, Snarky, your example is not taking premultiplying into account. I took your red mask sprite into Adobe AfterEffects and used it as a color mask (basically a red alpha). I put one of my characters between a background made from your mask. I then told AE to premultiply the mask to black and voila! The result was as expected.

Despite what I am saying your simply repeatedly producing the same missinterpritation of this technique. I have practically solved this to good end.

1. There is no reason why the halo from the background should be there. I have explained, in detail, that with the use of the color coded mask, and selectivly premiltiplying the mask over a black background, or any chosen color for that matter, effectivly removes any halo and ALL black. Like the way a blue screen keyer works.

2.

Pre-multiplied alpha is only able to solve this problem if the background is a known, flat color.

The background IS a known flat color. That is the whole idea Snarky.

3. I suggest you play around with Adobe AfterEffects and try some premultiplying with my mask image against a photograph and a sprite in between. It works for me in AE and should work practically and to good effect in the form of an AGS plugin or module.

Cheers.

[Gregjazz]

If there was a way you could select the main color what the walk-behinds are, whenever there's a character or object partially under it, it could use that color to do the alpha blending rather than the background. That's better than nothing.

[Snarky]

Very well explained, Snarky!

I would have to disagree. With all due respect, Snarky, your example is not taking premultiplying into account. I took your red mask sprite into Adobe AfterEffects and used it as a color mask (basically a red alpha). I put one of my characters between a background made from your mask. I then told AE to premultiply the mask to black and voila! The result was as expected.

A background made from the mask? That doesn't make sense. The mask corresponds to one particular background: namely the first image in my post. Also, "put one of my characters between a background"? There is no between. The background is a single layer, and the characters go on top of it.

Despite what I am saying your simply repeatedly producing the same missinterpritation of this technique. I have practically solved this to good end.

Look, I read the pages you linked to, and I understood them. Pre-multiplied alpha is not magic. It allows objects with soft edges against a solid-colored background to be lifted off that background while keeping the soft edges. It does not allow you to recreate information that simply does not exist in the image.

A single image and a mask does not provide enough information to extract two separate layers. If a pixel in my image is green, and the mask shows that it's a 50% mix of the upper layer and the lower layer, how do I know whether the upper layer is supposed to be yellow and the lower blue, or vice versa, or whether they were both green in the first place? Pre-multiplied alpha doesn't begin to solve this problem. It assumes that the bottom layer is one uniform, known color. That crucial assumption makes the problem solvable, even trivial.

1. There is no reason why the halo from the background should be there. I have explained, in detail, that with the use of the color coded mask, and selectivly premiltiplying the mask over a black background, or any chosen color for that matter, effectivly removes any halo and ALL black. Like the way a blue screen keyer works.

It is not a question of removing the black. The black is only in the mask, anyway, it's not part of the image.

A blue screen keyer works because the thing you're keying out is a relatively solid blue color. There's no way to do that for an arbitrary background.

The background IS a known flat color. That is the whole idea Snarky.

No it bloody well is not! In the pictures I posted, for instance, the background is a wooden floor and violet walls with a green stripe. How does that equal a known flat color?

You need to get your head around what is actually going on here. Maybe this explanation will finally make it sink in:

AGS draws the background image. Then it draws the characters on top of that, but only the parts that are in the black part of the mask, unless the feet of the character are below the baseline of that particular walkbehind. This gives the illusion of the character being "behind" those particular parts of the background (hence: walkbehind). Essentially, the walkbehind mask is merely the inverse of the alpha channel for the characters layer, if we ignore the color coding and baselines.

If AGS allowed soft edges in the mask, it would simply mean that characters would be drawn semi-transparently at those pixels. You can say "premultiplied" as much as you like; it doesn't matter, since we're not actually drawing any part of the background on top of the character: we're drawing the character on top of the (unchanged!) background.

Now, if we were worried about halos around anti-aliased characters, that would be a different thing. Since character sprites come against a known background, you could use pre-multiplied alpha to make nice soft edges. Easy.

Not surprisingly, AGS already supports alpha-blended characters gracefully.