Suggestion: Color coded anti-aliased masks

[Wretched]

Firstly your idea although handy won't work for the following reasons.
i) You would have to use a mask with an alpha channel, you cannot use shades of colours fading to black to represent individual masks because of the accuracy of say 8 bits per RGB channel, example (255,128,0) would at some point reach (16,0,0) as it fades to black and would therefore be indestinguishable from a (255,0,0) area. So you would need a seperate alpha channel, or be limited to only having 7 mask colours.
ii) AGS automaticaly assigns walkbehinds to different shades. Ok, that's possible, now suppose you add another walkbehind somewhere in your mask. AGS will now probably automatically assign different values to all the walkbehinds. Great! not. Ok you would need to add a automatic import flag into AGS.
iii) AGS automatically calculates baselines. Fine for the first time you import your background, but you would need to be able to stop it from doing it everytime as you would lose any edits you had made to baselines that were auto generated not where you wanted them. Ok you would need to add a automatic import flag into AGS.
iv) Using the alpha channel in your mask will cause background artifacts to be seen through the characters, as has been amply explained by Snarky, though I don't quite agree with his reasoning, but that might be just me not reading it right, his result is correct.
v) All this can be eliminated by simply using an object with it's own alpha and baseline instead of the walkbehind. If you want to remove the overheads of lots of layers of objects, just cut out the 100% transparent and 0% transparent from the object graphic and use normal walkbehinds to do the job in these areas. I've already used this technique in Bog's Adventures in Underworld with no problems.
vi) 99% of AGS users probably don't even know what an alpha channel is. I think AGS should develop in a way to encourage beginners more and not get itself bogged down in advanced techniques, that can be solved by a little work anyway. But that's maybe a bit off topic.

[Scorpiorus]

So, basically the idea is to have an additional alpha channel information taken from the walkbehind mask which is to be used along with the one in the sprite itself when drawing that sprite onto the background?

alpha = sprite.alpha * mask.alpha; // where alpha range: [0..1]

screen = (1.0 - alpha)*sprite + alpha*background;

[Snarky]

Right, except I think you got the coefficients switched in the last formula. It should be:

screen = alpha*sprite + (1.0 - alpha)*background;

I found some of the claims made for premultiplied alpha a bit fishy, so I did some additional reading. To clarify:

1) Premultiplied alpha is an implementation detail in how computers represent transparent images internally, used to optimize layer compositioning. It doesn't change the result of the composition, it just means you can leave out one step (a multiplication, not surprisingly) in the process.
2) The methods described under the label "premultiplied alpha" are simply those of Porter and Duff's seminal SIGGRAPH paper. All modern alpha compositing is based on the principles described in that paper. In other words, they are exactly the regular ways that layers and alpha-transparency work, which anyone who's used Photoshop is already familiar with.

"Premultiplied alpha", then, is just a buzzword, and there's no magic technique that will get around the problems of alpha-blended walkbehinds. It would be nice if at some point AGS could support multi-layered backgrounds, or provide more efficient display of object layers. All kinds of cool transparency and parallax effects could then be achieved. In the mean time, I really like Wretched's solution of using objects just for the edges of walkbehind areas. Nice hack!

[Scorpiorus]

Right, except I think you got the coefficients switched in the last formula.

Yeah indeed, that's it. Also in the first formula I type mask.alpha while meaning the sprite part of alpha taken from mask, ie. it's meant to be already inverted. So, to avoid confusion let's make some redifinition: now mask.alpha actually is an alpha channel of the mask itself (not inverted), then:

alpha = sprite.alpha * (1.0 - mask.alpha);
screen = alpha*sprite + (1.0 - alpha)*background;


Anyway, my original input just was to sum up what we are after and make the suggestion clear.

How much it is needed/useful at the moment (vs. worth the hassle to implement) is another question of course

[subspark]

AGS draws the background image. Then it draws the characters on top of that

It doesnt matter in what order AGS processes are run, the way the technique works gives ags all ther nessicarey information prior to ags performing its tasks.

No it bloody well is not! In the pictures I posted, for instance, the background is a wooden floor and violet walls with a green stripe.

Well I'm sorry Snarky, but thats not the idea at all. Keying out the constant black color in the mask is the methodology behind this tool. If you have been following at all, you will note that the only instance where the background image (trees houses whatever) comes into play is what areas of that background the mask makes opaque/transparent. All proccesses are done using the color coded mask.

Look, I read the pages you linked to, and I understood them.

With all due respect my friend, I doubt that very much.

I really apprecite your interest in my topic and the tool i have proposed, however it would be wise to fully understand the clockwork behind the technique described before assuming that its flawed with no reasonable evidence to suggest as such. With every effort to discredit this approach of convenient one-step-is-all masking it is ever more clear to me that your on a completely different page and what I have proposed is every bit as valid as the positive results i got in After Effects.

A background made from the mask? That doesn't make sense. The mask corresponds to one particular background: namely the first image in my post. Also, "put one of my characters between a background"? There is no between

What I am describing is in After Effects. Yes I took took your mask and used Photoshop's powerfull tools to come up with a simple background that matched your mask. So with the first layer in AE being the background derived from and based on your mask, I then put the character on layer 2. Next i duplicated the background and used your mask to chop out the black areas with premultiplication. The result was essentially the same as having alpha blended foreground sprites but used every process described succesfully to get there.

Yes I will admit, it is a difficult technique to grasp at first glance. I understand it may be quite easy to overlook some of the finer details so my apologies if my explination of it is not straight forward enough.

Lastly, Snarky I found your last reply to be rather resentfull and rude. I posted this idea in the hopes that it would be explored with keen eyes and a welcoming attitude for new ideas. Whether you take new ideas on with a smile or the fat end of a baseball bat, I have no interest in flaming or ego-quake if you will.

I have provided you just like everybody else with the information one needs to make a mental picture of how this tool will function.

Everybody else is either interested in learning more about what I have described or joining the hate club. I have a long history with AGS and I'm not about posting suggestions without any groundwork to support my claims neither do I have time for childrens games.

Good day.

[scotch]

I don't see anyone flaming, it's just all the things you've explained don't actually make sense the way you've explained them.Ã,  I certainly can't replicate them in Photoshop or logically, it just doesn't work.Ã,  Perhaps you could show some images from your programs showing what you're doing?Ã,  It isn't like you're talking to people who don't have any experience with graphics programming or computer art in general. I'd love to know how to use this technique, but as far as I can see it just isn't happening without some potentially complicated algorithms for guessing information in the background image, to get rid of the haloing.

If you're not sure how to explain the process technically, just take some screenshots as I suggested, perhaps we can work out what aftereffects is doing.

[Pet Terry]

Yeah, I too would be interested in seeing what you have achieved with the technique you keep mentioning over and over again. It would be a neat feature to have, but I just can't seem to understand how it could possibly work.

[Snarky]

AGS draws the background image. Then it draws the characters on top of that

It doesnt matter in what order AGS processes are run, the way the technique works gives ags all ther nessicarey information prior to ags performing its tasks.

I thought that maybe when stated this clearly, it would become obvious why the technique you describe won't work. Let's try one more time: The only thing walkbehinds do is define areas of the screen where character sprites are invisible (not drawn). If you added shades to the walkbehind mask, they would define areas of the screen where character sprites are drawn with transparency.

Since you're not modifying the background image, you can't fix the fact that when walkbehind regions in the background image are partially transparent, you can see the "background" through them. When you're then drawing the character sprite transparently on top of that, you can still see "background", which is what causes the artifacts.

Look, I know you think this technique solves the problem, but I'm telling you: You still haven't understood what the problem is.

No it bloody well is not! In the pictures I posted, for instance, the background is a wooden floor and violet walls with a green stripe.

Well I'm sorry Snarky, but thats not the idea at all. Keying out the constant black color in the mask is the methodology behind this tool. If you have been following at all, you will note that the only instance where the background image (trees houses whatever) comes into play is what areas of that background the mask makes opaque/transparent. All proccesses are done using the color coded mask.

See, this is where we have the fundamental miscommunication. You're focusing on the mask, but that's not where the problem is. What the mask does is simple: it just acts as an (additional) alpha channel for the character and object layers. The thing you need to focus on here is how the actual background image and character sprite get composited in those areas where the mask indicates partial transparency.

A background made from the mask? That doesn't make sense. The mask corresponds to one particular background: namely the first image in my post. Also, "put one of my characters between a background"? There is no between

What I am describing is in After Effects. Yes I took took your mask and used Photoshop's powerfull tools to come up with a simple background that matched your mask. So with the first layer in AE being the background derived from and based on your mask, I then put the character on layer 2. Next i duplicated the background and used your mask to chop out the black areas with premultiplication. The result was essentially the same as having alpha blended foreground sprites but used every process described succesfully to get there.

That sounds correct (it's what I did to create the example showing the artifacts), so I can only think that you either did something wrong (that is, something that can't be done with just a background image and a walkbehind mask), or you happened to create a background where the artifacts are not noticeable. Halos are most noticeable if the walkbehind object and the "background" color have high contrast, and the character color is similar to the walkbehind. Transparent sprites are most noticeable if the "background" texture has a distinct, high-contrast pattern.

Why not use the background I provided? After all, that's the background the mask was made for/from. It would be much more compelling evidence for your claim.

Actually, the example I provided isn't ideal, since the walkbehind layer is all black. This would make it possible to reconstruct the two layers from the flattened background + mask. I'll provide another one instead.

Lastly, Snarky I found your last reply to be rather resentfull and rude. I posted this idea in the hopes that it would be explored with keen eyes and a welcoming attitude for new ideas. Whether you take new ideas on with a smile or the fat end of a baseball bat, I have no interest in flaming or ego-quake if you will.

I freely admit that I'm getting increasingly frustrated with your insistence on the viability of this technique, in the face of detailed explanations of how and why it breaks down. The reason I persevere is that I'm convinced that once you get that Eureka! moment, you'll turn into a perfectly reasonable human being.

OK. Another example.

First of all, do you agree that if what you propose works, we should be able to do partially transparent walkbehinds? I mean, that's the fundamental claim you're making, right?

OK, let's see if that works:


The background image. (I clearly missed my calling as an artist.) Notice that the steam from the pot is partially transparent.


The walkbehind mask. I didn't bother to color it, but you can pretend that it is red or some color instead of white. Notice that the steam is grey, to show that it's partially transparent.

These two bitmaps are the only things AGS knows about!

So what does this represent? Well:


The "background" part of the background image. Call it the "walkinfrontof" if you like.


The "foreground" part of the background image. In other words, the walkbehind region. (Pink is transparent, and the steam is still partially transparent.)

And they've been composited thus:

Composition of our background image. Keep in mind that our background image is a flat bitmap. This just shows how we created it.

OK.... Now let's add a character, or rather an object into the mix:

It's a ball, I think. (Pink, again, is transparent.)

OK, we have our materials. Now what do we do with them?

This would be great if it worked
What we would really like to do is to put the object on top of the "walkinfrontof", then take the walkbehind and mask it with the walkbehind mask and put that on top of the object:

masked by


Composite it like so...


And get this. Ta-da!

However, we can't do that!

What actually happens
What actually happens is that we take the object and mask it with the inverse of the walkbehind mask, then slap that on top of the background:

(positioned in the correct position on the screen, of course) masked by the inverse of


Composite it like so...


And get this. Ugh! Notice that you can see the wallpaper through the ball where it's behind the steam. There's also a faint green halo between the pot and the ball.

What you call "premultiplying" is just the step I've called "masking". If you paid close attention, you will have noticed that we're masking the object, not the walkbehind layer (which doesn't actually exist as far as AGS is concerned) or the background image. Because of this (and I feel like I've repeated this a hundred times now), there's no way to not see the wallpaper if you're seeing the steam. The two are part of the same image, and you cannot separate them.

Edit:
What you're describing you did with AfterEffects should be (but apparently wasn't) something very much like this:

Mask (premultiply) the background with the walkbehind mask:
masked by

Which gives the walkbehind layer:

Notice that we can still see the green wallpaper and red pattern through the steam. (In this picture, black is transparent. It is, in fact, the premultiplied layer exactly as it is represented internally.)

Draw the background, then the object, then the walkbehind layer:


Which, again, looks like this:

...which is still wrong.

I'd really like to see how you would get this to work properly using the "premultiplying" method you keep talking about, using only the background image and the walkbehind mask. Posting a link to a PSD file would work swell.

[subspark]

once you get that Eureka! moment, you'll turn into a perfectly reasonable human being

Forgive m,e if I am wrong but was that an insult?

Okay. I know where your at. I know where your at at it's what your trying to explain that is detering you from other possibilities.

The difference is with the way you describe my technique in AGS is, the item the mask affects. In After Effects you create a mask and use it to make a selected object transparent.
The result is like drawing the character around the object in the background essentially making it appear occluded by a particular object. Theres no particuarly beneficial reason why AGS should use the mask to affect the background and bring it forward in front of the character like you described in your 15th image. This is unessicary complication and limited in a number of crucial ways.

Chris could better offer his advice seeing he wrote the program; There may be more than one approach of making objects transparent in the engine.
If theres one thing I may have overlooked which seems to be the basis of your argument Snarky it is that AGS draws transparently 'backwards'. Instead of using a mask to cut the character or object like a cookie cutter with AA, it brings the background forward removing the ability to treat the background and objects as seperate layers. Am I correct in my understanding that all objects and characters are drawn on the screen and collapsed at runtime? If so I must admit that the way alpha's and masks work in AGS is contrary to any other instance of masking I have come across in my history of Computer Graphics. After Effects asside, if I did what I described in say combustion, or Nuke, the result would be the same. Is there any reason why AGS draws its transparency from the background?

Perhaps Chris could bring meaning and reason to this post. It seems to have walked astray and into some frustrated entanglement of conflicting ideas. We really don't know at this point whether AGS is capable of drawing transparency the way I have assumed.

Yes Snarky you are right to question my approach based on this factor but I do not beleive your frustration is justified. Patience and tollerance is something I hope every grown adult is adhere to on these forums.

My sincere apologies for sounding like I was ignoring your feedback however you must understand that in the history of computer graphics, AGS is the first tool I have come across that draws its transparency like it does. I don't know why it does it like that, I dont think any of us besides CJ really knows why. We don't even know if it can draw transparency using the golden standard method every imaging software has since adopted.

So if what you say is correct, Snarky, you'll excuse my ignorance. Such a hitch in my design was unexpected and understandable overlooked.
But we will see if this 'hitch' is really the gobstopper we all think it is. AGS has a track record of offering multiple techniques as a means of getting around some older and obsolete limitations inherent with its design.

A re-write is long overdue. All in good time I guess.

Cheers.

[Gilbert]

Just use sprites, I don't see much advantages in using such masks.

[Snarky]

once you get that Eureka! moment, you'll turn into a perfectly reasonable human being

Forgive m,e if I am wrong but was that an insult?

On the contrary.

Okay. I know where your at. I know where your at at it's what your trying to explain that is detering you from other possibilities.

The whole "walkbehind" technique, where you take a flat bitmap and assign different regions to different virtual layers, has inherent limitations. The solution is to use actual layers (which AGS provides, although inefficiently, in the form of objects). That's not exactly news.

The difference is with the way you describe my technique in AGS is, the item the mask affects. In After Effects you create a mask and use it to make a selected object transparent.
The result is like drawing the character around the object in the background essentially making it appear occluded by a particular object. Theres no particuarly beneficial reason why AGS should use the mask to affect the background and bring it forward in front of the character like you described in your 15th image. This is unessicary complication and limited in a number of crucial ways.

I must be counting differently from you. I assume you're talking about this:



That is NOT how AGS does it. The picture illustrates what you described: "So with the first layer in AE being the background derived from and based on your mask, I then put the character on layer 2. Next i duplicated the background and used your mask to chop out the black areas with premultiplication." It shows why this would cause artifacts even in AfterEffects.

AGS doesn't do alpha-blended walkbehinds, so none of the pictures illustrate exactly what AGS does. However, this picture shows how it would work if you simply added alpha-blending to the walkbehind masks, as you proposed:

Chris could better offer his advice seeing he wrote the program; There may be more than one approach of making objects transparent in the engine.
If theres one thing I may have overlooked which seems to be the basis of your argument Snarky it is that AGS draws transparently 'backwards'. Instead of using a mask to cut the character or object like a cookie cutter with AA, it brings the background forward removing the ability to treat the background and objects as seperate layers.

No. No. No. Just look at the images, will you, and read what I wrote! "If you paid close attention, you will have noticed that we're masking the object, not the walkbehind layer (which doesn't actually exist as far as AGS is concerned) or the background image."

Am I correct in my understanding that all objects and characters are drawn on the screen and collapsed at runtime? If so I must admit that the way alpha's and masks work in AGS is contrary to any other instance of masking I have come across in my history of Computer Graphics. After Effects asside, if I did what I described in say combustion, or Nuke, the result would be the same. Is there any reason why AGS draws its transparency from the background?

Alpha transparency and masks work the same way in AGS as they do in familiar image editing apps. Based only on the background image and the mask, there's no way to get the ball behind the steam but in front of the wallpaper WHATEVER application you use. The whole notion of walkbehinds rarely comes up in image editing, because it's not needed. It's a trick from the days of 2D computer graphics for creating an illusion of depth when you only have a flat bitmap to move sprites across. It really is very useful, and very simple, which is why it's still being used.

Look, I've posted exhaustive descriptions and some twenty images to explain this now, and you still seem to be confused. I don't know what more I can do. I give up

[subspark]

Thats your 15th image based on the post before your last.

Forgive m,e if I am wrong but was that an insult?

On the contrary.

On the contrary WHAT. I'd say your avatar message says it all.

As much as you have offered to explain the problem here I don't beleive there is any problem with my technique. I had a programmer look over it and he thinks it's quite mountable. When the tool is finished, I'll get him to explain how it is done.
My dear Snarky, if you feel that you have exhausted yourself and are giving up trying to explain flaws in my design that arent really there then thats fine with me. Saves me the headache of going back over my design and explaining it lamens terms. However I don't beleive there is a simple way to explain this. So I will use what knowledge I have gathered during many years of compositing and put it to good use in the realisation of a plugin. With a programmer this shouldn't be a problem.

Now could somebody lock this goddamn thread. It's original intention has been extinguished apparently and serves no more puspose than to boil everyones blood. I regret I posted it in the first place.
Beers.

[Kweepa]

Yes, it's probably best if you write a plugin to prove to yourself that the technique you originally described is insufficient.

Of course it's perfectly possible to get the effect you desire, assuming you're drawing the backgrounds. That doesn't require a plugin, just a bit more data - using objects for the foreground.

Steve

[subspark]

Sure, I agree mate. But theres more than one way to do anything in practice. This should prove an interesting challenge for us. I'll get Andre to post a walkthru on how we acheived it in the forseeable future.

Thanks for your support.

[Snarky]

Thats your 15th image based on the post before your last.

No, that's actually the 17th one. You missed the small one, the image with the ball (twice). Anyway, it shows the same thing that I guessed: creating a foreground layer by "premultiplying" the background with the walkbehind mask.

Forgive m,e if I am wrong but was that an insult?

On the contrary.

On the contrary WHAT. I'd say your avatar message says it all.

I'd say that giving you the benefit of assuming you would be a reasonable person if you just got this misapprehension out of your head is a compliment, not an insult.

As much as you have offered to explain the problem here I don't beleive there is any problem with my technique.

That's OK. I've given up trying to convince you, since you don't seem to comprehend even the most basic parts of my posts. In response to:

"What you're describing you did with AfterEffects should be (but apparently wasn't) something very much like this"

You wrote:

"If theres one thing I may have overlooked which seems to be the basis of your argument Snarky it is that AGS draws transparently 'backwards'. Instead of using a mask to cut the character or object like a cookie cutter with AA, it brings the background forward removing the ability to treat the background and objects as seperate layers."

That indicates a level of misunderstanding that I just can't help you with. No one else seems to have much trouble understanding what the problem is.

I had a programmer look over it and he thinks it's quite mountable. When the tool is finished, I'll get him to explain how it is done.
My dear Snarky, if you feel that you have exhausted yourself and are giving up trying to explain flaws in my design that arent really there then thats fine with me. Saves me the headache of going back over my design and explaining it lamens terms. However I don't beleive there is a simple way to explain this. So I will use what knowledge I have gathered during many years of compositing and put it to good use in the realisation of a plugin. With a programmer this shouldn't be a problem.

Knock yourself out. Of course, it would probably be a better idea to mock up the method in Photoshop first: Using only a background image layer, a walkbehind mask layer (with soft edges) and an object layer, create a layer with the effect of an object partially hidden behind a soft walkbehind. Then post it here. Your claim would then either be proven or disproven. It would save your coder friend some work.

[Ali]

I understand why alpha blended backgrounds wouldn't work properly, especially for glass, steam or water. I still thing the wouldn't-really-work method could be good for anti-aliasing though.

When I use an anti-aliased backdrop I always try to draw the walkbehinds slightly smaller than the foreground parts of the image. That way the AA part of the backdrop won't appear in front of a character.

If a dodgy method of alpha blending was employed wouldn't it be possible to do the same thing to achieve preferable results:

(my drawings are normally better than this)


The first image shows what my scenes look like now. The second shows what they could look like with this not-properly-but-sort-of method. They aren't 'correct' because the character overlaps slightly, but that would be less noticeable at higher resolutions. I think the overlapping on the right image is less bothersome than the aliasing on the left image.

Do let me know If I've misunderstood the problem. I don't think I have, I just reckon we could work around it if alpha blended walkbehinds were implemented.

[Kweepa]

Ali, you have understood correctly.

However, it's debatable that your technique looks better - in my opinion, the left image looks better, because the yellow block isn't cutting into the red one so much.

I don't see the point in implementing something that only "sorta" works, given how small of a niche it would supply, how much it would have to be explained, etc. If you really want alpha blended walkbehinds, why not just use objects? If you're worried about efficiency, use Wretched's technique described earlier.

[Ali]

That makes sense, I'd missed Wretched's post. His technique does sound quite nifty.

Sorry for extending this thread unnecessarily!

[Gregjazz]

What if you had it so that there would be three things required for alpha-blended walkbehinds:

- Background (normal)
- Background (without anti-aliasing)
- Alpha channel

So you would have a copy of the background without the anti-aliasing (which the alpha channel would refer to)

I believe this would be one solution, if someone REALLY needs alpha-blended walkbehinds, though it is more work.

But yes, I would rather not have support for alpha-blended walkbehinds if they only half work.

[Pumaman]

Taking this thread back to basics for a moment, I do agree that allowing 24-bit BMP import for walk-behind masks, with AGS then automatmically assigned a walk-behind colour to each unique colour in the bitmap, is a good idea and should be supported.