Topic: Automated process for drawing pixel-art foliage?

[Monsieur OUXX]

Hello,

Normally Critics Lounge is to sho something that you've made and ask for constructive criticism.
But it's also a space for discussing the drawing techique used -- so, not to set a precedent, but I'd like to ask you tips on a specific topic : drawing foliage in pixel art.

Foliage in pixel art looks differently for every pixel-artist, but in the end I still think you can categorize the result in 3 families :

- Messy artifact-y foliage : That's usually when the artist simply scaled down a picture of a tree.      Caracteristics :       x There are almost as many pixels as there are leaves       x There are almost as many shades of green/brown as there are leaves       x The contrast is usually very high and there are way too many black pixels -- because in the original tree picture, the areas that were in the shadows (especially the self-cast shadows of the tree) were rather dark, and the artist failed to adapt that original shading to match the overall shading of his/her pixel scene.     Example 1 (background foliage): PLEASE NOTE: I'm not criticising the guys who drew the backgrounds  for this "Tanis" demo as I think they made an outstanding pixel art  job for everything else. Only the foliage bothers me.       Exampe 2: these ones were probably drawn by hand, and yet they still have this messy aspect and exagerated contrast :       - Overdrawn/overstylized foliage : That's when the artist actually drew the leaves one by one and got carried away with the shading. It gives everything a cartoony look or a "pillow shading" look.     Example 1: (see the leaves in the foreground)       Example 2 :  (notice how each leaf is drawn one by one in the middle tree) (The crystal shard team produces amazing graphics but that's not the style  we're aiming for)       - The magic look : (the look Id like to achieve) the foliage has exactly the right level of detail and number of colors. It has a crisp ascpect and yet it's not busy.     Example 1 (Fate of Atlantis as always      Example 2        Strangely, FotAQ artist succeeded here. Even though there's a lot of copy and paste.

So, now, the discussion: any ideas on how to achieve that render effect?


I was thinking the following transformations/filters in photoshop:
- gaussian blur (to remove unnecessary details)
- color quantizer (it's like a better posterize) to reduce the number of colors
- denoiser to remove the few rogue pixels remaining and giving an overall smooth (yet crisp) aspect
- and of course at the end a brightness-contrast (or better yet: curves) adjustment.

Any better idea?

[ThreeOhFour]

For the final image, this would be pretty simple to replicate using photoshop brushes, using pencil mode to stay away from soft edges, as this is basically what they've done. Just start with darkest shades, and work up through your ramp as you paint layers. I suggest some editing afterwards to lose the obvious copypaste look, though some don't bother with this.

Edit: Something similar I've done in the past is painting one small piece of foliage very carefully, then copying and pasting it around the image on all areas of foliage. Then I paint the seams in and add variety. This saves a lot of time and if you paint it in enough, it's basically impossible to see any repetition.

[Monsieur OUXX]

For the final image, this would be pretty simple to replicate using photoshop brushes, using pencil mode to stay away from soft edges, as this is basically what they've done. Just start with darkest shades, and work up through your ramp as you paint layers. I suggest some editing afterwards to lose the obvious copypaste look, though some don't bother with this.

Edit: Something similar I've done in the past is painting one small piece of foliage very carefully, then copying and pasting it around the image on all areas of foliage. Then I paint the seams in and add variety. This saves a lot of time and if you paint it in enough, it's basically impossible to see any repetition.

I understand what you're saying, and I totally agree that it is the "manual" method that gives the best results. I used to do that all the time with Deluxe Paint.
BUT I'm trying to find powerful new tricks to semi-automate the processes.

[ThreeOhFour]

If you're keen on using Photoshop trickery, have you considered using photo textures over scenes and then reducing colours? It's not an "automated process" but it's a very fast way of working, and is pretty common.

[Monsieur OUXX]

If you're keen on using Photoshop trickery, have you considered using photo textures over scenes and then reducing colours? It's not an "automated process" but it's a very fast way of working, and is pretty common.

That's indeed the kind of things I'm discussing. I've suggested a similar solution in the first post; but I was thinking that maybe someone had actually pushed their research further than I.

[Ilyich]

That last FotAQ background looks like it was painted with a custom scatter brush, which is a technique I personally quite like for foliage, and have used extensively in Quest for Infamy, admittedly with mixed results.

x1 x2 x3 x4

It's also a bit harder to use for cleaner pixel-art styles, but I've done a (very) rough test, and it seems reasonably easy and efficient:

x1 x2 x3 x4

The basic steps here are:

- Block out the general foliage shape with your darkest colour.
- Draw your brush - it's just this blob with 3 leaves in my case, but for better results you can create a variety of brushes, ranging from single leaves to meatier clusters that can fill an entire area.
- Set your brush to scatter and start layering that foliage, from darker to lighter. I'm only using 3 colours here, but you can use more, nobody can stop you. I'd suggest adding some size and rotation jittering for the lower levels, so it doesn't look too samey, but leaving the brush 'as is' for the brightest layer, to get that variety in detail and sharp highlights.

Some additional fun can be had by adding separate gradients to these layers of foliage, adding colour jitters to your brushes, or by cleaning it up by hand at the end, but I am too lazy to try any of that out.

Not sure if this method is helpful or painfully obvious(and it's pretty much exactly what Ben's already described, only I have pictures), but that's roughly how I would approach corner-cutting when painting lots of trees.
And if anyone wants to use that brush sample - feel free to take it, of course.

[Mandle]

This looks to be an incredibly useful thread!

(Could even be included as part of a workshop/video tutorial in the future?)

[Monsieur OUXX]

- Set your brush to scatter
(...)
- add some size and rotation jittering

Are we talking Photoshop settings here? I'm so bad at Photoshop brushes. Every single time I have to check Google to even remember how to create one.

[Ilyich]

Are we talking Photoshop settings here? I'm so bad at Photoshop brushes. Every single time I have to check Google to even remember how to create one.

Sorry, should've elaborated more on those in the first place! Here's a quick rundown on the most important Photoshop brush controls for this kind of task:

- Creating your brush - draw the shape or pattern you want to use in black on white (or transparent) background, then go to Edit -> Define Brush Preset, give it a nice name ("Sampled Brush 6" in my case), and now you have your very own custom brush!

- Press F5 (or Window -> Brush) to open the detailed Brush settings.

Here, in the Brush Tip Shape tab find your new brush, select it, and set the appropriate Spacing.
Spacing sets the distance between each brush tip placed, so setting it low means smooth, unbroken lines, while setting it high means you'd see each individual brush mark separately.

- Now, on to Scattering!
 
Scatter basically moves your brush marks around randomly, the higher you set it - the further apart they'll jump. Ticking the 'Both Axes' box would mean that your marks would scatter in every direction, making the pattern more random, but harder to control.
Count refers to the amount of marks being spawned
Count Jitter adds randomized variation to the Count parameter

- For more customization and randomization, Shape Dynamics is a good next step.

I won't go into detail about what all of these do, since it's easier to just try them out and see, but the general idea is that 'Jitter' parameters add some variation to different aspects of your brush.
Using this with the 'Pencil' tool for cleaner pixel art can be pretty problematic, since it mangles small brushes quite badly, but can still be useful for creating complicated patterns and shape edges.
 
There are similar controls for the colors in Color Dynamics and for opacity in Transfer.

And here's what it looks like when you just draw a straight line with these different settings -  a picture is worth a thousand words, after all.

x1 x2 x3 x4

[Slasher]

PaintShop Pro has a Leaf brush

Great Ideas

[Monsieur OUXX]

Sorry, should've elaborated more on those in the first place!
(...)

Thanks A LOT for this.
And now, you "shouldn't" have. But the fact that you did anyway makes you awesome

[Mandle]

Wow Ilyich, that is an amazing mini-tutorial!!!