I started a new game and tried to import masks that worked in the previous version, it says invalid colors are found and removed and leaves me with 1 color which only allows me to have 1 walkbehind etc.
Are you sure the mask is 256 colours with only the first 16 used?
Are you sure that the mask actually worked in the previous version? It used to import invalid masks without warning you about it.
Yes a few months ago I seemed to be able to use any colors. I even made some rooms using randomly colored masks.
What type of mask are you talking about? With walkable areas, you could probably get away with it; but for other mask types, you wouldn't be able to set up the interactions for randomly coloured areas.
Anyway, make sure you do as Gilbert says and use the first colours in the palette. An easy way to start would be to use this as a base:
http://www.adventuregamestudio.co.uk/stdpal256.pcx
which has the first 16 palette entries set to the correct colours for drawing masks.
When I first used AGS, I made a test room with 3 trees, I needed them to be 3 different walk behinds etc. because they were all in different spots. I made masks with 3 colors, I just chose a color out of my paint palette and made the walk behinds etc as needed with no problems. I have no pallete since I'm using 32 bit color.
Even if your game is hi-colour, the masks are 8-bit paletted images, which use the first 16 colours in the palette.