Almost any way you slice it, you’ll be running into the issue that you’ll
need enough polygons on your model to support crisp edges. See here:
http://www.zbrush.info/docs/index.php/Polypainting
(you can learn a great deal more by checking out the rest of
that site as well)
to get an idea of what’s going on (this ties in directly with masking, since
that can only effect the polygons beneath it similar to polypainting).
Additionally, when I’ve got a very specific shape or need for control for
a mask I’ll use Zapplink and make the mask in Photoshop with black,
then ‘Mask by Intensity’ under masking palette…sometimes I’ll
go back and forth multiple times to get the mask over the entire model.
Hope it helps.
WailingMonkey