Anti Aliasing in Post Topic is solved

Please use this part to report bugs & errors, ask questions & "How to..."
Post Reply
oldmandarin
Posts: 8
Joined: 12 Jan 2017, 04:41

Anti Aliasing in Post

Post by oldmandarin »

Hey all!

Does anyone know why applying AA in post (or using the effects panel for AA) after drawing is the standard for Japanese productions? Does it have something to do with using colored lines?

Thanks!
User avatar
schwarzgrau
Posts: 1238
Joined: 23 Jan 2012, 22:08
Location: Offenbach / Germany
Contact:

Re: Anti Aliasing in Post

Post by schwarzgrau »

I'm not really sure, but I guess it's easier to colorize if you don't have semi-transparent edges.
Windows 11 22H2 / TVP 11.7.0 PRO WIBU / Cintiq 22HD
Windows 11 22H2 / TVP 11.7.0 PRO WIBU / Mobile Studio Pro 16" (2019)
Android 13 / TVP 11.7.0 / Galaxy Tab 7 FE
INSTAGRAM
Jet
Posts: 51
Joined: 31 Jul 2015, 10:48
Location: Japan

Re: Anti Aliasing in Post

Post by Jet »

I think it was mainly so the frames could be stored at a high resolution, if I remember right Studio Ghibli stored them as 5K, 150DPI, TIFF using CCIT Group 4 compression (aliased). Coloring techniques were then made around that workflow, HDD's weren't as cheap back in the day like they are today.
OS: Win 11 / RAM: 128GB / GPU: RTX 3080 Ti / Display: Cintiq 27
User avatar
Hironori Takagi
Posts: 279
Joined: 14 May 2018, 10:15
Location: Tokyo, Japan
Contact:

Re: Anti Aliasing in Post

Post by Hironori Takagi »

Hello, oldmandarin
I will explain the flow using not anti-aliased images in Japan below.
The big reason is that it is easy to paint without gaps.

I can not answer a detailed question because I am not good at English.
Thank you.
20180717.jpg
TVPaint 11.7.2(Nov 14 2023), Windows11 Pro, HP Spectre x360 Convertible 14-ea0xxx / TVPaint 11.7.1(Dec 22 2022) Mac OS 11.6, Apple MacBookPro M1 2020
oldmandarin
Posts: 8
Joined: 12 Jan 2017, 04:41

Re: Anti Aliasing in Post

Post by oldmandarin »

Hironori Takagi wrote: 17 Jul 2018, 02:11 Hello, oldmandarin
I will explain the flow using not anti-aliased images in Japan below.
The big reason is that it is easy to paint without gaps.

I can not answer a detailed question because I am not good at English.
Thank you.
20180717.jpg
This makes so much sense! I've also just discovered the OLM smoothing plugins for AE, so I'm going to try that with this method as well. Thank you so much (and to everyone else who replied) !

Also, follow up question, how do you colorize your blue and red lines? Or do you just color to the line, and turn the layer off?
User avatar
melanie
Posts: 439
Joined: 12 Sep 2017, 07:13

Re: Anti Aliasing in Post

Post by melanie »

You can use the new FX Line Colorize Enhanced to color your lines:

https://youtu.be/M-6mnirDBbc



The full tutorial will be out very soon!
Melanie • Animator
oldmandarin
Posts: 8
Joined: 12 Jan 2017, 04:41

Re: Anti Aliasing in Post

Post by oldmandarin »

I spent a decent amount of time messing with the Line Colorize Enhanced, and unfortunately can't get it to work. At best, the blue only partially disappears. I have the FX applied to the colored line layer as well.
LCE ex.jpg
User avatar
melanie
Posts: 439
Joined: 12 Sep 2017, 07:13

Re: Anti Aliasing in Post

Post by melanie »

Is your colorization layer under your line layer?
Is your blue line completely opaque? If not, increase the value of the range.
Melanie • Animator
oldmandarin
Posts: 8
Joined: 12 Jan 2017, 04:41

Re: Anti Aliasing in Post

Post by oldmandarin »

Yes, the line layer is above, and the opacity is 100%. I tried boosting the range all the way up to 255, and still no change :/

I can send you the working file if possible!
User avatar
melanie
Posts: 439
Joined: 12 Sep 2017, 07:13

Re: Anti Aliasing in Post

Post by melanie »

Of course, you can send me your file in a Private Message :)
Melanie • Animator
User avatar
Hironori Takagi
Posts: 279
Joined: 14 May 2018, 10:15
Location: Tokyo, Japan
Contact:

Re: Anti Aliasing in Post

Post by Hironori Takagi »

oldmandarin wrote: 23 Jul 2018, 21:08
Hironori Takagi wrote: 17 Jul 2018, 02:11 Hello, oldmandarin
I will explain the flow using not anti-aliased images in Japan below.
The big reason is that it is easy to paint without gaps.

I can not answer a detailed question because I am not good at English.
Thank you.
20180717.jpg
This makes so much sense! I've also just discovered the OLM smoothing plugins for AE, so I'm going to try that with this method as well. Thank you so much (and to everyone else who replied) !

Also, follow up question, how do you colorize your blue and red lines? Or do you just color to the line, and turn the layer off?
Hello, oldmandarin.
Thank you for your feedback.

In Japan's way of making, use "paint bucket" and "filled stroke" for images with anti-aliasing turned off.
I will introduce the video of Youtube although it is explanation of Japanese.
Although FX Line Colorize Enhanced is effective, there are aspects that have not been able to respond to the special needs of Japan yet.
TVPaint 11.7.2(Nov 14 2023), Windows11 Pro, HP Spectre x360 Convertible 14-ea0xxx / TVPaint 11.7.1(Dec 22 2022) Mac OS 11.6, Apple MacBookPro M1 2020
Jet
Posts: 51
Joined: 31 Jul 2015, 10:48
Location: Japan

Re: Anti Aliasing in Post

Post by Jet »

I miss two tools from Retas PaintMan.

Color Drag:
Image

This tool is similar to a color smudge tool, it picks up whatever color you click on and allows you to drag into only unfilled areas. So if there is many colors with unfilled gaps it allows quick color dragging without eyedropping. It's adjustable, you can choose gap size etc.

Fill Unfilled Area Brush:
Image

It's a brush that paints a transparent green mark, when you let go it disappears and fills any small gaps you painted over, it's adjustable, you can pick the size of unfilled area to be filled, size of the brush head, etc and it also allows you to include the red, green, blue color trace lines as if they were unfilled areas, so if you draw highlights using red lines they can be changed easily to a highlight color.

This works similar to shape fill in TVP but a brush design is faster to use, like a marquee shape fill brush I guess.
OS: Win 11 / RAM: 128GB / GPU: RTX 3080 Ti / Display: Cintiq 27
oldmandarin
Posts: 8
Joined: 12 Jan 2017, 04:41

Re: Anti Aliasing in Post

Post by oldmandarin »

Hironori Takagi wrote: 30 Jul 2018, 07:18
oldmandarin wrote: 23 Jul 2018, 21:08
Hironori Takagi wrote: 17 Jul 2018, 02:11 Hello, oldmandarin
I will explain the flow using not anti-aliased images in Japan below.
The big reason is that it is easy to paint without gaps.

I can not answer a detailed question because I am not good at English.
Thank you.
20180717.jpg
This makes so much sense! I've also just discovered the OLM smoothing plugins for AE, so I'm going to try that with this method as well. Thank you so much (and to everyone else who replied) !

Also, follow up question, how do you colorize your blue and red lines? Or do you just color to the line, and turn the layer off?
Hello, oldmandarin.
Thank you for your feedback.

In Japan's way of making, use "paint bucket" and "filled stroke" for images with anti-aliasing turned off.
I will introduce the video of Youtube although it is explanation of Japanese.
Although FX Line Colorize Enhanced is effective, there are aspects that have not been able to respond to the special needs of Japan yet.
Thank you so much, this is supremely helpful!
oldmandarin
Posts: 8
Joined: 12 Jan 2017, 04:41

Re: Anti Aliasing in Post

Post by oldmandarin »

Jet wrote: 30 Jul 2018, 12:19 I miss two tools from Retas PaintMan.

Color Drag:
Image

This tool is similar to a color smudge tool, it picks up whatever color you click on and allows you to drag into only unfilled areas. So if there is many colors with unfilled gaps it allows quick color dragging without eyedropping. It's adjustable, you can choose gap size etc.

Fill Unfilled Area Brush:
Image

It's a brush that paints a transparent green mark, when you let go it disappears and fills any small gaps you painted over, it's adjustable, you can pick the size of unfilled area to be filled, size of the brush head, etc and it also allows you to include the red, green, blue color trace lines as if they were unfilled areas, so if you draw highlights using red lines they can be changed easily to a highlight color.

This works similar to shape fill in TVP but a brush design is faster to use, like a marquee shape fill brush I guess.
Woah, this is really cool. I would second this for TVP!
User avatar
Nexxorcist
Posts: 10
Joined: 11 Aug 2016, 00:03
Location: Los Angeles, CA
Contact:

Re: Anti Aliasing in Post

Post by Nexxorcist »

I just wanted to say this is AMAZING. I came on to look for an OLM Smoother type of thing in TV Paint, cause I was so used to coloring characters with anti-alias off in Photoshop, and boom, Anti-alias in the FX Panel with awesome directions from Hironori Takagi.

This program really is the gift that keeps on giving. Arigato! :D
Post Reply