Even though After Effects CS5 introduced the awesome time-saving Roto Brush, there would still be shots that would need manual roto work. Even the spline tools in Mocha for AE (ships free with AE CS5) can sometimes fail with some shots. The only thing left to do would be to put on the miners cap, gear up with a pen tool and get to work. So if you do find yourself in such a situation and are new to the art of rotoscoping then the following points from Imagineer Systems could be useful. Thanks to TheBlackbox.org for the tip.
1) There is no such thing as a perfect matte. Rotoscoping is an art form that takes into account the background image, the movement of the object, and the new elements to be composited in the background.
2) Try to start your shape at its most complex point in time, where it will need the most control points.
3) Break a complex shape into multiple simple shapes. If you are rotoscoping a humanoid form and an arm becomes visible, consider rotoscoping the arm as its own element, rather than adding extra points on the body that will serve no purpose when the arm is obscured.
4) Imagine you are the animator who created the shot. What would your dope sheet look like? No matter the medium, whether CG, live action or otherwise, most movements are rarely linear. They normally move in arcs; they normally accelerate in and out of stopped positions. Try and understand the mechanics behind how things are moving in your shot. This will help you to minimize keyframes.
5) Watch and study the shot before you start working. Where are the changes in directions? These will normally have keyframes. Where are the starts and stops? Are there camera moves that can be stabilized to make your work easier?
6) Don’t be afraid to trash your work and start over. Beginning roto artists often make the mistake of trying to fix a flawed approach by adding more and more keyframes. Experienced roto artists learn to quickly identify an inferior approach and are unashamed to trash their work and start over, often many, many times. It is very difficult to get a good matte without a conscious effort to keep the keyframes to a minimum.