Building Blocks

This page is dedicated to explaining the logic behind pychubby.

68 landmarks

pychubby relies on the standard 68 facial landmarks framework. Specifically, a pretrained dlib model is used to achieve this task. See pychubby.data for credits and references. Once the landmarks are detected one can query them via their index. Alternatively, for the ease of defining new actions a dictionary pychubby.detect.LANDMARK_NAMES defines a name for each of the 68 landmarks.

LandmarkFace

pychubby.detect.LandmarkFace is one of the most important classes that pychubby uses. To construct a LandmarkFace one needs to provide

  1. Image of the face
  2. 68 landmark points

Rather than using the lower level constructor the user will mostly create instances through the class method estimate which detect the landmark points automatically.

Once instantiated, one can use actions (pychubby.actions.Action) to generate a new (warped) LandmarkFace.

LandmarkFaces

pychubby.detect.LandmarkFaces is a container holding multiple instances of LandmarkFace. It additionally provides functionality that allows for performing the Multiple action on them.

Action

Action is a specific warping recipe that might depend on some parameters. Once instantiated, one can use their perform method to warp a LandmarkFace. To see already available actions go to Gallery or read how to create your own actions Custom Actions.

ReferenceSpace

In general, faces in images appear in different positions, angles and sizes. Defining actions purely based on coordinates of a given face in a given image is not a great idea. Mainly for two reasons:

  1. Resizing, cropping, rotating, etc of the image will render the action useless
  2. These actions are image specific and cannot be applied to any other image. One would be better off using some graphical interface.

One way to solve the above issues is to first transform all the landmarks into some reference space, define actions in this reference space and then map it back into the original domain. pychubby defines these reference spaces in pychubby.reference module. Each reference space needs to implement the following three methods:

  • estimate
  • inp2ref
  • ref2inp

The default reference space is the DefaultRS and its logic is captured in the below figure.

https://i.imgur.com/HRBKTr4.gif

Five selected landmarks are used to estimate an affine transformation between the reference and input space. This trasformation is endcoded in a 2 x 3 matrix A. Transforming from reference to input space and vice versa is then just a simple matrix multiplication.

\[\textbf{x}_{inp}A = \textbf{x}_{ref}\]
\[\textbf{x}_{ref}A^{-1} = \textbf{x}_{inp}\]

DisplacementField

Displacement field represents a 2D to 2D transformations between two images. To instantiate a pychubby.base.DisplacementField one can either use the standard constructor (delta_x, delta_y arrays). Alternatively, one can use a factory method generate that creates a DisplacemetField based on displacement of landmark points.