Building Blocks¶
This page is dedicated to explaining the logic behind pychubby
.
68 landmarks¶
pychubby
relies on the standard 68 facial landmarks framework. Specifically,
a pretrained dlib
model is used to achieve this task. See pychubby.data
for credits
and references. Once the landmarks are detected
one can query them via their index. Alternatively, for the ease of defining new actions
a dictionary pychubby.detect.LANDMARK_NAMES
defines a name for each of the 68 landmarks.
LandmarkFace¶
pychubby.detect.LandmarkFace
is one of the most important classes that pychubby
uses.
To construct a LandmarkFace
one needs to provide
- Image of the face
- 68 landmark points
Rather than using the lower level constructor the user will mostly create instances through the class
method estimate
which detect the landmark points automatically.
Once instantiated, one can use actions (pychubby.actions.Action
) to generate a new (warped)
LandmarkFace
.
LandmarkFaces¶
pychubby.detect.LandmarkFaces
is a container holding multiple instances of LandmarkFace
. It
additionally provides functionality that allows for performing the Multiple
action on them.
Action¶
Action is a specific warping recipe that might depend on some parameters. Once instantiated,
one can use their perform
method to warp a LandmarkFace
. To see already available
actions go to Gallery or read how to create your own actions Custom Actions.
ReferenceSpace¶
In general, faces in images appear in different positions, angles and sizes. Defining actions purely based on coordinates of a given face in a given image is not a great idea. Mainly for two reasons:
- Resizing, cropping, rotating, etc of the image will render the action useless
- These actions are image specific and cannot be applied to any other image. One would be better off using some graphical interface.
One way to solve the above issues is to first transform all the landmarks into some reference space,
define actions in this reference space and then map it back into the original domain. pychubby
defines these reference spaces in pychubby.reference
module. Each reference space needs to implement
the following three methods:
estimate
inp2ref
ref2inp
The default reference space is the DefaultRS
and its logic is captured in the below figure.
Five selected landmarks are used to estimate an affine transformation between the reference and input space. This trasformation is endcoded in a 2 x 3 matrix A. Transforming from reference to input space and vice versa is then just a simple matrix multiplication.
DisplacementField¶
Displacement field represents a 2D to 2D transformations between two images.
To instantiate a pychubby.base.DisplacementField
one can either use the standard
constructor (delta_x
, delta_y
arrays).
Alternatively, one can use a factory method generate
that creates a DisplacemetField
based on
displacement of landmark points.