{"id":1823,"date":"2022-08-16T11:48:35","date_gmt":"2022-08-16T10:48:35","guid":{"rendered":"https:\/\/wp.coventry.domains\/e2edu\/?page_id=1823"},"modified":"2022-08-24T15:37:31","modified_gmt":"2022-08-24T14:37:31","slug":"pose-generation-with-a-gan","status":"publish","type":"page","link":"https:\/\/wp.coventry.domains\/e2edu\/pose-generation-with-a-gan\/","title":{"rendered":"Pose Generation with a GAN"},"content":{"rendered":"\n

Summary<\/h2>\n\n\n\n

The following tutorial introduces the use of a simple generative adversarial network for generating synthetic dance poses. This model can be trained on motion capture data. The model uses conventional neural network layers (ANN) to create poses.<\/p>\n\n\n\n

This tutorial forms part of a series of tutorials on using PyTorch to create and train generative deep learning models. The code for these tutorials is available here<\/a>.<\/p>\n\n\n\n

After 400 epochs of training, the model generates pose that look like this when rendered as skeletons.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Imports<\/h2>\n\n\n\n

The following modules need to be available and imported for this example. The \u201ccommon\u201d module with all its submodules is included when downloading the tutorial files.<\/p>\n\n\n\n

import torch\nfrom torch.utils.data import Dataset\nfrom torch.utils.data import DataLoader\nfrom torch import nn\nfrom collections import OrderedDict\n\nimport os, sys, time, subprocess\nimport numpy as np\nsys.path.append(\"..\/..\")\n\nfrom common import utils\nfrom common.skeleton import Skeleton\nfrom common.mocap_dataset import MocapDataset\nfrom common.quaternion import qmul, qnormalize_np, slerp\nfrom common.pose_renderer import PoseRenderer\nimport matplotlib.pyplot as plt<\/code><\/pre>\n\n\n\n
Compute Device<\/h2>\n\n\n\n
If a GPU is available for running the model, this device can be selected as follows.<\/p>\n\n\n\n
device = 'cuda' if torch.cuda.is_available() else 'cpu'\nprint('Using {} device'.format(device))<\/code><\/pre>\n\n\n\n
Read Motion Capture Data<\/h2>\n\n\n\n
The file from which data is loaded is a \u201cpickled\u201d dictionary containing motion capture data. More information about this format is available\u00a0here<\/a>. The MocapDataset class is used to import such a file. Information about the MocapDataset class is available\u00a0here<\/a>. In the following code excerpt, a motion capture file is loaded and a sequence of poses represented by joint rotations (quaternions) is obtained. <\/p>\n\n\n\n
mocap_data_path = \"..\/..\/data\/Mocap\/MUR_Nov_2021\/MUR_PolytopiaMovement_Take2_mb_proc_rh.p\"\nmocap_fps = 50\n\n# load mocap data\nmocap_data = MocapDataset(mocap_data_path, fps=mocap_fps)\nif device == 'cuda':\n    mocap_data.cuda()\nmocap_data.compute_positions()\n\n# gather skeleton info\nskeleton = mocap_data.skeleton()\nskeleton_joint_count = skeleton.num_joints()\nskel_edge_list = utils.get_skeleton_edge_list(skeleton)\n\n# obtain pose sequence\nsubject = \"S1\"\naction = \"A1\"\npose_sequence = mocap_data[subject][action][\"rotations\"]\n\n<\/code><\/pre>\n\n\n\n
The edge list obtained from a skeleton can be passed to the constructor of the PoseRenderer class. An instance of the this class can be used for visualising poses. More information about the PoseRenderer class is available here<\/a>. An instance of the PoseRenderer class can be created as follows:<\/p>\n\n\n\n
skel_edge_list = utils.get_skeleton_edge_list(skeleton)\nposeRenderer = PoseRenderer(skel_edge_list)<\/code><\/pre>\n\n\n\n
Create Dataset<\/h2>\n\n\n\n
To create a dataset from the motion capture data that can be used for training, several steps are undertaken: remove sequence excerpts in which poses are invalid or otherwise unsuitable for training, collect information about the remaining pose sequence, declare and define a Dataset class to hold the data, split the data into a training and test set, and instantiate DataLoaders from the training and test set.<\/p>\n\n\n\n
The removal of unwanted sequence excerpts and the collection of information about the remaining pose sequence is conducted as follows:<\/p>\n\n\n\n
mocap_valid_frame_ranges = [ [ 500, 6500 ] ]\n\nposes = []\nfor valid_frame_range in mocap_valid_frame_ranges:\n    frame_range_start = valid_frame_range[0]\n    frame_range_end = valid_frame_range[1]\n    poses += [pose_sequence[frame_range_start:frame_range_end]]\nposes = np.concatenate(poses, axis=0)\n\npose_count = poses.shape[0]\njoint_count = poses.shape[1]\njoint_dim = poses.shape[2]\npose_dim = joint_count * joint_dim\n\nposes = np.reshape(poses, (-1, pose_dim))<\/code><\/pre>\n\n\n\n
A custom dataset class for poses is created by subclassing the Dataset class.<\/p>\n\n\n\n
class PoseDataset(Dataset):\n    def __init__(self, poses):\n        self.poses = poses\n    \n    def __len__(self):\n        return self.poses.shape[0]\n    \n    def __getitem__(self, idx):\n        return self.poses[idx, ...]<\/code><\/pre>\n\n\n\n
This custom dataset class is instantiated as follows:<\/p>\n\n\n\n
full_dataset = PoseDataset(poses)<\/code><\/pre>\n\n\n\n
This dataset contains all data. The dataset can be split into two datasets, one for training and one for testing, as follows:<\/p>\n\n\n\n
test_percentage  = 0.2\n\ndataset_size = len(full_dataset)\n\ntest_size = int(test_percentage * dataset_size)\ntrain_size = dataset_size - test_size\n\ntrain_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])\n\ntrain_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\ntest_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)<\/code><\/pre>\n\n\n\n
DataLoaders are created from these two datasets as follows:<\/p>\n\n\n\n
batch_size = 16\n\ntrain_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\ntest_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)<\/code><\/pre>\n\n\n\n
Create Models<\/h2>\n\n\n\n
As has been explained in the previous article<\/a> on creating synthetic images with a GAN, both a Generator model and a Critique model need to be implemented. The Generator model takes as input a tensor containing noise and produces as output a tensor the represents synthetic poses. The Critique takes as input a tensor representing poses and produces as output a tensor that classifies the poses as ether real or fake. Both models employ only conventional artificial neural networks (ANN).<\/p>\n\n\n\n
Create Critique Model<\/h3>\n\n\n\n
For converting input poses into output classes, the Critique model passes the input poses in the form of a one dimensional feature vector through several ANN layers. These layers successively reduce the dimension of the feature vector down to 1. Each ANN layer with the exception of the last one is followed by a leaky ReLU activation function. The last ANN layer is followed by a Softmax activation function to obtain normalised class probabilities.<\/p>\n\n\n\n
The class definition of the Critique model is as follows:<\/p>\n\n\n\n
class Critique(nn.Module):\n    def __init__(self, pose_dim, dense_layer_sizes):\n        super().__init__()\n        \n        self.pose_dim = pose_dim\n        self.dense_layer_sizes = dense_layer_sizes\n        \n        # create dense layers\n        dense_layers = []\n        \n        dense_layers.append((\"encoder_dense_0\", nn.Linear(self.pose_dim, self.dense_layer_sizes[0])))\n        dense_layers.append((\"encoder_dense_relu_0\", nn.ReLU()))\n        \n        dense_layer_count = len(dense_layer_sizes)\n        for layer_index in range(1, dense_layer_count):\n            dense_layers.append( (\"encoder_dense_{}\".format(layer_index), nn.Linear(self.dense_layer_sizes[layer_index-1], self.dense_layer_sizes[layer_index] ) ) )\n            dense_layers.append( (\"encoder_dense_relu_{}\".format( layer_index ), nn.ReLU() ) )\n\n        dense_layers.append( ( \"encoder_dense_{}\".format( len( self.dense_layer_sizes ) ), nn.Linear( self.dense_layer_sizes[-1], 1) ) )\n        dense_layers.append( ( \"encoder_dense_sigmoid_{}\".format( len( self.dense_layer_sizes ) ), nn.Sigmoid() ) )\n        \n        self.dense_layers = nn.Sequential(OrderedDict(dense_layers))\n        \n    def forward(self, x):\n        #print(\"x 1 \", x.shape\n        yhat = self.dense_layers(x)\n        #print(\"yhat \", yhat.shape)\n        return yhat<\/code><\/pre>\n\n\n\n
The constructor of the Critique model class takes two arguments: the dimensions of a pose and a sequence of unit counts for the ANN layers (with the unit count of 1 for the last layer missing since this layer is added anyway). The model class can be instantiated as follows:<\/p>\n\n\n\n
crit_dense_layer_sizes = [ 128, 64, 16 ]\n\ncritique = Critique(pose_dim, crit_dense_layer_sizes).to(device)<\/code><\/pre>\n\n\n\n
The shapes of the input and output tensors for this model are as follows:<\/p>\n\n\n\n