{"id":1641,"date":"2022-08-14T17:17:23","date_gmt":"2022-08-14T16:17:23","guid":{"rendered":"https:\/\/wp.coventry.domains\/e2edu\/?page_id=1641"},"modified":"2022-08-24T14:51:41","modified_gmt":"2022-08-24T13:51:41","slug":"pose-sequence-generation","status":"publish","type":"page","link":"https:\/\/wp.coventry.domains\/e2edu\/pose-sequence-generation\/","title":{"rendered":"Pose Sequence Generation (RNN)"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Summary<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The following tutorial introduces the use of a simple autoregressive model for generating sequences of dance poses. This model can be trained on motion capture data. The model uses a long short term memory (LSTM) network to predict the continuation of a sequence of poses.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This tutorial forms part of a series of tutorials on using PyTorch to create and train generative deep learning models. The code for these tutorials is available <a rel=\"noreferrer noopener\" href=\"https:\/\/github.coventry.ac.uk\/ad5041\/PyTorch_ML_Tutorials\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">After 1000 epochs of training, the model generates pose sequences that look like this when rendered as skeleton animations.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-vimeo wp-block-embed-vimeo wp-embed-aspect-18-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Original Versus Predicted Dance Pose Sequence\" src=\"https:\/\/player.vimeo.com\/video\/739681174?h=c95e60f7fc&amp;dnt=1&amp;app_id=122963\" width=\"788\" height=\"394\" frameborder=\"0\" allow=\"autoplay; fullscreen; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><figcaption>Original Versus Predicted Dance Pose Sequence. The predicted dance sequence has been created by the autoregressive model described in this article. The model has been trained for 1000 epochs on the following motion capture recording: MUR_Fluidity_Body_Take1_mb_proc_rh.p<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Imports<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The following modules need to be available and imported for this example. The &#8220;common&#8221; module with all its submodules is included when downloading the tutorial files. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import torch\nfrom torch.utils.data import Dataset\nfrom torch.utils.data import DataLoader\nfrom torch import nn\nfrom collections import OrderedDict\n\nimport os, sys, time, subprocess\nimport numpy as np\nimport math\nsys.path.append(\"..\/..\")\n\nfrom common import utils\nfrom common.skeleton import Skeleton\nfrom common.mocap_dataset import MocapDataset\nfrom common.quaternion import qmul, qnormalize_np, slerp\nfrom common.pose_renderer import PoseRenderer<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Compute Device<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">If a GPU is available for running the model, this device can be selected as follows.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>device = 'cuda' if torch.cuda.is_available() else 'cpu'\nprint('Using {} device'.format(device))<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Read Motion Capture Data<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The file from which data is loaded is a &#8220;pickled&#8221; dictionary of motion capture data. More information about this format is available <a rel=\"noreferrer noopener\" href=\"https:\/\/wp.coventry.domains\/e2edu\/motion-capture-data\/\" target=\"_blank\">here<\/a>. The MocapDataset class is used to import such a file.  Information about the MocapDataset  class is available <a rel=\"noreferrer noopener\" href=\"https:\/\/wp.coventry.domains\/e2edu\/utility-classes-and-functions\/\" target=\"_blank\">here<\/a>. In the following code excerpt, a motion capture file is loaded and information about the motion capture data is collected.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mocap_data_path = \"..\/..\/data\/Mocap\/MUR_Nov_2021\/MUR_Fluidity_Body_Take1_mb_proc_rh.p\"\nmocap_fps = 50\n\n# load mocap data\nmocap_data = MocapDataset(mocap_data_path, fps=mocap_fps)\nif device == 'cuda':\n    mocap_data.cuda()\nmocap_data.compute_positions()\n\n# gather skeleton info\nskeleton = mocap_data.skeleton()\nskeleton_joint_count = skeleton.num_joints()\nskel_edge_list = utils.get_skeleton_edge_list(skeleton)\n\n# obtain pose sequence\nsubject = \"S1\"\naction = \"A1\"\npose_sequence = mocap_data&#091;subject]&#091;action]&#091;\"rotations\"]\n\npose_sequence_length = pose_sequence.shape&#091;0]\njoint_count = pose_sequence.shape&#091;1]\njoint_dim = pose_sequence.shape&#091;2]\npose_dim = joint_count * joint_dim\npose_sequence = np.reshape(pose_sequence, (-1, pose_dim))<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Create Dataset<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To create a dataset from motion capture data that can be used for training, several steps are undertaken: remove sequence excerpts in which poses are invalid or otherwise unsuitable for training, split pose sequences into an input pose sequence and an output pose, declare and define a Dataset class to hold the data, split the data into a training and test set, and instantiate DataLoaders from the training and test set.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The removal of sequence excerpts and the splitting into input sequences and output poses is done in parallel. The input sequences are the sequences which are fed into the model and for which the model is supposed to predict the next pose. The output poses are the next poses that the model needs to learn to predict. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sequence_length = 128\nsequence_offset = 2\nmocap_valid_frame_ranges = &#091; &#091; 500, 6500 ] ]\n\n# prepare training data\n# split data into input sequence(s) and output pose(s)\ninput_pose_sequences = &#091;]\noutput_poses = &#091;]\n\nfor valid_frame_range in mocap_valid_frame_ranges:\n    frame_range_start = valid_frame_range&#091;0]\n    frame_range_end = valid_frame_range&#091;1]\n    \n    for seq_excerpt_start in np.arange(frame_range_start, frame_range_end - sequence_length - 1, sequence_offset):\n        #print(\"valid: start \", frame_range_start, \" end \", frame_range_end, \" exc: start \", seq_excerpt_start, \" end \", (seq_excerpt_start + sequence_length) )\n        input_pose_sequences.append( pose_sequence&#091; seq_excerpt_start : seq_excerpt_start + sequence_length ] )\n        output_poses.append( pose_sequence&#091; seq_excerpt_start + sequence_length : seq_excerpt_start + sequence_length + 1 ] )\n\ninput_pose_sequences = np.array(input_pose_sequences)\noutput_poses = np.array(output_poses)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A custom dataset class for the input pose sequences and output poses is created by subclassing the Dataset class.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>class SequencePoseDataset(Dataset):\n    def __init__(self, input_poses_sequences, output_poses):\n        self.input_poses_sequences = input_poses_sequences\n        self.output_poses = output_poses\n    \n    def __len__(self):\n        return self.input_poses_sequences.shape&#091;0]\n    \n    def __getitem__(self, idx):\n        return self.input_poses_sequences&#091;idx, ...], self.output_poses&#091;idx, ...]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This custom dataset class is instantiated as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>full_dataset = SequencePoseDataset(input_pose_sequences, output_poses)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This dataset contains all data. The dataset can be split into two datasets, one for training and one for testing, as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>test_percentage = 0.2\n\ndataset_size = len(full_dataset)\n\ntest_size = int(test_percentage * dataset_size)\ntrain_size = dataset_size - test_size\n\ntrain_dataset, test_dataset = torch.utils.data.random_split(full_dataset, &#091;train_size, test_size])<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">DataLoaders are created from these two datasets as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>batch_size = 16\n\ntrain_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\ntest_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Create Model<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">An autoregressive model is created that takes as input a sequence of poses and outputs a single pose which represents the continuation of the input sequence. The model employs an LSTM network for autoregression. The ReLU function is used as activation function. The output of the last LSTM layer is passed through a conventional artificial neural network. The last layer of this network outputs the predicted poses. The model is implemented by subclassing the nn.Module class. The class definition is as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>class AutoRegressor(nn.Module):\n    def __init__(self, pose_dim, rnn_layer_count, rnn_layer_size, dense_layer_sizes):\n        super(AutoRegressor, self).__init__()\n        \n        self.pose_dim = pose_dim\n        self.rnn_layer_count = rnn_layer_count\n        self.rnn_layer_size = rnn_layer_size\n        self.dense_layer_sizes = dense_layer_sizes\n        \n        # create recurrent layers\n        rnn_layers = &#091;]\n        \n        rnn_layers.append((\"autoreg_rnn_0\", nn.LSTM(self.pose_dim, self.rnn_layer_size, self.rnn_layer_count, batch_first=True)))\n        self.rnn_layers = nn.Sequential(OrderedDict(rnn_layers))\n        \n        # create dense layers\n        dense_layers = &#091;]\n        dense_layer_count = len(self.dense_layer_sizes)\n        \n        if dense_layer_count &gt; 0:\n            dense_layers.append((\"autoreg_dense_0\", nn.Linear(self.rnn_layer_size, self.dense_layer_sizes&#091;0])))\n            dense_layers.append((\"autoregr_dense_relu_0\", nn.ReLU()))\n\n            for layer_index in range(1, dense_layer_count):\n                dense_layers.append((\"autoreg_dense_{}\".format(layer_index), nn.Linear(self.dense_layer_sizes&#091;layer_index-1], self.dense_layer_sizes&#091;layer_index])))\n                dense_layers.append((\"autoregr_dense_relu_{}\".format(layer_index), nn.ReLU()))\n        \n            dense_layers.append((\"autoregr_dense_{}\".format(len(self.dense_layer_sizes)), nn.Linear(self.dense_layer_sizes&#091;-1], self.pose_dim)))\n        else:\n            dense_layers.append((\"autoreg_dense_0\", nn.Linear(self.rnn_layer_size, self.pose_dim)))\n        \n        self.dense_layers = nn.Sequential(OrderedDict(dense_layers))\n    \n    def forward(self, x):\n        #print(\"x 1 \", x.shape)\n        x, (_, _) = self.rnn_layers(x)\n        #print(\"x 2 \", x.shape)\n        x = x&#091;:, -1, :] # only last time step \n        #print(\"x 3 \", x.shape)\n        yhat = self.dense_layers(x)\n        #print(\"yhat \", yhat.shape)\n        return yhat<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The constructor of the model class takes the following arguments: the dimension of a single pose, the number of recurrent layers to create, the number of units in each LSTM layer, and a list of units per layer in the artificial neural network that follows the LSTM network. The model class can be instantiated as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ar_rnn_layer_count = 2\nar_rnn_layer_size = 512\nar_dense_layer_sizes = &#091; ]\n\nautoreg = AutoRegressor(pose_dim, ar_rnn_layer_count, ar_rnn_layer_size, ar_dense_layer_sizes).to(device)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The shapes of the input and output tensors for this model are as follows: <\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>input tensor: batch_size x sequence_length x pose_dim<\/li><li>output tensor: batch_size x pose_dim<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Optimiser and Loss Functions<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To update the model weights during training, the Adam optimiser is used. This optimiser is instantiated as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ar_learning_rate = 1e-4\n\nar_optimizer = torch.optim.Adam(autoreg.parameters(), lr=ar_learning_rate)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The overall loss of the model is obtained from a weighted sum of two individual losses. The two losses quantify the following: the deviation of the generated joint rotations from unit quaternions, and the deviation of the joint rotations of the predicted poses from the target poses. The two loss functions are defined as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def ar_norm_loss(yhat):\n    _yhat = yhat.view(-1, 4)\n    _norm = torch.norm(_yhat, dim=1)\n    _diff = (_norm - 1.0) ** 2\n    _loss = torch.mean(_diff)\n    return _loss\n\ndef ar_quat_loss(y, yhat):\n    # y and yhat shapes: batch_size, seq_length, pose_dim\n    \n    # normalize quaternion\n\n    _y = y.view((-1, 4))\n    _yhat = yhat.view((-1, 4))\n    _yhat_norm = nn.functional.normalize(_yhat, p=2, dim=1)\n    \n    # inverse of quaternion: \n    _yhat_inv = _yhat_norm * torch.tensor(&#091;&#091;1.0, -1.0, -1.0, -1.0]], dtype=torch.float32).to(device)\n    # calculate difference quaternion\n    _diff = qmul(_yhat_inv, _y)\n    # length of complex part\n    _len = torch.norm(_diff&#091;:, 1:], dim=1)\n    # atan2\n    _atan = torch.atan2(_len, _diff&#091;:, 0])\n    # abs\n    _abs = torch.abs(_atan)\n    _loss = torch.mean(_abs)   \n    return _loss<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The  overall loss is calculated by function named &#8220;ar_loss&#8221;. This function is defined as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ar_norm_loss_scale = 0.1\nar_quat_loss_scale = 0.9\n\ndef ar_loss(y, yhat):\n    _norm_loss = ar_norm_loss(yhat)\n    _quat_loss = ar_quat_loss(y, yhat)\n    \n    _total_loss = 0.0\n    _total_loss += _norm_loss * ar_norm_loss_scale\n    _total_loss += _quat_loss * ar_quat_loss_scale\n    \n    return _total_loss, _norm_loss, _quat_loss<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Training and Testing Functions<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">For conducting a single training and testing step, two separate functions are declared: &#8220;ar_train_step&#8221; and &#8220;ar_test_step&#8221;. These functions take as input two tensors, one representing a batch of input pose sequences, the other a batch of output poses. Both functions compute and return loss values. The &#8220;ar_train_step&#8221; function also calculates gradients from the loss functions and updates the trainable parameters of the model.  The &#8220;ar_test_step&#8221; suppressed gradient calculation and leaves the trainable model parameters unchanged.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def ar_train_step(pose_sequences, target_poses):\n\n    pred_poses = autoreg(pose_sequences)\n\n    _ar_loss, _ar_norm_loss, _ar_quat_loss = ar_loss(target_poses, pred_poses) \n\n    #print(\"_ae_pos_loss \", _ae_pos_loss)\n    \n    # Backpropagation\n    ar_optimizer.zero_grad()\n    _ar_loss.backward()\n\n    ar_optimizer.step()\n    \n    return _ar_loss, _ar_norm_loss, _ar_quat_loss\n\ndef ar_test_step(pose_sequences, target_poses):\n    \n    autoreg.eval()\n \n    with torch.no_grad():\n        pred_poses = autoreg(pose_sequences)\n        _ar_loss, _ar_norm_loss, _ar_quat_loss = ar_loss(target_poses, pred_poses) \n    \n    autoreg.train()\n    \n    return _ar_loss, _ar_norm_loss, _ar_quat_loss<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The function named &#8220;train&#8221; performs the actual training by calling the train and test step functions repeatedly. This function takes as arguments the train and test Dataloaders and the number of epochs. It then runs through two loops. The outer loop iterates over all epochs. The inner loop iterates over all batches. The inner loop exists in two versions, one iterates over the batches provided by the train Dataloader, and the other iterates over the batches provided by the test Dataloader. In each of these inner loops, the loss returned by the loss functions are added to a dictionary. This dictionary contains the history of the training process. The definition of the function is as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def train(train_dataloader, test_dataloader, epochs):\n    \n    loss_history = {}\n    loss_history&#091;\"ar train\"] = &#091;]\n    loss_history&#091;\"ar test\"] = &#091;]\n    loss_history&#091;\"ar norm\"] = &#091;]\n    loss_history&#091;\"ar quat\"] = &#091;]\n\n    for epoch in range(epochs):\n        start = time.time()\n        \n        ar_train_loss_per_epoch = &#091;]\n        ar_norm_loss_per_epoch = &#091;]\n        ar_quat_loss_per_epoch = &#091;]\n\n        for train_batch in train_dataloader:\n            input_pose_sequences = train_batch&#091;0].to(device)\n            target_poses = train_batch&#091;1].to(device)\n            \n            _ar_loss, _ar_norm_loss, _ar_quat_loss = ar_train_step(input_pose_sequences, target_poses)\n            \n            _ar_loss = _ar_loss.detach().cpu().numpy()\n            _ar_norm_loss = _ar_norm_loss.detach().cpu().numpy()\n            _ar_quat_loss = _ar_quat_loss.detach().cpu().numpy()\n            \n            ar_train_loss_per_epoch.append(_ar_loss)\n            ar_norm_loss_per_epoch.append(_ar_norm_loss)\n            ar_quat_loss_per_epoch.append(_ar_quat_loss)\n\n        ar_train_loss_per_epoch = np.mean(np.array(ar_train_loss_per_epoch))\n        ar_norm_loss_per_epoch = np.mean(np.array(ar_norm_loss_per_epoch))\n        ar_quat_loss_per_epoch = np.mean(np.array(ar_quat_loss_per_epoch))\n\n        ar_test_loss_per_epoch = &#091;]\n        \n        for test_batch in test_dataloader:\n            input_pose_sequences = train_batch&#091;0].to(device)\n            target_poses = train_batch&#091;1].to(device)\n            \n            _ar_loss, _, _ = ar_train_step(input_pose_sequences, target_poses)\n            \n            _ar_loss = _ar_loss.detach().cpu().numpy()\n            \n            ar_test_loss_per_epoch.append(_ar_loss)\n        \n        ar_test_loss_per_epoch = np.mean(np.array(ar_test_loss_per_epoch))\n        \n        if epoch % model_save_interval == 0 and save_weights == True:\n            autoreg.save_weights(\"results\/weights\/autoreg_weights_epoch_{}\".format(epoch))\n        \n        loss_history&#091;\"ar train\"].append(ar_train_loss_per_epoch)\n        loss_history&#091;\"ar test\"].append(ar_test_loss_per_epoch)\n        loss_history&#091;\"ar norm\"].append(ar_norm_loss_per_epoch)\n        loss_history&#091;\"ar quat\"].append(ar_quat_loss_per_epoch)\n        \n        print ('epoch {} : ar train: {:01.4f} ar test: {:01.4f} norm {:01.4f} quat {:01.4f} time {:01.2f}'.format(epoch + 1, ar_train_loss_per_epoch, ar_test_loss_per_epoch, ar_norm_loss_per_epoch, ar_quat_loss_per_epoch, time.time()-start))\n    \n    return loss_history<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The &#8220;train&#8221; function can be called as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>epochs = 100\nmodel_save_interval = 100\nsave_weights = False\n\n# fit model\nloss_history = train(train_dataloader, test_dataloader, epochs)<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Save Training History and Model Parameters<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The loss history can be exported as CSV file and a graphic plot by calling the corresponding functions of the common.utils module.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># save history\nutils.save_loss_as_csv(loss_history, \"results\/histories\/history_{}.csv\".format(epochs))\nutils.save_loss_as_image(loss_history, \"results\/histories\/history_{}.png\".format(epochs))<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The parameters of the trained model can be saved as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># save model weights\ntorch.save(autoreg.state_dict(), \"results\/weights\/autoreg_weights_epoch_{}\".format(epochs))<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Generate and Visualise Predicted Poses Sequences<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Once the model has been trained, it can be used to generate new pose sequences. These pose sequences are created by starting with an existing pose sequence and then extending this sequence by predicting one pose at a time. Once a predicted  pose sequence has been obtained, it can be used to generate a skeleton animation. The PoseRenderer class can be used for visualisation purposes. More information about the PoseRenderer class is available <a rel=\"noreferrer noopener\" href=\"https:\/\/wp.coventry.domains\/e2edu\/utility-classes-and-functions\/\" data-type=\"URL\" data-id=\"https:\/\/wp.coventry.domains\/e2edu\/utility-classes-and-functions\/\" target=\"_blank\">here<\/a>. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An instance of the PoseRenderer class can be created as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>skel_edge_list = utils.get_skeleton_edge_list(skeleton)\nposeRenderer = PoseRenderer(skel_edge_list)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two convenience functions are declared for rendering a sequence of poses as skeleton animation. The animation is exported in &#8220;<a rel=\"noreferrer noopener\" href=\"https:\/\/en.wikipedia.org\/wiki\/GIF\" target=\"_blank\">.gif<\/a>&#8221; format. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The first function named &#8220;create_ref_sequence_anim&#8221; creates animations from excerpts of the original motion capture data. This function takes as arguments the index of the first frame in a pose sequence, the number of poses following this first frame, and the name of the file the animation is exported as. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def create_ref_sequence_anim(start_pose_index, pose_count, file_name):\n    \n    start_pose_index = max(start_pose_index, sequence_length)\n    pose_count = min(pose_count, pose_sequence_length - start_pose_index)\n    \n    sequence_excerpt = pose_sequence&#091;start_pose_index:start_pose_index + pose_count, :]\n    sequence_excerpt = np.reshape(sequence_excerpt, (pose_count, joint_count, joint_dim))\n\n    sequence_excerpt = torch.tensor(np.expand_dims(sequence_excerpt, axis=0)).to(device)\n    zero_trajectory = torch.tensor(np.zeros((1, pose_count, 3), dtype=np.float32)).to(device)\n    \n    skel_sequence = skeleton.forward_kinematics(sequence_excerpt, zero_trajectory)\n\n    skel_sequence = np.squeeze(skel_sequence.cpu().numpy())\n    view_min, view_max = utils.get_equal_mix_max_positions(skel_sequence)\n    skel_images = poseRenderer.create_pose_images(skel_sequence, view_min, view_max, view_ele, view_azi, view_line_width, view_size, view_size)\n    skel_images&#091;0].save(file_name, save_all=True, append_images=skel_images&#091;1:], optimize=False, duration=33.0, loop=0)\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The second function named &#8220;create_pred_sequence_anim&#8221; creates animations from predicted pose sequences. This function takes the same arguments as the previous function. The start_pose_index argument marks the beginning of the predicted pose sequence. To create a first predicted pose, an excerpt of the original motion capture sequence that immediately precedes this first pose is used as input for the model. Accordingly, the start_pose_index must be higher than the length of the sequence that is input into the model. All successive poses are predicted by extending the input sequence by one predicted pose at the time. The animation is created from predicted poses only. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def create_pred_sequence_anim(start_pose_index, pose_count, file_name):\n    autoreg.eval()\n    \n    start_pose_index = max(start_pose_index, sequence_length)\n    pose_count = min(pose_count, pose_sequence_length - start_pose_index)\n    \n    start_seq = pose_sequence&#091;start_pose_index - sequence_length:start_pose_index, :]\n    start_seq = torch.from_numpy(start_seq).to(device)\n    \n    next_seq = start_seq\n    \n    pred_poses = &#091;]\n    \n    for i in range(pose_count):\n        with torch.no_grad():\n            pred_pose = autoreg(torch.unsqueeze(next_seq, axis=0))\n    \n        # normalize pred pose\n        pred_pose = torch.squeeze(pred_pose)\n        pred_pose = pred_pose.view((-1, 4))\n        pred_pose = nn.functional.normalize(pred_pose, p=2, dim=1)\n        pred_pose = pred_pose.view((1, pose_dim))\n\n        pred_poses.append(pred_pose)\n    \n        #print(\"next_seq s \", next_seq.shape)\n        #print(\"pred_pose s \", pred_pose.shape)\n\n        next_seq = torch.cat(&#091;next_seq&#091;1:,:], pred_pose], axis=0)\n    \n        print(\"predict time step \", i)\n\n    pred_poses = torch.cat(pred_poses, dim=0)\n    pred_poses = pred_poses.view((1, pose_count, joint_count, joint_dim))\n\n\n    zero_trajectory = torch.tensor(np.zeros((1, pose_count, 3), dtype=np.float32))\n    zero_trajectory = zero_trajectory.to(device)\n    \n    skel_poses = skeleton.forward_kinematics(pred_poses, zero_trajectory)\n    \n    skel_poses = skel_poses.detach().cpu().numpy()\n    skel_poses = np.squeeze(skel_poses)\n    \n    view_min, view_max = utils.get_equal_mix_max_positions(skel_poses)\n    pose_images = poseRenderer.create_pose_images(skel_poses, view_min, view_max, view_ele, view_azi, view_line_width, view_size, view_size)\n\n    pose_images&#091;0].save(file_name, save_all=True, append_images=pose_images&#091;1:], optimize=False, duration=33.0, loop=0) \n\n    autoreg.train()<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">These two functions can be called as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>seq_start_pose_index = 1000\nseq_pose_count = 200\n\ncreate_ref_sequence_anim(seq_start_pose_index, seq_pose_count, \"ref_{}_{}.gif\".format(seq_start_pose_index, seq_pose_count))\ncreate_pred_sequence_anim(seq_start_pose_index, seq_pose_count, \"pred_{}_{}.gif\".format(seq_start_pose_index, seq_pose_count))\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Summary The following tutorial introduces the use of a simple autoregressive model for generating sequences of dance poses. This model can be trained on motion capture data. The model uses a long short term memory (LSTM) network to predict the continuation of a sequence of poses. This tutorial forms part of a series of tutorials [&hellip;]<\/p>\n","protected":false},"author":2154,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","footnotes":""},"class_list":["post-1641","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/1641","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/users\/2154"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/comments?post=1641"}],"version-history":[{"count":56,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/1641\/revisions"}],"predecessor-version":[{"id":3196,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/1641\/revisions\/3196"}],"wp:attachment":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/media?parent=1641"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}