{"id":1833,"date":"2022-08-16T11:52:08","date_gmt":"2022-08-16T10:52:08","guid":{"rendered":"https:\/\/wp.coventry.domains\/e2edu\/?page_id=1833"},"modified":"2022-08-24T16:37:55","modified_gmt":"2022-08-24T15:37:55","slug":"granular-dance-2","status":"publish","type":"page","link":"https:\/\/wp.coventry.domains\/e2edu\/granular-dance-2\/","title":{"rendered":"Granular  Dance"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Summary<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The following tutorial introduces a system entitled &#8220;Granular Dance&#8221; that can generate synthetic sequences of dance poses of arbitrary length. This system combines an adversarial autoencoder with a method for sequence concatenation. The autoencoder model uses a combination of conventional neural network (ANN) and long short term memory (LSTM) layers to create short pose sequences. The concatenation method blends joint rotations to create smooth transitions between these short pose sequences. The article focuses on the explanation of the blending mechanism and provides some examples how the latent space of pose sequence encodings can be navigated to discover potentially interesting synthetic pose sequences. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This tutorial forms part of a series of tutorials on using PyTorch to create and train generative deep learning models. The code for these tutorials is available <a rel=\"noreferrer noopener\" href=\"https:\/\/github.coventry.ac.uk\/ad5041\/PyTorch_ML_Tutorials\" target=\"_blank\">here<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">After 400 epochs of training, the pose sequences reconstructed by the autoencoder look like this when rendered as skeleton animation.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-vimeo wp-block-embed-vimeo wp-embed-aspect-18-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"model128_ref_versus_pred_sequence_4000\" src=\"https:\/\/player.vimeo.com\/video\/507600887?h=aad047d66b&amp;dnt=1&amp;app_id=122963\" width=\"788\" height=\"394\" frameborder=\"0\" allow=\"autoplay; fullscreen; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Sequence Blending<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This article focuses on the explanation of the sequence blending method. All other aspects such as the creation of the dataset, the model architectures, and the training have been explained in a <a rel=\"noreferrer noopener\" href=\"https:\/\/wp.coventry.domains\/e2edu\/pose-sequence-generation-with-an-adversarial-autoencoder\/\" target=\"_blank\">previous article<\/a> on &#8220;Pose Sequence Generation with an Adversarial Autoencoder&#8221;. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For those who have experimented with the code examples in the <a rel=\"noreferrer noopener\" href=\"https:\/\/wp.coventry.domains\/e2edu\/pose-sequence-generation-with-an-adversarial-autoencoder\/\" target=\"_blank\">previous article<\/a>, it will have become apparent that the concatenated sequences contain discontinuities at the position of the concatenation. To avoid such discontinuities, &#8220;Granular Dance&#8221; employs a blending mechanism between successive sequences. The sequence blending mechanism is inspired by two methods from computer music that combine short sound fragments to generate longer sounds: Granular Synthesis and Concatenative Synthesis. In Granular Dance, a <a rel=\"noreferrer noopener\" href=\"https:\/\/en.wikipedia.org\/wiki\/Window_function\" target=\"_blank\">window function<\/a> (Hanning) is superimposed on a pose sequence. But rather than controlling an amplitude as for audio, this function blends the joint orientations of the overlapping pose sequences by using spherical linear interpolation (<a rel=\"noreferrer noopener\" href=\"https:\/\/en.wikipedia.org\/wiki\/Slerp#:~:text=In%20computer%20graphics%2C%20Slerp%20is,purpose%20of%20animating%203D%20rotation.\" target=\"_blank\">SLERP<\/a>).<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"450\" src=\"https:\/\/wp.coventry.domains\/e2edu\/wp-content\/uploads\/sites\/3486\/2022\/08\/GranularDance_sequence_blending-1024x450.png\" alt=\"\" class=\"wp-image-2322\" srcset=\"https:\/\/wp.coventry.domains\/e2edu\/wp-content\/uploads\/sites\/3486\/2022\/08\/GranularDance_sequence_blending-1024x450.png 1024w, https:\/\/wp.coventry.domains\/e2edu\/wp-content\/uploads\/sites\/3486\/2022\/08\/GranularDance_sequence_blending-300x132.png 300w, https:\/\/wp.coventry.domains\/e2edu\/wp-content\/uploads\/sites\/3486\/2022\/08\/GranularDance_sequence_blending-768x337.png 768w, https:\/\/wp.coventry.domains\/e2edu\/wp-content\/uploads\/sites\/3486\/2022\/08\/GranularDance_sequence_blending-1536x675.png 1536w, https:\/\/wp.coventry.domains\/e2edu\/wp-content\/uploads\/sites\/3486\/2022\/08\/GranularDance_sequence_blending-788x346.png 788w, https:\/\/wp.coventry.domains\/e2edu\/wp-content\/uploads\/sites\/3486\/2022\/08\/GranularDance_sequence_blending.png 1928w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Pose Sequence Blending. This figure schematically depicts the operation of the pose sequence blending mechanism. Prior to blending, the result pose sequence is populated with a base pose (bottom). Short pose sequences are blended one after the other with the result pose sequence (top) using quaternion SLERP. The bell shaped curves represent Hanning windows which control the amount of SLERP.<\/figcaption><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">The principle of sequence blending employed by Granular Dance involves the following steps: <\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>A Hanning window is created with a length that is equal to the size of the pose sequence that the autoencoder works with.  <\/li><li>An empty output pose sequence that eventually holds the concatenated pose sequence is created. <\/li><li>The output pose sequence is populated with an arbitrary pose that provides initial joint angles for interpolation. <\/li><li>Then for every input pose sequence, a quaternion SLERP is conducted between the corresponding excerpt of the output pose sequence and an input pose sequence.  Quaternion SLERP is calculated pose by pose for all joint rotations by stepping along the two pose sequences and taking the SLERP interpolation value from the Hanning window. This results in a pose sequence which is then added at the corresponding position to the output pose sequence. <\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The code for conducting sequence blending appears in the function named &#8220;decode_sequence_encodings&#8221;. This function takes the following arguments: a list of encodings of pose sequences, the size of the overlap between successive pose sequences, a base pose for initially populating the resulting pose sequence, and a file name under which the resulting pose sequence is saved as skeleton animation. This function is defined as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def decode_sequence_encodings(sequence_encodings, seq_overlap, base_pose, file_name):\n    \n    decoder.eval()\n    \n    seq_env = np.hanning(sequence_length)\n    seq_excerpt_count = len(sequence_encodings)\n    gen_seq_length = (seq_excerpt_count - 1) * seq_overlap + sequence_length\n\n    gen_sequence = np.full(shape=(gen_seq_length, joint_count, joint_dim), fill_value=base_pose)\n    \n    for excerpt_index in range(len(sequence_encodings)):\n        latent_vector = sequence_encodings&#091;excerpt_index]\n        latent_vector = np.expand_dims(latent_vector, axis=0)\n        latent_vector = torch.from_numpy(latent_vector).to(device)\n        \n        with torch.no_grad():\n            excerpt_dec = decoder(latent_vector)\n        \n        excerpt_dec = torch.squeeze(excerpt_dec)\n        excerpt_dec = excerpt_dec.detach().cpu().numpy()\n        excerpt_dec = np.reshape(excerpt_dec, (-1, joint_count, joint_dim))\n        \n        gen_frame = excerpt_index * seq_overlap\n        \n        for si in range(sequence_length):\n            for ji in range(joint_count): \n                current_quat = gen_sequence&#091;gen_frame + si, ji, :]\n                target_quat = excerpt_dec&#091;si, ji, :]\n                quat_mix = seq_env&#091;si]\n                mix_quat = slerp(current_quat, target_quat, quat_mix )\n                gen_sequence&#091;gen_frame + si, ji, :] = mix_quat\n        \n    gen_sequence = torch.from_numpy(gen_sequence)\n    gen_sequence = gen_sequence.view((-1, 4))\n    gen_sequence = nn.functional.normalize(gen_sequence, p=2, dim=1)\n    gen_sequence = gen_sequence.view((gen_seq_length, joint_count, joint_dim))\n    gen_sequence = torch.unsqueeze(gen_sequence, dim=0)\n    gen_sequence = gen_sequence.to(device)\n    \n    zero_trajectory = torch.tensor(np.zeros((1, gen_seq_length, 3), dtype=np.float32))\n    zero_trajectory = zero_trajectory.to(device)\n    \n    skel_sequence = skeleton.forward_kinematics(gen_sequence, zero_trajectory)\n    \n    skel_sequence = skel_sequence.detach().cpu().numpy()\n    skel_sequence = np.squeeze(skel_sequence)\n    \n    view_min, view_max = utils.get_equal_mix_max_positions(skel_sequence)\n    skel_images = poseRenderer.create_pose_images(skel_sequence, view_min, view_max, view_ele, view_azi, view_line_width, view_size, view_size)\n\n    skel_images&#091;0].save(file_name, save_all=True, append_images=skel_images&#091;1:], optimize=False, duration=33.0, loop=0) \n    \n    decoder.train()<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Generate and Visualise Pose Sequences<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">As in the <a href=\"https:\/\/wp.coventry.domains\/e2edu\/pose-sequence-generation-with-an-adversarial-autoencoder\/\">previous article<\/a> on &#8220;Pose Sequence Generation with an Adversarial Autoencoder&#8221;, several convenience functions are provided for reconstructing and visualising sequences of poses.  The previously described function &#8220;decode_sequence_encodings&#8221; is one of them. All the other convenience functions are identical with those in the previous article. These other functions are skipped here. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the following, some examples of using the convenience functions to experiment with pose sequences and pose sequence encodings are presented.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Create an Animation for a Single Original Pose<\/strong> <strong>Sequence<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A single original pose sequence can be obtained and saved as animation as follows<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>seq_index = 100\n\ncreate_ref_sequence_anim(seq_index, \"results\/anims\/orig_sequence_{}.gif\".format(seq_index))<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Create an Animation for a Single Reconstructed Pose<\/strong> <strong>Sequence<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A single pose sequence can be reconstructed and saved as animation as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>seq_index = 100\n\ncreate_rec_sequence_anim(seq_index, \"results\/anims\/rec_sequence_{}.gif\".format(seq_index))<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The next examples all employ sequence blending. For this, two settings have to be specified: the size of the overlap between successive pose sequences and the pose to be used to initially populate the output pose sequence. This can be done as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>seq_overlap = 32\nbase_pose = np.reshape(pose_sequence&#091;0], (joint_count, joint_dim))\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Create an Animation from Several Reconstructed Pose<\/strong> <strong>Sequences<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A list of pose sequences can be reconstructed and saved as animation as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>start_seq_index = 100\nend_seq_index = 612\nseq_indices = &#091; frame_index for frame_index in range(start_seq_index, end_seq_index, seq_overlap)]\n\nseq_encodings = encode_sequences(seq_indices)\ndecode_sequence_encodings(seq_encodings, seq_overlap, base_pose, \"results\/anims\/rec_sequences_{}-{}.gif\".format(start_seq_index, end_seq_index))\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Create an Animation from a Random Walk in Latent Space<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In a more interesting example, a single pose sequence is encoded and the encoding is used as starting point for a random walk within latent space. The random walk generates a list of increasingly randomised pose sequence encodings which are then decoded into pose sequences, concatenated and saved as animation. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>start_seq_index = 4000\nseq_frame_count = 32\n\nseq_indices = &#091;start_seq_index]\n\nseq_encodings = encode_sequences(seq_indices)\n\nfor index in range(0, seq_frame_count - 1):\n    random_step = np.random.random((latent_dim)).astype(np.float32) * 2.0\n    seq_encodings.append(seq_encodings&#091;index] + random_step)\n\ndecode_sequence_encodings(seq_encodings, seq_overlap, base_pose, \"results\/anims\/seq_randwalk_{}_{}.gif\".format(start_seq_index, seq_frame_count))\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Create an Animation by Following a Trajectory in Latent Space with an Offset<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this example, a list of pose sequences that follow each other in the original motion capture recording is encoded into a list of latent vectors. These latent vectors represent a trajectory in latent space. This trajectory is followed at an offset by adding a vector to each pose sequence encoding. The resulting encodings are then decoded  into pose sequences, concatenated and saved as animation. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>seq_start_index = 4000\nseq_end_index = 5000\n    \nseq_indices = &#091; seq_index for seq_index in range(seq_start_index, seq_end_index, seq_overlap)]\n\nseq_encodings = encode_sequences(seq_indices)\n\noffset_seq_encodings = &#091;]\n\nfor index in range(len(seq_encodings)):\n    sin_value = np.sin(index \/ (len(seq_encodings) - 1) * np.pi * 4.0)\n    offset = np.ones(shape=(latent_dim), dtype=np.float32) * sin_value * 4.0\n    offset_seq_encoding = seq_encodings&#091;index] + offset\n    offset_seq_encodings.append(offset_seq_encoding)\n    \ndecode_sequence_encodings(offset_seq_encodings, seq_overlap, base_pose, \"results\/anims\/seq_offset_{}-{}.gif\".format(seq_start_index, seq_end_index))\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Create an Animation by Interpolating Between Pose Sequence Encodings <\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Two pose sequences are encoded and new encodings are created by gradually interpolating between the initial pose sequence encodings. Each interpolated encoding is decoded into a pose sequence. All these pose sequences are concatenated and saved as animation. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>seq1_start_index = 1000\nseq1_end_index = 2000\n\nseq2_start_index = 4000\nseq2_end_index = 5000\n\nseq1_indices = &#091; seq_index for seq_index in range(seq1_start_index, seq1_end_index, seq_overlap)]\nseq2_indices = &#091; seq_index for seq_index in range(seq2_start_index, seq2_end_index, seq_overlap)]\n\nseq1_encodings = encode_sequences(seq1_indices)\nseq2_encodings = encode_sequences(seq2_indices)\n\nmix_encodings = &#091;]\n\nfor index in range(len(seq1_encodings)):\n    mix_factor = index \/ (len(seq1_indices) - 1)\n    mix_encoding = seq1_encodings&#091;index] * (1.0 - mix_factor) + seq2_encodings&#091;index] * mix_factor\n    mix_encodings.append(mix_encoding)\n\ndecode_sequence_encodings(mix_encodings, seq_overlap, base_pose, \"results\/anims\/seq_mix_{}-{}_{}-{}.gif\".format(seq1_start_index, seq1_end_index, seq2_start_index, seq2_end_index))<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Summary The following tutorial introduces a system entitled &#8220;Granular Dance&#8221; that can generate synthetic sequences of dance poses of arbitrary length. This system combines an adversarial autoencoder with a method for sequence concatenation. The autoencoder model uses a combination of conventional neural network (ANN) and long short term memory (LSTM) layers to create short pose [&hellip;]<\/p>\n","protected":false},"author":2154,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","footnotes":""},"class_list":["post-1833","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/1833","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/users\/2154"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/comments?post=1833"}],"version-history":[{"count":27,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/1833\/revisions"}],"predecessor-version":[{"id":3447,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/1833\/revisions\/3447"}],"wp:attachment":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/media?parent=1833"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}