{"id":440,"date":"2022-08-01T10:51:24","date_gmt":"2022-08-01T09:51:24","guid":{"rendered":"https:\/\/wp.coventry.domains\/e2edu\/?page_id=440"},"modified":"2022-08-23T17:39:32","modified_gmt":"2022-08-23T16:39:32","slug":"sequenceprediction","status":"publish","type":"page","link":"https:\/\/wp.coventry.domains\/e2edu\/sequenceprediction\/","title":{"rendered":"Sequence Prediction"},"content":{"rendered":"\n

In many applications of machine learning, the order in which data that is feed to a model is not relevant. For sequence prediction applications on the other hand, the order of the data matters. These applications require an explicit ordering of the data. <\/p>\n\n\n\n

Types of Sequence Prediction<\/h2>\n\n\n\n

There of four types of sequence prediction: sequence continuation, sequence classification, sequence generation, and sequence to sequence translation.<\/p>\n\n\n\n

Sequence Continuation<\/h3>\n\n\n\n

Sequence continuation deals with predicting the next value in a given input sequence. Example applications are weather forecasting or stock market predictions.<\/p>\n\n\n\n

Sequence Classification<\/h3>\n\n\n\n

Sequence classification deals with predicting a class label for a given input sequence. Example applications are sentiment analysis of text or DNA classification.<\/p>\n\n\n\n

Sequence Generation<\/h3>\n\n\n\n

Sequence Generation deals with generating a new output sequence. There are two versions of this depending of the type of input provided. If the input is also a sequence, then the new output sequence if created by iteratively performing a sequence continuation. In each sequence continuation step, a predicted value is generated. A new input sequence is created from the previous input sequence by copying the previous input sequence but without its first value and adding the newly predicted value at its end. This new input sequence is then used for the next sequence continuation step. Example applications of this type of sequence generation are text and music score generation. Alternatively, the input from which a new sequence is generated is a single value. An example application of this type of sequence generation is image captioning. <\/p>\n\n\n\n

Sequence to Sequence Translation<\/h3>\n\n\n\n

Sequence to sequence translation deals with generating a new output sequence from a given input sequence. A common example application is text translation. <\/p>\n\n\n\n

Sequence Prediction with Recurrent Neural Networks<\/h2>\n\n\n\n

As has been mentioned in the article on artificial neural networks<\/a>, recurrent neural networks (RNN) are particularly useful for sequence prediction tasks since they are able to capture the sequential characteristics of data. RNNs can be employed in one of four basic setups that different from each other with respect to the type of input and output data that they handle.<\/p>\n\n\n\n

One to Many<\/h3>\n\n\n\n

In a One to Many setup, a RNN takes as input a single value and outputs a sequence. Such a setup can be used for a sequence generation task such as image caption.<\/p>\n\n\n

\n
\"\"
One to Many<\/figcaption><\/figure>\n<\/div>\n\n\n

Many to One<\/h3>\n\n\n\n

In a Many to One setup, a RNN takes as input a sequence and outputs a single value. Such a setup can be used for sequence continuation and sequence classification tasks.<\/p>\n\n\n

\n
\"\"
Many to One<\/figcaption><\/figure>\n<\/div>\n\n\n

Many to Many with Identical Length of Input and Output<\/h3>\n\n\n\n

In a Many to Many setup, a RNN takes as input a sequence and outputs a sequence. The input and output sequences are synced and have the same length. Such a setup can be used for a video classification task in which each frame in the video is labelled individually.<\/p>\n\n\n

\n
\"\"
Many to Many with Identical Length of Input and Output<\/figcaption><\/figure>\n<\/div>\n\n\n

Many to Many with Non-Identical Length of Input and Output<\/h3>\n\n\n\n

In another Many to Many setup, a RNN takes as input a sequence and outputs a sequence. The input and output sequences are not synced and can have a different length. Such a setup can be used for sequence to sequence translation tasks. <\/p>\n\n\n

\n
\"\"
Many to Many with Non-Identical Length of Input and Output<\/figcaption><\/figure>\n<\/div>\n\n\n

Sequence Prediction with Convolutional Neural Networks<\/h2>\n\n\n\n

RRNs are experiencing an increasingly steep competition from convolutional neural networks when it comes to sequence prediction tasks. For operating on sequences of data, convolutional neural networks (CNN) employ one dimensional convolution along the length of the sequence. In addition, two important mechanisms have been introduced to CCNs to make them suitable for sequence prediction. These are: dilated kernels and causal convolution. <\/p>\n\n\n\n

Dilated Kernels<\/h3>\n\n\n\n

Contrary to normal kernels, dilated kernels sample values in a matrix with gaps between the locations of the values. This allows the kernel to spread over a larger area of the matrix which, in the case of sequences, corresponds to a longer distance across the sequence. By employing kernels with dilations, a CCN can learn relationships between sequence values that lie far apart from each other without having to resort to unpractically large kernels. <\/p>\n\n\n

\n
\"\"
Dilated One-Dimensional Kernels. The figure on the left depicts the operation of a normal kernel without dilation. The figure on the right depicts an operation with dilated kernels where the size of the dilation increased from layer to layer. \u00a9 Xiang Hao et. al. 2019<\/figcaption><\/figure>\n<\/div>\n\n\n

Causal Convolution<\/h3>\n\n\n\n

Causal convolution is a type of convolution in which only the values in the left half of the kernel are used in convolution whereas the kernel values in the right half are replaced by zero. This is achieved by masking the kernel. The purpose of masking the kernel is to conduct convolution only on values that are at the same position or further back in a sequence than the current value that is predicted. If the dimension along the sequence represents time, then what causal convolution achieves is to make it blind to values that lie in the future. This is essential for many sequence prediction tasks such as sequence continuation. <\/p>\n\n\n

\n
\"\"
Causal Convolution in a CNN. \u00a9Tho Nguyen et. al. 2020<\/figcaption><\/figure>\n<\/div>\n\n\n

The reason that CNNs are starting to surpass RNNs in sequence prediction tasks is that CNNs can operate in parallel on input data (which RNNs can’t), they are good at exploiting local dependencies, and the distance between positions in a sequence is logarithmic. Convolutional Neural Networks can work in parallel because each value in an input sequence can be processed at the same time and does not necessarily depend on the previous values. The distance in position between output value and first value in an input sequence is in the order of log(N) which is much better than that of an RNN, in which the distance is in the order of N.<\/p>\n\n\n\n

Transformer Model<\/h3>\n\n\n\n

The use of RNNs for sequence to sequence translation tasks is obsolete by now. This is partially because they have been superseded in efficiency by CNNs. But also because of the introduction of attention mechanisms. These mechanisms help with preserving the influence of important values even if they are located far back in an input sequence. Attention mechanisms have been introduced in the context of text to text translation. There, models with attention have been shown to significantly surpass models without. <\/p>\n\n\n

\n
\"\"
Architecture of a Transformer Model. \u00a9Gaurav Ghati 2020<\/figcaption><\/figure>\n<\/div>\n\n\n

Transformer models are models that employ multiple forms of attention. The level of sophistication of this approach exceeds the scope of this article. Readers interested in this topic can find more information for instance in an article by Gaurav Ghati<\/a>. <\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

In many applications of machine learning, the order in which data that is feed to a model is not relevant. For sequence prediction applications on the other hand, the order of the data matters. These applications require an explicit ordering of the data. Types of Sequence Prediction There of four types of sequence prediction: sequence […]<\/p>\n","protected":false},"author":2154,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","footnotes":""},"class_list":["post-440","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/440","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/users\/2154"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/comments?post=440"}],"version-history":[{"count":51,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/440\/revisions"}],"predecessor-version":[{"id":3002,"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/pages\/440\/revisions\/3002"}],"wp:attachment":[{"href":"https:\/\/wp.coventry.domains\/e2edu\/wp-json\/wp\/v2\/media?parent=440"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}