Recurrent Neural Network (RNN): How Machines Process Text

Turn to page 6, paragraph 8, line 11. Imagine your computer being able to do this faster than you.

We are at a stage of artificial intelligence where computers can read and write (literally).  Our brain patterns have been replicated in computers, in the form of RNNs.

One of the most distinguished features of RNNs is their ability to self-correct and self-learn, which makes them indispensable for data classification and processing.

Deep neural networks like RNN have replaced machine learning (ML) algorithms, which initially dominated the field, and are now implemented worldwide. It has been monumental in replicating human intelligence and brain mechanisms in computers for language translation, text sequence modeling, text recognition, time series analysis, and text summarization.

RNNs are an advanced version of artificial neural networks capable of processing correct and contextual counterparts of text sequences. They are flexible, adaptive, and efficient deep learning systems that accept several inputs and predict contextual outputs.

Just like RNNs, artificial neural network (ANN) software is used across commercial and noncommercial industries to prototype and develop smart and self-efficient machines. These machines know and can spot systemic errors by finding co-relations within input components. 

Google’s autocomplete, Google Translate, and AI text generators are all examples of RNNs designed to mimic a human brain. These systems are specifically modeled to adjust to user input, assign neurons, update weightage, and generate the most relevant response.

The key quality of an RNN is its memory or activation state, which stores output vectors of previous words in a sentence. This allows RNNs to understand the relationship between the subject and the verb and derive contextual meaning to generate a response.

Let’s learn more about how RNNs are structured and the different types of RNNs that can be used for text generation and translation.

Types of Recurrent Neural Network (RNN) 

Different industries have their preferences when choosing the right recurrent neural network algorithm. Companies can use the following types of RNNs to process text sequences for their business operations.

Let's look at different types of recurrent neural network systems you can use:

• One-to-one: This recurrent neural network maps a single input to a single output and processes the user’s statements sequentially. It also functions on a first come, first served basis, where the input entered first would be processed first in the same order.
  

• One-to-many: This neural network breaks down one individual text sequence into a series of outputs.  The input text sequence can have several output variations. One-to-many RNNs can evaluate different scenarios and process a series of responses.
  

• Many-to-one: This type of RNN produces one instance of output from a cluster of sequences it receives as input. Thus, more than one input sequence is mapped to a singular output. This technique can also be used to develop voice recognition apps and home assistants.
  

• Many-to-many: This type of RNN implementation accepts multiple input entries and maps them into an equal number of response statements. It is effective during sentimental analysis, text generation, text summarization, and audio mapping.

Types of RNN based on model upgrades

Apart from the above types, RNNs can also be categorized based on prediction accuracy and storage capacity. Software developers and engineers mostly deploy these four types of RNN systems for sequential word processing. 

• Vanilla RNNs (or simple RNNs): Vanilla RNNs feature a simple architecture where the output emitted from one node is fed to another node, along with the previously hidden state data. These RNNs are great for experimentation and help data engineers and scientists develop a conceptual understanding of the technology.
  

• Long-short-term memory (LSTM): LSTM networks are an upgraded version of RNNs with a unique forget cell. The entire network distributes the weights and parameters smartly. By analyzing crucial words that can impact future words, LSTM networks can interpret language more concisely. LSTM models were used for voice assistants, text recognition, music composition, audio detection, and anomaly detection.
  

• Gated recurrent units (GRU): Like LSTM networks, GRUs use a gated mechanism to filter out impactful words from non-impactful ones. A GRU's architecture is simpler than that of an LSTM. It is also trained with fewer parameters and can be developed easily. A GRU merges the hidden and input cells to optimize the RNN system. GRUs were deployed in speech recognition apps, text analysis, healthcare and medicine, and other commercial industries.
  

• Bidirectional RNNs: Bidirectional RNNs work in forward and backward directions to build correlations between words. These RNN models are effective in predicting future words in a sentence. They can also be used for speech recognition or conversational AI, where the tone and style of speech are essential to address. Additionally, bidirectional RNNs are more complex and deal with multiple sequences and user commands.
  

• Deep RNNs: Deep RNNs consist of recurrent units stacked on each other. These stacks can process multiple sequential units of data at the same time. Deep RNNs are used in advanced engineering or industrial automation sectors. Language translators and AI chatbots are also powered with deep RNNs to accommodate more user commands and produce accurate responses.
• RNNs with attention mechanism: RNNs with attention mechanism (now called large language models or LLMs) focus on specific blocks of sequence and selectively choose them to gauge the impact of those tokens on future output. The attention mechanism also helps the RNN focus on the dense weightage given to selective words and attend to critical parts of a sentence, RNNs with attention mechanisms have been used in Google algorithms, language generation and summarization, and AI writing assistants.
  

Understanding the RNN working methodology

RNNs consist of three main layers: the input layer, the output layer, and the activation or hidden layer. These layers work together to analyze the input text and compute the true values of output. 

Let’s go through these layers in detail.

The input, hidden, and output layer

RNNs have three major layers across their architecture: input, output, and hidden. These layers are pre-built within the neural network and receive dispersed neurons, weights, and parameters. 

1. Input layer

The input layer is largely the data declaration layer, where the RNN seeks user input. The input could be words, characters, or audio, but it has to be a sequence. Within the input layer, an automatic activation a[0] is triggered. This vector contains as many values as the length of the target sequence entered by the user. If the sentence has four words, the activation would be a [0,0,0,0]. This automatic activation ensures that the right decision nodes are activated as the word values are passed from one layer to another for correct prediction.

2. Hidden layer

The hidden layer is also the computation layer, where the RNN triggers the activation value and maps words to subsequent neurons. The value is computed as a vector output, which is an array of 0 and 1. The vector output, with the activation value, is supplied to another instance of the RNN function. 

At the same time, it analyzes the second word of the input sequence. The hidden layer stores the contextual derivation of words and their relationship with each other within itself, also known as the memory state, so that the RNN does not forget the previous values at any point.

3. Output layer

After the last word and the last time step, the RNN converts all the vector embeddings into a classified vector that exits through the output layer. The output layer parses the earlier word vectors and activations into a newly generated sequence. 

It also gives a loss value for all the words. Loss is the residue that every layer of RNN emits. It is the deviation from the right context of a particular word and is reduced through backpropagation through time (BPTT). The cycle is repeated until the values get normalized, and the system pushes out an accurate output.

Understanding the RNN training curve

RNN architecture is simple. It processes one word at a time and gathers the context of that word from previous hidden states. The hidden state connects the previous word output with the next word input, passing through temporal layers of time.

RNNs assess each word and its impact on the sequence in a tiered manner. The words are converted into vector representations, and new words are supplied at every algorithm stage.

Here is a detailed explanation. In the following image, the input x, at time step t-x is fed to RNN with a zero activation value. The output (vector y) is fed to the next node, and so on until the end. 

Named entity recognition

Named entity recognition is a strategy where the main subject within a sequence is encoded with a numeric digit while other words are encoded as zero. This is also known as hot encoding, where for each x, you have a y vector counterpart, and the subject is addressed differently as a special digit. With named entity recognition, the RNN algorithm can decipher the acting subject and attempt to draw correlations between the main vector and other vectors. 

Example of named entity recognition within an RNN

Consider this statement, “Bob got a toy Yoda,” as a user input fed to the RNN system. In the first stage, the words will be encoded through hot encoding and converted into embeddings with a specific value. For each word, an x variable is assigned.

Say, for “Bob,” your input variable becomes x bob,  which gives you y bob,  as a vector representation of the subject. The output, y bob, is stored in the memory state of RNN as it repeats this process with the second word in the sequence. 

The second word is then supplied to the network, which still remembers the previous vector. Even if new words are added, the neural network already knows about the subject (or named entity) within the sequence. It derives context from the subject and other words through constant loops that process word vectors, passing activations, and storing the meaning of words in its memory. 

With named entity recognition, RNN can also assign random vector representations to words or components, but the subject or main entity and other words are adjusted to make sense. 

RNNs share their weights and parameters with all words and minimize error through backpropagation through time (BPTT).

Sequence-to-sequence modeling

RNNs process sequential word tokens via time travel and hidden state calculation. The algorithm's loop continues until all the input words are processed. The entire mechanism is carried out within the hidden or computational layer. Unlike feedforward neural networks, RNNs travel back and forth to identify newer words, assign neurons, and derive the context in which they are used. 

RNNs are sensitive to the order of the sequence. The network works by carefully analyzing each token and storing it in memory. This is done by assigning equal weightage to each word token and giving it equal importance. 

The neural network fires the activation function right after it processes the first part of the input and stores it in its memory. As the network works with other words, the memory supplies the previous words and activation functions attached to them. 

The newer words and the previous meanings allow the RNN to predict the meaning and translate the word. Apart from translations, sequential modeling also helps with time series, natural language processing (NLP), audio, and sentences.

Vector representation

The key to understanding the complex semantics of words within a sequence depends on how well you understand the anatomy of the human brain. Humans receive electrical signals that travel through the optic fiber to the brain, which receives a central nervous system response transmitted through stimuli. In the same way, RNN attempts to fire the right neuron based on weightage assigned to different vector representations (the numeric values assigned to words).

RNNs take a scientific approach to solving sequence problems. The network assigns a random vector (like 1,0,1,1), which consists of as many numeric digits as the tokens within a sequence. 

Vector representation simply means that for x component, we have a y vector. As the neurons move from one word to another, the previous output’s context is delivered to the new input. RNN understands the previous word's output better if it remains in a numeric vector format.

Activation function 

RNN works as a series of time-unfolding events. Each time the neural network is triggered, it demands an activation function to activate its decision nodes. This function performs the major mathematical operation and transmits the contextualized meaning of previous words of text.

At each time step, the network must ensure that no erratic values have been passed. This is another reason neural networks share equal parameters and weightage with all the words within a sequence. The activation function is a propeller that methodizes the neurons and powers them to calculate the weightage of every word in a sequence. 

Let’s say you declare an activation function at the start of your sequence. If the first word is Bob, the activation will be bootstrapped as [0,0,0,0]. As the RNN moves sequentially, the neurons attend to all the words, fire the decision nodes, and pass values to the activation function. 

The activation function remains the same until the final word of the sequence is processed. The names of the function at each time step might differ. The activation function also helps solve the vanishing gradient problem which occurs when the gradients of a network become too small.

Recurrent connections

RNNs are known to time travel across their algorithmic layers, identify output counterparts, and complete one round of analysis to generate first set of responses. This can also be known as recurrent connections. It sounds very similar to feedforward neural networks. However, the feedforward neural network gets confused when new words are added to the text sequence or the order of the words is rearranged. 

In RNNs, the network remembers the previous state of words as a memory state and doesn’t let it alter the output course. Recurrent connections enable an RNN to revisit the sequence, ensure no errors, minimize loss function through BPTT, and produce accurate results.

LSTM vs. GRU cells

While processing long paragraphs or large corpus of data, RNNs suffer from short-term memory. This problem was addressed and resolved through advanced RNN architectures like long short-term memory (LSTM) and gated recurrent units (GRUs).

Long short term memory (LSTM) is an upgraded RNN primarily used in NLP and natural language understanding (NLU). The neural network has great memory and doesn’t forget the named entities defined at the beginning of the sequence. 

It contains a “forget” state between the input and output states. The network processes the first set of input tokens and then transfers the value to the forget state, which masks it as 0 or 1. The masking asserts what part of the input can pass on to the next time step and what can be discarded. 

The LSTM mechanism enables the network to remember only important semantics and establish long-term connections with previous words and sentences written at the beginning. It can read and analyze named entities, complete blank spaces with accurate words, and predict future tokens successfully. LSTMs are used in voice recognition, home assistants, and language apps. 

A gated recurrent unit (GRU) was designed to address the limitations of RNNs. This mechanism controls the flow of data so that more data can be stored and the system remembers the sequence for a long period. The unit has two gates: forget and reset. The forget gate decides what words should be carried to the next layer and how much candidate activation should be invoked. The reset gate helps forget unnecessary words and resets the value of weights granted to those words.

GRUs' mechanism is simpler than LSTM and proves more precise for long-range sequences and sequential modeling. GRUs are used for different applications, such as sentiment analysis, product reviews, machine translation, and speech recognition tools.

Decoding

The decoder layer of an RNN accepts the output from the encoder layer from all time steps, vector normalizations, and last activation values to generate newer strings. The decoder layer is primarily used for NLP, language translation, time-series data, and transactional recordkeeping.

If you want to convert an English sentence, like “My name is John,” into German, the RNN would activate neurons from the training dataset, assign pre-determined weights to entities, and figure out a person’s name from the sequence to replicate brain signals. 

Once the algorithm identifies the main named entity, it assigns specific values to other neurons. It passes the data to the decoder, which accepts the vector values and searches for the nearest possible values. It also uses cluster grouping or k-nearest neighbor techniques, a prominent machine learning method, to decode the input. The decoder then publishes the most suitable output — Ich hiese John.

Time travel

Although an RNN appears to have several layers and innumerable stages of analysis, it is initialized only once. The backend console follows a time travel approach, and the operation isn’t visible in real time. The command line interface of an RNN algorithm compiles on a word-to-word basis, travels back in time to adjust parameters, and supplies newer words along with the previous context. 

This process is also known as time unfolding. Only a few neurons out of the entire dataset are shortlisted for it. This method of execution also speeds up the runtime execution and generates a fast response.

Loss function 

With each instance of RNN, the output vector also carries a little bit of residue, or loss value, across to the next time step. As they traverse, the loss values are listed as L1, L2, and so on and until LN. After the last word, the last RNN calculates an aggregate loss and how much it deviates from the expected value. The loss is backpropagated through various time steps and leveraged to adjust weights and parameters. This is also known as the cross-entropy loss function and is mainly visible in sentence prediction or sequence modeling tasks.

Mathematically, if p(x) is the probability of receiving an expected value and q(x) is the actual probability distribution,

It is also worth noting that the usage and value of the loss function can vary based on the type and version of RNN architecture used. However, cross-entropy loss is widely used in sequence modeling and sequence prediction. 

Advantages of RNN

RNNs offer a wide range of benefits that make them suitable for several data-processing tasks across businesses.

• Temporal memory: RNNs maintain a hidden state that allows them to store the context of previous words in the sentence and remember their meaning. This temporal memory helps RNN to simulate brain functions and derive the relationship between different words.
  

• Variable input and output lengths: RNN is trained to look at the first input and interpret the user's thought process. Altering or modifying the input would not affect the initial understanding.  It also shares the old weights and parameters as the words change so that the output remains unchanged. RNNs are the ideal choice for tasks where the sequence length can vary.
  

• Parameter sharing and memory efficiency: RNNs do not waste their parameters. Each word shares the same set of parameters allocated to the input. The RNN optimizes memory space and analyzes a sentence's components by sharing similar parameters. The pre-allocation of parameters enables an RNN to allow leniency and flexibility to the user.
  

• Contextual understanding: Recurrent connections help RNNs break down the user's sentiment and contextualize the input sequence. It analyzes the words and the sentence’s tone, style, and structure to meet the user’s needs.
  
• End-to-end learning: RNNs support end-to-end learning, where the entire model, including word extraction and prediction, is learned directly from data. They have the special ability to interpret data from any language and translate it with 100% accuracy. This is also called self-supervised learning.

Even though RNNs have achieved considerable feats in predicting results and mimicking the human brain’s mechanism, they still have some disadvantages.

Disadvantages of RNNs

RNNs process words sequentially, which leaves a lot of room for error to add up as each word is processed. This leads to the model's erratic behavior and the following disadvantages. 

• Vanishing gradient problem: The vanishing gradient problem occurs when the loss value becomes small during the backpropagation stage and is close to zero at one specific time frame. The slope area becomes negligible, and the output value gets skewed and converges, resulting in diminishing gradient areas.
• Exploding gradient problem: As RNNs receive various inputs, confounding variables within the sequence can shoot up the value of the end output. This is known as the exploding gradient problem. It also happens when the weights or parameters of an RNN are incorrect, leading to the prioritization of the wrong parts of a sequence. 

Even with these disadvantages, RNNs are a massive achievement in ML and AI, as they give computers a sixth sense. With RNNs, many smart and intelligent applications have been developed that can respond like humans. 

RNN vs. deep neural networks

RNNs and deep neural networks are artificial neural networks. However, while deep neural networks can be used across automotive, retail, medicine and other industries, RNNs are mostly used in content creation and content analysis within marketing sector. 

RNNs are flexible as they process text sequences unbiased and less complexly. The algorithm shares its weights and parameters with newer words, stores the context in a memory registry, and supplies older words continuously till the algorithm deduces the meaning of the sequence. RNN also works with a temporal domain, where it registers the exact meaning of the sequence and revisits the layer to extract meanings.  They are mostly used in language translation, natural language processing, natural language understanding (NLU), time series analysis, and weather forecasting.

Deep neural networks are a branch of deep learning that enables computers to mimic the human brain. These neural networks are made up of several layers of neurons and are used for automation tasks and self-assist tasks within different industries. Deep neural networks have been successfully used for image recognition, image processing, facial recognition, object detection, and computer vision. While both RNNs and deep neural networks are multi-layered, only RNNs have recurrent connections with text sequences. A deep neural network is designed to extract, pool, and classify features as a final object. 

RNN vs. convolutional neural network (CNN)

RNNs are used for sequential problems, whereas CNNs are more used for computer vision and image processing and localization. 

Recurrent neural networks (RNNs) are well-suited for sequential tasks like text generation, speech recognition, and language translation. These networks address the sequence chronologically and draw connections between different inter-related words. 

In an RNN, the order of a sequence matters. Even if the user modifies the input or adds new tokens, RNN allocates pre-trained weights and parameters to adapt to the situation. RNN is a highly adaptive, flexible, agile, and informed system that strives to replicate human brain functions.

Convolutional neural networks (CNNs) are deep neural networks that detect, evaluate, and classify objects and images. A CNN works with a support vector machine (SVM) to predict the class of image data. This unsupervised learning method extracts key features, image coordinates, background illumination, and other image components. It also builds feature maps and data grids and feeds the data to support a vector machine to generate a class. 

CNNs have been a breakthrough discovery in computer vision and are now being trained to fuel automated devices that don’t require human intervention. 

How RNNs are revolutionizing marketing and advertising: a Google case study

Marketing and advertising industries have adopted RNNs to optimize their creative writing and brainstorming processes. Tech giants like Google, IBM, Accenture, and Amazon have also deployed RNN within their software algorithms to build a better user experience.

One notable RNN case study is Google Neural Machine Translation (GNMT), an update to Google’s search algorithm. GNMT embeds GRU and LSTM architecture to address sequential search queries and provide a more fulfilling experience to internet users. 

It encodes the sequence within the code, parses it into a context vector, and sends the data to the decoder to understand the sentiment and show appropriate search results. GNMT aimed to understand actual search intent and personalize the user’s feed to enhance the search experience. 

The algorithm was heavily utilized in language translation, multilingual audiences, intent verification, and agile search engine optimization to achieve quick responses from the audience.  Given the adaptive nature of RNN, it was easy for Google to decode search queries with varying lengths and complexities and even interpret the query correctly if the user types a wrong keyword. 

As RNN training consists of large corpora of source-target keywords and sentence strings, the algorithm can learn the direction of keywords, display contextualized results, and correctly predict the user’s behavior. The name GNMT suggests the grave similarity between this search algorithm and natural brain stimulation in humans.

As GNMT trains on an increasing number of source data corpora, it improves and delivers translation and response quality for search queries.

Mathematical formula for RNN

The mathematical derivation of RNN is straightforward. Let’s understand more about it through the following example.

Here is how RNN looks at an oncoming sequence. The flow in which RNN reads a sentence is chronological.

• xt: Input vector at time step t ( the word Tom)
• ht: hidden vector at time step t
• yt: output vector at time step t 

Look at the diagram below, where the arrows indicate the flow of information from one vector to another.

• xt is the input to both ht and h(t-1).
• yt is the output of ht and xt.
• ht-1 and xt is used to compute the value of ht. 
• ht and xt are used to compute yt.

The computation at each time step involves:

• Reading the output of h(t-1) and input at ht
• Computing the hidden state (ht) based on input x(t) and previous vector yt
• Calculating yt with h(t-1) and h(t) values

As the algorithm also uses pre-declared weights and parameters, they affect the equation.

• W(hx): Weight of neurons connecting input x(t) and hidden state (ht)
• W(hh): Weight matrice connecting current hidden state (ht) and hidden state at previous time step (h(t-1))
• W(hy): Weight matrice at xt and ht
• bh and by: Bias vectors for input and output vectors, respectively
• f = activation function (usually a linear or a ReLU activation)

To calculate loss, you must backpropagate the neural network at each time step. Here is how:

These formulas also calculate the loss gradient at yt by analyzing the weights at hidden states ht and h(t-1). The loss function helps update the weights and parameters. The weights can be updated by adjusting gradient descents and using variants like Adam or RMSProp.

RNN applications across industries

RNNs are used for various sequence-based tasks across B2B and B2C industries. Here are a few applications:

• Home assistants: Voice assistants like Amazon’s Alexa and Apple’s Siri use bidirectional RNNs to replay voice commands and dictate them to the device to perform specific tasks like playing a song or switching off home lights.
• OTT platforms: OTT streaming provides a theatre-like experience to its users by implementing real-time product feedback via sentimental analysis. The RNN behind OTT platforms like Netflix and Amazon Prime works continuously on prompt data and improves the functioning, recommendation lists, and streaming quality of these platforms.
• Social media platforms: Social media platforms like Facebook and Instagram use next-gen RNNs like large language models to power conversational assistance. A recent revelation, Meta AI, helps with conversation starters, icebreakers, and other prompts to encourage people to get creative and grow their audience.
• Search generative experience: Search generative experience, or SGE, has been launched to optimize the SERP time. By providing content for search queries directly on the results page, this algorithm enables quick purchase decision making.
  
• Language translators: Language translators are based on machine translation and are used to deliver the right translation of a particular statement entered by the user.

Future of RNN

RNNs have already marked an era for future innovations. The advanced upgrade to RNNs, known as LLMs, has marked a significant milestone in the AI industry. These models are powered by generative AI and AI sparsity to create a storytelling experience. Premium LLMs like ChatGPT, Gemini, Claude, and Google LaMDA are accelerating the speed of content creation and distribution across business industries.

LLMs also help IT companies speed up their app development process by building code syntaxes, function threads, and global class definitions. By submitting a well-defined prompt, users can receive automated code and run it directly on their compilers for quick results.

RNNs were a milestone in deep learning and are getting better at replicating human emotions, becoming more self-aware, and making fewer errors. 

Recurrent neural network: Frequently asked questions (FAQs)

What is RNN used for?

 RNN is used for sequence prediction, sequential modeling, voice recognition, sentiment analysis, NLP machine translation, and conversational chatbots. RNN’s intelligent neuron monitoring enables it to deal with variable text sequences and be agile and precise with output.

How many layers are there in an RNN?

An RNN consists of three layers: an input layer, an output layer, and a hidden layer, also known as the computational layer. In addition to these three layers, RNNs are powered by different types of activation functions, such as softmax, linear, tanh, and relu, to represent the sequence in terms of probability distributions.

Why is RNN used for classification?

RNNs are good at gathering enough data about a particular sequence. They can build bridges between different words in a sequence and store the context within their memory so that it isn’t lost. RNNs also retain their memory for a long time, just like humans. This trait is important for text classification and recognition, where the sequence of the words impacts the actual meaning.

What is the loss function in RNN?

The loss function in RNN calculates the average residual value after every round of the probability distribution of input. The residual value is then added at the last round and backpropagated so that the network updates its parameters and stabilizes the algorithm.

Why is RNN used for time series analysis?

As RNN works on the principle of time unfolding, it has a good grasp of previous inputs, enabling it to understand and judge the data better over long periods. This is why an RNN can link two or more data values precisely if it deals with a time series dataset. An RNN is also used with CNN layers to add more pixels to the image background and classify the image with more accuracy.

Dive into the depths of data roots

Neural networks have improved the performance of ML models and infused computers with self-awareness. From healthcare to automobiles to e-commerce to payroll, these systems can handle critical information and make correct decisions on behalf of humans, reducing workload.

Don’t let data stress you out! Learn the intricacies of your existing data and understand the intent behind words with our natural language processing guide.