How Google Translate's AI Works for Sign Language
Blog
25 January 2021

How Google Translate's AI Works for Sign Language

Jakarta, CNN Indonesia - There are thousands of languages ??in the world. Almost all of the most popular languages ??can be translated directly using the Google Translate software.


For live translation, some software uses voice over the microphone to translate messages without the user typing them.

But what about the millions of people who can't hear or speak?


This is how Artificial Intelligence (AI) on Google Translate works in sign language to help users with speech and hearing disabilities with the use of images.

Convolution Neural Nets (CNN)

Convolution Neural Nets (CNN) are a way that computers work in the form of convolutional neural networks to classify images or visual images.

This system has been trained to describe thousands of photos and classify each image category for translation.

CNN consists of several layers. Layers usually consist of convolutional, combined, and fully-connected layers.

The convolutional layer consists of a kernel (filter) with a convoluted size. On top of the pixels, multiplying and summing the values ??are then ultimately formed in a new, smaller, simplified matrix.

The new, smaller matrix is ??important because it highlights the features in the image and is also easier to train because less is needed to identify the weight of the image.

The next layer is the merge layer. This layer aims to reduce the size of the matrix.

To reduce the size of the matrix, it is necessary to pass the kernel (filter) over the feature matrix and take the highest or average image pixel value.

Next, move on to the third layer, which is fully-connected layers. This is where image classification occurs.

First the matrix will be flattened to a vector and then through the neutral net (neural network). This neutral net is similar to an Artificial Neutral Net (Artificial Neural Network).

CNN classifies the images using the softmax activation function which gives the input probability of a particular class.

How AI Works in Translating Sign Language

- Data preparation

The sign language image data that has been collected are organized into 29 folders with 3,000 images in each folder for each letter of the alphabet.

Three additional folders are space for deleting images. Also, to speed up training, sign language images are downsized to 48x48 from 200x200.

- Convolutional and Merged Layers

There are five actions in the convolutional and compound layers to translate sign language in Google Translate, namely:

1. In the first row, the number of filters and filter size are defined. For the first layer, 64 3x3 filters are used. Image size is specified as 48x48.

After the matrix undergoes convolution, it forms a feature matrix then goes through batch normalization. This will reduce the shift in hidden layer values.

It also makes the matrix easy to train as it stabilizes the weight and improves image accuracy.

After that, run the Re-LU function to bring some non-linearity to the layer allowing CNN to understand the complex images that have been inserted.

2. The next line is where the merge occurs. A 2x2 compound filter size is specified and uses maximum coalescing to reduce the size of the matrix.

3. Finally, the matrix nodes are removed via the neural net. The benefit of deleting nodes is that the network will reduce its sensitivity to the weight of each node.

It also allows the network to become more general in predictions and increase accuracy.

In the model used, 25 percent of the nodes are removed and will be replaced by the next new layer.

- Fully-connected Layers

After convolution, the next stage is the fully-connected layers. But before that, the data was flattened into a single column vector.

After that, the data can pass through the neural network to a denser layer. This dense layer carries on the input from the previous step and outputs all the matrices into the neurons. Neurons are connected in passing data from one layer to the next, in this case there are 256 neurons.


Then the data goes through batch normalization.
Then the ULT function is used for activation.
Finally, 25 percent of the nodes are deleted using the dropout.
There are two layers that are fully connected, so this code is looped with 512 nodes.


After passing through the second fully connected layer, its data output is fed via the softmax function which is used to give the probability that the image belongs to one of the 29 sign language image classes.


Finally, according to Towards Data Science, the final few lines will set the pace of learning and evaluate the accuracy of the model.
Google Middle Develops Sign Language Translator Through Video Conferencing

Reporting from AI Google, the AI ??uses a camera to capture the movement of 21 points on the palm of the hand to interpret text and speech.

However, this technology is not perfect enough because it only translates part of the conversation. Other aspects such as facial expressions, speed, and dialect of speech are still neglected.

"We plan to expand this technology with more stable and robust tracking, expand the number of sign languages ??we can detect properly, and support dynamic motion over time."

"We believe publishing this technology can provide an impetus for new creative ideas and applications by members of the research and developer community at large," wrote Google.

This technological development from Google cannot be separated from the important role of the existence of smart phones or smartphones. The existence of this smartphone will make it easier for this technology to be carried anywhere.

Google is currently working on making Ai work more complex and stable. Similar technology is also made by Microsoft. And a number of other private companies also continue to improve this communication technology.

Facebook Boss Shares How To Make Money on Instagram

Facebook Boss Shares How To Make Money on Instagram

Jakarta, CNN Indonesia - Facebook CEO Mark Zuckerberg revealed a new feature of Instagram to help creators or influencers make money through uploaded content. Instagram will help match brand owners with content creators that fit the audience or market they are trying to reach.

Read More
Getting to know Blockchain, the forerunner of crypto money like Bitcoin

Getting to know Blockchain, the forerunner of crypto money like Bitcoin

Jakarta, CNN Indonesia - Some people must have heard the term blockchain. But some may not know what blockchain is and why it is now so popular and underlies the rise of bitcoin and cryptocurrencies.

Read More
Starting April 25, the Government will temporarily close the entry gate for people traveling from India

Starting April 25, the Government will temporarily close the entry gate for people traveling from India

JAKARTA, KOMPAS.com - The Indonesian government, through the Committee for Handling Covid-19 and National Economic Recovery (KPC-PEN), officially closed the door for every traveler who came from India following a spike in Covid-19 cases in the country.

Read More
Discord Rejects Microsoft Acquisition Offer

Discord Rejects Microsoft Acquisition Offer

Jakarta, CNN Indonesia - An audio chat startup, Discord Inc, has reportedly stopped talking about being bought by a number of companies, one of which is Microsoft Corp. Discord prefers to remain independent and intends to IPO.

Read More
Gojek-Telkom Collaborate to Develop Startups in Eastern Indonesia

Gojek-Telkom Collaborate to Develop Startups in Eastern Indonesia

Jakarta, CNN Indonesia - Having succeeded in becoming the leading super-app in Southeast Asia, Gojek has again taken an initiative to empower young people, especially in eastern Indonesia by utilizing technology.

Read More
500 Million LinkedIn Data Leaks, Sold Rp.29 Thousand Per Data

500 Million LinkedIn Data Leaks, Sold Rp.29 Thousand Per Data

Jakarta, CNN Indonesia - Hundreds of millions of LinkedIn user data was reported to have leaked in cyberspace. There are 500 million data being auctioned in a hacker forum which contains detailed information from user data.

Read More
See All Blog
Icon