GestureSync Pro is a real-time hand gesture recognition system designed to bridge the communication gap

between sign language users and the general public. The system utilizes computer vision and deep learning

techniques to recognize American Sign Language (ASL) gestures and convert them into meaningful text and

speech output.

A webcam is used to capture live video input, and MediaPipe is employed to extract hand landmarks for efficient

feature representation. A Convolutional Neural Network (CNN) model is trained on a large dataset of hand

gestures to accurately classify ASL alphabets. The system further integrates heuristic logic and a hold-to-confirm

mechanism to improve prediction stability and reduce false detections.

Gesture Recognition, Human-Computer Interaction (HCI), Machine Learning, MediaPipe, Real-Time System,

Sign Language Recognition, TensorFlow.js

Communication is one of the most essential aspects of human interaction, enabling individuals to share ideas,

emotions, and information effectively. However, for people who are deaf or mute, communication often relies

on sign language, which is a visual language based on hand gestures, facial expressions, and body movements.

While sign language serves as an effective medium among its users, it is not widely understood by the general

population. This lack of understanding creates a significant communication barrier, making it difficult for deaf

and mute individuals to interact in everyday situations such as education, healthcare, workplaces, and public

services. As a result, there is a growing need for technological solutions that can bridge this communication

sign language recognition has emerged as a particularly important application due to its potential to improve

accessibility for individuals with hearing and speech impairments.

Several researchers have worked on gesture recognition using ML and deep learning techniques.

Alotaibi et al. proposed feature fusion techniques for improved accuracy.

Gupta & Singh used CNN and SVM models for gesture classification.

Jena et al. developed a gesture-to-text system using ML.

Kumar et al. introduced multimodal deep learning for gesture recognition.

Most existing systems either focus only on recognition or require specialized hardware. The proposed system

The system follows these steps:

1. Capture video using webcam

2. Extract hand landmarks using MediaPipe

3. Classify gestures using CNN

4. Apply hold-to-confirm mechanism

5. Generate sentences using AI

6. Convert text to speech

7. Architecture

The system consists of four layers:

1. Frontend: React + Vite

2. ML Model: CNN (TensorFlow.js)

3. Hand Tracking: MediaPipe

4. Speech Output: Web Speech API

The CNN model classifies ASL alphabets, while the heuristic engine detects common gestures. A hold-to-

confirm mechanism ensures accuracy by validating gestures over time.

The block diagram below presents the complete signal and data flow of GestureSync Pro

from raw webcam input through to spoken speech output, organized across its four

processing layers.

Fig 3.2 System Block Diagram

stream serves as the primary input to the system and is continuously fed into the next processing stage in real

time.

The captured video frames are passed to the MediaPipe Hands module, which detects and extracts 21 three-

dimensional hand landmarks per frame at 60 frames per second. These landmarks correspond to the key

anatomical points of the hand including the wrist, knuckles, and fingertips, and together they form a precise

spatial representation of the hand pose at any given moment.

The raw landmark coordinates are subjected to Linear Interpolation smoothing with a factor of alpha equal to

0.6. This interpolation step reduces frame-to-frame jitter and ensures that minor unintentional hand movements

do not produce erratic or unstable recognition results, thereby improving the overall reliability of the downstream

recognition engines.

After smoothing, the landmark data is simultaneously processed by two independent recognition engines

scenarios. This performance reflects a balanced evaluation under both controlled and real-world conditions,

including variations in lighting, background, hand size, and user behavior. Higher accuracy was observed in

controlled environments with clear hand visibility and stable gestures, while performance slightly decreased

during stress tests involving rapid motion, occlusions, and inconsistent lighting. The integration of heuristic-

based gesture detection and CNN-based classification, combined with real-time processing using TensorFlow.js

and MediaPipe Hands, contributed to reliable recognition in most practical use cases. Although not perfect, the

75% accuracy demonstrates the system’s effectiveness as a real-time, browser-based ASL recognition solution,

with the scope for further improvement through enhanced training data, model optimization, and environmental

robustness.

Table 4.2 – CNN Architecture

The table and chart above illustrate the layered architecture of the Convolutional Neural Network (CNN)

designed for ASL alphabet recognition. The network accepts a 64×64 RGB image as input and progressively

extracts features through three convolutional blocks, each consisting of a Conv2D layer, Batch

Normalization, and MaxPooling. The filters double with each block — from 32 → 64 → 128 — allowing the

model to learn increasingly complex visual patterns while the spatial dimensions reduce from 62×62 down to

6×6. After the final pooling layer, the feature maps are flattened into a 4,608-dimensional vector and passed

into a fully connected Dense layer with 512 ReLU units. A 50% Dropout layer follows to prevent overfitting

during training. Finally, a Softmax output layer maps the learned features to 29 classes, covering the 26 ASL

alphabet letters plus additional gesture categories. The bar chart visually reinforces how filter depth grows across