Indonesian sign language (BISINDO) gesture detection using Yolov11-Pose

Abstract

Sign language serves as the primary means of communication for deaf individuals, yet existing recognition systems for Bahasa Isyarat Indonesia (BISINDO) have not fully exploited keypoint-based pose estimation, particularly for dynamic alphabetic gestures. This study developed a BISINDO gesture detection system using the YOLOv11m-Pose algorithm integrated with MediaPipe hand landmark extraction, Letterbox image preprocessing, and a multi-phase class representation strategy that divided dynamic letters into two movement-phase classes. The main novelty of this study lies in the integration of YOLOv11-Pose with MediaPipe hand keypoint extraction and a multi-phase representation strategy for dynamic letters, which has not been previously applied to BISINDO alphabet detection. A dataset of 1.450 images across 29 classes was collected, augmented to 8.700 samples, and split into training, validation, and test sets. Evaluation on 870 test images yielded an overall accuracy of 99,43%, a macro precision of 99,44%, macro recall of 99,43%, and a mAP50 of 99,11%. All six dynamic letter classes achieved perfect prediction scores, confirming the effectiveness of the multi-phase representation approach. These results demonstrated that the proposed system was capable of reliable BISINDO alphabet detection and provided a solid foundation for further development toward full support for sign language communication.