App for translating Auslan to text
- Look more into dynamic gesture recognition
- LS-HAN (and Hierarchical Attention Networks in general)
- CNNs with recurrence
- FlowNets for optical flow?
- Implementations must be lightweight enough to run well on mobile devices
- Determined by amount of memory and FLOPs required to run inference model
- Look into different serialization formats (possibly looking at protobufs for now)
- Create and/or look for a larger and much more diverse training set for network
- Implement data augmentation?
- Make prettier GUI 😅
Just some existing methods for gesture recognition - some are possible candidates for implementation or for inspiration for developing different approach.
- Video-based Sign Language Recognition without Temporal Segmentation
- Spatial-Temporal Graph Convolutional Networks for Sign Language Recognition
- DeepASL
- One-shot Learning Hand Gesture Recognition - uses 3D Conv, higher resource complexity, but discrminatory evolution might be beneficial
Looking into voice-text (will be used to translate speech/text into auslan)
- LIGHTWEIGHT AND EFFICIENT END-TO-END SPEECH RECOGNITION USING LOW-RANK TRANSFORMER
- Attention Is All You Need
- SPEECH RECOGNITION WITH DEEP RECURRENT NEURAL NETWORKS
- Deep Speech: Scaling up end-to-end speech recognition
- TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices
About 0.09% loss after 20 epochs (simple CNN with Batch Norm. and Max Pooling, static fingerspelling ASL).
- Currently only works with static input, no recurrence or temporal window (for dynamics)
