Explore our collection of hands-on, didactic modules designed for working with multimodal data.
Each coding module introduces a specific topic — from automatic gesture detection and speech analysis to 3D motion tracking — providing a step-by-step guide to the tools, libraries, and techniques used in multimodal research.
For questions about a specific module, please contact the module’s authors directly.
If you’d like to contribute a new module yourself, please reach out to w.pouw@tilburguniversity.edu or other members of the core team.
This module provides a Python desktop app that leverages SAM2 to automatically track any object in videos and detect spatial overlaps between a specified target and scene objects.
By Davide Ahmar, Babajide Owoyele & Wim Pouw
Comprehensive toolkit for audio analysis, speech transcription, and speaker identification using multiple Python libraries.
by Marianne de Heer Kloots
Extract and visualize speech features using Praat's Python interface.
Complete gesture detection and analysis toolkit for research and real-time applications using machine learning.
by Wim Pouw, Bosco Yung, Sharjeel Shaikh, James Trujillo, Gerard de Melo, Babajide Owoyele
Motion detection to aid automatic gesture annotation (based on SPUDNIG).
By James Trujillo (based on Ripperda, Drijvers & Holler)
Track face, hands, and body using MediaPipe, with masking/blur and tracing.
By Wim Pouw & Sho Akamine
Multi‑purpose pipeline for post‑synchronizing video and audio recordings from different devices.
by Hamza Nalbantoğlu & Šárka Kadavá
Scripts for systematically masking facial information at different intensities.
by Wim Pouw & Annika Schiefner
Calculating interpersonal movement synchrony, time lags, and pseudo‑pairs.
by James Trujillo
YOLO pose tracking for multiple persons + interpersonal distance calculation.
By Wim Pouw, Arkadiusz Białek, and James Trujillo
Kinematic feature extraction pipeline with exercises for communicative motion analysis.
By Wim Pouw
Train a model to automatically annotate bodily gestures.
By Wim Pouw
Use decision trees to make sense of high‑dimensional data.
By Alexander Kilpatrick
Compare overlap between ELAN annotations using the multimodal‑annotation‑distance tool.
By Camila Antônio Barros
Create movement‑sound time series animations embedded in a video.
By Wim Pouw
Calculate gaps/overlaps and entropy in conversations with 2+ speakers.
By James Trujillo
Implement gesture networks and spaces using dynamic time warping.
By Wim Pouw
Python pipeline for OpenPose tracking and 3D triangulation with Pose2Sim.
By Šárka Kadavá & Wim Pouw
Example of a dynamic dashboard displaying audio‑visual and static data.
By Wim Pouw
Track head directions using MediaPipe, alongside face/hand/body tracking.
By Wim Pouw
Use batch scripting to run OpenPose on a set of videos.
By James Trujillo & Wim Pouw
Record synchronously from multiple cameras—handy for DIY 3D motion tracking.
By Šárka Kadavá & Wim Pouw
Set up a 3D motion tracking system with multiple 2D cameras.
By Wim Pouw
Wrangle and preprocess multiple streams into a single time‑series dataset.
By Wim Pouw
Create a unified long time‑series dataset from multiple modalities.
By Wim Pouw
Extract a smoothed amplitude envelope from a sound file.
By Wim Pouw
Example analysis of motion tracking data using kinematic features.
By James Trujillo
Set up kinematic and speech‑acoustic features to train classifiers for gesture types.
By Wim Pouw
Measure temporal synchrony of speech and gesture (or other visual signals).
By James Trujillo
EnvisionBox has been shortlisted for the Leo Waaijers open science award 2025!
New module on automatic object tracking and object-to-object overlap detection using SAM2 (Python App)
Satellite event for envisionBOX: Summer school 23-27 June 2025 Amsterdam on "Unboxing Multimodality"
New module on multi-person tracking with YOLO and social proximity calculation (Python)
New module on turn-taking dynamics and entropy (Python)
New module on 3D pose tracking with OpenPose and Pose2sim (Python)!
New module on creating video-embedded animations (Python)
New module on 3D pose tracking from 2D video data (Python)!
New module that goes into dynamic dashboards (Python)!
New module that goes into full body tracking with Mediapipe (Python)!
New module that goes into multimodal data wrangling in Python!
You can get in touch with any contributor of particular teaching modules. Emails are generally provided in the modules themselves. If you want to reach out to the main overseeing contributors then you can contact any of these people: