Building Open eXchange (BOX) for learners of multimodal social signal processing and analyses
This platform is to promote a community of learners and a culture of sharing knowledge and labor within fields that work with multimodal data to understand embodied minds in interaction and communication. Here you can learn new programming skills and routines that will help you qualify, process, analyze, and classify, multimodal data streams reflecting the diversity of ways that human and non-human animals communicate.
Subscribe to our newsletter to receive content updates and summer/winter school announcements. You can also put in a request for new modules and share your experiences via our 2-minute survey.
As researchers, we often communicate the results of studies and describe methods, but we don't always tailor our communication to teach or guide readers on how to use or reproduce our methods. This platform aims to promote literacy and use of each other's methods, and everyone is welcome to contribute.
Each contributed module introduces code with instructions on how to use it in practice. The modules are designed to introduce learners to new concepts and routines, rather than just sharing code with other experts.
Each module contributor provides the appropriate citation for their module, so they are properly acknowledged when they have been helpful to others in the community.
We envision this platform expanding in scope to host lectures on general theoretical and methodological frameworks, as well as a well-curated update bulletin featuring new papers and tools of interest to the community. If you have ideas and time to help out, please reach out.
This module provides a quick demo of our new (and still experimental) PyPi package envisionHGdetector, that allows you to automatically annotate hand gesture stroke events using a convolutional neural network trained on SAGA, TEDM3D, and Zhubo dataset.
by Wim Pouw
This module provides an introduction to calculating interpersonal movement synchrony, including time-lag assessment and pseudo-pair calculation.
by James Trujillo
This module uses the very reliable YOLO ultralytics pose tracking for multiple persons for top view or other perspectives, and shows an simple calculation of interpersonal distance between two persons.
By Wim Pouw, Arkadiusz Białek, and James Trujillo
This module contains a kinematic feature extraction pipeline with excercises for students of communicative motion analysis.
By Wim Pouw
This module takes you through training a model to automatically annotate bodily gestures.
By Wim Pouw
This module takes you through using decision trees to make sense of high-dimensional data.
By Alexander Kilpatrick
This module takes in annotations in ELAN and allows to compare the overlap between them using the multimodal-annotation-distance tool.
By Camila Antônio Barros
This module takes in a video, and then creates movement-sound time series animations embedded in the video.
By Wim Pouw
This module introduces calculating turn-taking measures, such as gaps and overlaps, as well as entropy, from conversations with 2 or more speakers.
By James Trujillo
This module demonstrates how to implement gesture networks and gesture spaces using dynamic time warping.
By Wim Pouw
This module provides a python pipeline for openpose tracking and 3D triangulation with Pose2Sim.
By Šárka Kadavá & Wim Pouw
This module provides an example of a dynamic dashboard that displays audio-visual and static data.
By Wim Pouw
This module shows how to track the face, hands, and body using MediaPipe, with the option of masking the individual in the video.
By Wim Pouw
This module shows a way to track head directions, next the face, hands, and body tracking using MediaPipe.
By Wim Pouw
This module demonstrates how to use batch scripting to run OpenPose on a set of videos.
By James Trujillo & Wim Pouw
This module demonstrates how to record from multiple cameras synchronously, which is very helpful for creating your own 3D motion tracking pipeline.
By Šárka Kadavá & Wim Pouw
This module shows how to set up a 3D motion tracking system with multiple 2D cameras, using anipose and human pose tracking with DeepLabCut.
By Wim Pouw
This module provides an overview of how to wrangle multiple data streams (motion tracking, acoustics, annotations) and preprocess them (smoothing) to create a single long time series dataset ready for further processing.
By Wim Pouw
In this module an overview is provided how to wrangle multiple data streams (motion tracking, acoustics, annotations) and preprocess them (smoothing) so that you end up with one long timeseries dataset ready for further processing.
By Wim Pouw
This module demonstrates how to extract a smoothed amplitude envelope from a sound file.
By Wim Pouw
This module provides an example of how to analyze motion tracking data using kinematic feature extraction.
By James Trujillo
This module introduces a practice dataset and provides R code for setting up a kinematic and speech acoustic feature dataset that can be used to train a machine classifier for gesture types.
By Wim Pouw
This module introduces the use of Cross-Wavelet analysis as a way to measure temporal synchrony of speech and gesture (or other visual signals).
By James Trujillo
New module on multi-person tracking with YOLO and social proximity calculation (Python)
New module on turn-taking dynamics and entropy (Python)
New module on 3D pose tracking with OpenPose and Pose2sim (Python)!
New module on creating video-embedded animations (Python)
New module on 3D pose tracking from 2D video data (Python)!
New module that goes into dynamic dashboards (Python)!
New module that goes into full body tracking with Mediapipe (Python)!
New module that goes into multimodal data wrangling in Python!
You can get in touch with any contributor of a particular teaching module by e-mailing them at the address provided in the module. If you want to reach out to the main overseeing contributors, please contact one of the following people: