Building Open Exchange (BOX) for learners of multimodal social signal processing and analyses
This platform is to promote a community of learners and a culture of sharing knowledge and labor within fields that work with multimodal data to understand embodied minds in interaction and communication. Here you can learn new programming skills and routines that will help you qualify, process, analyze, and classify, multimodal data streams reflecting the diversity of ways that human and non-human animals communicate.
Subscribe to our newsletter to receive content updates and summer/winter school announcements.
As researchers, we often communicate the results of studies and describe methods, but we don't always tailor our communication to teach or guide readers on how to use or reproduce our methods. This platform aims to promote literacy and use of each other's methods, and everyone is welcome to contribute.
Each contributed module introduces code with instructions on how to use it in practice. The modules are designed to introduce learners to new concepts and routines, rather than just sharing code with other experts.
Each module contributor provides the appropriate citation for their module, so they are properly acknowledged when they have been helpful to others in the community.
We envision this platform expanding in scope to host lectures on general theoretical and methodological frameworks, as well as a well-curated update bulletin featuring new papers and tools of interest to the community. If you have ideas and time to help out, please reach out.
This module demonstrates how to implement gesture networks and gesture spaces using dynamic time warping.
This module provides an example of a dynamic dashboard that displays audio-visual and static data.
This module shows how to track the face, hands, and body using MediaPipe, with the option of masking the individual in the video.
This module demonstrates how to use batch scripting to run OpenPose on a set of videos.
This module shows how to set up a 3D motion tracking system with multiple 2D cameras, using anipose and human pose tracking with DeepLabCut.
This module provides an overview of how to wrangle multiple data streams (motion tracking, acoustics, annotations) and preprocess them (smoothing) to create a single long time series dataset ready for further processing.
In this module an overview is provided how to wrangle multiple data streams (motion tracking, acoustics, annotations) and preprocess them (smoothing) so that you end up with one long timeseries dataset ready for further processing.
This module demonstrates how to extract a smoothed amplitude envelope from a sound file.
This module provides an example of how to analyze motion tracking data using kinematic feature extraction.
This module introduces a practice dataset and provides R code for setting up a kinematic and speech acoustic feature dataset that can be used to train a machine classifier for gesture types.
This module introduces calculating turn-taking measures, such as gaps and overlaps, as well as entropy, from conversations with 2 or more speakers.
New module on 3D pose tracking from 2D video data (Python)!
New module that goes into dynamic dashboards (Python)!
New module that goes into full body tracking with Mediapipe (Python)!
New module that goes into multimodal data wrangling in Python!
Satellite event for envisionBOX: Summer school in "Pracitical Approaches to Human Multimodal Behavior"
You can get in touch with any contributor of a particular teaching module by e-mailing them at the address provided in the module. If you want to reach out to the main overseeing contributors, please contact one of the following people: