Building Open Exchange (BOX) for learners of multimodal social signal processing and analyses
This platform is to promote a community of learners and a culture of sharing knowledge and labor within fields that work with multimodal data to understand embodied minds in interaction and communication. Here you can learn new programming skills and routines that will help you qualify, process, analyze, and classify, multimodal data streams reflecting the diversity of ways that human and non-human animals communicate.
Subscribe to our newsletter to receive content updates and summer/winter school announcements.
As researchers we are often communicating results of studies and describing our methods. We do not always tune our communication so as to teach or guide readers to use or reproduce the methods we use. This platform is to promote literacy and use of each others methods. Everyone can contribute.
Each contributed module is designed to introduce code with instructions how to use code in practice. The modules are designed to introduce learners to new concepts and routines as opposed to sharing code with other experts.
Each contributor to a module also provides the appropriate citation for their module. In this way the contributor is properly acknowledged when they have been helpful to others in the community.
We can imagine that this platform can expand in scope. It may host lectures focused on introducing general theoretical and methodological frameworks. It may involve a well-curated update bulletin figuring new papers and tools that might be of interest to the community. If you have ideas and time to help out, please reach out.
This module shows an implementation of gesture networks and gesture spaces, utilizing dynamic time warping.
This module provides an example of a dynamic dashboard with audio-visual and static data.
In this module an overview is provided how to track facial, hands, and body using Mediapipe, with the optionally of masking the individual in the video.
This module demonstrates running openpose on a set of videos using batch scripting.
This module demonstrates how to set up a 3D motion tracking system with multiple 2D cameras, running anipose and human pose tracking with deeplabcut.
In this module an overview is provided how to wrangle multiple data streams (motion tracking, acoustics, annotations) and preprocess them (smoothing) so that you end up with one long timeseries dataset ready for further processing.
In this module an overview is provided how to wrangle multiple data streams (motion tracking, acoustics, annotations) and preprocess them (smoothing) so that you end up with one long timeseries dataset ready for further processing.
This module provides an example of extracting a smoothed amplitude envelope from a sound file.
This module provides an example of how to analyze motion tracking data using kinematic feature extraction.
This module introduces a practice dataset and provides code in R for setting up a kinematic and speech acoustic feature dataset that can be used to train a machine classifier for gesture types
New module on 3D pose tracking from 2D video data (Python)!
New module that goes into dynamic dashboards (Python)!
New module that goes into full body tracking with Mediapipe (Python)!
New module that goes into multimodal data wrangling in Python!
Satellite event for envisionBOX: Summer school in "Pracitical Approaches to Human Multimodal Behavior"
You can get in touch with any contributor of particular teaching modules. Emails are generally provided in the modules themselves. If you want to reach out to the main overseeing contributors than you can reach out to any of these people: