ofxDabDataProc is an openFrameworks addon for processing and analysing sequences of numerical data with a specific focus on motion capture. The addon is available here.
Processing Pipeline
The pipeline receives motion capture data via OSC and sends the analysis results also via OSC. Accordingly, the pipeline acts as an in between link between software that produces motion data and sends this data via OSC and software that receives the processed data via OSC and uses it for further processing or to control for example real-time media.
The pipeline stores all data processing units and automatically handles the passing data from one processing unit to the next. The pipeline can be populated with instances of data processing classes. These instances need to be connected to each other for data to flow from one output of a processing unit into one or several inputs of other processing units. By creating these connections, a tree of processing units is created. Based on the topology of the tree, the pipeline automatically determines the proper data flow from the root node of the processing tree to its leaves.
The two core elements of the pipeline are the Data and DataProc classes. Instances of the Data class represent data as a named sequence of one dimensional features which are stored as floating point numbers. The DataProc class represents a basic data processing unit. These instances have a name, can be connected to each other, and possess a process function that handles data processing. The DataProc class does nor process any data and only serves as base class from which useful data processing classes can be derived from.
Data Processing Classes
The addon provides a number of data processing classes that perform a variety of operations.
- DataProcInput: A processing unit that is associated with an already created Data instance and that makes this instance available as input to other data processing classes. The DataProcInput is located at the beginning of a processing pipeline and the data associated with it is updated based on incoming network communication.
- DataProcCombine: Merges multiple Data instances into a single Data instance by concatenating the instances together.
- DataProcFilter: Removes features from a Data instance by dropping all features whose indices are not specified when instantiating the filter.
- DataProcScalar: Converts a sequence of features into a single feature by performing either a min, max, or average operation.
- DataProcScale: Scales data by multiplying all its features with a scalar value.
- DataProcPosGlobalToLocal: Converts position data from a representation in world coordinates into a local representation relative to a root position. The root position is assumed to be the first position value in the feature sequence.
- DataProcRotGlobalToLocal: Converts rotation data (quaternions) from a representation in world coordinates into a local representation relative to a root position. The root rotation is assumed to be the first rotation value in the feature sequence.
- DataProcQuatEuler: Converts rotation data from quaternion format into Euler format.
- DataProcDerivative: Calculates the temporal derivate of data.
- DataProcDerivativeQuat: A special version for calculating the temporal derivate of rotation data in quaternion format.
- DataProcLowPass: Calculates a moving average by using a weighted sum of the old data and new data.
- DataProcLowPassQuat: A special version for calculating a moving average for rotation data in quaternion format.
- DataProcTimeEffort: Calculates the Laban Time Effort Factor (urgency of movement). See Article on Movement Analysis.
- DataProcFlowEffort: Calculates the Laban Flow Effort Factor (continuity of movement). See Article on Movement Analysis.
- DataProcWeightEffort: Calculates the Laban Weight Effort Factor (strength of movement). See Article on Movement Analysis.
- DataProcSpaceEffort: Calculates the Laban Space Effort Factor (directness of movement). See Article on Movement Analysis.
- DataProcSpaceEffort2: Also calculates the Laban Space Effort Factor but uses a different method than the one described in the Article on Movement Analysis. Instead of scaling the cumulative joint position differences by an overall position difference (which becomes zero when there is no motion or the motion returns to the initial position), this version calculates the deviation of the directions at each step from the overall direction of a motion.
Communication
Communication is based on the Open Sound Control (OSC) protocol. Input data and processed data are received and sent as OSC messages, respectively. These messages consist of an address that is meant to specify the type of data contained within. For example the positions of skeleton joints obtained through motion capture could be represented by the message address “/mocap/joint/pos”. The arguments of the message represent the actual values of the data as a sequence of floats.
The DataMessenger class manages all communication. It contains instances of the DataReceiver class for receiving input data and instances of the DataSender class for sending processed data. An instance of a DataReceiver class is associated with a Data instance that stores the incoming data. An instance of a DataSender class is also associated with a data instance whose content it reads for sending. This data instance is typically part of a data processing unit.
Visualisation
The visualisation creates live-updated time plots of the data that is being processed. The DataPlot class provides the core functionality for creating time plots. Each instance of this class is associated with a particular data instance and is updated automatically when the data changes.
Programming Tutorial
Classes
The addon provides a several classes for a range of different purposes. To provide a better overview, the classes are grouped into the following categories: Processing Pipeline, Communication, Visualisation
The classes for creating a processing pipeline are:
- dab::Data
- dab::DataProcPipeline
- dab::DataProc
- dab::DataProcInput
- dab::DataProcCombine
- dab::DataProcFilter
- dab::DataProcScalar
- dab::DataProcScale
- dab::DataProcPosGlobalToLocal
- dab::DataProcRotGlobalToLocal
- dab::DataProcQuatEuler
- dab::DataProcDerivative
- dab::DataProcDerivativeQuat
- dab::DataProcLowPass
- dab::DataProcLowPassQuat
- dab::DataProcTimeEffort
- dab::DataProcFlowEffort
- dab::DataProcWeightEffort
- dab::DataProcSpaceEffort
- dab::DataProcSpaceEffort2
The classes for receiving and sending data are:
- dab::DataMessenger
- dab::DataReceiver
- dab::DataSender
The classes for visualising data are:
- dab::DataPlot
dab::Data
Numerical data is represented and stored by instances of the Data class. The class is instantiated by providing a name, the feature dimension, and the number of features in the sequence. Data instances for position and rotation (quaternion) data of a motion capture skeleton with 63 joints can be created as follows:
#include "dab_data.h"
std::shared_ptr<dab::Data> mJointPosData;
std::shared_ptr<dab::Data> mJointRotData;
int jointCount = 63
mJointPosData = std::shared_ptr<dab::Data>(new dab::Data("jointPositions", 3, jointCount));
mJointRotData = std::shared_ptr<dab::Data>(new dab::Data("jointRotations", 4, jointCount));dab::DataProcPipeline
A new instance of a DataProcPipeline can be created by calling its constructor without any arguments. This instance initially contains no data processing units. These have to be explicitly added to the pipeline later on.
#include "dab_data_proc.h"
std::shared_ptr<dab::DataProcPipeline> mDataProcPipeline;
mDataProcPipeline = std::shared_ptr<dab::DataProcPipeline>(new dab::DataProcPipeline());dab::DataMessenger
In order to receive input data and send processed output data as OSC messages, the DataMessenger class has to be instantiated. After that, the class member functions for creating an OSC receiver and sender can be called. These functions expect a name and port number for the receiver and a name, ip address, and port number for the sender.
#include "dab_data_messenger.h"
std::shared_ptr<dab::DataMessenger> mDataMessenger;
mDataMessenger = std::make_shared<dab::DataMessenger>();
unsigned int mOscReceivePort = 23456;
std::string mOscSendAddress = "127.0.0.1";
unsigned int mOscSendPort = 9004;
mDataMessenger->createReceiver("MocapReceiver", mOscReceivePort);
mDataMessenger->createSender("MocapProcSender", mOscSendAddress, mOscSendPort);Once an OSC receiver has been created, it can be associated with instances of the Data class in which the incoming OSC data are stored. The DataMessenger class provides the function “createDataReceiver” for this which takes an instance of the Data class, the address of the OSC message containing the incoming data, and the name of the receiver that was created just before.
mDataMessenger->createDataReceiver(mJointPosData, "/mocap/joint/pos", "MocapReceiver");
mDataMessenger->createDataReceiver(mJointRotData, "/mocap/joint/rot", "MocapReceiver");dab::DataProcInput
As has been mentioned before, instances of the DataProcInput class differ significantly from all the other data processing units. This difference is also reflected in the instantiation in the class. This class is created by passing as arguments to the constructor the name of the processing unit and a list of pointers to externally created data instances.
#include "dab_data_proc_input.h"
std::shared_ptr<dab::DataProcInput> mJointPosInputProc;
std::shared_ptr<dab::DataProcInput> mJointRotInputProc;
mJointPosInputProc = std::shared_ptr<dab::DataProcInput>(new dab::DataProcInput("joint pos input", { mJointPosData }));
mJointRotInputProc = std::shared_ptr<dab::DataProcInput>(new dab::DataProcInput("joint rot input", { mJointRotData }));dab::DataProc
All subclasses of the DataProc class (with the exception of DataProcInput) are instantiated in a similar manner. Almost all their constructors take three arguments: a name of the processing unit, a name for the internally created data instance that holds the processes data, and a class specific additional argument.
The constructors of the DataProcPosGlobalToLocal and DataProcRotGlobalToLocal classes take as third argument a vector of a vector of indices that specify the hierarchy of skeleton joints. The outer vector contains the indices of the parent joints and the inner vector contains the indices of each parent’s child joints.
#include "dab_data_proc_pos_global_to_local.h"
#include "dab_data_proc_rot_global_to_local.h"
std::vector< std::vector<int> > mJointConnectivity;
/*
Some code for specifying the joint connectivity
*/
std::shared_ptr<dab::DataProcPosGlobalToLocal> mJointPosGlobalToLocalProc;
std::shared_ptr<dab::DataProcRotGlobalToLocal> mJointRotGlobalToLocalProc;
mJointPosGlobalToLocalProc = std::shared_ptr<dab::DataProcPosGlobalToLocal>(new dab::DataProcPosGlobalToLocal("joint pos global local calc", "jointPosLocal", mJointConnectivity));
mJointRotGlobalToLocalProc = std::shared_ptr<dab::DataProcRotGlobalToLocal>(new dab::DataProcRotGlobalToLocal("joint rot global local calc", "jointRotLocal", mJointConnectivity));The constructor of the DataProcScale class takes as third argument a float value that represents the scaling factor.
#include "dab_data_proc_scale.h"
std::shared_ptr<dab::DataProcScale> mJointPosScaleProc;
mJointPosScaleProc = std::shared_ptr<dab::DataProcScale>(new dab::DataProcScale("joint pos scale", "jointPosition", 0.01));The constructors of the DataProcDerivative and DataProcDerivativeQuat classes lack a third argument.
#include "dab_data_proc_derivative.h"
#include "dab_data_proc_derivative_quat.h"
std::shared_ptr<dab::DataProcDerivative> mJointVelocityProc;
std::shared_ptr<dab::DataProcDerivativeQuat> mJointRotQuatVelocityProc;
mJointVelocityProc = std::shared_ptr<dab::DataProcDerivative>(new dab::DataProcDerivative("joint velocity calc", "jointVelocity"));
mJointRotQuatVelocityProc = std::shared_ptr<dab::DataProcDerivativeQuat>(new dab::DataProcDerivativeQuat("joint rot quat velocity calc", "jointRotQuatVelocity"));
The constructor of the DataProcLowPass and DataProcLowPassQuat classes take a float value as third argument. This value specifies the weighted sum of the old and new data. The sum is calculated as follows: new data = old data * scaling + new data * (1.0 – scaling).
#include "dab_data_proc_lowpass.h"
#include "dab_data_proc_lowpass_quat.h"
std::shared_ptr<dab::DataProcLowPass> mJointVelocitySmoothProc;
std::shared_ptr<dab::DataProcLowPassQuat> mJointRotQuatVelocitySmoothProc;
mJointVelocitySmoothProc = std::shared_ptr<dab::DataProcLowPass>(new dab::DataProcLowPass("joint velocity smooth", "jointVelocity", 0.9));
mJointRotQuatVelocitySmoothProc = std::shared_ptr<dab::DataProcLowPassQuat>(new dab::DataProcLowPassQuat("joint rot quat velocity smooth", "jointRotQuatVelocity", 0.9));The constructor for the DataProcScalar takes as third argument the type of reduction to perform. The enum ScalarMode specifies these types.
#include "dab_data_proc_scalar.h"
std::shared_ptr<dab::DataProcScalar> mJointVelocityScalarProc;
mJointVelocityScalarProc = std::shared_ptr<dab::DataProcScalar>(new dab::DataProcScalar("joint velocity scalar calc", "jointVelocityScalar", dab::DataProcScalar::Max));The constructor for the DataProcFilter class takes as third argument a vector of indices for those features that should not be removed in the output data.
#include "dab_data_proc_filter.h"
std::vector<int> mTorsoJointIndices;
/*
Some code for specifying the feature indices
*/
std::shared_ptr<dab::DataProcFilter> mJointPosTorsoFilterProc;
mJointPosTorsoFilterProc = std::shared_ptr<dab::DataProcFilter>(new dab::DataProcFilter("joint position torso filter", "JointPosTorso", mTorsoJointIndices));The constructor for all the Laban Effort Factor processing classes take as third argument an integer value that specifies the length of the temporal window over which the Effort Factor is calculated. These processing classes as posses a function named “setWeights” to set the weighting factor for each feature (joint). This weighting factor controls how much a particular feature (joint) contributes to the overall Effort Factor.
#include "dab_data_proc_weight_effort.h"
#include "dab_data_proc_time_effort.h"
#include "dab_data_proc_flow_effort.h"
#include "dab_data_proc_space_effort.h"
#include "dab_data_proc_space_effort_2.h"
std::vector<float> mJointWeights;
/*
Some code for specifying the weights
*/
std::shared_ptr<dab::DataProcWeightEffort> mJointWeightEffortProc;
mJointWeightEffortProc = std::shared_ptr<dab::DataProcWeightEffort>(new dab::DataProcWeightEffort("joint weight effort calc", "jointWeightEffort", 10));
mJointWeightEffortProc->setWeights(mJointWeights);The constructor for the DataProcCombine class lacks a third argument.
#include "dab_data_proc_combine.h"
std::shared_ptr<dab::DataProcCombine> mWeightEffortCombineProc;
mWeightEffortCombineProc = std::shared_ptr<dab::DataProcCombine>(new dab::DataProcCombine("weight effort combined", "WeightEffortCombined"));
The constructor for the DataProcQuatEuler class lacks a third argument
#include "dab_data_proc_quat_euler.h"
std::shared_ptr<dab::DataProcQuatEuler> mJointRotQuatEulerProc;
mJointRotQuatEulerProc = std::shared_ptr<dab::DataProcQuatEuler>(new dab::DataProcQuatEuler("joint rot euler calc", "jointRotEuler"));
Instances of data processing classes must be added to the processing pipelines in order to be actively used during data processing. The DataProcPipeline class provides the function “addDataProc” for this purpose.
mDataProcPipeline->addDataProc(mJointPosGlobalToLocalProc);The instances of the data processing classes must also be connected to each other so that the output data from one processing unit becomes the input data for another processing unit. The DataProcPipeline class provides the function “connect” for this purpose. The first argument to this function is a pointer to a data processing unit that provides the output data, the second argument is a pointer to a data processing unit that uses the output data as input.
mDataProcPipeline->connect(mJointPosInputProc, mJointPosGlobalToLocalProc);If a data processing unit receives its input from several other data processing units, then the “connect” function is called multiple times with the second argument staying the same.
mDataProcPipeline->connect(mJointSpaceEffortProc, mSpaceEffortCombineProc);
mDataProcPipeline->connect(mJointTorsoSpaceEffortProc, mSpaceEffortCombineProc);
mDataProcPipeline->connect(mJointLeftArmSpaceEffortProc, mSpaceEffortCombineProc);
mDataProcPipeline->connect(mJointRightArmSpaceEffortProc, mSpaceEffortCombineProc);
mDataProcPipeline->connect(mJointLeftLegSpaceEffortProc, mSpaceEffortCombineProc);
mDataProcPipeline->connect(mJointRightLegSpaceEffortProc, mSpaceEffortCombineProc);dab::DataMessenger (again)
To send the content of the output data from a data processing unit via OSC to a destination software, the output data has to be assigned to an existing DataSender instance using the “createDataSender” function of the DataMessenger class. This function takes as arguments a pointer to a Data instance, a string for the address of the OSC message to be created, and the name of the existing DataSender instance. A pointer of the output data is obtained using the “data” function of the DataProc class. Since a subclass of the DataProc class can potentially create more than one output data, the “data” function returns a vector of pointers for Data instances.
mDataMessenger->createDataSender(mJointPosGlobalToLocalProc->data()[0], "/mocap/joint/locpos", "MocapProcSender");Once data has been assigned to a DataSender, this data is sent as OSC message whenever the data is updated. To pause this sending, the DataSender can be set to inactive for a specific Data instance. For this, the “setDataSenderActive” function of the DataMessenger class is used. This function takes as first argument a pointer to the data and as second argument a boolean value. This value is set to false to stop the sending of the data or to true to resume the sending.
mDataMessenger->setDataSenderActive(mJointPosGlobalToLocalProc->data()[0], false);dab::DataPlot
Each instance of the DataPlot class can be assigned an instance of a Data class. The DataPlot class then creates one or several time plots, one for each dimension of the features stored by the Data class. The constructor of the DataPlot class takes five arguments: a string for the name of the plot, a pointer to an instance of the Data class, an integer specifying the length of the data history to be plotted, an ofVec2f for specifying the position of the plot in the application window, and another ofVec2f to specify the width and height of the plot. Furthermore, the DataPlot class contains a function named “setDataRange” to specifiy minimum and maximum values within which the data values should be normalised for displaying.
#include "dab_data_plot.h"
std::vector<std::shared_ptr<dab::DataPlot>> mDataPlots;
mDataPlots.resize(16);
ofVec2f plotPos(-59.0, 30.0);
ofVec2f plotSize(400.0, 200.0);
int plotHistoryLength = 20;
mDataPlots[0] = std::make_shared<dab::DataPlot>("Pos", mJointPosData, plotHistoryLength, plotPos, plotSize);
mDataPlots[0]->setDataRange({ -500.0, -500.0, 0.0 }, { 500.0, 500.0, 400.0 });Since displaying time plots is computationally demanding, it is recommended to only display of few or a single plot at a time. The example in the addon contains a mechanism for switching the display between one plot at a time.