Quantifying 3D range of motion using off-the-shelf, low-cost cameras

Quantifying 3D range of motion using off-the-shelf, low-cost cameras

After a gestation of three-and-a-half years, my mPhil thesis, about a low-cost, easy-to-use setup for 3D ROM studies, has finally been submitted. It'll be circulated eventually, but, in this article, I'll be giving a personal summary of what the thesis features and the story behind it. 

The background

Right, a brief overview of the science behind it all:

Reliable reconstructions of joint ranges of motion (henceforth ROM) in fossil archosaurs are important in understanding aspects of their appearance, behaviours and palaeoecology. However, this is complicated by the fact that soft tissues, which markedly affect ROM, rarely fossilise - whilst this was initially paid little attention, there's been a shift to taking this into account.

(This is particularly relevant to my supervisor - Dr Peter Falkingham - whose main area of research is in dinosaur locomotion and footprints.)

Fortunately, the digital revolution has enabled new insights into this, meaning that researchers can construct studies into functional morphology without the limitations of handling large, heavy and often surprisingly fragile bones, and advanced workflows, such as XROMM, enabling 3D motion to be reconstructed in a wide range of species. Whilst, traditionally, a lot of this was prohibitively complicated and/or expensive, there's been a shift in research to working on low-cost easy-to-use digitation and analysis methods, with the ultimate aim of widening the scope of these studies for researchers of all disciplines.

This study was part of that, with the ultimate aim to create a low-cost, easy-to-use workflow using XROMM methods and photogrammetry and an experimental setup involving two off-the-shelf GoPro cameras - assessing its accuracy and precision using a LEGO wand and ascertaining its effectiveness on biological data through dissecting an ostrich foot. Was it successful? Sort of. 

I'm not going to go into length about the trials and tribulations I underwent doing this thesis - but, suffice to say, the process was often aggravating and caused a great deal of stress (three COVID lockdowns didn't help) and, due to various complications, the project (initially pencilled in for the beginning of June) didn't end up submitted until the middle of July. Still, the end result was worth all the effort.

I owe a gigantic debt of thanks to a lot of people for their invaluable aid, support and encouragement throughout. To those who helped; you're all listed in the acknowledgements. Hope I didn't forget anybody.

How was it done?

One of the earliest things Peter and I did was sit down and make a timetable for the project - now, given most of it ended up disregarded,  I won't go into it in considerable detail except for saying that April 2021 was pencilled in as the most likely date for the dissections (Hahahahahaha...)

And so I got to work. 

Here's a few paragraphs about the experimental setup:

Two GoPro cameras were placed at roughly 45 degrees to each other, and were remotely triggered either using a WiFi remote or through voice commands (when triggered simultaneously, one of the cameras started recording slightly later than the other), with a Samsung mobile phone stopwatch being used to ensure the cameras were fully synchronised. A typical data capture session takes about 2-5 minutes.

For camera calibration (and to measure accuracy and precision), a calibration object made out of LEGO blocks (as seen in the picture below) was used - Lego was chosen because Lego bricks are manufactured to a high degree of precision.

After the footage was taken, it was transported into Blender and the individual videos were synchronised (basically when the time on the stopwatch was identical in both videos thoroughout the timeline). It was then trimmed, exported and loaded into XMALab, where the individual markers were tracked and rigid body transformations calculated. The RBTs were then integrated with a photogrammetric 3D model of the object (created using photographs of the object and Autodesk Metashape) in Autodesk Maya.

After familiarising myself with the hardware (a process that took six months... right up to the time the campus ended up closing - funny), the first item on the agenda was to assess the accuracy and precision - this was done using a LEGO wand with markers placed at a known distance (as seen in the picture below). A secondary aim was to assess the versatility of the setup by testing the effects of differing camera settings (e.g. differences in resolution, framerate and field of view) and differing calibration positions on accuracy and precision.

As an aside, several hardware shifts happened early on in the experimentation. Initially, the aim was to use three Raspberry PI cameras to collect video data - however, they proved to be a nightmare to work with. Only one of the cameras was consistently reliable - the third developed a fault after a couple or so weeks and couldn't be used and another only worked intermittently.

After returning from the first COVID lockdown (and I had rethought the project over the six months out), I ended up switching to a pair of GoPro Hero 7 Black Cameras - due to the fact that they are relatively inexpensive (£270 each) and can provide high picture quality at a wide range of settings.

A couple of other shifts (one hardware, one software) occurred in the photogrammetry side of things too - issues with picture quality led to a switch to a Sony NEX-6 camera, whilst both inconsistent model quality and lengthy processing time in Meshroom necessitated a switch to Agisoft Metashape.  

Eventually, after two-and-a-half years, the dissection was ready to be carried out. Now, the delay of almost a year meant that the initial plans - to do an ostrich foot and an ostrich neck - had to be changed when the ostrich neck turned out to have frozen solid. In addition, plans to do practice dissections on chicken material were mothballed after one try - due to issues with the compromise between marker retention and bone breakage.

So the ostrich foot was extended and flexed at three levels of sequential dissection - one with all soft tissues attached (skin-on), one with the integument removed (skin-off) and one with underlying musculature removed. One trial was recorded for each level of dissection. As in the accuracy and precision, footage was then synchronised in Blender and exported into XMALab, where the markers were tracked and rigid body transformations calculated. The foot was then skeletonised, the bones (the tarsometatarsus and each individual phalange) were digitised using photogrammetry and loaded into Metashape.

This data was then integrated in Maya - the photogrammetric models were transferred in and aligned in a reference pose (based on where they naturally rested, rather than geometric calculations, leading to minor issues), the centroids of the markers were calculated and locators placed (with a digital model of the intact foot used to reconstruct 3D co-ordinates of the lost ones). The RBTs were then imported into Maya to animate the bone models - to ensure they matched, the GoPro camera positions and image planes were exported too, with scientific rotoscoping being used to correct the alignment when necessary. 

To calculate bone rotation, joint co-ordinate systems were set at the reference pose and aligned with the bones using locators - rotation about the Z-axis (flexion-extension) was specifically focused on, with the Maya graph editor used to visualise the kinematics, with an average across the duration of max extension/flexion being taken. 

So... Does It Work?

Sort of.

From the accuracy and precision results, the workflow maintains marker-based XROMM's ability to produce accurate and precise results - and, whilst changing field of view negatively affected accuracy and precision, it remained high at a variety of framerates, resolutions and differing calibration positions. So the setup can both produce accurate and precise results and is versatile. 

For biological data, the question's a bit murkier - whilst it can potentially be used on biological data in an effective manner... it's clear that there is a considerable degree of refinement needed - the results on the dissection had a number of limitations, due to the fact it was comparatively poor quality (great deal of high-frequency noise in the max extension/flexion figures, as seen below), due to factors such as a lack of marker retention, manual errors made during marker tracking (due to markers blending in) and a possible lack of uniformity in scaling.

However, this is common in XROMM workflows - and is an unavoidable side effect of the realities of working with biological data. Despite its ability to produce excellent results in optimal scenarios, the simple reality is there's always going to be problems with marker insertion, tracking and reconstruction when dealing with biological data.

Now, it's important to note that I encountered multiple pitfalls - as mentioned before, several switches in hardware were made due to repeated technical difficulties and issues with image quality and there are some trade-offs between accuracy, precision and reprojection error when considering optimal settings. 

The main pitfall is that whilst this method is certainly cost-effective, the intensive file management that it requires is a possible offset, since the manual synchronisation of video data can lead to the trimming, synchronising and managing of data files being a time-consuming process (at the end of the ostrich dataset, I had roughly 18 files in total - raw data, synchronised footage, the Blender file, the photogrammetric model of each phalange and the Maya integration scene).

For biological data, I should emphasise that you should really take care with certain aspects such as marker choice and placement - especially if you are doing it on a smaller, more delicate bone (e.g. chicken bones). As well as this, marker retention is another issue you are going to encounter - markers falling out/moving during dissection/skeletonisation complicated aspects of marker tracking and reconstruction in Maya. 

So... what inferences can we make for fossil archosaurs from the biological data? A surprising lot. 

Analysis of the foot concluded that overall ROM increased as soft tissue was removed, with considerable variation between individual phalanges (being highest in phalange 1 and decreasing in the more distal phalanges) due to differing levels of soft tissue covering. In addition, removing integument had a smaller impact that removing underlying musculature. This supports the results of previous studies on other joints. 

Now, this is interesting for a bunch of reasons - ostriches are the most cursorial of living birds and adaptations for fast locomotion are seen in the ostrich foot... which is reflected in many aspects of the ROMs reported (e.g. digit 3, phalange 1 having the highest ROM due to the role of the metatarsophalangeal joint in energy storage and release).

This opens up an interesting theory - soft tissue reconstructions of pedal ROM, based on the extant phylogenetic bracket and osteological correlates of soft tissue, could be used to assess degrees of cursoriality in fossil bipedal archosaurs. A comparison of pedal ROM in a broader range of taxa could be useful in this regard... 

Conclusions

To wrap this up, the main takehomes are this:

• The setup cost less than £550 - not counting the laptop used for data analysis and the camera used for photogrammetry.
• Whilst the setup could produce accurate and highly precise results  with a variety of camera settings, peak accuracy and precision was seen at the 2.7kp/60fps/linear configuration - however, past a certain point, accuracy and precision wasn't significantly affected by a change in resolution.
• The workflow can potentially be used in an effective way on biological data... but a lot of refinement's needed.
• Soft tissue removal in an ostrich foot was found to markedly increase the ROM - further reinforcing the need for extensive ROM knowledge in extant analogues when reconstructing the movements of extinct taxa and provides a possible framework by which researchers can assess degrees of cursoriality in fossil bipedal archosaurs.

Finally, I'd like to say what a significant personal achievement this was for me - it's the end result of three-and-a-half years of long, hard and often aggravating work in times that were sometimes stressful. Getting it out was what I was trying to do.