This is the seventh tutorial in the series on how to create statistical shape models. So far, we’ve only visualized our model through manual inspection in Scalismo-UI, but a number of additional useful visualization methods exist.
If you are to include your model on a website or in a presentation, I often find it much more engaging to make a small animation of how the model deforms. A typical strategy I use is to write the principal components from 0 to 3 standard deviation, back to -3 and to 0, and do so for the top principal components. Before diving into the code, let’s look at the resulting animation that we are looking to represent:
The first step is to write meshes to a folder for each change to the model parameters. First, we define the values each component should take, i.e., from 0
to 3.0
to 0
to -3.0
and to 0
. Depending on your model, it might make sense to only show +/- 2.0
or +/- 1
standard deviation as 3.0
standard deviation is a very unlikely shape.
val valstop = 30
val stepSize = 5
val rangeInts =
(0 to valstop by stepSize) ++ (valstop to -valstop by -stepSize) ++ (-valstop to 0 by stepSize)
val rangeDouble = rangeInts.map(f => f.toDouble / 10.0)
Next, we define the components that we want to capture, in this case, components 0, 1, and 2. Then, we create a function to write the steps to a folder
val outputDir = new File("data/tmp/")
outputDir.mkdirs()
def processComponent(comp: Int): Unit = {
rangeDouble.foreach { v =>
val modelPars = DenseVector.zeros[Double](ssm.rank)
modelPars(comp) = v
val instance = ssm.instance(modelPars)
MeshIO.writeMesh(
instance,
new File(outputDir, s"${"ssm_%04d".format(cnt)}.ply")
)
cnt += 1
}
}
val components = Seq(0, 1, 2)
components.foreach { comp =>
println(s"Component: $comp")
processComponent(comp)
}
The next step is to load the mesh sequence into Paraview. Select all the meshes from the output folder and drag them to the browser area in Paraview.
To smoothen the mesh visualization, apply the Generate Surface Normals
filter to the mesh.
Next, set the canvas size to the dimensions that you would like the animation to be under View -> Preview -> Custom
, in this example, I choose an image size of 800x800
pixels.
If the Animation view is not visible, enable it view -> Animation View
. Set Mode to Sequence
, set the start time to 0
, the end time to the number of meshes in your exported folder, and the No. Frames
to the frame rate times the animation duration. In our case, we chose 25FPS and the clip to last 4 seconds, which means 100 frames.
Using the Play button at the top, check how the animation looks like. And check that the mesh is positioned correctly in the scene.
The last step in Paraview is now to export the animation. I recommend exporting it as PNG files and converting those to a GIF afterward. Alternatively, Paraview also has a video export option.
To export File -> Save Animation...
, select a name, folder, and in the Animation Optionts
, remember to choose Transparent Background
.
Finally, we need to convert the PNG files to a GIF. This can be done using (Imagemagick)[https://imagemagick.org/index.php].
With the terminal, CD
into the folder with the PNG files and execute
convert -delay 4 -loop 0 -alpha set -dispose previous *.png ssm.gif
This will create the final GIF file. NOTE that the -delay is in 100ths of a second, so 4 is 0.04 seconds, meaning a frame rate of 25 fps.
If the animation isn’t smooth enough, then go back and export a more dense set of meshes.
Another helpful visualization technique is to show the point correlation in the model. For this, we will select a point on the model and color the mesh based on the correlation with the selected point. This method is also good to use when manually defining the model kernels.
val model: PointDistributionModel[_3D, TriangleMesh] = ???
val lmId: PointId = ???
val dist: Array[Double] = model.reference.pointSet.pointIds.toSeq.map { pid =>
val cov = model.gp.cov(lmId, pid)
sum((0 until 3).map(i => math.abs(cov(i, i))))
}.toArray
val colorMesh = ScalarMeshField[Double](model.reference, dist)
val ui = ScalismoUI()
ui.show(model, "model")
ui.show(colorMesh, "dist")
To begin with, we can try to visualize the manually created model
val dataDir = new File("data/vertebrae/")
val modelFile = new File(dataDir, "gpmm.h5.json")
val model = StatisticalModelIO.readStatisticalMeshModel(modelFile).get
val modelLmFile = new File(dataDir, "ref_20.json")
val landmarks = LandmarkIO.readLandmarksJson[_3D](modelLmFile).get
val lmId = model.reference.pointSet.findClosestPoint(landmarks(6).point).id
As the model was created with a single Gaussian kernel, we correctly see that the further we get away from the selected landmark point, the less the points correlate with the landmark.
If we now instead use the created PCA model, we see that the kernel has incorporated that points on the mesh’s back side correlate with the selected landmark point. This is most likely connected with the size of the mesh.
The last visualization method I want to show is how we can visualize deformation fields. This method is useful to check how smooth the deformation fields of a model are, as well as debugging correspondence estimation functions (such as ICP).
val modelFile = new File(dataDir, "pca.h5.json")
val model = StatisticalModelIO.readStatisticalMeshModel(modelFile).get
val ref = model.reference
val sample = model.sample()
val deformations = ref.pointSet.pointIds.map (id => sample.pointSet.point(id) - ref.pointSet.point(id)).toIndexedSeq
val deformationField = DiscreteField3D(ref, deformations)
So, in summary, always visualize your models and try out different visualization methods to better understand the inner workings of your model.
]]>This is the sixth tutorial in the series on how to create statistical shape models.
When it comes to model evaluation, nothing beats visual inspection of the models anatomical properties. With Scalismo-UI, we can inspect the individual principal components of the model one by one, as shown in the previous tutorials.
Load in the model and inspect the components. Then, try to randomly sample from the model to see if the shapes look realistic and anatomically plausible. And, very importantly, check that the model does not produce any kind of artifacts, such as self-intersection, inverted surface, or similar.
As a second step, consider the application where the model will be used and see if an automatic evaluation can be created. This could, for instance, be partial mesh completion. In such a case, you can synthetically remove parts of a mesh and check the model’s reconstruction with the ground truth. A good first step for this is to use the training data itself to begin with. For the training data, we know that the model should be able to represent it precisely. Then, after it is validated, we can move on to check the model’s performance on a test dataset where we also know the complete shape. And finally, the model can be used on partial data with no ground-truth data. This data can, of course, only be qualitatively evaluated.
Instead of comparing the complete meshes with a ground-truth mesh, another common evaluation strategy is landmark-based evaluation. Here, the model’s ability to identify specific landmark points is checked. For the model, we need to create a landmark file where the landmarks are placed on the template mesh. Then, we extract the position of the same point ID after fitting. The task is then to compare the landmarks with landmarks clicked manually by a human observer and possibly other automatic methods if available.
Besides qualitatively looking at the model or looking at the model performance on a specific application task, we can also look at some fundamental properties of the model, e.g.
These three properties are referred to as Specificity, Generalization, and Compactness. The metrics were introduced by Martin Styner in his 2003 paper: Evaluation of 3D correspondence methods for model building and has since become the standard when evaluating statistical shape models. Even though the methods provide good insight into a single model, the metrics are more useful when comparing different models, e.g., built on different datasets or built using different techniques.
The three metrics are usually reported by limiting the model to only use a certain number of the principal components. A 2D plot can then be created by plotting the number of principal components used on the X-axis and the metric value itself on the Y-axis.
Now, let’s go over our Vertebrae example and see how we can calculate each of the Metrics for the model.
Generalization is measured on data items not in the dataset, so we will build our model on 7 of the Vertebrae samples and test on the remaining 3. A different strategy is to do the average of a leave-one-out calculation if limited data is available.
First, let’s define our training and test data items and build our model to be evaluated.
val dataDir = new File("data/vertebrae/")
val ref = MeshIO.readMesh(new File(dataDir, "ref_20.ply")).get
val registeredDataFolder = new File(dataDir, "registered")
val allPLYFiles = registeredDataFolder
.listFiles()
.filter(_.getName.endsWith(".ply"))
.sorted
.map(MeshIO.readMesh(_).get)
val trainingData = allPLYFiles.take(7)
val testData = allPLYFiles.drop(7)
val dataCollectionTraining = DataCollection.fromTriangleMesh3DSequence(ref, trainingData)
val dataCollectionTesting = DataCollection.fromTriangleMesh3DSequence(ref, testData)
val ssm = PointDistributionModel.createUsingPCA(dataCollectionTraining)
Then, I’ll use Scalismo’s built-in ModelMetrics functionality to compute the model’s Generalization. To the function, we just need to pass the model and the data collection to be used for the calculation, in this case, our testing data.
ModelMetrics.generalization(ssm, dataCollectionTesting).get
Let’s now call the function with the model limited to a certain number of principal components
val generalization = (1 until ssm.rank).map(i => {
val m = ssm.truncate(i)
val g = ModelMetrics.generalization(m, dataCollectionTesting).get
(i, g)
})
Finally, let’s plot the output data
Note again that the X-axis specifies the number of principal components we use from the model, while the Y-axis is the average distance in millimeters between the model and the test meshes.
In the plot we see how the model Generalizes better the more principal components are used as would be expected.
If we compute the Generalization on the training data. When using all the principal components, the model should be able to precisely describe the data, i.e., a generalization of ~0.
We also see that this is the case. This, again, can be used to check if there is a need for all the principal components in the model. If we, e.g., have a model with 100 principal components, but already after 50, it can perfectly describe both the training data and test data, this could hint towards some data not being needed in the model.
For the specificity, we again use the function available in Scalismo’s ModelMetrics. To calculate the specificity value we pass in the training data, NOT the test data. We then specify a value describing the number of samples that the function should take. The higher this number is, the more precise the specificity number will be, but it will at the same time take longer to calculate. I usually use a value of at least 100. Internally, what is happening is that the function will produce 100 random samples from our model and then calculate the average distance between the sample and each item in the training data set. For the final specificity value, only the closest training data item is used. In other words, for each sample, the function finds the most similar item in the training data and calculates the average distance between the two. It then takes the average of all the 100 random samples that were produced.
val specificity = (1 until ssm.rank).map(i => {
val m = ssm.truncate(i)
val s = ModelMetrics.specificity(m, trainingData.toSeq.toIterable, 100)
(i, s)
})
For the specificity, we should see that the more principal components that are used, we should be able to create shapes that are further away from the training data, i.e. a higher specificity value as also found in our case.
For specificity, the Axis’s describe the same as for Generalization - number of principal components and average distance in millimeters.
Finally, the compactness is a measure the model’s ability to use a minimal set of parameters to represent the data variability. It is straightforward to calculate from the variance information stored about each principal component in the model.
val compactness = (1 until ssm.rank).map(i => {
val c = ssm.gp.variance.data.take(i).sum
(i, c)
})
We see that each added component adds a lot of new information to the model. Again, if we had a model with 100 principal components, we might see the values flatten out after some time, suggesting that the remining principal components provide little to no extra flexibility to the model.
Also in this plot, the X-axis is the number of principal components that we use from the model, while the Y-axis is the total variance mm^2
in the model.
While the model metrics can be used to evaluate individual models, they are more useful when comparing different models. As described, these models could be built from different datasets or using different techniques to create the model. Another useful area would be to compare the PCA model
to different augmented models
, as introduced in one of the previous tutorials. This can help to check if the model is augmented enough to generalize better but at the same time not too much to suddenly produce unrealistic shapes, i.e., high specificity.
So, in summary, always visualize your models and try to formulate an application-specific valuation for your model. Finally, I showed the commonly used metrics to evaluate statistical shape models: generalization, specificity, and compactness.
In the next tutorial, I’ll go over:
This is the fifth tutorial in the series on how to create statistical shape models.
In this tutorial, we’ll first manually code up a model fitting function to understand all the aspects of what goes into non-rigid registration using the Gaussian Processes and the Scalismo library.
At first, let’s load in a model and a target mesh. Before doing so, be sure to have executed steps 1 and 2 in the src/prepare_data
folder as this will first align all our target data and create the model to use.
val dataDir = new File("data/vertebrae/")
val gpmmFile = new File(dataDir, "gpmm.h5.json")
val lmsFile = new File(dataDir, "ref_20.json")
val gpmm = StatisticalModelIO.readStatisticalTriangleMeshModel3D(gpmmFile).get
val lms = LandmarkIO.readLandmarksJson[_3D](lmsFile).get
val targetMesh = MeshIO.readMesh(new File(dataDir, "aligned/sub-verse010_segment_20.ply")).get
val targetLms = LandmarkIO.readLandmarksJson[_3D](new File(dataDir, "aligned/sub-verse010_segment_20.json")).get
val ui = ScalismoUI()
val modelGroup = ui.createGroup("modelGroup")
val targetGroup = ui.createGroup("targetGroup")
ui.show(modelGroup, gpmm, "gpmm")
ui.show(targetGroup, targetMesh, "target")
Since the target data in this example is very noisy, our goal is not to do a perfect fit, but instead to capture the overall size of the target data. Since we have landmark points available both for the model and the target mesh, we can start out trying to “fit” our model to these landmark points.
val lmsData = lms.zip(targetLms).map{ case(lm1, lm2) => (gpmm.reference.pointSet.findClosestPoint(lm1.point).id, lm2.point)}.toIndexedSeq
val lmPosterior = gpmm.posterior(lmsData, 1.0)
val lmFit = lmPosterior.mean
For some applications, this might be good enough. We can even continue adding landmarks to get the surfaces closer and closer together. You might also want to play around with the uncertainty value when calculating the posterior, this value should be seen as the uncertainty of the landmark observation.
Instead of adding more landmarks, we want to find the “corresponding points” automatically. To do so, we will implement a form of iterative closest point (ICP)algorithm, which uses the same principle as above, to calculate a posterior model given some observations and then we take the most likely shape, i.e. the mean from the distribution as our proposal. To find the corresponding points, we simply estimate this to be the closest point on the target surface. To begin with, we can then assign a large uncertainty value to our observation. The idea is then to iteratively move the model closer to the target mesh, by estimating new corresponding points in each iteration and lowering the uncertainty. To begin with, I will just show you a very simple implementation method that has the same structure as the rigid ICP alignment we saw in Tutorial 2
def nonrigidICP(model: PointDistributionModel[_3D, TriangleMesh], targetMesh: TriangleMesh3D, numOfSamplePoints: Int, numOfIterations: Int) : TriangleMesh3D =
val numOfPoints = model.reference.pointSet.numberOfPoints
val ptIds = (0 until numOfPoints by (numOfPoints / numOfSamplePoints)).map(i => PointId(i))
def attributeCorrespondences(movingMesh: TriangleMesh3D) : IndexedSeq[(PointId, Point[_3D])] =
ptIds.map( (id : PointId) =>
val pt = movingMesh.pointSet.point(id)
val closestPointOnMesh2 = targetMesh.pointSet.findClosestPoint(pt).point
(id, closestPointOnMesh2)
)
def fitting(movingMesh: TriangleMesh3D, iteration: Int, uncertainty: Double): TriangleMesh3D =
println(s"iteration: $iteration")
if (iteration == 0) then
movingMesh
else
val correspondences = attributeCorrespondences(movingMesh)
val posterior = model.posterior(correspondences, uncertainty)
fitting(posterior.mean, iteration - 1, uncertainty)
fitting(model.reference, numOfIterations, 1.0)
A ton of configuration possibilities exist for the ICP algorithm, for instance how the closest points are taken, which is calculated in the attributeCorrespondence
function. This could be either the closest Euclidean point on a target surface, the closest vertex on the target surface (as done), closest point along the surface normal, we could also estimate the closest points from the target to the model instead, and many more methods exist to make it more robust. The same goes for the uncertainty
value, which can either be manually set for all correspondent pairs or we can come up with a way to calculate the uncertainty based on the distance between the model surface and the target surface for each point. In the example, the uncertainty is a standard multivariate normal distribution, but we could also provide different uncertainty in different directions.
When running the fitting, we can either make use of the original model gpmm
or we can use the model that is conditioned on the landmark observations.
// Use posterior/conditioned model from landmarks
val icpFit = nonrigidICP(lmPosterior, targetMesh, 100, 50)
// Use complete initial model
val icpFit = nonrigidICP(gpmm, targetMesh, 100, 50)
To evaluate the fit to the target, some common metrics to use are the average distance and Hausdorff distance. These can be used to quickly get an idea about the quality of the fit.
def evaluate(mesh1: TriangleMesh3D, mesh2: TriangleMesh3D, description: String): Unit =
val avg1 = MeshMetrics.avgDistance(mesh1, mesh2)
val avg2 = MeshMetrics.avgDistance(mesh2, mesh1)
val hausdorff1 = MeshMetrics.hausdorffDistance(mesh1, mesh2)
val hausdorff2 = MeshMetrics.hausdorffDistance(mesh2, mesh1)
println(s"$description - avg1: $avg1, avg2: $avg2, hausdorff1: $hausdorff1, hausdorff2: $hausdorff2")
evaluate(targetMesh, lmFit, "lmFit")
evaluate(targetMesh, icpFit, "icpFit")
In my case, the output was:
lmFit - avg1: 2.03, avg2: 1.65, hausdorff1: 8.20, hausdorff2: 8.20
icpFit - avg1: 1.81, avg2: 1.22, hausdorff1: 8.67, hausdorff2: 8.67
This means that the average distance improved, but we have a slightly larger maximum distance found. Also, note that the distance from the target to the fit and from the fit to the target might not be the same as internally, the distance functions are using the closest point to decide the point on the opposite mesh.
In its full length: Generalized Iterative Non-Rigid Point Cloud and Surface Registration Using Gaussian Process Regression, is a framework built on top of Scalismo which implements some more sophisticated ways to perform non-rigid registration. And full disclosure, I’m one of the maintainers of the repository which is based on my PhD. thesis. The main principles behind the GiNGR framwork are exactly what we went through in the manual example that we coded up. We need to select a deformable model for the fitting, then we need to decide how the corresponding points are being estimated in each iteration and finally, we need to set the observation uncertainty.
The framework already comes with multiple different examples of how to perform fitting, but also automatic methods to calculate the models.
In the prepare_data/03_registration.scala
I’ve made use of the GiNGR framework where I make use of the Coherent Point Drift
implementation.
This method is very good in correcting minor rigid alignment offsets between the model and the target as well as giving a coarse fit to the data. As our data is very noisy, I’m already stopping after the coarse fit, as we would otherwise just start explaining the noise in the data with our model.
But let’s try to run the examples from the GiNGR repository, to get a feeling for how it can be used to fit very close to the target mesh. For this, let’s look at the MultiResolution demo. This demo first solves a small rigid offset in the target mesh as well as giving a rough fitting using the Coherent Point Drift
implementation. Notice how we use runDecimated
instead of run
, internally, this method will decimate both the model and the target mesh to speed up the fitting. In the second step, we use a less decimated model, still using the same fitting method. Finally, we switch to the Iterative Closest Point
algorithm, as explained at the start of this tutorial, where we do an additional step with the full resolution to fit all the intricate details in the mesh.
The average distance and max distances after each step:
STEP 1 - average2surface: 1.76 max: 9.31
STEP 2 - average2surface: 0.57 max: 5.92
STEP 3 - average2surface: 0.21 max: 5.25
By no means are all of these steps necessary in all cases. Always start with a simple model and one of the methods and try to identify what areas can be improved. Also, if speed is not an issue, you can of course skip decimating the meshes and just use the complete meshes.
If you would like to know more in detail about the technical aspects of GiNGR, we’ve also put out a white paper, which you can find on arXiv.
And now, finally, when we compute the fits of all the items in our Vertebrae dataset, we can refer to tutorial 1 on how to calculate our statistical shape model.
And that’s the end of the practical steps to create your first statistical shape model. The most crucial part is to look at your data and from there, decide how e.g. the kernel parameters need to look as well as the noise assumption during the fitting stage. If the dataset is very noisy, it does not make sense to create a model with thousands of basis functions that can perfectly fit the data. And likewise, if you have perfectly clean data, you need to add enough basis-functions to your model for it to be able to describe the data in detail. Also, remember to look at the official Scalismo tutorial on Model fitting.
The remaining tutorials are focused on model evaluation and different ways to visualize your created statistical models.
In the next tutorial I’ll go over:
This is the fourth tutorial in the series on how to create statistical shape models.
Defining the deformation space that the template can undergo might seem daunting with the endless possibilities of combinations.
Commonly asked questions on the Scalismo mailing list are “What parameters should I use for my model kernels” and “What kernels to choose”.
With a few heuristics in mind and a clear plan for defining your kernels, this task becomes a lot simpler.
For simplicity of this tutorial, I will not go into the formal definition of kernels. For this, I suggest you take a look at the tutorial from the scalismo tutorials from the Scalismo website or the instruction video from the statisticial shape modelling course at the University of Basel.
When talking about Gaussian Processes, a Kernel function describes how two data points are connected by describing their covariance. Simply said, when data point 1 moves, how much influence does this have on data point 2, if any at all? And also, how is the distance between point 1 and point 2 measured?
In this tutorial we will mainly look at the Gaussian kernel, and how we can modify it to, e.g. be symmetrical around a defined axis using the “symmetry kernel”, only be active in certain areas using the “Change Point Kernel”, or how to mix Gaussian kernels with different properties. Finally, I’ll show the PCA kernel, as introduced in the first video, and how we can make it more flexible by augmenting it with an analytically defined kernel.
5 different types:
Many more kernels exist than just the Gaussian kernel - but I often find mixing different Gaussian kernels sufficient, instead of using other kernel types.
For the Gaussian Kernel, we have two parameters to set:
val sigma = 100
val scaling = 1
val kernel = GaussianKernel3D(sigma, scaling)
val diagonal = DiagonalKernel3D(kernel, 3)
val gp = GaussianProcess3D[EuclideanVector[_3D]](diagonal)
One thing we need to remember when working with kernels is the dimensionality we work with. I will only show 3D examples, so it would also be possible to model the covariance between dimensions. But for simplicity, we always assume that the 3 dimensions are independent when analytically defining kernels. This is done using the DiagonalKernel.
With the kernel defined, we use it to define a Gaussian Process, which we’ll use later for regression purposes. The Gaussian Process that we have defined is continuous, but we are actually only interested in its values at the positions of our reference mesh. We can sample random deformations from the model:
val ref = MeshIO.readMesh(new File("data/vertebrae/ref_20.ply")).get.operations.decimate(1000)
val sampleDeformation = gp.sampleAtPoints(ref)
val interpolatedSample = sampleDeformation.interpolate(TriangleMeshInterpolator3D())
// val interpolatedSample = sampleDeformation.interpolate(NearestNeighborInterpolator3D()) // Alternative interpolator
val sample = ref.transform((p : Point[_3D]) => p + interpolatedSample(p))
Internally, the sampleAtPoints create a huge covariance matrix. So depending on your memory, you might need to decimate the mesh first to get the code snippet to work.
We can get around the problem by approximating the covariance matrix instead of explicitly calculating it.
val lowRankGP = LowRankGaussianProcess.approximateGPCholesky(
ref,
gp,
relativeTolerance = 0.1,
interpolator = NearestNeighborInterpolator3D()
)
val sampleDeformation = lowRankGP.sample()
val sample = ref.transform((p : Point[_3D]) => p + sampleDeformation(p))
The relativeTolerance
specifies the approximation error that is allowed. Setting it to 0.0 will mean that the low-rank approximation will precisely describe the full covariance matrix. Usually, a value around 0.01 is desired. But, to begin with, I often put a higher value like 0.5 or 0.1 to quickly calculate the function and visualize it. The interpolator to use very much depends on your application. Either the nearest neighbor or the triangle mesh interpolators are good choices to try out.
And now with the low-rank function, we should be able to sample without having to decimate our reference mesh first.
A more convenient way to visualize samples from a Gaussian process is to build a Point Distribution Model from the low-rank Gaussian process. This allows us to directly sample meshes that follow the distribution represented by the Gaussian process and not have to deform the reference mesh manually from the given deformations.
val pdm = PointDistributionModel3D(ref, lowRankGP)
val sampleFromPdm : TriangleMesh[_3D] = pdm.sample()
A Point distribution model can also be directly viewed in Scalismo, and we have access to all its parameters as well as a handle to sample from the model.
val ui = ScalismoUI()
ui.show(pdm, "pdm")
After having found the correct parameters to use for the model, it can be stored in a file and directly read again from disk, to avoid computing the model when we use it in the following tutorials.
StatismoIO.writeStatisticalTriangleMeshModel3D(pdm, new File("pdm.h5.json"))
val pdmRead = StatismoIO.readStatisticalTriangleMeshModel3D(new File("pdm.h5.json")).get
When inspecting the model, it is important to remember what the model will be used for. Our goal is to make the model flexible enough to represent all the other shapes in our dataset this also means that when we randomly sample from our model, it is perfectly fine that the deformations look exaggerated and produce non-natural shapes. The most important part is that the deformations are smooth such that the mesh does not intersect with itself. This also means that e.g. the mesh we see here is far from flexible enough to represent other meshes as it mainly shifts the position of the mesh around.
For the sigma value, I like to use a value that is 1/2 or 1/3 of the longest distance in the mesh. By looking at the scene view in Scalismo, we can get a feeling for the size of the mesh.
Here we see that the mesh is 65mm on the X-axis, 77mm on the Y-axis and 39mm on the Z-axis. With the current value of sigma to 100, this means that all points will have some correlation, what this ends up practically meaning is that the deformations will translate the mesh around. Let’s instead try to set Sigma to 35. Afterward, we can tune the scaling value if the magnitude of the deformations is not large enough.
I would recommend starting with a simple model like this with a simple Gaussian kernel and then only making it more advanced if needed.
And how do you know if more is needed? If the model has problems representing the meshes in your dataset, then more is needed. E.g. if some local curvatures are not nicely captured. You will only get to know so after running the non-rigid registration as introduced in the next two videos.
For completeness of this video, let’s continue adding some local deformations to our model by combining a kernel with a large sigma and one with a small sigma, I.e. a global and a local kernel. I typically visualize each model separately on the mesh and then combine the kernels afterward.
val kernelCoarse = GaussianKernel3D(35, 10)
val kernelFine = GaussianKernel3D(10, 3)
val kernel = kernelCoarse + kernelFine
val diagonal = DiagonalKernel3D(kernel, 3)
An alternative kernel is the symmetry kernel. To showcase this kernel, I’ll use the reference mesh from the Basel Face Model. First, let’s look at how a random sample from the face model looks like with the kernel
val kernel = GaussianKernel3D(100, 10)
val diagonal = DiagonalKernel3D(kernel, 3)
In the kernel, we’ll define the mesh to be symmetrical around the Z-axis. In reality, faces are of course not fully symmetrical, but it is a good global kernel to have, we can then always add local deformations to it.
case class xMirroredKernel(kernel : PDKernel[_3D]) extends PDKernel[_3D]:
override def domain = kernel.domain
override def k(x: Point[_3D], y: Point[_3D]) = kernel(Point(x(0) * -1.0 ,x(1), x(2)), y)
def symmetrizeKernel(kernel : PDKernel[_3D]) : MatrixValuedPDKernel[_3D] =
val xmirrored = xMirroredKernel(kernel)
val k1 = DiagonalKernel3D(kernel, 3)
val k2 = DiagonalKernel3D(xmirrored * -1f, xmirrored, xmirrored)
k1 + k2
val diagonal = symmetrizeKernel(GaussianKernel3D(100, 10))
Another kernel is the change point kernel. For this, let’s stick with the face mesh and make one side of the face with one kind of kernel and the other side with an inactive kernel. In this way, we should see that only half of the face deforms when we sample from the model.
case class ChangePointKernel(kernel1 : MatrixValuedPDKernel[_3D], kernel2 : MatrixValuedPDKernel[_3D]) extends MatrixValuedPDKernel[_3D]():
override def domain = EuclideanSpace[_3D]
val outputDim = 3
def s(p: Point[_3D]) = 1.0 / (1.0 + math.exp(-p(0)))
def k(x: Point[_3D], y: Point[_3D]) =
val sx = s(x)
val sy = s(y)
kernel1(x,y) * sx * sy + kernel2(x,y) * (1-sx) * (1-sy)
val diagonal = ChangePointKernel(
DiagonalKernel3D(GaussianKernel3D(100, 10), 3),
DiagonalKernel3D(GaussianKernel3D(1, 0), 3)
)
Make note of the s function, which defines which kernel to choose. This can either be binary to fully activate a kernel in a certain area, and fully deactivate it in others, or it can be made smooth as in the given example, such that the two kernels will have a smooth transition around the Z-axis in this case.
The final kernel I want to show is another mixture of kernels. This kernel could e.g. be used to iteratively include more data into your model. We start out with 10 meshes that are registered, from this, we can create a PCA kernel as also shown in the first video. Of course, 10 principal components rarely contain all small possible deformations, so we can augment the model e.g. with a Gaussian kernel, to make it more flexible.
val augmentedPDM = PointDistributionModel.augmentModel(pdm, lowRankGP)
And that’s the end of the practical guide to choosing your kernels and hyperparameters. Really the most crucial part is visualizing your models at every step of the way. Also, remember to look at the official Scalismo tutorial on Gaussian Processes and Kernels as the kernels are introduced there as well.
In the next tutorial I’ll show you:
This is the third tutorial in the series on how to create statistical shape models.
Choosing a good reference mesh is a very important step - but unfortunately also often overlooked. By mastering this step, you’ll be able to create shape models that generalize better and can be magnitudes faster and easier to use. It is very important to keep the final application of the shape model in mind when choosing a reference shape. A model used for animation in contrast to a model used for segmentation or for feature point extraction can have very different requirements.
So let’s go over the most common strategies:
For the semi-manual reference creation, I will use two different programs, besides the Scalismo library. These are Meshlab and Meshmixer. Meshlab will be used to align the mesh and apply different simple filters to the mesh: such as smoothening, sharpening, or triangulation decimation. Meshmixer will be used for manual correction of the mesh. Oftentimes, I even go back and forth a few times between the two programs. Many more options exist out there, such as Blender, Unity, Maya and many more.
So, let’s get started and create a reference mesh. For this, I’ll choose a random mesh from the Vertebra dataset as mentioned in the introduction tutorial of the series.
The very first thing I normally do is to align the mesh to the origin and rotate it in the direction you are interested in. This can either be done with Scalismo as showed in the previous tutorial, or we can do it semi-manually in e.g. MeshLab.
I typically try out an automatic alignment first, in MeshLab, this is found under filters -> Normals, Curvature and Orientation -> Transform: Align to Principal Axis
This will center the mesh and align the principal axes of the mesh to the x, y and z axes.
Alternatively, the center of the mesh can be manually set with the Transform: Translate, Center, set Origin
module.
Likewise, if the rotation of the mesh is not as you want, each individual axis rotation can be specified with the rotation transform Transform: Rotate
.
Aligning the reference is not strictly necessary, but it will make it much easier to e.g. specify region-specific kernels which we’ll see in one of the following tutorials.
Next up, we will inspect the mesh in Meshmixer. Now it is all a matter of using the different sculpt tools to clean up our mesh - or using the select tool to select items that we do not want in the reference. If the triangulation is very coarse, I usually start out applying the refinement method to the mesh and thereafter either draw, flatten, drag, move or use one of the smoothening tools to clean up the mesh.
The goal here is to remove noise as well as all possible biases that the mesh might have. We can also manually increase the triangulation in some areas of the mesh and decrease it in others, again, depending on the use case of the model.
The last step I do is to decimate the mesh. You might want to avoid this step if you have already manually defined the coarseness of the mesh everywhere you want. Again, the amount of decimation completely depends on your usage of the model. The more the mesh is decimated, the fewer points it will have, thereby it will be faster to compute and take up less space. But it will also not look as good when rendering it.
Let’s try out different decimation levels in MeshLab and see what it looks like. I prefer the mesh output from quadratic edge decimation algorithm filters -> Remeshing, Simplification and Reconstruction -> Simplification: Quadratic Edge Collapse Decimation
.
The same decimation algorithm is also available directly in Scalismo, so use it there you can simply load in the mesh, perform the decimation operation and save the mesh again. Like this, you can even have different coarseness levels of your reference mesh available
val mesh = MeshIO.readMesh(new File("mesh.ply")).get
val decimated = mesh.operations.decimate(1000)
MeshIO.writeMesh(decimated, new File("decimated.ply"))
That’s all there is to it. Some simple steps that will save you hours if not days of work down the line.
In the next tutorial I’ll show you:
This is the second tutorial in the series on how to create statistical shape models.
Aligning your dataset can be tedious, but the time spent here can be well worth it, as it will make it much simpler to establish point correspondence and thereby create great statistical shape models.
In this tutorial, I’ll go over 4 different methods for aligning your dataset. We will start out with two automatic methods that do not require any user input.
Also, have a look at the official Scalismo documentation, there is also a guide on rigid alignment link.
Origin alignment is an automatic method that works well if the orientation of the meshes is similar. This could also additionally be refined with rigid ICP alignment which we will discuss briefly.
val file = new File("data/vertebrae/raw/sub-verse010_segment_20.ply")
val mesh: TriangleMesh[_3D] = MeshIO.readMesh(file).get
val origin = mesh.pointSet.points.map(_.toVector).reduce(_ + _) / mesh.pointSet.numberOfPoints
val translation = Translation3D(EuclideanVector3D(origin.x, origin.y, origin.z))
val alignedMesh = mesh.transform(translation.inverse)
If you have metadata or some domain-specific knowledge that hints about the applied rotation, translation and even scaling, you can also build up your own complete transformation by specifying each of the components. Classes exist that then apply the different transformations one after the other, as hinted by the name. In the case of TranslationAfterScalingAfterRotation3D
, the rotation will be applied first, then scaling and then translation.
val translation = Translation3D(EuclideanVector3D(origin.x, origin.y, origin.z))
val rotation = Rotation3D(phi = 0, theta = 0, psi = 0, center = Point3D(0, 0, 0))
val scaling = Scaling3D(1.0)
val transformation = TranslationAfterScalingAfterRotation3D(translation, scaling, rotation)
val alignedMesh = mesh.transform(transformation)
If the meshes have a major directional axis, then the PCA alignment could be useful. Note however that the axis directions might be opposites, so you will need to manually go over and rotate the meshes by 180 degrees around some of the axis. For this example, I’ll show you a few femur bones where there is a clear major axis direction.
def alignmentPrincipalAxises(mesh: TriangleMesh3D): RotationAfterTranslation[_3D] =
val N = 1.0 / mesh.pointSet.numberOfPoints
val center = (mesh.pointSet.points.map(_.toVector).reduce(_ + _) / mesh.pointSet.numberOfPoints).toPoint
val cov = mesh.pointSet.points.foldLeft[SquareMatrix[_3D]](SquareMatrix.zeros)((acc, e) => acc + (e - center).outer(e - center)) * N
val SVD(u, _, _) = breeze.linalg.svd(cov.toBreezeMatrix)
val translation = Translation3D(center.toVector).inverse
val rotation = Rotation3D(SquareMatrix[_3D](u.toArray), Point3D(0, 0, 0)).inverse
RotationAfterTranslation(rotation, translation)
val translation = Translation3D(EuclideanVector3D(100,100,100))
val rotation = Rotation3D(phi = Math.PI/8, theta = Math.PI/4, psi = Math.PI/2, center = Point3D(0, 0, 0))
val transformation = TranslationAfterRotation(translation, rotation)
val targetMesh = mesh.transform(transformation)
val transformationPCA = alignmentPrincipalAxises(targetMesh)
val orientedMesh = targetMesh.transform(transformationPCA)
For the manual annotation, let’s start an instance of Scalismo-UI and load a mesh.
val ui = ScalismoUI()
Then we click the landmarking tool and start clicking landmarks. Either you can make use of the order the landmarks are defined, or as an alternative, I like to give meaningful names to the landmarks. If you have a large dataset, you can code up a semi-automatic method to help automatically change the landmarking names.
After the landmarks are clicked for each and every mesh in the dataset, it is time to align our data using Scalismo.
We first load in the reference landmarks file as these are the ones we would like to align our data to. Then we read in the target mesh and landmark file. We then calculate the transformation using the landmark transformation, apply the transformation to the mesh and the landmarks and save the mesh in a new aligned folder. Of course, feel free to overwrite the original file. Also, a good idea is to visualize the actual aligned mesh output to be sure that all your landmarks were correctly clicked.
val lms = LandmarkIO.readLandmarksJson[_3D](new File(dataDir, "ref_20.json")).get
val meshFile: File = ???
val jsonFile: File = ???
val mesh = MeshIO.readMesh(meshFile).get
val landmarks = LandmarkIO.readLandmarksJson[_3D](jsonFile).get
val transform = LandmarkRegistration.rigid3DLandmarkRegistration(landmarks, lms, Point3D(0,0,0))
val alignedMesh = mesh.transform(transform)
val alignedLms = landmarks.map(lm => lm.copy(point = transform(lm.point)))
MeshIO.writeMesh(alignedMesh, new File("alignedMesh.ply"))
LandmarkIO.writeLandmarksJson[_3D](alignedLms, new File("alignedLms.json"))
In the above example, all the landmarks are paired based on their index in the landmark files. If we instead have names of the individual landmarks, we can instead match the names in the two landmark files:
val commonLmNames = landmarks.map(_.id) intersect lms.map(_.id)
val landmarksPairs = commonLmNames.map(name => (landmarks.find(_.id == name).get.point, lms.find(_.id == name).get.point))
val transform = LandmarkRegistration.rigid3DLandmarkRegistration(landmarksPairs, Point3D(0,0,0))
Finally, let’s perform the alignment using automatic ICP alignment. This implementation is also one of the tutorials provided on the Scalismo website Scalismo Rigid ICP tutorial
def alignmentRigidICP(reference: TriangleMesh3D, target: TriangleMesh3D, numOfPoints: Int, iterations: Int): TriangleMesh3D =
def attributeCorrespondences(movingMesh: TriangleMesh3D, ptIds : Seq[PointId]) : Seq[(Point3D, Point3D)] =
ptIds.map((id : PointId) =>
val pt = movingMesh.pointSet.point(id)
val closestPointOnMesh2 = target.pointSet.findClosestPoint(pt).point
(pt, closestPointOnMesh2)
)
def ICPRigidAlign(moving: TriangleMesh3D, ptIds : Seq[PointId], numberOfIterations : Int) : TriangleMesh3D =
if (numberOfIterations == 0) then
moving
else
val correspondences = attributeCorrespondences(moving, ptIds)
val transform = LandmarkRegistration.rigid3DLandmarkRegistration(correspondences, center = Point(0, 0, 0))
val transformed = moving.transform(transform)
ICPRigidAlign(transformed, ptIds, numberOfIterations - 1)
val ptIds = (0 until reference.pointSet.numberOfPoints by 50).map(i => PointId(i))
ICPRigidAlign(reference, ptIds, iterations)
This method will iteratively estimate the corresponding points between the two meshes, calculate the transformation difference between the meshes and apply the transformation to one of the meshes. This method works well if the orientation of the meshes has already been solved. Often I use this as an additional alignment step after aligning the meshes with a few landmarks.
If one of the meshes is flipped around an axis, the method might end up in the wrong orientation.
In reality, you might often end up using a mixture of the above-mentioned methods. For the vertebras, I have defined a few manually clicked landmarks as also available on the GitHub repository. Depending on the dataset you are working with, a different mixture might be more useful.
And that’s basically all there is to rigidly aligning data. Of course, a lot of more advanced methods exist out there, but for building simple statistical shape models from a relatively small set of data items, I am confident that the provided methods will take you far.
In the next tutorial:
This is the first tutorial in a series on how to create statistical shape models.
In short, a statistical shape model captures the statistical variability in a set of object. Usually the set of objects are from the same shape class, so e.g. we could measure the statistics of hand variability in one model and head variability in another model, but we typically do not combine this into one model.
Let’s start out by looking at an example of vertebrae shapes.
Here we visually inspect an already created statistical shape model. We see that each of the components shows some variability in the geometry. We can also randomly set all the parameters in the model to create a new novel instance.
In my previous video, I already showed some use cases of statistical shape models - so if you are still not sure whether this is for you, have a look at the demo applications linked in the description.
The code for creating such a model is straightforward. So let’s switch to an IDE and get typing. For the coding part, I’ll be using the Scala command line interface, or Scala-CLI for short. I will use the SCALA programming language and the Scalismo library for all shape model related stuff. I’ll use VS code as my development environment.
First install scala-cli
if you haven’t already, use scala-cli —version
to check your system version. It needs to be at least version 1.x
.
Either start a blank project by creating an empty folder or clone the how-to-shape-model repo from github. This repo also contains the data that I’ll be using in this tutorial series.
CD into the folder, now we can execute the scala code from the terminal using scala-cli project.scala src/main.scala
Alternatively, we can open up the project in VS code. We just need to initialize the folder we are in using scala-cli ide-setup .
Then open vs code with code .
The scala version that we use and the external libraries we will be using are specified in the project.scala
file
//> using scala "3.3"
//> using dep "ch.unibas.cs.gravis::scalismo-ui:1.0-RC1"
//> using dep "ch.unibas.cs.gravis::gingr:1.0-RC1"
We will use the newer Scala 3 format and stick to the modern Python-like styling with indentation instead of curly brackets.
Now, let’s load in the data. The data is stored in the data folder folder in .PLY
format. We need to specify one of the meshes as the reference mesh. In The one of the next tutorials, I’ll will go over good practices of choosing this mesh. For now, a random one can be chosen.
val dataDir = new File("data/vertebrae/")
val dataFolder = new File(dataDir, "registered")
val meshes = dataFolder.listFiles().filter(_.getName.endsWith(".ply")).map(MeshIO.readMesh(_).get).toIndexedSeq
val ref = meshes.head
val dataCollection = DataCollection.fromTriangleMesh3DSequence(ref, meshes)
val dataCollectionAligned = DataCollection.gpa(dataCollection) // OPTIONAL: Generalized Procrustes analysis
val ssm = PointDistributionModel.createUsingPCA(dataCollectionAligned)
val ui = ScalismoUI()
ui.show(ssm, "ssm")
And that’s basically all there is to it. Now you might ask why I would need a whole tutorial series to explain this simple method that only takes up a few lines of code. The reason for this is the requirement that these meshes need to be in point correspondence with one another. Before explaining this phenomenon, let’s try to build a model from meshes that are not in point correspondence.
Instead of using the registered folder, let’s instead use the raw folder. And let’s also print out the number of vertices in each mesh:
...
val dataFolder = new File(dir, "raw")
meshes.foreach(_.pointSet.numberOfPoints)
...
If we are lucky to compute the model, we will end up with a model, where the deformations make little to no sense. Alternatively, you will get an error regarding the number of points in the meshes that were used. This will happen in the meshes in the dataset have a different number of vertices than the reference mesh.
By printing out the number of points in each mesh, we clearly see that each mesh has a different number of points.
Let’s have a closer look at our meshes. For this, let’s visualize the same point ID on all the meshes in our dataset. In the dataset that works, we see that the same point ID corresponds to the same anatomical point on all the meshes.
val dataDir = new File("data/vertebrae/")
val dataFolder = new File(dataDir, "aligned")
val ui = ScalismoUI()
val pointId = PointId(1000)
dataFolder.listFiles().filter(_.getName.endsWith(".ply")).foreach { f =>
val mesh = MeshIO.readMesh(f).get
val lm = Seq(Landmark3D(f.getName, mesh.pointSet.point(pointId)))
ui.show(mesh, f.getName)
ui.show(lm, "landmarks")
}
If we do the same for the aligned
meshes, we see that this isn’t the case.
Let’s look at a simple case of 3 hands^{1}. What the shape model contains is essentially the mean deformation and variance for each point in the mesh - and of course the covariance to neighbouring points. So in the case of the hands, we will find the mean hand size as well as the variability at each point. The corresponding points are here visualized with colors, so the same point color is located at the same anatomical point on each hand.
When meshes are extracted from images e.g. by using the marching cubes algorithm or by scanning an object, they will not by default be in point correspondence. They will rarely have the same number of points. For this, we can perform non-rigid registration between a reference mesh and all the meshes in our dataset to obtain this property. This is also often referred to as fitting.
A simple way to explain this is that we choose 1 reference mesh and we then find a deformation field that deforms the reference mesh to approximate each of the meshes in the dataset. As an example, we can overlay two hands and show the deformation needed to warp one hand into the other hand on the 11 given points.
In practise, our shapes will not only be discretized to a few points, but instead consist of thousands or even millions of points to more accurately approximate the surface.
Each of the meshes in the dataset will in other words get the same point structure as the reference mesh, which is why it is important to choose a good reference mesh.
For this tutorial series, we will use a shape dataset of vertices from the vertebrae segmentation challenge at MICCAI (VerSe: Large Scale Vertebrae Segmentation Challenge 2020). For simplicity, I’ve already extracted the mesh from 10 of the segmentation masks and added those to my repository which I have linked in the description.
To run all of the examples shown in this and future tutorials, you just need to execute all the Scala scripts in the prepare_data
folder one by one as mentioned in the readme
file of the repository.
In the following videos we will go over:
And then extra videos, not specifically on building models are
Hands from Shape modeling course ↩