- The researcher initially focused on studying partial differential equations in mathematics, specifically the stability analysis of the Kahn-Hilliard equation.
- The Kahn-Hilliard equation describes phase separation and coarsening in metallic alloys.
- The researcher's interest shifted to image processing when they discovered that the same equation could be used for image restoration, similar to content aware fill in Photoshop.
- The researcher's PhD focused on image restoration, specifically the problem of replacing damaged or occluded parts of an image.
- During their postdoc, the researcher explored inverse imaging problems, where the observed data is not an image directly but a transform of the image, such as in CT scans.
- The researcher discussed the challenges of reconstructing images from limited and noisy data in inverse imaging problems.
- Denoising in image reconstruction algorithms is based on the assumption that edges, or boundaries between different objects, are the most important information in an image.

- The focus of image denoising research is preserving the edges and colors in an image.
- Fourier techniques can remove high frequencies and noise, but they also remove high frequencies that correspond to edges.
- Total variation regularization and median filtering are effective techniques for preserving edge sharpness.
- Deep neural networks are increasingly being used for image denoising, outperforming handcrafted methods in certain scenarios.
- However, handcrafted models still have value in scenarios where neural networks have not been trained on specific types of images.
- Different scanners and acquisition methods can produce variations in image data, which can affect the performance of image denoising algorithms.
- The challenge with neural networks is interpreting the millions of parameters and understanding why certain outcomes occur.
- There is ongoing research to combine the strengths of handcrafted models and neural networks in image denoising.

- The speaker has been working on handcrafted models before exploring neural networks.
- They have been using parameter estimation to learn certain parts of the models.
- The motivation for using machine learning comes from the interpretation it offers.
- They are interested in combining handcrafted models with neural networks, particularly in computer tomography.
- The computational aspect of this approach is done sequentially, not all at once.
- Stochastic optimization is used in machine learning to minimize loss over a finite training set.
- The goal is to approximate the loss over an infinite amount of images to improve generalization.
- The speaker collaborates with academia, particularly in the medical field, to develop practical applications.
- They focus on getting high-resolution images from limited data and tracking dynamic processes.

- The challenge of obtaining high resolution over time is discussed, particularly in the context of reconstructing moving organs in the body.
- Collaboration with people in plant sciences involves monitoring forest health and constituents using aerial photographs, hyperspectral imaging data, and LiDAR measurements.
- Hyperspectral imaging provides information about the material properties of trees based on their spectral component.
- LiDAR measurements create 3D models of trees, which has applications in various fields such as archaeology.
- The potential use of machine learning methods to enhance low-resolution images, such as CCTV footage, is mentioned.
- The speaker expresses interest in working with art conservation, specifically in virtual restoration of frescoes and illuminated manuscripts.
- The concept of virtual restoration is explained, where a digital copy of the artwork is created and manipulated without physically altering the original.
- The speaker highlights an exhibition at the Fitzwilliam Museum that showcased a virtual restoration of an illuminated manuscript by removing overpainting.

- The speaker suggests looking at UCLA's applied math faculty and their online lecture material for cool developments in image processing in the US.
- They mention several classical names in the field, including Stan Osher, Andrea Bertozzi, Malik, Perona, and Stefano Soatto.
- The speaker advises starting with introductory books on mathematical approaches to image processing before delving into more recent research.
- They mention the possibility of catching up or applying the information learned.

- The researcher initially studied partial differential equations, specifically the Kahn-Hilliard equation in mathematics.
- They discovered that the same equation could be used for image restoration in image processing.
- The researcher's PhD focused on replacing damaged or occluded parts of an image.
- They explored inverse imaging problems and discussed the challenges of reconstructing images from limited and noisy data.
- Image denoising research focuses on preserving edges and colors in an image.
- Deep neural networks are increasingly used for image denoising, but handcrafted models still have value in certain scenarios.
- Different scanners and acquisition methods can affect the performance of image denoising algorithms.
- The speaker is interested in combining handcrafted models with neural networks, particularly in computer tomography.
- They collaborate with academia to develop practical applications, such as obtaining high-resolution images from limited data and tracking dynamic processes.
- Collaboration with people in plant sciences involves monitoring forest health using aerial photographs, hyperspectral imaging data, and LiDAR measurements.
- The potential use of machine learning methods to enhance low-resolution images, such as CCTV footage, is mentioned.
- The speaker expresses interest in working with art conservation, specifically in virtual restoration of frescoes and illuminated manuscripts.
- They suggest looking at UCLA's applied math faculty and their online lecture material for developments in image processing in the US.

Carola Schönlieb - http://www.damtp.cam.ac.uk/user/cbs31/Home.html - is an applied mathematician at the University of Cambridge. 

She’s also a Turing Fellow at the Alan Turing Institute - https://www.turing.ac.uk/ - and the head of the Image Analysis group - http://www.damtp.cam.ac.uk/research/cia/ - at Cambridge’s Department of Applied Mathematics and Theoretical Physics - http://www.damtp.cam.ac.uk/ 

In this episode we cover mathematical approaches to image processing.

The YC podcast is hosted by Craig Cannon - https://twitter.com/craigcannon

We ought to start with a little bit of your background.

So what did you start researching and then what are you researching now?

So I started out my research in mathematics in Austria, in Vienna, where I actually didn't look at image processing or imaging at all.

I started out with so-called partial differential equations, which are equations of a function and its derivatives.

So, you take an express change over time or space And they are models for various natural phenomena in physics and biology.

Lots of things are explained by these differential equations.

And my first paper, again, had nothing to do with image processing.

It was actually on the Conhilly equation, which is an equation that describes phase separation and coarsening and alloys in metallic alloys, for instance.

So when you cool them down to a certain temperature, you have a you have a mixture of 2.

And if you cool them down to a certain temperature, they are starting to separate from each other and coarsen out and build these larger areas.

And so there is an equation that models this kind of phenomenon, which is the Kahn-Hilliard equation.

And my first paper was on the stability analysis of a certain type of solutions to this Kahn-Hilliard equation.

Stability analysis meaning that if you perturb your initial condition a little bit, how much is your stationary solution that is when you let time evolve infinitely?

How, you know, when a stationary state is a state where the system is in no change, how much do these stationary states differ from each other when you just perturb the initial condition a little bit?

And this is in the context of creating alloys or building structure alloys for structures?

Well, the purpose is a lot with these differential equations to simulate certain phenomena.

And so if you understand how stable these stationary states are, so if you are in a stationary state and then you perturb the stationary state a little bit, is it going back to the same stationary state?

Or is it going somewhere completely different?

So, you kind of understand how these systems react to perturbations that are naturally occurring because we are in real life and things happen.

Yeah, so it's more an understanding of the physical processes involved in, you know, mixture of alloys for instance or things like that.

And were you at a technical university where you would be like focusing on alloys or this was a personal interest?

So it was just so actually, you know, a lot of applied mathematics on the continent, which is everything else in the UK, basically here in Europe, is applied mathematics very much means that what you're doing is inspired by applications, but eventually you end up with a mathematical problem.

So it was really the driving factor was, well, we were interested in analyzing this equation and there were techniques coming up that are kind of cool.

Yeah, so it was just a kind of intellectual interest in this equation.

was the driving factor for this particular paper.

But then, during writing this paper, research at UCLA, researchers at UCLA, in particular the group of Andrea Bertozzi, used this same equation to do image restoration.

Image restoration meaning you have a digital image and there are parts of this image which are damaged for some reason or which are where you have objects which are occluding some other object of interest that you want to get rid of the occlusion or something like this.

So you have 1 part in the image that you somehow want to replace by something that is suggested by the surrounding area of this region.

So is this similar to like content aware fill in Photoshop?

But this predates the Photoshop development, I assume.

And I mean, also the content aware fill is actually very much based on some of the things that have been initiated by people like Andrea Vatozzi.

So, I mean, the technique is different in what Photoshop is using, but it's still based on research in mathematics, in fact.

It's a differential equation, maybe if you want, if you wish to this more is not a connected equation, but it's a different type of differential equation that is non local is taking patches in images and kind of copy and pasting them into the region that you want to replace.

But anyway, so she used the Kanelian equation to do that.

And that was a kind of eye opening moment.

And then I moved into image processing, still sticking to differential equations at the time.

And actually looking at image restoration.

So at this Photoshop content aware fill type problem.

And yeah, and that and that was basically my PhD.

And during my postdoc, then I moved more and more into what is called inverse imaging problems where what you are observing or what you're measuring in the first place is not an image, like when you take a photo, you know, the digital image is an image.

So, but there are certain applications like in biomedical imaging, where what you're observing is not an image directly, but is some transform of this image, like an image tomography, for instance.

Think about CT, for instance, computer tomography.

What you are, what the CT, what the tomograph is measuring, are projections of your three-dimensional object, which is whatever you have in your body.

And from that, you want to reconstruct the object.

So projections, meaning in the CT sense, a particular sense, which is that you send x-rays through the body.

And what you're measuring, so you're sending them through, what you're measuring at the other end is the attenuation that they feel when they travel through the body, depending on which type of tissues they hit.

And so that's what you're measuring on the other hand, and you can model that by saying, you, what you're measuring is a line, is an integral along the line that the x-ray takes through your body, where you're integrating over the attenuation that it feels.

And so from that, and that is a very old problem, it goes back to Radon.

What you're measuring is not an image, but it's the Radon transform of your image, which are line integrals over the image density that you want to reconstruct, right?

The object consists then where the density is different in different parts of your body and then you can see organs in your body and stuff like that.

And so the likelihood of there to be some amount of it missing that you need to fill or recreate or denoise is much higher than an image.

That's quite obvious because, well, first of all, we are in a finite dimensional world.

So you know, you don't have all possible infinitely many line integrals of your body measured.

And then it's not even, you know, it's not even, that would be still okay if you're measuring as many line integrals as you're corresponding to the resolution of the image that you then want to compute from these line integrals.

But then very often it's not like that because you don't want to, you want a very high resolution image because you want to look at all the details in the body.

But you don't want to measure so many line integrals because you don't want to radiate the patient so much.

You don't want to send tons of x-rays through the patient.

You don't have as much data as you want for a high-resolution image to reconstruct.

And then there is noise because these are measurements

and there is always noise and measurements

and so were you doing denoising work as well at the same time

it's it's it's it's integrated in the reconstruction approach so in in the in the mathematical algorithm that reconstructs an image or the three-dimensional inside of your body from these line measurements, the denoising is integrated into this reconstruction step coming from these line integrals reconstructing a 3D

And so what I know about denoising mostly through audio, like a Fourier transform and that kind of thing.

So with images, it depends what you think is important in the image that will determine how you're going to denoise it.

A very successful assumption that has been made for designing image denoising approaches is and has been and still is that the most important information that visually guides you of what this image is showing you, but also that helps you if you later want to quantify something in the image, are the edges in the image.

Where are boundaries between different objects?

When you think about it, what really makes an impression on you of what this image shows are colors and the boundary between these colors.

And to preserve those and not make them blurry, blurred out, is something that a lot of research in image denoising has gone into.

So image denoising methods, which can preserve edges in an image.

And so the Fourier type techniques are good.

They can smoothen out the noise by taking away the high frequencies.

But they will take away the high frequencies everywhere, which means they will also take away the high frequencies that correspond to edges where the image function is changing rapidly.

This is a very high frequency component of your image.

But this is a component you would like to keep.

So you want to differentiate between the high frequency components in the image which are just noise, and the high frequency components which correspond to these very characteristic features that you want to keep.

And so, you know, there are various techniques, but 1 very successful 1 is total variation regularization, for instance, which is a technique that has been used a lot by people in image denoising to, you know, that models this assumption that you have sharp discontinuities.

Median filtering is a maybe simpler thing to understand or that people might have heard about, which is not exactly total variation denoising, but it's related.

So median filtering instead of Gaussian filtering.

Maybe where Gaussian filtering corresponds to your Fourier taking away the high frequencies type of thing.

You know, it's so funny when I was doing Photoshop of the onion, we were always actually interested in blurring edges because 1 of the most obvious things to spot a Photoshop is a sharp edge and a soft edge in the same photo.

So for instance, like If I were to cut you out and then put you in front of the White House, if the photo has a slight blur, so like the depth of field in the photo is like, say, like a 1.4 aperture, which creates a very, very like shallow depth of field.

But if you're crispy, someone can immediately spot that you were dropped into the photo.

So it was all about blurring the edges to trick someone into thinking that it was in the same photo.

So in your context, these algorithms that will handle the edge sharpness, are they hand-coded or are you using machine learning to create them?

And this is maybe something that is now, you know, more and more being replaced by other things where where image denoising nowadays, I think the best image denoising approaches are actually coming from deep neural networks.

So you know, these handcrafted methods get more and more beaten in terms of performance by some of these neural network approaches.

They get beaten in certain scenarios, though, they get beaten on the type of examples they have seen already, or similar type of images that they have seen already, right?

If you present them with something completely different, right?

If you only train them on photographs of animals or whatever, and then you present them with a CT image or with a CT scan, they will not be able to handle that.

So, that is 1 of the things I think we're still handcrafted models have a certain justification of existence in a sense, because there is not, there is still, you know, although we can do GPU programming and everything, there is still not enough computational power to train a machine to know everything, to learn everything about the world.

And so I think a lot, so while, you know, in certain scenarios, if you know what you want to apply your image denoising approach to

Well, it was like the ImageNet thing from like almost 10 years

But if you, if you want, you know, think about, for instance, 1 big thing in CT, let's say, or in different types of biomedical imaging, let's say MRI, magnetic resonance tomography, The type of image that you get, the resolution, the contrast and everything very much depends on how you do the acquisition.

How many, let's say in the CT case, how many x-rays you have been shooting through the patient.

But also, and that is actually connected to what I just said, also the type of scanner you're using.

Are you using a G or Siemens or Toshiba or whatever?

They have different settings and they have different ways of going from the measurements to an image.

And so, you know, if you train an algorithm, for instance, a neural network on 1 of these scanners, it doesn't mean that it works on images of another scanner.

So they're producing entirely different data.

I thought they were just basically the same tools inside with a different logo.

Well, so This is the other interesting thing.

You might not spot also visually what the difference is.

But this is 1 of the things that also people start, you know, more and more hopefully start to, you know, do some research and understanding this, that even small perturbances that are consistent in small differences that are consistent between the different scanners might contribute to your algorithm then failing.

You know, I don't know if you have seen these adversarial errors where you do a little perturbation, and then all of a sudden, it classifies the image into something completely different.

So, yeah, so I think the really very exciting and for mathematicians, in particular, the exciting opportunity that neural networks are now offering in contrast to these handcrafted models are that they can go beyond just saying, yeah, I want an algorithm that preserves edges.

Which is a very simplistic view of the world.

But on the other hand, that there are lots of unknowns in these algorithms.

On the 1 hand, that mathematicians, I think, should be exploring and try to bring some of the analysis and some of the methodologies that help us to understand why these handcrafted models work, because we we can prove properties about the denoising abilities of these methods of how stable they are, for instance, to perturbations in the images, we know, we know how that works.

So we can prove things about that we have error estimates and things like this.

And to bring those over to neural networks, I think would is very exciting.

But for that, bringing some structure into these neural networks is also important.

And that might, on the other hand, when you think about these neural networks having these 100, millions of parameters that are adapting themselves to the data, maybe In some cases, it would be better to not have a million parameters, but have an intelligent structural way of reducing the search space.

And as such, bring some structure into the problem, which helps you make statements about stability and things like that.

And also statements about what the algorithm is actually doing.

Because, so that is another thing, right?

Because since When you look at these handcrafted models, you have started with hypotheses, right?

You have started with hypotheses of edges are important in images.

And then you've come up with a mathematical algorithm that is exactly doing what you want it to do, right?

Or then you have to make sure that it's actually doing what you want it to do, right?

Or, you know, then you have to make sure that it is actually doing what you want it

Maybe you have to change your model in a certain way.

But you understand why things are happening.

If you have millions of parameters, and then, you know, you train this algorithm to do something, and then you get a parametrization that is a 1 million different parameters, how are you ever going to interpret that?

There are ways, you know, where machine learning people are trying to interpret classification results, for instance.

You have these salient features that you can detect in the image what was important for the classification to do this or this.

And I think, yeah, there are lots of very, very cool opportunities.

And so are you guys working on hand stitching the 2 together at this point?

Like, What's the status of the current research?

Yeah, so there are different people are trying to do different things.

So I can first tell you what I've been doing over the last couple of years.

So the last couple of years, what I've been doing is I've been trying to starting with these more handcrafted models, nothing to do yet with neural networks, I started with the handcrafted models.

And then for certain parts in these models, where I wasn't quite sure about our edges, really the only thing I'm looking for?

For instance, I've tried to parametrize them in a certain way.

But not with a million parameters, but maybe with 10 parameters or something like this.

And then learn these parameters from actual examples that I would like my handcrafted model to spit out.

And this is what we call by level optimization or parameter estimation.

I mean, people have been doing this for a long time.

But now, I think the motivation comes more from, you know, there is a certain interpretation in terms of machine learning that is kind of exciting that where people are, yeah, yeah, more, more, more, more interested in.

And levels of parametrization vary in this context.

But the good thing is you have an handcrafted model in the end, that you still understand,

And that you can still prove things about, you still have guarantees on your solution.

You know, you have guarantees that if you you don't have these adversarial errors that if you perturb a little bit, you get a completely different result.

The other thing is, and this is more blue sky, and this actually goes a little bit against what I said before, which is we have been starting to use deep neural networks for problems in computer tomography, for instance.

And there at the moment, we cannot prove a lot of things.

But we can see some ways of how to combine these more handcrafted models with neural networks in the sense of what you feed them with, for instance, the prior information you feed them with the data, maybe not just the measurements, but maybe also the information that the measurements are actually line integrals of the 3D object that you want to reconstruct.

And doing this in a kind of iterated fashion where you always go back to the fact that actually remember neural network, these are line measurements that I'm feeding you with.

Remember this And then you do another sweep through your network.

But then how does that work in the context of building out a model around, say, like, I mean, I don't even know in an MRI how many images are created or lines are monitored, but like, say you have 10,000 images, but you want to create a combination of a hand-coded algorithm and a machine learning system.

So What I understand you're saying is like you're giving it more data than just like the original source material.

And so how do you do that with a more like at larger scale?

So computationally we are doing this in a sequential manner.

So we're not, so you can do it in different ways, but in a sequential manner means that you're not feeding it to 10,000 images at the same time, but you're doing it bit by bit and you're adapting your objective towards this.

Another thing about computational performance is also, of course, that the optimization that is underlying, but this is not just a problem that we have.

problem that, you know, neural networks have in general is that you do not necessarily need to solve your optimization problem, your training, exactly.

And maybe sometimes, or most of the time, you actually don't want it, want to save it exactly, because you only have a finite amount of training examples.

And so when you think about what these neural networks are doing, they're trying to minimize a loss over the training examples that you have.

But this loss is only an approximation of many, many, many, many more images that you want your neural network to work for.

And so very often you do not want to solve that exactly.

You don't want to minimize your loss exactly for this training set.

And so there are different types of optimization methods that people are using.

But the main thing in machine learning is stochastic optimization.

So you don't minimize, you know, exactly for all the variables that you have, but you randomly pick a certain amount in every sweep through the network that you're optimizing for and then you randomly change which ones you're optimizing the next sweep and so on.

And just so I understand, minimizing lost, why don't you want to do that?

So what you're minimizing, if you're so if you're so the loss, let's say could be the least squares error.

Between let's, let's go back to denoising.

Let's say you want you want to train your neural network to optimally denoise images by saying, for this training set where I have both noisy and clean images, I want that if I sum over the difference between the denoised image, so you feed your neural network with a noisy image, it gives you a denoised image, you want that this denoised image is closest in the least square sense to the clean image that you know in this case, because you have a training set, you have a label, you have a true label for this noisy image, which the label in this case is your ground truth image.

And you want your denoising method, which is this neural network to produce denoised images such that all of them are, in the least square sense, closest to the original label, to the ground label, which is the clean image.

you want that to work over all the images in the training set.

See all Y Combinator transcripts on Youtube

Mathematical Approaches to Image Processing with Carola Schönlieb