- The video discusses a preprint from Facebook about unsupervised topic segmentation of meetings using BERT embeddings.
- The paper aims to divide a piece of text, such as meeting transcripts, into multiple blocks for different topics.
- The authors propose an unsupervised approach due to the difficulty of annotating meetings.
- The method involves using BERT or sentence BERT to create vector representations of blocks or sentences.
- Sliding window and similarity matching techniques are used to identify topic boundaries.
- The authors evaluate the approach using ground truth data and show both correct and incorrect predictions.
- The video encourages viewers to like, subscribe, and share the content.

#bert #research #nlp
⏩ Abstract: Topic segmentation of meetings is the task of dividing multi-person meeting transcripts into topic blocks. Supervised approaches to the problem have proven intractable due to the difficulties in collecting and accurately annotating large datasets. In this paper, we show how previous unsupervised topic segmentation methods can be improved using pre-trained neural architectures. We introduce an unsupervised approach based on BERT embeddings that achieves a 15.5% reduction in error rate over existing unsupervised approaches applied to two popular datasets for meeting transcripts.
 
Sign-up for Email Subscription - https://forms.gle/duSwrYAGw6zUhoGf9

⏩ OUTLINE:
0:00 - Understanding Problem Statement
02:05 - Example of Meeting Transcript
02:42 - Topic Segmentation Method
04:06 - Sentence Representation Methods(RoBERTa and Sentence-BERT)
06:20 - Segmentation Scheme
07:43 - Results

⏩ Paper Title: Unsupervised Topic Segmentation of Meetings with BERT Embeddings
⏩ Paper: https://arxiv.org/abs/2106.12978
⏩ Code: https://github.com/gdamaskinos/unsupervised_topic_segmentation
⏩ Author: Alessandro Solbiati, Kevin Heffernan, Georgios Damaskinos, Shivani Poddar, Shubham Modi, Jacques Cali
⏩ Organisation: Facebook

Please feel free to share out the content and subscribe to my channel :)
⏩ Subscribe - https://youtube.com/channel/UCoz8NrwgL7U9535VNc0mRPA?sub_confirmation=1

BERT use-cases in NLP: https://www.youtube.com/watch?v=uhnKsGDyhEg&list=PLsAqq9lZFOtX-WN8lldIOI7p-p0lBzjtY

**********************************************
If you want to support me financially which is totally optional and voluntary ❤️ 
You can consider buying me chai ( because I don't drink coffee :) ) at https://www.buymeacoffee.com/TechvizCoffee
❤️ Support using Paypal - https://www.paypal.com/paypalme/TechVizDataScience

**********************************************
⏩ Youtube - https://www.youtube.com/c/TechVizTheDataScienceGuy
⏩ LinkedIn - https://linkedin.com/in/prakhar21
⏩ Medium - https://medium.com/@prakhar.mishra
⏩ GitHub - https://github.com/prakhar21
⏩ Twitter - https://twitter.com/rattller
*********************************************

Tools I use for making videos :)
⏩ iPad - https://tinyurl.com/y39p6pwc
⏩ Apple Pencil - https://tinyurl.com/y5rk8txn
⏩ GoodNotes - https://tinyurl.com/y627cfsa

#techviz #datascienceguy #nlproc #machinelearning #unsupervisedlearning
About Me:
I am Prakhar Mishra and this channel is my passion project. I am currently pursuing my MS (by research) in Data Science. I have an industry work-ex of 3 years in the field of Data Science and Machine Learning with a particular focus on Natural Language Processing (NLP).

So in this video, we'll be going through this preprint from Facebook, which is titled as unsupervised topic segmentation of meetings with BERT embeddings.

But before moving forward, if you're new to this channel, and still wondering what am I doing over here.

I'm currently a master's student at IIIT Bangalore.

Now on this channel, I've majorly tried to explain research papers in the machine learning domain.

So yeah, definitely if that excites you, or any of your friends, make sure to subscribe to the channel and click the bell icon so that you don't miss out on any of the notifications.

So about this paper at very high level what the authors discuss is about dividing a piece of text into multiple blocks for different topics.

Let's call this as T1 and let's call this as T2.

And they particularly experiment with the transcripts of meetings, which again is kind of a prime use case because due to pandemic people have started extensively using zoom Google meets and all and let's say if you record that meeting and get a transcript for that and let's say want To summarize to a stakeholder in terms of what the discussions were, then clearly this works easily fits in that setting.

Because for a recording of let's say 1 hour, you will get a big transcript where you'll have multiple speakers, so to say, ABC, then A would again speak all of that will happen.

Now the idea is can you divide these continuous conversations or dialogues into multiple topics by identifying these boundary points as close as possible.

However, if we talk about the data in which they are operating, which is like the transcripts, which again is an output from the ASR system, It is expected that the output will have a lot of noise in terms of having proper punctuations, proper pauses, and also some confusion in terms of who speaks at that moment and what ASR transcribes.

So all those nuances in the data make it even harder to solve.

So in this paper, authors propose an unsupervised way of performing this task.

They didn't go with supervised because, again, it's a pain to kind of annotate these meetings in the first place.

So yeah, that's the entire idea and the problem statement that the authors are trying to solve.

Let's move forward and see the exact method.

So if you see this figure, this again is kind of a conversation that has happened in let's say 1 of the meetings, where you have speaker identification that has already been tagged.

So this work tries to create this column what you see.

Where in where you have the value of 1 indicates a topic change.

So since all of this is 0, we are talking about just 1 topic.

And that's what they have manually labeled as well over here.

So now the captions are the input to your model.

And this Boolean vector of topic change is what you want to predict.

So that's what they have mentioned over here.

If you talk about the input, it is nothing but the list of M utterances from S1 to Sm and let's consider the ground truth topics are n that get propagated in these M instances.

So for example if I start my conversation as saying hello and everybody says hello over there and then let's say we start off by talking about a problem statement And let's say after that the solution.

So if each of them had 4 utterances, so we have a total of 12 instances, which is the value of m.

And since clearly you can see like the first is the greeting segment, then you have the problem statement definition, then you have the solution.

So these are 3 topics over which all these 12 instances propagate.

Now each of the topic is defined as a block, we have a starting index of that utterance and the ending index of that utterance.

So which is what they have written over here.

And finally, what we need is a label sequence y, which is y1 to ym, which means for every utterance we need whether it's a 0 or 1, where a 1 would indicate a shift of topic from the previous topic, and 0 indicates the absorption of this utterance in the ongoing topic.

So yeah, that's about how you formally define this entire statement.

So as a part of the solution, authors define a pretty simple approach, which is based on sliding window and then doing a similarity matching of those windows.

So the first thing that we need is a vector representation of a block or for a sentence.

So they play around with the BERT architecture and also try the sentence BERT.

So with BERT they particularly make use of Roberta and extract the fixed length vector via max pooling of the second last layer in the entire architecture.

So this way you get a representation for the input block or a sentence.

The way this would work is let's say you pick a pre-trained Roberta model and do not fine-tune it let's say.

You have a sentence s that has let's say 4 words you pass it through the model m which is roboto encoder you get some 768 dimension representation for each of these words that's what they have written as the output you get is number of words into the hidden representation for every word So in this case this would become 3 x 768 and then they perform max pooling to get a fixed length vector for the entire sentence which is let's say if each of this vector is for example 4 length vector then for every position you take the max across all of these words and that's the first entry in the index for the fixed length vector that you have and similarly you'll have for second position third position and fourth positions so now this is the vector that you use as the representation for the sentence And the second model that they try out is sentence BERT, which was trained on natural language inference task, where you give 2 sentences, s1 and s2.

And the task is to predict whether these 2 sentences contradict each other or not, or if they don't hold any relation which is neutral, or if the first sentence entails the other 1 or not.

So this can be treated as a 3 class classification, you can use a softmax layer at the very last for getting this probability distribution.

But the way they encode each of these sentence and learn the representation is to pass them through the BERT model, each of them independently, where the weights are shared.

And then you do a pooling over here which is mean pooling.

And finally once you get the representation for both of them, you join them by a certain function f and then pass it through a softmax layer.

And in the original paper authors have played around with mean pooling, max pooling and even tried out the CLS vector embedding but the mean pooling worked the best.

So yeah these are 2 techniques that the authors try to get the representation for a block or for a sentence.

And once we have the representation, the next idea is to create blocks or window over these utterances.

So as a first step, we start by computing word embeddings for every utterance in the meeting script.

Then we divide this entire corpus into blocks of case consecutive utterances and generate the embedding for the entire block now by performing max pooling over the embeddings that we get for every utterance in that block.

So max pooling is typically robust to noisy speech data because with that you kind of pick the most representative aspect from any of the axis in that feature vector because of which any noise or unclear segment which is not representative enough is left out.

So once we get the embedding representation for every block The next task is to compute cosine similarity between the adjacent blocks ri and ri plus 1.

So that's like if you have this block b1 and this is let's say block b2, you compute similarity between these 2.

And finally to produce a value of 0 or 1, they check if the semantic similarity calculated is lower than a certain threshold.

So if the similarity is less than mean minus variance, then you put it as 1 or else it's 0.

Or visually you can think of if the similarity lies, let's say, towards the tail of this distribution, then you consider those blocks to be pretty different.

Rather the similarity is pretty high and close towards mean, you assign the value to be 0.

So yeah, that's the entire idea and the approach what the authors discuss.

So if you see this figure, So this red color is the ground truth where the topic has diverged and these blue ones are what the model has predicted.

So we can see right the model was able to predict just 1 boundary rather the original segmentation had a lot of topic shifts whereas this is correct and this is also more or less correct and similarly for the second 1 if you see this is 1 of the bad examples that they have mentioned where the model has kind of not performed really good so this is only correct whereas this is really off compared to what the diversions were and clearly no markers were made by the model for this 1.

So here 3 is like false positive because you have said you have a marker but it was not in the ground truth data.

Fifth is false negative because you have said that you don't have any marker but it was present in the original meeting Whereas 1, 2 and 4 can be seen as true positives.

Okay, so yeah, I think now we're done with the paper.

So if you like such content, make sure to hit that like button and subscribe to the channel.

Also share it across with the friends to whosoever is interested in such content.


Speaker 0: Hi there, welcome back to the new video. So in this video, we'll be going through this preprint from Facebook, which is titled as unsupervised topic segmentation of meetings with BERT embeddings. But before moving forward, if you're new to this channel, and still wondering what am I doing over here. So my name is Prakhar. I'm currently a master's student at IIIT Bangalore.

Now on this channel, I've majorly tried to explain research papers in the machine learning domain. So yeah, definitely if that excites you, or any of your friends, make sure to subscribe to the channel and click the bell icon so that you don't miss out on any of the notifications. So cool. So about this paper at very high level what the authors discuss is about dividing a piece of text into multiple blocks for different topics. Let's call this as T1 and let's call this as T2.

And they particularly experiment with the transcripts of meetings, which again is kind of a prime use case because due to pandemic people have started extensively using zoom Google meets and all and let's say if you record that meeting and get a transcript for that and let's say want To summarize to a stakeholder in terms of what the discussions were, then clearly this works easily fits in that setting. Because for a recording of let's say 1 hour, you will get a big transcript where you'll have multiple speakers, so to say, ABC, then A would again speak all of that will happen. Now the idea is can you divide these continuous conversations or dialogues into multiple topics by identifying these boundary points as close as possible. However, if we talk about the data in which they are operating, which is like the transcripts, which again is an output from the ASR system, It is expected that the output will have a lot of noise in terms of having proper punctuations, proper pauses, and also some confusion in terms of who speaks at that moment and what ASR transcribes. So all those nuances in the data make it even harder to solve.

So in this paper, authors propose an unsupervised way of performing this task. They didn't go with supervised because, again, it's a pain to kind of annotate these meetings in the first place. So yeah, that's the entire idea and the problem statement that the authors are trying to solve. Let's move forward and see the exact method. So if you see this figure, this again is kind of a conversation that has happened in let's say 1 of the meetings, where you have speaker identification that has already been tagged.

So this work tries to create this column what you see. Where in where you have the value of 1 indicates a topic change. So since all of this is 0, we are talking about just 1 topic. And then we have the topic shift. And then we talk about another topic.

And that's what they have manually labeled as well over here. So now the captions are the input to your model. And this Boolean vector of topic change is what you want to predict. Cool. Let's move forward.

So that's what they have mentioned over here. If you talk about the input, it is nothing but the list of M utterances from S1 to Sm and let's consider the ground truth topics are n that get propagated in these M instances. So for example if I start my conversation as saying hello and everybody says hello over there and then let's say we start off by talking about a problem statement And let's say after that the solution. So if each of them had 4 utterances, so we have a total of 12 instances, which is the value of m. And since clearly you can see like the first is the greeting segment, then you have the problem statement definition, then you have the solution.

So these are 3 topics over which all these 12 instances propagate. So here the value of n is now 3. Now each of the topic is defined as a block, we have a starting index of that utterance and the ending index of that utterance. So which is what they have written over here. And finally, what we need is a label sequence y, which is y1 to ym, which means for every utterance we need whether it's a 0 or 1, where a 1 would indicate a shift of topic from the previous topic, and 0 indicates the absorption of this utterance in the ongoing topic.

So yeah, that's about how you formally define this entire statement. So as a part of the solution, authors define a pretty simple approach, which is based on sliding window and then doing a similarity matching of those windows. So the first thing that we need is a vector representation of a block or for a sentence. So they play around with the BERT architecture and also try the sentence BERT. So with BERT they particularly make use of Roberta and extract the fixed length vector via max pooling of the second last layer in the entire architecture.

So this way you get a representation for the input block or a sentence. The way this would work is let's say you pick a pre-trained Roberta model and do not fine-tune it let's say. You have a sentence s that has let's say 4 words you pass it through the model m which is roboto encoder you get some 768 dimension representation for each of these words that's what they have written as the output you get is number of words into the hidden representation for every word So in this case this would become 3 x 768 and then they perform max pooling to get a fixed length vector for the entire sentence which is let's say if each of this vector is for example 4 length vector then for every position you take the max across all of these words and that's the first entry in the index for the fixed length vector that you have and similarly you'll have for second position third position and fourth positions so now this is the vector that you use as the representation for the sentence And the second model that they try out is sentence BERT, which was trained on natural language inference task, where you give 2 sentences, s1 and s2.

And the task is to predict whether these 2 sentences contradict each other or not, or if they don't hold any relation which is neutral, or if the first sentence entails the other 1 or not. So this can be treated as a 3 class classification, you can use a softmax layer at the very last for getting this probability distribution. But the way they encode each of these sentence and learn the representation is to pass them through the BERT model, each of them independently, where the weights are shared. And then you do a pooling over here which is mean pooling. And finally once you get the representation for both of them, you join them by a certain function f and then pass it through a softmax layer.

And in the original paper authors have played around with mean pooling, max pooling and even tried out the CLS vector embedding but the mean pooling worked the best. So yeah these are 2 techniques that the authors try to get the representation for a block or for a sentence. And once we have the representation, the next idea is to create blocks or window over these utterances. So as a first step, we start by computing word embeddings for every utterance in the meeting script. Then we divide this entire corpus into blocks of case consecutive utterances and generate the embedding for the entire block now by performing max pooling over the embeddings that we get for every utterance in that block.

So max pooling is typically robust to noisy speech data because with that you kind of pick the most representative aspect from any of the axis in that feature vector because of which any noise or unclear segment which is not representative enough is left out. So once we get the embedding representation for every block The next task is to compute cosine similarity between the adjacent blocks ri and ri plus 1. So that's like if you have this block b1 and this is let's say block b2, you compute similarity between these 2. And finally to produce a value of 0 or 1, they check if the semantic similarity calculated is lower than a certain threshold. So if the similarity is less than mean minus variance, then you put it as 1 or else it's 0.

Or visually you can think of if the similarity lies, let's say, towards the tail of this distribution, then you consider those blocks to be pretty different. Rather the similarity is pretty high and close towards mean, you assign the value to be 0. So yeah, that's the entire idea and the approach what the authors discuss. So if you see this figure, So this red color is the ground truth where the topic has diverged and these blue ones are what the model has predicted. So we can see right the model was able to predict just 1 boundary rather the original segmentation had a lot of topic shifts whereas this is correct and this is also more or less correct and similarly for the second 1 if you see this is 1 of the bad examples that they have mentioned where the model has kind of not performed really good so this is only correct whereas this is really off compared to what the diversions were and clearly no markers were made by the model for this 1.

So here 3 is like false positive because you have said you have a marker but it was not in the ground truth data. Fifth is false negative because you have said that you don't have any marker but it was present in the original meeting Whereas 1, 2 and 4 can be seen as true positives. Okay, so yeah, I think now we're done with the paper. So if you like such content, make sure to hit that like button and subscribe to the channel. Also share it across with the friends to whosoever is interested in such content.

I'll meet you in the next 1. Bye bye and take care.

See all TechViz - The Data Science Guy transcripts on Youtube

Unsupervised Topic Segmentation of Meetings with BERT Embeddings (Research Paper Walkthrough)