- The speaker discusses the topic of continuous deployment for data science and mentions recent news about AI.
- They highlight the need for public data in AI and data science, but also the potential risks of releasing sensitive information.
- The speaker shares their experience with AI models and how they sometimes struggle to provide meaningful insights or generalize well.
- They discuss the issue of discrimination in AI and the difficulty in determining when it becomes ethically problematic.
- The speaker raises concerns about the impact of AI on content sharing and search engine optimization.
- They suggest that AI can be useful for educational purposes, providing guidance and recommendations, and assisting in peer programming.
- However, they believe that AI may be less useful in generating novel insights from data.

- Finding interesting and new insights in data is rare and cannot be generalized
- AI models may drown out minorities and not include them in summarizations
- Data science is about triggering strange and interesting effects in data
- Proprietary and private data may not be shared for AI analysis
- Machine general intelligence may not reach human levels, but side applications can augment human intelligence
- Human involvement is still necessary for finding new insights and making use of data
- The speaker recommends using AI for marketing purposes but acknowledges that it can go in weird directions.

Speaker: Michael Berthold

The world is abuzz with controversy over the use of AI. Hear a sober take on how AI is useful for sanity checking or for skipping the annoying 60% so you don’t have to start from scratch--but less useful in (many) other cases.

I actually wanted to talk about something different today.

Talk a little bit more about continuous deployment for data science.

But then in the last couple of months, there was a few minor little things happened in the news around AI and stuff.

And I thought it would make sense to at least comment on that 1.

And I'm going to do this with a gigantic bold disclaimer.

I have no idea what's going to happen in the future.

So these are very random thoughts, some of it inspired by conversations with colleagues.

But maybe there's a bit or a piece in there that makes you think a little bit further as well.

I think we are for sure in for a very interesting journey.

Not quite sure where that's going to end.

So first of all, AI and data science, JET, GPT, OpenAI, whatever you want to call those things, Everybody now I hear that the madman in San Francisco just bought what, 100,000 GPU cards and will now build his own AI as well.

But anyway, it does require, and That's 1 of the big points, lots of public data.

That coming to data science, you think, okay, so if it requires lots of public data and also I need to upload my data to kind of be fused with that public data so I get interesting insights, You wonder if that's really what you want to do, right?

And there were already a couple of instances where people were releasing quite sensitive information by asking innocent questions with that sensitive information through public sources.

The fine print says that every query that you enter will be analyzed, not will be, but can be analyzed by humans.

Not only will it be looked at to analyze what's happening by other people, but it will also be used to retrain that AI system.

So you may not really want to do that if it's reasonably sensitive, interesting data that you have inside your organization.

The other big piece is, of course, what do we actually want to get from an AI when you feed in data?

You don't want to start a conversation with that data.

You want to get insights into the data, right?

I was playing a little bit and I was uploading the Iris data first.

And it said, oh, this is the well-known Iris data and it gave me a full analysis of the Iris data.

So I re-normalized the iris data and I changed the labels.

So I wouldn't even call it anonymization, but it wasn't quite as obviously the iris data anymore.

And it said, based on the information provided, you essentially gave me nothing.

Just said, okay, there are apparently 5 columns in the data.

So I asked, can you also build a decision tree?

And then it ended up explaining to me what a decision tree is.

And I said, yeah, great, but can you actually build it on the data I just shared?

And then it started to speculate around the data and went off in a totally different direction.

It started talking, if you can't read it down here, it said there's a car purchase column in the data.

There is no car purchasing column in the data.

So it was actually generalizing into something, what's a typical conversation around data, and a typical conversation around data seems to be ending there.

To be fair, if you continue the conversation a little bit, it's actually, I think it's wrong anyway to talk about a conversation.

All you give it is you provide prompts, your reactions.

But if you continue providing these prompts and you dive a little bit deeper, it very very honestly says, I'm a language model.

I can't really find patterns in data, right?

But maybe there's something other out there that can do that.

And I think that's a really interesting part.

So if you now think about all of these people that are building programs, Python for instance, by interacting with JetGPT, they are building up on the same type of content, right?

But if I upload data, I'm not interested in JetGPT, that AI providing more data.

I'm actually asking it to do something more with it, abstract from it, learn something interesting from it.

So that's not the same as from some text extrapolate more text.

Then there's of course all of this topic of generalization, hallucination, what people call it, which I find interesting.

Fundamentally what it does, it learns from existing data points, right?

But what does generalization really mean?

It essentially says find regularities and then you have data points here and then you find something in the middle.

It looks very similar, so I'm going to find something similar to that 1.

That's really what ultimately generalization is.

If you try that with KNIME, it was actually a really interesting conversation.

That's not really a conversation, but anyway.

And it gave us beautiful, if you read this text, it reads totally meaningful.

So if you want to do this, you should use the interactive authentication node instead of the Google authentication node and then everything will be fine.

The only problem is we don't have that node.

The interactive authentication node doesn't exist.

So this could be, it is clearly a problem with the analytics platform.

If you look at that 1, you say, yeah, we should probably build that.

We could prioritize the development of this node so it gets done quicker so that JetGPT is right.

Or we have to admit that we have a lot of nodes in KNIME, but it's not from a generalization perspective.

Every now and then there are unexpected gaps, and these unexpected gaps come from us having to prioritize work over other things.

So Generalization, the ability to have a model that generalizes well, assumes that the data that you see is reasonably well generalized, so that you can actually fill in meaningful gaps.

Reality doesn't always generalize well if you just look at the data.

And 1 big problem of that that we talk about in other aspects as well, this generalization problem, also comes up when it's about discrimination.

Because fundamentally what you're doing is you're learning from past data points, You learn something in a way you could argue that the poor interactive authentication node got discriminated.

Or some other node that is also doing authentication got discriminated because it was not mentioned.

That's fundamentally what data mining always does.

We are trying to segment, we are trying to group, we are trying to differentiate, we are trying to discriminate between different data points.

In many ways that's not a discrimination that we consider bad because we're looking at temperature or some other totally fine attributes.

But you don't really know when it is bad.

And finding out from an AI response when its response is actually based on data where you would consider that as ethically a problem, then that's a lot harder to do.

And then there's of course finally the sharing part.

I think that that was something I got out of a conversation with a colleague who said 1 of the problems that he foresees with that type of AI is people will become a lot more careful about publicly sharing interesting content.

Because when I post blog posts and they are reasonably interesting, not all of my blog posts are, but some I hope are, then I hope that I get a little bit of credit for that.

When you build up a bit of personal reputation, yeah, the stuff that Michael writes, every now and then it's useful.

Lots of weird stuff, but sometimes useful stuff.

So the question is, will people still share as openly as they were before?

Because the chances of people actually...

Search engines right now point you to interesting content.

AI summarises interesting content or generalises from interesting content.

So I have a hunch, or that person is postulated, that we're going to be back to more locked, more non-public content, because people don't want that.

Or at least you exclude that from training material for AI models.

And then there was this bold statement that SEO is that, because search engine optimisation, getting search engines to point to your page is going to change into something.

Getting AI to summarize in such a way that people will go to your page to finally find the content, that's going to be really rough.

So I think that's an interesting little side effect here as well.

So some early thoughts for me on the AI for data science.

I think it's likely very useful for educational purposes.

It will point you every now and then to things that don't yet exist.

For guidance, why don't you try this out next?

Right now, I mean, KNIME Analytics Platform has this built-in recommendation engine, marketing likes to call it AI, which really only looks at pairs of nodes, right, and then says, okay, it's very likely that you want to use this node.

We can do a lot more with AI models to kind of help people to say, hey, why didn't you look at this or try out this somewhere else?

I think it's going to be super useful in many disciplines, not just data science, to skip the boring 60% where you just say, OK, can you get me a workflow that joins 2 Excel files and an Oracle database and does this and provides a scatter plot?

You have a workflow and you say, hey, it doesn't quite do what I'm interested in doing, but now I start from something instead of starting from scratch.

It's not going to give you the fully developed solution.

And the other piece is, think peer programming.

Because you have 1 of the 2 developers is always kind of looking over the shoulder of the other 1 and make sure they catch stupid mistakes early.

Having an AI where you essentially say, do you find something fishy here?

And the AI just saying, this looks weird and this I've never seen before.

Maybe it's right, but at least it kind of points you to the instances where you might want to double check what's going on.

Likely less useful in my mind for novel insights into data.

Because you're really finding something interesting, new in the data, is very likely something that nobody has seen before in similar types of data.

So it's not something that you can generalize from because they are by definition very rare instances, very rare insights.

For highly sparse information data, if at the end of the day 1 of the big worries of all of these AI models that are summarizing is also that the minorities get drowned out.

If company A has 10,000 more mentions and stuff in articles talking about it than company B, chances that AI summarises and will also mention B are much, much smaller.

And then to me, I don't think this idea of hypothesis generation, when I was still at the university, we had this 1 talk about sparking insights.

Data science ultimately in many, many instances is about triggering this, that's weird effect.

Where you look at your data and say, that's weird, that can't be right.

And then you dig a little bit deeper and then you start realizing, ooh, that's a strange little interesting effect.

That of course, in 95% of those cases you'll find out that, oh, somebody really screwed up.

But the remaining 5% triggers a new interesting discovery, a new type of hypothesis.

So I think it's not going to be that great.

So I was relatively, and then of course for the analysis, I mentioned that proprietary private data, competitively super interesting data.

Why do we have representatives of pharma companies here?

I'm not quite sure they would upload all of their screening data to a public AI to get some interesting insights.

That type of stuff you need to do in isolation, maybe with a local copy, there may be ways around that.

But if I'm doing that with my private copy, nobody else is going to benefit from that either, so my AI doesn't keep learning on it.

So it's going to be isolated little discoveries.

So I was a little bit delusional, so I looked around, maybe other people have a stronger opinion about that or a better 1.

Here's a beautiful quote that I came across where somebody said, in 3 to 8 years, we will have a machine with the general intelligence of an average human being.

That's a quote from Marvin Minsky from 1970.

So I'm, maybe I'm too old, I have too many déjà vus, but I'm carefully optimistic, but I'm also skeptical.

I don't think we'll be running into this human intelligence type of thing.

I see us much more working with much smarter side applications that augment human intelligence.

They will help us to do a lot of stuff, the mundane tasks a lot easier, faster, and maybe even get a bit more inspiration what to try out.

But the last step in finding new insights and data and making something useful with your data is still going to be the human.

And that's when I said, this stuff does work.

And then, of course, I clicked let it revise a couple more times and it went into completely weird directions.


Speaker 0: I actually wanted to talk about something different today. Talk a little bit more about continuous deployment for data science. But then in the last couple of months, there was a few minor little things happened in the news around AI and stuff. And I thought it would make sense to at least comment on that 1. And I'm going to do this with a gigantic bold disclaimer.

I have no idea what's going to happen in the future. So these are very random thoughts, some of it inspired by conversations with colleagues. But maybe there's a bit or a piece in there that makes you think a little bit further as well. I think we are for sure in for a very interesting journey. Not quite sure where that's going to end.

So first of all, AI and data science, JET, GPT, OpenAI, whatever you want to call those things, Everybody now I hear that the madman in San Francisco just bought what, 100,000 GPU cards and will now build his own AI as well. Everybody's trying to play in this game. But anyway, it does require, and That's 1 of the big points, lots of public data. That coming to data science, you think, okay, so if it requires lots of public data and also I need to upload my data to kind of be fused with that public data so I get interesting insights, You wonder if that's really what you want to do, right? And there were already a couple of instances where people were releasing quite sensitive information by asking innocent questions with that sensitive information through public sources.

So now read the fine print, right? The fine print says that every query that you enter will be analyzed, not will be, but can be analyzed by humans. So this will be spilled. Not only will it be looked at to analyze what's happening by other people, but it will also be used to retrain that AI system. So you may not really want to do that if it's reasonably sensitive, interesting data that you have inside your organization.

The other big piece is, of course, what do we actually want to get from an AI when you feed in data? You don't want to start a conversation with that data. You want to get insights into the data, right? So I tried that over the weekend. I was playing a little bit and I was uploading the Iris data first.

And it said, oh, this is the well-known Iris data and it gave me a full analysis of the Iris data. And I said, that's a bit fishy. So I re-normalized the iris data and I changed the labels. So I wouldn't even call it anonymization, but it wasn't quite as obviously the iris data anymore. And suddenly it was very confused.

And it said, based on the information provided, you essentially gave me nothing. Just said, okay, there are apparently 5 columns in the data. So I asked, can you also build a decision tree? And then it ended up explaining to me what a decision tree is. And I said, yeah, great, but can you actually build it on the data I just shared?

And then it started to speculate around the data and went off in a totally different direction. It started talking, if you can't read it down here, it said there's a car purchase column in the data. There is no car purchasing column in the data. So it was actually generalizing into something, what's a typical conversation around data, and a typical conversation around data seems to be ending there. To be fair, if you continue the conversation a little bit, it's actually, I think it's wrong anyway to talk about a conversation.

All you give it is you provide prompts, your reactions. It's not really a conversation. But if you continue providing these prompts and you dive a little bit deeper, it very very honestly says, I'm a language model. I can't really find patterns in data, right? But maybe there's something other out there that can do that.

And I think that's a really interesting part. So if you now think about all of these people that are building programs, Python for instance, by interacting with JetGPT, they are building up on the same type of content, right? They're refining that type of content. But if I upload data, I'm not interested in JetGPT, that AI providing more data. I'm actually asking it to do something more with it, abstract from it, learn something interesting from it.

So that's not the same as from some text extrapolate more text. Then there's of course all of this topic of generalization, hallucination, what people call it, which I find interesting. Fundamentally what it does, it learns from existing data points, right? And then it generalizes from that. But what does generalization really mean?

It essentially says find regularities and then you have data points here and then you find something in the middle. It looks very similar, so I'm going to find something similar to that 1. That's really what ultimately generalization is. If you try that with KNIME, it was actually a really interesting conversation. That's not really a conversation, but anyway.

And it gave us beautiful, if you read this text, it reads totally meaningful. It comes straight out of a manual. So if you want to do this, you should use the interactive authentication node instead of the Google authentication node and then everything will be fine. Sounds cool, right? The only problem is we don't have that node.

Right? The interactive authentication node doesn't exist. So this could be, it is clearly a problem with the analytics platform. If you look at that 1, you say, yeah, we should probably build that. And there's a ticket to build that.

So we could do 2 things. We could prioritize the development of this node so it gets done quicker so that JetGPT is right. Or we have to admit that we have a lot of nodes in KNIME, but it's not from a generalization perspective. Every now and then there are unexpected gaps, and these unexpected gaps come from us having to prioritize work over other things. So Generalization, the ability to have a model that generalizes well, assumes that the data that you see is reasonably well generalized, so that you can actually fill in meaningful gaps.

Reality doesn't always generalize well if you just look at the data. And 1 big problem of that that we talk about in other aspects as well, this generalization problem, also comes up when it's about discrimination. Because fundamentally what you're doing is you're learning from past data points, You learn something in a way you could argue that the poor interactive authentication node got discriminated. Because it was outlined here. Or some other node that is also doing authentication got discriminated because it was not mentioned.

So a lot of discrimination isn't bad. That's fundamentally what data mining always does. We are trying to segment, we are trying to group, we are trying to differentiate, we are trying to discriminate between different data points. In many ways that's not a discrimination that we consider bad because we're looking at temperature or some other totally fine attributes. But you don't really know when it is bad.

And finding out from an AI response when its response is actually based on data where you would consider that as ethically a problem, then that's a lot harder to do. And then there's of course finally the sharing part. I think that that was something I got out of a conversation with a colleague who said 1 of the problems that he foresees with that type of AI is people will become a lot more careful about publicly sharing interesting content. Because when I post blog posts and they are reasonably interesting, not all of my blog posts are, but some I hope are, then I hope that I get a little bit of credit for that. When you build up a bit of personal reputation, yeah, the stuff that Michael writes, every now and then it's useful.

Lots of weird stuff, but sometimes useful stuff. AI tentatively doesn't give credit. So the question is, will people still share as openly as they were before? Because the chances of people actually... Search engines right now point you to interesting content.

AI summarises interesting content or generalises from interesting content. So I have a hunch, or that person is postulated, that we're going to be back to more locked, more non-public content, because people don't want that. Or at least you exclude that from training material for AI models. And then there was this bold statement that SEO is that, because search engine optimisation, getting search engines to point to your page is going to change into something. Getting AI to summarize in such a way that people will go to your page to finally find the content, that's going to be really rough.

So I think that's an interesting little side effect here as well. So some early thoughts for me on the AI for data science. I think it's likely very useful for educational purposes. How do I do that? With little caveats.

It will point you every now and then to things that don't yet exist. For guidance, why don't you try this out next? Right now, I mean, KNIME Analytics Platform has this built-in recommendation engine, marketing likes to call it AI, which really only looks at pairs of nodes, right, and then says, okay, it's very likely that you want to use this node. We can do a lot more with AI models to kind of help people to say, hey, why didn't you look at this or try out this somewhere else? I think it's going to be super useful in many disciplines, not just data science, to skip the boring 60% where you just say, OK, can you get me a workflow that joins 2 Excel files and an Oracle database and does this and provides a scatter plot?

You have a workflow and you say, hey, it doesn't quite do what I'm interested in doing, but now I start from something instead of starting from scratch. It's not going to give you the fully developed solution. That's extremely unlikely. And the other piece is, think peer programming. Why do you do peer programming?

Because you have 1 of the 2 developers is always kind of looking over the shoulder of the other 1 and make sure they catch stupid mistakes early. Having an AI where you essentially say, do you find something fishy here? And the AI just saying, this looks weird and this I've never seen before. Maybe it's right, but at least it kind of points you to the instances where you might want to double check what's going on. Likely less useful in my mind for novel insights into data.

Because you're really finding something interesting, new in the data, is very likely something that nobody has seen before in similar types of data. So it's not something that you can generalize from because they are by definition very rare instances, very rare insights. For highly sparse information data, if at the end of the day 1 of the big worries of all of these AI models that are summarizing is also that the minorities get drowned out. They're not part of the summarisation. If company A has 10,000 more mentions and stuff in articles talking about it than company B, chances that AI summarises and will also mention B are much, much smaller.

And then to me, I don't think this idea of hypothesis generation, when I was still at the university, we had this 1 talk about sparking insights. Data science ultimately in many, many instances is about triggering this, that's weird effect. Where you look at your data and say, that's weird, that can't be right. And then you dig a little bit deeper and then you start realizing, ooh, that's a strange little interesting effect. That of course, in 95% of those cases you'll find out that, oh, somebody really screwed up.

So it was an error in the data. But the remaining 5% triggers a new interesting discovery, a new type of hypothesis. So I think it's not going to be that great. So I was relatively, and then of course for the analysis, I mentioned that proprietary private data, competitively super interesting data. Why do we have representatives of pharma companies here?

I'm not quite sure they would upload all of their screening data to a public AI to get some interesting insights. That type of stuff you need to do in isolation, maybe with a local copy, there may be ways around that. But if I'm doing that with my private copy, nobody else is going to benefit from that either, so my AI doesn't keep learning on it. So it's going to be isolated little discoveries. So I was a little bit delusional, so I looked around, maybe other people have a stronger opinion about that or a better 1.

Here's a beautiful quote that I came across where somebody said, in 3 to 8 years, we will have a machine with the general intelligence of an average human being. Thought, that's bold. That's a quote from Marvin Minsky from 1970. That was the first AI wave. So I'm, maybe I'm too old, I have too many déjà vus, but I'm carefully optimistic, but I'm also skeptical.

I don't think we'll be running into this human intelligence type of thing. I see us much more working with much smarter side applications that augment human intelligence. They will help us to do a lot of stuff, the mundane tasks a lot easier, faster, and maybe even get a bit more inspiration what to try out. But the last step in finding new insights and data and making something useful with your data is still going to be the human. So that's a bit frustrating.

So I figured let's ask 1 other question. Would you recommend mine? And that's when I said, this stuff does work. Man, this is, Ritu, did you see this? I mean, it's marketing.

Perfect. And then, of course, I clicked let it revise a couple more times and it went into completely weird directions. But anyway, cool stuff.

See all KNIMETV transcripts on Youtube

Opening - Michael Berthold