In the 11th episode of Speaking of AI, LXT’s Phil Hall interviews Karin Golde, founder of West Valley AI. Phil describes Karin as living at the intersection of AI, data, human AI alignment, and product strategy. Join them as they dive into what this means, connect over their shared background as linguists, and discuss leadership development, bias in AI and misconceptions about AI hallucinations.
Introducing linguist, leader and founder of West Valley AI, Karin Golde
PHIL:
Today’s guest on Speaking of AI is someone who resides in the Bay Area, but she lives at the intersection of AI, human AI alignment and product strategy. She has a PhD in linguistics. She’s held senior leadership roles at NetBase, Quid, Clara Analytics and Amazon Web Services. And she’s the founder of West Valley AI. Please welcome Karin Golde. Hi Karin.
KARIN:
Hi, thanks Phil, it’s nice to be here.
What does it mean to live at the intersection of AI data, human AI alignment and product strategy?
PHIL:
Great. So from your perspective, what does it mean to live at the intersection of AI data, human AI alignment and product strategy?
KARIN:
Well, what I’ll say about that, let’s see. So living at the intersection of AI data and strategy and product strategy, it all comes back to kind of a human-centric approach. Language is a human construct. We sometimes think of it as just black-and-white marks on a page, just some text that’s living independent of us. But really it’s all about how do we use language to achieve our goals? And so to do that, we really have to think holistically about how do we approach language from a technical perspective, but also from a usability perspective, and an ethical perspective as well.
How has the role of the linguist changed over time?
PHIL:
In 1998, Fred Jelinek reportedly said, every time I fire a linguist, the performance of the speech recognizer goes up. Was that a fair comment back then? Has the role of the linguist changed over time? And is linguistics relevant to our industry and in particular AI today?
KARIN:
Yes, well, I will say I’m familiar with that quote and I think it comes from a place that, you know, linguists at heart, we love language and we can’t let go of that love, right? So when you see, and let me give maybe a little more context for some of your listeners too, about what he was talking about there. Back then, there were still a lot of symbolic systems in play in the AI industry, which wasn’t, which wasn’t called AI at that point. But there were systems that were trying to recreate grammar as we hypothesized that we use it as humans.
So it would be a matter of doing careful parsing of sentences, making sure that you understand the subject, verb, object, any adjective phrases, what do they describe, what are they attached to? So you get into these very complex linguistic representations of language and then you operate off of that when you’re building a system to interpret what language means. This can be a very manual process in a lot of ways, you know, building out these rules, testing them, making sure that the sentence structure is coming out the way you expect it.
And so I think people got frustrated with the slowness of linguists building those systems. And what they found was that you can take statistical approaches and just look for probabilities, just use math to pick up on the features of language and output something like a classification. What class does this text belong to, or extract information from it. Where are the names of people or places and so forth? And so I think what he was referring to there was, I can just use math to do this faster. And there’s always been a tension in the industry between those two points of view. And I’m sympathetic to both, really.
In today’s AI landscape, is linguistics still relevant?
PHIL:
Yeah, look, me too. Full disclosure for anybody that’s watching this, I’m a linguist as well. But with that in mind and with the deep learning techniques that are prevalent today, is linguistics relevant or has it, have things moved on to a point beyond there?
KARIN:
Well, yeah, I guess there’s this basic analytical stance towards language that you get from being trained as a linguist, which I think still holds a lot of value. So it’s very subtle, but this is something that I’ve really seen in practice at the various companies I’ve worked at over the years. You can always tell from the conversations in a meeting who has linguistic training and who doesn’t when they’re looking through language data. So, we know that language is this intermediary between humans and knowledge, right? But by itself, language does not constitute knowledge. The knowledge comes from both the semantics of the text, the meaning of the individual words, how they get put together to make higher-level meanings. So that literal meaning, but also it comes from the pragmatics, you know, some additional conclusions, inferences that we make as individual humans.
So, for example, if we read a restaurant review and it says, I had to raise my voice to talk to the people at my table, right? We take that to mean the restaurant was noisy. We don’t take that to mean that the people at the table were hard of hearing or something. We assume it’s relevant to the restaurant review. And what I’ve observed is that linguists look at that and they look at the language analytically and they think, wow, you know, this is gonna be really complex to programmatically capture, or restaurant noise, because it’s not gonna just be people saying the restaurant was noisy, there’s gonna be all kinds of ways to indirectly express that. People who have not had training in linguistics sometimes will kind of skip that analytical step and think like, well, it’s obvious. You can see that the restaurant was noisy just by reading this. And they don’t see the difference between the semantics of what’s on the page versus the pragmatics of what you as a human bring to interpreting the meaning of that statement.
What advice would you give to an institution that’s designing a contemporary linguistics course?
PHIL:
Yeah. So I was recently asked to provide input into course design for a bachelor of linguistics course at Macquarie University here in Sydney. And the background to that is that in my various roles over the last 25 years, I’ve hired a lot of linguists and so they came to me as somebody who might be able to offer some suggestions on what makes a linguist employable.
So, what advice would you give to an institution that’s designing a contemporary linguistics course? And are there particular skills that you feel are must-haves and other linguistic areas that you’d suggest are maybe less relevant in this field?
KARIN:
Yeah, so I’m going to say first and foremost is data collection. And I think talking to people who are doing university courses right now, it seems like that is being addressed better, at least than when I was in school, or maybe it was just my program. But it wasn’t really an emphasis. You kind of assumed the data was there, or you had sort of armchair explanations of like, well, this sounds right, kinds of things. It wasn’t quite so much data-driven. But the data collection part itself, I think, is really fundamental to both linguistics and AI.
So as a linguist, you want to base any kind of theoretical hypotheses you have on actual data that you’ve collected and tested out on that data. It’s also really valuable, of course, to have linguistic data on more scarce phenomena like low-resource languages. So, it’s a very valuable exercise in itself to do this data collection. But how to do that ethically? How are you going to crowdsource it? Are you going to go into communities and ask for it? How are you going to compensate people for it? How do you manage the data through its life cycle? So how do you learn about data governance principles? That would be probably my number one recommendation for building linguistics courses. And like I said, I think that people who do any kind of machine learning should be thinking about that as well.
That’s also, it’s a gap over there, too. And so the more people know about data collection, the better. I think the other thing that linguists do bring that is still really valuable and should still be a core part of courses is understanding multilingual considerations. So, the industry is really dominated by English, right? And that’s partly a business thing. It’s where the money usually is when you’re first building a product, that’s the first market you want to serve as an English-speaking market. And then the rest of the languages are developed as an afterthought. And that’s unfortunate, right? From a few perspectives.
But taking, if a linguist comes in and takes a perspective that English is just one of the many ways to represent meaning, then you can help avoid developing technologies that kind of take English as the norm and everything else as the weird language. I mean, English is just as idiosyncratic as any other language. So you really want to build from the start for all languages and not just for English.
What advice do you have for people who are pivoting into a leadership role?
PHIL:
Yeah, great advice. I will actually pass that advice on to the people I’m speaking to, in fact. For my next question, I’d like to get your thoughts on the concept of leadership. So, in today’s workplace, it seems to me that a lot of weight is placed on the value of leadership skills, but that outside of, say, MBA programs or specialized training programs like military officer training and so forth. There’s not a lot of weight placed on this in educational settings. Now you’ve held some impressive leadership roles and you’ve listed ethical leadership as one of your skills.
What advice do you have for people who are strong individual contributors in an organization and are maybe approaching the point where their next career move is going to require a pivot into a leadership orientation? And of course, what from your perspective is ethical leadership?
KARIN:
Yeah, that’s a great question because I think most people just kind of get thrust into a leadership position and it’s assumed that they’ll figure it out because they’ve had managers before and so they’ve seen people do the work so now just do what you saw people doing.
And it’s really not that simple. I think the other misconception is that some people are kind of born leaders and others aren’t. But leadership skills really are skills that can be learned. And there’s no one type of leader that, you know, you have to be this kind of person to be a leader.
I think pretty much anyone who has the desire to do so can look at the skills they need to grow and say, yes, this is for me, or no, I don’t want to go in that direction. So I think first and foremost, treat it as a learning exercise. Expect to have to learn. Don’t expect it to just kind of come to you. So that would be the first thing.
What constitutes ethical leadership?
PHIL:
And regarding ethical leadership as a concept, I guess I have a take on it, but I suspect that’s idiosyncratic. And I’d love to know if you think there are a set of boundaries around what constitutes ethical leadership. I think that would be an interesting thing for people to hear.
KARIN:
Yeah, so I think as far as ethical leadership, I think the hardest part that I’ve had to deal with is a couple things. One is the issue of fairness to your team, right? So you have to think about what’s good for your team as a whole, what’s good for the company as a whole when you’re dealing with individuals. And that doesn’t mean like being unfair to people or throwing them under the bus, it just means figuring out are they contributing to the team the way you expect?
If not, where is the issue? Is it that your expectations aren’t clear? Is it that they have a gap in the skills? Can that gap be closed? All of these things have to be issues that are addressed immediately and upfront. And I think that there’s a lot of ways in which doing that right means being an ethical leader, just kind of letting problems fester. And it can have a negative effect on the rest of your team. It can be, you know, the other people can perceive it unfair if you let somebody kind of continue to underperform without addressing it. So those are kind of the management aspects of it.
And then the other way in which it can be difficult is when you don’t agree with the decisions of senior leadership, but you still need to communicate those to your team. So I think that’s where you have to really be in touch with your own value system, and understand to what extent does it overlap with the organization and where is there possible disconnects? And you have to decide whether the decisions that are being made are just ones that you disagree with because you think it would be better to do something differently. Maybe we would get more customers or it would be more fun or are the decisions you disagree with crossing an ethical boundary for you. And that’s really tricky to be aware of because once you’re aware of that, you have to make some hard decisions. Your approach can’t be just go to the team and complain about senior management. That’s not the thing that you can do. What you have to do is figure out, are you going to disagree and commit on this one? Or are you going to push back? Or are you going to even leave the company because that’s just not a good fit for you?
What challenges are you helping clients solve, where are they getting AI right, and where are they struggling?
PHIL:
Yes, there’s a lot in there that resonates for me. So now that you’ve entered the world of consulting, what challenges are you helping clients to solve? And where are they getting AI right? Where are they struggling?
KARIN:
Yeah, well, the people I’ve been talking to, I think the biggest issue that they’re having right now is with evaluation. So, especially with the current systems that you can adopt a little bit more easily than you could previously, you could build machine learning models, but it would take a team of machine learning engineers or data scientists. And now you have more of an opportunity to use APIs to directly connect to large language models like ChatGPT and get your answers directly and have that be part of your system. And that feels really magical and it often will work really well in a demo setting or even in kind of as a beta mode. But then the question is, okay, great.
Now we have it in production because it felt good enough to release, but how do we really know it’s working? How do we know if it’s going off the rails? Those kinds of evaluation systems, I think, are a really interesting area of development right now. There’s some very interesting platforms out there that you can use to semi-automate evaluation of responses in production. So I would say that that’s one of the biggest issues that people are facing right now.
What are key considerations when somebody’s building a data pipeline to ensure that it’s fit for purpose?
PHIL:
We’ve talked a little bit about data. In fact, you’ve emphasized the importance of data and data collection quite strongly. What are key considerations when somebody’s building a data pipeline in order to ensure that it’s fit for purpose? And do you think companies are investing enough in data?
KARIN:
Hmm. You know, I think data has gotten a bit more of a, a bit more public attention, certainly lately with people being concerned about web data being used for pre-training large language models and so forth. It’s definitely always been an issue though. What data are you going to use and how are you going to build a pipeline that goes all the way from collecting it, to training the model, to holding out some for evaluating the model and so forth. So I’m not going to say, I think that you can always do more.
So I think to answer your question, are companies doing enough? I think they can definitely do more. And there are certainly some, I’m thinking right now about the data-centric AI movement in a way. Right, so this was something that Andrew Ng, who is a fairly well-known researcher in AI, he coined this term or made it popular at least around 2019 or so, about five years ago. And it’s definitely taken on in certain circles. Certain people do understand that there is this concept of you can iterate on getting better and better data and get better results that way than iterating on trying to train a better and better model using fancier algorithms and so forth.
So I think that there is a certain amount of kind of penetration into the industry of recognizing the importance there. But, you know, as more and more people kind of get into AI beyond the original sort of set of data scientists, machine learning engineers, now it’s like software developers and product managers and everybody is sort of in the game. I think we do need to continue to raise awareness about the importance.
Do you think that bias will limit the extent to which AI can reach its full potential?
PHIL:
Great. Well, that’s a good lead into my last small set of questions here. So these are all based around ideas of bias. So in terms of AI itself, do you think that bias is going to limit the extent to which AI can reach its full potential?
KARIN:
Yeah, bias is difficult because to a certain degree, bias is kind of the whole point of machine learning models. Machine learning models are very agnostic. They’re just a tool to interpret whatever data you give them. And so you do want them to learn and pick up from the patterns that they see, not to anthropomorphize, you know, that are consumed by the models. What counts as undesirable bias is a much trickier question. And we often think about, well, how might the output adversely affect protected classes? I think is one way in which we often think about bias, which is a very important consideration.
So for example, you know, if you’re building a machine learning model to screen resumes, and there was a famous example of this happening at Amazon in 2018, which they quickly corrected, but, you know, it’ll be difficult to say for any one resume, whether it was rejected or accepted based on, you know, cues to the person’s identity. What they found in the case of Amazon’s resume screening process that they were using internally was that it was discriminating against people who, who went to historically black colleges or all women’s colleges. And it wasn’t so much that it was picking up on any identity characteristics in particular, but since those people had been discriminated against in the past, it picked up on that and kind of continued and amplified that discrimination.
So if you look at the overall patterns, you can kind of start to see where the problems are and you can build test data sets that…that probe for a particular bias once you’re aware of it or once you suspect it might occur. So you can have resumes that are identical except for certain characteristics like whether the applicant went to historically black university and you can test for bias in the system and try to correct it with additional training sets. But you really have to be aware of it and looking out for it. It’s not going to just jump out at you. And you also have to have a clear set of values about what counts as negative bias for your application.
How do we deal with the inherent bias of real-world unstructured data?
PHIL:
Okay, my next question, maybe my next question should have been before the previous question, but in terms of bias in data, my view is that real-world unstructured data is inherently biased. That doesn’t mean necessarily negatively biased, but everything’s from a perspective.
If it’s real-world data, everything has a perspective. Is this a problem or is it just a fact of life that we… Yeah, we deal with it as humans. Do we just deal with it from a technology point of view as well?
KARIN:
Yeah, it’s a really good question because, as you’re kind of saying here, we get a lot of training data from the web. And especially these days for large language models, we use news and blogs and message boards and so forth. And we often think of this as like, that’s just a mirror of the world. And so any bias that’s in there is just, it’s going to be reflected by our systems. And so there’s really not much we can do about it. But it’s really not just a mirror reflection of the bias. It’s also an amplification of that bias.
So because these models deal in probabilities, if something is probable, it becomes the norm. And it kind of it erases what other diversity there even originally was. So I saw a study recently that analyzed AI-generated images. And it looked at the actual distributions for ethnicity and gender across different types of occupations and found that the images generated really amplified the existing tendencies. So for example, all of the images of flight attendants that the AI generated were all women. But in reality, in the US, only 65 % of flight attendants are women. So it’s not even just that it repeats what we’re telling it, it really takes that to be gospel at some point.
PHIL:
Yeah, yeah, I’ve observed a parallel to that with music selection algorithms. I’m looking for things that I haven’t heard before. But it doesn’t matter, if I pick a piece of jazz music. It really doesn’t matter what genre of jazz, what period, whatever I pick within an hour or so, I’ll get Kind of Blue by Miles Davis. It doesn’t matter where I try and direct it. I know that that’s coming.
KARIN:
Yeah, and you know, speaking of, I have these problems too, because I love discovering music through algorithms, but it’s not great. And I think there’s also a bias against older music, right?
Because newer music hasn’t had a chance to really pick up a lot of features that the algorithm can learn from. And so, it kind of, like you said, keeps going back to the same old standards that it’s sure that you’re gonna like.
Within the industry, as a founder of a startup, have you faced negative gender bias? Is this a barrier that you’ve had to overcome?
PHIL:
Yeah, yeah, it’s a few like that and you like that. You’re going to love this one because everybody does. And then my final question on that topic of bias is not really a technological one. It’s just a social one. So within the industry, as a founder of a startup, have you faced negative gender bias? Is this a barrier that you’ve had to overcome?
KARIN:
It’s a really good question. So for me personally, I would say like kind of over the course of my career, I’ve probably, I’ve been in situations definitely where I get a weird vibe, right? And there’s never anything overt. So, you know, one answer is that yes, I have my suspicions, but you can never really pin it down. And it’s kind of, it’s also, it’s like the water you’re swimming in, right?
And I actually, I was talking a while back to a trans woman who had been in the software industry working as a man for quite a while and then transitioned to being a woman and she said the way she gets treated is way different. So there are a few people out there who can really give us some A/B testing and report back. So it’s definitely out there. It’s just, it’s really hard to know if that’s the way that’s your, what your experience has always been, like whether it’s different from other people’s.
Let’s talk about AI hallucinations…
PHIL:
Great. I’ve got one more question. This is kind of putting you on the spot. We’ve gone through a range of different areas here. If you were running the interview, is there a question that I should have asked you but I didn’t? And if there is, what is it and what’s the answer?
KARIN:
I think I want to… I want to talk about hallucinations because it’s a thing that bugs me. So we kind of touched on this a little bit, the way that people kind of misinterpret language models and the way that people think of language as being inherently knowledge carrying, whereas actually it’s only part of the picture.
So let me back up and explain that a little bit more. So large language models are frequently correct. So when you ask ChatGPT a factual question, it often gives you the right answer. If you aren’t able to fact check that answer, a lot of times it just sounds right. So it probably is right. And so you kind of get into that habit of trusting it. Well, now and then you find something that you know isn’t right, or you go and fact check something and you find it’s not right. And so people talk about these as being hallucinations, that the system has and there’s all kinds of techniques that are developed to try to limit that and have it just repeat factual information. I feel like this is, sometimes when I hear people talk about this, it feels like foundationally they’re just not thinking about it the right way.
Hallucinations are really a feature, they’re not a bug. That this is the way the whole system was designed was to spit out probable sequences of words that sound like human language. You are the one who takes that language as a human and creates knowledge from it. The knowledge does not reside in there. So it’s not like hallucinations occur because the language model generally stays on this nice narrow track. Sometimes it just kind of goes off the rails and you just have to put it back on the rails and now it’s back on track again. There are no rails, right? It’s just gonna go wherever it goes and it just happens to coincide a lot with things that we know are true.
So I think we need to really get comfortable with that and not try to make large language models do something they weren’t designed to do. The other use cases should really call for where you need creativity and not critical decision-making. I think in general, structured knowledge… Maybe another question for me is what are my predictions for the future? And so I’ll answer my own question. I think structured knowledge will become increasingly valued as a result. I think things like knowledge graphs, ontologies, which really incorporate facts and give like complex networks of facts, a home which is transparent in a way that large language models are not, those are going to play an increasingly important role as complements to large language models.
PHIL:
Karin Golde, thank you for making the time to speak today. It’s been a genuine pleasure and extremely informative. I’m sure people are going to thoroughly enjoy hearing your views on things.
KARIN:
Well, thank you very much. It was wonderful to be here.
PHIL:
Yeah, I look forward to meeting in person when I’m in the area sometime.
KARIN:
Absolutely.
PHIL:
Thanks again. Bye bye. Thank you.
