Was having a related conversation with an employee this morning (I manage a software engineering organization). He asked an LLM how to separate the parts of a date in Excel, and got a pretty good explanation of how do it with the text to columns wizard, and also how to use a formula to get each part. He was happy because he felt it would have taken him much longer to figure it out himself.
I was saying I thought that was a good use of an LLM - it’s going to give a tailored answer - but my worry is that people will do less scrubbing of an answer coming from an AI than one they saw on a forum. I said we should think of it like a tailored Google search.
For comparison, I googled “Excel formula separate parts of a date” and one of the top results was a forum discussion that had the exact solutions the LLM gave, using the same examples. On the one hand, to get it from the forum you had to wade through all the wrong answers and discussions. On the other hand, that discussion puts the answer given in the context of a bunch of others that are off the mark, and I think make people less likely to assume it’s correct.
In any case, it’s still just synthesizing from or regurgitating training data.
I think LLMs are better for more fluffy stuff, like writing speeches etc.
Excel solutions are often very specific. A vague question like separating a date can be solved in many ways, using a variety of formulas, the text-to-column wizard, VBA, import queries or even just formatting, all depending on what you really need, what the input is and what locality is used and other things.
The text-to-column method is great, because it transforms whatever the input is into a date type, making it possible to treat it as and make calculations as an actual date. It’s not always the right solution though, for instance if the input is ambiguous.
It’s fine that he learned to use this method, but I wonder what he’d ask the LMM in a case where it isn’t the right solution and what it’ll come up with then. He didn’t actually learn to separate a date from the input. He learned to use the text import wizard.
In my experience it’s preferable to learn these things on a more basic level if only just to be able to search more specifically for the right answer, because there is a specific answer. Having a language model run through a bunch of solutions and presenting the most popular one might just be a waste of time and leading you into a wild goose chase.
You might have missed where I said it explained both the text to columns wizard and a formula. He used the formula, which is what he was looking for. He’s a top notch software developer, he just doesn’t use Excel much.
But I agree with your broader point. I keep having to remind people that the “LM” part is for “language model.” It’s not figuring anything out, it’s distilling what an answer should look like. A great example is to ask one for a mathematical proof that isn’t commonly found online - maybe something novel. In all likelihood, it’s going to give you one, and it will probably look like the right kind of stuff, but it will also probably be wrong. It doesn’t know math (it doesn’t know anything), it just has a model of what a response should look like.
That being said, they’re pretty good for a number of things. One great example is lesson plans. From what I understand, most teachers now give an LLM the coursework and ask it to generate a lesson plan. Apparently they do an excellent job and save many hours of work. Anything that involves summarizing information is good, especially as that constrains the training data.
Was having a related conversation with an employee this morning (I manage a software engineering organization). He asked an LLM how to separate the parts of a date in Excel, and got a pretty good explanation of how do it with the text to columns wizard, and also how to use a formula to get each part. He was happy because he felt it would have taken him much longer to figure it out himself.
I was saying I thought that was a good use of an LLM - it’s going to give a tailored answer - but my worry is that people will do less scrubbing of an answer coming from an AI than one they saw on a forum. I said we should think of it like a tailored Google search.
For comparison, I googled “Excel formula separate parts of a date” and one of the top results was a forum discussion that had the exact solutions the LLM gave, using the same examples. On the one hand, to get it from the forum you had to wade through all the wrong answers and discussions. On the other hand, that discussion puts the answer given in the context of a bunch of others that are off the mark, and I think make people less likely to assume it’s correct.
In any case, it’s still just synthesizing from or regurgitating training data.
I think LLMs are better for more fluffy stuff, like writing speeches etc.
Excel solutions are often very specific. A vague question like separating a date can be solved in many ways, using a variety of formulas, the text-to-column wizard, VBA, import queries or even just formatting, all depending on what you really need, what the input is and what locality is used and other things.
The text-to-column method is great, because it transforms whatever the input is into a date type, making it possible to treat it as and make calculations as an actual date. It’s not always the right solution though, for instance if the input is ambiguous.
It’s fine that he learned to use this method, but I wonder what he’d ask the LMM in a case where it isn’t the right solution and what it’ll come up with then. He didn’t actually learn to separate a date from the input. He learned to use the text import wizard.
In my experience it’s preferable to learn these things on a more basic level if only just to be able to search more specifically for the right answer, because there is a specific answer. Having a language model run through a bunch of solutions and presenting the most popular one might just be a waste of time and leading you into a wild goose chase.
You might have missed where I said it explained both the text to columns wizard and a formula. He used the formula, which is what he was looking for. He’s a top notch software developer, he just doesn’t use Excel much.
But I agree with your broader point. I keep having to remind people that the “LM” part is for “language model.” It’s not figuring anything out, it’s distilling what an answer should look like. A great example is to ask one for a mathematical proof that isn’t commonly found online - maybe something novel. In all likelihood, it’s going to give you one, and it will probably look like the right kind of stuff, but it will also probably be wrong. It doesn’t know math (it doesn’t know anything), it just has a model of what a response should look like.
That being said, they’re pretty good for a number of things. One great example is lesson plans. From what I understand, most teachers now give an LLM the coursework and ask it to generate a lesson plan. Apparently they do an excellent job and save many hours of work. Anything that involves summarizing information is good, especially as that constrains the training data.