Task-specific LLM evals that do and don't work
Comments
sfink
jrm4
Geez. This is such a reminder of how many "current" negative labels of this are ambivalent, probably useless, and possibly dangerous, e.g. "Toxic" and cousins "problematic" and "not okay."
And FWIW, I believe not saying this from any specific political-sided perspective. I very much like labels like "racist," "homophobic" etc. Not because they are always correct, but because they are relatively much CLEARER and force one to be serious about whether or not they want to use that label.
Havoc
A lot of models have also been overly chat trained. Responding with stuff like “sure I can help you with that”
That’s just unwanted noise if you’re trying to use them as a code building block in an application. So you need to force json or similar…which I suspect harms accuracy over free form
TeMPOraL
Unfortunately, that "unwanted noise" is a space for the models to compute; trying to eliminate it gives suboptimal responses. What you can do instead is try to corral it - let the model "think" like it wants, but guide it to add markers wrapping the thinking and/or result, then filter out the thinking in UI (for interactive applications) or as an intermediate/post-processing step (for hidden "building blocks").
If you're using Anthropic models, you may actually get improvements from prompting the model to maintain a tagging discipline; see https://docs.anthropic.com/en/docs/build-with-claude/prompt-....
hedgehog
As other people pointed out here you can also add "verbosity sinks" as text fields in structured output, recently I've also been experimenting with tool calls to support guided self-talk in a way that doesn't necessarily all accumulate in the context (e.g. if not all the tool parameters get echoed back).
glaugh
Thank you (and teMPOral) for these comments, this sounds potentially useful to me.
I hate to ask this, but I'm struggling to find any thorough posts or articles or papers about this, do you have any links you could point me toward?
jacobr1
Here is a short example that came up for me last week.
I had a set of documents I wanted to classify according a taxonomy that is well known (so it is exists in the training data of all the major llm models I tested)
If I have prompt like, `You are an expert classification system. Using the Classification Approach Foo, consider the following and output the category in JSON format, such as {"class":"bar"} `
This works ok, but it works much better if I tell it to output {"class":"bar", "reason": "baz"} and improved with some other approaches like adding "related_class" or "parent_category" which would otherwise be redundant.
Also including some few-shot examples helped, but the biggest benefit came from the "reason" field. Trying justification or other synonyms seems to produce the same output.
I suspect this is something similar to CoT.
potatoman22
Have you tested moving the "reason" field before the "class" field? That may encourage better CoT instead of having the model justify the class after it already picked it. Anecdotally, I saw a 5% boost in performance from a NER system from having the model output the entity's class at the end rather than the beginning.
te_chris
This has worked for me for sentiment classification, i had a score and a summary field and the results were much more accurate.
andai
Does it still work if the class field is output before the reason?
jacobr1
Yes. But, reason first did work (very slightly) better.
glaugh
Great, thank you (and hedgehog), that makes sense
hedgehog
Speaking only for myself these ideas are a combination of things I've seen scanning new papers and informal discussions with other people working in the area. Feel free to shoot me an e-mail though, maybe I can point you somewhere more specific.
Edit: The "verbosity sink" name is inspired by the idea from the paper below although they're not actually at all the same thing.
Jerrrry
It may be the self-aware human bias tainting this reasoning, but it seems convergent with our own psyche/brain processes, and/or inherent to the way we commonly express conscious thoughts.
Percolating tokens that allow a more "accurate" latent space appear to be more accurate, but are nearly actually useless noise. Almost a virtual shower thought.
Because people only put the answer at the end of a grammatically correct statement, with the more "reasoned" statements being more articulately percolated/logically sound, and that is expressed grammatically. These statements are inferred to be associated with intellectual boiler-plate. They may be correlated and not actually causative, but that would require a multiple component architecture with embeddings being used as a proto-"qualia" and that is getting hairy.
Facts should "only" have to be read once, and should be explicitly defined with a more secure of a confidence completely. Implicit inferences from those explicit facts should be emitted from a different, less confident module; with the chat boilerplate being tacitly composed finally when presenting the output to the user.
Of course separating the baby from the bathwater is the hard (not impossible) part.
pizza
It seems to me that it would make sense to just include more <BOS>-like meta tokens at the beginning in such cases, and have them as a prefixed scratch space that can be suppressed by treating them as non-output tokens.
bubaumba
it should be possible to ask model to think aloud (or step-by-step) and then give summary. in one or two prompts. give only summary back to user.
imtringued
We have Marco o1 at home.
behnamoh
marco o1 at home: https://www.reddit.com/r/LocalLLaMA/comments/1gyx1hj/macroo1...
TeMPOraL
That's... a good result, actually. No, I'm serious.
This reads exactly like my inner thought process on a novel or tricky task I'm asked to solve, especially when I know I'm tired (or drunk, back in the times I consumed alcohol on a regular basis), and need to spell everything out (out loud or in a text file).
Hell, it's exactly how I expect a kid who just learned about fractions would think. I have a vague recollection I processed such tasks this explicitly as a kid, until I understood the topic.
LLMs pulling this off reliably? That's huge progress. I used to say[0] that GPT-4 is best imagined as a 4 year old kid that memorized half the Internet. But this? This is 8 year old's stuff.
--
[0] - I currently prefer comparing it to "inner voice", and its performance and propensity to hallucinations to a smart schoolkid that's being asked questions by the teacher about things they only read about but didn't fully process, and who's pressured into giving some answer, as saying "I don't know" is an instant F and public humiliation. Such kid will be forced to extrapolate on the spot, but if they're smart enough and remember enough, they'll often get it at least partially right. I know that from personal experience :).
caturopath
Yeah, I don't know if someone thought this was bad or something, but it seems like valid reasoning. We may need to give these models a sense of things that require more and less detailed reasoning and better knowledge of what they know internally, but the "research work" the model claims to be, it seems like it's doing a good job.
The poster also shared in a comment https://preview.redd.it/u8vs29hq5w2e1.png?width=2704&format=... which did get the intended laugh out of me, but even that seems fair enough. I'm currently traveling in a country where most people speak a language I don't know well. You better believe I've been thinking through even trivial greetings, considering the setting, formality, appropriate follow ups, etc.
TeMPOraL
Looking at that example, I too feel it's a thought process I could go through once or twice. I think this highlights an important difference between humans and LLMs: a human can think such things explicitly in their head, or on paper (I do it in text editor more frequently than I'd care to admit), once or twice, and then it sticks, quickly becomes more of a "system 1" thing. With LLMs, the closest to that outside training/fine-tuning would probably be prompt caching. It would be great if we could figure out some kind of on-line learning scheme so the model could internalize its own thoughts and persist them between conversations, but in the latent space, not prepended as token input.
caturopath
> ou better believe I've been thinking through even trivial greetings
Even after thinking through what to say, I used the wrong greeting in a shop half an hour ago and the person working there called me on it.
behnamoh
OP here. I liked Macro-o1 (Marco1?) but as you pointed out, we need to teach these models to spend their system 2 thinking more economically.
caturopath
One of the things complicating the example is that the counting-letters task is actually a legitimately hard one for it, even if it's trivial for us: it's bad at letter stuff because of the way it represents text with tokens. I believe the problem exists, but the letter thing isn't necessarily a great example of something that it should recognize as trivial.
iknownthing
interesting
msp26
> which I suspect harms accuracy over free form
Untrue in my testing. If you want to use chain of thought, you can always throw in a `thoughts` field (json field/xml tags) before the rest of your output.
n2d4
If you want to be really sure, you can also first ask it to respond in chat format, and then ask it again to respond in JSON format, if you can afford the cost.
msp26
It really isn't necessary when using constrained decoding (aka structured outputs) which guarantees that you'll get JSON output in the correct structure.
qeternity
This is not true at all. Just because you can force the logits to give syntactically valid outputs, doesn't mean you're going to get a useful result.
Constrained generation, without a proper understanding of the model's natural response tendencies, can give horrible results.
msp26
I agree with you completely. I was talking about the parsing being easy with this, not referring to the outputs being correct in reality.
You can get awful results with poorly defined constraints.
imtringued
Depends on the way you do constrained generation. If all you do is reject tokens using a grammar, then yeah it is bad. If your software inserts things like field names and braces instead of forcing the model to produce them token by token and then afterwards rejecting the wrong tokens, then you should be good to go.
potatoman22
Why is it bad to reject tokens using a grammar?
petesergeant
This isn’t a problem in practice. Most of my prompts ask the LLM to do a bunch of chain of thought before asking them to spit out JSON. I extract the JSON, which works 97.5% of the time, and have a retry step being real specific about “here’s the conversation so far but I need JSON now” that handles the rest. Adding examples really helps.
imtringued
https://lmsys.org/blog/2024-02-05-compressed-fsm/
I'm not trying to shill sglang specifically, just pointing out that there's a better way, btw.
hansvm
...with the obvious caveat that the distribution of responses isn't the same
Elaborating slightly, retrying till the schema is adhered to has a different distribution from greedily selecting tokens adhering to the schema.
The simplest toy example I can come up with for that property is a universe of answers "aa", "ab", "bc", all of which the model is equally likely to output for a given prompt with normal auto-regressive invocations. The schema, in regex, is ".[bc]". Retry-till-success produces "ab" 1/2 of the time and "bc" the other half. Greedily adhering to the schema produces "ab" 2/3 of the time and "bc" the remaining third.
Last I checked large-scale LLMs, it was a problem in the wild for large string fields. They tend to want to finish the string with ellipses (this creating an incorrect response), but when they made that mistake they'd tend to truncate the entire json record and generate something that doesn't adhere to the schema. Retry-till-success has a high successful parse rate. Greedily adhering to the schema converts those ellipses errors into syntactically correct garbage.
Other such bugs can be much harder to quantify (model explainability is hard), but I'd be cautious employing the technique without a lot of case studies for your particular problem domain.
ActionHank
I also firmly believe that number of tokens served is a metric that is tracked and encouraged to go up, because more tokens mean more charges. o1 "does more" by using a whole lot more tokens for a very slight bump in usefulness.
baq
The problem with such takes is 1) the facts are also consistent with performance plateauing for fundamental reasons and 2) there are good, perhaps better than them, competitors with similar issues.
ActionHank
I think we’re on the same page here. In lieu of growing the product because of fundamental constraints they’ve opted to grow business metrics to net more revenue which in turn looks good on paper to investors.
phillipcarter
I've not had that experience when I include in the prompt for a coding LLM "only respond with the code".
Though it's worth noting that I often do want an explanation, and currently my workflow is to just use a different LLM.
michaelt
There were some models in the past [1] that were extremely keen to produce chatty noise, even when you explicitly asked them not to.
Of course this was back in May 2023, so things might have improved since then.
Kuinox
Are you not using instruct tuned models ?
TeMPOraL
Obviously they are, that's why they have this problem. Or did the terms "instruction tuning" and "instruct models" change their meaning when I wasn't looking?
knicholes
Shoot, maybe someone edited something, but I don't see anyone else in this conversation using the terms "instruction tuning" and "instruct models"?
iamwil
Writing task-specific evals are pretty important, and lots of people are just going off of vibes right now. If this all seems too much all at once, and you don't know where to start, we wrote a jargon-free issue for getting started with system evals.
The basic idea for system evals is to find a way to define a qualitative trait you want in the LLM responses using a corpus of examples, rather than being able to define it exactly using prompts. Then through systematic improvements, you nudge your LLM-driven task to adhere closer and closer to the given examples, for some metric of closeness. That way, you can be more sure you're not regressing on LLM responses as you try to make improvements. This is standard stuff for data scientists, but this way of working can be a little foreign to web engineers (depending on prior experience). It just takes a little adjustment to get up to speed.
vessenes
This is a fantastic resource. Super detailed, super practical, thanks for putting this up, Eugene! I learned a few things and love the practical engineering and stats angle on these assessments.
sails
Has anyone seen any good eval techniques for the OpenAI structured output api?
The toxicity example was thought-provoking.
I hope it's uncontroversial to say that there's nothing "toxic" about that continuation by itself. (My expectation from that beginning is that it would then continue on with a modest beginnings story of how the father worked hard, etc.)I guess the idea is that it is the leading portion of a toxic output, and if you prevent that beginning, you'll prevent the problematic continuance? At the cost of many possible non-toxic continuations.
I've never seen an actual labeled example before. Is this the form they usually take, or is this one quoted because it's innocuous and therefore uncontroversial to insert into a document about LLM evals?