That said, I still get the analogy by saying a car used 4 gallons per hour - it still indicates how fuel efficient something is. Especially if compared to something else
But the problem is you don’t have anything to tie that to. if you have a car that gets 4 gph but goes 100 mph then it’s more efficient than a car that gets 3 gph but only goes 50mph but even with those you miss out on the actual efficiency which for a car is usually transporting people.
So if car A gets 4 gph at 100 mph and transports 2 people it gets 50 passenger miles per gallon of gas which is finally an actually useful metric
For LLMs that becomes much harder to quantify but a useful metric might be wh per minute of time saved or mL of water per minute of time saved. Unfortunately to quantify those you would need to do much more in depth analysis and probably also factor in false readings and time lost from that
I get what you’re saying now.
That said, I still get the analogy by saying a car used 4 gallons per hour - it still indicates how fuel efficient something is. Especially if compared to something else
But the problem is you don’t have anything to tie that to. if you have a car that gets 4 gph but goes 100 mph then it’s more efficient than a car that gets 3 gph but only goes 50mph but even with those you miss out on the actual efficiency which for a car is usually transporting people.
So if car A gets 4 gph at 100 mph and transports 2 people it gets 50 passenger miles per gallon of gas which is finally an actually useful metric
For LLMs that becomes much harder to quantify but a useful metric might be wh per minute of time saved or mL of water per minute of time saved. Unfortunately to quantify those you would need to do much more in depth analysis and probably also factor in false readings and time lost from that