Have you ever attempted to fill up one of those monster context windows up with useful context and then let the model try to do some useful task with all the information in it?
I have. Sometimes it works, but often it’s not pretty. Context window size is the new MHz, in terms of misleading performance measurements.
I find there comes a point where, even with a lot of context, the AI just hasn’t been trained to solve the problem. At that point it will cycle you round and round the same few wrong answers until you give up and work it out yourself.
I think that giving the LLM an API to access additional context and then making it more of an agent style process will give the most improvement.
Let it request the interface for the class your using, let it request the code for that extension method you call. I think that would solve a lot, but I still see a LOT of instances where it calls wrong class/method names randomly.
This would also require a lot more in depth (and language specific!) IDE integration though, so I forsee a lot of price hikes for IDEs in the near future!
To actually answer your question - yes, but the only times I actually find it useful is for tests, for everything else it’s usually iffy and takes longer.
Intelligently loading the window could be the next useful trick
Have you ever attempted to fill up one of those monster context windows up with useful context and then let the model try to do some useful task with all the information in it?
I have. Sometimes it works, but often it’s not pretty. Context window size is the new MHz, in terms of misleading performance measurements.
I find there comes a point where, even with a lot of context, the AI just hasn’t been trained to solve the problem. At that point it will cycle you round and round the same few wrong answers until you give up and work it out yourself.
I think that giving the LLM an API to access additional context and then making it more of an agent style process will give the most improvement.
Let it request the interface for the class your using, let it request the code for that extension method you call. I think that would solve a lot, but I still see a LOT of instances where it calls wrong class/method names randomly.
This would also require a lot more in depth (and language specific!) IDE integration though, so I forsee a lot of price hikes for IDEs in the near future!
To actually answer your question - yes, but the only times I actually find it useful is for tests, for everything else it’s usually iffy and takes longer.
Intelligently loading the window could be the next useful trick