How do you approach Context Engineering when building with OpenAI models?
by•
Lately, I have been experimenting with how to feed context into GPT models more effectively.
For example, when fine-tuning or working with larger context windows, I have noticed that the dilemma is in organizing the surrounding information, rather than the prompt itself. Last week, I came to know that it's called Context Engineering.
I am curious how others here approach it:
How do you decide what information is essential vs noise when building AI workflows?
Have you noticed cases where too much context actually weakens the model’s output?
Would love to know about strategies or templates that worked for you while engineering the model context.
769 views
Replies
Needle
Hi @ashok_nayak creator of @Needle here. We aimed to solve that issue by combining agentic RAG & MCP tool calling, so that the Needle agent makes the right decisions based on the user's intended query. You may want to give it a spin.
@ashok_nayak @jan_heimes Agree with the approach. By the way did you find any solution to overcome rag limitations in search ? https://arxiv.org/abs/2508.21038
I don't think there is a recipe, it all comes down to a system of trial and error.
@d_ferencha Yeah, that's true...it feels like that.
@d_ferencha We've built one actually, patent is pending, can't wait to show folks!
AI Context Flow
Hi @ashok_nayak we have been working on something called as AI Context Flow - a browser extension that adds relevant memory into your prompts.
Regarding your two questions:
- RAG is by far the most important technique you need to use to see what's essential vs what's noise. Larger context windows are not really the solution
- Yes, too much context does weaken the results. There are two problems that happen when you add too much context. Read up on Lost in the middle and Context Rot (Needle in a haystack) benchmarks.
@hira_siddiqui1
I will give AI Context Flow a try, thanks for recommending it.
BTW, was it released on Product Hunt before, or you are yet to launch it here?
And yes, I will read the resources you shared.
AI Context Flow
AI Context Flow
@ashok_nayak please follow our page on PH so you get notified on the day we launch.
AI Context Flow
@ashok_nayak its called AI Context Flow
I treat context like a storyboard, each piece should move the model toward the outcome. When I overfeed details, the model rambles. Simplicity plus hierarchy has worked far better for me.
@moses_habila thanks for sharing your thoughts.
Any practical examples/templates you would like to share?
I'll answer the easier question first:
Yes, too much context is bad for model output quality
This is for a couple reasons, but as a rule of thumb if you can make the context smaller, do it.
Deciding which information to cut is tougher. Without knowing your exact situation it's tough to give specific advice, but here are a few generic pointers:
Long prompts tend to duplicate information, that is either explicit or can be inferred through context.
Passing in stuff into prompt directly as json is fine, just make sure that you're pre-filtering out garbage either programmatically or using another LLM
Try to see if the task you're having it do can be broken into sub-tasks. Doing this often reduces your context size.
The trickiest part is figuring out if your changes actually led to improvements. It's good practice to have some test cases setup beforehand.
@vsteppp thanks for the detailed answer.
Triforce Todos
Less is often more with context. A clean structure beats a big dump of info.
Cal ID
I’ve found that context engineering is all about trimming to essentials. Less is more – Really!
Too much info just muddies the output. Good structure, relevance, and constant testing seem to drive the best results for me.
Hi Ashok, I've been in the weeds on the same topic recently and while I don't have a solution I do have some reading that you may find helpful.
How Long Contexts Fail - explains why less is often more
How to Fix Your Context & Context Engineering for Agents - explain how to manage context
At the end of the day context engineering is informed by what you're trying to build. The more generalised you want your agent to be the harder it is to have a non-deterministic system produce a consistent output.
12-Factor Agents - Principles for building reliable LLM applications is an excellent resource for diving deeper into building agents.
@coire I love this type of answer.
Very helpful. Thanks a lot!
Hey @ashok_nayak I love what guys are doing in HumanLayer, they released 12 factor agents, and one of the pages is about owning your context, from what info you put there, to structure which you use and more, you can read it here https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-03-own-your-context-window.md
I hope it helps
@kacper_wdowik Thank you!
I will give it a read. Appreciate it!