Congrats on the launch! Finally got some time to try this out.
Tried this on a couple of personal documents and compared to asking on
chatGPT (both 4 and 3.5), so far the results weren't great.
For context, the questions needed a little inference and danswer said the info isn't present in the document versus chatGPT which inferred the answer from a related statement.
I do intend to perform a much bigger test around documents but curious to hear thoughts on why this might be the case.
Hi, it may be an indexing issue. There's a precanned "Information not found" message that we show if document retrieval failed. A couple common causes for this are:
- Not provisioning enough resources and processes are dying (we run NLP models locally so the system isn't totally lightweight)
- Access is not correctly configured, either while pulling in from the source or at the user level during query time.
Assuming nothing is wrong with the setup, it is most likely because of how we prompt the LLM. Questions that require reasoning or are more open ended, generally aren't "safe" for the LLM to answer. So the system is constrained on purpose. For example if someone were to ask: "How can I increase revenue by 30% next quarter". It's not safe for the system to just propose some actions and it's likely better to just search the documents and say there wasn't any answer in the docs (unless of course some doc explicitly states plans for increasing revenue).
I do intend to perform a much bigger test around documents but curious to hear thoughts on why this might be the case.