I've been using ChatGPT 4 the past couple weeks and also Phind just last night with a new library version. While yes, i did find that Phind was wrong a lot (though i don't think it was fully hallucinations, just wrong library version combinations), i think there's a more important point to be made.
Unless we get a very near breakthrough on self-validating accuracy of these models or models+plugin combinations, i suspect it may be a useful skill to learn to use LLMs to explore ideas even when hallucination is a risk.
Ie searching with Google is a skill we have to acquire. Validating results from Google is yet another skill. Likewise i feel it could be very useful to find a way to use LLMs in a way where you get the benefits while managing to mitigate the risk.
For me these days that usually translates to low risk environments. Things i can validate easily. ChatGPT was a good starting off point for researching ideas. It's also very useful to know how niche your subject matter is. The less results you find on Google for your specific edge case the more likely ChatGPT will struggle to have real or complete thoughts on the matter.
Likewise i imagine similarly this is true for Phind. Yea, it can search the web, but as my tests last night showed it still happily strings together incorrect data. Old library versions, notably. I'd say "Given Library 1.15, how do i do X?". It did eventually give me the right answer, but it happily wrote up coding examples that were a mix of library versions.
I imagine Phind will, to me, be similarly useful (if not more?) than ChatGPT, but you really have to be aware of what it might do wrong. .. because it will, heh.
We definitely still have work to do in this area and the feedback we've gotten here is incredibly helpful. Having the AI be explicitly aware of specific library versions so it doesn't mix-and-match is a high priority.
Unless we get a very near breakthrough on self-validating accuracy of these models or models+plugin combinations, i suspect it may be a useful skill to learn to use LLMs to explore ideas even when hallucination is a risk.
Ie searching with Google is a skill we have to acquire. Validating results from Google is yet another skill. Likewise i feel it could be very useful to find a way to use LLMs in a way where you get the benefits while managing to mitigate the risk.
For me these days that usually translates to low risk environments. Things i can validate easily. ChatGPT was a good starting off point for researching ideas. It's also very useful to know how niche your subject matter is. The less results you find on Google for your specific edge case the more likely ChatGPT will struggle to have real or complete thoughts on the matter.
Likewise i imagine similarly this is true for Phind. Yea, it can search the web, but as my tests last night showed it still happily strings together incorrect data. Old library versions, notably. I'd say "Given Library 1.15, how do i do X?". It did eventually give me the right answer, but it happily wrote up coding examples that were a mix of library versions.
I imagine Phind will, to me, be similarly useful (if not more?) than ChatGPT, but you really have to be aware of what it might do wrong. .. because it will, heh.