Study Overview and Findings

A recent study by researchers from Stanford University and UC Berkeley, which is still awaiting peer review, suggests that ChatGPT may be exhibiting a decline in performance. The research, titled How Is ChatGPT’s Behavior Changing over Time?, was first released on July 18, 2023, with a revised version available on August 1, 2023.

The study evaluated the outputs of the March and June 2023 versions of both GPT-3.5 and GPT-4 across four specific tasks:

1. Solving mathematical problems
2. Responding to sensitive or risky inquiries
3. Code generation
4. Visual reasoning

Results revealed that the March 2023 iteration of GPT-4 achieved a remarkable 97.6% accuracy in identifying prime numbers. In contrast, the June 2023 update plummeted to just 2.4% accuracy. Interestingly, GPT-3.5's performance improved in June compared to its March counterpart. Furthermore, the June version of GPT-4 displayed increased caution when addressing sensitive questions compared to March. Both GPT-4 and GPT-3.5 also exhibited more formatting errors in code generation during June than in March.

These findings emphasize the potential for rapid fluctuations in the output quality of LLMs, underscoring the need for ongoing assessment.

The first video titled "Is ChatGPT Getting Dumber?!" delves into the concerns surrounding ChatGPT’s declining performance and explores the implications of these findings.

The Need for Continuous Monitoring

While the researchers did not specify the reasons behind the observed drop in accuracy, many speculate that efforts to enhance certain aspects of complex AI models may inadvertently degrade other functionalities. To mitigate this risk, it is crucial that LLMs undergo regular evaluations to confirm their performance aligns with expectations, allowing for timely adjustments when necessary.

The second video, "ChatGPT is Getting Dumber," provides a broader context to the discussion, examining user experiences and the perceived decline in the quality of responses over time.

Engagement and Feedback

Do you think ChatGPT's responses have worsened compared to previous months? Have you encountered any challenges while using ChatGPT? We encourage you to share your thoughts or questions in the comments section!