One-sentence summary:
ChatGPT/OpenAI released an update at 1 AM on Monday, March, 20th, causing the potential breach of sensitive data of over 1.2% of users within a 9-hour window.
Who was involved?
OpenAI, ChatGPT, 1.2% of users in a 9-hour window on March 20th, 2023, and, potentially, other ChatGPT users.
What was the timeline?
March 20, 2023: At 1 AM PST, ChatGPT/OpenAI introduces a redis-py library bug during a change pushed into production. Breach starts.
March 20, 2023: At 10 AM PST, breach allegedly stops per ChatGPT.
March 24, 2023: OpenAI publishes a breach notice on its website and confirms the bug is patched.
What occurred?
OpenAI updated their open-source redis-py library in production at 1 AM, resulting in server-side caching errors and the breach of sensitive data for approximately 9 hours (OpenAI, 2023). Approximately 1.2% of active users between the hours of 1 and 10 AM PST were affected, and the data potentially breached included: “user’s first and last name, email address, payment address, credit card type and the last four digits (only) of a credit card number, and credit card expiration date” (OpenAI, 2023). An “outage” was also associated with the incident, however, this appeared to be an incident response (Abrams, 2023).
Estimated costs:
Potential, internal incident response costs, potential lost business and operational costs due to downtime, investigation, and troubleshooting,
Involved laws:
TBA or N/A (see disclaimer)
Root cause:
Change management: open-source Redis-py library update introduced code conflict into production, allowing “corrupted data” from other users to be returned from a Redis cache to any valid user’s requests.
Lessons learned:
Extensive regression testing is mandatory for software-based companies before pushing code into production. Just take a look at the remediation steps by OpenAI:
- Extensively tested our fix to the underlying bug.
- Added redundant checks to ensure the data returned by our Redis cache matches the requesting user.
- Programatically examined our logs to make sure that all messages are only available to the correct user.
- Correlated several data sources to precisely identify the affected users so that we can notify them.
- Improved logging to identify when this is happening and fully confirm it has stopped.
- Improved the robustness and scale of our Redis cluster to reduce the likelihood of connection errors at extreme load. (OpenAI, 2023)
Another thought: extensive monitoring is required during a push to production (even at 1 AM). Monitoring social media should also be a part of this effort, as Twitter users reported seeing the bug in real time (Abrams, 2023).
Sources:
Abrams, L. (2023, March 24). OpenAI: ChatGPT payment data leak caused by open-source bug. BleepingComputer. https://www.bleepingcomputer.com/news/security/openai-chatgpt-payment-data-leak-caused-by-open-source-bug/
OpenAI. (2023, March 24). March 20 ChatGPT outage: Here’s what happened. Retrieved April 21, 2023, from https://openai.com/blog/march-20-chatgpt-outage
OpenAI (logowik). (n.d.). openai-chatgpt-4038641.logowik.com.webp. logowik.com. https://logowik.com/content/uploads/images/openai-chatgpt-4038641.logowik.com.webp