Technical Incident Report
Incident Overview
Date of Incident: Thursday, 7th December
Services Affected: VoIP Systems - v5 & Sophia
Impact: Some customers experienced one-way audio during reinvite scenarios.
Description of the Incident On the morning of December 7th, we received reports from several customers experiencing issues with our end-to-end VoIP system. Specifically, during reinvite scenarios, the called party was encountering one-way audio issues. Our monitoring systems also detected empty RTP (Real-Time Protocol) packets, which was an unusual occurrence.
Initial Investigation A team was immediately assembled to investigate the issue. The following steps were taken as part of the initial investigation:
Verification of Customer Reports: We confirmed the reported issue by simulating reinvite scenarios and observed the one-way audio problem. Analysis of RTP Packets: Examination of the RTP packets revealed that they were empty, which was consistent with the one-way audio reports. Check with External Parties: We reached out to our external partners and telecom providers from whom the calls originated to ascertain if there were any recent changes or issues on their end.
Internal Systems Review: Our team conducted a thorough review of our internal systems. We verified configurations and recent change logs to rule out any internal updates that could have caused the issue.
Root Cause Analysis:
Despite extensive investigation, no recent changes or anomalies were found in both our and our partners' systems that could directly be linked to the incident. This led to the conclusion that the issue might be related to the way reinvite scenarios were being handled in our VoIP infrastructure.
Resolution and Recovery
To mitigate the issue and restore service for the impacted customers, we made the following changes:
Re-invite Behavior Modification: We temporarily adjusted the re-invite behavior in our VoIP system. This change was specifically targeted at the customers who were experiencing the issue. Result: After implementing this change, the functionality was restored, and customers reported the resolution of the one-way audio problem.
Post-Incident Actions
Monitoring: Enhanced monitoring has been put in place to closely observe the reinvite scenarios and RTP packet integrity.
Further Investigation: A team has been assigned to continue investigating the root cause of the empty RTP packets in reinvite scenarios. This includes a deeper dive into the interplay between our systems and those of our external partners.
Customer Communication: We have reached out to all affected customers, explaining the incident and the steps taken to resolve it. An apology for the inconvenience and assurance of our commitment to service quality were communicated.
Conclusion
The incident was resolved on the quickly after it was reported, with minimal long-term impact on our customers. We appreciate the patience and cooperation of all parties involved and remain committed to providing high-quality, reliable VoIP services.