On 19 June 2022, Ran Mochary wrote about how Gong moved non-urgent Datadog alerts to Sentry. Datadog stayed. OpsGenie stayed for pages. Sentry became the inbox for everything else.
Gong’s product records sales calls and deals, so production has to stay up. There is no infrastructure team. Developers rotate as the on-call duty owner, and that week had become a full-time job of reading alerts.
Datadog monitors forwarded to OpsGenie. The duty owner triaged them and handed work to other people. Two problems piled up. Non-urgent and repeating alerts buried the ones that were hurting customers. Transient alerts closed themselves, often overnight, and the duty owner missed them. OpsGenie was a poor inbox for work that could wait.
The counts he gives: 371 Datadog monitors sent alerts to OpsGenie. 316 of those were not urgent. In one week, non-urgent alerts came from 150 different monitors, some of them more than once.
What they tried before the bridge
They tried Datadog’s Amazon SNS integration. The payload was partial, so they still had to call the Datadog API. Server-side encryption on the AWS side stopped that integration.
Webhooks were the next idea. An existing webhook server also handled loads they did not want to mix in, including Zoom’s native recordings. A new webhook server added nothing they could not get from API Gateway and SQS. Those already gave them authentication, rate limits, and encryption.
Sentry was already how they tracked production errors. It can assign an event to a person or a team. It groups events that share a fingerprint, so a repeating monitor becomes one issue. OpsGenie opened a new event every time.
The path they shipped
Datadog’s webhook integration posts to an API Gateway endpoint. AWS WAF sits in front of it. API Gateway writes to SQS. A periodic task on Gong’s servers reads the queue, turns each Datadog event into a Sentry event, and sends it with the Sentry SDK.
Lambda could have read the queue. They used the Gong frameworks they already had.
Urgent monitors still page through OpsGenie. Sentry gets the rest, and those issues can land on the team that owns them, inside a normal work queue. The duty owner is supposed to see only what has to be fixed now.
They do not trust the pipe blindly. A beacon fires a chosen Datadog monitor every 10 minutes. Another periodic task asks the Sentry API whether those events arrived. If they did not, OpsGenie pages someone to fix the bridge.
Some events can die before SQS. API Gateway execution logs and access logs are the debug trail, and CloudWatch logging is off until you turn it on. He points at AWS docs for the gateway-and-queue setup and for enabling those logs. The post says kinks remain. It gives no cost number, and it does not say the 150-monitor week got smaller after the change. The result he claims is that on-call spends its time on urgent alerts, because assignment of the rest is automatic.
What I would copy
Leave the paging tool in place for the urgent set. Gong’s split is “316 monitors are not urgent”, which only works if someone has already marked them. Do that inventory before you build a bridge.
Put a beacon on the bridge. Theirs is 10 minutes, Datadog to Sentry, and the failure path is OpsGenie. A quiet Sentry project looks like a quiet system.
Turn on API Gateway logs before the first real alert. He says events can vanish on the way to SQS, and the log groups are how you see that.
The Datadog → Sentry playbook reads like a product replacement. Gong did not do that replacement. If you follow them, Datadog is still the source of the monitor. Sentry is the inbox. Both bills continue, and the playbook’s “both systems take production actions” warning is the steady state, not a mistake.