And boosted their company culture
“For us [at Honeycomb], the cost [of AWS Lambda] is a function of three things,” wrote engineering manager Ben Hartshorne on the Honeycomb blog, describing his team’s recent cost-cutting efforts. According to Ben:
- ARM architecture for the Lambda was cheaper than x86
- Smaller Lambdas with less RAM were cheaper than larger ones
- Lambdas that finished quicker were cheaper than slower ones
After identifying the three cost-saving variables of architecture, RAM size, and duration, the Honeycomb team needed to capture this data. Their goal was to identify and segment AWS costs per customer and per query.
The team instrumented Honeycomb’s query engine to send telemetry of a customer-specific identifier along with the source of the query directly to the Honeycomb app, segregating queries by user action.
The result was that each of Honeycomb’s Lambdas logged its customer id, user action (for example, manually running a query or loading a dashboard), configured memory, and other information about the query.
Using this equivalent of a black box flight recorder, they calculated cost as duration (in milliseconds) times a price per architecture multiplied by the memory size divided by the maximum size of a Lambda (10240GB).
Based on their initial set of data, they determined that arm64 architecture cost $1.33e-7 per GB, while x86 architecture cost $1.67e-7 per GB. Multiplying either of these numbers by RAM and duration equals cost.
Not only were their calculations (using a “derived column” in Honeycomb to automate things) useful for tracking costs, but they identified customers who incurred Lambda bills significantly larger and smaller than average.
The last thing the team did before actually cutting costs was to set up a Slack trigger that automated notifications whenever any individual “big spender” customer spent more than $100 on AWS Lambda in a single day.
The team’s biggest question was whether “our chosen Lambda size was correct,” wrote Ben. To find out, they used AWS Lambda Power Tuning to experiment with tons of different configurations and then compared costs.
An image in Ben’s article shows that Honeycomb’s Lambda costs dropped from $10,272.67 to $5,200.15 on what I presume is a monthly basis, meaning they had a drop of nearly 50% in their AWS Lambda spending.
“Woooo! 50% lower Lambda costs while providing a consistent customer experience is a big win.”
That’s some serious savings, all from what looks like to be a straightforward monitorability project. While they set up Slack notifications (very cool!) for overspending, the real win came from tracking and experimentation.
In addition to what I’m sure was a morale boost for the team after they saved the company $60,000 a year (by my estimate), this project also had a lasting impact on the company culture at Honeycomb.
“Having the effects [on AWS spending] reflected in real dollars,” Ben wrote, “makes it easier to [know] when […] optimization work [is] really worth the effort [for our developers].” Now that’s what I call management!
By rigorously measuring and tracking their AWS cost data, and then tweaking their Lambda strategy methodically, Honeycomb didn’t just save costs — they also identified objective measures for how to best use developer time.
Great job to everyone at the Honeycomb team, and I applaud how they’ve incorporated cost saving into their culture. I hope this article inspires you to slash (or at least track) your own spending on AWS Lambdas.