Guided Memory Optimization: Practical Guide

by Sabbir Ahmed

In this guide, we have demonstrated how to optimize memory usage when scraping Amazon. It is legal to scrape Amazon for publicly available product prices, titles, and other information...

TL;DR:

Reduce infrastructure costs by optimizing application memory usage. And optimize application memory with pprof and benchmark tests.

Understanding Memory-Hungry I/O Operations

In application development, performing I/O operations on JSON can be memory-intensive, significantly impacting an application's performance and cost, especially in a cloud environment.

Initial State: Memory-Heavy I/O Operations

In the below image, we observe the memory usage profile of an application using io.ReadAll() to handle JSON file operations. Here, io.ReadAll() consumes approximately 54% of the total application memory. While this level of consumption is manageable, there's considerable potential for optimization.

Optimizing Memory Usage: Transition to Buffers

To reduce memory usage, one effective strategy is to minimize reliance on direct I/O operations. My preferred method is using a buffer. Handling a buffer can be complex, particularly in concurrent scenarios, but it enables us to stream bytes efficiently. By utilizing in-memory fixed allocations, we can better predict runtime memory usage.

Benchmark using buffer

In the second image, we see the implementation of the buffer strategy. The memory consumption drops to approximately 36%, resulting in a 20% reduction in overall application memory usage.

Performance Impact and Infrastructure Cost Savings

A critical point to note is that memory optimization does not always have a direct impact on response time/latency, as evidenced by the third image.

Instead, the primary benefit of memory optimization lies in cost reduction. Memory expenses in cloud/hosted solutions increase exponentially. Therefore, reducing memory usage is a fundamental step in decreasing application infrastructure costs.