The Story Nobody Tells You
It was a Tuesday morning at 10 AM when Slack exploded with alerts.
Our payment API was down. Not crashed. Not slow. Down.
I pulled up the logs and saw it immediately: OutOfMemoryError on every server. The heap was full. The garbage collector was running constantly. Everything was fighting for the last bits of RAM.
The weird part? Traffic was completely normal. No DDoS attack. No unusual spike. Everything looked fine until you looked at one thing: the rate limiter.
We had implemented rate limiting the "textbook" way. Create a map, track IPs, count requests. Super simple. Seemed brilliant at 2 AM when I coded it.
What I didn't account for was that map never cleaned up after itself.
Every unique IP address that hit our API got an entry in a ConcurrentHashMap. Scrapers with rotating IPs? That's 10,000 entries. A crawler farm testing different user agents? Another 50,000 entries. Each entry sitting in memory forever, never cleaned up, never garbage collected.
Three weeks later, we had 2 million IP entries in that map. The heap was maxed out. The JVM was dying.
This is what nobody tells you about in-memory rate limiting. It's not the algorithm that's the problem. It's memory management.
The Textbook Example That Breaks Production
Here's the code I shipped. It looks reasonable. It probably shows up in a dozen tutorials.
// DON'T DO THIS IN PRODUCTION
private final Map<String, Integer> requestCounts = new ConcurrentHashMap<>();
private final Map<String, Long> lastResetTime = new ConcurrentHashMap<>();
public boolean isAllowed(String clientIp) {
long now = System.currentTimeMillis();
long lastReset = lastResetTime.getOrDefault(clientIp, now);
// Reset every minute
if (now - lastReset > 60_000) {
requestCounts.put(clientIp, 0);
lastResetTime.put(clientIp, now);
}
int count = requestCounts.getOrDefault(clientIp, 0);
if (count >= 100) {
return false; // Limit exceeded
}
requestCounts.put(clientIp, count + 1);
return true;
}
This code has a fatal flaw hidden in plain sight. Can you spot it?
ConcurrentHashMap never deletes anything. Every IP that hits your API gets an entry that stays in memory forever. Under normal traffic, this is fine. You get maybe 1,000 active IPs at any time. But under real-world conditions:
- A web scraper rotates through 50 different IPs
- A DDoS attack with spoofed IPs sends traffic from 100,000 different sources
- A buggy client library retries with slightly different headers
- A penetration tester probes your API with automation
Suddenly you have millions of entries. Your heap fills up. The garbage collector can't keep up. Your entire service becomes unresponsive.
This happened to us. We watched it happen in real-time. The only solution that day was to restart every server and implement proper cache eviction.
Why This Matters More Than You Think
Rate limiting isn't optional anymore. It's how you:
- Prevent your API from being hammered by bots
- Protect customers from accidentally hitting your service too hard
- Ensure fair resource allocation across all users
- Meet SLA requirements by avoiding overload
But rate limiting done wrong becomes a liability. It causes the exact problem you're trying to prevent.
I've talked to developers who stopped using rate limiting entirely because they worried it would crash their system. That's worse. That leaves you completely unprotected.
The solution isn't to skip rate limiting. It's to implement it correctly.
The Clean Production Solution
The fix requires two pieces:
First, the Token Bucket Algorithm. Instead of hard-resetting counters every minute, you use a bucket metaphor. Imagine a bucket that holds tokens. Every request consumes one token. Tokens refill at a constant rate. If the bucket is empty, the request is rejected. This smooths out traffic bursts naturally.
Second, Self-Cleaning Cache. Use Caffeine, which automatically removes stale entries. If an IP hasn't made requests in 10 minutes, it gets deleted. If you exceed a maximum size, the least-recently-used entries are evicted.
Together, these solve the problem completely.
Setting It Up: Step 1 - Add Dependencies
First, add Bucket4j (token bucket implementation) and Caffeine (cache with expiration):
<dependency>
<groupId>com.bucket4j</groupId>
<artifactId>bucket4j-core</artifactId>
<version>8.10.1</version>
</dependency>
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
<version>3.1.8</version>
</dependency>
That's it. Two dependencies. No heavy infrastructure needed. No Redis cluster. No microservice overhead.
Step 2 - The Rate Limiter Service
This is where the magic happens. The service creates a bucket for each IP and manages eviction:
package com.example.ratelimit.service;
import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import io.github.bucket4j.Bandwidth;
import io.github.bucket4j.Bucket;
import io.github.bucket4j.Refill;
import org.springframework.stereotype.Service;
import java.time.Duration;
@Service
public class RateLimiterService {
// Purges inactive IPs after 10 mins; capped at 50k active entries
private final Cache<String, Bucket> bucketCache = Caffeine.newBuilder()
.maximumSize(50_000)
.expireAfterAccess(Duration.ofMinutes(10))
.build();
public Bucket resolveBucket(String clientIp) {
return bucketCache.get(clientIp, this::createBucket);
}
private Bucket createBucket(String clientIp) {
// Capacity: 20 tokens; Refill: 20 tokens every 1 minute
Refill refill = Refill.greedy(20, Duration.ofMinutes(1));
Bandwidth limit = Bandwidth.classic(20, refill);
return Bucket.builder()
.addLimit(limit)
.build();
}
}
Let me break down what's happening:
Caffeine Configuration:
maximumSize(50_000): Never store more than 50,000 IPs in memoryexpireAfterAccess(Duration.ofMinutes(10)): Remove any IP that hasn't made a request in 10 minutes
This means your cache stays bounded. As new IPs come in, old ones get deleted. Memory usage is predictable.
Bucket Configuration:
20 tokens capacity20 tokens refill every 1 minute
This means each IP can make 20 requests per minute. After that, they have to wait until the bucket refills.
You can adjust these numbers based on your needs. Want stricter limits? Use 10 tokens per minute. Want more generous? Use 100 tokens per minute.
Step 3 - Intercepting Requests with a Filter
Picture Description: Diagram showing HTTP request flow through the rate limit filter. Request arrives → Filter checks bucket → Either allow or reject with 429 status.
Now you need to actually use this rate limiter. The cleanest way is an HTTP filter that runs on every request:
package com.example.ratelimit.filter;
import com.example.ratelimit.service.RateLimiterService;
import io.github.bucket4j.Bucket;
import io.github.bucket4j.ConsumptionProbe;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
@Component
public class RateLimitFilter extends OncePerRequestFilter {
private final RateLimiterService rateLimiterService;
public RateLimitFilter(RateLimiterService rateLimiterService) {
this.rateLimiterService = rateLimiterService;
}
@Override
protected void doFilterInternal(HttpServletRequest request,
HttpServletResponse response,
FilterChain filterChain) throws ServletException, IOException {
String clientIp = extractClientIp(request);
Bucket bucket = rateLimiterService.resolveBucket(clientIp);
// Try to consume 1 token
ConsumptionProbe probe = bucket.tryConsumeAndReturnRemaining(1);
if (probe.isConsumed()) {
// Request allowed
response.setHeader("X-Rate-Limit-Remaining",
String.valueOf(probe.getRemainingTokens()));
filterChain.doFilter(request, response);
} else {
// Rate limit exceeded
long waitSeconds = Math.max(1,
TimeUnit.NANOSECONDS.toSeconds(probe.getNanosToWaitForRefill()));
response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
response.setContentType("application/json");
response.setHeader("Retry-After", String.valueOf(waitSeconds));
String jsonResponse = String.format(
"{\"status\":429,\"error\":\"Too Many Requests\"," +
"\"message\":\"Rate limit exceeded. Try again in %d seconds\"," +
"\"retryAfterSeconds\":%d}",
waitSeconds, waitSeconds);
response.getWriter().write(jsonResponse);
}
}
private String extractClientIp(HttpServletRequest request) {
String xForwardedFor = request.getHeader("X-Forwarded-For");
if (xForwardedFor != null && !xForwardedFor.isBlank()) {
return xForwardedFor.split(",")[0].trim();
}
return request.getRemoteAddr();
}
}
Picture Description: Code snippet showing the two paths - successful request (green checkmark) vs rate limited request (red X with 429 response).
What's happening here:
The Filter Chain: Every HTTP request passes through this filter. It runs once per request (hence OncePerRequestFilter).
Consuming Tokens: bucket.tryConsumeAndReturnRemaining(1) tries to consume 1 token. This either succeeds (the request is allowed) or fails (the bucket is empty).
The Client IP: This is critical. You need to extract the real client IP, not the proxy IP. If you're behind Cloudflare or AWS ALB, the client IP is in the X-Forwarded-For header. The code checks this first, falls back to getRemoteAddr() if the header is missing.
429 Response: When rate limited, return HTTP 429 (Too Many Requests). Include the Retry-After header so clients know when to retry. Return JSON so mobile apps and JavaScript can parse it.
What Happens Under the Hood (Real-World Example)
Let's walk through a real scenario:
Time 0:00 - IP 192.168.1.1 makes a request. Bucket doesn't exist, so it's created with 20 tokens. The filter consumes 1 token. Bucket now has 19 tokens. Request is allowed.
Time 0:01 - Same IP makes 19 more requests. Each consumes a token. Bucket is now empty (0 tokens).
Time 0:02 - Same IP makes another request. Bucket is empty. tryConsumeAndReturnRemaining() fails. The system calculates how long until the next refill (58 more seconds). Response is 429. Client gets told to retry in 58 seconds.
Time 1:00 - The 1-minute window expires. The bucket refills completely. Bucket has 20 tokens again. The same IP can make 20 more requests.
Time 10:00 - If the IP hasn't made any requests for 10 minutes, Caffeine automatically deletes the bucket from the cache. Memory is freed.
Time 10:01 - A new IP (or a different client from the same IP) makes a request. A brand new bucket is created for them. This cycle repeats.
This is elegant. No manual cleanup. No memory leaks. No OutOfMemoryError.
Customizing for Your Use Case
The numbers I chose (20 requests per minute, 10-minute expiration) are arbitrary. You need to adjust them for your API.
Conservative (useful for payment processing, sensitive operations):
Refill refill = Refill.greedy(5, Duration.ofMinutes(1)); // 5 requests per minute
Bandwidth limit = Bandwidth.classic(5, refill);
Moderate (good for most APIs):
Refill refill = Refill.greedy(50, Duration.ofMinutes(1)); // 50 requests per minute
Bandwidth limit = Bandwidth.classic(50, refill);
Generous (for public read-only APIs):
Refill refill = Refill.greedy(500, Duration.ofMinutes(1)); // 500 requests per minute
Bandwidth limit = Bandwidth.classic(500, refill);
Picture Description: Comparison chart showing different rate limiting strategies (Conservative vs Moderate vs Generous) with example use cases for each.
You can also adjust Caffeine settings:
// Stricter memory constraints
.maximumSize(10_000) // Fewer active IPs
.expireAfterAccess(Duration.ofMinutes(5)) // Shorter expiration
// More generous
.maximumSize(100_000) // More active IPs
.expireAfterAccess(Duration.ofMinutes(30)) // Longer expiration
Start with moderate settings. Monitor memory usage. Adjust from there.
Testing This Locally
Here's a quick test to verify it works:
# Make 5 rapid requests
for i in {1..5}; do
curl -i http://localhost:8080/api/test
done
# Next request will be rate limited
curl -i http://localhost:8080/api/test
# Output:
# HTTP/1.1 429 Too Many Requests
# Retry-After: 58
# {"status":429,"error":"Too Many Requests",...}
You'll see the first 5 requests succeed (or however many you configured). Request 6 gets a 429. The Retry-After header tells you how many seconds to wait.
Real Production Problems I've Seen
Problem 1: Trusting the Client IP Wrong
If you use request.getRemoteAddr() and you're behind a reverse proxy, every request will have the proxy IP. They'll all share the same bucket. One user can exhaust limits for everyone.
Solution: Always check X-Forwarded-For first. Configure your proxy to add this header.
Problem 2: Rate Limiting Too Aggressively
I once saw a company rate limit at 10 requests per second. Legitimate batch operations hit the limit. Support tickets exploded.
Solution: Monitor your actual usage patterns. See what normal clients do. Set limits 2-3x higher than your 95th percentile.
Problem 3: Not Informing Clients
When rate limited, some APIs just return 429 with no explanation. Clients don't know when to retry.
Solution: Always include the Retry-After header. Include a helpful error message. Give clients a way to upgrade to higher limits.
Problem 4: Running Out of Memory Anyway
Even with Caffeine, if you set maximumSize() too low on a high-traffic API, eviction might not keep up.
Solution: Monitor cache hit rate and eviction rate. If eviction rate is high, increase maximum size. If memory is still an issue, move to distributed rate limiting with Redis.
When to Move to Redis
In-memory rate limiting works great until it doesn't. When should you graduate to Redis?
You need Redis when:
- Multiple servers: If you're running your app on 5 servers, rate limits aren't synchronized. One user could hit 100 requests per minute by distributing 20 to each server.
- Extreme traffic: High-frequency requests from many IPs can make Caffeine eviction the bottleneck.
- Complex rules: You need per-user limits, per-resource limits, tiered limits, or limits that depend on authentication status.
For those cases, Redis is cleaner:
// Redis example (high level)
@Service
public class RedisRateLimiter {
@Autowired
private RedisTemplate<String, String> redisTemplate;
public boolean isAllowed(String clientIp) {
String key = "rate_limit:" + clientIp;
Long count = redisTemplate.opsForValue().increment(key);
if (count == 1) {
redisTemplate.expire(key, Duration.ofMinutes(1));
}
return count <= 100;
}
}
But for most APIs, in-memory with Caffeine is simpler and faster.
Copy Code Function (JavaScript for Your Blog)
Here's a JavaScript snippet you can add to copy code blocks:
Picture Description: Screenshot of a code block with a "Copy" button in the top-right corner, showing the button state before and after clicking (with "Copied!" feedback).
<script>
// Add copy button to code blocks
document.addEventListener('DOMContentLoaded', function() {
document.querySelectorAll('pre').forEach(function(block) {
// Create copy button
const button = document.createElement('button');
button.innerHTML = '📋 Copy';
button.className = 'copy-button';
button.style.cssText = `
position: absolute;
top: 8px;
right: 8px;
padding: 6px 12px;
background: #2d3748;
color: #e2e8f0;
border: none;
border-radius: 4px;
cursor: pointer;
font-size: 13px;
font-weight: 500;
transition: all 0.2s ease;
`;
button.onmouseover = function() {
this.style.background = '#1a202c';
};
button.onmouseout = function() {
this.style.background = '#2d3748';
};
// Handle click
button.onclick = function() {
const code = block.innerText;
navigator.clipboard.writeText(code).then(function() {
button.innerHTML = '✓ Copied!';
button.style.background = '#22863a';
setTimeout(function() {
button.innerHTML = '📋 Copy';
button.style.background = '#2d3748';
}, 2000);
}).catch(function() {
alert('Failed to copy code');
});
};
// Wrap in container for positioning
block.style.position = 'relative';
block.appendChild(button);
});
});
</script>
Add this to your blog template. Every code block gets a copy button automatically. When users click it, the code is copied to their clipboard and they get visual feedback.
Key Takeaways
Never use unbounded maps for tracking anything time-based. ConcurrentHashMap doesn't clean up after itself. You will run out of memory. This isn't a theoretical concern. This is what happened to our production system.
Cache with expiration is non-negotiable. Caffeine is the gold standard in Java for this. It's lightweight, fast, and handles eviction automatically.
Extract client IP correctly. If you're behind a proxy (and most of you are), use X-Forwarded-For. Getting this wrong means your rate limiting doesn't work at all.
Give clients clear feedback. Return 429. Include Retry-After. Return JSON so clients can parse it. This is how they know what happened and when to retry.
Monitor your rate limiter. Watch cache size, hit rate, eviction rate. These metrics tell you whether your settings are right.
Start in-memory, move to Redis if needed. For single-server or low-traffic APIs, Caffeine + Bucket4j is plenty. For distributed systems, Redis is cleaner.
What I Learned From Breaking Production
That Tuesday morning when the API went down, we learned something valuable. It's easy to write code that works 99% of the time. It's hard to write code that works when edge cases hit.
Rate limiting seems simple. It's not. The simplest implementation breaks under load. The correct implementation requires thinking about memory management, client IP extraction, distributed systems, and operational monitoring.
This is why production engineering is different from writing tutorials. Tutorials show you the happy path. Production shows you what happens when everything goes wrong simultaneously.
The code I've shown you here is what we run now. It's been through the fire. It's handled millions of requests. It hasn't crashed once.
Use it. Adjust it for your needs. Monitor it. And maybe you won't get that 2 AM page about OutOfMemoryError.