A brief look at this extremely useful functionality
Usually, server-side systems are sized to process the expected volume of incoming requests. If the design is right, clients sending requests to the server receive the response within a delay considered acceptable for the specific system under normal working conditions.
However, sometimes working conditions can turn abnormal. A database can degrade its performance, the network slows down, and a sudden peak of requests comes in.
In such cases, requests start accumulating in queues, clients experience delays in the responses, and eventually, the entire system may collapse.
In such cases, the system is “resisting the flow of data.” This phenomenon is called backpressure.
This is the same phenomenon that happens on highways when traffic is heavy. Cars flow at a reasonable speed as long as there are no accidents or the density of cars is below a certain threshold. As soon as an accident occurs, cars start forming queues. And these queues will stay long, even after the road has been cleared, with all the annoyance we may have experienced sitting blocked in the middle of a highway.
There is nothing we can do with cars and highways, but with IT systems, we can find strategies to reduce the damage when abnormal conditions occur. One such strategy is the so-called “drop pattern with timeout,” which means cancelling a request if such request has been waiting for too long.
Let’s see in more detail how this pattern works, and let’s illustrate a possible implementation of such a pattern using Go.
Let’s consider an ideal system where the server is sized to respond to a constant flow of incoming requests.
In other words, has the following features:
- the server can serve a request with an execution time of
- requests arrive with an interval equal to
In such a situation, if the server stops responding for a period of duration, as soon as it comes live again, there will be a queue of requests waiting to be served. Since, in this ideal scenario, each request takes
E time to be executed and produce a response, after the stop, clients will experience a response time of
E+S, which means a delay of
S if compared to the normal condition. What is also important is that this delay will remain the same for all requests coming to the server after the problem arises and will never be reduced.
This is an extreme case but is useful to illustrate the fact that, if we do nothing, the negative effects of an incident can be propagated over time much after the incident has been solved (you may be stuck in a queue on a highway hours after the accident has been solved).
One of the common strategies to mitigate these prolonged negative effects is to introduce timeouts. If a request remains in the queue for more than a certain limit (the timeout), the request is terminated and removed from the queue.
This comes at the cost of having some requests fail, and some clients receive a timeout error message but it brings the benefit of limiting the delay for all the others when normal working conditions are re-established.
In our ideal case, if we set a timeout equal to
TO, requests coming in after the incident has been resolved will experience a response time at most equal to
TO (which is an improvement over the previous case if
TO < E+S).
This is how the “drop pattern with timeouts” works.
Implementing the “drop pattern with timeout” in Go is possible using channels, Context, and goroutines. All the code for this pattern implementation can be found in this repo.
The elements at play
To see the “drop pattern with timeout” in action, we need three basic elements:
- A server that can process requests with a certain throughput
- A generator of requests that we want the server to process
- Something, like a Waiting Room that sits in the middle and implements the “drop pattern with timeout” logic
Let’s see these elements in more detail.
Worker pool server
We will use a pool of workers to implement a server that can serve requests concurrently (the fact that the server is a pool of workers is not relevant to the patterns, it is just a common way to implement a server).
The worker pool is implemented following a standard Go pattern. A set of goroutines, each implementing one worker, are started. Each worker loops on the receiving end of the input channel of the pool,
As soon as a worker receives a request, it starts processing it. When the processing of a request is complete, the worker loops back to the receive on the input channel
reqCh ready to start processing a new request.
The code implementing the worker pool can be found here.
The implementation of the “drop pattern with timeout”
The implementation of the core of “drop pattern with timeout” is based on
context with timeout and on the use of the powerful
select on channels statement. In the example code, the logic is implemented by the
Requests that have to enter the Waiting room are sent to its input channel
inCh. The goroutine that drives the work of the Waiting room reads from this channel and, for each request, starts a new goroutine launching the
sendOrDrop function where the core logic of the “drop pattern with timeout” is implemented.
For each request, the
sendOrDrop function creates a
context with timeout and then runs the
select on channels statement.
In the first case of the
select checks if a send on the input channel of the Worker pool, the
reqCh, completes. If it completes, one worker in the pool has received the request and will process it.
The second case using
select checks if the
Done channel of the
context is closed. If this occurs, it means that the timeout has expired before the request could be received by the pool, and therefore, the request has to be dropped.
The code implementing the
WaitingRoom can be found here.
The request generator can be anything that generates requests, from a REST endpoint to a queue receiving messages.
In our example, for the sake of simplicity, we have a simple Go function that generates requests and sends them to the
The “drop pattern with timeout” is therefore implemented using two channels, one as input for the
WaitingRoom and one as input for the server (and as output for the
WaitingRoom itself), the
context with timeout and the
select on channels statement, all powered by the concurrency provided by the goroutines.
A working example of the “drop pattern with timeout” is implemented in this repo.