SECRET OF CSS

Drop pattern with timeout in Go with context and channels


A brief look at this extremely useful functionality

Usually, server-side systems are sized to process the expected volume of incoming requests. If the design is right, clients sending requests to the server receive the response within a delay considered acceptable for the specific system under normal working conditions.

However, sometimes working conditions can turn abnormal. A database can degrade its performance, the network slows down, and a sudden peak of requests comes in.

In such cases, requests start accumulating in queues, clients experience delays in the responses, and eventually, the entire system may collapse.

In such cases, the system is “resisting the flow of data.” This phenomenon is called backpressure.

This is the same phenomenon that happens on highways when traffic is heavy. Cars flow at a reasonable speed as long as there are no accidents or the density of cars is below a certain threshold. As soon as an accident occurs, cars start forming queues. And these queues will stay long, even after the road has been cleared, with all the annoyance we may have experienced sitting blocked in the middle of a highway.

There is nothing we can do with cars and highways, but with IT systems, we can find strategies to reduce the damage when abnormal conditions occur. One such strategy is the so-called “drop pattern with timeout,” which means cancelling a request if such request has been waiting for too long.

Let’s see in more detail how this pattern works, and let’s illustrate a possible implementation of such a pattern using Go.

Let’s consider an ideal system where the server is sized to respond to a constant flow of incoming requests.

In other words, has the following features:

  • the server can serve a request with an execution time of E
  • requests arrive with an interval equal to E
1*lAvz8M rTX9NKnkbYBVJ6Q
A system exactly sized to respond to requests with a specified delay E

In such a situation, if the server stops responding for a period of duration, as soon as it comes live again, there will be a queue of requests waiting to be served. Since, in this ideal scenario, each request takes E time to be executed and produce a response, after the stop, clients will experience a response time of E+S, which means a delay of S if compared to the normal condition. What is also important is that this delay will remain the same for all requests coming to the server after the problem arises and will never be reduced.

1*3UKiNZXyZ9g13gX7CIseIA
A stop in the server generates a delay which is propagated over time

This is an extreme case but is useful to illustrate the fact that, if we do nothing, the negative effects of an incident can be propagated over time much after the incident has been solved (you may be stuck in a queue on a highway hours after the accident has been solved).

One of the common strategies to mitigate these prolonged negative effects is to introduce timeouts. If a request remains in the queue for more than a certain limit (the timeout), the request is terminated and removed from the queue.

This comes at the cost of having some requests fail, and some clients receive a timeout error message but it brings the benefit of limiting the delay for all the others when normal working conditions are re-established.

1*F3a5tjpp2jMFNo2mzEW85Q
A timeout represents the maximum response time that a request can have

In our ideal case, if we set a timeout equal to TO, requests coming in after the incident has been resolved will experience a response time at most equal to TO (which is an improvement over the previous case if TO < E+S).

This is how the “drop pattern with timeouts” works.

Implementing the “drop pattern with timeout” in Go is possible using channels, Context, and goroutines. All the code for this pattern implementation can be found in this repo.

The elements at play

To see the “drop pattern with timeout” in action, we need three basic elements:

  • A server that can process requests with a certain throughput
  • A generator of requests that we want the server to process
  • Something, like a Waiting Room that sits in the middle and implements the “drop pattern with timeout” logic
1*ozQ0a7V0cUOwdMduAfz8rQ
The elements at play

Let’s see these elements in more detail.

Worker pool server

We will use a pool of workers to implement a server that can serve requests concurrently (the fact that the server is a pool of workers is not relevant to the patterns, it is just a common way to implement a server).

The worker pool is implemented following a standard Go pattern. A set of goroutines, each implementing one worker, are started. Each worker loops on the receiving end of the input channel of the pool, reqCh.

As soon as a worker receives a request, it starts processing it. When the processing of a request is complete, the worker loops back to the receive on the input channel reqCh ready to start processing a new request.

1*UYTS jK4WXzd3G VDVQN0Q
Worker pool

The code implementing the worker pool can be found here.

The implementation of the “drop pattern with timeout”

The implementation of the core of “drop pattern with timeout” is based on context with timeout and on the use of the powerful select on channels statement. In the example code, the logic is implemented by the WaitingRoom type.

1*plhlYeTXLOZ iIjes0GiPQ
The Waiting room implementing the “drop pattern with timeout”

Requests that have to enter the Waiting room are sent to its input channel inCh. The goroutine that drives the work of the Waiting room reads from this channel and, for each request, starts a new goroutine launching the sendOrDrop function where the core logic of the “drop pattern with timeout” is implemented.

For each request, the sendOrDrop function creates a context with timeout and then runs the select on channels statement.

In the first case of the select checks if a send on the input channel of the Worker pool, the reqCh, completes. If it completes, one worker in the pool has received the request and will process it.

The second case using select checks if the Done channel of the context is closed. If this occurs, it means that the timeout has expired before the request could be received by the pool, and therefore, the request has to be dropped.

The code implementing the WaitingRoom can be found here.

The request generator can be anything that generates requests, from a REST endpoint to a queue receiving messages.

In our example, for the sake of simplicity, we have a simple Go function that generates requests and sends them to the WaitingRoom.

The “drop pattern with timeout” is therefore implemented using two channels, one as input for the WaitingRoom and one as input for the server (and as output for the WaitingRoom itself), the context with timeout and the select on channels statement, all powered by the concurrency provided by the goroutines.

1*L8a CATBkXVTA4ccCz2V3w

A working example of the “drop pattern with timeout” is implemented in this repo.



News Credit

%d bloggers like this: