Testing Javascript with iterables

|

I wanted to write tests for the retryAxios function described in my recent post about retrying back-end service calls.

Rather than mocking Axios itself with either moxios or Axios mock adapter, I wanted a simple mock of an Axios function. The mock needs to be configurable to return a known sequence of responses, where each response is a promise that either resolves or rejects.

My solution was to create a function that returns a closure:

const mockAxios = (...returns) => {
  const iter = returns[Symbol.iterator]();
  return () => {
    const { done, value } = iter.next();
    if (done) {
      throw new Error('Testing error: mockAxios iterator done');
    }
    return mockRequest(value);
  };
};
  • The rest parameter returns contains an array that specifies how each function call will return.
  • mockAxios returns a closure that iterates through returns, calling mockRequest for each value in that iterable.
  • If the iterable is exhausted the test itself is in error.

mockRequest is a bare-bones implementation of the behaviour of Axios functions:

class HttpError extends Error {
  constructor(response, ...params) {
    super(...params);
    this.response = response;
  }
}

class NetError extends Error {
  constructor(code, ...params) {
    super(...params);
    this.code = code;
  }
}

const mockRequest = async (retVal) => {
  if (Number.isInteger(retVal)) {
    const response = { status: retVal };
    if (retVal >= 400 && retVal <= 599) {
      throw new HttpError(response);
    }
    return response;
  }
  throw new NetError(retVal);
};
  • If retVal is an HTTP success status code, it returns a response object with that code.
  • If retVal is an HTTP failure status code, it throws a simplified HTTP error object with that code.
  • Otherwise it returns a simplified network error object with code retVal.
  • mockRequest is async so it wraps the return value in a resolved or rejected promise.

Here is a simple example of using mockAxios in a Jest test case, with inline comments for annotation:

test('retryAxios should return first successful call if within retry limit', async () => {
  // Calling expect.assertions() is useful when testing asynchronous code.
  expect.assertions(1);
  const data = await retryAxios(
      // The iterable with delays after retriable failures.
      [5, 5],
      // A mock Axios function that returns server failure, then network failure,
      // then success.
      mockAxios(500, 'ECONNRESET', 301)
  );
  expect(data.status).toBe(301);
});

The test code has been added to my Gist for retryAxios.

Retrying back-end service calls with back-off

|

Recently I worked on a single-page ES6 application that calls many back-end services using the axios promise-based library.

Back off!

We wanted to implement the back-off pattern to retry service calls when they suffer transient failures. With back-off, the client does not retry immediately. Instead, it sleeps for a short delay before retrying again. If the same request continues to fail, the client sleeps for progressively longer delays before each retry.

There is the axios-retry plugin that adds retry capability to Axios but it works by counting the number of retries without providing any back-off delays.

A solution

We implemented retry with back-off by looping through an iterable of delay periods in milliseconds:

const retryAxios = async (delays, axiosFunc, ...axiosArgs) => {
  // Extract the iterator from the iterable.
  const iterator = delays[Symbol.iterator]();
  while (true) {
    try {
      // Always call the service at least once.
      return await axiosFunc(...axiosArgs);
    } catch (error) {
      const { done, value } = iterator.next();
      if (!done && isRetriable(error)) {
        await sleep(value);
      } else {
        // The error is not retriable or the iterable is exhausted.
        throw error;
      }
    }
  }
};

Notes:

  • delays is an ES6 Iterable. An array is a simple example.

  • A while loop is used so the service is called at least once. This requires the iterable to be unpacked manually so the done attribute of the object returned by the iterator’s next method can be checked after the service is called.

  • The sleep function is the usual promise-based mechanism to wait for a time period:

const sleep = delay => new Promise(resolve => setTimeout(resolve, delay));

Transient failures

We consider network and server errors as transient, thus suitable to be retried:

// Network error rules from https://github.com/softonic/axios-retry.
const isNetworkError = error => (
  !error.response
  && error.code
  && Boolean(error.code)
  && error.code !== 'ECONNABORTED'
  && isRetryAllowed(error)
);
const isServerError = error => (
  error.response
  && error.response.status >= 500
  && error.response.status <= 599
);
const isRetriable = error => isNetworkError(error) || isServerError(error);

The isRetryAllowed function is from the is-retry-allowed module: it checks the error code against lists of values that can be retried and those that cannot.

Back-off implementations

A simple implementation of delays is an array, for example:

const delays = [50, 50, 100, 100, 200, 500, 1000, 1000, 1000, 1000];

This example provides ramp-up to repeated 1-second delays.

It is common to use exponentially-increasing delays between retries. This ExponentialDelays class returns an iterable of a specified number of delay times in milliseconds:

class ExponentialDelays {
  constructor(initialDelay, retryCount) {
    this.initialDelay = Math.max(1, initialDelay);
    this.retryCount = Math.max(0, retryCount);
  }
  // Iterator generator
  * iterator() {
    let delay = this.initialDelay;
    for (let retry = 0; retry < this.retryCount; retry += 1) {
      yield delay;
      delay *= 2;
    }
  }
  // Implement iterable protocol.
  [Symbol.iterator]() {
    return this.iterator();
  }
}

Some examples

  // GET with simple array of delays.
  const delays = [50, 50, 100, 100, 200, 200, 500, 1000, 1000, 1000];
  const { data } = await retryAxios(delays, axios.get, 'https://api.example.com/people/12345');
  // POST with exponential set of delays.
  const delays = new ExponentalDelays(10, 6);
  const { data } = await retryAxios(delays, axios.post, 'https://api.example.com/people', {
    firstName: 'Osbert',
    lastName: 'Sitwell',
  });
  // Using a custom axios instance as a function with 10 equally-spaced delays.
  const customAxios = axios.create({
   timeout: 30000,
   withCredentials: true,
   maxRedirects: 0,
  });
  const delays = new Array(10).fill(100);
  const { data } = await retryAxios(delays, customAxios, {
    method: 'put',
    url: 'http://api.example.com/people/12345',
    data: {
      firstName: 'Sacheverell',
      lastName: 'Sitwell',
    }
  });

A note on ESLint

The code for retryAxios shown here violates a number of default ESLint rules (no-await-in-loop, no-constant-condition and consistent-return). Comments to selectively disable those rules have been omitted here for clarity, but are present in the complete code in this Gist.

Careful what you promise

|

Background

I was working recently on a JavaScript front-end project using ES6 async / await syntax. We created a client to back-end services hosted on AWS Lambda via AWS API Gateway.

The client is successful at fetching temporary auth tokens and credentials from specified endpoints, using the credentials to call signed services, and automatically refreshing tokens or credentials when they are about to expire.

Our code calls services using the Axios library, which returns promises. A simple example method is:

const getData = async (url, config) => {
  const { data } = await axios.get(url, config);
  return data;
}

A refactoring mistake

I was refactoring the function’s code to add AWS signing headers to the config argument before passing it to Axios. (That task was previously done elsewhere.) Given a function addAwsHeaders that adds the headers, I wrote something like this:

const getData = async (url, config) => {
  const { data } = await axios.get(url, addAwsHeaders(url, config));
  return data;
}

But my tests failed with unresolved promises. This is because the addAwsHeaders function is asynchronous (the auth tokens or credentials may be stale and new ones requested) and returns a promise that my code was not resolving.

A correct versions is:

const getData = async (url, config) => {
  const awsHeaders = await addAwsHeaders(url, config);
  const { data } = await axios.get(url, awsHeaders);
  return data;
}

It is important to pass the resolved value from the promise, not the promise itself, to the function.

I voted Yes for marriage equality

|

Today I voted Yes in the Australian postal survey for marriage equality. It was an easy decision. Quite simply, love is love, this is a matter of fundamental human equality, and the details of other people’s love for each other is none of my business.

I voted yes

Is it OK to vote “no” as the naysayers’ slogan goes? Of course, because everybody is free to vote as they choose.

But it is not OK to base your decision on misleading allegations about the ill-effects on children of same-sex parenting, curbs on religious freedoms, or fears about the follow-on effects (to name a few). All of these claims have been refuted in a number of places, if you care to look. Many countries have already accepted marriage equality and those fears have not materialised there.

Do the people making dire claims about the effects of marriage equality really believe the things they say, given their dubious veracity, or is there some underlying, unexpressed reason? I think they actually find ‘non-standard’ sexuality at least unsettling and probably repulsive.

I think there is a spectrum of views from dislike (I won’t do that) to disgust (I don’t like to think about that) to disapproval (I don’t want you do that). Disapproval is a moral judgement about other people’s behaviour. And moral judgement invites people to extend their views about what is, and is not, their own business.

Some handy Git aliases

|

I got some useful Git aliases a while ago and was starting with a new client where I needed to recreate them so I decided the document them here.

git s is useful for short status display, defined like this:

git config --global alias.s 'status --short'

git lg is a short, graphical log display, defined like this:

git config --global alias.lg 'log --oneline --decorate --graph'

Example output is:

Git log output with --decorate

Then I decided to augment it with useful extra information, including colouring.

git config --global alias.lg 'log --graph --format="%C(auto)%h %C(cyan)%cn%C(auto)%d %s %C(magenta)%cr"'

Git log output with --format

The %C(auto) specification does not colour committer name or date, so I coloured those explicitly.

Documentation is here: