I have been experimenting with running TeamCity in AWS, using the CloudFormation stack provided by JetBrains. This stack uses Docker images from JetBrains and runs them in AWS Elastic Container Service.

However the default configuration does not allow Docker-in-Docker builds. This is the situation where a TeamCity agent, itself running in a Docker container, needs to run a build step with Docker or Docker Compose.

The page for the JetBrains Docker image for agents gives two options for starting the agent container from the command line:

• Docker from the host
• Run in privileged mode

This post is about how to modify the JetBrains CloudFormation template to start the agent container in those two ways.

Docker from the host

This technique maps /var/run/docker.sock from the Docker host into the running container. Declare a host volume and mount it in the container (see comments):

  AgentTaskDefinition:
      Type: AWS::ECS::TaskDefinition
      Condition: ShouldLaunchAgents
      DependsOn:
        - PublicLoadBalancer
        - TCServerNodeService
      Properties:
        PlacementConstraints:
          - Type: memberOf
            Expression: attribute:teamcity.node-responsibility == buildAgent
        # Define the host volume to map.
        Volumes:
          - Name: "dockerSock"
            Host:
              SourcePath: "/var/run/docker.sock"
        ContainerDefinitions:
          - Name: 'teamcity-agent'
            Image: !Join [':', ['jetbrains/teamcity-agent', !Ref 'TeamCityVersion']]
            Cpu: !Ref AgentContainerCpu
            Memory: !Ref AgentContainerMemory
            Essential: true
            Environment:
              - Name: SERVER_URL
                Value: "https://teamcity.tawh.net"
            LogConfiguration:
              LogDriver: 'awslogs'
              Options:
                awslogs-group: !Ref ECSLogGroup
                awslogs-region: !Ref AWS::Region
                awslogs-stream-prefix: 'aws/ecs/teamcity-agent'
            # Mount the host volume in the container.
            MountPoints:
              - ContainerPath: "/var/run/docker.sock"
                SourceVolume: "dockerSock"

I have used this method successfully.

Run in privileged mode

Set privileged mode in the ECS Task defintion for the agent by adding Privileged: true to the container definitions:

  AgentTaskDefinition:
      Type: AWS::ECS::TaskDefinition
      Condition: ShouldLaunchAgents
      DependsOn:
        - PublicLoadBalancer
        - TCServerNodeService
      Properties:
        PlacementConstraints:
          - Type: memberOf
            Expression: attribute:teamcity.node-responsibility == buildAgent
        ContainerDefinitions:
          - Name: 'teamcity-agent'
            Image: !Join [':', ['jetbrains/teamcity-agent', !Ref 'TeamCityVersion']]
            Cpu: !Ref AgentContainerCpu
            Memory: !Ref AgentContainerMemory
            # Run this container in privileged mode.
            Privileged: true
            Essential: true
            Environment:
              - Name: SERVER_URL
                Value: !GetAtt [PublicLoadBalancer, DNSName]
            LogConfiguration:
              LogDriver: 'awslogs'
              Options:
                awslogs-group: !Ref ECSLogGroup
                awslogs-region: !Ref AWS::Region
                awslogs-stream-prefix: 'aws/ecs/teamcity-agent'

I have not tried this method yet.

A while back I worked on a Spring Boot application that stores and works with Swagger files. It has a controller that needs to capture file paths at the end of request URIs.

We need to extract three variables: project, repo and path, where path may traverse multiple layers. Some examples:

This naive attempt at a solution does not work:

@GetMapping(path = "/swagger/{project}/{repo}/{path}")
public ResponseEntity<String> swaggerFile(@PathVariable("project") String project,
                                          @PathVariable("repo") String repo,
                                          @PathVariable("path") String path) {

With this, when using the first URI, the path variable gets the value swagger without the file extension. The second URI does not match at all.

The file extension can be captured by using a regular expression, for example:

@GetMapping(path = "/swagger/{project}/{repo}/{path:.+}")

Here, given the first URI, the path variable gets the value swagger.yaml as desired. But the second URI still does not match at all.

There is no simple way to extract a deep path into a single variable. I think this is because the Spring classes split the path into segments using slash characters before matching each segment with a regular expression.

A working solution is to get the entire path from the servlet request object and parse it locally.

@GetMapping(path = "/swagger/{project}/{repo}/**")
public ResponseEntity<String> swaggerFile(@PathVariable("project") String project,
                                          @PathVariable("repo") String repo,
                                          HttpServletRequest request) {
    String prefix = String.format("/swagger/%s/%s", project, repo);
    String path = request.getRequestURI().substring(prefix.length() + 2);
    // Use path
}

I have been working with Kotlin recently and really enjoying it compared to Java.

A really useful feature is the ability to define extension functions on existing classes.

The built-in scope functions let, run, also, apply and with are really handy and are described well in Coping with Kotlin scope functions.

It can be difficult to remember which one to use. It helps me to think first of whether the function returns the receiving object or the value of the lambda.

Return the receiver

The functions also and apply return the receiver. For example, here is a natural way to get a Jackson object mapper and configure it in one expression:

val mapper = jacksonObjectMapper().apply {
    configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
    configure(DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES, false)
}

Within the lambda, this is set to the object returned by jacksonObjectMapper().

Similarly, also returns the receiver and passes it to the lambda as an argument instead of as self. This feels natural for side-effects that do not change the receiver, like logging:

return serviceResponse.also {
    log.info("Returning service response: $it")
}

Return the value of the lambda

The functions let, run and with return the lambda. They can be used as idiomatic null checks, for example:

val result = value?.let { transform(it) } ?: defaultValue

This has the meaning, If value is not null, transform it; else return a default value.

run is similar, except that it applies directly to the receiver (available as this in the lambda).

There is a with function like that in Visual Basic, except that it returns the value of the lambda (which can be ignored).

with(something) {
    setValue(newValue)
}

Quick mnemonic

• If the scope function starts with a (also, apply) it returns the receiver;
• else (let, run, with) it returns the lambda.

I wanted to write tests for the retryAxios function described in my recent post about retrying back-end service calls.

Rather than mocking Axios itself with either moxios or Axios mock adapter, I wanted a simple mock of an Axios function. The mock needs to be configurable to return a known sequence of responses, where each response is a promise that either resolves or rejects.

My solution was to create a function that returns a closure:

const mockAxios = (...returns) => {
  const iter = returns[Symbol.iterator]();
  return () => {
    const { done, value } = iter.next();
    if (done) {
      throw new Error('Testing error: mockAxios iterator done');
    }
    return mockRequest(value);
  };
};

• The rest parameter returns contains an array that specifies how each function call will return.
• mockAxios returns a closure that iterates through returns, calling mockRequest for each value in that iterable.
• If the iterable is exhausted the test itself is in error.

mockRequest is a bare-bones implementation of the behaviour of Axios functions:

class HttpError extends Error {
  constructor(response, ...params) {
    super(...params);
    this.response = response;
  }
}

class NetError extends Error {
  constructor(code, ...params) {
    super(...params);
    this.code = code;
  }
}

const mockRequest = async (retVal) => {
  if (Number.isInteger(retVal)) {
    const response = { status: retVal };
    if (retVal >= 400 && retVal <= 599) {
      throw new HttpError(response);
    }
    return response;
  }
  throw new NetError(retVal);
};

• If retVal is an HTTP success status code, it returns a response object with that code.
• If retVal is an HTTP failure status code, it throws a simplified HTTP error object with that code.
• Otherwise it returns a simplified network error object with code retVal.
• mockRequest is async so it wraps the return value in a resolved or rejected promise.

Here is a simple example of using mockAxios in a Jest test case, with inline comments for annotation:

test('retryAxios should return first successful call if within retry limit', async () => {
  // Calling expect.assertions() is useful when testing asynchronous code.
  expect.assertions(1);
  const data = await retryAxios(
      // The iterable with delays after retriable failures.
      [5, 5],
      // A mock Axios function that returns server failure, then network failure,
      // then success.
      mockAxios(500, 'ECONNRESET', 301)
  );
  expect(data.status).toBe(301);
});

The test code has been added to my Gist for retryAxios.

Recently I worked on a single-page ES6 application that calls many back-end services using the axios promise-based library.

Back off!

We wanted to implement the back-off pattern to retry service calls when they suffer transient failures. With back-off, the client does not retry immediately. Instead, it sleeps for a short delay before retrying again. If the same request continues to fail, the client sleeps for progressively longer delays before each retry.

There is the axios-retry plugin that adds retry capability to Axios but it works by counting the number of retries without providing any back-off delays.

A solution

We implemented retry with back-off by looping through an iterable of delay periods in milliseconds:

const retryAxios = async (delays, axiosFunc, ...axiosArgs) => {
  // Extract the iterator from the iterable.
  const iterator = delays[Symbol.iterator]();
  while (true) {
    try {
      // Always call the service at least once.
      return await axiosFunc(...axiosArgs);
    } catch (error) {
      const { done, value } = iterator.next();
      if (!done && isRetriable(error)) {
        await sleep(value);
      } else {
        // The error is not retriable or the iterable is exhausted.
        throw error;
      }
    }
  }
};

Notes:

• delays is an ES6 Iterable. An array is a simple example.

• A while loop is used so the service is called at least once. This requires the iterable to be unpacked manually so the done attribute of the object returned by the iterator’s next method can be checked after the service is called.

• The sleep function is the usual promise-based mechanism to wait for a time period:

const sleep = delay => new Promise(resolve => setTimeout(resolve, delay));

Transient failures

We consider network and server errors as transient, thus suitable to be retried:

// Network error rules from https://github.com/softonic/axios-retry.
const isNetworkError = error => (
  !error.response
  && error.code
  && Boolean(error.code)
  && error.code !== 'ECONNABORTED'
  && isRetryAllowed(error)
);
const isServerError = error => (
  error.response
  && error.response.status >= 500
  && error.response.status <= 599
);
const isRetriable = error => isNetworkError(error) || isServerError(error);

The isRetryAllowed function is from the is-retry-allowed module: it checks the error code against lists of values that can be retried and those that cannot.

Back-off implementations

A simple implementation of delays is an array, for example:

const delays = [50, 50, 100, 100, 200, 500, 1000, 1000, 1000, 1000];

This example provides ramp-up to repeated 1-second delays.

It is common to use exponentially-increasing delays between retries. This ExponentialDelays class returns an iterable of a specified number of delay times in milliseconds:

class ExponentialDelays {
  constructor(initialDelay, retryCount) {
    this.initialDelay = Math.max(1, initialDelay);
    this.retryCount = Math.max(0, retryCount);
  }
  // Iterator generator
  * iterator() {
    let delay = this.initialDelay;
    for (let retry = 0; retry < this.retryCount; retry += 1) {
      yield delay;
      delay *= 2;
    }
  }
  // Implement iterable protocol.
  [Symbol.iterator]() {
    return this.iterator();
  }
}

Some examples

  // GET with simple array of delays.
  const delays = [50, 50, 100, 100, 200, 200, 500, 1000, 1000, 1000];
  const { data } = await retryAxios(delays, axios.get, 'https://api.example.com/people/12345');

  // POST with exponential set of delays.
  const delays = new ExponentalDelays(10, 6);
  const { data } = await retryAxios(delays, axios.post, 'https://api.example.com/people', {
    firstName: 'Osbert',
    lastName: 'Sitwell',
  });

  // Using a custom axios instance as a function with 10 equally-spaced delays.
  const customAxios = axios.create({
   timeout: 30000,
   withCredentials: true,
   maxRedirects: 0,
  });
  const delays = new Array(10).fill(100);
  const { data } = await retryAxios(delays, customAxios, {
    method: 'put',
    url: 'http://api.example.com/people/12345',
    data: {
      firstName: 'Sacheverell',
      lastName: 'Sitwell',
    }
  });

A note on ESLint

The code for retryAxios shown here violates a number of default ESLint rules (no-await-in-loop, no-constant-condition and consistent-return). Comments to selectively disable those rules have been omitted here for clarity, but are present in the complete code in this Gist.