Octopus Deploy server in AWS and polling tentacles

|

I am using Octopus Deploy on a current project to deploy to a number of targets in tightly-controlled, on-premises environments. We are using polling tentacles so we don’t need to get ingress firewall rules manually created for every deployment target.

Tentacle-to-server communication

Octopus Deploy server is deployed into AWS and needs to be configured to securely accept connections from polling tentacles:

  • the Octopus web portal on its assigned port
  • the Octopus server for tentacle instructions, usually on port 10943

The first connection is HTTP or HTTPS and can be secured simply in AWS with any load balancer that presents a certificate and offloads TLS, forwarding HTTP requests to the server.

The second connection is HTTPS but must be secured from end to end. On installation, both server and tentacle generate a self-signed certificate, which they use to secure all communication with each other. This means the Octopus Deploy server cannot be deployed behind a device that offloads the TLS certificate.

AWS Load Balancers

The current generation of AWS Elastic Load Balancers come in two types: Application Load Balancers and Network Load Balancers.

Application Load Balancers can route traffic based on host, header, path etc. and are very flexible. But they can only accept HTTP and HTTPS connections and always offload TLS in the latter case.

Network Load Balancers do not support complex routing rules but can offload certificates for some requests and allow TCP passthrough of others. This solution meets our needs:

  • A TLS listener on port 443 offloads the certificate on requests to the web portal, which are forwarded over HTTP.

  • A TCP listener on port 10943 passes requests through unchanged to port 10943 to the same server.

Security Group differences

AWS application and network load balancers work differently with security groups. Application load balancers have security groups attached to them and apply ingress rules. In contrast, network load balancers do not have security groups attached; here the security rules of target instances apply, using their listening ports.

In our configuration the security group for the server specifies port 10943 for the traffic that passes through the load balancer, and port 80 for the web portal traffic.

Docker-in-Docker builds in TeamCity agents on AWS ECS

|

I have been experimenting with running TeamCity in AWS, using the CloudFormation stack provided by JetBrains. This stack uses Docker images from JetBrains and runs them in AWS Elastic Container Service.

However the default configuration does not allow Docker-in-Docker builds. This is the situation where a TeamCity agent, itself running in a Docker container, needs to run a build step with Docker or Docker Compose.

The page for the JetBrains Docker image for agents gives two options for starting the agent container from the command line:

  • Docker from the host
  • Run in privileged mode

This post is about how to modify the JetBrains CloudFormation template to start the agent container in those two ways.

Docker from the host

This technique maps /var/run/docker.sock from the Docker host into the running container. Declare a host volume and mount it in the container (see comments):

  AgentTaskDefinition:
      Type: AWS::ECS::TaskDefinition
      Condition: ShouldLaunchAgents
      DependsOn:
        - PublicLoadBalancer
        - TCServerNodeService
      Properties:
        PlacementConstraints:
          - Type: memberOf
            Expression: attribute:teamcity.node-responsibility == buildAgent
        # Define the host volume to map.
        Volumes:
          - Name: "dockerSock"
            Host:
              SourcePath: "/var/run/docker.sock"
        ContainerDefinitions:
          - Name: 'teamcity-agent'
            Image: !Join [':', ['jetbrains/teamcity-agent', !Ref 'TeamCityVersion']]
            Cpu: !Ref AgentContainerCpu
            Memory: !Ref AgentContainerMemory
            Essential: true
            Environment:
              - Name: SERVER_URL
                Value: "https://teamcity.tawh.net"
            LogConfiguration:
              LogDriver: 'awslogs'
              Options:
                awslogs-group: !Ref ECSLogGroup
                awslogs-region: !Ref AWS::Region
                awslogs-stream-prefix: 'aws/ecs/teamcity-agent'
            # Mount the host volume in the container.
            MountPoints:
              - ContainerPath: "/var/run/docker.sock"
                SourceVolume: "dockerSock"

I have used this method successfully.

Run in privileged mode

Set privileged mode in the ECS Task defintion for the agent by adding Privileged: true to the container definitions:

  AgentTaskDefinition:
      Type: AWS::ECS::TaskDefinition
      Condition: ShouldLaunchAgents
      DependsOn:
        - PublicLoadBalancer
        - TCServerNodeService
      Properties:
        PlacementConstraints:
          - Type: memberOf
            Expression: attribute:teamcity.node-responsibility == buildAgent
        ContainerDefinitions:
          - Name: 'teamcity-agent'
            Image: !Join [':', ['jetbrains/teamcity-agent', !Ref 'TeamCityVersion']]
            Cpu: !Ref AgentContainerCpu
            Memory: !Ref AgentContainerMemory
            # Run this container in privileged mode.
            Privileged: true
            Essential: true
            Environment:
              - Name: SERVER_URL
                Value: !GetAtt [PublicLoadBalancer, DNSName]
            LogConfiguration:
              LogDriver: 'awslogs'
              Options:
                awslogs-group: !Ref ECSLogGroup
                awslogs-region: !Ref AWS::Region
                awslogs-stream-prefix: 'aws/ecs/teamcity-agent'

I have not tried this method yet.

Capturing multi-part paths in Spring controllers

|

A while back I worked on a Spring Boot application that stores and works with Swagger files. It has a controller that needs to capture file paths at the end of request URIs.

We need to extract three variables: project, repo and path, where path may traverse multiple layers. Some examples:

URI project repo path
/swagger/AAA/domain-api/swagger.yaml AAA domain-api swagger.yaml
/swagger/BBB/domain-api/swaggers/domain.json BBB domain-api swaggers/domain.json

This naive attempt at a solution does not work:

@GetMapping(path = "/swagger/{project}/{repo}/{path}")
public ResponseEntity<String> swaggerFile(@PathVariable("project") String project,
                                          @PathVariable("repo") String repo,
                                          @PathVariable("path") String path) {

With this, when using the first URI, the path variable gets the value swagger without the file extension. The second URI does not match at all.

The file extension can be captured by using a regular expression, for example:

@GetMapping(path = "/swagger/{project}/{repo}/{path:.+}")

Here, given the first URI, the path variable gets the value swagger.yaml as desired. But the second URI still does not match at all.

There is no simple way to extract a deep path into a single variable. I think this is because the Spring classes split the path into segments using slash characters before matching each segment with a regular expression.

A working solution is to get the entire path from the servlet request object and parse it locally.

@GetMapping(path = "/swagger/{project}/{repo}/**")
public ResponseEntity<String> swaggerFile(@PathVariable("project") String project,
                                          @PathVariable("repo") String repo,
                                          HttpServletRequest request) {
    String prefix = String.format("/swagger/%s/%s", project, repo);
    String path = request.getRequestURI().substring(prefix.length() + 2);
    // Use path
}

Kotlin scope functions

|

I have been working with Kotlin recently and really enjoying it compared to Java.

A really useful feature is the ability to define extension functions on existing classes.

The built-in scope functions let, run, also, apply and with are really handy and are described well in Coping with Kotlin scope functions.

It can be difficult to remember which one to use. It helps me to think first of whether the function returns the receiving object or the value of the lambda.

Return the receiver

The functions also and apply return the receiver. For example, here is a natural way to get a Jackson object mapper and configure it in one expression:

val mapper = jacksonObjectMapper().apply {
    configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
    configure(DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES, false)
}

Within the lambda, this is set to the object returned by jacksonObjectMapper().

Similarly, also returns the receiver and passes it to the lambda as an argument instead of as self. This feels natural for side-effects that do not change the receiver, like logging:

return serviceResponse.also {
    log.info("Returning service response: $it")
}

Return the value of the lambda

The functions let, run and with return the lambda. They can be used as idiomatic null checks, for example:

val result = value?.let { transform(it) } ?: defaultValue

This has the meaning, If value is not null, transform it; else return a default value.

run is similar, except that it applies directly to the receiver (avaiable as this in the lambda).

There is a with function like that in Visual Basic, except that it returns the value of the lambda (which can be ignored).

with(something) {
    setValue(newValue)
}

Quick mnemonic

  • If the scope function starts with a (also, apply) it returns the receiver;
  • else (let, run, with) it returns the lambda.

Testing Javascript with iterables

|

I wanted to write tests for the retryAxios function described in my recent post about retrying back-end service calls.

Rather than mocking Axios itself with either moxios or Axios mock adapter, I wanted a simple mock of an Axios function. The mock needs to be configurable to return a known sequence of responses, where each response is a promise that either resolves or rejects.

My solution was to create a function that returns a closure:

const mockAxios = (...returns) => {
  const iter = returns[Symbol.iterator]();
  return () => {
    const { done, value } = iter.next();
    if (done) {
      throw new Error('Testing error: mockAxios iterator done');
    }
    return mockRequest(value);
  };
};
  • The rest parameter returns contains an array that specifies how each function call will return.
  • mockAxios returns a closure that iterates through returns, calling mockRequest for each value in that iterable.
  • If the iterable is exhausted the test itself is in error.

mockRequest is a bare-bones implementation of the behaviour of Axios functions:

class HttpError extends Error {
  constructor(response, ...params) {
    super(...params);
    this.response = response;
  }
}

class NetError extends Error {
  constructor(code, ...params) {
    super(...params);
    this.code = code;
  }
}

const mockRequest = async (retVal) => {
  if (Number.isInteger(retVal)) {
    const response = { status: retVal };
    if (retVal >= 400 && retVal <= 599) {
      throw new HttpError(response);
    }
    return response;
  }
  throw new NetError(retVal);
};
  • If retVal is an HTTP success status code, it returns a response object with that code.
  • If retVal is an HTTP failure status code, it throws a simplified HTTP error object with that code.
  • Otherwise it returns a simplified network error object with code retVal.
  • mockRequest is async so it wraps the return value in a resolved or rejected promise.

Here is a simple example of using mockAxios in a Jest test case, with inline comments for annotation:

test('retryAxios should return first successful call if within retry limit', async () => {
  // Calling expect.assertions() is useful when testing asynchronous code.
  expect.assertions(1);
  const data = await retryAxios(
      // The iterable with delays after retriable failures.
      [5, 5],
      // A mock Axios function that returns server failure, then network failure,
      // then success.
      mockAxios(500, 'ECONNRESET', 301)
  );
  expect(data.status).toBe(301);
});

The test code has been added to my Gist for retryAxios.