2015-06-23

Clocker 1.0.0 Technology Preview

Peter at Project Calico has already described some of the cool things that Calico can do to link Virtual Machines and Docker Containers using Software-Defined Networking. Here I want to show how some of the new features in Clocker can use Calico to orchestrate the deployment of applications that span both platforms. To make this work, we will need the latest version of Clocker, which is the 1.0.0 technology preview, currently available for testing.

New Features

There are several new and updated features in this release. The major points are listed below, but we will concentrate on how to link OpenStack Virtual Machines and Docker Containers in this post.

  1. The ability to start Calico with an external etcd cluster, using local proxies on each Docker host.
  2. Support for Swarm as the entry point for the cluster of Docker hosts.
  3. An OpenStack aware virtual network entity for mixed-infrastructure applications.
  4. Configurable setting to switch off sshd installation and use docker exec instead of ssh.
  5. Change from Ubuntu base image to use Alpine for faster container startup times.

OpenStack and Docker

Peter's blog post describes how to create an OpenStack cloud with the Calico driver for Neutron. Apache Brooklyn can be configured to deploy to a JcloudsLocation defined using the jclouds:openstack-nova provider. By saving the connection details in the brooklyn.properties it can be referenced in a blueprint by name, such as my-openstack-cloud. To create a Calico based Docker Cloud that is linked to this OpenStack cloud we only need to make sure that Clocker deploys the correct version of the Calico Node software, and that they connect to the same etcd data store that Calico is using on the OpenStack compute nodes are using. We can do this by configuring the Calico nodes to use an EtcdProxy entity that is connected to the correct remote URL. Here is the snippet of the YAML blueprint describing the SDN provider specification:

sdn.provider.spec:
  $brooklyn:entitySpec:
    type: brooklyn.networking.sdn.calico.CalicoNetwork
    brooklyn.config:
      calico.version: 0.4.8
      etcd.version: 2.0.11
      sdn.network.cidr: 50.3.0.0/16
      sdn.network.size: 24
      etcd.node.spec:
        $brooklyn:entitySpec:
          type: brooklyn.entity.nosql.etcd.EtcdProxy
          brooklyn.config:
            etcd.cluster.name: "controller.example.org"
            etcd.cluster.url: "http://192.168.0.123:2380"

Once we have our Calico Docker and OpenStack Cloud running, we can deploy applications there, in the same way that we would deploy applications to a standalone Docker Cloud. These applications will consist of containers connected to Calico networks, and will only be able to connect to services running on Virtual Machines in the OpenStack cloud through the Internet. For private communication between VMs and Containers to take place on a Calico network, we must write a blueprint that describes this. Calico connects containers together by adding them to a profile on Docker. Calico also create profiles on OpenStack when a security group is created, so we must ensure that our application has a way to link the security group name used on OpenStack to the profile name needed on Docker. We do this with a new VirtualNetwork sub-class which creates a new security group in OpenStack. The UUID of this new security group is then used by Calico as a profile name for Docker Containers.

Demo Application

We have created a simple demo application to illustrate linking Docker and OpenStack with Calico. This consists of a Riak cluster running on Virtual Machines in OpenStack, and a cluster of Node.js applications, fronted by an HAProxy load balancer, all running in Docker Containers. The Clocker blueprint for this application needs to describe the OpenStack location where we will create a RiakCluster entity with several RiakNode children. These will use VMs in a security group, created by the OpenStackVirtualNetwork entity. Finally, we will create a ControlledDynamicWebAppCluster which is a load-balanced group of web services, each of which will be a NodeJsWebAppService running some client code that access Riak to update a simple counter and display it on a web page.

The complete YAML blueprint is shown below, and the application can be found at csabapalfi/riak-counters-example.

id: riak-nodejs-demo
name: "Riak and Node.js Demo Application"

services:
- type: brooklyn.networking.OpenStackVirtualNetwork
  location: my-openstack-cloud
  networkId: data
  cidr: 50.2.0.0/24

- type: brooklyn.entity.nosql.riak.RiakCluster
  location: my-openstack-cloud
  initialSize: 3
  id: cluster
  brooklyn.config:
    install.version: 2.0.5
    provisioning.properties:
      minRam: 7g
      minCores: 8
      securityGroup: data

- type: brooklyn.entity.webapp.ControlledDynamicWebAppCluster
  location: my-docker-cloud
  name: Web Cluster
  brooklyn.config:
    initialSize: 2
    dynamiccluster.memberspec:
      $brooklyn:entitySpec:
        type: brooklyn.entity.webapp.nodejs.NodeJsWebAppService
        brooklyn.config:
          appName: riak-counters-example
          gitRepoUrl:
            "https://github.com/csabapalfi/riak-counters-example"
          appFileName: index.js
          docker.openPorts:
          - 3000
          shell.env:
            RIAK_NODES: >
              $brooklyn:component("cluster")
              .attributeWhenReady("riak.cluster.nodeListPbPort")
          network.list:
          - data
          - web
    controlleddynamicwebappcluster.controllerSpec:
      $brooklyn:entitySpec:
        type: brooklyn.entity.proxy.haproxy.HAProxyController
        brooklyn.config:
          docker.image.name: haproxy
          docker.image.tag: 1.5.9
          install.dir: "/usr/local/sbin/"
          run.dir: "/usr/local/etc/haproxy/"
          network.list:
          - web

There are a few things to note about this blueprint. Firstly, the OpenStackVirtualNetwork is created with the name data, and this network is used by both the RiakNode and the NodeJsWebAppService entities. The second network refernced is named web and is also used by the Node.js entities, as well as the HAProxyController which is acting as a load balancer. The second network will be created by Clocker automatically, without the need for any explicit entity definition, using its default settings and will be used only with Docker Containers. The specification for the controller entity for the web cluster uses a Brooklyn entity, the HAProxyController, but Clocker can simply use a pre-existing image from Docker Hub (in this case haproxy:1.5.9) and will reconfigure it appropriately when new web services are added to the controlled group. A similar method could have been used with the Node.js entity, if desired, and in many cases when a blueprint will only be used in a Clocker context this is desirable.

The running application displays a counter and a unique identifier for each Node.js server connected. The number of these services can be increased either automatically with a Brooklyn policy when scaling criteria are met, or manually by invoking the resize effector on the cluster. To learn more about how to write and run Brooklyn and Clocker blueprints, look at the this introduction or more in-depth documentation about creating YAML.

Clocker is available on GitHub under the Apache 2.0 license. You can provide feedback with any comments or problems on the incubator-brooklyn-dev mailing list, or add issues to the GitHub repository. If you have an idea for an improvement or a new feature, just fork the code and issue a pull request! See the Trello board for the current roadmap and task list.

2015-03-15

Clocker News and Announcements

It's been a long time since the last update to Clocker, at least in the fast-moving world of Docker. Since then we have had multiple releases of the Docker Engine, now at version 1.5.0, and the announcement of beta releases of the Compose, Swarm and Machine tools, Socketplane.io joined Docker, competing container software has appeared, like Rocket from CoreOS, and there have been several new minimalist, Docker-specific operating systems and distributions annouced. So far, Clocker has caught up to the latest Docker release, and supports some of its new features, and we are working on adding support for Compose and Swarm. As it stands, Machine is not as useful for Clocker, since we leverage the powerful jclouds library and the mechanisms provided by Brooklyn for VM creation and deployment. The most exciting integration is going to be using the Compose language as a blueprint for Clocker, allowing re-use of existing application definitions.

The main feature of 0.8.0 is the pluggable Software-Defined Networking capability. In the previous releases it was possible to enable Weave as a SDN mechanism that provided networking between containers on different hosts. In this release I have made the SDN provider a generic interface and written multiple implementations, porting the original Weave code across to work this way also. The latest SDN provider is the Project Calico for Docker, from Metaswitch, which will be merged into the code later this week. I am very excited about this integration, as it shows that Clocker can be used to orchestrate Docker CLouds in various configurations with whatever choice of features is optimal. This opens the way for further integrations like Flocker from ClusterHQ for storage, using their Powerstrip extension prototyping mechanism.

Clocker 0.8.0 Developer Preview

This release of Clocker is a developer preview or early-access release of 0.8.0 which will be available until the final 0.8.0 is shipped. Since the development for 0.8.0 was dependent on a SNAPSHOT release of jclouds it was carried out on a feature branch, and was less visible. I have now merged that code with master and made customised releases of both jclouds and Brooklyn available, thanks to Cloudsoft, on their Artifactory server. This means it will be possible to clone and build the latest code and test the SDN features I have been building. I would appreciate any comments about usability and any bugs found, and will try and incorporate them into the final release. Over the next week I will work through the outstanding issues and tasks, particularly finalising the Project Calico integration, and generally improve the quality of the released code.

Please check back for further updates before the final 0.8.0 announcement, with release notes and further documentation for new features...

To test the code, use the following commands to download and run a Docker Cloud with weave networking in SoftLayer. You may need to configure your credentials for the Brooklyn location first. If you don't have a SoftLayer account, any cloud supported by Apache jclouds is suitable.

% wget --quiet -O brooklyn-clocker-dist.tgz http://git.io/ppnt
% tar zxf brooklyn-clocker-dist.tgz
% cd brooklyn-clocker
% ./bin/clocker.sh jclouds:softlayer:lon02

Remember to send any bug reports or feedback on this pre-release to clocker@cloudsoft.io or create an issue on GitHub, and I will investigate.

Clocker Talks and Presentations

One of the things that has kept me busy recently is speaking about Clocker, and you can view the slides from a couple of these presentations online. At DockerCon in Amsterdam I gave an overview of Clocker and its Weave integration, then this month in London I spoke during the Docker track at QCon. There I explained more about the ways that Clocker can create a Docker Cloud and optimise its deployment to fit your specific needs. The slides are linked to below, and there is a longer list on the Clocker Wiki. I will link to the video of the QCon talk when it is available.

I will also be speaking at ApacheCon North America about Clocker, as well as my colleagues Andrea Turli and Alex Heneveld, so please come along if you are attending. The dates for your diary are listed below.

Finally, if you are in Edinburgh this week, Peter White from Metaswitch will be speaking at the Edinburgh Docker Meetup on Thursday. The Meetup is sponsored by Cloudsoft and will be at CodeBase in Argyle House on Thursday 19 March from 18h00 onwards. There will be beer and pizza beforehand, followed by Peter's talk on Project Calico and a demonstration of the Clocker integration, and plenty of time for discussion about Docker and DevOps...

https://github.com/brooklyncentral/clocker/

Clocker is available on GitHub under the Apache 2.0 license. You can provide feedback with any comments or problems on the incubator-brooklyn-dev mailing list, or add issues to the GitHub repository. If you have an idea for an improvement or a new feature, just fork the code and issue a pull request! See the Trello board for the current roadmap and task list.