+U Maxout!

cats and dogs for AWS

Posted on In , Disqus:

When I worked on a wireless basestation project, there is an IPL layer to configure and manage uboot, uboot to configure and manage build images, image starts HAL, the HAL calls WatchCat, WatchCat wathes soft WatchDog, then the watchdog propagate Apps which talks via USL (Universal Software Layer). Now AWS runs in an eco as well. There are plenty of Cats and Dogs which can prepare, configure, watch(monitor) even take care your services, containers, apps, or, hosts.

This post summarizes the eco services as Ansible, Datadog, CircleCI, Terraform and draws a list for further self-study.

  • Management
    • Ansible, no preinstalled package.
      • Simple to deploy as it only requests Python and SSH connection to target. As Fabric, Ansible is also based on paramiko.
      • It will transfer executables to targets and call them via preconfigured YAML as DSL.
      • More than 450 modules available.
      • Three ways to run:
        • cli ad-hoc run ansible;
        • Playbook;
        • With automation framework: Tower.
      • Fabric automates CLI in Py Lib, ansible automates the process.
    • Chef
      • Use recipes to reuse processes;
      • In Ruby based DSL, (looks more like scripts)
      • Needs CouchDB and RabbitMQ on server side
      • Communication to RESTful via HTTPs with pem key file.
      • One Chef server, multiple knives (Chef Workstation, just the laptop of administrators), multiple Chef Nodes (clients, the target to manage, support 1st class CLX as docker)
    • Puppet
      • Just install by gem;
      • No intermedia software to install
    • SaltStack,
      • YAML config and Python lib, similar to Ansible;
      • Needs a package SaltStack minion installed on target as prerequisite.
  • Infrastructure Creators (Infrastructure as Code)
    • Packer,
      • Creating AMI via JSON config.
    • Terraform,
      • Create infrastructure via DSL;
      • Support multiple platforms: Azure, AWS, Openstack, et.
    • cloudtools,
      • Open source, https://github.com/cloudtools;
      • troposphere: Py lib to create AWS CloufFormation;
      • awacs: Py lib to create AWS Access Policy.
  • DevOps
    • CircleCI: free for open source projects
    • Travis: free for open source projects
  • Monitors
    • DataDog,
      • Perf Analysis, as TingCloud, 14 days free trial.
      • Needs target installation;
      • Follow StatD protocol to send timely message via UDP to DataDog servicce then present to Web GUI with visualization.
    • Netflix ICE,
      • Opensource toolset focus on top-bottom cost and usage analysis.
    • pagerduty,
      • Event driven reporter, 14 days free trial.
    • Scalr,
      • Needs rpm installaiton, free for open source on-premise mgt.

"Thoughts and Notes"

The author also thinks that JSON or YAML based plain CloudFormation is not easy for human beings to edit directly. However, it seems most of the attempts are going to adapt a DSL based of Ruby or YAML. Based on my first sight opinion, this kind of problem is okay to adapt Template to help. For example, we can image to create CloudFormation via Jinjia2 flavor template. It is a smaller work and easy to develop. Some template rather than introducing something like YAML again. Web service developers are supposed to be familiar with at least one template as ASP, JSP, or Jinjia2(in Flask or Django). Another way, still a DSL, maybe more programmable, in Scapy style as Python interpreted dsl to describe wrapped resources and linkages. Moreover, could this DSL run in an interactive way, possibly if we just call boto3 and make a lightweight way of feedbacks.

Trying to make more practices with Anisble and check if one of them is still a valid idea to work further.

Jan 31, 2017: After an inpiring discussion with some people this afternoon, knowing more are also sufferring on cloudformation plain edition or drafting, add above thoughts and notes part.

EOF