KubeCon 2015

Learn how contributors to the Go build system modified build.golang.org to perform all Linux 386 and x64 builds on Kubernetes and Google Container Engine. From cluster discovery and authentication to optimizing Kubernetes API calls and using the watch API, learn about the ins and outs of using Kubernetes from a Go client to coordinate a large distributed build system.

To Go build system (https://build.golang.org), like the Go programming language itself, is an open source project. Written in Go, the build system is responsible for building, testing, and validating every commit made to the Go programming language across every platform and architecture the language supports (almost 40!). Many contributors commit many changes, resulting in hundreds of builds and thousands of tests running each day.

Linux 386 and x64 builds previously ran on Google Compute Engine. Each build required a VM boot, then the build and tests ran for several minutes, and the VM was terminated. Although GCE is much better than EC2 for this type of workload (with ~40s boot times, 10-minute minimum billing, and per-minute billing thereafter), Kubernetes offered the possibility of even faster build times and the ability to optimize for scale and cost (especially as the popularity of Go grows).

Docker Swarm is a powerful solution to manage your Kubernetes cluster on any Platform. In this talk, we will show how you can quickly deploy and manage a Kubernetes or a Nomad cluster on Digital Ocean, Google Cloud Platform and on your local environment in the exact same way.

The Deis project, an open source PaaS platform, recently announced in our public roadmap meeting that we are transitioning our entire platform onto Kubernetes. We call it the Great Kubernetes Rebase. In the process, we’ve gained valuable insight into what it means to build a complex platform in the Kubernetes way.

This presentation is a crisply paced walk through the tough lessons we've learned as we've tried (and sometimes failed) to re-think our application.

Among the things this presentation will cover:

• Turtles all the way down: Running a VM in a container in a pod on a kubelet… in VirtualBox, just so that we can safely build Docker images.
• Namespaces in theory and practice.
• Running an HA Postgres cluster inside Kubernetes.
• Ditch OSX: Productive developers do it on Linux.
• Storage: The data’s gotta go somewhere!

The secret to using secrets: They’re good for credentials, but they’re also good for a few other things.

We take stability very seriously: Continuous testing inside the cluster.

Kubernetes: it’s new; it’s cool; and you want it! But you have questions… Is deployment easy? Does it scale? Can it be upgraded in place? Will it work with your existing services? Can it handle big data workloads?

You wish you could learn just one platform and never have to learn another one, but that's not how software works. Today’s epic swell is tomorrow's choppy ripples. Today's cloud nine is tomorrow's smog. What you really need to learn is how to surf… on clouds.

So catch the wave and we’ll ride the white fluffies together. We’ll start with Kubernetes, the cloud container orchestration engine Google seeded, and Mesos, the scheduling framework from which Twitter and Apple are hanging ten. Then we’ll throw on the afterburner with the Mesosphere Datacenter Operating System (DCOS) and deploy orchestrators, like Kubernetes and Marathon, alongside distributed services, like Spark and Cassandra, to open up a universe of possibilities.

OpenShift 3 has been intertwined with Kubernetes development since the very beginning, and many of the developer focused workflows it has focused on has informed and been informed by the core design of Kubernetes. Today it offers workflow and tooling for application developers using Kubernetes as a development platform while still allowing those developers to manage the lower level components of the system. This talk will describe those patterns and how they enhance the core Kubernetes objects, and how those patterns are being moved into the Kubernetes core over time.

We will cover:

• Deploying and redeploying applications based on external changes – how does a developer rely on the platform to manage change, and how can the new Deployment object evolve to offer extended capabilities like triggers, hooks, and custom logic.
• Building and rebuilding images based on administrator or developer source code, and how immutable image infrastructures like Kubernetes and Docker are dependent on automated end to end build chains.
• Allowing administrators to offer a low cost way to expose hundreds or thousands of applications through a front-end load balancer without requiring dedicated load balancers, and how the Ingress resource allows abstraction of high availability and external exposure while allowing administrative control.
• Hosting toolchains on top of Kubernetes that allow better development experience – specifically cheap Git hosting and mirroring and scale out Jenkins slaves.
• Administrative policies that can limit and control developer action to allow sharing of the cluster.

The talk will also touch on future design directions for OpenShift 3 and Kubernetes and how the platform can grow to better serve large scale application hosting.

With the rise of modern containers comes new problems to solve – especially in networking. Numerous container SDN solutions have recently entered the market, each best suited for a particular environment. Combined with multiple container runtimes and orchestrators available today, there exists a need for a common layer to allow interoperability between them and the network solutions.

As different environments demand different networking solutions, multiple vendors and viewpoints look to a specification to help guide interoperability. Container Network Interface (CNI) is a specification started by CoreOS with the input from the wider open source community aimed to make network plugins interoperable between container execution engines. It aims to be as common and vendor-neutral as possible to support a wide variety of networking options — from MACVLAN to modern SDNs such as Weave and flannel.

CNI is growing in popularity. It got its start as a network plugin layer for rkt, a container runtime from CoreOS. Today rkt ships with multiple CNI plugins allowing users to take advantage of virtual switching, MACVLAN and IPVLAN as well as multiple IP management strategies, including DHCP. CNI is getting even wider adoption with Kubernetes adding support for it. Kubernetes accelerates development cycles while simplifying operations, and with support for CNI is taking the next step toward a common ground for networking. For continued success toward interoperability, Kubernetes users can come to this session to learn the CNI basics.

This talk will cover the CNI interface, including an example of how to build a simple plugin. It will also show Kubernetes users how CNI can be used to solve their networking challenges and how they can get involved.

Containers are revolutionizing the way we deploy and maintain our infrastructures: reducing development overhead, streamlining dev / test / ops, and enabling highly scalable, dynamic infrastructures. But containers still have a key problem: monitoring and troubleshooting them is impractical, painful, and sometimes plain impossible. Even basic things like understanding what is using CPU, memory, or disk bandwidth inside a container are difficult - let alone finding out who a container is talking to on the network or tracking malicious activity. This complexity is even further complicated by the addition of orchestration layers such as Kubernetes.

In this 30-minute presentation, Gianluca Borello will cover the current state of the art for container and Kubernetes monitoring and visibility, including real use-cases and pros / cons of each. He will then focus on advanced container visibility techniques, such as:

* visualizing a container’s network activity
* understanding detailed resource usage (CPU, memory and disk I/O) of pods, containers and individual processes running inside containers
* following process and user activity inside containers and across pods
* collecting logs from multiple containers
* understanding application performance from a microservice perspective, across pods and RCs, regardless of the physical location of the underlying containers

The presentation will include live interaction with Kubernetes environments, and hopes to help DevOps engineers to deploy a Kubernetes infrastructure in production with confidence and peace of mind. Many visibility tools and techniques will be covered, although special emphasis will be put on sysdig, an open source container and system troubleshooting tool that the presenter has helped author. GitHub link: https://github.com/draios/sysdig

eBay inc runs one of the largest Openstack based private clouds in the world offering various infrastructural services to its developers. However, developing cloud-scale services using only infrastructural primitives poses a really hard problem. We have been playing around with Kubernetes from early on with the intention of adopting it as the standard base for developing web scale platform services.

In this presentation cover:

• The motivations behind adopting Kubernetes
• The technical challenges and solutions for making Kubernetes work on Openstack, specifically on the following areas of:
  •  Networking
  •  Storage
  • Identity and Access Management
• Future plans and priorities
• Q&A

Once you get past the work of installing and configuring Kubernetes to your liking, the real challenge begins. Without any documentation to guide you, you'll need to figure out which people, groups and projects will access the system, what resources they can consume, where resources live and how these pieces will interact. And then you'll have to do that again and again, tweaking the configuration each time the requirements of your organization change. The more successful your infrastructure is, the more complex the process of actually managing your Kubernetes installation becomes.

In this talk, you'll learn how a powerful, policy-driven tool can help you with the complexity of managing systems like Kubernetes at scale. A comprehensive and pervasive policy approach with automatic enforcement enables Operations and IT teams to model and control the steady state of a system they want, while still giving your developers the freedom to claim the resources they need. This lets everyone associated with your organization's Kubernetes installation win.

This talk will include:

- How a policy system works in this context
- How to structure policy so it is most effective in a distributed, multi-cloud system
- Crucial points for policy enforcement
- Initial work to add a comprehensive policy system to Kubernetes as part of the Cloud Native Initiative

Pachyderm is a containerized data analytics solution that's completely deployed using Kubernetes. We take all the amazing tools and potential in the container ecosystem and unlock that power for massive-scale data processing. In this talk we'll show you how to leverage Docker, Kubernetes, and Pachyderm, to build incredibly robust and scalable data infrastructure. We'll start by discussing the key components of a modern data-drive company and how your infrastructure choices can have a massive impact on your product and scalability roadmap. We'll then dive into some architecture details to show how Kubernetes, Docker, and Pachyderm all work in tandem to create a cohesive data infrastructure stack. Finally, we will demonstrate some high-level use cases and powerful benefits you get from the architecture we've outlined.

Following the Kubernetes 1.0 release, the community has organized into a number of Special Interest Groups, one of which is focused on building large clusters. This presentation will give an update on the activities of the Kubernetes SIG, the Samsung SDS rationale for involvement, and a view on what's ahead. Outline:

Explain the goals of the group, when/how we meet, and how to get involved

Briefly explain why Samsung SDS has prioritized scalabilty and why we are interested in large clusters

List a short summary of SIG activities “what we've done/learned so far”

Give an update on the Intel-donated 1000 node cluster

Present a view on a few of the scalability challenges ahead.

Like many other companies, SoundCloud migrated to a microservices architecture over the last couple of years. Today, there are several hundreds of services with thousands of container instances running in our datacenters. In this talk, I’ll give a brief overview of the current state of our infrastructure and how a typical service is deployed and can communicate with other services.

To make it simple for teams to prototype, deploy and operate several services on their own, we built our own container runtime environment, called Bazooka. I’ll give an overview of Bazooka, its features and design decisions, but also the shortcomings and problems we faced over time:

• automated scheduling.
• resource management for services with different load profiles.
• monitoring of highly dynamic deploys or the requirements of stateful services.

With the rise of Docker and a general shift towards container-based environments, SoundCloud started to build more and more of its development workflows based on these new solutions. When it became clear that our existing system needed an overhaul to support additional requirements and overcome its shortcomings, we started to look into other container management technologies and available open-source options. In the second part, I’ll present our evaluation process and some of the requirements we defined for a suitable candidate. The attendee will learn which features and properties of Kubernetes make it the ideal choice for us.

Finally, I’ll talk about the current state of our Kubernetes migration, some challenges we need to solve to integrate Kubernetes in our existing infrastructure, and present some open issues we are working on in order to eventually deploy and run all our services with Kubernetes.

Wikimedia Tool Labs is a free and fully open source cluster environment made available to anyone who asks, to experiment with anything Wikimedia related. It's an important part of the Wikimedia universe – a study has found that when our bot infrastructure is down, the amount of time it takes to remove vandalism from Wikipedia almost doubles! (http://stuartgeiger.com/wikisym13-cluebot.pdf).

In the 10 years it has been active it has accumulated lots of 'jury-rigged' homegrown solutions - based primarily on the last open source version of OpenGridEngine. This has led to interesting custom setups that users have embraced, got to work once and expect to keep working. Most users are volunteers, so we can't force them to change their code to adapt to new things too frequently
It is quite hard to follow most modern deployment practices with our current architecture, and most code does not (some are still CGIs!). Kubernetes seems to allow us to both provide a legacy, backwards compatible interface for users who are unwilling to change, and a more modern, 'native' kubectl interface for people who are - 'best of both worlds'. This 'backwards compatibility' setup is made possible by replicating our current runtime environment via docker images, thus allowing people to continue using their current setups with no changes. People who want to move to saner systems can just treat us as 'yet another kubernetes cluster' as much as possible - helping commoditize our infrastructure, which is A Good Thing.

This talk will cover:
# What Tool Labs is, and why it is important
# A quick overview of our current setup, and why it is hard to change
# Ongoing kubernetes based work to overhaul this setup
# What kubernetes is going to offer to its users
# Call for people to get involved - all of our infrastructure is open source and open to outside contributions!

Moving your application into a container and deploying it to
production is a great first step towards taking advantage of
containerization. This gets you past "works on my machine", and Docker
makes this easy. But the real value of containers -- fast immutable
deployments, maximizing resource utilization, and bare-metal
performance -- comes from an architecture optimized for containers.
This is container-native architecture.

Tim will explore the story of a real-world large scale production
microservices deployment of Docker, and the challenges faced in both
design and operations of migrating this kind of multi-faceted
application to a container-native architecture.

There are plenty of useful things you can do with Ruby and a bunch of servers. This talk isn't about useful things. This talk will show off asinine, amusing, and useless things you can do with Ruby and access to cloud computing.

Sentiment analysis based on emoji? Why not? Hacky performance testing frameworks? Definitely! Multiplayer infinite battleship? Maybe? The world's most inefficient logic puzzle solver? Awesome!

Aja uses Kubernetes to deploy distributed computing solutions to ridiculous computing problems!

In this talk, Connor and Niklas will talk about their thoughts on the next decade of cluster computing. They have worked on Apache Mesos, Kubernetes and Mesos Frameworks; from design of subsystems to tooling and operationalizing at scale.  They will discuss past, present and future trends in public and private cloud computing and unique opportunities for the cluster computing communities.  By the end of the talk, they hope you will leave with a fresh perspective on scheduling and orchestration, at a deeper level than "Mesos vs. Kubernetes vs. Omega vs. Borg …"

Lithium has chosen to use Kubernetes as its orchestration system for microservices. We'll share our insights getting our first service into our production cloud. The path to getting the gleaming effect wasn't easy but now we're pushing the limits with deploying new services.

If faced with the task of migrating a monolithic application to a microservice architecture, Kubernetes might be a counter-intuitive first step. However, operating an application from within a Kubernetes cluster can enable some very low-risk / high-reward refactorings without rewriting much application code, if any.

Many will encourage one to start refactoring a monolithic application at business boundaries. Instead, focus on the parameterization and configuration of your application. Many of the contents of configuration files are orthogonal to the business logic of the application: server names, credentials, locations of certificates, hashing algorithms, etc. Strive to reduce the configurability of your application as much as possible. This exercise will point you at the first and easiest places to utilize some of the best features of Kubernetes.

Services provide the ideal abstraction for external systems which require only a simple network connection. In many cases, the services required by a monolithic application are treated as ubiquitous and guaranteed. Their names can be used as hard-coded defaults. Services that cannot be deployed in the cluster, such are large databases, or those provide by 3rd parties, can make use of “services without selectors”.

Pods, and the shared network namespace they provide, allow your application to assume that everything is running locally, even when they are not.

Ambassador containers are helpful in cases where a service requires any amount of additional logic to use correctly, such as custom headers, service location, or authentication. Examples: An open smtp-relay which deals with SASL authentication to an upstream mail server, or twemproxy in front of shards of redis or memcached servers.

Use the adapter pattern to convert incoming https requests into http, and eliminate the need for your application to properly set up server-side TLS.

Secrets can be used in cases where configuration files are still required. Kubernetes allows the mount path of secrets to match the default location required by your application.

Deploy your monolith to a Kubernetes cluster, where it will provide the freedom to change many of its moving pieces. Each externalized dependency adds to a virtuous cycle of increased modularity. Refactored code often produces reusable functions, utilities, and libraries. A refactored Kubernetes application often produces reusable pods and services.

Part and parcel of many microservices-based deployments is an agile approach to development, testing, and delivery. Continuous integration and continuous delivery are specific implementations of these agile processes. By their very nature microservices are small, decoupled, and focused on accomplishing single tasks – and the most successful implementations rely on continuous testing and deployment to minimize the feedback loop and reduce the burden on operations.

Containers and Kubernetes are a natural extension of microservices deployments; the former allows applications to be encapsulated in their own operating environment and the latter makes it easy to orchestrate containers inside of a distributed cluster. The goals of Kubernetes are to accelerate development and simplify operations by treating clusters of containers as a single system – but how do you map those concepts back to microservices and continuous delivery?

In this talk, we will cover the basics of Kubernetes and show how to set up continuous delivery pipelines using Jenkins and Jenkins Workflow to go from code to deployment, without developers having to interact with the production deployment infrastructure. The goal is an end-to-end set of steps to automate deployment and delivery of an application composed of several microservices.

The next major release of Carbon will rely on lightweight containers to overcome the limitations described above by offloading multi tenancy support into k8s using namespaces and quotas. It provides isolation and the controllability of resource usage of each and every tenant. For large scale deployment we will use k8s service discovery, monitoring, health checking, auto healing and load balancing capabilities.

In this session we will discuss how we reduced complexity of the WSO2 middleware platform by using k8s with the design of microservices.

A few months ago, Vungle’s infrastructure was showing its age. As the company moved more toward microservices and needing globaly distributed infrastructure, our old approach of deploying a single app to a group of Ubuntu machines with Chef (either with autoscaling or manually) was starting to become a bottleneck. We were also worried that we were not utilizing our server resources well. We were already using Docker to streamline our development environments and CI systems, so moving production to a Docker-based system seemed like an obvious choice. After evaluating the options (Kubernetes, Mesos, Fleet, etc), we decided to go with Kubernetes on CoreOS.

This talk will focus on the technical decisions we made about our Kubernetes infrastructure to allow us to scale all over the globe, some of the issues we faced and how we worked around them, and the benefits we have seen.

Some highlights:

• Setting up clusters in VPCs using CloudFormation
• Moving from legacy infrastructure
• Exposing services to the outside world
• Making complex http routing easily configurable by services
• Communication between clusters
• Limitations in AWS support
• Integration into Deployment process

Vungle has benefitted greatly by embracing containers as the basic method for packaging services, and Kubernetes has really allowed us to become container-native all the way into production. It’s a lot of work to get it right, but putting in the effort is really paying off.