SRE / DevOps / Kubernetes Weekly Collection#13(Week 18)

Yoshiki Fujiwara

Jul 9, 2020·14 min read

• In this blog post series, I collect the following 3 Weekly Mailing List I subscribe to, leave some comments as an aide-memoire and useful links.
• Actually, I have already published the same content in my Japanese blog and am catching-up in English in this series.
• I hope it contributes to the people browsing this kind of information as a reference.

DEVOPS WEEKLY ISSUE #487 April 26th, 2020
SRE Weekly Issue #216 April 26th, 2020
KubeWeekly #214 May 1st, 2020

DEVOPS WEEKLY ISSUE #487 April 26th, 2020

News

A detailed write up from the recent Failover Conf on all things chaos engineering. Lots of notes and links to related content.

• The title is “Failover Conf Wrapup”.
• This is a detailed summary of the Failover Conf held the other day. There were comments such as the session I participated in, I wanted a long break time (I can not help setting lunchtime breaks because there are parts that can not be helped because of different time zones).
• When the session video is released, he will update the blog and link to it.

A good post for any software developers needing to operate databases, in this case PostgreSQL. Covers tracing, query profiling, slow query logs and more.

• The title is “Introduction to Profiling and Optimizing SQL Queries for Software Engineers”.
• “Recently, it’s rare to have a dedicated Database Admins (DBAs) in the z development team. As a development team, you have to take responsibility for the database as it adopts microservices architecture, cloud, and DevOps processes. Therefore, acquiring a solid database skill is an inevitable path to becoming a seasoned programmer,” says an article discussing profiling and optimizing SQL queries.

A look at applying feature branches to infrastructure, automatically provisioning new Terraform workspaces for testing distinct changes.

• The title is “From “Feature branches” to “Feature environments” with Terraform”.
• As a general practice of the development team, refer to “Separate the code of the Feature that is being developed by breaking the Feature branch from other team members” and explain the advantages of creating and utilizing the Feature environment through Terraform The article you are doing.

Related to the above post, this article covers patterns for using source control branches effectively.

• The title is “Patterns for Managing Source Code Branches”.
• Following the above article, an article explaining Git’s branch strategy. It concludes that “the optimal strategy is different in each context between OSS and commercials”, and I would like to always consider the optimal one in consideration of merit/demerit/culture/value to be provided.

A tutorial on managing exceptions in policy, using Open Policy Agent and Conftest. Looking at testing different types of Dockerfiles.

• The title is “Creating Exceptions Lists for Conftest in Rego”.
• OSS tool “conftest” that can write structured config tests such as YAML, JSON, Dockerfile, HCL/HCL2 does not support context as one of the constraints, so explain how to create “Exceptions Lists” The article you are doing.

An excellent recorded conversation, and transcript, on securing software. Covering the challenge of increasing complexity, the cost of vulnerabilities and lots more.

• The title is “The Security of Software”.
• A transcription of a live streaming video from Trend Micro with Tanya Janca , Founder, Security Trainer, and Coach of SheHacksPurple.dev as a guest.
• The theme is advice and tips for developing a security mindset for the development team. The video is also embedded.

A writeup of the new cf-for-k8s project, which provides a way to run Cloud Foundry on top of Kubernetes.

• The title is “DEPLOY CLOUD FOUNDRY TO GOOGLE KUBERNETES IN 10 MINUTES”. While touching on the recent movement of CF and future, it is a story to deploy CF in the environment of GKE as the title.
• Also check out the blog “Cloud Foundry Becomes More Kubernetes Native with cf-for-k8s” that was announced by the cf-for-k8s release team.

A look under the hood of Docker images, exploring the underlying overlay filesystem and the OCI specification.

A set of tips for managing your cloud bills, looking at optimising applications, understanding your invoices, precommits and other ways to cut costs.

• The title is “8 Ways to Cut Cloud Consumption Costs”.
• An article proposing “8 ways of thinking for optimizing the use of the organization’s cloud”.

Tools

Paketo is a new project, providing a set of curated, high quality, cloud native buildpacks for common languages and frameworks. Use with pack or other tools to build container images quickly with minimal configuration.

• Introducing an OSS tool “Pake to Buildpacks” that uses the framework of “Cloud Native Buildpacks”, a sandbox project of CNCF. It is said that it will be easy to build and patch containerized applications.
• Click here for the io page.

SRE Weekly Issue #216 April 26th, 2020

Articles

How to create an incident response playbookAwesome resource!
In each section, they explain what to include, why to include it, and an example from their playbook.
Blake Thorne — Atlassian

• An article explaining “How to create a playbook for disability” by Atlassian. The comment of mine was the same with the editor of SRE Weekly, so I will skip it.
• The “Incident Management Handbook” created by Atlassian is distributed free of charge at this link , so if you are interested, please enter your affiliation and email address and you can receive it by email.

Failover Conf Wrapup
I didn’t make it to Failover Conf, and it sounds like I missed a great time, so I’m especially grateful for this writeup.
Rich Burroughs — FireHydrant

• I will skip it because I mentioned it in DevOps Weekly above.

Failover Conf, a Recap of Gremlin’s Epic Virtual EventAnd this one!Hannah Culver — Blameless

• She talks about the significance of the Failover Conf, keynotes, good sessions, and efforts as good events. This person’s article often appears in this blog, and I like it a lot because It provides me with inspiration.
• The phrase “Gremlin gave us everything we wanted socially from a conference sans the free tee shirts.” was my favorite expression.

COVID-19 Oncall Survey
I’m a little late with this one, sorry folks! Survey ends tomorrow, April 27.

This is an anonymous survey to look at the impact that COVID-19 has had on oncall teams in tech.FireHydrant

• The COVID-19 Oncall Survey was an anonymous survey that confirmed how COVID-19 affected the on-call team in the tech industry. The deadline was 4/27, so it was closed.

Incident Analysis: How Learning is Different Than Fixing

Most post-incident review documents are written to be filed, not written to be read.

This slide deck is awesome and well worth the read.John Allspaw — Adaptive Capacity Labs

• Since I touched it upon on DEVOPS WEEKLY ISSUE #486 last week , I will skip it.

How to build robust anomaly detectors with machine learning
A deep dive into the math behind anomaly detection.Nikita
Butakov — Ericsson

• The theme is “How to establish a robust anomaly detection system with a machine learning model”. As the theme for the future is how to integrate this learning model into the production environment once it has been developed,” that blog seems to be interesting, so I look forward to it.

Advice for On-call Teams During COVID-19This article brings together thoughts on on-call work during the pandemic from folks at different companies.Rich Burroughs — FireHydrant

• An article introducing “Advice to On-Call Teams in the COVID-19 Crisis” by Rich Burroughs, Senior Developer Advocate of FireHydrant, who is the same author as “Failover Conf Wrapup” above.
• He already wrote the advice himself in another blog post as a self-introduction, but other people also gave advice from a different perspective, so he was introducing the name and content of the proposer.

Shadowing a Site Reliability Engineer A frontend engineer shares their key takeaways from their time shadowing.Laura Montemayor — GitLabA frontend engineer shares their key takeaways from their time shadowing.Laura Montemayor — GitLab

• Laura Montemayor , a frontend engineer at GitLab, had the opportunity to work (shadow wing) with the company’s SRE team, so they shared their experiences and insights.

Outages

• GitHub
• DataDog
• Poloniex
• DigitalOcean
• Apple Pay
• ShipStation
• Sendy
• Sharp online store and IoT devices
  ○ Sharp retooled one of its factories to produce masks and started selling them commercially. The increased load caused problems with their online store and existing consumer IoT devices.
• Discord
• Fastly
  Also a control plane issue earlier the same day.Full disclosure: Fastly is my employer.
• reddit

KubeWeekly #214 May 1st, 2020

The Headlines

Editor’s pick of the highlights from the past week.

CNCF Announces Helm Graduation

“When we set out to build Helm, we had two goals in mind,” said Matt Butcher, principal software development engineer at Microsoft and Helm maintainer. “First, make it easy for Kubernetes newcomers to get started deploying cloud native applications. Second, provide a package manager that can scale with enterprise needs. With today’s graduation announcement, we’re happy to see how these two goals have been fulfilled on a larger scale than we ever imagined!”

Help us congratulate the Helm team on this exciting milestone.

• Announced that the CNCF recognized the OSS package manager “Helm” as the 10th highest grade “Graduation” that represents the maturity of the CNCF.

KubeCon + CloudNativeCon is Going Virtual

Did you hear the news? CNCF’s flagship conference will gather the community virtually from August 17–20, 2020. Join Kubernetes, Prometheus, Envoy, CoreDNS, containerd, Fluentd, Jaeger, Vitess, TUF, OpenTracing, gRPC, CNI, Notary, NATS, Linkerd, Helm, Rook, Harbor, etcd, Open Policy Agent, CRI-O, TiKV, CloudEvents, Falco, Argo & Dragonfly as the community gathers for four days to further the education and advancement of cloud native computing.

• KubeCon + CloudNativeCon Europe 2020 will be held online and the schedule will be from August 17th to August 20th.
• The cost is $75 USD. Since the hurdles to participate have dropped all at once, I think I should take part during the summer vacation.

ICYMI: CNCF Webinars

Weekly recap of CNCF member and project webinars that you might have missed.

You can view all CNCF recorded and upcoming webinars here

CNCF Ambassador Webinar: CKA / CKAD

Christian Jantz, Cloud Strategist @ Level 25

• Level 25 Cloud Strategist and CNCF Ambassador Christian Jantz’s Webinar video explaining best practices for “CKA/CKAD”.
• I explain the environment and tips very carefully, so if you look at it before taking the exam, your anxiety will be alleviated. Personally, I would like to take CKA after summer when Corona calms down.

CNCF Member Webinar: Kuma: Service Mesh and the Future of Application Connectivity

Marco Palladino CTO and co-founder @Kong and Kevin Chen, Developer Advocate @Kong

• A webinar video entitled “The Future of Service Mesh and App Connectivity” by Marco Palladino, CTO and co-founder of Kong, and Kevin Chen, Developer Advocate of Kong.
• It explains carefully from the background of the service mesh, and the intention to “communicate” comes from gestures and words. There is a demo while explaining it, so I would like to carefully review it.

CNCF Member Webinar: Building Zero Trust based Authentication in Healthcare with SPIRE

Bobby Samuels, Vice President, AI Technology @Anthem, Inc., Frederick Kautz, Head of Edge Infrastructure @Doc.ai, Emiliano Berenbaum, Chief Technologist, HPE Labs @Hewlett Packard Enterprise (HPE), and Madhukesh Wali, Software Engineer @HPE

• Bobby Samuels, AI Technology Vice President, Anthem, Frederick Kautz, Head of Edge Infrastructure, Doc.ai; Emiliano Berenbaum, HPE Labs Chief Technologist, HPE; Madhukesh Wali, Software Engineer, HPE With the theme of “Establishing zero trust-based certification in the healthcare industry with SPIRE”, he introduces zero truss architecture, design trade-offs, core competencies, a wide range of use cases and ecosystems. Webinar video.
• Presenters share the roles and relay. There is also a demo in the latter half.

CNCF Member Webinar: Best Practices In Implementing Container Image Promotion Pipelines

Baruch Sadogursky, Head of DevOps Advocacy @JFrog

• JFrog’s Head of DevOps Advocacy, Baruch Sadogursky, titled “Best Practices for Implementing a Promotion Pipeline for Container Images’’, comparing different approaches, listing desired features, and creating a pipeline using OSS. I’m doing a webinar video.

CNCF Community Webinar: How to Conduct a GREAT Live Stream

Alex Lustenberg, Jorge Castro, Chris Short

• Alex Lustenberg, Jorge Castro, and Chris Short titled “How to Conduct a GREAT Live Stream” to explain how to use YouTube, OBS, Twitch and more to get involved in the community for CNCF member projects. Webinar video.
• I think that the environment is different from the United States, but I think it will be helpful for those who are considering live streaming. They talk for about two hours .

The Technical

Tutorials, tools, and more that take you on a deep dive into the code.

Life of a Vitess Cluster

Sugu Sougoumarane

• An article by PlanetScale’s CTO and Vitess co-creator Sugu Sougoumarane explaining what’s happening behind the scenes when a cluster of Vitess comes up.
• “This article can be a learning tool or a troubleshooting guide,” he said. When I take care of Vitess, I think I will read the materials around here.

Two-phased Canary Rollout with Open Source Gloo

Author: Rick Ducott | GitHub | Twitter

• Introduction article of kubernetes.io of OSS tool “ Gloo ” which does 2 stage canary rollout .

Kubernetes state management with Pulumi and Python

Maksim Ramanouski, HSBC

• The author was looking for a tool to automate the cluster configuration after Kubernetes provisioning, but he was not satisfied with Ansible/Terraform/YAML files/kustomize/kpt etc, but he tried Pulumi and liked it.
• When Pulumi was featured in Episode #76 of the Kubernetes Podcast, he was glad to see a developer thinking about the same thing. He said that he tried it because it had good support, like, languages(Python, Javascript, Typescript, Go, .NET, etc.), a number of cloud providers, YAML, and Helm charts.

Simulating Clock Skew in K8s Without Affecting Other Containers on the Node

Chengwen Yin, PingCAP

• Introducing the new function “Time Chaos” that can simulate the “ clock skew “ phenomenon of Chaos Mesh™ . The feature is that it can be executed without affecting another container on the same node.
• I thought, “Chaos engineering is everything.” First of all, we must start by understanding the phenomena/principles caused by chaos.

Inside Kinvolk Labs: Investigating Kubernetes performance issues with BPF

Alban Crequy, Kinvolk Labs

• A story of investigating Kubernetes performance issues using BPF.
• It uses the CPU profiler , a BPF tool from The BPF Compiler Collection (BCC) project, a toolkit for efficient kernel tracing and program manipulation .
• BPF is also read, so I have to put my hand on it.

kubernetes-azure-devops-dragdrop

A small node-js app triggering a small Azure-Devops pipeline to deploy multiple HELM-Charts into any Kubernetes cluster.

• GitHub page of OSS tool “kubernetes-azure-devops-dragdrop” that applies a single Azure-Devops pipeline to Helm charts to multiple Kubernetes clusters by drag and drop.

Istio Upgrades Reloaded | A demonstration of revisions tech in 1.6

Steven Dake, IBM

• Video from IBM’s YouTube “IBM Developer” channel. A demo video introducing the “revision” function of Istio v1.6.
• An article explaining routing from outside the cluster of Kubernetes to services running inside.

An (Opinionated) Guide to Understanding Kubernetes Network Architecture

Richard Li

• An article explaining routing from outside the cluster of Kubernetes to services running inside.

Encrypting and storing Kubernetes secrets in Git

Atul Bhosale, codemancers

• For GitOps, the secret of Kubernetes is encrypted and stored in Git.
• Kubeseal is used for encryption of Secret .

Why strace doesn’t work in Docker

Julia Evans

• An article that delves into a hypothesis about why strace cannot be used with Docker.
• I got a glimpse of the process from the process to the source code, which was helpful.

The Editorial

Articles, announcements, and morethatgive you a high-level overview of challenges and features.

Open Policy Agent, with Tim Hinrichs and Torin Sandall

Craig Box and Adam Glick, Kubernetes Podcast from Google

• Kubernetes Podcast by Google employees. The current co-hosts are Craig Box and Adam Glick.
• Guest guests include Open Policy Agent (OPA) creator and Styra CTO and Co-founder Tim Hinrichs, and the company’s VP of Open Source, Torin Sandall.
• The topics of interest in News of the week are: This week, I’m focusing on the following, but there are many topics.
  ○ Anthos for AWS is now Generally Available
  ○ TechCrunch coverage
  ○ Changes to Kubernetes release cycles for 2020
  ○ Aqua Security announces Dynamic Threat Analysis

Announcing Vitess 6

Morgan Tocker

• Vitess 6 GA release article on cncf.io. A short period of 12 weeks since the last release.
• The major updates listed here are improved SQL support, Kubernetes Topology Service, and GA for VReplication-based workflows.

Essential Kubernetes Extensions Explained

Oleg Chunikhin, Kublr

• Kubernetes is just one piece of the puzzle as an element necessary for development in a production environment, and extensions and plugins are the key in the Kubernetes stack,” the article that explains.

Tidepool Replatforms Diabetes Data Service from Custom Tooling to Open Source with Gloo and Kubernetes

Betty Junod

• Platform update case introduction article using OSS tools Gloo and Kubernetes from in-house tools of Tidepool which is NPO (nonprofit organization) (attached on September 25, 2019).

Container security meets Kubernetes: What IT pros need to know

Matthew Heusser, TechRepublic

• He points out what IT professionals need to know when working with Kubernetes, and touches on services and tools such as StackRox, splunk, PagerDuty, SumoLogic, and Istio.

Couchbase Updates Operator for Database Running on Kubernetes Clusters

Mike Vizard, Container Journal

• An introductory article on the update of Couchbase Operator.

How to scale a deployment within a Kubernetes cluster

Jack Wallen, TechRepublic

• It explains how to create Kubernetes resource Deployment, kubectl apply command which applies the difference of the updated YAML file, and kubectl scale command which edits replicas by command.

Upcoming CNCF webinars

You can check some Recorded Webinars and Upcoming Webinars here. The following are posted as Upcoming CNCF webinars at that moment.

Project webinar: Kubernetes 1.18
Kubernetes release team
May 1, 2020 9:00 AM Pacific Time

Member Webinar: Making the Most of Helm 3
Dan Garfield, Full-stack engineer @Codefresh
Anna Baker, Software Engineer/Technical Writer and DevOps Evangelist @Codefresh
May 5, 2020 10:00 AM Pacific Time

Member Webinar: The KUbernetes Test TooL (kuttl)
Gerred Dillon, Principal Engineer @D2iQ
May 6, 2020 10:00 AM Pacific Time

Member Webinar: Encrypting data in Kubernetes deployments. Protect your data, not just your Secrets
Maksim Yankovskiy, VP of Engineering @Zettaset
May 6, 2020 1:00 PM Pacific Time

Member Webinar: How AWS uses Firecracker and Fargate to run serverless Kubernetes pods in Amazon EKS
Mo Ziyuan 莫梓元 解决方案架构师 @AWS
This webinar will be delivered in Chinese.
May 7, 2020 10:00 AM China Standard Time

Member Webinar: Everything you need to know about Storage for Kubernetes
Alex Chircop, Founder and CEO @StorageOS
May 7, 2020 10:00 AM Pacific Time

Member Webinar: How OpenTelemetry is Eating the World
Steve Flanders, Director of Engineering @Splunk
May 8, 2020 10:00 AM China Standard Time

Member webinar: Data Services for Cloud Native Workloads
Diamanti
May 12, 2020 10:00 AM Pacific Time

Member Webinar: Piraeus: Dynamic Provisioning, Resource Management and High Availability for Local Persistent
Philipp Reisne, CEO @Linbit
Sun Liang, 资深存储架构师 @DaoCloud
Alex Zheng, 资深存储工程师 @DaoCloud
This webinar will be delivered in Chinese.
May 13, 2020 10:00 AM China Standard Time

Member Webinar: The Rosetta Stone Guide to Compliance in a Cloud-Native World
Cynthia Burke, Program Manager @Capsule8
May 13, 2020 10:00 AM Pacific Time

Member webinar: Cloud Native Monitoring: Scaling Prometheus
Aaron Newcomb, Director, Product Marketing, Monitoring @Sysdig
Carlos Arilla Navarro, Technical Marketing Engineer @Sysdig
May 19, 2020 10:00 AM Pacific Time

Member Webinar: Take Your Monitoring to the Next Level
Liran Haimovitch, Co-Founder & CTO @Rookout
Mickael Alliel, DevOps @Rookout
May 21, 2020 10:00 AM Pacific Time

Project Webinar: Harbor, the trusted cloud native registry for Kubernetes
Michael Michael, VMware
May 28, 2020 10:00 AM Pacific Time

Member webinar: Kubernetes Cost Allocation Done Right
Webb Brown, Co-founder and CEO @Kubecost
June 24, 2020 10:00 AM Pacific Time

Member Webinar: Pivoting Your Pipeline from Legacy to Cloud Native
Tracy Ragan, CEO of DeployHub and CDF Board Member
June 30, 2020 10:00 AM Pacific Time

How about those articles? Do you have any interest in any?

Actually, I have some contents which I can not digest at this stage, I’ll make use of this aide-memoire and links for catching-up for myself too.

Bye now!!

Yoshiki Fujiwara