SRE / DevOps / Kubernetes Weekly Collection#38(Week 43)

  • In this blog post series, I collect the following 3 Weekly Mailing List I subscribe to, leave some comments as an aide-memoire and useful links.
  • Actually, I have already published the same content in my Japanese blog and am catching-up in English in this series.
  • I hope it contributes to the people browsing this kind of information as a reference.

DEVOPS WEEKLY ISSUE #512 October 18th, 2020
SRE Weekly Issue #240 October 18th, 2020
KubeWeekly #238 October 23rd, 2020

DEVOPS WEEKLY ISSUE #512 October 18th, 2020


Measurement is one of the pillars of devops, but what KPIs should you measure to access your adoption of devops practices? This post has some suggestions and discussion.

  • The title is “How to Ensure the Success of DevOps in Your Organization”.
  • It describes the business value that DevOps offers, focusing on the following five most important KPIs that can help you measure the success of DevOps in your organization.
  1. Deployment Frequency
  2. Change Lead Time
  3. Defect Volume and Escape Rate (Error Budget)
  5. Application Performance

Versioning software is one of those problems that on the surface appears simple and quickly goes downhill from there. This post explores some of the nuances of the semantic versioning approach.

  • The title is “The Semantics of Semantic Versioning?”.
  • The following two perspectives on semantic versioning, which the author did not write in the essay “ Putting the Semantics into Semantic Versioning “ that he participated in as a co-author, are explained.
  1. Downstream.
  2. Upstream.

Not quite an experience report, but some observations about the new Hashicorp workflow tool Waypoint.

  • The title is “Waypoint”.
  • About Waypoint announced by Hashicorp , the author philosophically explains why he believes the design of Waypoint holds a lot of promise and potential. For tutorials and instructions on how to use it , refer to the official documentation and tutorials.

An interesting post on the barriers to entry for new technologies in large organisations coming from security due-diligence.

  • The title is “The Infosec Apocalypse”.
  • From the author’s point of view, the rise of InfoSec and vulnerability detection tooling makes it very difficult for existing or new entrants to see similar success without a massive coordinated push backed by big enterprise companies like Microsoft and Oracle. ing.
  • If you would like to discuss this, please join them on the Twitter thread , so you can follow the link.

A balanced post on maturity models for devops. Discusses pros and cons and some of the areas such models should generally cover.

  • The title is “DevOps Maturity Model What Is A Better Alternative?”.
  • It explains the following important areas that the “DevOps Maturity Model” is paying attention. The model evaluates teams and organizations and reviews DevOps skills and abilities.
    ○ Culture
    ○ Automation for DevOps Maturity Model
    ○ Process
    ○ Levels
    ○ Start
    ○ Mature
    ○ Final

Kubernetes is built to be extended, and this presentation covers some of the most common integration points, including custom resources.

  • The title is “Cloud Native Ambassador Day — Extending Kubernetes”.
  • Gianluca Arbezzano, CNCF Ambassador, introduces various integration points that can be used to extend Kubernetes, such as shared informers, controller, and kubectl plugins.

There are several best practices when it comes to writing Dockerfiles. This post describes a few of them and shows how to check for issues using Open Policy Agent.

  • The title is “Dockerfile Security Best Practices”.
  • The following list summarizes common security issues and their workarounds. We have also created OPA (Open Policy Agent) rules that can be used to statically analyze Dockerfiles using conftest for each problem. The.rego rule set is in this repository.
    ○ Do not store secrets in environment variables
    ○ Only use trusted base images
    ○ Do not use ‘latest’ tag for base image
    ○ Avoid curl bashing
    ○ Do not upgrade your system packages
    ○ Do not use ADD if possible
    ○ Do not root
    ○ Do not sudo


Boundary provides a secure way to access hosts and critical systems without having to manage credentials or expose your network. It’s a modern alternative to juggling SSH keys and managing bastion hosts.

  • The io page of a new open source Boundary announced by Hashicorp, which provides a secure way to access hosts and critical systems without managing credentials or exposing the network.
  • Click here for the GitHub page.

Waypoint is a new developer workflow tool that allows for defining build, deploy, and release lifecycle as code, and supports a plugin model with plugins for Docker, Kuberenetes, AWS, Azure, GCP, Netlify and more.

  • Following the above, the io page of the new open source “Waypoint” announced by Hashicorp. It’s the second appearance in this mailing list since it was mentioned in the above article. Last week there was a lot of talk around this series of announcements.
  • Click here for the GitHub page.

Use cert-manager in Kubernetes to automatically issue certificates for your Puppet infrastructure.

  • The Puppet Certificate Authority Issuer GitHub page that plugs into cert-manager as an external issuer to communicate with Puppet CA to get a certificate issued for the Kubernetes environment.

SRE Weekly Issue #240 October 18th, 2020


Google Cloud Issue Summary — Google Chat — 2020–09–17

This interesting post-incident analysis is marked as “Google Customer Confidential — Not for publication or distribution”, but Google linked it directly from their public status page. I normally would not include a seemingly “leaked” incident report like this, but in this case I think the “confidential” label is erroneous.


  • There is a phrase “Important: Google Customer Confidential — Not for publication or distribution” as the editor rushes in, but it is a dignified public material. 9/17 Google Chat trouble report.

40 milliseconds of latency that just would not go away

I keep re-learning and re-forgetting about TCP_NODELAY.

Rachel By the Bay

  • I didn’t know Nagle’s algorithm. I want to dig deeper around here, including the TCP_NODELAY option.

“Manual” and “Automated” are just words

The distinction between the two is a lot more nuanced than it may seem. What are we really trying to say with those words?

Michael Nygard

  • As the title suggests, rather than being trapped in the mere swamp of the words “Manual” and “Automated,” breaking down the questions will help you understand how processes provide availability, stability, security, and more. The explanation is given by taking incident response as an example.

Heroku incident #2110 follow-up

This incident from the week before last involved a Let’s Encrypt API rate limit.

  • Heroku follow-up report for an incident. SSL connections to the domain were affected between October 7, 2020 19:30 UTC and October 8, 2020, at 01:46 UTC.

Fixing Linux filesystem performance regressions

Don’t you hate when you’re minding your own business upgrading your OS, and you run smack into a kernel bug in the ext4fs code?

…ext4 performance on kernel versions above 4.5 and below 5.6 suffers severely in the presence of concurrent sequential I/O on rotating disks.

Ryan Underwood — LinkedIn

  • They share the challenges and lessons they faced with the titled challenges.

Identifying and protecting against the largest DDoS attacks

Google discusses DDoS attacks and how they deal with them, including a 2.5Tbps attack in 2017.

Damian Menscher — Google

  • An article that introduces the exponential growth of DDoS attack volumes by Google, and its efforts and countermeasures.
  • In some cases, the details of the specific defense approach are given to the third SRE book, “Building Secure and Reliable Systems.” “Working together for collective security” is interesting and I think it’s an approach to take.

How I Broke git push heroku main

I love these first-hand incident stories. This one is from an engineer at Heroku who was a contributing factor in an incident last month.

Damien Mathieu — Heroku (Salesforce)

  • Like the editor, I enjoyed reading this story. Expectations for the following swelled in the following writing part. I’m glad to hear that he didn’t get fired.
    ○ In this post, we will look at one of those incidents, #2105, see how it happened (spoiler: I messed up), and what we’re doing to avoid it from happening again (spoiler: I’m not fired).


KubeWeekly #238 October 23rd

The Headlines

Editor’s pick of the highlights from the past week.

Local Kubernetes Clusters Overview

Ellen Körbes & team

A local development cluster is a great way to get started with Kubernetes. But when looking for a local cluster, there are way too many options. This article aims to clarify how to choose the best one for your use case, and provides useful general recommendations. (An overview of Kind, Microk8s, K3s, Minikube, and Docker for Desktop.)

  • The following tools for running a Kubernetes cluster locally are briefly introduced in about 7 minutes, including features and usage. Personally, it’s about time I need to try it in earnest, so I will take it.
    Docker for Desktop

Join us for KubeCon + CloudNativeCon North America Virtual 2020!

The countdown to KubeCon + CloudNativeCon North America is on. Have you reserved your spot?

Whether you are new to cloud native or well-versed, you can choose between two different ticket options including a Full Pass and Keynote Only! Don’t forget that special early-bird pricing is available through October 31, 2020, a savings of $50 off registration. Don’t delay — act fast!

  • KubeCon + CloudNativeCon North America is approaching, and the discount for paid participation tickets is until the end of October, so as a reminder.

ICYMI: CNCF Webinars

You can view all CNCF recorded and upcoming webinars here.

CNCF Member webinar: Delivering cloud-native apps to Kubernetes using werf

Dmitry Stolyarova CTO, @Flant

  • It shows how developers / release engineers / SREs can benefit from infrastructure, release management, and CI/CD pipelines using the CLI tool “werf”.

CNCF Member webinar: How to migrate NF or VNF to CNF without vendor lock-in

Grzegorz Sikora, VP Business Development @OVOO, Rafał Myśliwiec, Software Engineer @OVOO, and Paweł Kulpa, Software Engineer @OVOO

  • It explains how to use ONF EPC, CNCF, or other open source projects to achieve significant cost savings and move from legacy network capabilities or vendor-specific VNFs to cost-effective, flexible, and promising.
  • I am not used to the diagram when CockroachDB is mixed in multiple topology diagrams…..

CNCF Member webinar: Deploying Kubernetes to bare metal using cluster API

Seán McCord, Principal Senior Software Engineer @Talos Systems, Inc.

  • Along with the title, it introduces Talos Systems’ new collection of open source tools based on the Cluster API project.
  • Demonstrations of Kubernetes deployment and subsequent OS upgrades are provided, explaining future plans for these tools.

CNCF Member webinar: The abc’s of Kubernetes security

Roger Klorese, Senior Product Manager @SUSE & Danny Sauer, Senior Software Engineer @SUSE

  • It guides users in getting started with areas of security concern, from system hardware to the OS stack, Kubernetes itself, and deployed apps. It is explained at the beginning that it is a viewpoint as an organizational stance rather than a technical viewpoint.

CNCF Member webinar: K8s audit logging deep dive

Randy Abernethy, Managing Partner @RX-M

  • It briefly describes the more complex and deep issues that Kubernetes operators face when trying to make Kubernetes auditing a comprehensive, efficient, and secure implementation, with the following points:
    ○ Multiple API servers
    ○ Mutating Admission Controller
    ○ Webhooks
    ○ Aggregated APIs
    ○ Webhook audit log backends
    ○ Massive API throughput requirements

The Technical

Tutorials, tools, and more that take you on a deep dive into the code.

Waypoint — Solving Build, Deploy and Release problem

Saiyam Pathak, Civo

  • An article by Civo that introduces “Waypoint,” which was featured in another article last week. Explains how to install and use Waypoint, build and deploy sample apps, and release them to a Civo Kubernetes cluster. The experience looks good.

Flink on PaaSTA: Yelp’s new stream processing platform runs on Kubernetes

Antonio Verardi, Engineering Manager, Yelp

  • As the title suggests, it introduces the configuration of PaaSTA , a new stream processing platform using Apache Flink that runs on Yelp’s Kubernetes.

Understanding and mitigating CVE-2020–8563: vSphere credentials leak in the cloud-controller-manager log

Kaizhe Huang, Sysdig

  • It describes which parts of Kubernetes are affected by a problem that can cause sensitive data breaches (CVE-2020–8563) and how to mitigate it.

COBOL is Kubernetes and OpenShift

JJ Asghar, IBM and Chris Short, Red Hat

  • A YouTube video of IBM Developer Advocate JJ Asghar and Kube Weekly Editor Chris Short talking about the subject. I personally have no experience or knowledge with COBOL, so I thought, “Let’s dig deeper when the opportunity meets…” and it never happened yet.

Quick tip: How Prometheus can make visualizing noisy data easier

Ronald McCollam, Grafana Labs

  • Using Prometheus, it explains how to make visualizing noisy data easier using temperature sensor data as an example.

Introducing pvc-autoresizer

Akihiro Ikezoe, Kintone

  • It describes a new Kubernetes-native OSS tool “pvc-autoresizer” that automatically increases the size of PVCs(Persistent Volume Claims) on a Kubernetes cluster.
  • Click here for the GitHub page.

Container networking is simple

Ivan Velichko

  • It explains the container network organizing approach and answers the following questions.
    ○ How to virtualize network resources to make containers think each of them has a dedicated network stack?
    ○ How to turn containers into friendly neighbors, prevent them from interfering, and teach to communicate well?
    ○ How to reach the outside world (e.g. the Internet) from inside the container?
    ○ How to reach containers running on a machine from the outside world (aka port publishing)?

A Hitchhiker’s Tour to Containerizing a Java application (talk)

Nicolas Frankel

  • The following solutions, including Benefit/Downsides, are carefully explained with demonstrations according to the theme.
    ○ Multi-stage build
    ○ Jib
    ○ Layer-aware multi-stage build
    ○ Buildpacks
    ○ Spring Boot Maven plugin

Introducing HA MicroK8s, the ultra-reliable, minimal Kubernetes


Kubernetes 1.19: The future of traffic ingress and routing

Michael Vittrup Larsen, eficode

  • Kubernetes 1.19 focuses on the movement around “Ingress” resources upgraded from beta to GA, scaling issues and solutions.

The Editorial

Articles, announcements, and morethatgive you a high-level overview of challenges and features.

Research, Steering and Honking, with Bob Killen

Adam Glick and Craig Box, Kubernetes Podcast from Google

Fear and loathing in YAML

Chris Short, Red Hat

  • An article by Chris Short of Red Hat, editor of KubeWeekly, DevOps’ish, etc., who feels the fear, loathing, and struggles with YAML.

Cloud Foundry coalesces around Kubernetes

Frederic Lardinois, TechCrunch

  • TechCrunch’s article. It explains the movement of re-platform to Kubernetes of the Cloud Foundry project in 2020, touching on the movement such as the change of executive director.

Helm turns 5, and GitHub gives the gift of charts

Matt Butcher & Matt Farina

  • Helm blog post commemorating Helm’s 5th anniversary. The transition of the Git repository and the following announcements are made.
    ○ Helm’s stable and incubator repositories are hosted directly on GitHub.
    ○ The official Helm GitHub Actions are now available on the GitHub marketplace.
    ○ Check out Helm Chart Releaser for how to host Helm charts on GitHub.

For PayIt, cloud native is a ‘competitive advantage’ for getting government services online


  • “PayIt” is featured in the CNCF case study article. I was wondering what happened to the government agencies in Japan.
  • There may be various ways of thinking, but if the procedures at the government office are less complicated and the functions such as redistribution can be performed, I think it’s okay for the people at the government office to play and live.
    ○ If the service is provided properly, they can do other jobs or do activities rooted in the community.
    ○ Many people have not been able to enjoy the value despite officers’ efforts, and I feel that the structure and standing position of Japanese government offices are now in a lose-lose relationship, especially in terms of procedures and redistribution.

Experiment with OpenTelemetry: Play with the Future Cloud Native Metrics Framework, Now

Ambassador Podcast

As the title suggests, it gives an overview of “ OpenTelemetry “, which is a Cloud Native Metrics Framework, and provides a method for starting in a sandbox environment using “ K8s Initializer “. It also provides links to materials to get you started with OpenTelemetry and Kubernetes.

Upcoming CNCF webinars

You can check some Recorded Webinars and Upcoming Webinars here. The following are posted as Upcoming CNCF webinars at that moment.

Member Webinar: The truth about the service mesh data plane
Christian Posta, Global Field CTO
Idit Levine, CEO and Founder

Oct 27, 2020 10:00 AM Pacific Time

Member Webinar: Admission controllers: one part of your Kubernetes security and governance toolkit
Gunjan Patelm, Cloud Architect @Palo Alto Networks
Robert Haynes, Cloud Security Evangelist @Palo Alto Networks
Oct 28, 2020 7:00 AM Pacific Time

Member Webinar: Event-driven architecture with Knative events
Nicolás López, Senior Software Engineer @Google
Bryan Zimmerman, Product Manager @Google
Oct 29, 2020 10:00 AM Pacific Time

Member Webinar: Managing your policies and standards
Ahmed Badran, Chief Technology Officer @Magalix
Nov 4, 2020 7:00 AM Pacific Time

Member Webinar: Security in the world of service meshes
John A. Joyce, Principal Engineer @Cisco
Nov 4, 2020 1:00 PM Pacific Time

Member Webinar: Building edge as a service
Dr. Bin Ni, CTO @Wangsu Science & Technology / CDNetworks
Nov 5, 2020 10:00 AM Pacific Time

Member Webinar: Developer-friendly platforms with Kubernetes and infrastructure as code
Lee Briggs, Staff Software Engineer @Pulumi
Nov 6, 2020 10:00 AM Pacific Time

Member Webinar: MicroK8s HA under the hood: Kubernetes with Dqlite
Konstantinos Tsakalozos, Senior Software Engineer @Canonica
l Nov 11, 2020 7:00 AM Pacific Time

Member Webinar: The what and why of distributed tracing
Dave McAllister, Sr. Technical Evangelist @Splunk
Nov 13, 2020 10:00 AM Pacific Time

Member Webinar: Metal³: Kubernetes-native bare metal host management
Maël Kimmerlin, Senior Software Engineer @Ericsson Software Technology
Dec 10, 2020 10:00 AM Pacific Time

How about those articles? Do you have any interest in any?

Actually, I have some contents which I can not digest at this stage, I’ll make use of this aide-memoire and links for catching-up for myself too.
Bye now!!

Yoshiki Fujiwara

An infra engineer in Tokyo, Japan. Grew up in Athens, Greece(1986–1992). #Network, #Kubernetes, #CKA, #CKAD, #Certified AWS SAP

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store