KubeCon | CloudNativeCon EU edition, en francaise ç’année.

JWorks and friends at KubeCon 2024

Introduction

This year the cloud-native conference is hosted in the capital of love: Paris, France. The convention center, Expo Port de Versailles, is a huge complex. Funnily enough, it’s so big that there are other (non-kubecon | cloudnativecon) events going on at the same time as Kubecon in the conference center. Pavilion 7, where the conference is hosted, is a very nice building which looks very recent. The biggest difference with the previous conferences we attended (Valencia and Amsterdam) is that everything is very close to each other. The building has 3 floors: the main entrance, the showcase floor and the talk-rooms-floor. All talks are held in rooms that are oriented in a large square, with the Paris Keynote room being in the middle. This makes it very easy to quickly switch between rooms, especially compared to the 5-10 minute walks we were used to in Valencia and Amsterdam.

The venue map

The Paris Keynote room could accommodate a very large crowd. Even from quite a few rows back, you could easily attend and perfectly follow all keynotes. Since this was the biggest room in the venue, it was used for very popular talks during the conference as well.

The Paris Keynote room

As is customary, the day before the start of KubeCon, all co-located events happened in Paris. This year there was a significant list of co-located events like ArgoCon, OpenTofu Day and Cloud Native StartupFest just to name a few.

The OpenTofu track was very interesting as the project is very new to the Linux Foundation (and hopefully soon part of the CNCF). Maintainers, users and enthusiasts shared their usage and views on OpenTofu. The maintainers gave a glimpse of insight into the possible future of OpenTofu. Spoiler alert, it might be moving closer to being a real programming language instead of a hybrid declarative with support for expressions.

Another very interesting track was the Cloud Native StartupFest. This track, hosted by Kelsey Hightower, was an interesting insight into the world of startups and venture capital. Kelsey created a great program for (potential) startup owners to attend. Some VCs explained how their selection process works and how they work with open-source projects. Someone explained the tradeoffs between open-source and open-core licenses from a business perspective. Tim Enwall shared lessons learned from being involved with startups for over 25 years. If you’re interested in how startups work, get funded or how open-source licensing is linked to business models, all of these talks are a must-watch for you.

Highlighted talks

Atlantis and OpenTofu

Atlantis is an automation solution that assists teams in managing their Infrastructure as Code rollouts by integrating the automation process into their Source Control Management system (E.g. GitHub). It can run plans on code branches and provide the plan output in the pull request associated with the branch. When a user wants to apply the changes (the plan is approved), Atlantis can roll out the required changes directly. The biggest selling point for this is that the user never has to leave the SCM to manage rollout out changes.

The core maintainers of Atlantis shared the history of the project and how they became owners of the project. They shared what their journey looked like and provided an insight into what the future holds for Atlantis.

Today, Atlantis doesn’t natively support OpenTofu as a first-class citizen yet, but efforts are ongoing to change this (Epic link: https://github.com/runatlantis/atlantis/issues/3741). Since Atlantis supports custom workflows, there is a way to use OpenTofu already today with Atlantis.

If you’re running Terraform or OpenTofu setups at scale, this is surely a project to check out. One of the few downsides is that Atlantis expects to receive webhooks from your SCM. This requires Atlantis to be exposed on the public internet if you’re using a SaaS-based SCM. There are workarounds:

• limit the IPs that can access the endpoint to those linked to your SCM
• Put a reverse proxy in front of Atlantis and heavily filter the traffic
• (our favorite) use natively integrated CI solutions like AWS CodePipeline and “fake” the webhook from there

Startup lessons from 25 years and 5 startups & Panel: Venture Captial and Open Source

These were two talks in the StartupFest co-located event of Kelsey Hightower. The first talk discussed six useful lessons for doing a startup. Tim Enwall from Fermyon shared his experiences and gave startup owners key things to consider when managing their startups. One of the takeaways that was mentioned multiple times during other talks as well was: don’t be afraid of giving away one or two percent more of shares when discussing Venture Capital. The key reason is that the percentages don’t make a difference in case the startup fails, 1 or 2 percent of 0 is still 0. On the other hand, if the startup does very well, you’re at the point where it does not significantly affect your outcome either because you’ll be earning tons of money anyway.

Another interesting lesson Tim shared was that VCs are a necessary evil. I’m not sure if this was planned or not, but the talk directly afterward was a panel with VCs. The VCs shared how to evaluate potential investments are provided suggestions on how a startup can make itself more interesting to a VC. The two takeaways here were: the best way to some product-market-fit is by having paying customers using your solution and secondly, show the VC data that backs any claims you’re making w.r.t. customer adoption and market interest in your product.

Kubernetes Maintainers Read Mean Comments - Tim Hockin (Google) & Davanum Srinivas (AWS)

Being an open-source developer can be a very satisfying and rewarding job. Engaging with a large community, particularly with famous open-source software, often leads to heartwarming feedback from dedicated users. Open-source is more than just the open development of a product. It has a widespread community, people contribute by either reporting issues on the issue tracker, by submitting a pull request to improve the software, participating in discussions on Slack or public forums, attending public meetings, … . Sadly, it can also be a thankless job and not all people realize that open source software is something that anyone can contribute to, and it doesn’t really have any limitations as to how someone can contribute either.

Tim Hockin (Google) and Davanum Srinivas (AWS) both maintain the Kubernetes project on Github. During this talk, they showed some of the meanest, weirdest, most ridiculous comments written by Kubernetes users on public forums such as Reddit and their Github issue tracker. Most comments are about frustrations why it takes too long for a certain feature to be implemented or for a particular bug to be fixed. While some of them are hilarious, they really do not have a place in an open source environment.

People often wonder how it is possible that a bug is open for a few years and still hasn’t been fixed, despite Kubernetes being a very popular product. Tim and Davanum answer this question by simply saying that there are only a few engineers who are 100% assigned to the Kubernetes project. Engineers who are not assigned to the project try to contribute, but it’s often hard because of other commitments, whether those commitments are personal or business related. As Kubernetes is globally adopted and used in a lot of international companies, there are a lot of issues coming in. Sometimes, it’s too much, and they simply don’t have the time for it. But as they say, the Kubernetes project is open source, anyone can create a pull request if people want to see a certain issue fixed.

They also want to highlight that, even though you have some frustrations with the product or things are not working as they are intended, you still have to remember that engineers are people aswell. There is no place for rudeness, discourtesy and impoliteness. You are allowed to express your frustrations, but do so in a kindly manner, without personally offending anyone.

Not every comment is a bad comment, and they wanted to show that by ending their presentation with a series of thankful and heartwarming posts and comments about Kubernetes and the time and effort that is being committed in the Kubernetes project every single day.

Keep Hackers Out of Your Cluster with These 5 Simple tricks - Christophe Tafani-Dereeper & Frederic Baguelin

Currently, there is a limited set of useful threat intelligence tools available for cloud and containers, surrounded by a sea of noise. Therefore, Christophe & Frederic conducted some research and shared their finding in this interesting talk.

The talk began by presenting a minimalist threat model for managed Kubernetes clusters operating in a cloud environment. You can compare a threat model to a map of your house, used to identify potential entrypoints for burglars. In this context, it helps anticipate and prevent cyber-attacks by understanding the thread vectors.

The model illustrated several possible threats, ranging from exploiting application vulnerabilities and compromising the cluster to injecting malicious code into images.

They explained their research process, which involved studying various sources and using honeypots. This led to an extensive breakdown of attacks into two main categories: Controller plane attacks, which target the K8s API server directly and Data plane attacks, which attempt to exploit workloads within K8s itself.

As promised in their title, they provided five simple security mechanisms to prevent these attacks, along with practical insights and tool recommendations.

1. Control Plane Security: Ensure proper management of the control plane basic, which are mostly covered with managed K8s. Enforce authentication on the API server and minimize its network exposure.
2. Block Cloud Metadata Service Access: Prevent workloads from accessing the cloud metadata service. While GKE Workload Identity provides enough protection, AWS IAM roles for Service Accounts & Azure Workload Identity do not. Blocking IMDS access using a network policy is required for these platforms.
3. Understand Workload Privileges: Be aware of the access levels and permissions associated with your workload.
4. Intentional Cluster Management: Be deliberate about what runs in your cluster. Avoid creating unnecessary resources that may be forgotten or pose security risks.
5. Application Security: Utilize modern frameworks and regularly update libraries, frameworks, and dependencies.

In conclusion, the talk not only provided valuable insights into the complexities of securing cloud-native environments but also offered practical guidance for addressing these challenges effectively.

Product Market Misfit: Adventures in User Empathy - Mitch Connors, Aviatrix

Far too often, software developers invest a considerable amount of time and effort into building features or products, only to discover they are either disliked or unused by their target audience - a scenario we’ve encountered firsthand in our day-to-day work.

In his talk, Mitch Connors delves into this issue and provides valuable insights on how to tackle it.

Developers often make the mistake of building what they believe their users need, rather than what they actually require. This misalignment originates from differing perspectives of developers and users. Connors effectively illustrates this through the use of some real-world examples.

His key advice for tackling this problem is to foster empathy for your users:

1. Stand in their shoes; experience what they experience. By doing so, a shared perspective can be developed.
2. Maintain an open mindset and never stop listening. Embrace criticism, especially if you truly love what you’ve built.

While Connors’ talk certainly differed from all others we attended, it was very interesting and engaging. We strongly believe every software developer, as well as anyone involved in creating products for groups of users, could greatly benefit from the insights he provides.

Keptn: a deep dive

The talk starts with explaining why Keptn was created and why other tools like ArgoCD are not solution for this (because App health degraded).

Keptn tries to solve the problem of deployments failing and being hard to debug or trace. Think about image tag missing, a configmap can’t be mounted as volume, scheduling errors and network policy changes.

Keptn uses observability to track application health and not just pod health, improves the signal-to-noise ratio and allows you to promote your applications easily into other environments using your favourite CI/CD tool.

Why are probes not enough you ask? Kubernetes only checks if an application is responding and giving a 200 http ok.

Keptn goes further if an application that before responded to requests in say 200 ms, but your new version of the application responds in 600 ms; would you consider it a healthy application? No, and neither does Keptn if you configure it right.

Keptn allows you to define SLI, SLO and SLAs. This makes it easier to track your applications deployments real health and improves your day two operations.

So what Keptn does is provide you with a tool that solves the problem of having no Metrics about your deployments; too limited observability capabilities and improves the release life cycle management.

With the help of Keptn, you can monitor if an application was deployed, check if your application is healthy, perform tasks before, during and after the release life cycle management.

The day after, if all your SLI, SLO and SLAs are met, it can automatically trigger your tool of preference to promote into the next environment.

Keptn adds a subset of the DORA observability metrics and traces to your kubernetes events, making it possible to ingest them in any tool that supports OpenTelemetry

Argo loves Keptn and is exploring ways to better integrate Keptn into ArgoCD. They are asking the community how to better integrate it.

Conclusion: Keptn is a tool that uses OpenTelemetry to add observability to your deployments, track your SLAs/SLIs/SLOs and assist you in promoting applications across environments.

Check out their documentation for use cases and how to set it up.

The IAC evolution

This was a panel discussion compromised of industry leaders discussing the evolution of Infrastructure as Code (IaC), tracing its journey from early pioneers like CFEngine, Puppet, and Chef.

The panel illuminated a common challenge faced by frontline engineers: the exhaustion of continually adapting to new Domain-Specific Languages (DSLs) to manage infrastructure effectively. This frustration highlights the need for a more sustainable approach to IaC.

A proposed solution during the discussion is the transition towards mainstream programming languages such as Go, Java, or Python, exemplified by platforms like Pulumi, now made possible by the growing maturity of provider APIs. Another approach suggested is the utilization of dedicated platform teams, responsible for providing infrastructure platforms using DSLs with a steeper learning curve, such as Terraform.

As a platform engineer myself, I’ve observed a rising demand for dedicated platform teams at my current customer, where we specialize in providing bootstrapped accounts through AWS Account Factory for Terraform (AFT) while also providing and maintaining dedicated infrastructure or CI/CD solutions for development teams.

The panel featured esteemed members of the industry:

• Eran Bibi: Chief Product Officer & Co-Founder of Firefly
• Mandi Walls: DevOps Advocate for PagerDuty
• Roni Fratch: Director of Engineering at env0
• Solomon Hykes: CEO & Co-Founder of Dagger

Their collective insights shed light on the challenges and opportunities in the ever-evolving landscape of Infrastructure as Code.

Noteworthy talks

• State of Platform Maturity in the Norwegian Public Sector: this talk provides a great insight into how the Norwegian public sector adopted Kubernetes and Cloud-native solutions to enhance their technology stack.
• Cloud-Agnostic Approach to Bin-Packing pods in Managed Kubernetes in AWS, GCP and Azure: this talk provides some real-world optimizations that can be leveraged to improve the node utilization by making changes to the default scheduler in Kubernetes.
• We tested and compared 6 database operations: This talk shares (in a very funny pirate theme) the journey Enix made to select their preferred database Operators. TLDR; CNPG for PostgreSQL and the maria-operator for MariaDB.
• The party must go on - Resume pods after spot instance shut down: This talk gives an introduction and live demo of CRUI and CRIT which allows to essentially pause/hibernate containers to disk and restart them later without issue. This is a must-watch if you want to benefit from spot instances for batch workloads that don’t support being paused of re-located half-way through their job.
• How Spotify Re-Created Our Entire Backend Without Skipping a Beat: Spotify had to recreate every Kubernetes cluster they have, migrate half a million pods, all with zero downtime. In this talk, they briefly explain their experience in doing so.
• Building Container Images the Modern Way: Using a Dockerfile to build your Docker images isn’t considered the best practice nowadays. This talk explores different image building tools through live demos, helping you understand which tool might be the best fit for your needs.
• Why Is This so HARD? Conveying the Business Value of Open Source: How do we explain the value of Open Source to business? On why communicating a project’s value & risks is essential for its growth and sustainability, and which strategies to employ to better surface and visualize these metrics.
• Keynote Panel Discussion: Revolutionizing Cloud Native Architectures with WebAssembly: This keynote introduces Spinkube, a way to run WASM workload natively in Kubernetes.
• Keynote: Welcome + Opening Remarks - Priyanka Sharma, Executive Director, Cloud Native Computing: This demo shows how AI workloads can be easily ran and integrated into an application using Ollama.

Showcase highlights

At KubeCon, the Showcase room was the largest room of the venue packed with booths from big names like AWS, Microsoft, Google, VMWare, and Docker, alongside smaller companies as well as startups. They were all there to show off their latest stuff and give away cool goodies like gadgets and merchandise. Some booths had fun games to play, while others held raffles with prizes like LEGO sets, drones, and gaming consoles. It was a lively scene, with attendees buzzing around to check out what each booth had to offer.

AWS

At this year’s KubeCon, AWS didn’t have as big a setup as they did last year, with a smaller booth and fewer activities. But they still had some neat stuff to check out, like cool gadgets and socks up for grabs. One standout was an arcade machine with a twist - the game was inspired by Space Invaders, but instead of monsters, you were shooting at Kubernetes pods in a cluster. Shooting down Deployments & Pod Autoscalers stopped the pods from respawning.

AWS staff were around to chat with, offering demos of their latest features and hot topics.

On the final day, they hosted a raffle where lucky attendees could snag one of two LEGO sets just by swinging by their booth and signing up for the raffle.

Conclusion

The conference in “la Ville Lumière” (“the City of Light”) was an incredible experience, combining education with enjoyment. While the venue at Paris Expo Porte de Versailles was impressive, there was room for improvement in the quality of the food, although the desserts were delicious.

KubeCon is a great way for companies to share their latest and greatest tech. We noticed a lot of engagement towards these companies from Kubecon attendees. The wide variety of conference booths showcasing interesting companies (and, of course, gadgets/activities) added to the fun.

AI was all the rage this year at KubeCon. Because this is still a very evolving space and not a lot of companies have effectively embedded into their portfolio, most AI-related talks were “this might be possible someday”-talks or “getting very deep into the weeds”-talks. It would have been nice to have some more 101-level talks about how to run AI workloads in Kubernetes and to see more demos of AI-K8s-related products.

Similar to the previous year, some talks were so popular that they exceeded the capacity of their assigned rooms. Some talks had to be moved to even bigger rooms so the capacity could be handled. This was a very big improvement over previous KubeCons where it happened more than once that you couldn’t attend a very popular talk. This was a very big improvement over previous KubeCons where it happened more than once that you couldn’t attend a very popular talk.

Overall, the event was amazing and there were a lot of interesting talks, leaving us eagerly anticipating next year’s edition in London. We hope to see you there!