Add Helm v3 proposal - #5
Conversation
Also refactor by adding a top-level helm-v3 directory
|
|
||
| ## Summary | ||
|
|
||
| - Tiller is gone, and there is only one functional component (`helm`) |
There was a problem hiding this comment.
This seems very counter to the advances of helm2, reverting to a helm1 behavior. This feels counter productive. A lot of the issues I have with helm2 were related to behaviour that continues to not use tiller. (helm client repo lists rather then on the server for example)
There was a problem hiding this comment.
Oh, and this also seems in the opposite direction that the majority of the k8s community is going. They are moving more and more towards operator like models with the continued focus on declarative requests from users. It also pushes it further away from the addon-manager use case.
There was a problem hiding this comment.
@kfox1111 This isn't going against the direction of Kubernetes for a few reasons:
- Kubernetes is moving to CRDs for storing custom objects. Helm 3 is moving that route with the introduction of multiple CRDs and the Application CRD is the common route others, not just helm, are rallying around. Helm 2 was created before CRDs or custom controllers were a viable option. Helm 3 is moving to the common k8s conventions.
- Tiller has been a pain point for many cases. Rudder was introduced as an experiment to try and get around some issues but it wasn't really adopted. Auth[nz] has been an issue because it's not user creds it's service account creds (tiller per namespace is a thing). The tiller API was almost never used by others for interaction because it wasn't the same style people expected. Functionality is shifting to common k8s conventions now.
- Many people are trying to work around tiller today. The new architecture is based around solving common problems while updating to the latest k8s conventions.
Operators solve a slightly different problem from custom controllers which is a little different from the package manager that Helm is. Operators should be used on an application. A package manager could easily install an operator as part of the package it installs. There are cases of charts that do that. Helm doesn't sit at the same level as an operator. Helm 3 doesn't conflict with operators because they solve different problems.
There was a problem hiding this comment.
the CRD's do make it more like where k8s is going, yes. But the external, imperative driven command is farther away now.
| - Tiller is gone, and there is only one functional component (`helm`) | ||
| - Charts are updated with a new Chart.yaml, overlays, libraries, schematized values, and the ext directory | ||
| - Helm will use a "lifecycle events" emitter/handler model. | ||
| - Helm has an embedded Lua engine for scripting some event handlers. Scripts are stored in charts. |
There was a problem hiding this comment.
Why pick a winner here? we have a container orchestration engine. helm can launch any code in a container and call out to it.
There was a problem hiding this comment.
See appendix on what we considered.
There was a problem hiding this comment.
It's worth adding some context. Helm has to work on varying operating systems. Linux, Windows, Mac, etc. Today we have both Linux and Windows based containers. We can not assume one environment for either the place Helm runs or the containers in use.
Running a server side component (tiller) has introduced many pain points. Auth[nz] and other security considerations is one of them. How would you deal with an environment that has many namespaces, which cannot know details of each other, and deal with things like running container based workflows to generate k8s objects without leaking any details to the wrong place? And do with a simple UX. Also, inject CI tests like validators on those generated k8s objects. And, where does GitOps fall in?
This is a complex problem with many use cases to consider. We've spent months talking through the different options. We've talked with people at companies who use Helm and talked with k8s core leaders.
|
|
||
| The reference implementation has two distinct parts: | ||
|
|
||
| 1. The command line façade, which translates commands, subcommands, flags, and arguments into a Helm operation |
There was a problem hiding this comment.
How about making it a kubectl plugin?
There was a problem hiding this comment.
We could do that. IIRC, it's fairly trivial to do both stand-alone and plugin. That's what the service broker folks did.
There was a problem hiding this comment.
At the moment kubectl plugins are alpha (API to them can break at any time), the API is in discussions to change (developers of other tools are asking for changes so they can plug them in), and they aren't ready for production.
Helm is used for production workloads.
I wouldn't be surprised if there's a helm plugin to kubectl some day. We are tracking the kubectl plugin work.
|
|
||
| By design, the Helm library _must_ be usable as a standalone library. | ||
|
|
||
| In Appendix A, we specify a second implementation of Helm 3 that provides a controller-oriented model with no client. The Helm Controller provides a CRD-based façade to receive commands, but executes them via the Helm library. |
There was a problem hiding this comment.
This sounds much better to me... Which will be implemented first? or will it be released in parallel?
There was a problem hiding this comment.
No commitments yet on timelines. Basically, it will come down to who is available for doing what work. That approach has some additional complexities that must be worked out, so at the moment it looks like CLI first, controller second.
| * An extension mechanism (Lua scripts stored in the chart) | ||
| * A few in-cluster CRDs: Release, ReleaseVersion (Neither of which has an accompanying controller) | ||
|
|
||
| In this model, Tiller is removed and there are no operators or controllers that run in-cluster. |
There was a problem hiding this comment.
The "simplification" doesn't really simplify things. It just shifts the complexity to the end user.
For example, if you have more then 1 person on your team, it would be easy to get into issues where multiple people are working on a release at the same time. Each helm instance may have differently configured repo sets.
There may not be locking between the two helm instances, or if there is, there will be the possibility of a network disconnect causing helm to prematurely loose access to running through the workflow.
There was a problem hiding this comment.
The repo issue exists today (Helm 2), and gets no better or worse in the proposed model. The network issue is not likely, since the manifests are shipped to kube API in one large block. No part of Kubernetes has ever supported the kinds of atomic transactions that would be needed to entirely prevent resource races. It's a problem we're not interested in solving. Some middle-ware deployment system would need to be created, and if it's done using CRDs, Helm would easily support it. That, in fact, is the plan: Reduce Helm's deployment logic and let new deployment middleware develop that Helm can use. I'll return to this point at the end.
|
|
||
| This implementation will necessarily have some limitations and restrictions that the client Helm does not; namely, it will not be able to handle large charts because there is a 1M limit to CRD size. | ||
|
|
||
| It is also assumed that the controller model may not choose to implement a number of the client features, such as dry runs, repository management, chart creation, etc. |
There was a problem hiding this comment.
dry run doesn't quite make sense by itself, but if all the info can be handed off from the operator to a local helm, then you could still do the dry run?
| Consequently, if Helm is to fit into such a paradigm, it is natural that Helm will provide at least most, if not all of these. | ||
|
|
||
| Helm is not a configuration management tool (such as Chef or Puppet). Helm does provide certain features of such systems, though. Helm's templating and configuration model is a common feature of configuration managers (and also of package managers). | ||
|
|
There was a problem hiding this comment.
Would like to see an explicit feature around 'upgrades' to a namespace. all of the package managers listed above have a "X upgrade" command that looks at all installed things, looks at the repos in question and upgrades the things that need upgrading. This significantly lowers the bar for managing software updates. Helm currently lacks this feature but would be a very significant advancement IMO.
| There were two main choices to be made about how to extend Helm with a better lifecycle event model: | ||
|
|
||
| 1. Do we load lifecycle hooks as extensions, as part of a chart, or both? | ||
| 2. What extension mechanism do we support? |
There was a problem hiding this comment.
Option 3, plugin running in cluster.
k8s can handle multiple arch/os options today. It also will behave identically on each users machine while lua may behave quite differently if admin 1 and admin 2 run it based on various machine differences like mac/windows/linux, what their home directory structure looks like, etc.
|
|
||
| In answer to the second question, we considered four alternatives before deciding on embedding a scripting engine. The following alternatives were considered as an alternative to embedding a scripting language. The pros/cons of embedding a scripting language leaned heavily in its favor. | ||
|
|
||
| ### Option A: Implementing Lifecycle Hooks as Docker Images |
There was a problem hiding this comment.
Option A.2, Implement Lifecycle hooks as Helm Charts....
| # Helm 3 Design Proposal | ||
|
|
||
| This document contains a proposal for Helm 3. It assumes deep familiarity with Helm 2, and assumes that much of Helm 2's architecture will simply carry over without major change. | ||
|
|
There was a problem hiding this comment.
General comment I'm going to put right at the top. after reading this through. The proposal is quite the opposite of the suggestion one of the k8s devs made at the helm summit. Make things more modular and follow the unix/k8s philosophy of having one tool do one thing and do it well.
One of the things I like about helm2 today is its fairly focused. it does basically two things. packaging multiple objects together, and templating those objects.
His idea was to break it up into at least 2 pieces. one that did the packaging of multiple objects, and another that did the templating. This would allow alternate template engines to be easily added for example.
With CRD's, this is totally possible.
upload a package release CRD. it takes it, pulls tarball, and instantiates each k8s object inside. those k8s objects, could be CRD's for a templater, that takes those and values, and templates them out to the real objects.
This would even let you instantiate a templated helm release crd that then instantiated multiple releases. lots of things that are difficult today like templating out values would naturally fall out of that natural breaking up of functionality.
Should we explore that idea further? The discussion at the helm summit seemed to be positive amongst the discussion members.
There was a problem hiding this comment.
Multiple template engines will be supported via Lua.
The main idea, which is very understated in the proposal (by design) is that we're trying to focus on the chart -> manifest process, and back off of the release management part. There are several tools in development that will take a CRD that defines an application and then install and manage that application (the Application WG is working on these). Our intention is to be able to describe charts that can be used to produce those CRDs, and then let the backing systems manage the ins and outs of actual application lifecycle management.
At this point, we still need to manage a release as a thing. So we can't completely strip out the install/upgrade/rollback/delete concepts. But now we're heading back toward the simplistic notions of those that are part of kubectl. So we're de-emphasizing lifecycle management.
Instead, we're now focusing on the workflow that is "load chart, accumulate values, render chart into manifest, (optionally) send manifest". For that reason, the vast majority of the changes in this proposal are all centered on how we make that a flexible, pluggable system.
Finally, keep in mind that this is just one proposal. It's not set in stone, though currently it's the only one that has gotten any agreement.
- Resolve open issues from implementation experience - Document content cache at $HELM_CACHE_HOME/content/ - Add archive-based plugin loading details - Add airgap support considerations (Open Issue helm#5) - Clean up rejected ideas section Signed-off-by: Scott Rigby <scott@r6by.com>
Also refactor by adding a top-level helm-v3 directory