This page explains how Kubernetes objects are represented in the Kubernetes API, and how you can express them in .yaml format.
Understanding Kubernetes Objects
Kubernetes Objects are persistent entities in the Kubernetes system.
Kubernetes uses these entities to represent the state of your cluster.
Specifically, they can describe:
- What containerized applications are running (and on which nodes)
- The resources available to those applications
- The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance
A Kubernetes object is a “record of intent”–once you create the object, the Kubernetes system will constantly work to ensure that object exists.
By creating an object, you’re effectively telling the Kubernetes system what you want your cluster’s workload to look like; this is your cluster’s desired state.
To work with Kubernetes objects–whether to create, modify, or delete them–you’ll need to use the Kubernetes API.
When you use the kubectl command-line interface, for example, the CLI makes the necessary Kubernetes API calls for you; you can also use the Kubernetes API directly in your own programs.
Kubernetes currently provides a golang client library for this purpose, and other language libraries (such as Python) are being developed.
Object Spec and Status
Every Kubernetes object includes two nested object fields that govern the object’s configuration: the object spec and the object status.
The spec, which you must provide, describes your desired state for the object–the characteristics that you want the object to have.
The status describes the actual state of the object, and is supplied and updated by the Kubernetes system.
At any given time, the Kubernetes Control Plane actively manages an object’s actual state to match the desired state you supplied.
Describing a Kubernetes Object
When you create an object in Kubernetes, you must provide the object spec that describes its desired state, as well as some basic information about the object (such as a name).
When you use the Kubernetes API to create the object (either directly or via kubectl), that API request must include that information as JSON in the request body.
Most often, you provide the information to kubectl in a .yaml file. kubectl converts the information to JSON when making the API request.
Here’s an example .yaml file that shows the required fields and object spec for a Kubernetes Deployment:
apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
One way to create a Deployment using a .yaml file like the one above is to use the kubectl create command in the kubectl command-line interface, passing the .yaml file as an argument. Here’s an example:
$ kubectl create -f https://k8s.io/docs/user-guide/nginx-deployment.yaml --record
The output is similar to this:
deployment "nginx-deployment" created
Required Fields
In the .yaml file for the Kubernetes object you want to create, you’ll need to set values for the following fields:
-
apiVersion- Which version of the Kubernetes API you’re using to create this object -
kind- What kind of object you want to create -
metadata- Data that helps uniquely identify the object, including anamestring, UID, and optionalnamespace
You’ll also need to provide the object spec field.
The precise format of the object spec is different for every Kubernetes object, and contains nested fields specific to that object.
The Kubernetes API reference can help you find the spec format for all of the objects you can create using Kubernetes.
Names
All objects in the Kubernetes REST API are unambiguously identified by a Name and a UID.
For non-unique user-provided attributes, Kubernetes provides labels and annotations.
Names are generally client-provided.
Only one object of a given kind can have a given name at a time (i.e., they are spatially unique).
But if you delete an object, you can make a new object with the same name.
Names are used to refer to an object in a resource URL, such as /api/v1/pods/some-name.
By convention, the names of Kubernetes resources should be up to maximum length of 253 characters and consist of lower case alphanumeric characters, -, and ., but certain resources have more specific restrictions. See the identifiers design doc for the precise syntax rules for names.
UIDs
UIDs are generated by Kubernetes.
Every object created over the whole lifetime of a Kubernetes cluster has a distinct UID (i.e., they are spatially and temporally unique).
Namespaces
Kubernetes supports multiple virtual clusters backed by the same physical cluster.
These virtual clusters are called namespaces.
When to Use Multiple Namespaces
Namespaces are intended for use in environments with many users spread across multiple teams, or projects.
For clusters with a few to tens of users, you should not need to create or think about namespaces at all.
Start using namespaces when you need the features they provide.
Namespaces provide a scope for names. Names of resources need to be unique within a namespace, but not across namespaces.
Namespaces are a way to divide cluster resources between multiple users (via resource quota).
In future versions of Kubernetes, objects in the same namespace will have the same access control policies by default.
It is not necessary to use multiple namespaces just to separate slightly different resources, such as different versions of the same software: use labels to distinguish resources within the same namespace.
Working with Namespaces
Creation and deletion of namespaces is described in the Admin Guide documentation for namespaces.
Viewing namespaces
$ kubectl get namespaces
NAME STATUS AGE
default Active 1d
kube-system Active 1d
kube-public Active 1d
Kubernetes starts with three initial namespaces:
-
defaultThe default namespace for objects with no other namespace -
kube-systemThe namespace for objects created by the Kubernetes system -
kube-publicThe namespace is created automatically and readable by all users (including those not authenticated).- This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster.
- The public aspect of this namespace is only a convention, not a requirement.
Setting the namespace for a request
To temporarily set the namespace for a request, use the --namespace flag.
$ kubectl --namespace=<insert-namespace-name-here> run nginx --image=nginx
$ kubectl --namespace=<insert-namespace-name-here> get pods
Setting the namespace preference
You can permanently save the namespace for all subsequent kubectl commands in that context
$ kubectl config set-context $(kubectl config current-context) --namespace=<insert-namespace-name-here>
# Validate it
$ kubectl config view | grep namespace:
Namespaces and DNS
When you create a Service, it creates a corresponding DNS entry.
This entry is of the form <service-name>.<namespace-name>.svc.cluster.local, which means that if a container just uses <service-name>, it will resolve to the service which is local to a namespace.
This is useful for using the same configuration across multiple namespaces such as Development, Staging and Production.
If you want to reach across namespaces, you need to use the fully qualified domain name (FQDN).
Not All Objects are in a Namespace
Most Kubernetes resources (e.g. pods, services, replication controllers, and others) are in some namespaces.
However namespace resources are not themselves in a namespace. And low-level resources, such as nodes and persistentVolumes, are not in any namespace.
Events are an exception: they may or may not have a namespace, depending on the object the event is about.
Labels and Selectors
Labels are key/value pairs that are attached to objects, such as pods.
Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users,
but do not directly imply semantics to the core system.
Labels can be used to organize and to select subsets of objects.
Labels can be attached to objects at creation time and subsequently added and modified at any time.
Each object can have a set of key/value labels defined.
Each Key must be unique for a given object.
"labels": {
"key1" : "value1",
"key2" : "value2"
}
We’ll eventually index and reverse-index labels for efficient queries and watches, use them to sort and group in UIs and CLIs, etc.
We don’t want to pollute labels with non-identifying, especially large and/or structured, data.
Non-identifying information should be recorded using annotations.
Motivation
Labels enable users to map their own organizational structures onto system objects in a loosely coupled fashion, without requiring clients to store these mappings.
Service deployments and batch processing pipelines are often multi-dimensional entities (e.g., multiple partitions or deployments, multiple release tracks, multiple tiers, multiple micro-services per tier).
Management often requires cross-cutting operations, which breaks encapsulation of strictly hierarchical representations, especially rigid hierarchies determined by the infrastructure rather than by users.
Example labels:
-
"release" : "stable","release" : "canary" -
"environment" : "dev","environment" : "qa","environment" : "production" -
"tier" : "frontend","tier" : "backend","tier" : "cache" -
"partition" : "customerA","partition" : "customerB" -
"track" : "daily","track" : "weekly"
These are just examples of commonly used labels; you are free to develop your own conventions. Keep in mind that label Key must be unique for a given object.
Syntax and character set
Labels are key/value pairs.
Valid label keys have two segments: an optional prefix and name, separated by a slash (/).
The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.
The prefix is optional.
If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (.), not longer than 253 characters in total, followed by a slash (/).
If the prefix is omitted, the label Key is presumed to be private to the user.
Automated system components (e.g. kube-scheduler, kube-controller-manager, kube-apiserver, kubectl, or other third-party automation) which add labels to end-user objects must specify a prefix.
The kubernetes.io/ prefix is reserved for Kubernetes core components.
Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.
Label selectors
Unlike names and UIDs, labels do not provide uniqueness. In general, we expect many objects to carry the same label(s).
Via a label selector, the client/user can identify a set of objects.
The label selector is the core grouping primitive in Kubernetes.
The API currently supports two types of selectors: equality-based and set-based.
A label selector can be made of multiple requirements which are comma-separated.
In the case of multiple requirements, all must be satisfied so the comma separator acts as a logical AND (&&) operator.
An empty label selector (that is, one with zero requirements) selects every object in the collection.
A null label selector (which is only possible for optional selector fields) selects no objects.
Note: the label selectors of two controllers must not overlap within a namespace, otherwise they will fight with each other.
Equality-based requirement
Equality- or inequality-based requirements allow filtering by label keys and values.
Matching objects must satisfy all of the specified label constraints, though they may have additional labels as well.
Three kinds of operators are admitted =,==,!=.
The first two represent equality (and are simply synonyms), while the latter represents inequality.
For example:
environment = production
tier != frontend
The former selects all resources with key equal to environment and value equal to production.
The latter selects all resources with key equal to tier and value distinct from frontend, and all resources with no labels with the tier key.
One could filter for resources in production excluding frontend using the comma operator: environment=production,tier!=frontend
Set-based requirement
Set-based label requirements allow filtering keys according to a set of values.
Three kinds of operators are supported: in,notin and exists (only the key identifier).
For example:
environment in (production, qa)
tier notin (frontend, backend)
partition
!partition
The first example selects all resources with key equal to environment and value equal to production or qa.
The second example selects all resources with key equal to tier and values other than frontend and backend, and all resources with no labels with the tier key.
The third example selects all resources including a label with key partition; no values are checked.
The fourth example selects all resources without a label with key partition; no values are checked.
Similarly the comma separator acts as an AND operator.
So filtering resources with a partition key (no matter the value) and with environment different than qa can be achieved using partition,environment notin (qa).
The set-based label selector is a general form of equality since environment=production is equivalent to environment in (production);
similarly for != and notin.
Set-based requirements can be mixed with equality-based requirements. For example:
partition in (customerA, customerB),environment!=qa
API
LIST and WATCH filtering
LIST and WATCH operations may specify label selectors to filter the sets of objects returned using a query parameter.
Both requirements are permitted (presented here as they would appear in a URL query string):
- equality-based requirements:
?labelSelector=environment%3Dproduction,tier%3Dfrontend
- set-based requirements:
?labelSelector=environment+in+%28production%2Cqa%29%2Ctier+in+%28frontend%29
Both label selector styles can be used to list or watch resources via a REST client.
For example, targeting apiserver with kubectl and using equality-based one may write:
$ kubectl get pods -l environment=production,tier=frontend
or using set-based requirements:
$ kubectl get pods -l 'environment in (production),tier in (frontend)'
As already mentioned set-based requirements are more expressive.
For instance, they can implement the OR operator on values:
$ kubectl get pods -l 'environment in (production, qa)'
or restricting negative matching via exists operator:
$ kubectl get pods -l 'environment,environment notin (frontend)'
Set references in API objects
Some Kubernetes objects, such as services and replicationcontrollers, also use label selectors to specify sets of other resources, such as pods.
Service and ReplicationController
The set of pods that a service targets is defined with a label selector.
Similarly, the population of pods that a replicationcontroller should manage is also defined with a label selector.
Labels selectors for both objects are defined in json or yaml files using maps, and only equality-based requirement selectors are supported:
"selector": {
"component" : "redis",
}
#or
selector:
component: redis
this selector (respectively in json or yaml format) is equivalent to component=redis or component in (redis).
Resources that support set-based requirements
Newer resources, such as Job, Deployment, Replica Set, and Daemon Set, support set-based requirements as well.
selector:
matchLabels:
component: redis
matchExpressions:
- {key: tier, operator: In, values: [cache]}
- {key: environment, operator: NotIn, values: [dev]}
matchLabels is a map of {key,value} pairs.
A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is “key”, the operator is “In”, and the values array contains only “value”.
matchExpressions is a list of pod selector requirements.
Valid operators include In, NotIn, Exists, and DoesNotExist. The values set must be non-empty in the case of In and NotIn.
All of the requirements, from both matchLabels and matchExpressions are ANDed together – they must all be satisfied in order to match.
Selecting sets of nodes
One use case for selecting over labels is to constrain the set of nodes onto which a pod can schedule.
See the documentation on node selection for more information.
Annotations
You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata.
Attaching metadata to objects
You can use either labels or annotations to attach metadata to Kubernetes objects.
Labels can be used to select objects and to find collections of objects that satisfy certain conditions.
In contrast, annotations are not used to identify and select objects.
The metadata in an annotation can be small or large, structured or unstructured, and can include characters not permitted by labels.
Annotations, like labels, are key/value maps:
"annotations": {
"key1" : "value1",
"key2" : "value2"
}
Here are some examples of information that could be recorded in annotations:
-
Fields managed by a declarative configuration layer.
- Attaching these fields as annotations distinguishes them from default values set by clients or servers, and from auto-generated fields and fields set by auto-sizing or auto-scaling systems.
Build, release, or image information like timestamps, release IDs, git branch, PR numbers, image hashes, and registry address.
Pointers to logging, monitoring, analytics, or audit repositories.
Client library or tool information that can be used for debugging purposes: for example, name, version, and build information.
User or tool/system provenance information, such as URLs of related objects from other ecosystem components.
Lightweight rollout tool metadata: for example, config or checkpoints.
Phone or pager numbers of persons responsible, or directory entries that specify where that information can be found, such as a team web site.
Instead of using annotations, you could store this type of information in an external database or directory, but that would make it much harder to produce shared client libraries and tools for deployment, management, introspection, and the like.