<p>I have a cluster with 3 control-planes. As any cluster my cluster also has a default <code>kubernetes</code> service. As any service it has a list of endpoints:</p>

<pre><code>apiVersion: v1
items:
- apiVersion: v1
  kind: Endpoints
  metadata:
    creationTimestamp: 2017-12-12T17:08:34Z
    name: kubernetes
    namespace: default
    resourceVersion: "6242123"
    selfLink: /api/v1/namespaces/default/endpoints/kubernetes
    uid: 161edaa7-df5f-11e7-a311-d09466092927
  subsets:
  - addresses:
    - ip: 10.9.22.25
    - ip: 10.9.22.26
    - ip: 10.9.22.27
    ports:
    - name: https
      port: 8443
      protocol: TCP
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
</code></pre>

<p>Everything is ok, but I completely can't understand where do these endpoints come from? It is logical to assume from the <code>Service</code> label selector, but there's no any label selectors:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2017-12-12T17:08:34Z
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
  resourceVersion: "6"
  selfLink: /api/v1/namespaces/default/services/kubernetes
  uid: 161e4f00-df5f-11e7-a311-d09466092927
spec:
  clusterIP: 10.100.0.1
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8443
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  type: ClusterIP
status:
  loadBalancer: {}
</code></pre>

<p>So, could anybody explain how k8s services and endpoints work in case of built-in default <code>kubernetes</code> service? </p>

<p>Its not clear how you created multi-node cluster, but here are some research for you:</p>

<p><a href="https://kubernetes.io/docs/tasks/administer-cluster/highly-available-master/" rel="nofollow noreferrer">Set up High-Availability Kubernetes Masters</a> describe HA k8s creation. They have <a href="https://kubernetes.io/docs/tasks/administer-cluster/highly-available-master/" rel="nofollow noreferrer">notes</a> about default kubernetes service.</p>

<blockquote>
  <p>Instead of trying to keep an up-to-date list of Kubernetes apiserver
  in the Kubernetes service, the system directs all traffic to the
  external IP:</p>
  
  <p>in one master cluster the IP points to the single master,</p>
  
  <p>in multi-master cluster the IP points to the load balancer in-front of
  the masters.</p>
  
  <p>Similarly, the external IP will be used by kubelets to communicate
  with master</p>
</blockquote>

<p>So I would rather expect a LB ip instead of 3 masters.</p>

<p>Service creation: <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/master/controller.go#L46-L83" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/blob/master/pkg/master/controller.go#L46-L83</a></p>

<pre><code>const kubernetesServiceName = "kubernetes"

// Controller is the controller manager for the core bootstrap Kubernetes
// controller loops, which manage creating the "kubernetes" service, the
// "default", "kube-system" and "kube-public" namespaces, and provide the IP
// repair check on service IPs
type Controller struct {
    ServiceClient   corev1client.ServicesGetter
    NamespaceClient corev1client.NamespacesGetter
    EventClient     corev1client.EventsGetter
    healthClient    rest.Interface

    ServiceClusterIPRegistry rangeallocation.RangeRegistry
    ServiceClusterIPInterval time.Duration
    ServiceClusterIPRange    net.IPNet

    ServiceNodePortRegistry rangeallocation.RangeRegistry
    ServiceNodePortInterval time.Duration
    ServiceNodePortRange    utilnet.PortRange

    EndpointReconciler reconcilers.EndpointReconciler
    EndpointInterval   time.Duration

    SystemNamespaces         []string
    SystemNamespacesInterval time.Duration

    PublicIP net.IP

    // ServiceIP indicates where the kubernetes service will live.  It may not be nil.
    ServiceIP                 net.IP
    ServicePort               int
    ExtraServicePorts         []corev1.ServicePort
    ExtraEndpointPorts        []corev1.EndpointPort
    PublicServicePort         int
    KubernetesServiceNodePort int

    runner *async.Runner
}
</code></pre>

<p>Service periodically updates: <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/master/controller.go#L204-L242" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/blob/master/pkg/master/controller.go#L204-L242</a></p>

<pre><code>// RunKubernetesService periodically updates the kubernetes service
func (c *Controller) RunKubernetesService(ch chan struct{}) {
    // wait until process is ready
    wait.PollImmediateUntil(100*time.Millisecond, func() (bool, error) {
        var code int
        c.healthClient.Get().AbsPath("/healthz").Do().StatusCode(&amp;code)
        return code == http.StatusOK, nil
    }, ch)

    wait.NonSlidingUntil(func() {
        // Service definition is not reconciled after first
        // run, ports and type will be corrected only during
        // start.
        if err := c.UpdateKubernetesService(false); err != nil {
            runtime.HandleError(fmt.Errorf("unable to sync kubernetes service: %v", err))
        }
    }, c.EndpointInterval, ch)
}

// UpdateKubernetesService attempts to update the default Kube service.
func (c *Controller) UpdateKubernetesService(reconcile bool) error {
    // Update service &amp; endpoint records.
    // TODO: when it becomes possible to change this stuff,
    // stop polling and start watching.
    // TODO: add endpoints of all replicas, not just the elected master.
    if err := createNamespaceIfNeeded(c.NamespaceClient, metav1.NamespaceDefault); err != nil {
        return err
    }

    servicePorts, serviceType := createPortAndServiceSpec(c.ServicePort, c.PublicServicePort, c.KubernetesServiceNodePort, "https", c.ExtraServicePorts)
    if err := c.CreateOrUpdateMasterServiceIfNeeded(kubernetesServiceName, c.ServiceIP, servicePorts, serviceType, reconcile); err != nil {
        return err
    }
    endpointPorts := createEndpointPortSpec(c.PublicServicePort, "https", c.ExtraEndpointPorts)
    if err := c.EndpointReconciler.ReconcileEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts, reconcile); err != nil {
        return err
    }
    return nil
}
</code></pre>

<p>Endpoint update place: <a href="https://github.com/kubernetes/kubernetes/blob/72f69546142a84590550e37d70260639f8fa3e88/pkg/master/reconcilers/lease.go#L163" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/blob/72f69546142a84590550e37d70260639f8fa3e88/pkg/master/reconcilers/lease.go#L163</a></p>

<p>Also endpoint could be created manually. Visit <a href="https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors" rel="nofollow noreferrer">Services without selectors</a> for more info.</p>
