Cluster Membership, Bootstrap, and Safety¶

Movie cluster membership is a production-alpha extension above the actor and remoting runtime. It supplies static or dynamic seed discovery, stable bootstrap, process-incarnation identity, convergent membership, deterministic leadership, reachability, graceful leave, manual downing, rolling protocol gates, typed events, and telemetry. An explicit split-brain extension resolves network partitions with dependency-free keep-majority or an external lease.

Cluster traffic remains at-most-once. Use persistence and transactional outbox contracts when application delivery must survive process loss. Cluster sharding, singletons, the receptionist, and routers build on this membership contract.

Static seeds¶

Every node enables remoting before membership. Seed addresses include the actor-system name because association negotiation verifies it:

seed_remote = seed_system.enable_remoting("127.0.0.1", 2551)
seed = seed_system.enable_cluster(Movie::Cluster::ClusterSettings.new(
  cluster_name: "orders",
  roles: ["seed", "backend"]
))

worker_system.enable_remoting("127.0.0.1", 2552)
worker = worker_system.enable_cluster(Movie::Cluster::ClusterSettings.new(
  cluster_name: "orders",
  seed_nodes: [seed_remote.address],
  roles: ["worker"]
))
worker.await_up(10.seconds)

An empty static seed list forms a one-node cluster immediately. A configured seed whose list contains itself starts as Up and retries the other seeds. A non-seed starts as Joining until the leader admits its exact address and node UID. Join, welcome, and gossip processing is idempotent.

Dynamic discovery and bootstrap¶

Dynamic discovery is a separate SPI. DnsSeedDiscovery resolves A/AAAA records. KubernetesSeedDiscovery lists all discovery.k8s.io/v1 EndpointSlices with the selected Service label, deduplicates them, and keeps ready, non-terminating endpoints for the named TCP port. A custom provider implements SeedDiscovery#lookup.

All dynamically discovered nodes use the same actor-system name and remoting port. Static seeds and dynamic discovery are mutually exclusive:

discovery = Movie::Cluster::DnsSeedDiscovery.new(
  "orders-system",
  "orders-headless.default.svc.cluster.local",
  2552
)

system.enable_remoting("0.0.0.0", 2552)
cluster = system.enable_cluster(Movie::Cluster::ClusterSettings.new(
  cluster_name: "orders",
  discovery: discovery,
  bootstrap: Movie::Cluster::BootstrapSettings.new(
    required_contact_points: 3,
    stable_after: 5.seconds,
    join_timeout: 3.seconds
  )
))

Bootstrap sorts an unchanged contact set, tries an existing contact first, and allows only the lowest address to form after the join window. Configure required_contact_points as the minimum expected initial deployment size. Set new_cluster_enabled: false on nodes that may join but must never form a new cluster.

Automatic Kubernetes configuration expects a headless Service with a named TCP port and a service account allowed to get and list EndpointSlices in the namespace:

name: orders-system
remoting:
  enabled: true
  host: 0.0.0.0
  port: 2552
cluster:
  enabled: true
  name: orders
  discovery:
    method: kubernetes
    kubernetes:
      service: orders-headless
      namespace: default
      port-name: remoting
  bootstrap:
    required-contact-points: 3

Identity and membership¶

UniqueAddress combines the advertised remoting Address with the random process node UID authenticated by its association. Reusing a host and port after restart creates a different incarnation. Membership records move monotonically through:

Joining -> Up -> Leaving -> Exiting -> Removed
                  \-> Down -> Removed

Bounded tombstones prevent delayed gossip from resurrecting an old UID. cluster.max-members covers live members plus tombstones and fails closed before partial overflow. The lowest reachable Up or Leaving UniqueAddress is the local leader. ACKs carry a deterministic digest; cluster.converged? is true when all reachable Up members have acknowledged the current state.

Reachability and split-brain resolution¶

Cluster heartbeats use /system/cluster. A monotonic timeout adds an exact UID to the local snapshot.unreachable set; reachability alone never changes membership. Without SBR, an operator must resolve the ambiguity and invoke down(unique_address) on the current leader.

The default automatic policy needs no persistence or external coordinator. Enable it explicitly on every node after cluster membership:

split_brain = system.enable_split_brain

After the unreachable set remains unchanged for stable_after, keep-majority counts active (Up, Leaving, or Exiting) members. The larger partition survives. An even split is won by the side containing the lowest eligible UniqueAddress. Only the surviving partition's reachable cluster leader issues down commands, but every unreachable member is downed. If cluster.split-brain.role is set, only members with that role vote; non-voting unreachable members are still downed. The losing partition shuts its actor systems down. A continuously changing partition view also shuts down after stable-after + down-all-when-unstable rather than running ambiguously forever.

Keep-majority resolves membership without PostgreSQL, but it is not a linearizable storage fence. Persistent sharding and persistent singleton writes still use their independent shared-backend fencing epoch. Ordinary in-memory actors on a process paused longer than failure detection can execute until that process resumes and observes its terminal membership, so commands that require durable exactly-once effects still need idempotency or storage fencing.

For a stronger infrastructure policy, supply one strongly consistent external lease to every node. The PostgreSQL adapter reuses Movie persistence's pool and fenced lease table:

require "movie/persistence/postgres"

database = Movie::Database.get(system) # backend must be postgres
lease = Movie::Cluster::PostgresSplitBrainLease.new(
  database,
  cluster.settings.cluster_name,
  "primary"
)
split_brain = system.enable_split_brain(lease)

With lease-majority, after the unreachable set stays unchanged for stable_after, only the lowest reachable eligible member attempts the lease. Nodes that can observe that holder as reachable survive. A side that sees the holder as unreachable shuts its actor system down. Lease loss or renewal failure also fails closed. The holder downs unreachable incarnations; a paused loser shuts down when it resumes and observes its terminal membership record.

Use the same cluster and lease names everywhere, an HA PostgreSQL deployment independent of either application partition, and remoting.shared-secret. SQLite and process-local locks are not production leases. The process supervisor or Kubernetes controller must restart terminated nodes. Resolution time includes failure detection, stabilization, minority delay, and lease I/O, so probes must allow that envelope.

split_brain.stats reports state, partitions, acquisitions, renewals, failures, downed members, self-shutdowns, and releases. Persistent sharding and singleton writes retain their independent PostgreSQL fencing.

Rolling updates¶

Each member advertises minimum_protocol_version, protocol_version, and app_version. Compatibility is mutual: each current cluster protocol must fall inside the other's accepted range. JoinRejected gives await_up an immediate reason instead of a generic timeout. Membership admits at most two active application versions.

Keep serializers and actor message contracts readable in both directions during a rollout. Raise the minimum protocol only after all nodes understand the new version. This gate is independent of remoting's exact association protocol; changing that transport protocol still needs a capability-compatible rollout or a full restart.

Graceful leave¶

Use a two-step shutdown so peers receive the tombstone before transport stops:

cluster.leave
cluster.await_removed(10.seconds)
system.shutdown

Direct actor-system shutdown is abrupt. A non-leader retries its leave request, and peers retry the final Removed state for graceful leavers until its digest ACK arrives. Distributed extensions may install a fast non-blocking departure barrier with register_departure_guard.

Events and telemetry¶

cluster.snapshot returns defensive member and unreachable arrays. Subscribe a local actor for current state followed by effective membership, reachability, and leader transitions:

cluster.subscribe(events)
# CurrentState, MemberJoined, MemberUp, MemberLeaving, MemberExiting,
# MemberDown, MemberRemoved, UnreachableMember, ReachableMember, LeaderChanged
cluster.unsubscribe(events)

cluster.stats includes join/gossip/heartbeat, membership, discovery/bootstrap, compatibility rejection, capacity rejection, and subscriber counters.

Verification¶

crystal spec spec/movie/cluster -Dpreview_mt -Dexecution_context
MOVIE_CLUSTER_STRESS=1 crystal spec spec/movie/cluster/stress_spec.cr \
  -Dpreview_mt -Dexecution_context
MOVIE_CLUSTER_SAFETY_STRESS=1 MOVIE_POSTGRES_TEST_URL="$DATABASE_URL" \
  crystal spec spec/movie/cluster/safety_stress_spec.cr \
  -Dpreview_mt -Dexecution_context

The safety scenario runs three OS processes, pauses one, verifies PostgreSQL lease acquisition and majority survival, resumes the loser, and verifies its fail-closed shutdown. MOVIE_CLUSTER_SAFETY_STRESS_ROUNDS repeats the scenario for soak runs.

Movie does not yet provide multi-data-center topology, distributed pub/sub, CRDT data, or automatic remoting-protocol migration. Those are separate features, not implied by SBR or bootstrap.