End a run¶

Three things end a run, and the outcome names which of the three it was.

	Idle limit	Wall clock	`abort`
Outcome	`Settled`	`WallClockExpired`	`Aborted`
Deadline lives in	The store, pushed out on every event	The store, fixed when the run is created	Nothing, it is a call
Who notices	Any replica on any access, or the reaper	Same	The replica handling the call
What is checked	`now > idle_deadline`	`now > wall_deadline`	Nothing to check
Predicate asked	Yes, and `CONTINUE` pushes the deadline out again	No, it closes regardless	No
Names unfinished agents	Yes	Yes	No

Whoever wins the compare and set on the outcome closes it, so a run closes once however many callers reach a deadline together.

The first three sections take those in turn. The rest is who does the closing, how the outcome is read, and what a closed run still answers.

Silence¶

A run ends on silence when nothing has happened for the idle limit.

RunLimits(wall_clock=timedelta(hours=1), idle=timedelta(minutes=10))

A write to a level or a premise pushes the deadline out, and so do an acknowledgment and a notification reaching an agent. Reads do not, so an agent polling the board cannot hold a run open.

Registering pushes the deadline out through the notification it hands the new agent, so a run does not close on an agent that has just joined. Registering an agent that subscribes to nothing the board holds pushes nothing, because that agent was given nothing to do.

Choose the idle limit by how long an agent's slowest step takes. If it is shorter than that, a run closes while an agent is still thinking; if it is much longer, a finished run sits open.

Requiring something of the result¶

A termination predicate is asked when the idle limit passes, and answering CONTINUE re-arms the idle limit.

from blackboard import TerminationDecision


def both_levels_have_something(reader):
    if reader.read_level("platform") and reader.read_level("application"):
        return TerminationDecision.COMPLETE
    return TerminationDecision.CONTINUE


model = create_model(..., termination_predicate=both_levels_have_something)

Supplying no predicate lets silence close the run on its own.

The predicate runs without the control component's lock, so its decision is discarded if the board moved while it ran, and the predicate is asked again at the next deadline.

The wall clock¶

The wall clock closes the run whatever the predicate says, and it is what ends a run that never falls silent.

Agents that answer each other faster than the idle limit never let a run fall silent. Each write wakes the other agent, whose write pushes the deadline out again, and the deadline is never reached. Two agents answering each other every twenty seconds, under an idle limit of thirty, run until the wall clock passes and are both named unfinished.

RunLimits(wall_clock=timedelta(seconds=100), idle=timedelta(seconds=30))
# WallClockExpired(unfinished=frozenset({'a', 'b'}))

A termination predicate does not help here. It is asked when the idle deadline passes, and in a run that never falls silent that deadline never passes, so the predicate is never asked at all.

Choose the wall clock by how long the work is allowed to take. An idle limit shorter than the gap between events would end such a run, and would end every slower run with it, which is the failure the idle limit is sized against.

Closing by hand¶

A caller may close a run outright.

model.control.abort("the operator stopped the investigation")

Closing a run no process is watching¶

A run closes because nothing happened, so no request is in flight to notice. The process that opened the run notices through its own timer. Every other process notices by asking the store.

from blackboard import close_expired

closed = close_expired(store)  # names the boards it closed

Call it on whatever schedule suits the deployment, being a thread beside the service, a scheduled job, or a test advancing time by hand. This library takes no view on where it runs.

Several callers running it together is safe. Recording an outcome is a write only one caller wins, so a run closes once, one outcome is recorded and one on_closed fires, however many callers reach the deadline in the same instant. That is also why no lock and no elected leader appears anywhere.

If every caller runs it on the same interval they will tend to arrive together, which wastes queries rather than breaking anything. Draw each wait from a range, the way the delivery backoff already does.

Sweep is that loop written out, for an application that would otherwise write the same thread:

from blackboard import Sweep

with Sweep(store, control_for=runs.get) as sweep:  # every thirty seconds
    serve_until_shutdown()

It spreads its first pass over the interval, so replicas that started together do not query together afterwards. A pass that raises is logged and the loop continues, because a loop that died on one failure would leave every later run open. Closing it waits for the pass in flight.

Two things need a schedule, and a pass runs both. close_expired closes what nobody is watching, and relay_unsent sends what a write recorded and nothing has sent. Closing takes only the store; sending takes the callables as well, so control_for answers with the Control for a board, and is the same callable BoardService takes. A sweep given none reaps and nothing more.

Reaping runs first. Closing a run clears what it was owed, so a notification to a run about to end is not sent and then answered into a run that has closed.

It is a convenience over those two functions and nothing more, the way HttpNotifier is a convenience over the wire protocol. An application with a scheduler of its own calls them and ignores this.

Reading the outcome¶

outcome = model.control.wait_closed(timeout=timedelta(seconds=30))

match outcome:
    case Settled(unfinished=agents) if not agents:
        ...  # everyone finished
    case Settled(unfinished=agents):
        ...  # settled, but these never came back
    case WallClockExpired(unfinished=agents):
        ...  # ran out of time
    case Aborted(reason=reason):
        ...  # someone stopped it
    case None:
        ...  # still open when the timeout passed

The empty case takes a guard because frozenset() written as a pattern matches every frozenset, including a non-empty one, and would swallow the case under it.

Match the outcome rather than checking that it is not None, because unfinished carries a result that the outcome alone does not carry. A region left empty because nobody examined it and a region left empty because an agent looked and found nothing are different findings.

Settled and WallClockExpired name the agents still holding an unacknowledged notification. Aborted names none: abort closes the run and leaves unfinished empty, so a caller that stopped a run and wants to know who had not answered asks the store. store.read_agents returns each agent with how far it was told and how far it answered, and one told further than it has answered is one of those agents.

Being told instead of asking¶

wait_closed blocks the calling thread. A caller that would rather be told gives create_model an on_closed callback.

model = create_model(..., on_closed=record_the_outcome)

It is called once, with the same RunOutcome that wait_closed returns, on whichever thread closed the run: the caller's thread for abort, and the clock's for the idle limit and the wall clock. An exception it raises is suppressed, so a callback that fails does not reach whoever closed the run.

The outcome names no board, so a callback that serves several runs takes the identifier from the call that opened the run it belongs to. Serve a blackboard uses on_closed that way, to drop a closed run from the runs that the service reads.

After it closes¶

Reads keep working, so the result stays available. A write to a level or a premise comes back Rejected with the cause RUN_CLOSED. Registering an agent or declaring a region raises RunClosedError. The run explains why a write returns and a registration raises.

The names¶

TerminationPredicate is the type of the callable that create_model takes: it receives a BoardReader and answers a TerminationDecision.

on_closed, on create_model, is called once with the RunOutcome when the run ends, on whichever thread ended it. It is how a service holding many runs learns that one finished, where outcome answers when it is asked and wait_closed waits for the run to close.