Websocket Access¶

A web socket is provided to retrieve a stream of events. This can be accessed by hitting a URL in the form of:

wss://<IP>:<API_PORT>/api/v3/ws?token=<AUTH_TOKEN>

Named endpoints are also supported for connection labeling (the name is accepted but not otherwise interpreted by the server):

wss://<IP>:<API_PORT>/api/v3/ws/:name?token=<AUTH_TOKEN>

The URL will require authentication, but once authorized, the resulting web socket will receive events.

Authentication and Authorization¶

The websocket upgrade request is authenticated like any other API request, using any one of:

• An Authorization: Bearer <token> or Authorization: Basic <base64 user:pass> HTTP header.
• The token query parameter (lowercase). A value containing a colon is treated as a user:password pair; any other value is treated as an API token.
• A TLS client certificate.

Requests with no credentials receive a 401 response; invalid credentials receive a 403 response.

Once connected, events are additionally filtered by the authenticated principal's claims: an event is only delivered if the principal is allowed either the event action or the event's specific action on the event's object -- that is, a claim matching <type> event <key> or <type> <action> <key>.

Events can be filtered by submitting register filters to the websocket listener.

Register for Events¶

The events desired must be registered for. This is done by sending a register request to the server. By sending, register *.*.*, this will cause all events that the authorized user can see to be delivered to the websocket receiver.

Multiple registers (and the corresponding deregister) are allowed.

The general form is:

register type.action.key or register type.action.key.filter

The fields are:

• type - the type of object (e.g. profiles, machines, ...)
• action - the action of the object (e.g. create, save, update, destroy, ...)
• key - the specific key of the object (e.g. machine uuid, profile name, ...)
• filter - A filter that can be used for more specific matching of events beyond the type.action.key matches.

Some simple example are provided in the DRP source tree:

Acknowledgements and Error Handling¶

Every register and deregister message is acknowledged with an event of type websocket sent back over the socket, with Action set to register or deregister and Key set to the selector exactly as it was sent:

{"Time":"...","Type":"websocket","Action":"register","Key":"machines.update.*","Principal":"","Object":null,"Original":null}

Registering a selector that is already registered is a no-op, but it is still acknowledged.

Malformed input is fatal to the connection: any message other than a well-formed register or deregister command, or a selector that cannot be parsed, causes the server to close the websocket with close code 1003 (unsupported data) and a text message describing the problem.

Event specifiers¶

The type, action, and key fields in an event specifier can have the following values:

• * to indicate all possible values of that field count towards a match.
• A comma-separated list of values appropriate to the field with no intervening spaces -- object types for the type field, action names for the action field, and key values for the key field. The wildcard * cannot appear inside a comma-separated list (e.g. machines.create,delete.* is valid but machines.create,*,delete.* is not).

Wildcards only apply to a whole field; there is no partial or glob matching within a field (machines.cre*.* is not valid).

Machine events are published under a type that depends on the machine's role: machines, clusters, or resource_brokers. Register for each type you care about, or use a comma-separated list (e.g. machines,clusters.*.*).

Escape Sequences¶

If your specifier values contain the delimiter characters (. , *), you can escape them using backslash-encoded numeric sequences:

For example, to register for events on a machine key containing a literal asterisk, use register machines.*.some\3key.

There are a couple of special cases to enable efficient handling of certain job-related events:

• If the type is machine_jobs, then a job event whose Machine field matches the key of the event specifier will match the event assuming the other fields also match.
• If the type is work_order_jobs, then a job event whose WorkOrder field matches the key of the event specifier will match the event assuming the other fields match.

These specifier types are available when the server advertises the machine-jobs-register feature flag.

The filter field in the event specifier is optional. If it is present, it is interpreted as one of two filter languages, tried in this order:

1. The client compatible filter language described below.
2. If the filter does not parse as that language, the list-style filter language used by the list API endpoints.

Client compatible filter language¶

This language is designed to be easily implementable in client API libraries and operate identically on the native Go objects and their JSON serialization. It is available on the server side if the server advertises the websocket-changed-filter feature flag, otherwise it can also run on the client side as an event post-processor.

A filter may consist of several comma-separated terms at the top level; they are implicitly Anded together.

• value is the JSON representation of the value you want to filter on.
• fields is either a dot-separated string or (if it starts with /) an RFC6901 compliant JSON pointer specifying the field in a potentially nested struct or array to look up. Unless the first path segment is Object, Original, or event, the path is evaluated against the event's Object. Use an event prefix (e.g. event.Principal) to test the event envelope fields (Type, Action, Key, Principal, Time), or an Original prefix to test the object state before the change. In dot-separated paths, array indexes are written as [N], and the escape sequences above may be used for literal dots in a field name.
• term is any syntactically correct filter expression.
• terms is a comma separated list of syntactically correct filter expressions.
• And(terms) will match of all of the terms match. Terms are tested from left to right, and the first one that does not match stops testing. All is accepted as an alias.
• Or(terms) will match if any of the terms match. Terms are tested from left to right, and the first one that matches stops testing. Any is accepted as an alias.
• Not(term) returns the opposite of the term. It takes exactly one term.
• fields=Exists() checks to see if there is a value at fields at all, no matter what the value may be. It takes no arguments.
• fields=Eq(value) checks to see if the value at fields is equal to the passed in value. A missing field does not match.
• fields=Ne(value) checks to see if the value at fields is not equal to the passed in value. A missing field matches.
• fields=Re(value) checks to see if the value at fields matches the passed-in regular expression. The value at fields must be a string, and the regular expression is unanchored.
• fields=In(value,value...) checks to see if the value at fields matches any of the passed-in values. It is equivalent to Or(fields=Eq(value1),fields=Eq(value2),...), except that a missing field matches In.
• fields=Nin(value,value...) checks to see if the value at fields does not match any of the passed-in values. It is equivalent to Not(Or(fields=Eq(value1),fields=Eq(value2),...)), except that a missing field does not match Nin.
• fields=Changed(from,to) behaves specially if it is matching against either a models.Event or an api.RecievedEvent. from and to must both be valid JSON values, the same as value. The separating comma is required even when one side is omitted (e.g. Runnable=Changed(,true) matches a change to true from any other value), and if both are given they must differ.
• If the event being tested has non-nil Object and Original fields, Changed will return false if fields is not present in both objects or if it is present and identical in both objects.
• If from is specified and the value at fields in Original is not Eq to it, Changed will return false.
• If to is specified and the value at fields in Object is not Eq to it, Changed will return false.
• Otherwise, Changed will return true. If Changed is testing anything besides a models.Event or an api.RecievedEvent, it returns true if to is not specified or Eq(to) returns true.

List-style filter language¶

If the filter does not parse as the client compatible language, it is interpreted using the same syntax as the filters accepted by the list API endpoints. This support is advertised by the websocket-filters feature flag.

The filter is a space-separated list of Field=value or Field=Op(args) clauses, where each field must be an indexed field on the event's object type. Supported operations are Eq, Ne, Re, Lt, Lte, Gt, Gte, Between(low,high), Except(low,high), In(v1,v2,...), and Nin(v1,v2,...); a bare value means Eq. Multiple clauses for the same field are ORed together; clauses for different fields are ANDed.

A bare word that is not a Field=value clause is looked up as a stored filters object by that name and expanded in place, so saved filters can be reused directly:

register machines.update.*.my-saved-filter

Unlike the client compatible language, list-style filters can only match events whose Object is a complete object of the event's type; they cannot test the Original object or the event envelope fields.

Deregister Events¶

If you no longer wish to receive specific events you have registered for, you may use the deregister command. The command syntax is exactly like the register command.

The general form is:

deregister type.action.key or deregister type.action.key.filter

The deregister selector must exactly match a previous register message -- including the filter text, character for character.

CLI Event Tools¶

The drpcli tool includes built-in commands for watching and posting events without writing custom code.

Watching Events¶

# Watch all events
drpcli events watch

# Watch only machine events
drpcli events watch "machines.*.*"

# Watch machine creation events
drpcli events watch "machines.create.*"

# Watch events for a specific machine UUID
drpcli events watch "machines.*.a1b2c3d4-e5f6-7890-abcd-ef1234567890"

The watch command streams events to stdout and runs until interrupted (Ctrl+C). The optional filter argument uses the standard type.action.key glob pattern; if omitted, it defaults to *.*.*.

When the server supports registration filters, a fourth .filter component may be added as well:

# Watch machines whose Runnable flag transitions from false to true
drpcli events watch 'machines.update.*.Runnable=Changed(false,true)'

Posting Events¶

# Post a custom event inline
drpcli events post '{"Type":"custom","Action":"deploy-complete","Key":"my-machine"}'

# Post an event from a file or stdin
drpcli events post - < my-event.json

Post accepts a JSON or YAML-encoded Event object with Type, Action, Key, and optional Object fields. Custom events are published to the event bus and consumed by any registered WebSocket watchers.

Watching Logs¶

Server log messages are published as events with type log, so they can be streamed over the same websocket. drpcli logs watch is a shortcut for registering log.*.*.

Waiting on Object State¶

Every object type also has wait and await subcommands that use the websocket event stream to block until an object reaches a desired state. wait tests a single field for equality; await accepts one or more expressions in the client compatible filter language:

# Wait up to 300 seconds for a machine's Runnable flag to become true
drpcli machines wait a1b2c3d4-e5f6-7890-abcd-ef1234567890 Runnable true 300

# Wait for the machine to finish its workflow (default timeout is 24 hours)
drpcli machines await a1b2c3d4-e5f6-7890-abcd-ef1234567890 'WorkflowComplete=Eq(true)' --timeout 300

Both print complete, timeout, or interrupt on exit. Multiple await expressions are ANDed together.

Websocket Tools¶

For Go programs, the DRP client library (gitlab.com/rackn/provision/v4/api) provides complete event stream support: Client.Events() opens the websocket and negotiates the feature flags described on this page automatically, EventStream.Register subscribes using the selector syntax above, and EventStream.WaitForEvent blocks until a matching event arrives. drpcli and the DRP agent are built on this package.

Most modern languages provide websocket libraries that you can use to create listeners in a given language. Some examples include (this is not an exhaustive list):

• Golang: https://github.com/gorilla/websocket
• Golang: https://github.com/gobwas/ws
• Python: https://www.willmcgugan.com/blog/tech/post/announcing-lomond/
• Python: https://github.com/Lawouach/WebSocket-for-Python/tree/master/requirements
• Python: https://github.com/websocket-client/websocket-client
• Javascript: https://github.com/websockets/ws
• google is your friend ...

There are several extensions/add-ons for web browsers that will allow you to do basic testing of websocket listening. Here at RackN, we have used the following with some success:

• Chrome/Firefox: https://github.com/WangFenjin/Simple-WebSocket-Client
• Chrome: https://chrome.google.com/webstore/detail/smart-websocket-client/omalebghpgejjiaoknljcfmglgbpocdp?hl=en

There is a simple sample Python script available in the Digital Rebar Provision repo for reference, see the Websocket Integrations: page for further details.

Keepalive Behavior¶

The server sends a ping frame every 20 seconds. If a pong is not received within 30 seconds, the server closes the connection. Clients should ensure their pong handler resets the read deadline to maintain long-lived connections.

A slow consumer is not disconnected: if an event cannot be written to the connection within 10 seconds, the server logs the failure and drops that event for that connection. Clients that cannot keep up will silently miss events.

Event Format¶

Every event is a JSON object with the following fields:

By default, events include both the Object field (the current state) and the Original field (the state before the change). For clients that don't need both, the event format can be configured to reduce bandwidth.

Set the format using either the event-format query parameter or the X-Ws-Event-Format HTTP header on the websocket upgrade request:

wss://<IP>:<API_PORT>/api/v3/ws?Token=<AUTH_TOKEN>&event-format=object-only

The following formats are available:

Suppressed fields are sent as null rather than omitted. If both the query parameter and the header are supplied, the header takes precedence.

If an invalid format is provided, the server returns a 400 Bad Request error and the websocket connection is not established.

Subsystem Events¶

Most events describe the lifecycle of a stored object, so their Type is an object prefix (machines, profiles, ...) and their Key is that object's key. Several DRP subsystems also publish events that are not tied to any object type:

connections events are covered under Monitoring Connections below.

Note that the websocket type is not in this list. Registration acknowledgements are written directly to the requesting socket and never reach the event bus, so you cannot register for them or observe another client's registrations. See Acknowledgements and Error Handling.

File Transfer Events¶

The TFTP server publishes tftp.serve.<filename> and the static HTTP/HTTPS file servers publish static.serve.<url>. Both carry the same Object:

These events are published from a deferred handler, so they fire whether the transfer succeeded or failed. A FAILED event is therefore proof that a request arrived, which makes them the cheapest way to confirm a network-booting machine is reaching the endpoint. See TFTP Debugging FAQ for how this is used in practice.

FAILED is not on its own a fault indicator: PXE clients routinely issue a request purely to learn a file's size and abort it immediately, which is recorded as FAILED, then re-request the file for real. Expect a FAILED and a SUCCESS per file on a healthy boot.

{"Time":"...","Type":"tftp","Action":"serve","Key":"lpxelinux.0","Principal":"tftp","Object":{"Start":"...","End":"...","RequestSize":0,"ResponseSize":45056,"Status":"SUCCESS","Requestor":"10.10.20.76","Url":"lpxelinux.0"},"Original":null}

Log Events¶

Every server log line is published as an event with Type log, Action set to the level (trace, debug, info, warn, error, panic, fatal, audit), and Key set to the log service name (dhcp, static, render, bootenv, frontend, plugin, backend, runner, ...). Principal identifies the emitting subsystem, which is narrower than the service: TFTP log lines, for example, use service static with principal tftp.

drpcli logs watch is a shortcut for registering log.*.*. Register a specific level or service to narrow the stream, and filter on event.Principal to isolate one subsystem within a service:

# Only errors, from any service
drpcli events watch "log.error.*"

# Static file service lines emitted by the TFTP server
drpcli events watch 'log.*.static.event.Principal=Eq(tftp)'

Each service only publishes what its corresponding debug* preference allows, and those default to warn. Raising one with drpcli prefs set debugDhcp debug increases what reaches the stream.

Filtering Subsystem Events¶

Two constraints apply to these types that do not apply to object events:

• Keys frequently contain dots, and cannot be matched exactly. Selectors are split on ., so tftp.serve.lpxelinux.0 does not match the file lpxelinux.0 (the trailing 0 is parsed as a filter). The \1 escape sequence does not help here, because registration matches the raw key before escapes are decoded. Register tftp.serve.* and narrow with a filter expression instead.
• Only the client compatible filter language works. The list-style language requires the event's Object to be a complete instance of an object type, which these events do not have.

# Everything one machine fetches, over either protocol
drpcli events watch 'tftp,static.serve.*.Requestor=Eq(10.10.20.76)'

There is no role scope for tftp, static, or log, so no specific claim grants or restricts them. Any authenticated principal that registers for them will receive them.

Matched Filter Streaming¶

When a client registers multiple event filters, it can be useful to know which filters caused a particular event to be delivered. This avoids the need to re-run filter matching on the client side.

Enable this by setting the send-matched-filters query parameter or the X-Ws-Send-Filter-Match HTTP header on the websocket upgrade request. In both cases the value is ignored; the presence of the parameter or header enables the feature:

wss://<IP>:<API_PORT>/api/v3/ws?Token=<AUTH_TOKEN>&send-matched-filters=true

When enabled, the server sends two messages for every event, always in lockstep:

1. A JSON string array listing the registration specifiers that matched the event
2. The event itself

For example, if a client has registered profiles.save.* and profiles.*.*, and a profile save event occurs, the client will receive:

["profiles.save.*","profiles.*.*"]

followed immediately by:

{"Time":"...","Type":"profiles","Action":"save","Key":"global","Object":{...}}

Registration and deregistration acknowledgement events also include matched filters when this mode is enabled. The matched filter for these events is always ["websocket.*.*"].

Both options can be combined:

wss://<IP>:<API_PORT>/api/v3/ws?Token=<AUTH_TOKEN>&event-format=object-only&send-matched-filters=true

Monitoring Connections¶

Digital Rebar has an API endpoint for listing active websocket and REST Connections under GET /api/v3/connections. This list of connections contains information such as create time, principal, address, and if the connection is a websocket.

Connections for specific machines can be accessed via GET /api/v3/machines/:uuid/connections. Clusters and Resource brokers have the similarly named API endpoints.

New connections and disconnections can be monitored by the connections.create.<principal> event. This is useful as individual runner connections can be monitored with connections.create.runner:<uuid>

A combination of the machine connections API endpoint and connection events can be used to monitor a relative "online" state of a machine's runner, a user's portal session, or the connectivity of a plugin.

Example Information¶

Here is a simple walk through of basic testing on how to use websockets with Digital Rebar. Please note this is fairly basic, but it should get you started on how to interact with and use websockets. This example was tested, using the " Simple Websocket Client" in both Chrome and Firefox that is listed above.

We assume you have the DRP endpoint installed on your localhost in these examples. You can adjust the IP address/hostname to point to a remote DRP Endpoint, just ensure you have access to Port 8092 (by default, or the API port you specify if you changed the default).

URL: wss://127.0.0.1:8092/api/v3/ws?token=rocketskates:r0cketsk8ts

Note that the token information is a set of credentials with permissions to view events. This example uses the default username/password pair. You may also create and specify access Tokens for the websocket client to use.

In the Request input box, enter your register filter you'd like to receive events for.

Request: register profiles.*.*

This example will only output websocket events related to profiles. Now create and delete a few test parameters

# now create a `bar` param on the `global` profile
drpcli profiles set global param bar to blatz

# now remove the param from the `global` profile
drpcli profiles remove global param bar

...and you should see events like:

{"Time":"2017-12-21T23:26:43.412554192Z","Type":"profiles","Action":"save","Key":"global","Object":{"Validated":true,"Available":true,"Errors":[],"ReadOnly":false,"Meta":{"color":"blue","icon":"world","title":"Digital Rebar Provision"},"Name":"global","Description":"Global profile attached automatically to all machines.","Params":{"bar":"blatz","change-stage/map":{"centos-8-install":"packet-ssh-keys:Success","discover":"packet-discover:Success","packet-discover":"centos-8-install:Reboot","packet-ssh-keys":"complete-nowait:Success"},"kernel-console":"console=ttyS1,115200"}}}
{"Time":"2017-12-21T23:27:15.218761478Z","Type":"profiles","Action":"save","Key":"global","Object":{"Validated":true,"Available":true,"Errors":[],"ReadOnly":false,"Meta":{"color":"blue","icon":"world","title":"Digital Rebar Provision"},"Name":"global","Description":"Global profile attached automatically to all machines.","Params":{"change-stage/map":{"centos-8-install":"packet-ssh-keys:Success","discover":"packet-discover:Success","packet-discover":"centos-8-install:Reboot","packet-ssh-keys":"complete-nowait:Success"},"kernel-console":"console=ttyS1,115200"}}}