commit bfa2a877c6ef4bf23aa0767d4fe06f22c347684b
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Fri Feb 18 08:55:00 2022 -0600

    feat(ci): Make Gitlab CI process more scalable.

    This fixes up how the unit tests detect when they are running
    under the Gitlab CI infrastructure, and allow for the test
    phases to run in parallel to the extent feasible.

M   .gitlab-ci.yml
M   test/plugins.go
A   tools/multitest.sh
M   tools/test.sh
A   tools/test_prereqs.sh

commit 2b579b528ea8b78f6c560cd7772ba5d43256e0b3
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Feb 14 13:58:43 2022 -0600

    fix(consensus): Fix deadlock and cert rotation error

    Etags.Remove and Etags.RemoveAll have baked-in deadlocks that are too
    easy to trigger with external uses of Etags.SendArtifactOp.  Fix it by
    splitting SendArtifactOp into two functions, one that takes the locks
    and one that does not, and fix up all in-package callers of
    SendArtifactOp to take whatever locks are needed and then call

    The consensus HA TLS handshaking strategy could fail if a root cert
    rotate happened while one of the nodes was out of the cluster. Once
    that happened, the missing node would be unable to rejoin the cluster
    when it came back up due to it not having the most recent root
    certificate for TLS peer cert checking.  This has been resolved with
    two fixes:

    * On the client side (the one initiating the connection), we start by
      picking a cert that is signed by the most recent cert in the Roots
      array.  If the TLS handshake fails, we will iterate through the Roots
      array in order from most recent to oldest, retrying the connection
      with each until either the handshake passes or we run out of root
      certs.  If we run out of root certs, we wrap around and try again.

    * On the server side (the one receiving the connection), we ratchet
      the same way, except we move on to the next oldest cert after
      len(fsm.Roots) handshake failures instead of at every one.  If we
      run out of certs, we will omit an error message to the log and wrap

    These two behaviours ensure that all possible current common sets of
    mutually valid root certs are tested when creating TLS connections
    between consensus members.

    Additionally, the root cert rotation code was incorrectly calculating
    when to rotate the root certificates, leading to an insufficient
    number of known trusted roots and the possibility of the nodes not
    having enough certs in common to mutually authenticate their peers.
    We now handle this like so:

    1. On cluster create, we seed a TLS certificate that every node in the
       cluster uses for mutual peer authentication.  This cert is valid from
       time.Now().Add(-consensus.CertGrace) (to handle servers that do not
       have well-synchronized clocks) through time.Now().Add(RootCertExpire+CertGrace).
       This gives the initial cert a 3 month lifespan.

    2. Every RootCertExpire/(2*MaxTrustRoots+1), the leader of the
       consensus cluster checks to see if it needs to generate a new root
       certificate by checking to see if the first entry in the Roots
       array of the finite state machine is more than
       RootCertExpire/MaxTrustRoots old. If so, it generates a new
       self-signed cert, adds it to the front of the Roots array in the
       finite state machine, removes any certs in the Roots array that
       have expired, and publishes the new FSM via Raft.

    The failover tests have been updated to have much increased cert
    rotation rates in order to test this code.

M   .gitlab-ci.yml
M   consensus/raft.go
M   consensus/raftFSM.go
M   consensus/serverAPI.go
M   datastack/etags.go
M   failover_test/consensus_failover_test.go
M   go.mod
M   go.sum
M   server/args.go

End of Note