8.3.2.36. v4.4.1ΒΆ

commit b4bd5fcba48055831725daf95998d3f2a62bdc9d
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Fri Aug 7 09:53:42 2020 -0500

    fix(unittest): Sigh, don't check the size of incrementer

M   clitest/test-data/output/TestPluginsFromContent/plugin_providers.upload.incrementer.from/tools/build/incrementer/stdout.expect

commit 8b478f7c5c64fd1332147f2fe56562ea07d075fd
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Thu Aug 6 14:14:21 2020 -0500

    fix(contentLoad): Fix plugin-related content load bugs.

    The backend Validate function has no idea where the sockets for talking
    to running plugins were located, so it could wander off into lala land
    when it needed to communicate with a plugin. Fix this by making
    datastack responsible for keeping track of it instead.

    Identical content detection did not consider any content to be identical,
    due to content validation fields and the ReadOnly field being guaranteed
    to be different.  Before checking to see whether items are identical,
    set ReadOnly to be true (since at that point it is guaranteed to be
     anyways), and clear out validation.

    Add a unit test to test that plugin add/change/remove triggers the
    start/change/stop events to plugin providers as appropriate.

    Signed-off-by: Victor Lowther <victor.lowther@gmail.com>

M   backend/dataTracker.go
M   backend/dataTracker_test.go
M   backend/plugin_controller.go
M   backend/runningPlugin.go
M   backend/utils.go
M   clitest/plugins_test.go
A   clitest/test-data/incrPluginChangeOne.yaml
A   clitest/test-data/incrPluginCreate.yaml
A   clitest/test-data/incrPluginRemove.yaml
A   clitest/test-data/output/TestPluginsFromContent/contents.destroy.incrContent/stdout.expect
A   clitest/test-data/output/TestPluginsFromContent/contents.upload.test-data/incrPluginChangeOne.yaml/stdout.expect
A   clitest/test-data/output/TestPluginsFromContent/contents.upload.test-data/incrPluginCreate.yaml/stderr.expect
A   clitest/test-data/output/TestPluginsFromContent/contents.upload.test-data/incrPluginCreate.yaml/stdout.expect
A   clitest/test-data/output/TestPluginsFromContent/contents.upload.test-data/incrPluginRemove.yaml/stdout.expect
A   clitest/test-data/output/TestPluginsFromContent/plugin_providers.upload.incrementer.from/tools/build/incrementer/stdout.expect
M   datastack/stack.go
M   server/server.go

commit 2b1865b086cf9fc8f84ad32521dafa052e9ec94a
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Wed Aug 5 12:31:23 2020 -0500

    fix(indexCorruption): Indexes were being corrupted by duplicate adds/removes

    The index.Add and index.Remove code was not properly accounting for having duplicate items passed to Add and Remove,
    respectively.  That caused Remove to remove extra unrelated items in the case of being called with duplicates,
    potentially leading to a panic dues to slices being out of bounds.  Add was potentially allowing multiple items in the
    Index to have the same Key(), which violates our deeply held beliefs that all keys must be unique for sanity's sake.

    This fixes both of those issues by refactoring the Add and Remove codepaths to be aware of duplicates and handle them --
     in the case of the Remove codepath, duplicates are ignored, and in the case of the Add codepath, the last of the
     objects with identical Key() values will be the one that winds up in the index.

    Signed-off-by: Victor Lowther <victor.lowther@gmail.com>

M   datastack/index/index.go
M   datastack/index/index_test.go

commit 6778cb18e7ba484d9938ce9c04d9931b7581891a
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Tue Aug 4 08:26:38 2020 -0500

    fix(secureBoot): Grub does not work in UEFI with proxy DHCP.

    There is a known issue with Grub not reading DHCP options from
    the cirrent packet.  Until that is resolved upstream, detect the
    situation and fall back to not using secure boot mode.

M   backend/dhcpUtils.go
M   midlayer/pxe.go

commit f0a28e8ff500a752b6cbf007fec732db4640e8ab
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Thu Jul 30 12:22:27 2020 -0500

    build: update v4.4.0 provision and provision-plugins

M   go.mod
M   go.sum

commit e0e511b4add3ae9c1c1804048a276e80e7ca31bf
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Wed Jul 29 16:34:08 2020 -0500

    build: update to latest drpcli

M   go.mod
M   go.sum

commit f07844ef7ff8c1573c0dd2c8caaa99ba9161de9c
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Wed Jul 29 12:31:47 2020 -0500

    fix(template): Allow template IDs to start with nuimbers

A   clitest/catalog_test.go
M   go.mod
M   go.sum

commit 03c1bf97ecd14c6a422a3da6f91b899c547514ec
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Wed Jul 29 11:05:07 2020 -0500

    fix(embedded): add version to embedded drpcli

M   tools/build_cli.sh

commit 24708dbe1954377f3b844fbc5072d141020de27b
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Tue Jul 28 13:50:39 2020 -0500

    perf(contentPalLoad): Validate that botched updates get rolled back

M   backend/bootenv.go
M   backend/stage.go
M   backend/task.go
M   backend/template.go
M   clitest/contents_test.go
A   clitest/test-data/bob1.yaml
A   clitest/test-data/bob2.yaml
A   clitest/test-data/bob3.yaml
A   clitest/test-data/bob4.yaml
M   clitest/test-data/output/TestBootenvStageHandling/stages.create.30f84558be9d7caefb08a7672ef6a725/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.create.test-data/content.yaml/stderr.expect
A   clitest/test-data/output/TestContentLoadValidation/bootenvs.create.0fd684c07aaf1096509e7449b0e1c140/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/bootenvs.create.7284e44c2281470dc7a0ae35e0922b05/stderr.expect
A   clitest/test-data/output/TestContentLoadValidation/bootenvs.create.8fef9e18ebddc6023f741c0e379308da/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/bootenvs.create.be5b231134c9ddc72eb2b6c679fb0dd0/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/bootenvs.destroy.unknownBob/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/contents.destroy.bob/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/contents.upload.test-data/bob1.yaml.2/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/contents.upload.test-data/bob1.yaml/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/contents.upload.test-data/bob2.yaml/stderr.expect
A   clitest/test-data/output/TestContentLoadValidation/contents.upload.test-data/bob3.yaml/stderr.expect
A   clitest/test-data/output/TestContentLoadValidation/contents.upload.test-data/bob4.yaml/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.bootenvs/bob.2/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.bootenvs/bob.3/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.bootenvs/bob.4/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.bootenvs/bob.5/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.bootenvs/bob/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.stages/fred.2/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.stages/fred.3/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.stages/fred.4/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.stages/fred.5/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.stages/fred/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.uk/fred.2/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.uk/fred.3/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.uk/fred.4/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.uk/fred.5/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/files.static.uk/fred/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/machines.create.bob/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/machines.destroy.Name.bob/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/machines.update.Name.bob.050d0da24857c13849b0458615134e40/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/prefs.set.unknownBootEnv.ignore/stdout.expect
A   clitest/test-data/output/TestContentLoadValidation/prefs.set.unknownBootEnv.unknownBob/stderr.expect
A   clitest/test-data/output/TestContentLoadValidation/prefs.set.unknownBootEnv.unknownBob/stdout.expect
M   clitest/test-data/output/TestTaskCli/tasks.create.8d20432a4e21ef071b283910d9596673/stderr.expect
M   clitest/test-data/output/TestTaskCli/tasks.create.c63da551ae6c6bf9e78f37d69408c66c/stderr.expect
M   clitest/test-data/output/TestTaskCli/tasks.create.dedd6ced565015e77ca12954ca5e76b9/stderr.expect
M   clitest/test-data/output/TestTasksWithElevatedTokens/tasks.create.416f18fb9dc46b108e9d06571a7d1709/stderr.expect
M   clitest/test-data/output/TestTasksWithElevatedTokens/tasks.create.de8d8a847507a34220f8ff818a1aa85a/stderr.expect
M   go.mod
M   go.sum
M   pacman/PKGBUILD

commit eab5a14b56a921902b3c0636877e9fa5e0fd6c6a
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Tue Jul 28 11:12:31 2020 -0500

    Factor out common template munging code

M   backend/dataTracker.go
M   backend/stage.go
M   backend/template.go
M   backend/utils.go

commit f83312864d9f3f19a3d59a976820681a13b6bdce
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Tue Jul 28 13:06:43 2020 -0500

    feat(license): All HaId To match endpoint info as well

M   backend/license.go

commit bd7e346898df7975981fe8d7bd2bd6d86d2094dc
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Tue Jul 28 00:13:48 2020 -0500

    fix(backend): minor change with lease fill

M   backend/dataTracker.go

commit 3d8dc32d75ec487ac9bb7b85905dece2c8ca88d2
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Fri Jul 24 14:09:26 2020 -0500

    fix(template): Fix up template validation handling.

    It turns out that cramming all the tempaltes together to validate them
    can have some negative side effects, and did not give is the leyer of
    consistency checking I thought it had been giving us.

    Instead, rework template validation in the following ways:

    1. Disallow templates that have embedded {{define}} or {{begin}}
       blocks.  These let you create extra named and anonymous go
       templates, which can conflict with ones defined directly by Template
       objects.  Support for these may be readded at a later date once
       we work out exactly how to validate and ensure that all templates in
       a given sacope are uniquely named.

    2. Allow for incremental redefinition of the root template pool when
       templates are added and remove, and move the location of the root
       pool from the datatracker to the datastack.  This makes keeping track
       of the root pool simpler.

    Moving it to the datastack makes keeping validated root templates in
    sync with the datatrackers simpler.

M   backend/bootenv.go
M   backend/dataTracker.go
M   backend/stage.go
M   backend/task.go
M   backend/template.go
M   backend/utils.go
M   datastack/stack.go

commit 5d28529490a144b63c52ef2a7047a384700e03d5
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Fri Jul 24 14:13:43 2020 -0500

    fix(manager): close response body on proxy
    fix(manager): scope for endpoint joining should include all objects
    fix(manager): machine test should not set Endpoint unless it means it

M   backend/endpoint_manager.go
M   frontend/endpoint.go
M   go.mod
M   go.sum

commit 0d9d1488f0b8f8bcfe7d8bfd6581ae44ac0767dd
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Wed Jul 22 15:47:10 2020 -0500

    fix(pools): Clarify a few codepaths in the pool code per Greg.

    Still doesn't look quite right, but it is as right as we can
    get it without some sort of distributed transaction mechanism.

M   frontend/pools.go

commit 751d1f3df4feb1f2623eb6f159be3ccf6421316e
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Thu Jul 16 11:19:55 2020 -0500

    fix(contents): Improve how we validate content packs.

    This commit changes how we validate objects that are added, changed,
    or removed by adding, removing, or updating a content pack.  Instead
    of rewriting the indexes with the changes and then revalidating all of
    the data, we keep track of what has been added, removed, or changed by
    the index rewrites and then run the changes through the appropriate
    create/update/remove routines in the backend for their side effects.

    This allows us to precisely report on any errors that making the
    requested change would cause and return them, instead of getting
    unrelated errors mixed up in the output.

    It also refactors a bit of the split-manager-datastack to only expose
    objects from other endpoints to the frontend code where is is needed.
    The idea is that backend operations on local data should never
    accidentally pull in data from other endpoints.  There are a few
    places where that layering is intentionally violated -- mostly around
    pooling and endpoint object management -- and those areas need to be
    addressed before this commit can be accepted.

M   backend/bootenv.go
M   backend/contexts.go
M   backend/dataTracker.go
M   backend/endpoint_manager.go
M   backend/hooks.go
M   backend/license.go
M   backend/profiles.go
M   backend/requestTracker.go
M   backend/roles.go
M   backend/stage.go
M   backend/subnet.go
M   backend/task.go
M   backend/template.go
M   backend/utils.go
M   clitest/bootenv_test.go
M   clitest/contents_functional_test.go
M   clitest/test-data/output/TestAuth/roles.destroy.stage/stderr.expect
M   clitest/test-data/output/TestAuth/roles.destroy.task/stderr.expect
A   clitest/test-data/output/TestContentsFunctionalCli/contents.create.15403cc39b7ae9e66716216634f1cd9a/stderr.expect
A   clitest/test-data/output/TestContentsFunctionalCli/contents.create.740c331718d653ae77e7875d61c12c77/stderr.expect
A   clitest/test-data/output/TestContentsFunctionalCli/contents.create.d100b6546aa478bed0df87b28de5b4b6/stdout.expect
A   clitest/test-data/output/TestContentsFunctionalCli/contents.update.Pack1.2b39d1a2d37a7b6a6a516b308f64943e/stderr.expect
A   clitest/test-data/output/TestContentsFunctionalCli/contents.update.Pack1.9bd529bd15b272b95e51535ff4261dc8/stdout.expect
A   clitest/test-data/output/TestContentsFunctionalCli/contents.update.Pack1.fef6a27ccd18e22e0f1f99a6560ae342/stderr.expect
M   clitest/test-data/output/TestContentsFunctionalCli/profiles.list.3/stdout.expect
M   clitest/test-data/output/TestContentsFunctionalCli/profiles.list.4/stdout.expect
M   clitest/test-data/output/TestContextStuff/contexts.destroy.dangerzone/stderr.expect
M   clitest/test-data/output/TestJobCli/jobs.actions.john/stderr.expect
M   clitest/test-data/output/TestJobCli/jobs.log.john/stderr.expect
M   clitest/test-data/output/TestProfileInProfile/profiles.destroy.profile1/stderr.expect
M   clitest/test-data/output/TestProfileInProfile/profiles.destroy.profile2.2/stderr.expect
M   clitest/test-data/output/TestProfileInProfile/profiles.destroy.profile2/stderr.expect
M   clitest/test-data/output/TestSubnetCli/subnets.destroy.john.3/stderr.expect
M   clitest/test-data/output/TestUnknownBootEnvRerender/contents.destroy.fred.2/stderr.expect
M   clitest/test-data/output/TestUnknownBootEnvRerender/contents.destroy.fred.2/stdout.expect
M   clitest/test-data/output/TestUnknownBootEnvRerender/contents.destroy.fred/stderr.expect
M   clitest/test-data/output/TestUnknownBootEnvRerender/contents.destroy.fred/stdout.expect
M   clitest/test-data/output/TestUnknownBootEnvRerender/files.static.pxelinux.cfg/default.5/stdout.expect
M   datastack/index/index.go
M   datastack/stack.go
M   frontend/actions.go
M   frontend/bootenvs.go
M   frontend/endpoint.go
M   frontend/endpoints.go
M   frontend/frontend.go
M   frontend/info.go
M   frontend/jobs.go
M   frontend/machines.go
M   frontend/meta.go
M   frontend/params.go
M   frontend/pools.go
M   frontend/tasks.go
M   frontend/users.go
M   frontend/version_sets.go

commit ac592cf6af794f20a85855907d2f44dc9e3b4c04
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Mon Jul 20 09:54:46 2020 -0500

    refactor(manager): rework datastore to not mingle datastack objects

    This reworks the datastack objects from the main datastack.  This
    uses a merge of data on read.

M   backend/endpoint_manager.go
M   backend/requestTracker.go
M   datastack/endpoint_stack.go
M   datastack/stack.go
M   datastack/streamingSyncActive.go
M   datastack/streamingSync_test.go
M   datastack/wal.go
M   frontend/debug.go
M   frontend/system.go

commit 04e2e4fe5fb0c38cfed392c1fef86a603d220b17
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Jul 27 12:37:01 2020 -0500

    feat(waltool): Merge dr-backup into walfsck, making it dr-waltool.

    Along the way, rewrite walfsck to use a nice CLI interface with
    the ability to make it moduilar and add extra help as needed.

D   cmds/dr-backup/main.go
D   cmds/dr-humanize/main.go
A   cmds/dr-waltool/waltool.go
D   cmds/walfsck/walfsck.go
M   go.mod
M   go.sum
M   pacman/PKGBUILD
M   tools/build.sh

commit 84c85d1f9f346263d9da520baf6c15a0077f8305
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Jul 27 10:31:33 2020 -0500

    fix(dhcp): Lower log priority of packets from unconfigured interfaces.

    They don't need to be debug, as at that point they tend to overwhelm
    other, more desired logging if they are present at all.  Lower their
    logging to trace.

M   midlayer/dhcp.go

commit c27c2d12168dbb90018bbb76732225d089bb41be
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Wed Jul 22 12:52:32 2020 -0500

    perf(jobPurge): Speed up certain Machine and Job index operations.

    A performance trace from David Neal shows that the purgeJobs function
    was taking an excessive amount of time to determine what jobs to
    remove, accounting for around 20% of the total CPU time eaten by a
    perf trace that also included a content pack reload.  Closer analysis
    determined that half of that time was taken up by repeatedly
    converting UUIDs from theior native byte array format to a printable
    string for index comparison purposes.  Since they are byte arrays
    natively, we can just use the native byte array comparison and
    equality testing instead.

    Go ahead and make this change for every index operation that deals
    with UUIDs, since they were all doing string-based comparisons.

    Overall, the job purge work was unneeded in any case, since David has
    both of the purge parameters set to "never".  We now detect that case
    and avoid querying the Job index at all if true.

M   backend/jobs.go
M   backend/machines.go

commit 72a7e3456d0ba7219ecb73e12aebaa14949b8f32
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Tue Jul 21 17:10:36 2020 -0500

    fix(manager): AddEndpoint should return endpoint object on success

M   frontend/plugins.go

commit c8f14462f1b0abd878b5d014cd993b90b231e47c
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Tue Jul 21 12:48:41 2020 -0500

    fix(dhcp): Properly handle lease requested addresses in DISCOVER

    We were not properly ignoring the lease requested address in the
    DISCOVER codepath when there was conflicting non-expired lease with
    the address being requested, leading to a DISCOVER -> OFFER -> REQUEST
    -> NAK loop.  Fix it by noticing the situation and zeroing out the
    requested address, which will cause us to allocate a new address.

M   backend/dhcpUtils.go
M   backend/dhcpUtils_test.go

End of Note