8.3.2.93. v4.4.4ΒΆ

commit 14a9c290d1fa6461dbbd207f18ae5a1023563cdb
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Tue Aug 18 10:07:32 2020 -0500

    fix(revalidate): Fix data race when validating content bundles.

    So, it turns out we were still falling through to the old revalidate
    everything code after processing a content bundle change list.  Not
    only was this hiding a few bugs with the new code, it caused spurious
    data races in the parallel revalidate everything code.

    Rework changes to make content bundle changes validate as intended and
    thereby avoid the spurious data race.

M   backend/dataTracker.go
M   backend/profiles_test.go
M   backend/utils.go

commit 69b0dc6cce58a981779bb28de5fa7b604e99b06b
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Aug 17 11:57:18 2020 -0500

    perf(startup): Parallelize object processing on startup

    This adds parallelism in two places at system startup.

    The first is when bulk unmarshalling data from the WAL.  The
    max batch size has been increased from 4 megs to 32, and actual
    unmarshalling load is equally divided among the CPUs present on the
    system if the total amount of data being unmarshalled passes a certain
    threshold.

    The second is when validating data when building the datatracker.  The
    data load for parallelization is divided across all CPUs in the same fashion
    as when being bulk unmarshalled, unless the objects in question are
    backed by a RawModel -- trying to parallelize operations on them resulted
    in concurrent map access panics.

M   backend/dataTracker.go
M   datastack/index/index.go
M   datastack/stack.go

commit 3757b23649a227e8d80ad90912accc50dce278ce
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Aug 17 11:00:52 2020 -0500

    perf(startup): Cache sorted indexes.

    This adds functionality to the Index layer to recognize when we are
    asking for a sorted version of a base index, and caches the result
    for later use if we are.  The cache will be invalidated if anything
    is added, removed, or modified in the base layer of the index.

    To enable this to work properly, Names have been added to all Makers
    where applicable, to give each index a set of unique keys to work with.

M   backend/bootenv.go
M   backend/catalog_items.go
M   backend/contexts.go
M   backend/endpoints.go
M   backend/jobs.go
M   backend/lease.go
M   backend/machines.go
M   backend/param.go
M   backend/pluginProviders.go
M   backend/plugins.go
M   backend/pools.go
M   backend/profiles.go
M   backend/raw_model.go
M   backend/reservation.go
M   backend/roles.go
M   backend/stage.go
M   backend/subnet.go
M   backend/task.go
M   backend/template.go
M   backend/tenants.go
M   backend/user.go
M   backend/version_sets.go
M   backend/workflow.go
M   datastack/index/index.go
M   datastack/index/index_test.go

commit f9ace71745a0d1eff292174745408991f2e0b98c
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Fri Aug 14 13:32:55 2020 -0500

    Perf enchancements to machine processing during startup

M   backend/bootenv.go
M   backend/dataTracker.go
M   backend/machines.go

commit a52e38262335c33965eb4d99e793e770ba65beb2
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Fri Aug 14 09:40:32 2020 -0500

    feat(walTool): Add more debugging to dr-waltool.

M   backend/dataTracker.go
M   cmds/dr-waltool/waltool.go
M   datastack/stack.go

commit f67e3cf9fd2ec0b35453ddadc1937c900f369503
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Fri Aug 14 12:14:54 2020 -0500

    test(parProfiles): Add staged transaction blocking tests.

    Add a test to testParallelProfiles that tests transaction
    unblocking at scale.

M   backend/profiles_test.go

End of Note