8.3.2.145. v4.7.9ΒΆ

commit 8c1ea83433b85ad13470489a7df46fdde46ba996
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Mon Oct 25 13:30:22 2021 -0500

    feat(jobs): add a pref that will delay job purges until idle for that amount of time

    This removes all job purges from directly inline with job creation.

M   backend/dataTracker.go
M   backend/jobs.go
M   backend/machines.go
M   clitest/test-data/output/TestContentCli/contents.list.2/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.list.3/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.list.4/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.list.5/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.list/stdout.expect
M   clitest/test-data/output/TestContentsFunctionalCli/contents.list.2/stdout.expect
M   clitest/test-data/output/TestContentsFunctionalCli/contents.list/stdout.expect
M   frontend/job_create.go
M   server/args.go

commit 951b38237f951467b3e30e9342861f5691f5b203
Author: Greg Althaus <galthaus@austin.rr.com>
Date:   Mon Oct 25 11:58:31 2021 -0500

    fix(raft): Test fix for reuse of certs

M   consensus/raft_test.go

commit 52f9b3c85846d980b940b4a84383060077828866
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 14:50:44 2021 -0500

    Derp, fix silly test errors

M   clitest/00_aone_test.go
M   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.upload.dangerzone.from/tools/build/dangerzone.2/stdout.expect
M   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.upload.incrementer.from/tools/build/incrementer.2/stdout.expect

commit 854204cb0b55eaf2004178911275854aa59e0208
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 14:10:26 2021 -0500

    feat(contents): Add contentsLocked preference.

    You can use this new preference to lock out any API driven content
    changes.  Useful if you need to make sure no-one can change contents
    they should not, even if they have the rights to do so.

M   backend/dataTracker.go
M   clitest/00_aone_test.go
M   clitest/contents_test.go
M   clitest/test-data/output/TestContentCli/contents.list.2/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.list.3/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.list.4/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.list.5/stdout.expect
M   clitest/test-data/output/TestContentCli/contents.list/stdout.expect
M   clitest/test-data/output/TestContentsFunctionalCli/contents.list.2/stdout.expect
M   clitest/test-data/output/TestContentsFunctionalCli/contents.list/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/contents.destroy.john.2/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/contents.destroy.john/stderr.expect
A   clitest/test-data/output/TestPluginProviderLockout/contents.upload.4c4e8ae2275d5c4ad58d3e7b5442a889.2/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/contents.upload.4c4e8ae2275d5c4ad58d3e7b5442a889/stderr.expect
A   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.destroy.dangerzone.2/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.destroy.dangerzone/stderr.expect
A   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.destroy.incrementer.2/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.destroy.incrementer/stderr.expect
A   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.upload.dangerzone.from/tools/build/dangerzone.2/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.upload.dangerzone.from/tools/build/dangerzone/stderr.expect
A   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.upload.incrementer.from/tools/build/incrementer.2/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/plugin_providers.upload.incrementer.from/tools/build/incrementer/stderr.expect
A   clitest/test-data/output/TestPluginProviderLockout/prefs.set.contentLocked.false.2/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/prefs.set.contentLocked.false/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/prefs.set.contentLocked.true.2/stdout.expect
A   clitest/test-data/output/TestPluginProviderLockout/prefs.set.contentLocked.true/stdout.expect
M   frontend/content.go
M   frontend/plugin_providers.go

commit b2fb136341a973c3e69316f21660e1da966645da
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 12:01:42 2021 -0500

    Stream returing a single content bundle like we would any other object

M   frontend/content.go

commit 606bdb170df70d9ea1b06bd2e7f18647ce214918
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 11:58:51 2021 -0500

    Locking everything to get content summaries is not actually needed

M   frontend/content.go

commit 10b52f499d812ffc44b1206a25134bd130cdd009
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 11:51:17 2021 -0500

    Skip reflect conversion compatibility checking when all we care about is the actual underlying byte slice

M   datastack/index/index.go

commit 16a28f1d9ebd97778d0ba47bc42cd16e59292651
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 11:50:11 2021 -0500

    Skip machine uniqieness checking if hte name did not change

M   backend/machines.go

commit 5a97e10c2c7f85b3e5cdeeb3720382c042f97df2
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 11:29:25 2021 -0500

    Have generic frontend.Remove code use encodeItemTo

M   frontend/frontend.go

commit dd1201592ca62a14ab82937674f2f35fdf358d85
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 11:28:20 2021 -0500

    Relax models.Models restriction on generic frontend encoding stuff

M   frontend/frontend.go
M   mr/modelReflect.go

commit 36dd1b582d505b172936facd648b4e388c766002
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Mon Oct 25 11:26:09 2021 -0500

    Add perf test for messing with a million -ish jobs

M   perf_tests/machineCreate_test.go

commit 93ce7e82d2b3435dc7e054575c625bb82bba2426
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Thu Oct 21 14:31:41 2021 -0500

    perf(frontend): Optimize frontend memory usage.

    This refactors various frontend functions to reduce memory consumption
    in a few different ways:

    1. Add frontend.Resolve and frontend.RawFind methods.  The former is
    optimized for first-pass object availability testing, and the second
    is frontend.Find refactored to operate inside a transaction where the
    expectation is that the object fetched will not be modified or sent
    over the wire by the end of the API call.

    2. Replace frontend.processItem with a collection of functions that
    let us interleave JSON encoding and any required mangling on objects
    we need to send over the API (for param decoding, object sanitization,
    slimming, etc).  It utilizes a combination of reflection and unwinding
    the top layer of JSON encoding to let us make field-by-field decisions
    on whether to skip or mangle arbitrary object fields.  Use this new
    functionality for the main read and write paths through the API.

    3. Sending Events no longer carries around deep copies of the objects
    the event is being sent for.  Instead, we marshal them beforehand and
    arrange for the marshalled data to be slotted in directly.

    4. Standardize on the jsoniter codec object that all the packages use
    and get rid of some legacy use of encoding/json in the front end.

    On my dev box, this reduces the time spent in the machine create and
    list performance test from 40.07 seconds -> 15.47 seconds in the API,
    and (more importantly) 12.45 seconds -> 3.72 seconds in the garbage
    collector.

M   backend/dataTracker.go
M   backend/machines.go
M   backend/requestTracker.go
M   backend/utils.go
M   clitest/common_test.go
M   clitest/test-data/output/TestPluginProviderCli/plugin_providers.list.2/stdout.expect
M   clitest/test-data/output/TestSecureParams/91e88b16a9c5b7e77215714ef9eba6ff/stderr.expect
M   clitest/test-data/output/TestSecureParams/bb5d4e3f789d07bec23bc261352ea358/stderr.expect
M   clitest/test-data/output/TestSecureParams/e19676fa6a1f473aaa7f7124105765d8/stderr.expect
M   commit/buffer.go
M   commit/entry.go
M   commit/entry_easyjson.go
M   frontend/actions.go
M   frontend/bootenvs.go
M   frontend/endpoint.go
M   frontend/endpoints.go
M   frontend/frontend.go
M   frontend/jobs.go
M   frontend/meta.go
M   frontend/params.go
M   frontend/pools.go
M   frontend/users.go
M   frontend/websocket.go
A   mr/modelReflect.go
A   utils/json.jo.go

commit 359e9076cf920e2cb486e0327ffca7fcccdb1f03
Author: Victor Lowther <victor.lowther@gmail.com>
Date:   Sun Oct 24 13:20:51 2021 -0500

    perf(frontend): Start adding some basic frontend perf testing.

    So far, just single-threaded machines create.  To use this:

    1. cd perf_tests
    2. go generate
    3. go test -cpuprofile cpu.out.before
    4. hackety hack hack for great perf enhancements
    5. go test -cpuprofile cpu.out.after

    Then, marvel at your magnificence.  You can also use -memprofile
    and/or -trace to  better optimize other things.

    Other perf testy things will also appear here.

M   .gitignore
M   cmds/license/license.go
A   cmds/license/machinePerf.go
A   perf_tests/machineCreate_test.go
A   perf_tests/main_test.go
M   tools/test.sh

End of Note