commit e163e108792cd59d511d66b7ff7c136787735e5a
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Wed Nov 3 14:30:13 2021 -0500
fix(transact): Antagonize the perf tests even more.
Add content pack swapping as part of the job create and fail stress
test, and fix a bug on how and when we mark jobs as not current that
exposes that can cause occasional spurious job create failures.
M backend/jobs.go
M frontend/job_create.go
M perf_tests/machineCreate_test.go
commit 243873fc1792552741bac7d1928d17790945cb1b
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Thu Oct 28 20:01:11 2021 -0500
feat(transact): Refactor Transactions and fix resource leaks
The API presented to the rest of dr-provision made coding for access
to the transactions directly unweildy, and hid a couple of resource
leaks that only show up under significant contention.
1. Refactor the transact package to shift around how its functionality
is exposed. In particular, the transaction IDs themselves are hidden
while the transactions themselves are exposed, and have methods that
handle the heavy lifting. This also adds the methods to panic with an
appropriate TxError if lock acquisition fails, since that is the most
common use of this package by far. Everything else that uses
transactions has been refactored to reflect these changes.
2. It is possible under very high contention mixed with cancellations
for a goroutine handling a transaction to exit without releasing the
interlock tracker mutex. Mutex management in the transaction
interlock tracker has been refactored to use the with() pattern we use
elsewhere in the codebase, and all functions that need to use the
mutex have been refactored to account for these changes. This
resulted in splitting transact.Tx.waitOn into two smaller functions.
3. Cancelled transactions that were waiting to acquire a write lock on
something were not properly unblocking the key they were waiting on.
The transaction release codepaths have been refactored to account for
this situation and properly release the write blocks.
4. The perf tests now batch up API requests to force maximum parallelism
and place more stress on the anti-starvation code and the transact
code.
Do not run the perf test until you have a suitably high ulimit -n. I
recommend setting it to at least 100,000 open files.
M backend/dataTracker.go
M backend/dhcpUtils.go
M backend/endpoint_manager.go
M backend/jobs.go
M backend/pluginProviders.go
M backend/profiles_test.go
M backend/requestTracker.go
M backend/subnet.go
A cmod/contentChange.go
M datastack/stack.go
M midlayer/dhcp.go
M perf_tests/machineCreate_test.go
M test/serverGroup.go
A transact/interlockTracker.go
A transact/simpleKey.go
M transact/tx.go
M transact/tx_test.go
commit 8c1104d058b4b3375c3490716106ed61257cdedd
Author: Zaheena <zaheena@gmail.com>
Date: Tue Nov 2 13:12:34 2021 -0400
fix(server): one less place for AllLocked. this was causing hung goroutines
M server/args.go
commit 91274b46dc310f6905912155a4438a886f49f9ab
Author: Victor Lowther <victor.lowther@gmail.com>
Date: Tue Oct 26 09:05:47 2021 -0500
fix(mgrtests): Add missed updates to manager tests.
All that prefs listing needed to have the purgeIdleDelay pref added.
M manager-tests/test-data/output/TestAddMachines/client1.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestAddMachines/client2.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestAddMachines/manager1.prefs.list.2/stdout.expect
M manager-tests/test-data/output/TestAddMachines/manager1.prefs.list/stdout.expect
M manager-tests/test-data/output/TestAddMachines/manager1.prefs.set.manager.true.2/stdout.expect
M manager-tests/test-data/output/TestAddMachines/manager1.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestAddMachines/sub2-client1.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestAddMachines/sub2-client2.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestAddMachines/sub3-client1.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestAddMachines/sub3-client2.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestAddMachines/sub4-client1.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestAddMachines/sub4-client2.prefs.set.manager.true/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/client1.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/client1.prefs.list/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/client2.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/client2.prefs.list/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/manager1.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/manager1.prefs.list/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/sub2-client1.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/sub2-client2.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/sub3-client1.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/sub3-client2.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/sub4-client1.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestManagerFunctions/sub4-client2.prefs.list.e5b1e4bcad9ef0a7b91360e3033c60b9/stdout.expect
M manager-tests/test-data/output/TestSetupServers/client1.prefs.set.baseTokenSecret.01234567890123456789012345621091.systemGrantorSecret.01234567890123456789012345621091/stdout.expect
M manager-tests/test-data/output/TestSetupServers/client2.prefs.set.baseTokenSecret.01234567890123456789012345622091.systemGrantorSecret.01234567890123456789012345622091/stdout.expect
M manager-tests/test-data/output/TestSetupServers/manager1.prefs.set.baseTokenSecret.01234567890123456789012345611091.systemGrantorSecret.01234567890123456789012345611091/stdout.expect
M manager-tests/test-data/output/TestSetupServers/sub2-client1.prefs.set.baseTokenSecret.01234567890123456789012345631091.systemGrantorSecret.01234567890123456789012345631091/stdout.expect
M manager-tests/test-data/output/TestSetupServers/sub2-client2.prefs.set.baseTokenSecret.01234567890123456789012345632091.systemGrantorSecret.01234567890123456789012345632091/stdout.expect
M manager-tests/test-data/output/TestSetupServers/sub3-client1.prefs.set.baseTokenSecret.01234567890123456789012345641091.systemGrantorSecret.01234567890123456789012345641091/stdout.expect
M manager-tests/test-data/output/TestSetupServers/sub3-client2.prefs.set.baseTokenSecret.01234567890123456789012345642091.systemGrantorSecret.01234567890123456789012345642091/stdout.expect
M manager-tests/test-data/output/TestSetupServers/sub4-client1.prefs.set.baseTokenSecret.01234567890123456789012345651091.systemGrantorSecret.01234567890123456789012345651091/stdout.expect
M manager-tests/test-data/output/TestSetupServers/sub4-client2.prefs.set.baseTokenSecret.01234567890123456789012345652091.systemGrantorSecret.01234567890123456789012345652091/stdout.expect
End of Note
