Commit Graph

121 Commits

Author SHA1 Message Date
Zhe Wang
43446204ed Database Per-Range Lock (#11693)
* range lock framework

* improve the framework

* persist to txnStateStore

* fix bugs

* code clean

* code clean

* bug fix

* address comments

* add complex test workload and fix bugs found by the workload

* add workload correctness check and fix bugs

* code clean up

* add random range lock injection

* fix bugs in RandomRangeLock.actor.cpp

* enable random range lock injection in general workloads

* add rangelockcycle test

* disable random range lock in backup workloads

* nits

* add range lock ownership concept

* enable lock ownership to rangeLock

* api deal with tenant

* fix CI

* add test for multiple rangeLock owners

* nits

* address comments and renaming

* address comments
2024-10-23 16:25:56 -07:00
Syed Paymaan Raza
48064f6cf1 Make some codeprobes rare (#11607)
* Make BlobGranule code probes rare

* Make encryption related code probes rare

* fixup! Fix formatting
2024-08-26 22:33:38 -07:00
Jingyu Zhou
5d2deddb7d Reduce the chance to run some rare tests
E.g., StatusBuilderPerf and TLogVersionMessagesOverheadFactor are more like
performance tests, which shouldn't be running so many times.

Without the change, a 100k-run has this many for these tests:

   1318 tests/rare/CycleWithKills.toml
   1591 tests/rare/TLogVersionMessagesOverheadFactor.toml
   1647 tests/rare/ConfigDBUnitTest.toml
   1839 tests/rare/StatusBuilderPerf.toml

After the change, a 100k-run has:

    129 tests/rare/TLogVersionMessagesOverheadFactor.toml
    151 tests/rare/CycleWithKills.toml
    160 tests/rare/StatusBuilderPerf.toml
    375 tests/rare/ConfigDBUnitTest.toml
2024-08-02 17:24:30 -07:00
Syed Paymaan Raza
c3e7542cda Update end year in copyright header 2024-08-02 09:40:11 -07:00
Jingyu Zhou
995eec702b Fix ApplyMutationsError test failure due to hot shard throttling (#11551)
In the restore test, there are hot ranges with writes to customized range with
prefix "BeforeRestart" or "AfterRestart". As a result, FDB can return
transaction_throttled_hot_shard errors and cause test failure.
2024-08-01 22:28:22 -07:00
Sreenath Bodagala
d6f6b45125 - Handle errors thrown during replica consistency check 2024-04-30 21:37:50 +00:00
Sreenath Bodagala
a4430b9169 Compare storage replicas on reads (#11235)
* - Compare storage replicas on reads (in "loadBalance()")

* - Do consistency check on reads in loadbalance

* - Do replica consistency check in the case where loadBalance issues
requests to multiple storage servers

* - Address a state variable related bug

* - Code formatting

* - API simplification

* - Simplify code

* - Code formatting

* - Address a review comment
2024-04-11 16:08:54 -04:00
Dimitris Apostolou
a88114c222 Fix typos 2024-02-07 01:16:00 +02:00
Jingyu Zhou
ae6ef0ce90 Add different block size for tests 2023-07-20 21:28:58 -07:00
Jingyu Zhou
7d6481f6f9 Add a test case for getLogKey()
With input that exposes the bug.
2023-07-17 20:33:41 -07:00
Jingyu Zhou
944e45e630 Fix a bug of hash value for backup log keys
Need to be consistent with getLogRanges().
2023-07-17 14:31:16 -07:00
Ata E Husain Bohra
7779c908b3 EaR: Remove usage of ENABLE_CONFIGURABLE_ENCRYPTION knob (#10570)
Description

Given Configurable encryption has been checked in and being tested via
simulation for more than a month and also to avoid penalty of accessing
KNOBS in inline commit path, patch retires the KNOB and make
ConfigurationEncryption default EaR mode for FDB.

BlobCipher still supports the old format header and encryption semantics,
will remove the dead code as a followup PR.

Testing

devRunCorrectness - 100K
2023-06-30 17:48:09 -07:00
Hui Liu
788f3420c8 Fix tenant map update race when applying mutations (#10557) 2023-06-27 10:31:58 -07:00
A.J. Beamon
1af1346cde Change a buggify and decrease the number of transactions per second so that the dr upgrade test runs faster 2023-06-23 09:03:34 -07:00
Josh Slocum
fb950a9c81 adding blob ranges to backup keys to not lose blobbification on restore (#10059) 2023-05-04 13:55:20 -05:00
Nim Wijetunga
021bdccc32 propogate encryption errors properly (#10012)
propogate encryption errors properly
2023-04-19 11:35:29 -07:00
Nim Wijetunga
6e4e6ab2f4 Revert "Revert "Refactor GetEncryptCipherKeys (#9600)"" (#9903)
* Revert "Revert "Refactor GetEncryptCipherKeys (#9600)" (#9708)"
2023-04-05 10:03:48 -07:00
A.J. Beamon
807646675c Refactor the metacluster project into smaller files, and reorganize the namespaces. Move some metacluster and tenant testing helpers into the metacluster project. 2023-03-30 16:20:09 -07:00
A.J. Beamon
e61748c7d5 Move metacluster into its own directory and static library 2023-03-30 16:07:49 -07:00
A.J. Beamon
64b6a5d257 Allow boolean parameters to be nested inside of namespaces or classes 2023-03-30 15:09:59 -07:00
Ata E Husain Bohra
dbcab0b1bd Revert "Refactor GetEncryptCipherKeys (#9600)" (#9708)
This reverts commit 2702665e35.
2023-03-15 12:10:08 -07:00
Nim Wijetunga
2702665e35 Refactor GetEncryptCipherKeys (#9600)
* inital commit

* address pr comments
2023-03-08 17:05:03 -08:00
Nim Wijetunga
29819b0645 Change Feed Bug Fix + Encryption Asserts (#9457)
* add encryption asserts

* modify function name

* address pr comments

* address pr comments

* Trigger Build
2023-02-23 19:33:25 -08:00
Ata E Husain Bohra
99b23ac04d EaR: Configurable encryption support for Tlog mutations (#9394)
* EaR: Configurable encryption support for TLog mutations

Description

  diff-1 : Address review comments

Major changes includes:
1. Update the code involved in ensuring Tlog mutation encryption to be
compliant with "configurable encryption" feature.
2. Update ENABLE_CONFIGURABLE_ENCRYPTION flag to be 'true' by default
and BUGGIFY it.

Testing

devRunCorrectness - 100K
2023-02-16 19:01:59 -08:00
Nim Wijetunga
bf85c9f8af Backup Mutation Log Separates Tenant Map Modifications During Restore (#9292)
mutation log separates tenant map modifications
2023-02-14 16:46:09 -08:00
Nim Wijetunga
86f3665514 Handle EKP Tenant Not Found Errors (#9261)
handle EKP tenant not found errors
2023-02-01 19:15:38 -08:00
Nim Wijetunga
05a8a90830 Snapshot Backup Tenant Deletion Support (#9145)
Tenant deletion support for snapshot backups
2023-01-26 17:46:14 -08:00
A.J. Beamon
fd13bc04c8 Update the tenant maps to be keyed by ID 2023-01-23 14:09:12 -08:00
Nim Wijetunga
330ac71630 Tenant Deletion Support for Backup Mutation Log (#9103)
tenant deletion support for backup mutation log
2023-01-18 15:11:58 -08:00
A.J. Beamon
3353103d9d Fix filtering of potential backup mutations in commit proxy and backup worker; add code probe to ensure we are testing default backup sharing and add some tests to hit it 2022-09-27 15:25:30 -07:00
A.J. Beamon
a7c200b16c Add support for sharing mutation streams between a default backup and DR 2022-09-22 11:36:46 -07:00
A.J. Beamon
fda0d7223d Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants. 2022-09-22 10:00:13 -07:00
Nim Wijetunga
eadb769cfa Encrypt Backup Mutation Log (#8159)
* encrypt backup mutation log

* format

* address pr comments

* format

* fix bug

* revert knobs

* address pr comments
2022-09-20 15:43:39 -07:00
A.J. Beamon
4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
sfc-gh-tclinkenbeard
82adc1e856 Make g_simulator a pointer 2022-09-15 09:00:33 -07:00
Markus Pilman
bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Markus Pilman
bed799220a Addressed review comments, added test 2022-03-15 16:57:26 +01:00
Markus Pilman
dc973fb67e Allow List and first test 2022-02-22 11:15:16 +01:00
A.J. Beamon
f24adc7b6a Fix a bunch of places where we used old-style arguments. Allow hyphens for profiler args. 2021-12-14 09:59:14 -08:00
Renxuan Wang
6e89dc3da0 Check in the prototype of MutationLogReader. 2021-08-18 14:59:50 -07:00
Steve Atherton
2bf7b9b7a9 Remove many calls to .toString(), .printable(), and printable(...) from TraceEvent .detail() values to avoid double-escaping characters such that the same values look different in different events and can't be easily searched for. Also fixed a few warnings found by clang. 2021-08-08 00:04:54 -07:00
Steve Atherton
f596a81073 Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs. 2021-07-17 00:11:40 -07:00
Oleg Samarin
47f13ad1b0 Formatting status if the log version has not been set
Update fdbclient/DatabaseBackupAgent.actor.cpp

Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
Provide log version information to fdbdr status
2021-07-10 20:54:10 +03:00
sfc-gh-tclinkenbeard
382533c4e1 Add FDB_ prefix to BOOLEAN_PARAM macros 2021-07-09 05:42:14 -07:00
sfc-gh-tclinkenbeard
020371a78f Merge remote-tracking branch 'origin/master' into add-boolean-param 2021-07-07 16:50:51 -07:00
sfc-gh-tclinkenbeard
8b2d8448d7 Move BackupAgentBase method implementations to .cpp file 2021-07-04 19:09:03 -07:00
sfc-gh-tclinkenbeard
8370d92c20 Expand use of BOOLEAN_PARAM in backup code 2021-07-04 18:30:55 -07:00
sfc-gh-tclinkenbeard
cadfdc551e Use BOOLEAN_PARAM for backup agents 2021-07-04 16:27:26 -07:00
sfc-gh-tclinkenbeard
49c680b252 Use BOOLEAN_PARAM for TaskBucket 2021-07-04 13:14:25 -07:00