Commit Graph

3764 Commits

Author SHA1 Message Date
Jingyu Zhou
3f97fa046b Fix ASAN build errors (#11767)
* Fix ASAN build errors

This fixes https://github.com/apple/foundationdb/issues/10813

There are multiple problems:
- The boost 1.78 doesn't work well with ASAN build, which was fixed in later versions
- The compiling missed "context-impl=ucontext" flag for b2 command
- missing /usr/local/include/c++/v1/__config_site in the developer environment

* Use B2_ADDTTIONAL_BUILD_ARGS instead
2024-11-09 09:38:15 -08:00
Syed Paymaan Raza
aaba814512 Fix two cases of non-determinism in simulation (#11766) 2024-11-08 14:38:15 -08:00
Syed Paymaan Raza
7fd213739b Urgent consistency checker fixes (#11734)
* Drop duplicate or conflicted requests from urgent consistency checker clients

* Fix edge case in urgent consistency check causing infinite loop

* fixup! Fix edge case in urgent consistency check causing infinite loop
2024-10-25 22:37:42 -07:00
Zhe Wang
43446204ed Database Per-Range Lock (#11693)
* range lock framework

* improve the framework

* persist to txnStateStore

* fix bugs

* code clean

* code clean

* bug fix

* address comments

* add complex test workload and fix bugs found by the workload

* add workload correctness check and fix bugs

* code clean up

* add random range lock injection

* fix bugs in RandomRangeLock.actor.cpp

* enable random range lock injection in general workloads

* add rangelockcycle test

* disable random range lock in backup workloads

* nits

* add range lock ownership concept

* enable lock ownership to rangeLock

* api deal with tenant

* fix CI

* add test for multiple rangeLock owners

* nits

* address comments and renaming

* address comments
2024-10-23 16:25:56 -07:00
Vishesh Yadav
5215eb61bd Merge pull request #11718 from vishesh/dummythreadpool-init-fix 2024-10-21 23:46:53 +05:30
Vishesh Yadav
809c3a86a7 Call IThreadReceiver::init() in DummyThreadPool 2024-10-16 10:26:53 +05:30
Vishesh Yadav
42f5e84306 Log all incoming connections 2024-10-09 11:09:50 -07:00
Syed Paymaan Raza
1e626be639 [fdbserver] Fix some issues reported by clangd 2024-09-19 23:27:05 -07:00
Sepeth
3854dbfe4d Upgrade fmt from 8.1.1 to 11.0.2 (#11601)
And, added via cmake FetchContent, and removed contrib/fmt-8.1.1
2024-09-10 14:42:43 -07:00
hao fu
f092e19026 address comments 2024-09-05 16:07:01 -07:00
Vishesh Yadav
a84026cca5 [testing] Automatically discover unit-test and register as ctest (#11612)
* [testing] Automatically discover unit-test and register as ctest

This patch adds `collect_unit_tests()` to CMake which searches over
the codebase and finds all the unit-tests written using Flow's TEST_CASE
macro and register as ctest.

The test then can be then run using ctest command or directly via Test
Explorer in VSCode.

* Update CMakeLists.txt

* Check failed tests

* Update TestDirectory.py to create more unique directory

* Put the feature behind flag
2024-09-04 11:31:49 -07:00
Syed Paymaan Raza
48064f6cf1 Make some codeprobes rare (#11607)
* Make BlobGranule code probes rare

* Make encryption related code probes rare

* fixup! Fix formatting
2024-08-26 22:33:38 -07:00
Xiaoge Su
a3850c3589 Fix the misuse of BIO_get_mem_data
BIO_get_mem_data will return the length of the string, however, the
string is not zero-terminated. The length value determines where the
string terminates.
2024-08-06 05:08:14 +08:00
Syed Paymaan Raza
392bad2bd3 More copyright end year updates (#11556) 2024-08-05 14:00:32 -07:00
Syed Paymaan Raza
c3e7542cda Update end year in copyright header 2024-08-02 09:40:11 -07:00
Zhe Wang
74990e44bd Bulk Loading Framework (#11369) 2024-07-23 14:57:28 -07:00
Xiaoge Su
92ad7666c3 fixup! Reformat source 2024-07-23 09:52:49 -07:00
Xiaoge Su
9f15c57b26 Add peer address when TLS policy failed
See #5854
2024-07-23 09:52:49 -07:00
Xiaoge Su
e683076ff5 fixup! Reformat source 2024-07-23 09:52:49 -07:00
Xiaoge Su
bb29465713 fixup! Remove unused is_client flag 2024-07-23 09:52:49 -07:00
Xiaoge Su
390e88c0ae fixup! Address issues per comment
Also try to introduce [[unlikely]] to ASSERT, this is related to issue #11422
2024-07-23 09:52:49 -07:00
Xiaoge Su
4b70993d89 Better TraceEvent output for TLS failures
This patch adds more output about TLS failures, e.g.

<Event Severity="20" Time="1716265824.713579" DateTime="2024-05-21T04:30:24Z" Type="TLSPolicyFailure" ID="0000000000000000" SuppressedEventCount="0" Reason="Rule.Cert.Issuer" Rule="Rule{ verify_cert=1, verify_time=1, Subject=[ ], Issuer=[ ], Root=[ ] }" ThreadID="7547317051334743152" LogGroup="default" />

The failure data will include the rule, the reason of failure and the
value of corresponding fields.
2024-07-23 09:52:49 -07:00
Yao Xiao
c630fa2296 Fix wait (#11474) 2024-07-18 11:28:34 -07:00
Xiaoge Su
cf70d45e6d Add PeerAddress to all PeerAddr/Peer TraceEvent
This is to address #4846
2024-07-09 16:49:22 -07:00
Sreenath Bodagala
d7eb028b2a Enable replica consistency check on data movement (#11415)
* - Enable replica consistency check on data movement (and, randomly, on
all reads)

* - Address PR review comments
2024-06-17 17:07:32 -04:00
Xiaoge Su
20fddc8eb4 fixup! Reformat source 2024-06-17 11:41:06 -07:00
Xiaoge Su
afc04366fb Rewrite BUGGIFY related code
This is a rewrite of BUGGIFY function/macros. Seems the performance
improved a lot during the simulation, e.g.

fdbserver -r simulation -b on -f ../CycleTest.toml -s 99438

Without this patch:

Unseed: 54646
Elapsed: 494.091327 simsec, 14.586831 real seconds

With this patch:

Unseed: 54646
Elapsed: 494.091327 simsec, 12.580612 real seconds

I expected the improvement but did not expect a ~13% improvement.
2024-06-17 11:41:06 -07:00
dependabot[bot]
a3c1f44c7f Bump jinja2 from 3.1.3 to 3.1.4 in /flow/protocolversion
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.3...3.1.4)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 21:00:43 +00:00
Sreenath Bodagala
df2b7b4fe8 - Address PR review comments 2024-05-01 19:41:00 +00:00
Sreenath Bodagala
d6f6b45125 - Handle errors thrown during replica consistency check 2024-04-30 21:37:50 +00:00
Sreenath Bodagala
bd68263558 - Disabe replica consistency check related knob 2024-04-22 21:53:28 +00:00
Sreenath Bodagala
a4430b9169 Compare storage replicas on reads (#11235)
* - Compare storage replicas on reads (in "loadBalance()")

* - Do consistency check on reads in loadbalance

* - Do replica consistency check in the case where loadBalance issues
requests to multiple storage servers

* - Address a state variable related bug

* - Code formatting

* - API simplification

* - Simplify code

* - Code formatting

* - Address a review comment
2024-04-11 16:08:54 -04:00
Zhe Wang
308ff77e91 Consistency Check Urgent (Cherrypick from Release-7.1) (#11217)
* cherry-pick-distributed-consistency-checker

* code cleanup

* refactor code, decouple consistencyCheckerUrgent and consistency checker

* fix workload for consistencycheckurgent

* add new consistencycheckurgent role type

* fix CI

* address comments
2024-02-28 14:22:47 -08:00
Hao Fu
8555ac9b71 Implement checksum via LRU-like approach to save space (#11194) 2024-02-21 12:24:51 +08:00
Johannes M. Scheuermann
0370cc08e1 Add knob to allow fdbserver to abort under abnormal behaviour 2024-02-14 10:15:14 +01:00
Johannes Scheuermann
88905b1f16 Update link to abort function 2024-02-14 10:15:08 +01:00
Johannes M. Scheuermann
0f8a9cde4e Abort process when abnormal shutdown is initiated to allow coredumps to be generated 2024-02-14 10:15:01 +01:00
Boris Korzun
f7a443aea4 Add libfmt 10+ support (#11140) 2024-02-13 23:09:58 +03:00
He Liu
9d8d52cbb7 Added checksum in MutationRef (#11181)
* Append checksum to param2.

* Pass sim tests w/o validating checksums.

* Code cleanup.

* Renew checksum.

* Remove checksum for all private mutations.

* Added checksum validation at SS.

* Fixed VERSION_TIMESTAMP.

* Disable Mutation Checksum by default.

* Cleanup.

* cleanup.
2024-02-09 13:36:41 -08:00
Dimitris Apostolou
a88114c222 Fix typos 2024-02-07 01:16:00 +02:00
Johannes M. Scheuermann
b2b3a8e791 Fix complie issue for ALLOC_INSTRUMENTATION 2024-02-01 14:48:04 +01:00
dependabot[bot]
d08d0fc549 Bump jinja2 from 3.1.2 to 3.1.3 in /flow/protocolversion
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.2...3.1.3)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-30 17:03:41 +00:00
mavenraven
2bfff79fd3 Adds support for running protocol_version.py in an virtual env, similar to Sphinx. 2024-01-27 17:19:33 -05:00
Yao Xiao
2329e8327a Add log cleaner for rocksdb logs. (#11134)
Co-authored-by: yaoxiao-github <yaoxiao@Yaos-MacBook-Pro-14.local>
2024-01-17 14:51:15 -08:00
Markus Pilman
a2bb520847 Fix cancellation bug in Choose class 2023-12-08 17:46:21 +01:00
Markus Pilman
fd4a300d4c Address review comments and fix serious bug 2023-11-17 11:08:34 +01:00
Markus Pilman
afce7b35bf add concatenateStrings and remove onWorkerThread from mock network 2023-11-14 10:57:38 +01:00
Markus Pilman
f85ed2ac14 Removed ranges stuff to support old compilers. Added new _swiftEnqueue to test network impl 2023-11-14 10:22:13 +01:00
Markus Pilman
e07b3e35ca Added C++ Coroutine support to Flow 2023-11-14 10:10:11 +01:00
Dan Lambright
015167c17e Throttle commits against hot shards (#10970)
* throttle hot shards

* expire throttled shards over time

* add backoff

* Parallelize messaging from RK to CP

* Obtain shards from a single SS

* handle expired transactions

* bump transaction_throttled_hot_shard

* Change SevError to SevWarn for CannotMonitorHotShardForSS

* Add log per request
2023-10-31 12:01:34 -04:00