Commit Graph

588 Commits

Author SHA1 Message Date
Jingyu Zhou
fac3496c39 Add "checkall" debug command for fdbcli (#10687) (#10691)
* Add "checkall" for checking \xff\x02/blog/ keys

* Avoid GRV calls for getlocation

* Update comments

* Add non-stopping checking and remove verbose output

* Update checkall command to accept customized range

* Fix format

* Fix a compiling issue and output
2023-07-27 13:43:23 -07:00
Jingyu Zhou
bfbb084b10 Add getlocation and getall fdbcli debug commands
getlocation: returns the SS list for a key
getall: returns both the SS list and values on the SS for a key
2023-06-30 13:24:56 -07:00
Zhe Wang
aa47c0c722 [Release 7.3] Cherry-pick Add audit storage cancellation (#10386) (#10430)
* Add audit storage cancellation (#10386)

* list audits

* cancel audits and corresponding tests

* make audit storage dblock aware

* increase audit retry since we are able to cancel

* fix updateAuditState and fdb github ci

* fmt

* fix fdbcli audit_storage and fix CI issue

* fix fdb cli

* address comments

* fmt

* Fix audit storage actor cancel issue (#10443)

* init

* add testAuditStorageConcurrentRunForSameType test

* init (#10458)
2023-06-09 10:43:54 -07:00
He Liu
32e8ba7474 Added location_metadata fdbcli to query shard locations, assignements… (#10395) (#10428)
* Added location_metadata fdbcli to query shard locations, assignements, numbers etc.

* Added `listshards` to get some random physical/non-physical shards.

* Resolved comments.
2023-06-07 15:08:19 -07:00
Steve Atherton
d069d94cf7 Added explanation of \u support to fdbcli token parsing. Small tweak to rangeconfig hints. Reformatted rangeconfig help to not be intended because the help printer does its own line wrapping which makes it look very messy. 2023-05-03 15:52:51 -07:00
Steve Atherton
44c8a91f7d Move custom shard test setup to a separate function. Add JSON utf-8 escaped bytes to fdbcli token parsing. 2023-05-03 15:52:40 -07:00
Steve Atherton
5b34628a2b Add rangeconfig CLI. 2023-05-03 15:27:58 -07:00
Steve Atherton
8d35aa113d Changed KeyBackedTypes to an actor file. Added TypedKeySelectors for Map and Set classes and getRange() keySelector methods. Added debug macro for KeyBackedTypes. Rewrote KeyBackedRangeMap using keyselectors on KeyBackedMap. 2023-05-03 15:27:57 -07:00
Chaoguang Lin
7273723a43 Add the hotrange fdbcli command (#9570)
* Add the hotrange fdbcli command

* Remove the unnecessary state

* Add the doc about the hotrange command
2023-03-06 14:46:52 -08:00
Ankita Kejriwal
27f9c1e06e Merge branch 'main' of github.com:apple/foundationdb into fdbcli 2023-01-27 15:03:27 -08:00
He Liu
00203c8732 Validate Storage part II (#8471)
* Implemented AuditUtils.actor.cpp

Moved AuditUtils to fdbserver/

* Persist AuditStorageState.

* Passed persisted AuditStorageState test.

* Added audit_storage_error to indicate a corruption is caught.

Throw/Send audit_storage_error when there is a data corruption.

Added doAuditStorage() for resuming Audit.

* Load and resume AuditStorage when DD restarts.

* Generate audit id monotonically.

* Fixed minor issue AuditId/Type was not set.

* Adding getLatestAuditStates.

* Improved persisted errors and added AuditStorageCommand.actor.cpp for
fdbcli.

* Added `audit_storage` fdbcli command.

* fmt.

* Fixed null shared_ptr issue.

* Improve audit data.

* Change DDAuditFailed to SevWarn.

* Sev.

* set SERVE_AUDIT_STORAGE_PARALLELISM to 1.

* Moved AuditUtils* to fdbclient/.

* Added getAuditStatus fdbcli command.

* Refactor audit storage fdb cli commands.

* Added auditStorage in sim.

* Cleanup.

* Resolved comments.

* Resolved comments.

* Test disabling audit for sims.

* Cleanup.

Co-authored-by: He Liu <heliu@apple.com>
2023-01-15 21:46:14 -08:00
Ankita Kejriwal
f8cfe69488 Fix formatting in fdbcli.actor.cpp 2023-01-06 17:16:29 -08:00
Ankita Kejriwal
db520d423a Add fdbcli commands for checking the status of and clearing old idempotency ids 2022-12-22 23:56:55 -08:00
Ankita Kejriwal
fe06114bcd Update the help text for quota command 2022-12-14 11:01:33 -08:00
FoundationDB CI
86d6106dc1 format source code after switch to clang 15 2022-12-08 17:26:45 +00:00
Daniel Luan
4be05e5a5d Upgrade C++ Standard to 20 2022-12-06 14:19:06 -08:00
Hui Liu
5834517570 Add fdbcli blobrestore to start the full restore 2022-11-11 08:32:23 -08:00
Kevin Hoxha
2dac017ea2 fdbcli: Cancel checkStatus actor when prompting for description in setknob 2022-10-20 21:31:52 -07:00
sfc-gh-tclinkenbeard
9a1876dd11 Update comment for fdbcli quota command 2022-10-14 12:02:14 -07:00
sfc-gh-tclinkenbeard
3827b1ee62 Merge remote-tracking branch 'origin/main' into add-quota-clear-command 2022-10-14 11:55:45 -07:00
Kevin Hoxha
ff1b2df8f6 fdbcli: Add options for knob management
- setknob <knob_name> <knob_value> [config_class]
- getknob <knob_name> [config_class]
- Added new option to begin to specify if it's a configuration txn. Syntax is begin [config-txn]
- Added utility function for converting tuples to string
- Added knobmanagment test in fdbcli_tests.py
2022-10-11 15:32:01 -07:00
sfc-gh-tclinkenbeard
d67f3ca3b9 Update documentation for quota clear command 2022-10-09 19:43:44 -07:00
Trevor Clinkenbeard
ef13985feb Merge pull request #8320 from sfc-gh-tclinkenbeard/read-write-fungible
Make read and write quotas fungible
2022-09-28 10:59:37 -07:00
Junhyun Shim
fd621c0c86 Make TLSConfig check against coordinator addresses one-way
TLS parameters may be loaded by default system paths, which would break
non-TLS cases
2022-09-27 20:57:38 +02:00
sfc-gh-tclinkenbeard
7fc5c196c4 Make read and write quotas fungible 2022-09-25 21:00:11 -07:00
Josh Slocum
28f3cacfd7 Merge pull request #8285 from sfc-gh-jslocum/bg_history_fdbcli
Adding blobkey history fdbcli debug command
2022-09-22 16:56:59 -05:00
Josh Slocum
36df430b14 Merge branch 'main' into bg_history_fdbcli 2022-09-22 15:49:07 -05:00
A.J. Beamon
97a325adab Add an fdbcli command to get tenant group metadata 2022-09-22 13:24:21 -07:00
Josh Slocum
21c627ba97 Adding blobkey history fdbcli debug command 2022-09-22 13:03:53 -05:00
A.J. Beamon
64c8e0e2b1 Convert fdbcli tenant commands to a single command with subcommands. Update the structure of fdbcli tenant tests. 2022-09-21 16:22:58 -07:00
A.J. Beamon
4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
sfc-gh-ngoyal
1bd97fe628 Recruit new singleton for consistency checker. (#5804)
* Recruit new singleton for consistency checker.

* Recruit the consistency checker only if enabled.

* Add a yield in monitorConsistencyChecker().

* Minor fixes.

* Consistency check workload enhancements.

* Minor fixes and clarifications.

* clang format

* Clang format.

* Minor fixes, cleanup, debug tracing.

* Misc.

* Move the consistency scan information from dbconfig to a key backed object.

* Move consistency scan config out of db cofig to a state object and feature rename.

* ConsistencyCheck workload refactor.

* devFormat

* Update fdbcli/ConsistencyScanCommand.actor.cpp

* Review Comments.

Co-authored-by: negoyal <neelam.goyal@gmail.com>
Co-authored-by: Ata E Husain Bohra <ata.husain@snowflake.com>
2022-09-16 09:03:06 -07:00
Dennis Zhou
e6222be6c6 fdbcli: use ApiVersion::LATEST_VERSION instead of hardcoding 2022-09-12 17:08:57 -07:00
A.J. Beamon
726d5215a0 Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (#8108)
* Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (no need to guard)

* Enable the relaxed special key access in transactions that need to use special key-space APIs introduced in 7.2
2022-09-08 17:22:36 +02:00
Junhyun Shim
bcf0861a0a Make fdbcli check for TLSConfig and coordinator address mismatch
At least one of the coordinator addresses in cluster file must contain ":tls" suffix
if fdbcli's resolved TLS client configuration holds any of the TLS elements (key, cert, or CA)

Conversely, if none of the TLS elements are configured,
at least one of coordinator addresses must be without ":tls" suffix
2022-09-01 15:50:11 +02:00
Chaoguang Lin
3fed0456ca Add an verify option for \xff\xff/worker_interfaces special keys (#7873)
* Add the verify option for \xff\xff/worker_interfaces

* Remove unused code

* update documentations

* update documentations

* solve comments from review

* update some of the comments to be more clear
2022-08-15 14:05:07 -04:00
Vaidas Gasiunas
79571dd2b4 Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
A.J. Beamon
7f53b340f5 Merge branch 'main' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/ClientKnobs.cpp
#	fdbclient/include/fdbclient/ClientKnobs.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbserver/BlobManager.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
#	flow/include/flow/error_definitions.h
2022-07-28 11:43:21 -07:00
A.J. Beamon
7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
Jon Fu
6dbaf46ec8 restructure test workload impl and map 2022-07-26 13:53:04 -07:00
A.J. Beamon
f2a8af9155 Better API version handling for fdbcli tenant commands 2022-07-26 09:04:29 -07:00
A.J. Beamon
a64693518a Add support for tenant groups 2022-07-26 09:04:29 -07:00
A.J. Beamon
279296c29f Merge branch 'tenant-metadata-change' into feature-metacluster
# Conflicts:
#	fdbclient/SystemData.cpp
#	fdbclient/Tenant.cpp
#	fdbclient/include/fdbclient/SystemData.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-20 09:18:27 -07:00
Markus Pilman
1de37afd52 Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
A.J. Beamon
860d3843cc Merge remote-tracking branch 'origin/feature-tenant-groups' into feature-metacluster 2022-07-16 19:33:26 -07:00
A.J. Beamon
c08592368f Add support for tenant groups 2022-07-15 21:18:41 -07:00
A.J. Beamon
0fe6f6ad36 Merge remote-tracking branch 'origin/update-prefix-encoding-in-tenant-metadata-json' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbclient/include/fdbclient/TenantSpecialKeys.actor.h
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-14 14:32:29 -07:00
A.J. Beamon
cb499fa5db Merge branch 'main' into feature-metacluster 2022-07-13 15:28:34 -07:00
A.J. Beamon
91ccdbcb03 Use a more standard encoding in the JSON metadata reported for tenants. 2022-07-13 15:22:19 -07:00
A.J. Beamon
a3f970cf3a Parse integer options in fdbcli rather than passing the bytes unparsed. This allows specifing the string "1" instead of "\x01\x00\x00\x00\x00\x00\x00\x00". 2022-07-11 16:39:59 -07:00