345 Commits

Author SHA1 Message Date
Matthew Newhook
681dada0c9 Review feedback. 2024-04-30 14:29:54 +00:00
Matthew Newhook
5d3e226ac6 Hex decode add and remove prefixes for fdbrestore. 2024-04-27 13:56:30 +00:00
Johannes Scheuermann
d289caabe3 Correct proxy handling backup agent 7.3 (#10911)
* Correct the way how the backup agent makes use of the proxy and add proxy to trace events

* Correct format
2023-09-20 15:22:11 -07:00
Hao Fu
23390679c0 Add proxy to backup agent via global var (#10884)
backup agent itself does not have proxy info.
This changes adds the proxy via a global var.
2023-09-13 11:12:37 -07:00
Hui Liu
c04ef6465c Add TLS command line options to fdbbackup modify 2023-05-16 17:42:29 -07:00
Steve Atherton
8d35aa113d Changed KeyBackedTypes to an actor file. Added TypedKeySelectors for Map and Set classes and getRange() keySelector methods. Added debug macro for KeyBackedTypes. Rewrote KeyBackedRangeMap using keyselectors on KeyBackedMap. 2023-05-03 15:27:57 -07:00
Steve Atherton
216d0be2cf Add processID, networkAddress, and locality to layer status JSON for Backup Agents. (#9736)
* Add processID, networkAddress, and locality to layer status JSON for Backup Agents.

* Backup/dr agent determines network address to report in Layer Status only once, when the status updater loop begins, since it is a blocking call which connects to the cluster.  And lots of code cleanup.
2023-03-17 18:07:03 -07:00
Jingyu Zhou
ad778cbe5e Merge branch 'main' of https://github.com/apple/foundationdb into fix 2023-03-02 09:56:30 -08:00
Jingyu Zhou
9ec9f38a50 Allow fdbbackup, fdbrestore to read keyranges from a file 2023-02-27 13:06:30 -08:00
Zhe Wu
c5c434eb71 Add more comments in fdbbackup query command, and address comments 2023-02-27 10:00:49 -08:00
Zhe Wu
4fd3a90732 fix clang build error 2023-02-26 12:38:52 -08:00
Zhe Wu
ffa3467098 Explicitly using min and max restorable version from backup description in query command in stead of going throw snapshots 2023-02-26 12:17:07 -08:00
Zhe Wu
92a90c78d2 clean format 2023-02-25 18:13:55 -08:00
Zhe Wu
8a88df0e91 Query backup size from a specific snapshot 2023-02-25 17:38:27 -08:00
Zhe Wu
a94dd3a430 Fix fdbbackup query returning earliest version 2023-02-25 16:44:45 -08:00
Vaidas Gasiunas
f8b1da8bc6 An option to initialize client tracing in setupNetwork (#9209)
* client_config_tester: use a generic mechanism to set specific network options

* trace_initialize_on_setup option to initialize client traces on network setup without local IP address

* trace_initialize_on_setup: Addressing review comments

* Restore correct formatting

* trace_initialize_on_setup: Update go bindings

* Include PID for identification into trace file names by default

* Use the same naming pattern for trace files in all configurations

* Empty commit
2023-02-02 10:00:51 +01:00
Nim Wijetunga
05a8a90830 Snapshot Backup Tenant Deletion Support (#9145)
Tenant deletion support for snapshot backups
2023-01-26 17:46:14 -08:00
Xiaoxi Wang
bbcb3cc018 extract KeyBackedConfig, StorageWiggleData class; solve template resolution problem; solve MV txn and native api conflict by splitting RunTransaction file 2023-01-02 23:34:39 -08:00
sfc-gh-tclinkenbeard
49ab88e912 Fix more clang 15 warnings 2022-12-08 14:25:36 -08:00
FoundationDB CI
86d6106dc1 format source code after switch to clang 15 2022-12-08 17:26:45 +00:00
Nim Wijetunga
70d30d93aa Add CLI Argument to Override Snapshot Backup Encryption (#8661)
* add cli option

* fix compile error

* address pr comments

* trigger build

* trigger build
2022-11-07 20:39:18 -08:00
Nim Wijetunga
6f37f55917 Restore System Keys First in Backup/Restore Workloads (#8475)
* system key restore ordering

* restore system keys before regular data

* atomic restore backup fix

* change testing

* fix compile error

* fix compile issue

* fix compile issues

* Trigger Build

* only split restore if encryption is enabled

* revert knob changes

* Update fdbserver/workloads/AtomicSwitchover.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/AtomicSwitchover.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/BackupCorrectness.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbserver/workloads/AtomicRestore.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* add todo

* strengthen check

* seperate system restore for atomic restore

* address pr comments

* address pr comments

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-10-26 09:38:27 -07:00
Nim Wijetunga
025359a974 Add Options to restore user/system keys in backup agent (#8497)
* add options to restore user/system keys

* format help commands

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update documentation/sphinx/source/backups.rst

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbbackup/backup.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Update fdbbackup/backup.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* address pr comments

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-10-20 09:58:00 -07:00
Nim Wijetunga
232e23c87e Snapshot Backup Encryption (#8095)
* inital commit

* add encryption support

* refactor passing encryption keys

* disable encryption codepath

* pass enable encryption cli arg for fdbbackup

* change encryption key passing

* only enable encryption for backup if tenant mode is required

* revert server knobs

* ignore json trace files

* --amend

* integrate ekp

* add comments

* comments

* comments

* revert knobs

* fix bug

* modify encrypted backup logic

* use common prefix

* check boundaries between begin and end key

* return system encrypt domain

* modify

* add tenant entry cache to backup

* slight revert

* format

* Trigger Build

* format

* use tenant name to fetch keys

* Trigger Build

* Trigger Build

* merge

* fix getting ekp

* fix bug

* address pr comments

* Trigger Build

* Trigger Build

* remove faulty check
2022-09-29 14:45:47 -07:00
A.J. Beamon
fda0d7223d Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants. 2022-09-22 10:00:13 -07:00
A.J. Beamon
4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
Dennis Zhou
80a0816157 flow: switch from hard coded to ApiVersion like ProtocolVersion (#8071)
* flow: add ApiVersion to replace hard coding api version

Instead of hard coding api value, let's rely on feature versions akin to
ProtocolVersion.

* ApiVersion: remove use of -1 for latest and use LATEST_VERSION
2022-09-02 09:28:13 +02:00
Vaidas Gasiunas
79571dd2b4 Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
A.J. Beamon
7bca503528 Use narrow includes in key backed types; remove some unnecessary return statements. 2022-07-14 13:26:43 -07:00
A.J. Beamon
1dc78a3b3f Add support for different transaction types (Reference<ITransaction>, Reference<ReadYourWritesTransaction>) to key backed types 2022-07-01 10:02:39 -07:00
Markus Pilman
03d913a1de Flow compiling 2022-06-27 17:05:55 -06:00
Markus Pilman
d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
Renxuan Wang
13b2b455b1 HTTP proxy for backup.
Only support non-TLS blobs now.
2022-05-09 13:56:33 -07:00
sfc-gh-tclinkenbeard
06825775db Fix formatting of lines with TLS_OPTION_FLAGS 2022-05-02 22:56:06 -07:00
sfc-gh-tclinkenbeard
7f05221cfe Removed TLS_DISABLED macro 2022-05-02 22:15:27 -07:00
Yi Wu
994b8c92f8 Add option to limit resident memory and remove default memory limit (#6719)
Changing `memory` option to limit resident memory instead of virtual memory, in config file and fdbserver/fdbbackup/fdbcli command-line argument. Since `rlimit` doesn't support limiting virtual memory, the current implementation have both of fdbmonitor and the fdbserver/fdbbackup process checking process RSS periodically and kill and restart the process if the limit is exceeded.

Adding a new `memory_vsize` option to limit virtual memory, if backward-compatible behavior is desired.

closes #6671, closes #6672
2022-04-06 20:06:24 -07:00
Jingyu Zhou
f68fd28d73 Refactor duplicated code into IKnobCollection::setupKnobs() 2022-04-05 02:06:38 -07:00
Renxuan Wang
e1775627ab Add a check on proxy format. 2022-03-29 10:17:14 -07:00
Renxuan Wang
0a332ee1c1 Add proxy option to backup and restore params. 2022-03-28 17:10:49 -07:00
sfc-gh-tclinkenbeard
a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
sfc-gh-tclinkenbeard
8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
Jingyu Zhou
1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
Renxuan Wang
233c918ffb Replace printf() and fprintf() with fmt::print(). 2022-02-25 19:06:57 -08:00
Renxuan Wang
f7eb66441d Try eliminating warnings in macOS and Windows CI builds.
MacOS warnings are format warnings, e.g., `format specifies type 'long' but the argument has type 'Version' (aka 'long long')`.
Windows warnings are `ACTOR does not contain a wait() statement`.
2022-02-25 19:06:57 -08:00
A.J. Beamon
250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
sfc-gh-tclinkenbeard
ef711d405c Add fdbbackup tags command 2022-01-24 13:34:54 -08:00
A.J. Beamon
ff1cb58174 Convert hyphens to underscores for all prefix-based arguments (e.g. --knob-, --locality-) 2021-12-14 12:01:44 -08:00
A.J. Beamon
f24adc7b6a Fix a bunch of places where we used old-style arguments. Allow hyphens for profiler args. 2021-12-14 09:59:14 -08:00
A.J. Beamon
f29f487823 Unify flags (#25)
* Unify flags implementation and change help text in backup.actor.cpp
Description

Testing

* Keep LOG_GROUP unchanged

Description

Testing

* Transfer the hyphens to underscores for internal options and user's input, EXCEPT leading hyphens

Description

Testing

* Use a deep copy of the user's input flag to do the match

Description

Testing

* Convert the _ to - in Option arrays of backup.actor.cpp

Description

Testing

* Transter _ to - for files:
        TLSConfig.actor.h, fdbcli.actor.cpp, fdbserver.actor.cpp, FileConverter.h, FileConverter.cpp

Description

Testing

* Change another way to unify flag: using SO_O_ICASE_HYPHEN_AND_UNDERSCORE to determine whether we do the conversion in function IsEqual

Description

Testing

* Change the config command's name from SO_O_ICASE_HYPHEN_AND_UNDERSCORE to SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Update the comment for the SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Fix left underscore in SOption arrays

Description

Testing

* Convert _ to - in several files for commands

Description

Testing

* Make the FDBService and fdbmonitor backward compatible

Description

Testing

* Fix bugs about pointers

Description

Testing

* Check underscore and hyphen at the same time for --knob_, --localily_ and --test_
And fix bugs in fdbmonitor and FDBService
Description

Testing

* Simplify the function in fdbmonitor and FDBService about retrieving arguments.
And fix some documents in masterserver.actor.cpp

Description

Testing

* Convert _ to - for knob in the setKnob functions

Description

Testing

* Convert - to _ in the setKnob functions

Description
Since key in the knob related maps only contain _

Testing

* Rename varialbe name in the fdbmonitor and FDBService for clarification

Description

Testing

Co-authored-by: Chang Liu <chang.liu@snowflake.com>
2021-12-14 08:44:39 -08:00
sfc-gh-tclinkenbeard
464d9488ef Merge remote-tracking branch 'origin/master' into fix-unused-warnings 2021-12-01 23:52:09 -08:00