Thanks to visit codestin.com
Credit goes to github.com

Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
195 commits
Select commit Hold shift + click to select a range
16e0570
Sketch out interface
mcevoypeter Feb 13, 2023
9fa2538
Add clang-format configuration
mcevoypeter Feb 13, 2023
d590931
Stub out functions
mcevoypeter Feb 13, 2023
c01e091
Build library
mcevoypeter Feb 13, 2023
e934648
Add parameter for base address to pma_init()
mcevoypeter Feb 13, 2023
d505257
Implement pma_init()
mcevoypeter Feb 13, 2023
74c2f4e
Remove leading and trailing spaces from conditionals
mcevoypeter Feb 13, 2023
4f62939
Expose PMA length as parameter in pma_init()
mcevoypeter Feb 13, 2023
9539978
Implement map_file_()
mcevoypeter Feb 13, 2023
dd529bc
Add mapped heap and stack len fields to pma_t
mcevoypeter Feb 13, 2023
b6036cd
Implement pma_deinit()
mcevoypeter Feb 13, 2023
d3b5b2c
Test pma_init()
mcevoypeter Feb 13, 2023
64eda97
Install dummy page fault handler
mcevoypeter Feb 13, 2023
1599817
Allocate memory in page fault handler via mmap()
mcevoypeter Feb 13, 2023
414c1d9
Switch from MAP_SHARED to MAP_PRIVATE
mcevoypeter Feb 14, 2023
c9b1fe6
Add array to pma_t for tracking dirty pages
mcevoypeter Feb 14, 2023
9e9171b
Implement addr_to_page_idx_()
mcevoypeter Feb 14, 2023
ff7a677
Add ability to test static PMA functions
mcevoypeter Feb 14, 2023
b1029fd
Test addr_to_page_idx_()
mcevoypeter Feb 14, 2023
b7884c3
Test round_up_()
mcevoypeter Feb 14, 2023
199c65c
Always fail in pma_init() if HAVE_SIGSEGV_RECOVERY isn't defined
mcevoypeter Feb 15, 2023
1f358b5
Only handle SIGSEGV for PMA pages
mcevoypeter Feb 15, 2023
8f27d69
Switch addr_to_page_idx_() to 0-based and remove negative indices
mcevoypeter Feb 15, 2023
1a4ea49
Use two bits to track page status, not one
mcevoypeter Feb 15, 2023
9cd793e
Use page_status_() in handle_page_fault_()
mcevoypeter Feb 15, 2023
4bd36ec
Remove write protections from mapped page in page fault handler
mcevoypeter Feb 15, 2023
9f993a9
Set page status in map_file_()
mcevoypeter Feb 15, 2023
84078c5
Move rounding functions into separate file
mcevoypeter Feb 15, 2023
bc09978
Change return type of page_status_() to page_status_t
mcevoypeter Feb 15, 2023
83c7e3f
Handle empty files correctly
mcevoypeter Feb 15, 2023
6a41fc0
Replace u3e_live() with pma_init()
mcevoypeter Feb 15, 2023
d374162
Replace static with static_ for constants
mcevoypeter Feb 16, 2023
c903cee
Add set_page_status_range_()
mcevoypeter Feb 16, 2023
780dd64
Make addr as const where possible
mcevoypeter Feb 16, 2023
5631f16
Move kPageSz into separate header file
mcevoypeter Feb 16, 2023
1ffcabc
Stub out journal interface
mcevoypeter Feb 16, 2023
c30eeae
Implement journal_open()
mcevoypeter Feb 16, 2023
409647d
Add missing path to fprintf()
mcevoypeter Feb 16, 2023
1e99a05
Run clang-format
mcevoypeter Feb 16, 2023
3194118
Stub out read_all() and write_all()
mcevoypeter Feb 16, 2023
44b7708
Implement read_all() and write_all()
mcevoypeter Feb 16, 2023
977056e
Implement journal_append()
mcevoypeter Feb 16, 2023
3670781
Add path to journal_t
mcevoypeter Feb 16, 2023
4249bd0
Implement journal_destroy()
mcevoypeter Feb 16, 2023
b05ae5e
Implement journal_apply()
mcevoypeter Feb 16, 2023
5e8ca10
Allow stack-allocated journal_t via journal_open()
mcevoypeter Feb 16, 2023
119e372
Add crash recovery at end of pma_init() via pma_sync()
mcevoypeter Feb 16, 2023
2524f4e
Sync heap and stack files using journal in pma_sync()
mcevoypeter Feb 16, 2023
30bc597
Resize backing files and reapply mappings
mcevoypeter Feb 16, 2023
7126f48
Don't explicitly extend backing files
mcevoypeter Feb 16, 2023
f8588a5
Add total_len_()
mcevoypeter Feb 16, 2023
226333d
Use set_page_status_range_(
mcevoypeter Feb 16, 2023
af7e969
Add journal_sync()
mcevoypeter Feb 16, 2023
42c43d9
fsync() backing file after applying journal
mcevoypeter Feb 16, 2023
79bc3b3
Construct journal file path
mcevoypeter Feb 16, 2023
12e12cd
Return 0 on success from write_all()
mcevoypeter Feb 16, 2023
99c136a
Add new_file_() helper function to pma_tests
mcevoypeter Feb 17, 2023
7a47180
Allow NULL path as argument to sync_file_()
mcevoypeter Feb 17, 2023
4bda946
Add heap and stack file paths to pma_t
mcevoypeter Feb 17, 2023
9606485
Pass correct base address for stack when calling write_all() in sync_…
mcevoypeter Feb 17, 2023
5e081d4
Log error messages read_all() and write_all()
mcevoypeter Feb 17, 2023
75811e4
Add pma_sync() to empty backing files test
mcevoypeter Feb 17, 2023
2f8831d
Successfully test page status after read only
mcevoypeter Feb 17, 2023
8f1249d
Modify new_file_() to support creating large files
mcevoypeter Feb 17, 2023
88a2ac4
Add macros for dealing with GiB, MiB, and KiB
mcevoypeter Feb 17, 2023
d8e2eac
Unmap entire arena in pma_deinit()
mcevoypeter Feb 17, 2023
4de5105
Test file-backed PMA with sync and truncation
mcevoypeter Feb 17, 2023
bb3f22b
Add PMA_DEBUG macro to make it possible to debug SIGSEGV
mcevoypeter Feb 17, 2023
b30488f
Replace u3e_save() with pma_sync()
mcevoypeter Mar 1, 2023
a871bc8
Comment out heap logic in pma_sync()
mcevoypeter Mar 1, 2023
bf317ed
Add missing return statement to journal_sync()
mcevoypeter Mar 1, 2023
e87ccc1
Reset pma heap_len and heap_fd after sync_file_()
mcevoypeter Mar 1, 2023
ad1c38f
End print statements with \r\n
mcevoypeter Mar 1, 2023
6fda596
Apply journal directly to snapshot file
mcevoypeter Mar 3, 2023
6105ca4
Lint
mcevoypeter Mar 3, 2023
2a5a209
Correctly update stack len and heap and stack fd fields in pma_t
mcevoypeter Mar 3, 2023
116d19b
Properly declare and define inline utility functions
mcevoypeter Mar 3, 2023
85e70f0
Reformat separating banners
mcevoypeter Mar 3, 2023
ff579cc
Add --config=test to CI invocation of unit tests
mcevoypeter Mar 3, 2023
f147c3a
Rename pma_init() as pma_load()
mcevoypeter Mar 6, 2023
4a2452b
Rename pma_deinit() as pma_unload()
mcevoypeter Mar 6, 2023
6f6b563
Undef inline_
mcevoypeter Mar 6, 2023
602d742
Document page_status_t and pma_t
mcevoypeter Mar 6, 2023
e1ddd11
More efficiently remove unused mappings in pma_sync()
mcevoypeter Mar 6, 2023
81cf617
Document map_file_()
mcevoypeter Mar 6, 2023
a313471
Tweak handle_page_fault_() documentation
mcevoypeter Mar 6, 2023
4ccaf34
Tweak addr_to_page_idx_() documentation
mcevoypeter Mar 6, 2023
3e3bf00
Improve documentation for page status getter and setters
mcevoypeter Mar 6, 2023
8b35580
Improve sync_file_() documentation
mcevoypeter Mar 6, 2023
ea1b1dd
Document kPageSz
mcevoypeter Mar 6, 2023
907f4cf
Rename journal as wal (write-ahead log)
mcevoypeter Mar 6, 2023
5d8413a
Document wal_t and wal_entry_t
mcevoypeter Mar 6, 2023
f9bbfaf
Document wal interface
mcevoypeter Mar 6, 2023
e6b7f18
Return error from pma_load() if stack and heap overlap
mcevoypeter Mar 6, 2023
9fe978c
Document pma interface
mcevoypeter Mar 6, 2023
4fd6249
Factor out duplicated snapshot logic
mcevoypeter Mar 6, 2023
78ec3a0
Remove old snapshotting system
mcevoypeter Mar 7, 2023
3f9c81e
Place guard page at end of pma_sync()
mcevoypeter Mar 9, 2023
294cd40
Recenter guard page in fault handler
mcevoypeter Mar 9, 2023
da85a35
Revert "Recenter guard page in fault handler"
mcevoypeter Mar 10, 2023
4321ab6
Add OOM handler to pma_load() interface
mcevoypeter Mar 10, 2023
7d1af91
Bail meme from OOM handler
mcevoypeter Mar 10, 2023
562201e
Correctly recenter guard page on OOM
mcevoypeter Mar 10, 2023
d56e249
Properly apply write-ahead log to snapshot during crash recovery
mcevoypeter Mar 10, 2023
2fd2f12
Mark just-mapped pages as dirty, not clean
mcevoypeter Mar 14, 2023
1254a98
Remove reundant wal_apply() in map_file_()
mcevoypeter Mar 14, 2023
3f4a31e
Don't unmap at end of pma_sync()
mcevoypeter Mar 14, 2023
1b0e547
Validate write-ahead log
mcevoypeter Mar 14, 2023
92b4ef1
Replace %llu with %zu when printing size_t
mcevoypeter Mar 14, 2023
14efdea
Properly handle unaligned base and len in pma_load()
mcevoypeter Mar 14, 2023
db28dec
Remove inappropriate assert()s in wal_destroy()
mcevoypeter Mar 14, 2023
110b0da
Add basic error handling instructions
mcevoypeter Mar 14, 2023
990f71f
Set errno correctly in read_all() and write_all()
mcevoypeter Mar 17, 2023
5f611cb
Return success early from read_all() and write_all() for zero length
mcevoypeter Mar 17, 2023
93317ca
Fold typedef into same line as struct declaration
mcevoypeter Mar 17, 2023
10f20c6
Remove extra white space
mcevoypeter Mar 17, 2023
4e2d53a
Document that kBitsPerByte must be power of 2
mcevoypeter Mar 17, 2023
dac9cec
Remove stale documentation restricting use of PS_MASK
mcevoypeter Mar 17, 2023
d099663
Remove stale wal_entry_t
mcevoypeter Mar 17, 2023
a1e6166
Rename u3m_take_snapshot() as u3m_save_snapshot()
mcevoypeter Mar 17, 2023
f406c87
Correct error handling in wal_open()
mcevoypeter Mar 17, 2023
8da9bfb
Finish adding errno semantics
mcevoypeter Mar 17, 2023
e2c3798
Remove unnecessary declarations for inline arithmetic functions
mcevoypeter Mar 17, 2023
a6f9a35
Add PS_MAPPED_DIRTY to handle_page_fault_() switch statement
mcevoypeter Mar 17, 2023
11dee5c
Directly include pma.c in pma_tests.c
mcevoypeter Mar 17, 2023
732293a
Simplify PS_UNMAPPED assert
mcevoypeter Mar 17, 2023
09c62b4
Remove leading . from file extensions
mcevoypeter Mar 17, 2023
22922ba
Include page index in WAL checksum
mcevoypeter Mar 17, 2023
728ea16
Test write-ahead log
mcevoypeter Mar 17, 2023
eae3739
Back up snapshot
mcevoypeter Mar 20, 2023
3dda1a8
Use %lld for off_t, not %u
mcevoypeter Mar 20, 2023
60af462
Add missing return to wal_apply()
mcevoypeter Mar 20, 2023
0fb6f9c
Remove --config=test from CI unit test invocation
mcevoypeter Mar 20, 2023
04f3a02
Expose pma_center_guard_page() in public interface
mcevoypeter Mar 21, 2023
924fa18
Add global checksum to write-ahead log
mcevoypeter Mar 21, 2023
964dbf0
madvise() when heap or stack shrinks in pma_sync()
mcevoypeter Mar 22, 2023
9687351
Clarify documentation of kBitsPerPage
mcevoypeter Mar 22, 2023
be9e0ed
Revert "Remove unnecessary declarations for inline arithmetic functions"
mcevoypeter Mar 22, 2023
35d4bae
Source kPageSz from page.h in util tests
mcevoypeter Mar 22, 2023
6001287
Switch to 64KB pages
mcevoypeter Mar 22, 2023
9ef007a
Remove fragile round_up() test case
mcevoypeter Mar 22, 2023
44bf4be
Switch to 128KB pages
mcevoypeter Mar 22, 2023
4a22a78
Remove unnecessary trailing _ from variable names
mcevoypeter Mar 23, 2023
fd6d6cc
Use leading, not trailing, underscore for statically-scoped identifiers
mcevoypeter Mar 23, 2023
fd9700d
Rename pma_center_guard_page() as pma_adjust()
mcevoypeter Mar 23, 2023
a7aab03
Document PMA interface
mcevoypeter Mar 23, 2023
88cf858
Document PMA implementation
mcevoypeter Mar 23, 2023
9899773
Remove pma->max_sz
mcevoypeter Mar 23, 2023
fdac132
Update wal.h header guards
mcevoypeter Mar 23, 2023
044b198
Document WAL interface and implementation
mcevoypeter Mar 23, 2023
0876513
Use u3m_backup_snapshot() to back up snapshot
mcevoypeter Mar 23, 2023
8b16de5
Define NDEBUG on release builds
mcevoypeter Mar 23, 2023
ce5501d
Revert "Define NDEBUG on release builds"
mcevoypeter Mar 23, 2023
4b2bca5
Free pma in pma_unload()
mcevoypeter Mar 27, 2023
cb8544b
Verify len in _sync_file() is multiple of kPageSz
mcevoypeter Mar 27, 2023
b104999
Make explicit WAL usage pattern
mcevoypeter Mar 27, 2023
b3aab95
Mark pages as unmapped on failure in _map_file()
mcevoypeter Mar 27, 2023
3717c4c
Revert "Switch to 128KB pages"
mcevoypeter Mar 27, 2023
0c32314
Pass fd to _map_file() instead of path
mcevoypeter Mar 28, 2023
8c40f6f
Apply write-ahead log in pma_load(), not _map_file()
mcevoypeter Mar 28, 2023
d99a148
Use one WAL for both heap and stack
mcevoypeter Mar 28, 2023
144f599
Give wal_open() a directory path, not a file path
mcevoypeter Mar 28, 2023
a9d7ed3
Use fnctl(F_FULLFSYNC) on macOS
mcevoypeter Mar 28, 2023
8e8e64c
Don't truncate backing files
mcevoypeter Mar 28, 2023
d7da1ec
Don't create anonymous mappings when the backing file is nonexistent
mcevoypeter Mar 28, 2023
cb10b17
Remove unused fd parameter to _append_dirty_pages()
mcevoypeter Mar 29, 2023
767fc64
Replace "is there" with "if there" in pma_t documentation
mcevoypeter Mar 29, 2023
35aff84
Add @param[in] documentation for oom_handler_t's fault_addr
mcevoypeter Mar 29, 2023
e955a08
Fix argument offsets in pma_load() call
mcevoypeter Mar 29, 2023
b30ba3c
Remove extra space in fprintf()
mcevoypeter Mar 29, 2023
beab2d9
Remove now-unnecessary pma_adjust() parameter comments
mcevoypeter Mar 29, 2023
9601b32
Explain that i is unused because of read_all() semantics
mcevoypeter Mar 29, 2023
cfc5c26
Clarify failure comments at _map_file() call sites
mcevoypeter Mar 29, 2023
5a246f8
Clarify comment in _map_file() about anonymous mappings
mcevoypeter Mar 29, 2023
4ac859c
Replace int with c3_o in new u3m signatures
mcevoypeter Mar 29, 2023
1505ad8
Remove unused variable in PMA tests
mcevoypeter Mar 29, 2023
779881f
Revert "Switch to 64KB pages"
mcevoypeter Mar 30, 2023
930e4e4
Require stack and heap files to be both NULL or both non-NULL
mcevoypeter Mar 30, 2023
3e15ad1
Add retry logic to write_all()
mcevoypeter Mar 30, 2023
822c391
Add retry logic to read_all()
mcevoypeter Mar 30, 2023
b9676d4
Replace assert() with c3_assert()
mcevoypeter Mar 30, 2023
45285e0
Initialize tmp checksum variable to 0
mcevoypeter Mar 30, 2023
d393848
Lint
mcevoypeter Mar 30, 2023
a2d30db
Ensure global checksum integrity at end of wal_open()
mcevoypeter Mar 30, 2023
4902621
Add test for corrupt WAL entry checksum
mcevoypeter Mar 30, 2023
c49c88c
Ensure all 64-bit values passed to MurmurHash3_x86_32 are zero-initia…
mcevoypeter Mar 30, 2023
c9eacfd
Read metadata header in wal_open()
mcevoypeter Mar 30, 2023
fd3927e
Include page index in checksum in wal_append()
mcevoypeter Mar 30, 2023
fd3aacb
Add _page_checksum()
mcevoypeter Mar 30, 2023
2042bb5
Add WAL version number
mcevoypeter Mar 30, 2023
3d2326c
Replace mistaken use of data_path with meta_path
mcevoypeter Mar 30, 2023
80bc6d2
Revert "Don't unmap at end of pma_sync()"
mcevoypeter Mar 30, 2023
bb556f7
Mark untracked dirty pages as read-only at end of pma_sync()
mcevoypeter Mar 30, 2023
d0083d1
Reserve entire PMA in pma_load()
mcevoypeter Mar 31, 2023
6ac58b3
Address review feedback, MADV_DONTNEED ephemeral space
philipcmonk Apr 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,59 @@ title and description. Assuming that you properly included the "Resolves #N."
directive in the pull request description, merging will automatically close the
tracking issue associated with the pull request.

## C Style

### Error Handling

Our error handling is currently suspect, and one of our priorities moving
forward is to bring it in line with standard [POSIX error handling]. If you're
writing new code, please adhere to the [guidelines laid out by POSIX][POSIX
error handling].

Use `assert()` to programmatically document invariants that are known at compile
time. The following is an acceptable use of `assert()`:
```c
Comment on lines +74 to +76
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no mention of a debug assert here, i.e. c3_dessert. Especially in hotpaths, extra conditional jumps almost certainly hurt performance rather than improve the compiler's optimization capability.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. To address this, I've defined NDEBUG for release builds, which will replace assert() with (void)(0).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to revert the NDEBUG definition because it's leading to segfaults, potentially related to our current abuse of c3_assert().

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I expected that. But also, there's an important difference between c3_dessert and assert. dessert actually allows you to recover an assertion if you have a debugger attached. If you don't, the process will dump core. But if you do, it's effectively a conditional breakpoint in your code. There are some scenarios where that's useful

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm reluctant to introduce c3 as dependency of pma. ur is also implemented as a separate library that doesn't depend on c3, so there's a precedent. To your point about wanting the utility functions defined in a shared place, I wonder if it makes sense to define a separate utility package that includes these sorts of things. c3 is partially that but also partially a statement of opinion on how to use C due to its inclusion of loobeans, typedefs, etc.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to a separate utility package

static void
local_(const char *ptr)
{
assert(ptr);
// <snip>
}

int
global(const char *path)
{
if (!path) {
fprintf(stderr, "path cannot be NULL");
return -1;
}
local_(path);
}
```

The following is an *unacceptable* use of `assert()`:
```c
int
global(const char *path)
{
int fd = open(path, O_RDONLY);
assert(fd != -1);
}
```

Instead, if `fd == -1` is a non-fatal error, `return -1`; if `fd == -1` is
fatal, use `exit()`:
```c
int
global(const char *path)
{
int fd = open(path, O_RDONLY);
if (fd == -1) {
exit(errno); // replace with `return -1` if non-fatal
}
}
```


## Development Environment

Expand Down Expand Up @@ -153,3 +206,5 @@ jump to definition, cross-references, hovering, symbol renaming, etc.):
```console
bazel run //bazel:refresh_compile_commands
```

[POSIX error handling]: https://csresources.github.io/SystemProgrammingWiki/SystemProgramming/POSIX,-Part-1:-Error-handling/
1 change: 1 addition & 0 deletions pkg/noun/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ cc_library(
deps = [
"//pkg/c3",
"//pkg/ent",
"//pkg/pma",
"//pkg/ur",
"//pkg/urcrypt",
"@gmp",
Expand Down
Loading