Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement CLI v2 #7

Closed
wants to merge 4 commits into from
Closed

Implement CLI v2 #7

wants to merge 4 commits into from

Conversation

hlinnaka
Copy link
Contributor

As described in neondatabase/rfcs#5

This doesn't rely on any Postgres modifications, all the commands have
been implemented in the most naive way possible, with lots of copying data
around. For example, when you initialize a new data directory, we copy
the last snapshot on that timeline to the data directory, instead of doing
it lazily. Doesn't make use of a Page server either.

Take a look at https://github.com/zenithdb/zenith/blob/laptop-cli-v2/cli-v2-story.md for a little walk-through of how to use this. It's close to the examples given in https://github.com/zenithdb/rfcs/blob/6bbca945327a6cb1b4fc09d987235e7c07589f0c/CLI.md.

As described in neondatabase/rfcs#5

This doesn't rely on any Postgres modifications, all the commands have
been implemented in the most naive way possible, with lots of copying data
around. For example, when you initialize a new data directory, we copy
the last snapshot on that timeline to the data directory, instead of doing
it lazily. Doesn't make use of a Page server either.
@kelvich
Copy link
Contributor

kelvich commented Apr 15, 2021

Playing with this I can see several areas how CLI v1 and v2 are different:

  • One repo / multiple repos

    v1 creates ~/.zenith with absolute paths to repo directory and all stuff is stored there

    v2 initializes repo in a directory where it was initialized. That makes sense while we store everything in files without any daemon on top, but when add daemon into equation (like pageserver) things are more complicated -- should we launch one pageserver for all repos or should we launch separate pageserver for each repo. Single pageserver seems less redundant, but do we need tablespace-like functionality in pageserver now? So let's assume pageserver per repo model for now as a way to unite concepts here.

  • Timelines and terminology

    In v2 we have more or less 1-to-1 correspondence between datadir, timeline and branch. Each timeline consist of snapshots and timeline can have parent timeline.

    In v1 there is a 'pg' concept which represents datadir, timeline and branch ('pg create --snapshot=mysnap' is assumed to create new datadir from mysnap and corresponding storage in pageserver ~= timeline). And snapshot are not meant to be grouped into timelines, each snapshot can have it's own parent. From the technical angle that may be actually implemented as a timeline per snapshot. Also if we allow both CoW and non-CoW branches/datadirs implementation should take that into account and do slightly different things.

    In git "branch" command is necessary because file modifications are not going through git itself. But for us only way to create "commit" is to start some postgres and run some queries in it. So we have possibility to avoid that and just have tree of linked snapshots. But branch itself is popular concept on its own, so I'm not sure.

  • Storage

    Pageserver vs. local files. We may introduce .zenithconfig in repo during init and state there what kind of storage is used. All other commands can look into this config and use proper implementation. But having two set of commands depending on storage is not good, so let's decide what command do we want.

@hlinnaka
Copy link
Contributor Author

Superseded by #43

@hlinnaka hlinnaka closed this Apr 16, 2021
@hlinnaka hlinnaka deleted the laptop-cli-v2 branch May 7, 2021 08:06
knizhnik added a commit that referenced this pull request Apr 4, 2024
## Problem

Running test_pageserver_restarts_under_workload in POR #7275 I get the
following assertion failure in prefetch:
```
#5  0x00005587220d4bf0 in ExceptionalCondition (
    conditionName=0x7fbf24d003c8 "(ring_index) < MyPState->ring_unused && (ring_index) >= MyPState->ring_last", 
    fileName=0x7fbf24d00240 "/home/knizhnik/neon.main//pgxn/neon/pagestore_smgr.c", lineNumber=644)
    at /home/knizhnik/neon.main//vendor/postgres-v16/src/backend/utils/error/assert.c:66
#6  0x00007fbf24cebc9b in prefetch_set_unused (ring_index=1509) at /home/knizhnik/neon.main//pgxn/neon/pagestore_smgr.c:644
#7  0x00007fbf24cec613 in prefetch_register_buffer (tag=..., force_latest=0x0, force_lsn=0x0)
    at /home/knizhnik/neon.main//pgxn/neon/pagestore_smgr.c:891
#8  0x00007fbf24cef21e in neon_prefetch (reln=0x5587233b7388, forknum=MAIN_FORKNUM, blocknum=14110)
    at /home/knizhnik/neon.main//pgxn/neon/pagestore_smgr.c:2055

(gdb) p ring_index
$1 = 1509
(gdb) p MyPState->ring_unused
$2 = 1636
(gdb) p MyPState->ring_last
$3 = 1636
```

## Summary of changes

Check status of `prefetch_wait_for`

## Checklist before requesting a review

- [ ] I have performed a self-review of my code.
- [ ] If it is a core feature, I have added thorough tests.
- [ ] Do we need to implement analytics? if so did you add the relevant
metrics to the dashboard?
- [ ] If this PR requires public announcement, mark it with
/release-notes label and add several sentences in this section.

## Checklist before merging

- [ ] Do not forget to reformat commit message to not include the above
checklist

---------

Co-authored-by: Konstantin Knizhnik <knizhnik@neon.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants