Fix: support correct saving for scaled displays #85

AndreasBackx · 2024-02-10T22:59:15Z

In #78 it was discussed that master did not support scaled displays. The way screenshotting works is that screencopies are requested in logical regions (so after scaling, a pixel is not a real world pixel), but they're actually saved as "physical" regions (before scaling, aka the unit is 1 real world pixel). Some issues that I fixed were:

We created the final image as a "logical" size. So when someone took a screenshot of a 4K screen that was scaled 200%, the picture in memory would be 4K but we would create a 1080p size output file. A 4K output file would now be created.
Following the previous step, we need to convert the "logical" region to a "phyiscal" region, aka scale it as well.

But how and where do we scale? From all of the outputs, we take the maximum scale and make sure to scale all other outputs the same amount. This makes sure that our final image will lose the least amount of information. I'm having a hard time explaining this so I'll give some examples in hopes that it makes sense:

1x 1080p + 1x 4K @ 100% -> neither will be resized.
1x 1080p + 1x 4K @ 200% -> the 1080p one will be scaled 200%.
1x 4K @ 100% + 1x 4K @ 200% -> the 4K @ 100% will be scaled 200%.

This seemed the most logical default scaling to me. We could technically make it configurable, though the way it's setup now is that it assumes it will never scale it down.

Still have to do some testing when attaching multiple monitors, but it should work. ™ I am working on my desk setup so it might take me a while to have more than 1 monitor, I'd appreciate someone else testing it with 2 screens and different scaling.

Note: this PR does not fix the freeze overlaying. I actually removed it even because the changes wouldn't allow for it and it needs to be fixed in another PR.

Shinyzenith · 2024-02-15T19:44:10Z

The upscale policy seems acceptable to me. The code from a preliminary view looks fine to me! I don't have a dual monitor system at the moment.

@Decodetalkers Would you mind testing the PR?

Shinyzenith · 2024-02-15T19:45:02Z

Note: this PR does not fix the freeze overlaying. I actually removed it even because the changes wouldn't allow for it and it needs to be fixed in another PR.

Pardon me, I seem to be losing track due to the large changes recently, what overlaying issue is being discussed here? Is it the image that decode posted in the earlier pr which was due to us not scaling the overlay buffer to physical size?

AndreasBackx · 2024-02-16T00:36:20Z

Note: this PR does not fix the freeze overlaying. I actually removed it even because the changes wouldn't allow for it and it needs to be fixed in another PR.

Pardon me, I seem to be losing track due to the large changes recently, what overlaying issue is being discussed here? Is it the image that decode posted in the earlier pr which was due to us not scaling the overlay buffer to physical size?

There are still two issues with overlaying:

Hyprland rotation problem, will need to look into it further and either fix or report a bug.
Overlaying in general has not worked yet on a scaled output because this will require one of the newer wlr protocols that allows you to specify that a surface is fractional scaling aware. On a scaled display, it won't overlay correctly afaik because of this. I haven't looked into it much, though we might not even need the special protocol, I'm unsure. Anyhow, overlaying during a freeze is not scaled properly so does not cover the entire output.

Primarily referring to 2 here.

Decodetalkers · 2024-02-16T01:25:05Z

libwayshot/src/lib.rs


                surface.set_buffer_transform(output_info.transform);
-                surface.set_buffer_scale(output_info.scale);
+                // surface.set_buffer_scale(output_info.scale());


why comment this line? will this line never use again?

This accepts only an integer and if we were to round it, it'd still be incorrect. It needs fractional scaling support.

libwayshot/src/image_util.rs

Decodetalkers · 2024-02-16T01:28:23Z

Emm. you should resize in freeze stage, not in finally picture output stage, so I think you just need resize the picture you get in freeze is enough, and maybe this pr is not needed

Decodetalkers · 2024-02-16T01:40:20Z

Maybe you can add a option, for if it is preview image or not, preview image will always resize

AndreasBackx · 2024-02-16T02:27:34Z

Emm. you should resize in freeze stage, not in finally picture output stage, so I think you just need resize the picture you get in freeze is enough, and maybe this pr is not needed

We should not need to resize at all for a freeze as we'd lose quality. We already have a pixel perfect screenshot we simply need to overlay properly.

Also, we would not be able to anyhow. Freezing is done using the same buffer as the screenshot and resizing would mean reading it from there which would make the freeze noticeably lag behind the startup.

Decodetalkers · 2024-02-16T04:45:49Z

no, the fix induced many problem, so I create another pr. the size is still not right

#88

AndreasBackx · 2024-02-16T11:15:41Z

@Decodetalkers can you please explain the issues? You created a PR without any information. When looking at the PR you are doing exactly what I said in my previous comment will cause noticeable lag:

Freezing is done using the same buffer as the screenshot and resizing would mean reading it from there which would make the freeze noticeably lag behind the startup.

You are allocating reading from the existing buffer which takes a lot of time, resizing it, and then writing it to a new buffer. This will lose quality as well and resizing is not at all needed because we have a pixel perfect version of the output in the existing buffers.

Could you explain the issues or possibly point out where my logic is incorrect?

Shinyzenith · 2024-02-16T11:28:37Z

no, the fix induced many problem

Hi @Decodetalkers, can you kindly enumerate the problems induced so we can improve the PR? From what I can tell, apart from the buffer scale setter function which only accepts integers and not fractionally scaled surfaces -- other things look fine.

Shinyzenith · 2024-02-16T11:33:11Z

Also, we would not be able to anyhow. Freezing is done using the same buffer as the screenshot and resizing would mean reading it from there which would make the freeze noticeably lag behind the startup.

And this matters heavily as we ideally want the least amount of time from binary invocation to overlay display -- scaling would make us lag behind due to increased allocations.

Decodetalkers · 2024-02-16T11:47:01Z

@Decodetalkers can you please explain the issues? You created a PR without any information. When looking at the PR you are doing exactly what I said in my previous comment will cause noticeable lag:

Freezing is done using the same buffer as the screenshot and resizing would mean reading it from there which would make the freeze noticeably lag behind the startup.

You are allocating reading from the existing buffer which takes a lot of time, resizing it, and then writing it to a new buffer. This will lose quality as well and resizing is not at all needed because we have a pixel perfect version of the output in the existing buffers.

Could you explain the issues or possibly point out where my logic is incorrect?

Emmm. I have test your pr, but the size of picture is not correct, the overlay image become too big when the scale is 1.2, I have test your pr on sway, but it not works well

Decodetalkers · 2024-02-16T11:53:31Z

you not not adjust the picture on the overlay , I found it still be the size of picture, it will not fully show over the screen. at the beginning, what I concerned is that. the picture size by the wlr-screencopy won't be the size of the screen , so it need to resize to cover on the whole screen, but this pr do not solve that problem. because it does not resize the image it captured. so my problem still not solved.

The buffer scale always be integer, so it won't cover the fractional case, and the size by wlr-screencopy always be the real size of the image, so if is set with fractional size, the frame_copy.frame_format.size won't be the screen size, so it cause problem. So after I test and view this pr, I think this pr has not solve my problem, so I make another pr

Decodetalkers · 2024-02-16T12:02:10Z

wayshot/libwayshot/src/lib.rs

Lines 475 to 478 in 0a8e69d

    
           layer_surface.set_size( 
        
               frame_copy.frame_format.size.width, 
        
               frame_copy.frame_format.size.height, 
        
           );

I mean the size will always not be the size of the screen, so this is my problem

Decodetalkers · 2024-02-16T12:09:08Z

So the problem for me is that I think the picture should over the whole screen, the size should just be the size of screen, other things will be ok it is what I am thinking

Decodetalkers · 2024-02-16T12:13:12Z

I see, this pr is not about freeze overlay, so we are solving two problems

Decodetalkers · 2024-02-16T12:16:01Z

Emm. but without this pr, when use different scale, what will happened? @AndreasBackx I think it will works fine. only I concerned is that the pciture will not cover the screen, the picture shoted will be wrong

libwayshot/src/lib.rs

Decodetalkers · 2024-02-16T12:24:29Z

oh, I see, the logic you use will make the picture be best resized, greate work, have you test on muti screens? I do not have muti screens now

sorry I misunderstood your pr, you are doing greate work, thanks

Shinyzenith · 2024-02-20T23:38:14Z

I'd like to merge this PR. Before I do so, can anyone confirm that this works on multiple monitor system? I do not have access to one sadly.

Decodetalkers · 2024-02-21T01:45:59Z

it is ok, I will merge it

* Refactor: introduce FrameGuard, ScreenCapturer, and Logical/EmbeddedRegion * Add freeze functionality * Move to clap_derive so CLI arguments are typed * Renames `clap.rs` to `cli.rs` because otherwise there's confusion between the `clap` crate and local module. * `Cli` struct added that almost identically represents the current state of the CLI with no logical changes. --- ## `--help` Comparison Before: https://gist.github.com/AndreasBackx/5945b366e989159f4669e7ba30c13239 After: https://gist.github.com/AndreasBackx/8929c8bde080eac0cafd33128210b0cc Diff (ignoring whitespace changes due to table alignment): ```diff 1c1 < Screenshot tool for compositors implementing zwlr_screencopy_v1. --- > Screenshot tool for wlroots based compositors implementing the zwlr_screencopy_v1 protocol. 9d8 < -c, --cursor Enable cursor in screenshots 10a10 > -c, --cursor Enable cursor in screenshots 12c12 < -l, --listoutputs List all valid outputs --- > -l, --list-outputs List all valid outputs 14c14 < --chooseoutput Present a fuzzy selector for outputs --- > --choose-output Present a fuzzy selector for outputs 16a17 > ``` You can see that the only changes are: - About is longer, this is now using the value from Cargo.toml instead of a duplicate text that was shorter. - Some have a dash where the English words would have a space, e.g: "list-outputs" instead of "listoutputs". I've also made the old still work via an alias: https://gist.github.com/AndreasBackx/6025e91844e3d766d4264a01ae4d1a71 This seems like a tiny improvement? I plan to make further changes later, but I want to keep PRs separate. * Improve CLI design This "improves" (and that is subjective) the design of the CLI. I am aiming to get some feedback on what people think of the new design: ``` Screenshot tool for wlroots based compositors implementing the zwlr_screencopy_v1 protocol. Usage: wayshot [OPTIONS] [OUTPUT] Arguments: [OUTPUT] Where to save the screenshot, "-" for stdout. Defaults to "$UNIX_TIMESTAMP-wayshot.$EXTENSION" Options: --log-level <LOG_LEVEL> Log level to be used for printing to stderr [default: info] [possible values: trace, debug, info, warn, error] -s, --slurp <SLURP_ARGS> Arguments to call slurp with for selecting a region -c, --cursor Enable cursor in screenshots --encoding <FILE_EXTENSION> Set image encoder, if output file contains an extension, that will be used instead [default: png] [aliases: extension, format, output-format] Possible values: - jpg: JPG/JPEG encoder - png: PNG encoder - ppm: PPM encoder - qoi: Qut encoder -l, --list-outputs List all valid outputs -o, --output <OUTPUT> Choose a particular output/display to screenshot --choose-output Present a fuzzy selector for output/display selection -h, --help Print help (see a summary with '-h') -V, --version Print version ``` The main changes are: 1. `--debug` is now `--log-level` because this makes it easy to select more specifically what log level to use. I considered using `-v`, `-vv`... to increase verbosity but the `clap-verbosity-crate` uses `log` and not `tracing`. We could use it somewhat, but we'd pull in `log` (which seems fine) and we'd need to map from `log`'s Level to `tracing`'s Level enums (they use inverse ordering). 2. `--stdout` and `--file` has been made an optional positional argument. This because it's what other CLIs often do and I wasn't sure what to call the option otherwise because `--output` and `-O`/`-o` is often what others use but we use it here to refer to displays/monitors/Wayland outputs. This avoids that confusion hopefully and I've also clarified this in the documentation. * Additionally if a path is given, its extension will always be used. So you cannot save `jpg` to `foo.png`. Perhaps this behaviour can be changed, though I don't see a reason to support this weird edge case? When is someone saving `png` to `jpg`? 3. `--extension` is `--encoding` with aliases like `extension`. Again, let me know what you think. * Remove explicit panics and exits Let's bubble up errors instead of panicking and also not use `exit(...)` because it does not call destructors which might give issues. * Make region, sizing, and positioning data structs reusable We use different structs/fields to store the same three types of data everywhere: - Position (x,y) - Size (width,height) - Region(position, size) This makes it so they all reuse the same information. * First version of scaling fix. (#85) * chore: remove useless roundtrip (#105) * feat(clipboard): implement clipboard integration (#91) * feat(clipboard): implement clipboard integration Add the --clipboard flag and implement functionality to make image available on the clipboard using wl-clipboard-rs. * style(format): apply code formatting to cli.rs * feat(clipboard): implement fork wait for clipboard Add functionality offering image on clipboard persistently in the background * feat(clipboard): inform user about wayshot persisting in background with --clipboard * feat(clipboard): use tracing::warn instead of print if fork fails * feat(clipboard): Switch from the fork crate to the nix crate for daemonization * style(format): code formatting to wayshot.rs * style(typo): corrected a typo in the comments * docs(ManPage): Fix capitalization issue Signed-off-by: Shinyzenith <[email protected]> * docs: Document clipboard interaction Signed-off-by: Shinyzenith <[email protected]> * chore: Update Cargo.lock Signed-off-by: Shinyzenith <[email protected]> * feat: added time_stamp flag (#93) * feat: added time_stamp flag * Syntax Refactor and filename format fixed * file name format fixes * wayshot-{timestamp}.{extension} as default filename * chore: Update Cargo.lock Signed-off-by: Shinyzenith <[email protected]> --------- Signed-off-by: Shinyzenith <[email protected]> Authored-by: rachancheet <[email protected]> Co-authored-by: Shinyzenith <[email protected]> * chore: Update Cargo.lock Signed-off-by: Shinyzenith <[email protected]> * feat: Add support for webp (#98) Signed-off-by: Shinyzenith <[email protected]> --------- Signed-off-by: Shinyzenith <[email protected]> Co-authored-by: Shinyzenith <[email protected]> * refactor(libwayshot): Reduce allocations (#99) * refactor: remove unnecessary to_string() * refactor: fix clippy warnings * refactor: remove 1 level of indirection via &Vec<T> -> &[T] Typically `&Vec<T>` isn't something you want when you need a read-only view over a Vec, because it adds another level of indirection: Vec already stores a pointer to heap-allocated buffer internally. Using slices `&[T]` removes such unnecessary level of indirection and is considered a cleaner design. It is cache friendlier and can be better optimized by the compiler (not that it should matter in this case). * feat: account for directories in file path (#96) Signed-off-by: Shinyzenith <[email protected]> --------- Signed-off-by: Shinyzenith <[email protected]> Authored-by: rachancheet <[email protected]> Co-authored-by: Shinyzenith <[email protected]> * fix: Clipboard flag ignored if path is qualified to file (#108) * feat(clipboard): fix issue #106 clipboard flag ignored if path is qualified to file * feat(clipboard): clarify flag description of --clipboard * feat(clipboard): enable multiline comment description for --clipboard feature flag Signed-off-by: Shinyzenith <[email protected]> * feat(clipboard): reduce buffer allocations for encoding image * feat(clipboard): improve code quality * feat(clipboard): code style change: perform match inside function call --------- Signed-off-by: Shinyzenith <[email protected]> Co-authored-by: Shinyzenith <[email protected]> * chore: make clippy happy (#111) * feat(clipboard): fix interaction with freeze and select feature: Issue #109 (#110) * feat(clipboard): fix interaction with freeze and select feature: Issue #109 * feat(clipboard): handle callback fails * Destroy layer shell surfaces * [fix] unmap layer shell surfaces before destroying them * Implement missing conversion from string to webp enum (#121) * Replace nix with rustix (#120) * Update documentation with changes in freeze-feat (#116) * [docs] update manpage (1) with changes in freeze-feat * [fix] make arguments to slurp optional * [docs] update manpage(7) with cli changes * [docs] update README with new cli * Add screencopy dmabuf backend (#122) * [feat] rough MVP for screencpy+dmabuf * [feat] refactor MVP into libwayshot - create new constructor * [refactor] first draft of dmabuf API * [refactor] - Add error handling - Add example/demo for wayshot dmabuf API * [feat] import wayland-egl-ctx for use as MVP for dmabuf import functionality * [fix] correct modifier_lo parameter in waymirror-egl MVP * [feat] get waymirror-egl dmabuf MVP demo working * [feat] refactored dmabuf->eglImage API into libwayshot * [feat] - Implemented clean dropping of EGLImage - Improved API docs for dmabuf functions * [fix] libwayshot build error fixed * [fix] remove hardcoded GPU path from libwayshot constructor * [fix]] remove reduntant dmabuf_to_texture call from waymirror-egl * [docs] document WayshotConnection dmabuf constructor * [feat] Added helper/wrapper function to convert screencapture EGLImages into GL textures * [fix] improved logging in dmabuf API code * [doc] update waymirror-egl Readme * [fix] change logging level in waymirror-egl to Debug * [fix] remove unnecessary .gitignore in waymirror-egl * [ci/cd] attempting to fix the build * [ci\cd] add libegl system deps to fix github CI * [fix] remove unused egl_image struct field in Waymirror demo * fix: I find out a smart way to fix scale problem (#128) * fix: I find out a smart way to fix scale problem * Update libwayshot/src/lib.rs Co-authored-by: Aakash Sen Sharma <[email protected]> --------- Co-authored-by: Aakash Sen Sharma <[email protected]> * Fix for 'cargo test' (#114) * fixes for 'cargo test' & bump flake.lock due to outdated cargo * commit to pass `cargo fmt -- --check` test * add `cargo test` to Makefile --------- Co-authored-by: id3v1669 <[email protected]> --------- Signed-off-by: Shinyzenith <[email protected]> Co-authored-by: Access <[email protected]> Co-authored-by: Andreas Backx <[email protected]> Co-authored-by: Andreas Backx <[email protected]> Co-authored-by: Shivang K Raghuvanshi <[email protected]> Co-authored-by: Sooraj S <[email protected]> Co-authored-by: rachancheet <[email protected]> Co-authored-by: Gigas002 <[email protected]> Co-authored-by: Sergey A <[email protected]> Co-authored-by: id3v1669 <[email protected]> Co-authored-by: id3v1669 <[email protected]>

First version of scaling fix.

0a8e69d

AndreasBackx changed the base branch from main to freeze-feat-andreas February 12, 2024 11:03

Decodetalkers reviewed Feb 16, 2024

View reviewed changes

AndreasBackx mentioned this pull request Feb 16, 2024

fix: scale problem #88

Closed

Decodetalkers reviewed Feb 16, 2024

View reviewed changes

libwayshot/src/lib.rs Show resolved Hide resolved

Decodetalkers approved these changes Feb 16, 2024

View reviewed changes

Decodetalkers requested a review from Shinyzenith February 16, 2024 12:25

Decodetalkers merged commit 8f22e6d into waycrate:freeze-feat-andreas Feb 21, 2024

murlakatamenka pushed a commit to murlakatamenka/wayshot that referenced this pull request Mar 13, 2024

First version of scaling fix. (waycrate#85)

5e3eb37

nukelet pushed a commit to nukelet/wayshot that referenced this pull request Mar 21, 2024

First version of scaling fix. (waycrate#85)

37326d5

Uh oh!

Fix: support correct saving for scaled displays #85

Fix: support correct saving for scaled displays #85

Uh oh!

Conversation

AndreasBackx commented Feb 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shinyzenith commented Feb 15, 2024

Uh oh!

Shinyzenith commented Feb 15, 2024

Uh oh!

AndreasBackx commented Feb 16, 2024

Uh oh!

Decodetalkers Feb 16, 2024

Choose a reason for hiding this comment

Uh oh!

AndreasBackx Feb 16, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Decodetalkers commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Decodetalkers commented Feb 16, 2024

Uh oh!

AndreasBackx commented Feb 16, 2024

Uh oh!

Decodetalkers commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndreasBackx commented Feb 16, 2024

Uh oh!

Shinyzenith commented Feb 16, 2024

Uh oh!

Shinyzenith commented Feb 16, 2024

Uh oh!

Decodetalkers commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Decodetalkers commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Decodetalkers commented Feb 16, 2024

Uh oh!

Decodetalkers commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Decodetalkers commented Feb 16, 2024

Uh oh!

Decodetalkers commented Feb 16, 2024

Uh oh!

Uh oh!

Decodetalkers commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shinyzenith commented Feb 20, 2024

Uh oh!

Decodetalkers commented Feb 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

AndreasBackx commented Feb 10, 2024 •

edited

Loading

Decodetalkers commented Feb 16, 2024 •

edited

Loading

Decodetalkers commented Feb 16, 2024 •

edited

Loading

Decodetalkers commented Feb 16, 2024 •

edited

Loading

Decodetalkers commented Feb 16, 2024 •

edited

Loading

Decodetalkers commented Feb 16, 2024 •

edited

Loading

Decodetalkers commented Feb 16, 2024 •

edited

Loading