Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get_processed_image gives reversed byte order #50

Closed
Victor-N-Suadicani opened this issue Sep 29, 2022 · 12 comments
Closed

get_processed_image gives reversed byte order #50

Victor-N-Suadicani opened this issue Sep 29, 2022 · 12 comments
Assignees

Comments

@Victor-N-Suadicani
Copy link

So I tried the following:

    let document = pdfium.load_pdf_from_file(&args.file, None)?;

    for (page_n, page) in document.pages().iter().enumerate() {
        for (object_n, object) in page.objects().iter().enumerate() {
            match &object {
                PdfPageObject::Text(_) => println!("Got Text object"),
                PdfPageObject::Path(_) => println!("Got Path object"),
                PdfPageObject::Image(image_obj) => {
                    println!("Got Image object");
                    let image = image_obj.get_processed_image(&document)?;
                    image.save(format!("{page_n:0>3}_{object_n:0>3}.jpeg"))?;
                }
                PdfPageObject::Shading(_) => println!("Got Shading object"),
                PdfPageObject::FormFragment(_) => println!("Got FormFragment object"),
                PdfPageObject::Unsupported(_) => println!("Got Unsupported object"),
            }
        }
    }

This worked on the first image of a PDF I tried it on, after which I got an error Error: Pdfium(PdfiumLibraryInternalError(Unknown)). But the important point is that the image that did manage to get extracted had the colours all wrong - a mostly red image turned blue. I'm guessing this is due to #9 and that this doesn't use the set_reverse_byte_order flag?

@ajrcarey
Copy link
Owner

ajrcarey commented Sep 29, 2022

Hi @Victor-N-Suadicani, thanks for raising this. Yes, this sounds a lot like #9, doesn't it? Pdfium internally uses BGR8 byte ordering throughout; the upstream image crate got rid of their support for BGR8 (or, at the very least, made working with it a lot harder) in their move from 0.23 to 0.24. For rendering complete pages, it is possible to tell Pdfium to flip the byte order, as you noted, by setting a flag during rendering. For getting the processed image from an image object, Pdfium provides the function FPDFImageObj_GetRenderedBitmap(), but there is no provision for passing any rendering flags.

It may be necessary to do some byte flipping manually within pdfium-render. I'm happy to implement this, preferably by using functionality provided by the upstream image crate if possible, or by writing it myself if necessary.

Can you attach the sample PDF you are working with?

I'm curious as to what might have caused the error you mentioned, but perhaps it will become clear upon examining your PDF.

@Victor-N-Suadicani
Copy link
Author

Victor-N-Suadicani commented Sep 29, 2022

I can't share the PDF unfortunately but I get the same error on the image-test.pdf you have yourself in the test directory.

Also interestingly the images there seem to turn very tiny after extraction...

@ajrcarey
Copy link
Owner

Ok, that's fine. I'm going to work on this tomorrow.

@ajrcarey
Copy link
Owner

ajrcarey commented Sep 30, 2022

Ok. There's at least three problems here :/

  • Pdfium uses BGR/BGRA internally; image::DynamicImage dropped BGR support in 0.24.x, leaving only RGB/RGBA support intact. pdfium-render needs to seamlessly convert between BGR* and RGB* as required, and doesn't. In the first instance, as you've identified, pdfium-render needs to convert from BGR* to RGB* as required when extracting image data from PdfPageImageObject objects, otherwise the colors are inverted.
  • Additionally, when assigning image data to an image object, pdfium-render needs to convert from RGB* to BGR* as required, otherwise the colors are inverted.
  • Thirdly, Pdfium does not always return a valid FPDF_BITMAP handle when asked to render an image object to a bitmap during a call to FPDFImageObj_GetRenderedBitmap(). This is why you see a Pdfium(PdfiumLibraryInternalError(Unknown)). That function is convenient to use, since it returns a bitmap directly rather than raw pixel data, but it is marked as experimental in the Pdfium API. I will try to use the older, non-experimental FPDFImagObj_GetImageDataDecoded() function to see if it is more stable. Those cases for which FPDFImageObj_GetRenderedBitmap() does not return a valid bitmap, it does not set a error value, so there is no indication from Pdfium as to exactly what the problem was.

Created utils::pixels module for pixel color channel conversion. Added utils::pixels::bgr_to_rgba(), utils::pixels::bgra_to_rgba(), utils::pixels::rgb_to_rgba(), and utils::pixels::rgba_to_bgra() functions. Added correct color conversions when assigning a new image to, or extracting an image from, a PdfPageImageObject.

I'll continue tomorrow with the third problem noted above, with a view to pushing all changes by the end of tomorrow.

@ajrcarey
Copy link
Owner

Added new PdfPageImageObject::set_bitmap() function, to allow setting an object's image data from a BGR* bitmap without performing a double color conversion to and from image::DynamicImage first. Exposed utils::pixels::* conversion functions in PdfiumLibraryBindings. Updated README.md.

@Victor-N-Suadicani
Copy link
Author

Does FPDFImagObj_GetImageDataDecoded take into account the mask (as I believe FPDFImageObj_GetRenderedBitmap is designed to do, despite being buggy)? I was hoping to be able to extract images with transparency.

@ajrcarey
Copy link
Owner

ajrcarey commented Oct 1, 2022

No idea! My initial interest is in seeing whether FPDFImageObj_GetImageDataDecoded() has the same runtime behaviour as FPDFImageObj_GetRenderedBitmap(), i.e. does _GetImageDataDecoded() fail in the same places as _GetRenderedBitmap()? And the answer is no, it does not. _GetImageDataDecoded() always returns a valid byte stream. This suggests that bitmap rendering inside _GetRenderedBitmap() is the problem.

You're not the only person to encounter this problem. Based on the tiny hint at https://groups.google.com/g/pdfium/c/V-H9LpuHpPY, I looked at the transformation matrices of every image object in image-test.pdf. And lo and behold, where either of the matrix values a and d are negative values, _GetRenderedBitmap() fails. Where those matrix values are both positive, _GetRenderedBitmap() succeeds.

This suggests that this is indeed a bug in Pdfium, and you may wish to open a bug upstream. I am open to implementing a work-around in pdfium-render, if possible. For example, simply flipping the matrix a and d values from negative to positive results in a successful call to _GetRenderedBitmap() every time. Presumably, however, this affects the processed output.

I think at this point I'm going to push the changes I've made so far, and post my test code here, so you can play around with it and have a think about what you want to do.

@ajrcarey
Copy link
Owner

ajrcarey commented Oct 1, 2022

In fact, since this _GetRenderedBitmap() problem is separate from the original reason you opened the issue - the mismatched coloring - I'm going to create a new issue. This issue will remain focussed on the color byte ordering.

Spawned new issue #52 to handle the upstream issue. Implemented unit tests for utils::pixels::* color conversion functions. Scheduled for inclusion in crate version 0.7.21.

ajrcarey pushed a commit that referenced this issue Oct 1, 2022
ajrcarey pushed a commit that referenced this issue Oct 1, 2022
@ajrcarey
Copy link
Owner

ajrcarey commented Oct 1, 2022

You can experiment with my changes to PdfPageImageObject by adding pdfium-render as a git dependency in your project's Cargo.toml.

My sample code is basically the same as yours, but for completeness, here it is:

use pdfium_render::prelude::*;

fn main() -> Result<(), PdfiumError> {
    let bindings =
        Pdfium::bind_to_library(Pdfium::pdfium_platform_library_name_at_path("./"))
            .or_else(|_| Pdfium::bind_to_system_library())?;

    let pdfium = Pdfium::new(bindings);

    let document = pdfium.load_pdf_from_file("image-test.pdf", None)?;

    for (page_n, page) in document.pages().iter().enumerate() {
        for (object_n, object) in page.objects().iter().enumerate() {
            match &object {
                PdfPageObject::Text(_) => println!("Got Text object"),
                PdfPageObject::Path(_) => println!("Got Path object"),
                PdfPageObject::Image(image_obj) => {
                    println!("Got Image object");

                    let image = image_obj.get_processed_image(&document)?;
                    image
                        .save(format!(
                            "{page_n:0>3}_{object_n:0>3}-processed.jpeg"
                        ))
                        .map_err(|_| PdfiumError::ImageError)?;
                }
                PdfPageObject::Shading(_) => println!("Got Shading object"),
                PdfPageObject::FormFragment(_) => println!("Got FormFragment object"),
                PdfPageObject::Unsupported(_) => println!("Got Unsupported object"),
            }
        }
    }

    Ok(())
}

ajrcarey pushed a commit that referenced this issue Oct 2, 2022
@Victor-N-Suadicani
Copy link
Author

Thanks for the quick work! Using the latest master, the color problem has definitely been fixed. I'm able to extract many more images now as well, though I still run into an error later on in my own PDF (image-test.pdf works however). The biggest problem I think is that the images extracted in this way are quite small for some PDFs, much smaller than the highest possible resolution as far as I can tell.

@ajrcarey
Copy link
Owner

ajrcarey commented Oct 3, 2022

Ok, that's good news about the colors. Pdfium returning the wrong image size may be related to #52; in any case, it is definitely a problem in Pdfium, rather than pdfium-render. I propose we mark this issue - which was with the color conversion (or lack thereof) between RGBA and BGRA, and which definitely was a bug in pdfium-render, not Pdfium - as complete, and let's discuss options for working around upstream problems in Pdfium as part of #52.

@ajrcarey
Copy link
Owner

ajrcarey commented Oct 5, 2022

As there have been no further comments on this, and I believe the original problem has been fixed, I am closing this issue. Let's continue discussion about your upstream problems in #52.

Fix for reversed byte order released as part of 0.7.21. Published to crates.io.

@ajrcarey ajrcarey closed this as completed Oct 5, 2022
@ajrcarey ajrcarey self-assigned this Mar 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants