Skip to content
This repository was archived by the owner on Apr 10, 2025. It is now read-only.

Cut out objects with transparent background in the segmenter#47

Open
W7CH wants to merge 4 commits intomainfrom
feature/segmenter
Open

Cut out objects with transparent background in the segmenter#47
W7CH wants to merge 4 commits intomainfrom
feature/segmenter

Conversation

@W7CH
Copy link
Contributor

@W7CH W7CH commented Jul 31, 2024

This PR modifies the output of the segmenter process by cropping objects with transparent background. The previous implementation cropped objects with an entire region of interest from the original image, which often included unnecessary background and parts of other objects. The file affected is processing/segmenter/planktoscope/segmenter/__init__.py to update and add methods for object detection, cropping and saving.

Changes:

  • Define a method _save_object_image to save the object cutout with RGB channels along with the alpha channel (RGBA) for transparency
  • Convert the raw image to RGB mode in the method _open_and_apply_flat
  • Modify the method _stream to be compatible with RGBA mode and PNG file format of the object image
  • Define a method _crop_and_apply_mask that crops the raw image and mask to the bounding box of the detected object to ensure the cutout is as close as possible to the size of the object. This method adds an alpha channel to the cropped image, and sets the alpha channel to the segmentation mask. This will make areas outside the mask completely transparent and areas inside the mask opaque.
  • The nested function __augment_slice is no longer used within the method _slice_image
  • For each object extracted from the raw image, the cropped mask is applied as an alpha channel to create a transparent background. The resulting object image is saved as a PNG file because the JPEG format doesn't support transparency (it ignores the alpha channel info)

Testing:

These changes were tested in an external notebook environment with multiple images to ensure that each detected object is correctly isolated and saved in the expected format. The transparency was preserved in the output PNG files (see example).
replication of result

Instructions for testing this PR on the PlanktoScope

To test new changes to this PR on a PlanktoScope with internet access:

  1. Push your changes in one or more commits up to the feature/segmenter branch
  2. Wait for the CI checks/workflows on this PR to finish rebuilding the Docker container image for the segmenter. You should see a new entry at https://github.com/PlanktoScope/device-backend/pkgs/container/device-backend-processing-segmenter with a label (e.g. sha-55e23d1) corresponding to the latest commit (e.g. 55e23d1 ) which you had pushed to the branch. It should also have the label pr-47 attached.
  3. Go to the Cockpit Terminal of the PlanktoScope (usually this will be at http://planktoscope.local/admin/cockpit/system/terminal, or else you can access Cockpit from the PlanktoScope's landing page and then you can access the terminal from Cockpit's sidebar).
  4. Run the following command: forklift pallet switch github.com/PlanktoScope/pallet-standard@feature/segmentation-transparency (note: the URL given here corresponds to Bump device-pkgs to make segmentation background transparent pallet-standard#15, which deploys the latest pr-47 version of the Docker container image for the segmenter).
  5. Either reboot your PlanktoScope or run: sudo systemctl start forklift-apply. This command will disconnect Cockpit in your browser tab; if you wait ~10 seconds and refresh your browser tab, then you should be able to log in to Cockpit again.

Afterwards, the updated segmenter should be running. You can check its live-updating logs with Dozzle (usually the logs will be at http://planktoscope.local/admin/dozzle/show?name=apps_ps_backend_proc-segmenter-server-1, or else you can access the live-updating logs from a link in the Node-RED dashboard's "Administration" page).

@W7CH W7CH requested a review from tpollina August 1, 2024 15:36
@W7CH
Copy link
Contributor Author

W7CH commented Aug 5, 2024

A few examples of images segmented with transparent background:

Some points are not represented in the objects because they were not detected in the segmentation mask. This behavior had always existed, but segmentation with a region of interest made it possible to correct it.
00_25_56_357156_20_cutout
00_25_56_357156_9_cutout
00_25_54_141982_56_cutout
00_25_51_908646_13_cutout
00_25_49_701693_14_cutout
00_25_45_101719_103_cutout
00_25_45_101719_81_cutout
00_25_45_101719_28_cutout
00_25_42_755452_43_cutout
00_25_42_755452_2_cutout
00_25_40_432352_11_cutout
00_25_40_432352_4_cutout
00_25_38_251991_26_cutout
00_25_38_251991_18_cutout
00_25_35_964166_53_cutout
00_25_35_964166_49_cutout
00_25_35_964166_36_cutout
00_25_35_964166_8_cutout
00_25_35_964166_9_cutout
00_25_33_745790_63_cutout
00_25_33_745790_8_cutout
00_25_31_542437_34_cutout
00_25_31_542437_2_cutout
00_25_29_314502_38_cutout
00_25_29_314502_58_cutout
00_25_29_314502_10_cutout
00_25_27_130201_25_cutout
00_25_29_314502_36_cutout

@ethanjli
Copy link
Member

ethanjli commented Apr 3, 2025

Because nobody's approved this PR in the meantime since I've been making simplifications to github.com/PlanktoScope/pallet-standard, the original instructions for testing this PR are no longer valid. If we want to test this PR in the future, I can set up a new PR in github.com/PlanktoScope/pallet-standard for testing purposes; and the testing instructions will be the same as before.

I think what may be needed for approving/rejecting this PR is just bringing it up for discussion in a general PlanktoScope software meeting. I will add it to the agenda for the next such meeting.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants