Stitching equirectangular images with Neural Style Transfer applied.

[StijnWillemen]

Hi!

I have been using your stitcher for blending equirectangular images with a self-created overlap (by pasting a part of the picture on the opposite side).

Then applying a Neural Style Transfer which uses an input style image and original image to restyle the original image to the input style image as if it was created in that style.

Afterwards I split the image down the middle and swap the pieces so it can be blended.
Using the original unstyled image to find key-features but applying all transformations/blending to the styled image this
generates a very good result, as there is no visible line in the output image on the blended part.

But this creates a new stitch line on the opposite side of the image (in 360° view).
It seems this stitch is created because one of the two images slightly shifted vertically and there also seem to be a kind of fading color effect on the sides of the image which is visible when connecting the sides of the image.
(very bad quality because this screenshot is taken in a 360° web viewer)

Are there any parameters I could change to combat this effect?

Thanks in advance.

[lukasalexanderweber]

I don't know if I followed correctly. I'll try to explain your workflow in my own words:

1. You stitch some images together
2. You apply NST on the result
3. You now want to distribute the image content differently (e.g. put the person from center to left), so you

Afterwards I split the image down the middle and swap the pieces so it can be blended.

meaning to me:

you perform another stitch just to move the content.

---

A better sollution to just to move the content could be the following:

• Decide on which x value you want to swap the content
• Move the right part of your NST to the left and the left part to the right of this x

old_right_new_left = img[:, x:]
old_left_new_right = img[:, :x]
result = np.concatenate((old_right_new_left , old_left_new_right), axis=1)

The code is not tested and I'm not sure if I understood you correctly. Here are my sources:

https://stackoverflow.com/questions/15589517/how-to-crop-an-image-in-opencv-using-python
https://stackoverflow.com/questions/7589012/combining-two-images-with-opencv

[StijnWillemen]

Thanks for the quick response.

The position of elements does not matter as this is for 360° view in Virtual Reality but I split the images and blend them because there is a stitching line where the outer left and outer right side of the image meet when viewed in VR.

Here is a github docs page with images in steps to show the process.
https://modernwayz.github.io/AI-Project/docs/Algemeen.html#sift-advanced

And I use two images the original and the NST image where I use the original image as input for mapping key features and then transform the NST image. I do this because barely any key features can be found on a NST image.

But in the stitching process the camera angle is shifted slightly (I think) and there is some sort of color saturation which causes a new stitching line when bringing the left and right side together.

Sorry for the unclear explanation in my first post.
Here is my forked repository with the changes to use two different input imagesets and example images/experiments.
https://github.com/StijnWillemen/stitching

[lukasalexanderweber]

To me it seems the different color saturation comes from the NST. What about doing the stitch first and NST later?

[lukasalexanderweber]

But you can also play around with blending methods and blend widths. Also make sure that exposure error compensation and seam carving is run on the NST Images and not on the original.

[KaceCM]

Hello guys,

I'm starting again this topic because I'm using stitching library to create an algorithm able to stitch pictures together to create a 360° panorama (to be able to view it in a 360° viewer).

I'm stuck because I don't know at all how to make the beginning and the end of the panorama joinable. I have successfully joined all my pictures, but when it comes to the begining and the end, they are not joinable.

To take your picture in example, we can see that the end of it join almost perfectly the beginning of it.

In my case, this is the beginning of the panorama :

And this is the end of the panorama :

And if we want to join them, here is the result :

Can you help me to fix my problem using your library @lukasalexanderweber ?

Or can you tell me how you did it @StijnWillemen ?

Thanks a lot !

[lukasalexanderweber]

Do you have some resources to 360° viewers and the requirements they have on input Images? I heard a lot of users try to use such viewers but I dont have experience with them.

[StijnWillemen]

It just seems to me like you don't have enough overlap in these pictures, I used a variety of different methods because I had to apply a neural style transfer which removed all SIFT possibilities (because neural style transfer contains random factors).

All my results of the research are present here: https://modernwayz.github.io/AI-Project/ . Unfortunately the explanations are in Dutch (possible to translate with chrome's "translate this page" I hope) , but most result images speak for themselves and there are a lot of useful links to viewers, processing tools and results.

Specifically if there is a huge vertical stitching line you can use something like this https://360-image-viewer-rfqb.vercel.app/ where there are two different images blended on another splitting point and the resulting two images both have a stitching line in a different spot. When looking around in the viewer the images switch when you come close to either one of the lines to hide it from the observers view.
A hacky solution that works for most implementations.

[lukasalexanderweber]

Thanks for the Links! Can you tell us which settings you have used for panorama creation?

[lukasalexanderweber]

There is definetly need for this:

https://stackoverflow.com/questions/69043002/360-equirectangular-image-stitching

https://www.reddit.com/r/opencv/comments/ph1xw8/question_equirectangular_image_stitching_method/

https://answers.opencv.org/question/27176/can-opencv-stitcher-generate-a-full-spherical-panoramas-like-street-view/

It would be great to get this working and provide an example!

[KaceCM]

Yes, when I have the final panorama with your services, I transform it to an equirectangular image, then I transform it to a cubemap to be able to see it on my viewer. The problem is that the beginning of the panorama and the end of the panorama are not stitched (cause your services do it on all image except the beginning and the end)

So i'm wondering how can I do it on the first and last image to be able to stitch them together when I join the beginning and end of the panorama ?

[lukasalexanderweber]

I transform it to an equirectangular image

How do you do that?

I transform it to a cubemap

How do you do that?

Which viewer do you use?

Would a bit of more overlap help between the first and the last Image? I dont get it yet

[KaceCM]

Here is an example.
I did an entire panorama with your service, we can see it there (I blurred it myself, don't worry)

We see that the beginning of the panorama (left) and the end of the panorama (right) cannot be "bound" nor "sewn".

We can imagine that the panorama is positioned around a cylinder, and that the start and end of the panorama will be united on the white line.

If we take the panorama that I took for example above, that is what we will see if we put it around a cylinder:

We don't want that at all, because the beginning and the end are not connected. That's what I would like :

If the beginning and the end are perfectly cut, it would be perfectly positioned around a cylinder without even seeing a delineation.

I hope that it's clear for you !

[lukasalexanderweber]

So you can get your desired result manually by cutting it at the red crosses but it would be nice to have the option that the stitching package cuts it with the information we have, ist this correct?

[lukasalexanderweber]

@KaceCM have you tried cylindrical warper?