Ensure reports rel_object.internal_id are deduplicated

[OctaveLaventure]

Description

The ingestion should deduplicate rel_object.internal_ids, but an account reported duplicated values here

Environment

• OpenCTI version: 6.9.20

Reproducible Steps

No steps provided, analyze the Reports ingestion/creation code and try to find where a rel_object.internal_id field could eventually be created with multiple duplicated values, and to prevent it. If nothing comes up, close this issue until we get an actual repro case.

Expected Output

Actual Output

Additional information

Screenshots (optional)

[nino-filigran]

@OctaveLaventure do you think I can close the issue in the meantime?

[fruitcakej]

Hi @nino-filigran, @OctaveLaventure
Good news, we have the the json files from the source platform.
Pauline helped me add these to a bucket on a system with the stream importer connector installed
We then noted the discrepancy on data.
Can be seen on ticket (screenshots) OCTI1-3145

[fruitcakej]

Update from customer :
In my test environment, the duplication does not happen when the number of workers is just 1. So I believe this is something to do with the workers working in parallel.

[OctaveLaventure]

I believe I identified the source of the duplication issue, working on a fix @nino-filigran

@fruitcakej thank you for that last info, it helped a lot 🙏🏻