Skip to content

Feature/ingestor management 481#543

Draft
apo-mak wants to merge 5 commits intol5yth:mainfrom
apo-mak:feature/ingestor-management-481
Draft

Feature/ingestor management 481#543
apo-mak wants to merge 5 commits intol5yth:mainfrom
apo-mak:feature/ingestor-management-481

Conversation

@apo-mak
Copy link
Copy Markdown
Contributor

@apo-mak apo-mak commented Dec 8, 2025

This pull request introduces a new ingestor management system, allowing external contributors to register and manage their own API keys for data ingestion, rather than using a shared API token. It adds a dedicated database schema, admin API endpoints, and supporting backend logic for tracking, authenticating, and managing ingestors. Documentation and code changes ensure the feature is configurable and secure.

Based on : #481

Ingestor Management Feature

  • Added INGESTOR_MANAGEMENT and ADMIN_TOKEN environment variables, with documentation explaining how to enable ingestor management and use the new admin API endpoints. [1] [2]
  • Created a new ingestor database schema (data/ingestors.sql) to store ingestor registrations, API keys, metadata, and activity tracking.
  • Implemented App::Ingestors module (web/lib/potato_mesh/application/ingestors.rb) providing methods for ingestor CRUD operations, API key management, and request tracking.
  • Integrated ingestor authentication into the ingestion flow, allowing requests to be authenticated with either the shared API token or individual ingestor API keys, and recording usage statistics.
  • Added admin API endpoints and helper methods for ingestor management, including token validation, key masking, and database upgrades for schema support. [1] [2] [3] [4] [5] [6]

Mesh Ingestor Client Improvements

  • Updated mesh ingestor client to support reporting its version in the request headers for tracking and display in the admin interface. [1] [2] [3]

@apo-mak
init l5yth#481
4733d2f 
Adds ingestors table to database initialization

Ensures the ingestors schema is loaded during both initial database setup and upgrade checks for missing tables, supporting ingestion features.

Relates to l5yth#481
@apo-mak
feat: Add ingestor management functionality
5e5eb90 
- Introduced new API for managing ingestors, including registration, updates, and deletions.
- Added support for individual API keys for ingestors, allowing for more granular access control.
- Implemented version tracking for ingestors in the request headers.
- Created admin routes for listing, creating, updating, and deleting ingestors.
- Enhanced authorization checks for admin and ingestor API keys.
- Added masking functionality for API keys in responses for security.
- Updated configuration to enable or disable ingestor management features.
- Added comprehensive tests for ingestor management functionality.
@apo-mak
Merge branch 'main' into feature/ingestor-management-481
e68c119
@codecov
Copy link
Copy Markdown

codecov bot commented Dec 8, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 97.59036% with 6 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
web/lib/potato_mesh/application/ingestors.rb 95.55% 4 Missing ⚠️
data/mesh_ingestor/queue.py 75.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@apo-mak
test: Add comprehensive test coverage for ingestor management
e2d839e 
- Add HTTP route tests for all admin endpoints
- Add ingestor token authentication tests
- Add Config tests for ingestor_management_enabled? and admin_token
- Add edge case tests for nil/empty parameters
- Add Python tests for X-Ingestor-Version header
@apo-mak
fix: Use warn_log instead of info_log in admin routes and fix flaky test
8bf1b04 
- Replace info_log calls with warn_log (info_log doesn't exist)
- Fix list_ingestors ordering test by using different timestamps
github-advanced-security[bot]
github-advanced-security AI found potential problems Dec 11, 2025
View reviewed changes
Comment thread web/lib/potato_mesh/application/data_processing.rb Fixed
@apo-mak apo-mak force-pushed the feature/ingestor-management-481 branch from 8521094 to 8bf1b04 Compare December 11, 2025 12:38
@l5yth
Copy link
Copy Markdown
Owner

l5yth commented Dec 14, 2025

I would prefer that we discuss significant changes like this before creating a PR.

@apo-mak
Copy link
Copy Markdown
Contributor Author

apo-mak commented Dec 14, 2025 •
edited
Loading

I would prefer that we discuss significant changes like this before creating a PR.

Of course — I agree, and I’m happy to discuss this first.

I noticed this issue was already open: #481
, and thought it was worth tackling. After discussing it within our local community, it became clear that to onboard a large number of users (ingestors), we’ll likely need some form of user management — or perhaps something simpler like API key management.

I don’t have a strong opinion on the exact approach. My main goal is to enable onboarding as many ingestors as possible, with the aim of building a country-wide map view.

P.S. I’m aware this is a significant change, which is why I opened the PR as a draft 🙂

@l5yth l5yth mentioned this pull request Dec 16, 2025
Open
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Feb 6, 2026

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@apo-mak @l5yth @github-advanced-security