feat(workers): update workers and event producer for feature list changes

[jcscottiii]

feat: Report query errors in feature list diffs and notifications

This PR introduces the capability to capture, persist, and report errors encountered when executing search queries during the feature list diffing process. This ensures that users are notified when their saved searches or queries become invalid (e.g., due to deleted dependencies or cycles).

Key Changes:

• Core Diffing: Updated FeatureDiffer to capture specific non-fatal errors (like ErrSavedSearchNotFound) during feature fetching, instead of failing the entire run.
• Refactored FeatureFetcher Interface: Updated FetchFeatures to return a structured *FetchFeaturesResult containing both features and optional UserError objects, encapsulating backend error interpretation inside the adapter layer.
• Schema Updates: Added QueryErrors field to StateMetadata (v1) as a regular slice of QueryError using Go 1.24's omitzero tag, balancing consistency with EventID (which uses omitempty) and the project's use of omitzero for optional struct fields.
• Notifications:
  • Email: Added a warning banner in the email digest template to display query errors.
  • Slack: Updated the Slack payload builder to append query error blocks to the message.
• Workflow: Added logic to infer machine-readable error codes from error messages for better client handling.
• Linter and Architectural Governance: Enabled the depguard linter in .golangci.yaml and added rules to prevent package coupling. Specifically, prevented workers/*/pkg/** and lib/workertypes/** from importing spanneradapters, enforcing the Staff SWE pattern of orchestrated error handling and clean service boundaries.

Rationale for Multi-Layer Enums and "Duplication"

Reviewers may notice seemingly duplicate definitions of query error codes across different packages:

1. lib/workertypes/comparables: Used for internal diffing logic.
2. lib/workertypes: Used for summary representations.
3. lib/blobtypes/featurelist/v1 and featurelistdiff/v1: Used for permanent storage in diff snapshots.

Why are they duplicated?
This is an intentional application of the Multi-Layer Enum Strategy to ensure:

• Service Boundaries: We decouple internal pipeline logic from the external storage schema.
• Serialization Safety: Services interacting with these snapshots should not depend on the internal types of the differ worker.
• Explicit Translation: We use exhaustive switch statements (enforced by the linter) to map between these layers, ensuring that adding a new error code forces updates across all layers without silent failures or brittle type casting.

[jcscottiii]

FYI: For those new to Go. This is how you implement the error interface

[DanielRyanSmith]

Gemini finding:

Major Logic Error in executePlan

In workers/event_producer/pkg/differ/differ.go, the executePlan method fetches features for the previous query when plan.QueryChanged is true:

if plan.QueryChanged {
	result, err := d.client.FetchFeatures(ctx, plan.PreviousQuery)
	if err == nil {
		data.TargetSnapshot = comparables.NewFeatureMapFromBackendFeatures(result.Features)
	} else {
		// Fallback: If old query fails, we return nil TargetSnapshot.
		// Run() detects this and skips diffing, treating it as a silent reset.
		return executionData{
			OldSnapshot:    nil,
			TargetSnapshot: nil,
			NewSnapshot:    data.NewSnapshot,
			QueryErrors:    data.QueryErrors,
			SnapshotOrigin: data.SnapshotOrigin,
		}, nil
	}
}

The Bug: The interface for FetchFeatures was changed to return (*workertypes.FetchFeaturesResult, error). For "user errors" (like ErrSavedSearchNotFound, invalid queries, etc.), FetchFeatures now returns a valid *FetchFeaturesResult with a populated UserError field, but the err return value is nil.

Because err is nil, the code will mistakenly enter the if err == nil block rather than the else fallback block. Since result.Features will be empty/nil when there is a UserError, TargetSnapshot becomes an empty map (map[]) rather than nil.

The Impact: Instead of skipping diffing and silently resetting (the intended fallback behavior for a failed previous query), the diff engine will compare the new features against an empty TargetSnapshot. This will erroneously report every feature in the new query as "Added" and spam users with notifications about "new" features.

The Fix: You need to check if result.UserError == nil in addition to err == nil:

if plan.QueryChanged {
	result, err := d.client.FetchFeatures(ctx, plan.PreviousQuery)
	if err == nil && result.UserError == nil {
		data.TargetSnapshot = comparables.NewFeatureMapFromBackendFeatures(result.Features)
	} else {
		// Fallback ...

[jcscottiii]

@DanielRyanSmith good catch! fixed!

[DanielRyanSmith]

LGTM!