Add dvx import-url --git — git-tracked imports with URL provenance

Downloads files via `urllib`, computes MD5, writes `.dvc` with
`meta.git_tracked: true` and URL dep metadata (ETag, Last-Modified,
size, fetch date). Bypasses DVC cache and `.gitignore`.

- `dvx update` detects and handles git-tracked imports separately
- `dvx push`/`pull` skip git-tracked files in `get_transfer_status()`
- `DVCFileInfo.git_tracked` field for downstream consumers (audit, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: ryan-williams

specs/done/git-tracked-imports.md

@@ -0,0 +1,133 @@
+# Git-tracked imports with URL provenance
+
+## Context
+
+Some projects import small files (< 1MB) from HTTP URLs that should be:
+1. **Committed to Git** (small enough, useful for diffing/auditing)
+2. **Tracked by DVX** for provenance: source URL, download date, hash
+
+Current `dvx import-url` pushes files to the DVC remote (S3) and `.gitignore`s them. For small files this is overkill — Git is the better storage backend, and having the file directly in the repo makes diffs, CI, and collaboration simpler.
+
+### Motivating use case
+
+The [NJ crashes project][crashes] imports annual UCR (Uniform Crime Report) Excel files from NJSP:
+
+```
+crime/2018_Uniform_Crime_Report.xlsx          (~150KB)
+crime/2019_Uniform_Crime_Report.xlsx          (~130KB)
+...
+crime/2024-AGENCY-GROUPS-OFFENSE-SUMMARY-REPORT.xlsx  (~200KB)
+```
+
+These are small, rarely-changing files downloaded from government URLs. They should be Git-tracked for easy diffing (e.g. verifying quarterly vs annual 2023 data), but we want DVX to record:
+- **Source URL** (changes across years/site migrations, e.g. `nj.gov/njsp/ucr/` → `njsp.njoag.gov/crime-reports/`)
+- **Download date** (when we fetched it)
+- **Hash** (detect if upstream republishes/corrects data)
+
+[crashes]: https://github.com/hudcostreets/crashes
+
+## Implemented behavior
+
+### `dvx import-url --git` / `-G`
+
+A flag that bypasses DVC's cache/gitignore machinery:
+1. Downloads the file to the specified path (via `urllib`)
+2. Computes MD5, captures HTTP headers (ETag, Last-Modified, Content-Length)
+3. Creates a `.dvc` file with URL provenance and `meta.git_tracked: true`
+4. **Does NOT** add the file to `.gitignore`
+5. **Does NOT** cache the file in DVC
+
+```bash
+dvx import-url --git \
+  https://njsp.njoag.gov/wp/wp-content/uploads/2023%20Uniform%20Crime%20Report.xlsx \
+  -o crime/2023_Uniform_Crime_Report.xlsx
+```
+
+Resulting `.dvc` file:
+
+```yaml
+deps:
+- path: https://njsp.njoag.gov/wp/wp-content/uploads/2023%20Uniform%20Crime%20Report.xlsx
+  checksum: '"etag-value"'
+  size: 153600
+  mtime: '2025-01-30T00:00:00+00:00'   # Last-Modified as ISO 8601
+outs:
+- md5: a1b2c3d4...
+  size: 153600
+  hash: md5
+  path: 2023_Uniform_Crime_Report.xlsx
+meta:
+  git_tracked: true
+  import:
+    fetched: '2026-03-09'
+```
+
+`--no-download` is supported: creates `.dvc` from HEAD request metadata only (no md5 in outs).
+
+### `dvx update` behavior
+
+`dvx update crime/2023_Uniform_Crime_Report.xlsx.dvc`:
+- Detects `meta.git_tracked: true` and uses `update_git_import()` instead of DVC
+- HEAD request to check ETag/Last-Modified
+- If changed: re-download, update `.dvc` hash
+- If unchanged: no-op
+- `--no-download` supported: updates metadata only
+
+### `dvx push` / `dvx pull` behavior
+
+Files with `meta.git_tracked: true` are **skipped** by `get_transfer_status()` — they're in Git, not DVC cache.
+
+## Implementation
+
+### New files
+
+- **`src/dvx/git_import.py`**: Core logic
+  - `git_import_url(url, out, no_download)` — download + create `.dvc`
+  - `update_git_import(dvc_path, no_download)` — re-check and update
+  - `is_git_tracked_import(dvc_path)` — check if `.dvc` is git-tracked
+  - `_ensure_not_gitignored(path)` — remove from `.gitignore` if present
+
+### Modified files
+
+- **`src/dvx/cli/external.py`**: Added `-G`/`--git` flag to `import-url`, git-tracked handling in `update`
+- **`src/dvx/cache.py`**: `get_transfer_status()` skips git-tracked imports
+- **`src/dvx/run/dvc_files.py`**: Added `git_tracked: bool` field to `DVCFileInfo`, read from `meta.git_tracked`
+
+### Design decisions
+
+- **`meta.git_tracked` not `outs[].git_tracked`**: DVC allows arbitrary data in `meta` but rejects unknown keys in `outs`. Putting the flag in `meta` ensures DVC compatibility.
+- **No auto-`git add`**: DVX doesn't generally auto-stage files. The user runs `git add` themselves.
+- **`urllib` not DVC's downloader**: Keeps the git-tracked path completely independent of DVC's stage system, avoiding `.gitignore` and cache side effects.
+
+## Alternatives considered
+
+### Just use a manifest file
+
+A simpler approach: a `crime/sources.yaml` or similar that lists URLs and expected hashes. No DVX integration, just a convention.
+
+**Downside**: No automated freshness checking, no integration with `dvx status`/`dvx update` workflows, duplicates hash tracking that DVX already does.
+
+### `dvx add --import-url`
+
+Instead of a new flag on `import-url`, extend `dvx add` (which already handles git-tracked files) with an optional `--import-url` to record provenance. This might be a cleaner API since `dvx add` already means "track this file" and the URL is just metadata.
+
+```bash
+dvx add crime/2023_Uniform_Crime_Report.xlsx \
+  --import-url https://njsp.njoag.gov/wp/wp-content/uploads/2023%20Uniform%20Crime%20Report.xlsx
+```
+
+This is arguably better since `import-url` currently implies "DVC-managed remote storage". Could be added later as an alias.
+
+## Future work
+
+- `dvx status` upstream freshness checks (ETag/Last-Modified comparison via `--cloud`)
+- Batch import from manifest YAML
+- `dvx add --import-url` alias
+
+## Relationship to other specs
+
+- [http-import-last-modified]: Captures `Last-Modified` in `.dvc` deps — prerequisite for meaningful freshness checks
+- [blob-audit-lineage]: Audit trail for tracked files — `git_tracked` imports participate in lineage tracking
+
+[http-import-last-modified]: ./http-import-last-modified.md
+[blob-audit-lineage]: ../blob-audit-lineage.md

src/dvx/cache.py

@@ -900,8 +900,12 @@ def get_transfer_status(
     file_info = []  # [(dvc_file, data_path, md5, size), ...]
     errors = []
 
+    from dvx.git_import import is_git_tracked_import
+
     for dvc_file in dvc_files:
         try:
+            if is_git_tracked_import(dvc_file):
+                continue  # Git-tracked imports skip DVC cache
             md5, size, _is_dir = _get_output_info(dvc_file)
             data_path = dvc_file[:-4] if dvc_file.endswith(".dvc") else dvc_file
             file_info.append((dvc_file, data_path, md5, size))

src/dvx/cli/external.py

@@ -28,15 +28,31 @@ def import_cmd(url, path, out, rev):
 @click.command("import-url")
 @click.argument("url")
 @click.option("-F", "--fs-config", multiple=True, help="Filesystem config (key=value).")
+@click.option("-G", "--git", is_flag=True, help="Track in Git (not DVC cache). For small files.")
 @click.option("-N", "--no-download", is_flag=True, help="Track metadata only (no download).")
 @click.option("-o", "--out", help="Output path.")
 @click.option("-V", "--version-aware", is_flag=True, help="Track S3 version IDs.")
-def import_url(url, fs_config, no_download, out, version_aware):
+def import_url(url, fs_config, git, no_download, out, version_aware):
     """Import a file from a URL.
 
+    Use --git to commit the file to Git (instead of DVC cache) with URL
+    provenance. Good for small files (< 1MB) you want in the repo.
+
     Use --no-download to track metadata (ETag, size) without downloading.
     Use --fs-config allow_anonymous_login=true for public buckets.
     """
+    if git:
+        from dvx.git_import import git_import_url
+
+        try:
+            dvc_path = git_import_url(url=url, out=out, no_download=no_download)
+            action = "Tracked" if no_download else "Imported"
+            click.echo(f"{action} {url} (git-tracked)")
+            click.echo(f"  {dvc_path}")
+        except Exception as e:
+            raise click.ClickException(str(e)) from e
+        return
+
     fs_config_dict = dict(kv.split("=", 1) for kv in fs_config) if fs_config else None
     try:
         with Repo() as repo:
@@ -67,17 +83,37 @@ def update(no_download, recursive, targets):
 
     Re-checks source ETags and optionally re-downloads if changed.
     """
-    try:
-        with Repo() as repo:
-            repo.update(
-                targets=list(targets),
-                no_download=no_download,
-                recursive=recursive,
-            )
-            for target in targets:
-                click.echo(f"Updated {target}")
-    except Exception as e:
-        raise click.ClickException(str(e)) from e
+    from pathlib import Path
+
+    from dvx.git_import import is_git_tracked_import, update_git_import
+
+    dvc_targets = []
+    for target in targets:
+        dvc_path = Path(target)
+        if not dvc_path.suffix == ".dvc":
+            dvc_path = Path(f"{target}.dvc")
+        if is_git_tracked_import(dvc_path):
+            try:
+                changed = update_git_import(dvc_path, no_download=no_download)
+                status = "updated" if changed else "up to date"
+                click.echo(f"{target}: {status} (git-tracked)")
+            except Exception as e:
+                raise click.ClickException(f"{target}: {e}") from e
+        else:
+            dvc_targets.append(target)
+
+    if dvc_targets:
+        try:
+            with Repo() as repo:
+                repo.update(
+                    targets=dvc_targets,
+                    no_download=no_download,
+                    recursive=recursive,
+                )
+                for target in dvc_targets:
+                    click.echo(f"Updated {target}")
+        except Exception as e:
+            raise click.ClickException(str(e)) from e
 
 
 # =============================================================================