Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 134 additions & 5 deletions docs/plugins/import-export.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ Each item in the `collections` array can have the following properties:
| `disableJobsQueue` | boolean | Run exports synchronously for this collection. |
| `disableSave` | boolean | Disable save button for this collection. |
| `format` | string | Force format (`csv` or `json`) for this collection. |
| `hooks` | object | Lifecycle hooks — `before` and `after`. See [Hooks](#hooks). |
| `limit` | `number \| function` | Maximum documents to export. Set to `0` for unlimited (default). Overrides global `exportLimit`. |
| `overrideCollection` | function | Override the export collection config for this specific target. |

Expand All @@ -138,6 +139,7 @@ Each item in the `collections` array can have the following properties:
| `batchSize` | number | Documents per batch during import. Default: `100`. |
| `defaultVersionStatus` | string | Default status for imported docs (`draft` or `published`). |
| `disableJobsQueue` | boolean | Run imports synchronously for this collection. |
| `hooks` | object | Lifecycle hooks — `before` and `after`. See [Hooks](#hooks). |
| `limit` | `number \| function` | Maximum documents to import. Set to `0` for unlimited (default). Overrides global `importLimit`. |
| `overrideCollection` | function | Override the import collection config for this specific target. |

Expand Down Expand Up @@ -366,15 +368,130 @@ importExportPlugin({
})
```

## Hooks

Collection-level hooks let you intercept each batch of documents before it is written (to file on export, or to the database on import) and after it has been written. Hooks fire once per batch and work for both `csv` and `json` formats.

### Hook arguments

All hooks receive:

| Argument | Type | Description |
| -------------- | --------------------------- | -------------------------------------------------------------------- |
| `batchNumber` | number | Current batch, starting at `1`. |
| `data` | `Record<string, unknown>[]` | Transformed batch — flat rows for CSV export, nested docs otherwise. |
| `format` | string | Export/import format. Open-ended (`'csv' \| 'json' \| string`). |
| `originalData` | array | Source data before transformation. Read-only — do not mutate. |
| `req` | `PayloadRequest` | The full request object. |
| `totalBatches` | number | Total number of batches in this operation. |

`ImportAfterHook` additionally receives `result: ImportResult` (per-batch counts and errors) instead of `data`/`originalData`.

### before hooks — modify data

`before` hooks return the (optionally modified) `data` array, which replaces the batch going into the write step. Returning an empty array skips the write for that batch without aborting remaining batches.

```ts
import { importExportPlugin } from '@payloadcms/plugin-import-export'

importExportPlugin({
collections: [
{
slug: 'users',
export: {
hooks: {
// Mask sensitive fields before the batch is written to file
before: ({ data, format }) => {
return data.map((row) => {
const { passwordHash: _ph, ssn: _ssn, ...safe } = row
return safe
})
},
},
},
import: {
hooks: {
// Normalise incoming data before it is written to the database
before: ({ data }) => {
return data.map((doc) => ({
...doc,
email:
typeof doc.email === 'string'
? doc.email.toLowerCase()
: doc.email,
}))
},
},
},
},
],
})
```

### after hooks — logging and observability

`after` hooks fire after the write has completed. Their return value is ignored — use them for logging, metrics, or notifications.

```ts
importExportPlugin({
collections: [
{
slug: 'orders',
export: {
hooks: {
after: ({ batchNumber, totalBatches, data, format, req }) => {
req.payload.logger.info({
msg: `Export batch ${batchNumber}/${totalBatches} written`,
format,
docsInBatch: data.length,
})
},
},
},
import: {
hooks: {
after: ({ batchNumber, totalBatches, result, req }) => {
req.payload.logger.info({
msg: `Import batch ${batchNumber}/${totalBatches} complete`,
imported: result.imported,
updated: result.updated,
errors: result.errors.length,
})
},
},
},
},
],
})
```

### Using originalData for context

`originalData` gives you the raw source before any transformation. For exports it is the original DB documents; for imports it is the raw parsed file rows. This is useful when you need context from the full document while building modified flat rows.

```ts
export: {
hooks: {
before: ({ data, originalData }) => {
return data.map((row, i) => ({
...row,
// Add a computed column not present in the DB document
displayName: `${originalData[i]?.firstName} ${originalData[i]?.lastName}`,
}))
},
},
},
```

## Field Options

In addition to the above plugin configuration options, you can granularly set the following field level options using the `custom['plugin-import-export']` properties in any of your collections.

| Property | Type | Description |
| ---------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `disabled` | boolean | When `true` the field is completely excluded from the import-export plugin. |
| `toCSV` | function | Custom function used to modify the outgoing CSV data by manipulating the data, siblingData or by returning the desired value. |
| `fromCSV` | function | Custom function used to transform incoming CSV data during import. |
| Property | Type | Description |
| ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `disabled` | boolean | When `true` the field is completely excluded from the import-export plugin. |
| `toCSV` | function | **Deprecated since v4** — use `export.hooks.before` instead. Custom function to modify outgoing CSV data for a single field. |
| `fromCSV` | function | **Deprecated since v4** — use `import.hooks.before` instead. Custom function to transform incoming CSV data for a single field. |

### Disabling Fields

Expand All @@ -400,6 +517,12 @@ When a field is disabled:

### Customizing Export Data with toCSV

<Banner type="warning">
Comment thread
paulpopus marked this conversation as resolved.
Outdated
`toCSV` is **deprecated since v4**. Use `export.hooks.before` instead — it
works for both CSV and JSON formats and gives you access to the full document
batch. See [Hooks](#hooks).
</Banner>

To manipulate the data that a field exports, you can add `toCSV` custom functions. This allows you to modify the outgoing CSV data by manipulating the row object or by returning the desired value.

The `toCSV` function receives an object with the following properties:
Expand Down Expand Up @@ -445,6 +568,12 @@ const pages: CollectionConfig = {

### Customizing Import Data with fromCSV

<Banner type="warning">
`fromCSV` is **deprecated since v4**. Use `import.hooks.before` instead — it
works for both CSV and JSON formats and gives you access to the full document
batch. See [Hooks](#hooks).
</Banner>

To transform data during import, add `fromCSV` custom functions. This allows you to transform incoming CSV data before it's saved to the database.

The `fromCSV` function receives an object with the following properties:
Expand Down
119 changes: 103 additions & 16 deletions packages/plugin-import-export/src/export/batchProcessor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
*/
import type { PayloadRequest, SelectType, Sort, TypedUser, Where } from 'payload'

import type { ExportAfterHook, ExportBeforeHook } from '../types.js'

import { type BatchProcessorOptions } from '../utilities/useBatchProcessor.js'

/**
Expand Down Expand Up @@ -46,6 +48,11 @@ export interface ExportProcessOptions<TDoc = unknown> {
* The export format - affects column tracking for CSV
*/
format: 'csv' | 'json'
/** Lifecycle hooks for this export operation */
hooks?: {
after?: ExportAfterHook
before?: ExportBeforeHook
}
/**
* Maximum number of documents to export
*/
Expand All @@ -58,6 +65,8 @@ export interface ExportProcessOptions<TDoc = unknown> {
* Starting page for pagination (default: 1)
*/
startPage?: number
/** Total number of docs available (used to compute totalBatches for hooks) */
totalDocs?: number
/**
* Transform function to apply to each document
*/
Expand Down Expand Up @@ -115,7 +124,22 @@ export function createExportBatchProcessor(options: ExportBatchProcessorOptions
const processExport = async <TDoc>(
processOptions: ExportProcessOptions<TDoc>,
): Promise<ExportResult> => {
const { findArgs, format, maxDocs, req, startPage = 1, transformDoc } = processOptions
const {
findArgs,
format,
hooks,
maxDocs,
req,
startPage = 1,
totalDocs,
transformDoc,
} = processOptions

const effectiveDocs =
totalDocs !== undefined
? Math.min(totalDocs, maxDocs === Number.POSITIVE_INFINITY ? totalDocs : maxDocs)
: 0
const totalBatches = effectiveDocs > 0 ? Math.ceil(effectiveDocs / batchSize) : 1

const docs: Record<string, unknown>[] = []
const columnsSet = new Set<string>()
Expand Down Expand Up @@ -144,13 +168,28 @@ export function createExportBatchProcessor(options: ExportBatchProcessorOptions
)
}

for (const doc of result.docs) {
const transformedDoc = transformDoc(doc as TDoc)
docs.push(transformedDoc)
const batchNumber = currentPage - startPage + 1
const originalDocs = result.docs as Record<string, unknown>[]
const batchData = result.docs.map((doc) => transformDoc(doc as TDoc))

const dataToWrite =
hooks?.before && batchData.length > 0
? await hooks.before({
batchNumber,
data: batchData,
format,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
originalData: originalDocs as any,
req,
totalBatches,
})
: batchData

for (const row of dataToWrite) {
docs.push(row)

// Track columns for CSV format
if (format === 'csv') {
for (const key of Object.keys(transformedDoc)) {
for (const key of Object.keys(row)) {
if (!columnsSet.has(key)) {
columnsSet.add(key)
columns.push(key)
Expand All @@ -159,6 +198,17 @@ export function createExportBatchProcessor(options: ExportBatchProcessorOptions
}
}

if (hooks?.after && dataToWrite.length > 0) {
await hooks.after({
batchNumber,
data: dataToWrite,
format,
originalData: originalDocs,
req,
totalBatches,
})
}

fetched += result.docs.length
hasNextPage = result.hasNextPage && fetched < maxDocs
currentPage++
Expand Down Expand Up @@ -187,7 +237,22 @@ export function createExportBatchProcessor(options: ExportBatchProcessorOptions
async function* streamExport<TDoc>(
processOptions: ExportProcessOptions<TDoc>,
): AsyncGenerator<{ columns: string[]; docs: Record<string, unknown>[] }> {
const { findArgs, format, maxDocs, req, startPage = 1, transformDoc } = processOptions
const {
findArgs,
format,
hooks,
maxDocs,
req,
startPage = 1,
totalDocs,
transformDoc,
} = processOptions

const effectiveDocs =
totalDocs !== undefined
? Math.min(totalDocs, maxDocs === Number.POSITIVE_INFINITY ? totalDocs : maxDocs)
: 0
const totalBatches = effectiveDocs > 0 ? Math.ceil(effectiveDocs / batchSize) : 1

const columnsSet = new Set<string>()
const columns: string[] = []
Expand Down Expand Up @@ -215,15 +280,26 @@ export function createExportBatchProcessor(options: ExportBatchProcessorOptions
)
}

const batchDocs: Record<string, unknown>[] = []

for (const doc of result.docs) {
const transformedDoc = transformDoc(doc as TDoc)
batchDocs.push(transformedDoc)

// Track columns for CSV format
const batchNumber = currentPage - startPage + 1
const originalDocs = result.docs as Record<string, unknown>[]
const batchData = result.docs.map((doc) => transformDoc(doc as TDoc))

const dataToWrite =
hooks?.before && batchData.length > 0
? await hooks.before({
batchNumber,
data: batchData,
format,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
originalData: originalDocs as any,
req,
totalBatches,
})
: batchData

for (const row of dataToWrite) {
if (format === 'csv') {
for (const key of Object.keys(transformedDoc)) {
for (const key of Object.keys(row)) {
if (!columnsSet.has(key)) {
columnsSet.add(key)
columns.push(key)
Expand All @@ -232,7 +308,18 @@ export function createExportBatchProcessor(options: ExportBatchProcessorOptions
}
}

yield { columns: [...columns], docs: batchDocs }
yield { columns: [...columns], docs: dataToWrite }

if (hooks?.after && dataToWrite.length > 0) {
await hooks.after({
batchNumber,
data: dataToWrite,
format,
originalData: originalDocs,
req,
totalBatches,
})
}

fetched += result.docs.length
hasNextPage = result.hasNextPage && fetched < maxDocs
Expand Down
Loading
Loading