Using Saloon's cursor pagination to process ~72,000 records causes memory to grow linearly

[awebartisan]

Problem

Using Saloon's cursor pagination to process ~72,000 records causes memory to grow linearly (~3MB per 1000-item page) until exhaustion, despite following cleanup best practices.

Environment

• Saloon version: "^3.0"
• saloonphp/laravel-plugin: "^3.5",
• Guzzle version: ^7.8
• PHP version: 8.3.x
• Memory limit: 128M (exhausted at ~22k records)
• Use case: Bulk sync from paginated API (1000 items per page)

Code Structure

class MyApiConnector extends Connector implements HasPagination
{
    public function paginate(Request $request): CursorPaginator
    {
        return new class(connector: $this, request: $request) extends CursorPaginator {
            protected function getNextCursor(Response $response): int|string
            {
                return $response->json('cursor');
            }

            protected function isLastPage(Response $response): bool
            {
                return is_null($response->json('cursor'));
            }

            protected function getPageItems(Response $response, Request $request): array
            {
                // Returns array of DTOs created from response
                return array_map(
                    fn($item) => MyDataDTO::from($item),
                    $response->json('results')
                );
            }
        };
    }
}

Approaches Tried

1. Using ->items() generator (slow processing)

foreach ($connector->paginate($request)->items() as $item) {
    // Individual database operation per item
    MyModel::updateOrCreate(['id' => $item->id], $item->toArray());
}
✅ This worked (no memory issues) but took 19 minutes due to ~140k individual database queries

2. Using ->collect() for bulk processing

$connector->paginate($request)
    ->collect()
    ->chunk(250)
    ->each(fn ($chunk) => MyModel::insert($chunk->toArray()));
❌ Memory grows linearly - appears to accumulate all pages in memory

3. Manual pagination with aggressive cleanup

$cursor = null;
config(['app.debug' => false]); // Disable debug per guzzle/guzzle#1407

do {
    // Fresh connector instance each iteration
    $connector = new MyApiConnector();
    $request = new MyApiRequest();

    if ($cursor) {
        $request->query()->add('cursor', $cursor);
    }

    $response = $connector->send($request);

    // Read entire response as string (not stream)
    $body = $response->body();
    $json = json_decode($body, true);
    $cursor = $json['cursor'] ?? null;
    $items = $json['results'] ?? [];

    // Process in chunks and bulk insert
    foreach (array_chunk($items, 250) as $chunk) {
        $records = array_map(fn($item) => [...], $chunk);
        DB::table('my_table')->insert($records);
    }

    // Aggressive cleanup
    $response->close();
    unset($body, $json, $items, $response, $request, $connector);
    gc_collect_cycles();
    usleep(50000); // 50ms delay to give GC time

} while ($cursor);
❌ Still shows ~2-3 MB memory growth per 1000-record page

Memory Pattern

Start:        58.5 MB
After 1k:     70.5 MB  (+12 MB)
After 5k:     82.5 MB  (+24 MB)
After 10k:    92.5 MB  (+34 MB)
After 22k:   128 MB    (exhausted)

Consistent linear growth regardless of cleanup approach.

Current Workaround

ini_set('memory_limit', '1024M') - allows completion but doesn't scale if dataset grows to 100k+ records.

Questions

1. Is ->collect() expected to keep all pages in memory? The docs state it "only keeps one page in memory at a time"
2. Why does slow processing with ->items() not leak, but fast bulk processing does? Could this be a GC timing issue where fast processing doesn't give GC time to reclaim memory?
3. Are there Saloon-specific configuration options to prevent HTTP response accumulation?

Related

- guzzle/guzzle#1407 - Discusses similar memory accumulation with Guzzle's Pool class