Processing Uploads in the Background

Uploaded data shouldn't be sent to Quirrel, but be stored in your own database. Add a new DB entity for it:

model UploadedCSV {
  id          Number @id @default(autoincrement())
  data        String
}

Then when a user uploads something, you insert it into the database and enqueue the resulting record's ID into Quirrel.

// app/mutations/uploadCsvForProcessing
import db from "db"
import csvProcessingQueue from "app/api/process-csv"

export default async function uploadCsvForProcessing(data: string) {
  const record = await db.uploadedCsv.create({
    data: {data},
  })

  await csvProcessingQueue.enqueue(record.id)

  return record.id
}

Our Quirrel Queue then fetches the corresponding data from the database and does the required processing. After that's done, it deletes the database record (alternative: add a flag called "finishedProcessing" and set it to true).

// app/api/process-csv
import db from "db"
import {Queue} from "quirrel/blitz"

export default Queue("api/process-csv", async (uploadId: number) => {
  const upload = await db.uploadedCsv.findUnique({
    where: {id: uploadId},
  })

  await doYourProcessing(upload.data)

  await db.uploadedCsv.delete({where: {id: uploadId}})
})

Now when you want to know whether an upload has already been processed, you can look it up in your own database:

// app/queries/hasFinishedProcessing
import db from "db"

export default async function hasFinishedProcessing(uploadId: number) {
  const count = await db.uploadedCsv.count({where: {uploadId}})
  return count === 0
}