Skip to content

Overview

2. 🧠 CPU Compute Needed for Stacking

The most resource-intensive step is Heterogeneous Stacking (WCS Reprojection). This process relies heavily on the CPU and RAM to handle interpolation and alignment of thousands of pixels across many frames.

The general consensus from image processing professionals is: RAM first, then CPU cores, then fast storage.
Component Minimum for Usability Recommended for Efficiency Why?
RAM $16 \text{ GB}$ $\mathbf{32 \text{ GB} +}$ Stacking loads all images into memory for processing. Lack of RAM forces the use of slow disk space.
CPU Intel Core i5 / AMD Ryzen 5 (4-6 Cores) Intel Core i7/i9 or AMD Ryzen 7/9 (8-16 Cores) Stacking is a highly multi-threaded process. More cores means the process scales nearly linearly, dramatically reducing processing time.
Storage $512 \text{ GB}$ SSD $\mathbf{1 \text{ TB} + \text{ NVMe SSD}}$ Necessary for fast loading of image data and writing huge temporary files during the stacking process.

Computational Cost Analogy: Stacking 100 high-resolution images on a minimum-spec machine could take over 90 minutes. Upgrading the CPU core count to the recommended spec can cut that time down to 15–20 minutes for the same data set.

A. The Community Compute Model

Instead of a single server, you can distribute the processing load back to the community:

  1. Centralized Plate Solving: The core WCS (World Coordinate System) processing and metadata extraction must happen centrally (on scalable cloud compute) to ensure quality control and standardization.

  2. Distributed Stacking: Use a simple "Data Package" delivery system. A volunteer signs up to process data, downloads the standardized, WCS-corrected FITS files, and runs a standardized, provided script (e.g., a Python or PixInsight script) to perform the final Reprojection and Co-addition. The final, high-SNR image is then uploaded back to the central archive. This leverages the distributed CPU power of the community.

[!question] Is it cheaper to move the data or is it cheaper to compute it yourself?

Petascale Data Handling (from massive info dump.md)

With hundreds of advanced telescopes (larger detectors, faster cadence), daily raw data volume will be in the Terabytes, leading to Petabytes in archival storage. Below are the strategies for handling this.

Data Transfer and Archival

  • Compression at source: Raw FITS files must be compressed (using FITS compression / fpack) at the telescope node before transfer. Reduces bandwidth requirements by 30–50%.
  • High-bandwidth ingress: The central hub requires cloud-based Object Storage (S3 or Google Cloud Storage) for petabyte-scale data ingest. These services are designed for this scale.
  • Metadata separation: All FITS metadata (WCS, exposure time, weather, photometric zeropoint) should be immediately ingested into a SQL/NoSQL database for fast querying, physically separate from the bulk FITS archive.

Distributed CPU Compute (Stacking Pipeline)

You cannot stack all data on a single machine. The pipeline must be parallelised.

  • Workflow orchestration: Use Apache Spark, Dask, or a dedicated workflow manager (DIRAC, Prefect) to break the pipeline into thousands of small, independent tasks. Reprojection (WCS mapping) is performed simultaneously on all images for a given target, distributed across a cluster of cloud VMs.
  • Containerisation: The processing software (Astropy, Reproject, Photutils) should be packaged in Docker containers. This ensures identical processing environments across hundreds of cloud CPU cores regardless of underlying OS.
  • GPU acceleration: For convolution, deconvolution, and ML tasks (image classification, artifact removal), the pipeline must utilise NVIDIA GPUs. Substantial speed improvement over CPU-only for these workloads.

File Naming Convention

All submitted FITS files must include the following in the filename to make source and arrival order irrelevant:

ObsReq-{UUID}_TelID-{NodeID}_{ISO8601_StartTime}.fits

Example: ObsReq-550_TelID-T42_20251201T013000.fits

This ensures that even if files arrive out of order or from unexpected nodes, the stacking pipeline can correctly attribute and sort them.

Late-Binding Stacking

The stacking process is deliberately delayed until the entire mission is complete.

  1. All submitted images are dumped into a Raw Data Lake (cloud object store).
  2. The stacking pipeline is not triggered by a single file arriving, but by the Status flag in the persistence database changing to COMPLETE.
  3. When the pipeline runs, it collects all images tagged with the mission's UUID — the WCS metadata in each FITS header tells the reprojection algorithm everything it needs to align them, regardless of which telescope provided each frame or when they arrived.