Details on componets
Components¶
Central Server (you run this)
- Target database — every object of interest, with coordinates, magnitude, observation requirements, priority score, campaign membership
- Site registry — every participating telescope, with location, aperture, field of view, filters, mount type, automation level, availability windows
- Scheduler — matches targets to sites based on visibility, priority, weather, site capabilities
- Data ingest — receives uploaded images/photometry, validates format, stores with metadata
- Data products pipeline — runs automated photometry, astrometry, quality flagging on ingested data
- Alert ingest — receives external alerts (ZTF, Gaia, GCN, TNS), creates or updates targets automatically
- User/auth system — API keys for automated systems, accounts for humans
- Web dashboard — shows network status, recent observations, campaign progress, data quality metrics
Client Software (runs at each site)
- Polls server for target list periodically
- Translates targets into local mount commands
- Captures images
- Runs local photometry/astrometry (optional but encouraged)
- Uploads results to server
The client should be:
- Single executable or simple install
- Cross-platform (Windows for amateurs, Linux for professionals)
- Configurable via simple text file (INI or YAML)
- Works with ASCOM (Windows) or INDI (Linux) for hardware abstraction Minimum viable client:
loop forever:
targets = GET /api/targets?site=MYSITE&count=10
for target in targets:
if can_observe(target):
slew(target.ra, target.dec)
image = capture(target.exposure, target.filter)
photometry = extract(image)
POST /api/observations {target, photometry, image_metadata}
sleep(60)
That's it. A competent programmer writes this in a weekend.
Scheduling Algorithm¶
Each target has:
- Base priority (science importance)
- Time sensitivity (deadlines, event windows)
- Observation requirements (filters, cadence, total coverage needed)
- Current coverage (how much data already exists)
Each site has:
- Horizon limits
- Weather status (can be self-reported or ingested from APIs)
- Equipment capabilities
- Historical reliability
Scheduler runs every few minutes:
- Compute visibility windows for all targets from all sites
- For targets needing urgent observations, assign to best available site
- For targets needing long-term coverage, balance across sites to maximise geographic diversity
- Avoid redundant simultaneous coverage unless explicitly needed (e.g., occultations want redundancy; transit photometry wants diversity)
- Return ranked target list per site
Nothing in here requires machine learning or exotic optimisation. Greedy assignment with priority queues gets you 90% of optimal.
Data Model¶
Observation record:
{
observation_id: uuid,
target_id: uuid,
site_id: uuid,
timestamp_utc: datetime,
exposure_seconds: float,
filter: string,
airmass: float,
seeing_arcsec: float (optional),
photometry: {
magnitude: float,
error: float,
comparison_stars: [list of star IDs],
method: string
},
astrometry: {
ra: float,
dec: float,
error_arcsec: float
},
image_url: string (optional, if full image uploaded),
quality_flags: [list of strings],
raw_metadata: {FITS headers or equivalent}
}
Target record:
{
target_id: uuid,
name: string,
aliases: [strings],
ra: float,
dec: float,
target_type: string,
campaigns: [list of campaign IDs],
priority: float,
observation_requirements: {
filters: [strings],
cadence_hours: float,
total_hours_needed: float,
time_critical_windows: [{start, end, priority_boost}]
},
current_coverage: {
total_observations: int,
unique_sites: int,
recent_observations: int (last 7 days)
}
}
Technology Choices¶
For the server:
- Language: Python (FastAPI) or Go. Python has better astronomy library support. Go has better deployment simplicity. I'd lean Python.
- Database: PostgreSQL. It handles everything you need, scales to millions of records, and has good spatial query support for coordinate matching.
- Storage: S3-compatible object storage for images (Backblaze B2 is cheap), database for metadata and photometry.
- Hosting: Single VPS to start (Hetzner, DigitalOcean). Move to managed Kubernetes only when you actually need to.
For the client:
- Language: Python again, for ASCOM/INDI library compatibility and astronomy tools (astropy, photutils, sep).
- Distribution: PyInstaller for Windows executable, pip install for Linux.
Scaling Path¶
Phase 1: 10-50 sites Single server handles everything. Manual campaign management. Email list for coordination.
Phase 2: 50-200 sites Add caching layer for target lists. Automated quality flagging. Web dashboard for observers to see their contributions.
Phase 3: 200-1000 sites Sharded database for observations. Dedicated alert ingestion service. Automated campaign creation from external triggers.
Phase 4: 1000+ sites Regional relay servers to reduce latency. Machine learning for quality control and anomaly detection. Professional operations team.
You can start at Phase 1 and never need Phase 4. Most networks don't grow that big, and that's fine.
THE BACKBONE: COMPLETE IMPLEMENTATION PLAN¶
Philosophy¶
The backbone must be:
- Simple enough that one developer can build the MVP in three months
- Extensible enough that it can scale to thousands of sites
- Robust enough that failure of any component doesn't bring down the network
- Cheap enough that you can run it indefinitely on a shoestring
Architecture Overview¶
┌─────────────────────────────────────────────────────────────────┐
│ CENTRAL SERVER │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │
│ │ Target │ │ Site │ │ Observation │ │
│ │ Database │ │ Registry │ │ Database │ │
│ └─────────────┘ └─────────────┘ └─────────────────────────┘ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │
│ │ Scheduler │ │ Alert │ │ Data Products │ │
│ │ │ │ Ingest │ │ Pipeline │ │
│ └─────────────┘ └─────────────┘ └─────────────────────────┘ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ REST API ││
│ └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
│
│ HTTPS
│
┌─────────────────────────┼─────────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Site A │ │ Site B │ │ Site C │
│ Client │ │ Client │ │ Client │
└─────────┘ └─────────┘ └─────────┘
Database Schema (PostgreSQL)¶
-- Sites in the network
CREATE TABLE sites (
site_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
owner_name VARCHAR(255),
owner_email VARCHAR(255),
latitude DOUBLE PRECISION NOT NULL,
longitude DOUBLE PRECISION NOT NULL,
elevation_m DOUBLE PRECISION,
timezone VARCHAR(64),
aperture_mm INTEGER,
focal_length_mm INTEGER,
camera_model VARCHAR(128),
filters JSONB, -- ["B", "V", "R", "I", "Clear"]
fov_arcmin DOUBLE PRECISION,
limiting_mag DOUBLE PRECISION,
automation_level VARCHAR(32), -- 'manual', 'semi-auto', 'robotic'
is_active BOOLEAN DEFAULT true,
api_key_hash VARCHAR(128),
created_at TIMESTAMP DEFAULT NOW(),
last_seen TIMESTAMP
);
-- Targets to observe
CREATE TABLE targets (
target_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
ra_deg DOUBLE PRECISION NOT NULL, -- J2000
dec_deg DOUBLE PRECISION NOT NULL,
target_type VARCHAR(64), -- 'variable', 'exoplanet', 'asteroid', 'transient', etc.
magnitude DOUBLE PRECISION,
priority DOUBLE PRECISION DEFAULT 1.0,
cadence_hours DOUBLE PRECISION, -- desired observation cadence
min_observations INTEGER, -- minimum obs needed
active BOOLEAN DEFAULT true,
metadata JSONB, -- arbitrary additional info
created_at TIMESTAMP DEFAULT NOW(),
expires_at TIMESTAMP
);
-- Campaigns group targets with common goals
CREATE TABLE campaigns (
campaign_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
description TEXT,
start_date DATE,
end_date DATE,
priority_boost DOUBLE PRECISION DEFAULT 1.0,
is_active BOOLEAN DEFAULT true
);
CREATE TABLE campaign_targets (
campaign_id UUID REFERENCES campaigns,
target_id UUID REFERENCES targets,
PRIMARY KEY (campaign_id, target_id)
);
-- Individual observations
CREATE TABLE observations (
obs_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
target_id UUID REFERENCES targets,
site_id UUID REFERENCES sites,
observed_at TIMESTAMP NOT NULL,
exposure_sec DOUBLE PRECISION,
filter VARCHAR(16),
airmass DOUBLE PRECISION,
seeing_arcsec DOUBLE PRECISION,
magnitude DOUBLE PRECISION,
mag_error DOUBLE PRECISION,
comparison_stars JSONB,
quality_flags VARCHAR(64)[],
fits_header JSONB,
image_path VARCHAR(512),
created_at TIMESTAMP DEFAULT NOW()
);
-- External alerts ingested
CREATE TABLE alerts (
alert_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
source VARCHAR(64), -- 'TNS', 'GCN', 'ZTF', 'Gaia', etc.
external_id VARCHAR(128),
ra_deg DOUBLE PRECISION,
dec_deg DOUBLE PRECISION,
alert_type VARCHAR(64),
priority DOUBLE PRECISION,
payload JSONB,
received_at TIMESTAMP DEFAULT NOW(),
target_id UUID REFERENCES targets -- link if we created a target
);
-- Index for fast spatial queries
CREATE INDEX idx_targets_coords ON targets USING GIST (
ll_to_earth(dec_deg, ra_deg)
);
CREATE INDEX idx_sites_coords ON sites USING GIST (
ll_to_earth(latitude, longitude)
);
API Endpoints¶
GET /api/v1/targets
?site_id={uuid} # filter to visible from this site
&count={n} # how many targets to return
&priority_min={float} # minimum priority
&campaign={uuid} # filter by campaign
POST /api/v1/observations # submit observation data
Body: {target_id, site_id, observed_at, magnitude, ...}
GET /api/v1/sites # list all sites
GET /api/v1/sites/{site_id} # get site details
POST /api/v1/sites # register new site
PUT /api/v1/sites/{site_id}/heartbeat # site is online
GET /api/v1/campaigns # list campaigns
GET /api/v1/campaigns/{id} # campaign details and targets
POST /api/v1/alerts # submit external alert (for ingestion services)
GET /api/v1/observations # query observations
?target_id={uuid}
&site_id={uuid}
&after={datetime}
&before={datetime}
GET /api/v1/lightcurve/{target_id} # get aggregated light curve data
Scheduler Logic (Python pseudocode)¶
def get_targets_for_site(site_id: UUID, count: int) -> List[Target]:
site = get_site(site_id)
now = datetime.utcnow()
# Get all active targets
targets = db.query(Target).filter(Target.active == True).all()
scored = []
for target in targets:
# Is it visible from this site right now?
alt, az = compute_altaz(target.ra, target.dec, site.lat, site.lon, now)
if alt < 30: # below horizon limit
continue
# Base score from priority
score = target.priority
# Boost for campaign membership
for campaign in target.campaigns:
if campaign.is_active:
score *= campaign.priority_boost
# Boost for targets needing observations (coverage deficit)
recent_obs = count_recent_observations(target.target_id, hours=24)
if target.cadence_hours and recent_obs < (24 / target.cadence_hours):
score *= 1.5
# Boost for targets at optimal airmass
if alt > 60:
score *= 1.2
# Penalty for targets already observed recently from this site
site_recent = count_site_observations(target.target_id, site_id, hours=4)
if site_recent > 0:
score *= 0.5
scored.append((score, target))
# Sort by score descending
scored.sort(key=lambda x: -x[0])
return [t for _, t in scored[:count]]
Client Software (Python)¶
#!/usr/bin/env python3
"""
DATAS Client - Minimal implementation
Polls server for targets, observes them, uploads results.
"""
import requests
import time
import yaml
from astropy.coordinates import EarthLocation, AltAz, SkyCoord
from astropy.time import Time
import astropy.units as u
# Load config
with open('config.yaml') as f:
config = yaml.safe_load(f)
SITE_ID = config['site_id']
API_KEY = config['api_key']
API_URL = config['api_url']
# Hardware abstraction - implement for your setup
from hardware import mount, camera, focuser # You implement these
def get_targets(count=10):
"""Fetch target list from server."""
resp = requests.get(
f"{API_URL}/api/v1/targets",
params={'site_id': SITE_ID, 'count': count},
headers={'Authorization': f'Bearer {API_KEY}'}
)
resp.raise_for_status()
return resp.json()['targets']
def submit_observation(obs_data):
"""Upload observation to server."""
resp = requests.post(
f"{API_URL}/api/v1/observations",
json=obs_data,
headers={'Authorization': f'Bearer {API_KEY}'}
)
resp.raise_for_status()
return resp.json()
def is_observable(target, location):
"""Check if target is above horizon."""
coord = SkyCoord(ra=target['ra_deg']*u.deg, dec=target['dec_deg']*u.deg)
now = Time.now()
altaz = coord.transform_to(AltAz(obstime=now, location=location))
return altaz.alt.deg > 30
def observe_target(target):
"""Slew to target, take image, extract photometry."""
mount.slew(target['ra_deg'], target['dec_deg'])
mount.wait_until_settled()
exposure = target.get('exposure_sec', 60)
filter_name = target.get('filter', 'V')
camera.set_filter(filter_name)
image = camera.expose(exposure)
# Run photometry
photometry = extract_photometry(image, target)
return {
'target_id': target['target_id'],
'site_id': SITE_ID,
'observed_at': Time.now().isot,
'exposure_sec': exposure,
'filter': filter_name,
'magnitude': photometry['magnitude'],
'mag_error': photometry['error'],
'comparison_stars': photometry['comp_stars'],
'airmass': photometry['airmass']
}
def main_loop():
"""Main observation loop."""
location = EarthLocation(
lat=config['latitude']*u.deg,
lon=config['longitude']*u.deg,
height=config['elevation']*u.m
)
while True:
# Get target list
targets = get_targets(count=20)
for target in targets:
if not is_observable(target, location):
continue
try:
obs = observe_target(target)
submit_observation(obs)
print(f"Observed {target['name']}: {obs['magnitude']:.3f} mag")
except Exception as e:
print(f"Failed to observe {target['name']}: {e}")
# Wait before next cycle
time.sleep(60)
if __name__ == '__main__':
main_loop()
Technology Stack¶
Server:
- Python 3.11+ with FastAPI
- PostgreSQL 15+ with PostGIS extension
- Redis for caching target lists
- S3-compatible storage for images (Backblaze B2 = $0.005/GB/month)
- Caddy for reverse proxy and automatic HTTPS
Client:
- Python 3.9+ (for broad compatibility)
- PyInstaller for Windows distribution
- ASCOM/Alpaca for Windows hardware
- INDI for Linux hardware
- astropy for coordinate calculations
Hosting (Phase 1):
- Hetzner Cloud CX21: €5.49/month (2 vCPU, 4GB RAM, 40GB SSD)
- Backblaze B2: ~$5/month for 1TB storage
- Domain + Cloudflare: ~$15/year
Total: ~$15/month to start.