Deployment Case Study: Rubin Observatory
The Vera C. Rubin Observatory, located on Cerro Pachón in Chile, is one of the world’s most ambitious astronomy projects. Its Legacy Survey of Space and Time (LSST) will capture a full digital image of the sky every few nights for ten years, producing an unprecedented volume of optical astronomy data. The observatory’s challenge is not just the telescope, but the massive HPC and data pipeline required to process, store, and distribute the data to scientists worldwide.
Overview
- Location: Cerro Pachón, Chile (2,715 m elevation)
- Telescope: 8.4 m wide-field optical telescope
- Survey: Legacy Survey of Space and Time (LSST), 10 years
- Data Scale: ~20 TB per night, ~7 PB/year, ~60 PB total over survey
- Processing: U.S. DOE + global HPC centers (NCSA Tier-1 hub in Illinois)
- Timeline: First light 2025, full operations ~2026–2036
Data Pipeline
| Stage | Location | Function | Notes |
|---|---|---|---|
| Image Capture | Rubin Observatory, Chile | 3.2 gigapixel camera, 15 TB of raw images/night | Largest optical survey camera in the world |
| Transmission | High-speed fiber, Chile ? U.S. | 40 Gbps international links | Ensures near real-time data delivery |
| Tier-0 Processing | NCSA (Illinois, USA) | Initial reductions, image differencing, nightly alerts | 10 million+ alerts per night to global community |
| Archival Storage | U.S., Chile, international partners | ~60 PB dataset over 10 years | Science-ready database for cosmology and astrophysics |
| Global Access | Astronomy research centers worldwide | Federated access to LSST data products | Democratizes big science data |
HPC, Storage & Networking
- Compute: Tier-0 data center at NCSA, Tier-1/Tier-2 sites in the U.S., Europe, and Chile.
- Storage: Multi-petabyte archive distributed globally, with mirrored sites for redundancy.
- Networking: Dedicated high-speed fiber between Chile and U.S. backbones; 40–100 Gbps sustained throughput.
- Real-Time Processing: Image differencing pipeline must produce alerts within 60 seconds of capture.
Partners & Stakeholders
| Partner | Role |
|---|---|
| U.S. National Science Foundation (NSF) | Primary funding, oversight |
| U.S. Department of Energy (DOE) | Co-funding, HPC infrastructure |
| National Center for Supercomputing Applications (NCSA) | Tier-0 processing hub, Illinois |
| International Partners | Europe, Asia, Latin America supporting Tier-1/Tier-2 centers |
| Rubin Observatory Project Office | Site management, telescope + instrument operations |
Key Challenges
- Data Volume: 20 TB/night must be moved and reduced in near real time.
- Networking: International fiber capacity must scale to handle sustained petabyte transfers.
- Real-Time Alerts: 10 million transient alerts per night require low-latency compute pipelines.
- Storage Growth: Multi-petabyte archives must remain accessible for decades.
- Global Collaboration: Coordinating international Tier-1/Tier-2 centers adds complexity.
Strategic Importance
- Scientific Discovery: Will enable breakthroughs in dark matter, dark energy, time-domain astronomy, and asteroid tracking.
- Data Infrastructure: Demonstrates HPC + storage integration at science scale, akin to AI workloads.
- Technology Transfer: Techniques in streaming, distributed storage, and federated computing have crossovers with AI and enterprise data centers.