high throughput processing of space debris data

High-Throughput Processing of Space Debris Data

Andreas Schreiber, Michael MeinelGerman Aerospace Center (DLR)

Simulation and Software Technology,Berlin / Cologne, Germany

PyData Seattle

July 26, 2015

> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 1

• Space Debris

• BACARDI

• Skynet

Outline


Major research areas of DLR

• Aeronautics

• Space

• Transportation

• Energy

• Security

Software research and dev.

• Software engineering, HPC,distributed systems, …

DLRAerospace Research Center, Space Agency


http://de.wikipedia.org/wiki/Thomas_Reiter


Space Debris


Lower Orbit300 – 2000 km

Geostationary Orbit≈36,000 km

Space Flight


• Almost 5,000 rockets launched

• More than 6,000 satellites placed in orbit

• About 1,000 active satellites today

(that’s 7% of known objects in orbit)

About 50 Years of Space Flight


Defunct objects in space

• Disused satellites & rocket stages

• Fragments from explosions & collisions

• Released object (tools etc.)

• Slag of solid-fueled rockets

• … and other sources of small particles

Space Debris


Space Debris


January 2007: FengYun-1C

February 2009: CollisionIridium-33 & Kosmos-2251

Space Debris


Iridium-33 & Kosmos-2251:Debris fields after 50 minutes

Fengyun-1C debris one month after its disintegration

16,300 Objects in Catalogue


29,000 Objects Larger than 10 cm


750,000 Objects Larger than 1 cm


150M Objects Larger than 1 mm


ImpactLaboratory Impact Aluminum Sphere / Block


ImpactReal Debris Object on a Satellite Solar Panel


Distribution of Space Debris


Source: NASA Orbital Debris Quarterly News 2/2012

ISS

Many Earth ObservationSatellites (e.g., Envisat)

TerraSAR-X

Collision PredictionInternational Space Station

DLR.de • Chart 18 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015


http://rbth.com/news/2015/07/26/iss_successfully_ducks_from_space_debris_48023.html

Detecting Space Debris


Sources: FhG FHR, AIUB, ESA

Optical Radar Laser

SMARTnet

• Theory: complete coverage with 3 locations

• Northern / southern hemisphere for compensating seasonal variations (6 locations)

• Telerobotical operation

• Optimized scheduler for all telescopes

Operational Collision Avoidance Global network for monitoring the geostationary ring


Operational Collision Avoidance Global network for monitoring the geostationary ring


Object Identification with Optical Measurements

DLR.de • Chart 24 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015

• Separation of real / false tracklets above threshold of loss function (chi-squared distribution)

• Filter rate depends on accuracy, time difference, survey strategy, …

Object Correlation



BACARDI

Objective

• Database with preferably high completeness and high accuracy

• Primary source: sensor data and operator data

• Secondary source: externally generated ephemerides

BACARDIBackbone Catalogue of Relational Debris Information


Science and Research

• Database of more than 1M objects

• High-performance computational… • object correlation• orbit determination• propagation• object identification• detection of maneuvers and

fragmentations



Mission Support

• Orbit information

• Collision prediction

• Re-entry prediction

Real-time operation



BACARDI Overview


BACARDI Overview


Sensor network

BACARDI

Sensorscheduler

UserGroups

External orbital / object information

Optical Radar Laser

Database

User Interface

Processes

BACARDIInternal Nodes (Processes)


External Data SourcesExample: space-track.org


BACARDISystem Components


RDBMS

Cache

Import

Processing

Python

FORTRAN

Middlew

are

Security

Traceability

• Simple• Parallel• Fast• Huge number

Export


Skynet

Specialized middleware with following objectives

• Decentralized message queues

• Scalable, self-organizing network

• Minimal network overhead

• Platform independent• Operation: Linux• Development: Windows, Mac OSX

• Automatic provenance recording

SkynetNetwork for surveillance of the sky


ZeroMQ

• Decentralized network infrastructure and messaging

Protocol Buffers

• Highly efficient serialization of data

SQLAlchemy

• Database mapping

ZeroMQ and Protocol Buffers are available for a wide range of platforms and programming languages.

SkynetTechnology Decisions


SkynetModules


skyn

et

skynet.network Messaging

skynet.model Data Model

skynet.process Processes

skynet.record Provenance

• Socket library for messaging

• Multi-platform, multi-language

• Fast and small

• Many connection patterns

ZeroMQwww.zeromq.org


• Network layer based on ZeroMQ

• „Device“ as abstract endpoint

• XML configuration: ZDCF (ZeroMQ Device Configuration File)

• Automatically connected to other matching „devices“ (Peer-to-Peer)

• Compression of large data packets

• (De-)serialization via Protocol Buffer messages

• Prepared for encryption and signing

• Synchronous or asynchronous

skynet.network


skynet.network


• Specified by Google (https://github.com/google/protobuf)

• Serializing (encoding) structured data

• Efficient and extensible

• Language-neutral, platform-neutral

• Interface definition language to describe data structure

Protocol Buffers (“ProtoBuf”)


message Person { required string name = 1; required int32 id = 2; optional string email = 3;}

• All models are defined as Protocol Buffer messages

• Provides basic data type as ProtoBuf (e.g., 3D vector, 6D vector, orbital data, …)

• Interfaces for caching

• ORM for ProtoBuf messages with SQLAlchemy (database-neutral)

• Very few additional annotation• Compiled to Python module using protoc

• skynet.network device for database connection

skynet.model


If a model should be mapped to a database table, it should have the db_table message option extension set

skynet.modelModel Definition


// Commonly used message types.package bacardi.model.common;

import "skynet/model/proto/options.proto";

// A spatial vector with x, y and z double componentsmessage vec3d { required double x = 1; required double y = 2; required double z = 3;}

skynet.model


BACARDI Network Message

HEADER BLOCK 1 BLOCK 2 BLOCK N NULL…

DEST NULL SOURCE NULL… …

TYPE COMMAND NULL [SIG]… [KEY]

// Orbits and orbital parameters.package bacardi.model.orbit;

import "skynet/model/proto/options.proto";import "bacardi/model/common.proto";

// One entry of a propagation error dataset.message properr_item { required double days_prop = 1; required double altitude = 2; required double mean_f107 = 3; required common.vec6d std_state_rtn = 4;}

// Propagation error dataset.message properr { option (skynet.model.db_table) = { name: "properr"; }; required uint64 id = 1; repeated properr_item err = 2; optional common.comment comment = 3; required common.permit export_permit = 4;}

// Two-Line Element orbit dataset.message tle { option (skynet.model.db_table) = { name: "tle"; }; required uint64 id = 1; optional uint32 norad_id = 2 [(skynet.model.db_column).index = true]; optional string cospar_name = 3 [(skynet.model.db_column).length = 10, (skynet.model.db_column).index = true]; optional string norad_name = 4 [(skynet.model.db_column).length = 69]; optional string classification = 5 [(skynet.model.db_column).length = 1]; optional uint32 elem_num = 6; required common.mjd mjd_utc_epoch = 7; required uint32 theory = 8; required double incl = 9; required double raan = 10; required double ecc = 11; required double aop = 12; required double ma = 13; required double n0 = 14; optional double n_dot = 15; required double n_2dot = 16; required double bstar = 17; optional uint32 nrev_at_epoch = 18; optional double days_prop_epoch = 19;

optional properr properr = 20; optional tle replace_tle = 21 [(skynet.model.db_column).autofetch = false]; optional common.mjd mjd_utc_replace = 22; required common.entity source_entity = 23; required common.permit export_permit = 24;}

// Metadata for numerical propagation (Cowell).message cowell { option (skynet.model.db_table) = { name: 'cowell'; }; required uint64 id = 1;

required uint32 integrator = 2; optional double stepsize = 3; optional double rel_acc = 4; optional double abs_acc = 5; required uint32 grav_model = 6; optional uint32 grav_order = 7; optional uint32 grav_degree = 8; required bool is_drag = 9; optional uint32 density_model = 10; required bool is_srp = 11; required bool is_sun = 12; required bool is_moon = 13; optional uint32 sun_moon_pos = 14; required bool is_solid_tides = 15;

optional common.entity f107_source_entity = 16; optional common.entity kp_source_entity = 17; optional common.entity ap_source_entity = 18; optional common.entity taiutc_source_entity = 19; optional common.entity pole_source_entity = 20; optional common.comment comment = 21; required common.entity source_entity = 22; required common.permit export_permit = 23;}

// Osculating orbit.message osc { option (skynet.model.db_table) = { name: "osc"; }; required uint64 id = 1; required common.mjd mjd_utc_epoch = 2; required common.vec6d state = 3; optional common.utmat6d cov_state = 4; optional double mass = 5; optional double area_cd = 6; optional double area_cr = 7; optional double cd = 8; optional double cr = 9; optional double std_cd = 10; optional double std_cr = 11; required common.vec6d kep = 12; required cowell cowell = 13; optional osc replace_osc = 14 [(skynet.model.db_column).autofetch = false]; optional common.mjd mjd_utc_replace = 15;

required common.entity source_entity = 16; required common.permit export_permit = 17;}

// Ephemeridial dataset.message ephem { option (skynet.model.db_table) = { name: "ephem"; }; required uint64 id = 1; required common.mjd mjd_utc_start = 2; required common.mjd mjd_utc_stop = 3; optional double step = 4; optional uint32 inter_method = 5; optional uint32 inter_degree = 6; repeated osc osc = 7; required common.entity source_entity = 8; required common.permit export_permit = 9;}

// Basic orbit dataset.message orb { option (skynet.model.db_table) = { name: "orb"; }; required uint64 id = 1; optional tle tle = 2; optional osc osc = 3; optional ephem ephem = 4; optional common.mjd mjd_utc_od = 5;

optional common.comment comment = 6;}

// A maneuver.message man { option (skynet.model.db_table) = { name: "man"; }; required uint64 id = 1; required common.mjd mjd_utc_ignition = 2; optional double std_ignition = 3; optional double duration = 4; optional double std_duration = 5; optional double mass_flow = 6; optional double std_mass_flow = 7; required uint32 ref_system = 8; required common.vec3d dv = 9; optional common.vec3d std_dv = 10; required common.vec3b esti_dv = 11;

optional common.comment comment = 12; optional man replace_man = 13 [(skynet.model.db_column).autofetch = false]; optional common.mjd mjd_utc_replace = 14; required common.entity source_entity = 15; required common.permit export_permit = 16;}

Model ExampleOrbits and Orbital Parameters


skynet.modelInitialize a Database with All the Models


# Import skynet ORM and database.from skynet.model import mapping, db

# Search for models in 'bacardi.model'. # Alternatively, you could also pass a Python module.mapping.Mapper.scan_module('bacardi.model’)

# Connect to SQLite database store = db.DataStore('sqlite:///bacardi.db')

# Create all tables (SQL schemas) for # the registered models.store.create_tables()

Two ways of accessing data

• Sending a Query to the DataStore

• Using the db extension • More compact code and hence better readability• To retrieve the ISS data, one could simply:

skynet.modelData access


# Import data model for objects. from bacardi.model import object_pb2

# Get the obj with NORAD ID 25544 (ISS Zarya). iss = object_pb2.obj.get(norad_id=25544)

Abstraction layer for processes

• Decoupling of processes from network layer• Automatic caching, if needed

Easy integration of processes

• as sub process (data via disk I/O or named pipes) • as FORTRAN or C module with Python wrapper• as pure Python module

Worker device for multiple processes• One device per CPU / Core

skynet.process


Provenance is defined as a record that describes the people, institutions, entities, and activities involved in producing, influencing, or delivering a piece of data or a thing.

(W3C Provenance Working Group, http://www.w3.org/2011/prov)

Provenance


Agent

Entity Activity

used

wasGeneratedBy

wasDerivedFrom

wasStartedBy

wasEndedBywasAssociatedWith

actedOnBehalfOf

http://www.w3.org/2011/prov

Provenance recording

• Recording of all activities during runtime

• Multiple Provenance stores

• Provenance information stored as graph in Neo4j

Traceability and provability of all processed data

• Backtracking of each produced product (ephemerides, state vectors, correlated objects, …)

• Reproducibility of products and data generated

Work-in-progress

skynet.record


Technical

• Graphical User Interfaces• Web (Django) and desktop (PyQt)

• Workflow

Organizational

• Operational for simple workflows end of 2015 at DLR

• Extended SSA versions for DLR and German Army

• Other use cases for Skynet (e.g., data acquisition for rocket motors)

BACARDI and SkynetCurrent and Future Work



Thank You!

Questions?

[email protected]

www.DLR.de/sc | @onyame

high throughput processing of space debris data

Software

space agency pydata

c debris

debris fields

germany pydata seattle

catalogue pydata seattle

impact real debris object

objects larger

known objects