[email protected] [email protected] … · 2019-05-14 · motivation: batch...

[email protected] [email protected] [email protected]

Motivation: Batch processing patterns, security, QoS

Kubernetes Jobs, Asperathos & SCONE

Demonstration321


Extract-Transform-Load (ETL)

In this case, the goal is to collect data for one or more sources, apply some processing logic to transform (convert, filter, anonymize) then load on another system or database.

Business logic

Extract Transform

Business logic

Load


Bag of tasks

In this case, the inputs are independent work items. These inputs trigger the processing, which eventual returns results to users.

Tasks

WorkersWorkersWorkersWorkersOutputs

External information


Processing through independent tasksBoth patterns can be implemented through tasks in a work queue or set.

● A task item can be emails to be sent, frames to be rendered, files to be transcoded, ranges of keys in a NoSQL database to scan, and so on.

● Here we consider that a job is composed of a set of independent tasks, which are managed through a work queue.


What are the consequences of internal attackers or a compromised infrastructure?


Our threat model

The attacker is powerful and has root access to the infrastructure, for example:

● Works for the cloud provider or has impersonated a cloud admin;

● Has previously compromised the machine (BMC, firmware, OS);

● Exploits vulnerabilities in the host or hypervisor to get root access.

There are recent news about all cases above.

In addition, there is a large surface for attack, and the user has to trust everything he/she does not control (and bugs are especially bad in privileged software as may result in unrestricted access):

● Linux: ~20 MLOC● KVM: ~21 MLOC● OpenStack: ~9 MLOC


Threat model & security impact in our use case

Collect & Store

Billing application

System safety analysis

application

Data can be stolen or modified in storage

Applications can be changed to leak data or modify computation.


Approach: encryption

Collect & Store

Billing application


application


By adding encryption capabilities…● Encryption keys can be provided directly to applications;

data-at-rest is secure.Applications can be changed to leak data or modify computation.



Approach: attestation

Collect & Store

Billing application


application

By adding attestation capabilities…● Secrets handed only after attestation:

modified applications cannot be run.Applications can be changed to leak data or modify computation.


Orchestration & QoS

Without orchestration, even recurrent jobs may become unpredictable:

● Interferences in the cluster can delay the job execution;

● Irregular behavior of processing can delay the job.

Here the executions were finished after 15% overtime.

Data processing with Kubernetes,Asperathos & SCONE

Data processing with Kubernetes, Asperathos & SCONE

Kubernetes Jobs

Native Kubernetes resource for finite workloads:

● Define a set of pods and make sure a certain number of them terminate successfully.

● We can leverage a work queue or set to store the work items. These inputs trigger the processing, which eventually returns results to users.

● Good foundation to implement workload engines, as it benefits from the highly mature Kubernetes API and its ecosystem.


Kubernetes Jobs Patterns

There are several different patterns for parallel computation with Jobs:

● One Job per item vs. one Job for all items;

● One Pod per item vs. N pods can access M items;

● Direct input access from pods vs. Work queue for managing the items.

Using one job, N pods, and work queues makes it more efficient and enables adjustments, but requires modifications in the application. Asperathos simplifies this process...


Asperathos

● Open-source framework for orchestrating data processing applications.

● Plugin-based architecture provides high level of customization.

● Ability to control resources to meet user-defined & application-specific QoS constraints (e.g., makespan, latency, throughput).

● Current plugins allow workload execution in various infrastructures (OpenStack VMs, Kubernetes clusters, Spark clusters, Mesos, your own!).


Asperathos framework

Manager: triggers application executions.

Monitor: collects & normalizes relevant performance metrics.

Controller: controls orchestration based on normalized metrics.

Visualizer: hosts dashboards for cluster and application metrics.


Intel SGX

SGX uses the notion of an enclave:● New CPU instructions to manipulate

enclaves;● A memory region in a process for

which memory accesses are encrypted before leaving the processor;

● Code loaded into this enclave is measured (hashed) during load.

In practice:● Isolated memory regions for code and

data of the enclave; memory is protected through a hardware memory encryption engine;

● The processor can provide a proof of the code that has been loaded;

● Enclave memory can be accessed only by the enclave code (protection from privileged code – OS, hypervisor).


Intel SGX: Reduced attack surface

Based on the figure from Intel® Software Guard Extensions Tutorial Series: Part 1, Intel® SGX Foundation https://software.intel.com/en-us/articles/intel-software-guard-extensions-tutorial-part-1-foundation

https://software.intel.com/en-us/articles/intel-software-guard-extensions-tutorial-part-1-foundation


Secure execution with SCONE

Application is recompiled (GCC + modified libc) or runs within a recompiled runtime (e.g., Python).

Container image is regenerated and K8s manifest is updated with integrity information (hashes expected for the application and its dependencies).

A tool is used to upload the secrets to a SGX-enabled key manager while the, now secret-free, manifest can be submitted to the untrusted Kubernetes cluster.


A K8s Job Manifest: The common part (extract)apiVersion: batch/v1kind: Jobmetadata: name: copel-analyticsspec: parallelism: 2 template: metadata: name: copel-analytics spec: volumes: - name: dev-isgx hostPath: path: /dev/isgx containers: - name: worker image: 10.11.5.6:5000/worker:latest


… and an example of the difference (the ENV section)

env: - name: APP_KEY value: 'admin' - name: PY_SHA256_CHECK value: '{"/app/worker.py": "483e...f346","/app/redis/client.py": "fb2335...d534", ...}'

In the workstation: Secrets (APP_KEY) and integrity-sensitive data (application hash) included.

env: - name: SCONE_HEAP value: "200M" - name: SCONE_CAS_ADDR value: "22.65.15.87:18765" - name: SCONE_CONFIG_ID value: "user/app/worker" - name: SCONE_LAS_ADDR value: "las:18766"

To be sent to the (untrusted) cloud: Secret-free manifest.


SCONE secret provisioning - Successful attestation

The Configuration and Attestation Service (CAS)...

● Performs attestation of application;

● Runs inside SGX enclaves;● Stores secrets (credentials,

certificates).

Figure from Stefan Köpsell


SCONE secret provisioning - Failed Attestation

If application is compromised, attestation from CAS fails and secrets are not provisioned.

Figure from Stefan Köpsell

Demonstration

Demonstration

Smart grid data processingwith Kubernetes Jobs

Data is encrypted and stored in a scalable storage (e.g., OpenStack Swift).

Job task is written in high-level language (e.g., Python, Go, C, Fortran).

A task instance gets an item from the queue, processes it and puts it back on the storage.

Demonstration

Legacy application, written in Fortran.

● Cannot be modified.● Encryption is abstracted by running the

application with SCONE volumes mounted inside the worker.

● The application is wrapped in a container that downloads inputs and uploads outputs back to Swift.

Runs simulations periodically.

● QoS is important to make sure it finishes on time, especially when susceptible to interference (e.g., multi-tenant environments).

Case 1:System safety simulation

Demonstration

Our approach:

1) Recompile the application with the SCONE-gfortran compiler.

2) Enable transparent encryption.3) Submit to Asperathos.

a) Each pair (system network file, simulation parameters) becomes a work item in a queue.

b) Asperathos spawns Kubernetes workers to concurrently consume the queue.

c) Asperathos Controller keeps adjusting resource allocation (e.g., # of workers) to meet the user deadline.

Case 1:System safety simulation

Demonstration

Case 2:Smart meter data analysis

Application is written in Python and knows encryption.

● Uses a SCONE-compiled Python interpreter.

The encrypted input data is pushed to Swift.

The decryption key are pushed to an SGX-enabled Secret Management Service, and are injected inside the enclave after successful attestation.

The worker gets data from the queue and uploads the output metric to a DB or object storage.

Conclusion and links

Kubernetes Jobs eases the execution of useful task-based data processing jobs.

Intel SGX enables confidential computing, and SCONE simplifies the porting and development of applications.

Asperathos enables the combination of SCONE and Kubernetes to support application QoS and data confidentiality.3

21


Go to github.com/ufcg-lsd/asperathos and try the quick start to understand the orchestration components with a work queue example.

Go to sconedocs.github.io/blender/ and try the processing of sensitive tasks in remote untrusted hosts.

Explore other SCONE evaluation images and go back to Asperathos Github for other tutorials and examples.321


More Information

Asperathos, SCONE, and KubeJobs have been funded through the 3rd & 4th EU-Brazil H2020-MCTIC/RNP coordinated calls (SecureCloud, BigSea, and Atmosphere projects).

Github: github.com/ufcg-lsd/asperathos

Papers and technical reports:securecloud.works atmosphere-eubrazil.eu PrivacyByEvidence.works

http://github.com/ufcg-lsd/asperathos

http://securecloud.works

http://atmosphere-eubrazil.eu

http://privacybyevidence.works

[email protected] [email protected] [email protected]

[email protected] [email protected] … · 2019-05-14 · motivation: batch...

Documents