how to develop for openstack apis
TRANSCRIPT
Developing for Openstack APIs
Yoram Weinreb, CTO OfficeRan Ziv, Cloudify R&DGigaspaces
Download the deck
Agenda- Introduction to Openstack API- Exposing the APIs in an Orchestrator- Openstack API Quirks and Pitfalls- Testing with Openstack API
Introduction to OpenStack API
How do we interact with OpenStack?- Send Actions/Commands- Collect information
Interact with OpenStack
Openstack Endpoints RESTful API
Horizon Openstack CLI Tools
Openstack Official Python Client Libraries
SDKs for other Languages
RESTful endpoints
- API Versioning- For Example Identity ver. 1, 2, 3…- Interface
- Public- Internal- Admin
- Format (JSON, XML)
The northbound interfaces
Some Endpoint examples
- Identity API v3 - Compute API v2.1- Image service API v2- Block Storage API v2- Networking API v2.0- Object Storage API v1
- Bare Metal API v1- Orchestration API v1- Telemetry API v2- Clustering API v1- Data Processing v1.1- Database Service API- Shared File Systems API v2
Debugging OpenStack API
- CLI --debug- Python SDK debug params- VM Logs on horizon & CLI- Access to Openstack logs /
DevStack
Debugging OpenStack API
Demo time...
Debugging using Python SDK
Another option to turn on logging is with environment variables:
os.environ['NEUTRONCLIENT_DEBUG'] = 'true'
Viewing VM logs
- VM Logs on horizon- VM logs on CLI nova console-log [--length <length>] <server>
- VM logs in Python client: server.get_console_output()
Viewing OpenStack logs- Most logs are located at
/var/log/<service name>
- Devstack:[[local|localrc]]
DEST=/opt/stack/
LOGFILE=$DEST/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
Exposing OpenStack APIs in Cloudify
Exposing OpenStack APIs in Cloudify
- Cloudify is an open-source, pure-play orchestrator
- Based on the TOSCA open standard- Describe your application components, their
lifecycle, and their relationships to one another- Use either YAML format or the Cloudify Composer
- Supports both hybrid and multi cloud environments- Open architecture allows for extending Cloudify to
work on any environment through plugins
Exposing OpenStack APIs in Cloudify
- Cloudify supports Openstack via the Openstack plugin
- Strives to be unopinionated, support any use-case
- Offers simplicity and ease-of-use, yet not compromise on flexibility
- Robust at dealing with cloud errors
- Makes abstractions common to other environments where possible
- Resources are exposed via types defined by the plugin
- Prominent resource parameters are exposed explicitly- Provides sensible default values, syntactic
sugaring, etc.
- Other parameters are configurable via direct override
- Parameters for Openstack clients used are also configurable
Exposing OpenStack APIs in Cloudify
Exposing OpenStack APIs in Cloudify
- An example:
my_subnet_node:
type: cloudify.openstack.nodes.Subnet
properties:
resource_id: my-subnet
cidr: 1.2.3.4/24
dns_nameservers: [8.8.8.8]
Subnet: # direct override
enable_dhcp: false
Exposing OpenStack APIs in Cloudify
- A Server example- Keypair, SGs, networks/ports may be configured
directly or, preferably, via relationships
my_server_node:
type: cloudify.openstack.nodes.KeyPair
properties:
resource_id: my-server
image: Ubuntu-14.04 # translated to ID
flavor: x3.medium # translated to ID
- Some server operations take a while to complete
- VMs take long to boot (reach “ACTIVE”)- Sometimes boot in errored state, or have
connectivity issues- Cloudinit startup- retrieve a password from the metadata service- sshd service to start up
- An orchestrator needs to be able to handle asynchronous operations
Exposing OpenStack APIs in Cloudify
Exposing OpenStack APIs in Cloudify
Volumes considerations:
- Volumes may also require asynchronous handling
- Volumes often require a specific set of operations to become usable
- format- mkfs- mount
Exposing OpenStack APIs in Cloudify
Security-groups considerations:
- In Openstack, a default SG always exists and will be attached automatically to a new VM which doesn’t declare otherwise
- Orchestrator must create SGs prior to the VM and declare them at boot time
- When creating a new SG, it’s instantiated with default, permissive egress rules- disable_default_egress_rules: false
Exposing OpenStack APIs in Cloudify
- Can override Openstack clients configuration- Configurable on multiple scopes (deployment, node,
operation) to achieve both ease of use and flexibility
my_floating_ip_node:
type: cloudify.openstack.nodes.FloatingIP
properties:
openstack_config:
nova:
http_log_debug: true
neutron:
endpoint_url: https://1.2.3.4:9696
OpenStack APIQuirks and Pitfalls
OpenStack API Quirks and Pitfalls
- Some of the APIs aren’t necessarily intuitive
- Some due to historic and legacy reasons; some have been or will be declared as bugs
- The orchestrator can help with handling these quirks and making things more intuitive where possible
- A network may contain more than a single subnet
- When creating a server (or a port), required parameter is “network_id” - subnet is chosen arbitrarily
- To place a server on a specific subnet, must first create a port and supply it with the “fixed_ips” param
{'name': 'my-port',
'network_id': '4c6d….aba9',
'fixed_ips': [{'subnet_id': '4e70...f2a6'}]}
OpenStack API Quirks and Pitfalls
- Unlike the rest of the resource types, Keypairs are managed on a per-user basis, not per-tenant
- Can lead to funny behavior:- Breaks isolation between tenants - resource
cleanup on one tenant may affect another
- Heat stack created by one user may not be destroyable by another user on same tenant
- Keypair quotas are set per tenant but apply per user
OpenStack API Quirks and Pitfalls
- In Openstack, floating IPs are first allocated and then attached to a server or a port
- There’s no validation when attempting to attach a floating IP which is already attached to a resource
- The floating IP will simply detach from the previous resource
- race-condition scenarios- checking an IP is allocated doesn’t necessarily
mean it’s safe to attach it to a resource
OpenStack API Quirks and Pitfalls
- The Nova API for adding a SG to a server is not thread-safe (launchpad bug)
- An orchestrator that runs concurrent operations should verify the addition after making the API call
- Neutron API for adding a SG to a port is thread-safe- But not concurrency-friendly as it requires stating
all SGs rather than only the newly added one
OpenStack API Quirks and Pitfalls
- SG rules are defined over a port or a range of ports- ICMP rules have a “type” and a “code” but no port
association- Neutron’s security-groups API translates:
“port_range_max” → “code”
“port_range_min” → “type”
- E.g. - Allowing Ping from anywhere:
{'protocol': 'icmp',
'port_range_min': 0, # Ping’s ICMP Type
'port_range_max': 0, # Ping’s ICMP Code
'remote_ip_prefix': '0.0.0.0/24',
…}
OpenStack API Quirks and Pitfalls
- Nova’s API for adding a security-group to a server may receive either a security-group’s name or ID
- If the security-group was created using Nova (i.e. it’s a Nova-net security-group), passing the ID will fail
- Poses a problem when there are multiple security-groups with the same name
- Poses a problem for orchestrator - the code for a separate resource (Server) needs to be aware of the SG’s type
OpenStack API Quirks and Pitfalls
- Ports can be created either explicitly or implicitly (e.g. by connecting a server to a network
- Explicit ports should have to be deleted explicitly
- Up until Kilo release, deleting a server which was connected to a port that had been created explicitly would’ve deleted the port as well
- From an orchestrator’s POV, this is an abstraction breach, as one resource affects the lifecycle of another
OpenStack API Quirks and Pitfalls
- Keystone roles are assigned per tenant
- However, a user who’s assigned an admin role on one tenant becomes an admin across all tenants
- Should stick to setting admin users only within an “Admin” project
- Admin users can manage resources on any tenant- May actually see resources of tenant A even
while using tenant B
OpenStack API Quirks and Pitfalls
Testing with OpenStack API
- Unit tests- mock library- custom mocks- Mimic project
- Integration tests- Test the plugin’s operations against real
Openstack deployments, of different versions
- System tests- End-to-end tests for the plugin- The plugin is used to run many other
of Cloudify’s end-to-end tests
Testing with OpenStack API
Testing with OpenStack API
- Testing environments:- Integration tests and system tests run in parallel
over multiple Openstack tenants- Pretty good isolation- Easy clean-up
- Os-purge
- The tenants can be pre-existing- Tenant deletion requires clean-up anyway- Allows setting up an environment for
multiple tests to run serially- Take resources snapshot before and
after each test - clean-up delta
Testing with OpenStack API
- Testing environments:- Tenants for tests are set across multiple
Openstack deployments- Clouds have problems:
- VMs starting up in erroneous states- Connectivity issues- Maintenance and downtime- ...
Relying on a single environment is risky
- Possibly tests multiple Openstackversions / distributions
Testing with OpenStack API
- Testing environments:- When cleaning up a tenant’s resources after a
test, keypairs make an exception- since they’re per-user rather than per-tenant
- Each test cleans up any Keypairs it created on its own, whether it succeeds or fails
- An independent process is run when tests are inactive to clean up any Keypair leftovers from tests that’ve been stopped abruptly
Testing with OpenStack API
- Testing multiple Openstack versions:- Devstack
git clone git://git.openstack.org/openstack-dev/devstack -b <release branch name>
- Ravello- A double virtualized environment in the
cloud.- Define Ravello blueprints that will start and
stop isolated Openstack clouds in minutes
Testing with OpenStack API
Questions?
Thanks!Come visit us at booth #C20!
Exposing OpenStack APIs in Cloudify
- A more advanced example:
my_subnet_node:
type: cloudify.openstack.nodes.Subnet
properties:
resource_id: my-subnet
cidr: 1.2.3.4/24
relationships:
- target: my_network_node
type: cloudify.relationships.contained_in