OpenStack Summit October 2015 Tokyo

Openstack Notifications can play a big role in extending capabilities of the Openstack. Most of the actions in Openstack which manipulate the system state generate notifications which is posted to the messaging component (e.g. RabbitMQ) and can be consumed by any service outside the Openstack. Some of these notifications are used by Ceilometer to provide chargeback related data. 

In this talk we examine some use cases where one can tap into OpenStack notifications layer and build real world use cases that are so often needed by enterprise IT. 

Specifically, we will cover the following: 

• High level overview of Openstack notifications
• Configuring Openstack services to enable notifications
• Live demo demonstrating tapping in to notifications related to Glance image creation, VM creation, Network creation etc.
• Auditing/Compliance solutions -- a few examples.
• Detect unauthorized Glance images in the system
• Inspecting newly created Glance image to detect infected OS images
• Auditing network access whenever a VM becomes externally accessible
• Running network access tools as soon as a new VM becomes available on the network
• Activity Feed of various Openstack activities which helps Admins to monitor their private cloud.

OpenStack and Hadoop ecosystems have been enjoying parallel amounts of rapid growth and adoption over the last few years. Project Sahara was established in the OpenStack community to drive the overall “Data processing on OpenStack” theme, while Hadoop-focused companies such as Cloudera and Hortonworks have been offering their vision of management frameworks and deployment models around Hadoop.

At Mirantis, we noticed that many people got quickly confused about different options for adopting Hadoop as a technology. In this talk, we’ll make an effort to address part of this confusion and set the stage for further deep-dive conversations into how OpenStack can actually help in adopting Hadoop in a particular organization. As always, we’ll keep things as vendor-neutral as possible.

Specifically, we’ll talk about the following:

• An overview and roadmap of the Hadoop ecosystem -- components like YARN, Hive, HBase, and others that constitute a working Big Data solution


• Management frameworks and deployment models offered by different vendors such as Cloudera and Hortonworks


• Typical Hadoop logical and physical architecture deployment


• Architecting OpenStack for Hadoop workloads, including:
  
  
  
  • Picking the right hardware and sizing it properly
  
  
  • Doing Storage right (HDFS, Ceph, Swift, direct block device mapping?)
  
  
  • Doing Compute right (KVM or Baremetal? Making scheduling work)
  
  
  • Doing Networking right (Just Neutron, or do we need full-featured SDN?)
  
  
  • Leveraging extras that OpenStack has to offer (multi-tenancy, NUMA, CPU pinning)
  
  
  
  

By attending this presentation, you will gain a solid real-world understanding of the benefits and ways to build out a working Hadoop/Big Data solution on OpenStack.

If you have a service and want a UI for it, this session is for you! Come to this session to learn what all the fuss is all about. Learn how to write a dashboard plugin for your custom service as well as add visual enhancements to make your dashboard stand out.

Horizon is the official OpenStack dashboard providing access to your services via the web UI. One of its selling feature is its robust plugin mechanism. We will explore what you can do with our plugin model as well as discuss all of the options available to you for extensions. Furthermore, we will explain how the new client-side work will fit into this model and what best practices you should follow.

Finally, we will talk about the OpenStack services roadmap and plans we have for supporting it and where you help us make a difference by contributing! 

OpenStack service controllers produce large amounts of log data, and processing these logs can be a time consuming and difficult task. Data processing applications excel at automating these types of tasks and providing new windows into the underlying trends in the data produced.

We will discuss how to configure your stack to produce log data that can be consumed and analyzed in real-time by Spark applications running on the OpenStack Data processing service (sahara). These techniques can be used to inspect the current status and health of your services, decreasing the time spent between problem detection and solution. This also creates an exciting platform for developers to explore new transformations of log data that will unlock endless possibilities such as failure analysis, performance enhancement, and fraudulent activity detection.

This talk will cover a broad range of OpenStack and Apache technologies used in the development of this solution. We will discuss the following as they pertain to the greater problem space:

• Logging service data to Zaqar and Manila


• Launching Spark clusters with Sahara


• Visualizing and experimenting on data with Zeppelin


• Creating Spark jobs to deploy through Sahara


• Storing processed data to Trove databases

There are many paths to achieving improved log processing in OpenStack. Attendees should expect to learn about how they can improve the stability and performance of their stacks through the advanced techniques discussed in this session.

Let’s be honest, OpenStack native service API’s leave a lot to be desired when it comes to consistency, capabilities, and performance. Wouldn’t it be better if you could use the power of Elasticsearch to find your instances, images, and other resources?

Say hello to Project Searchlight. Based on Elasticsearch, this newly approved OpenStack project has a mission to provide advanced and scalable indexing and search across multi-tenant cloud resources. Originally a part of Glance as the Catalog Index Service, its scope has been expanded to cover resources from all possible OpenStack services using a simple plugin mechanism.

Come learn more about Project Searchlight and see how you can take advantage of it today!  In this session we will provide an overview of the technology, how it is being integrated into Horizon, deployment concepts, and tips for helping you to get started in creating your own Searchlight plugin.

Ceilometer has got significant performance, scalability and reliability problems.

Is there a magic formula to fix Ceilometer?

We added Monasca as a storage layer to Ceilometer and … we got Ceilosca!

The preliminary performance results were really encouraging; first, we could store in excess of 6 million samples with a single node deployment and second, we recorded reasonable query response times:

Query (6M samples)                        Average Time (s)

ceilometer sample-list -m instance      2.722209
ceilometer meter-list                            0.316250
ceilometer resource-list                     28.852132
ceilometer statistics -m image            3.066389

How we did it.
Ceilosca leverages the Ceilometer data collection pipeline via a Ceilometer-Monasca Publisher and the Ceilometer API via a Ceilometer-Monasca Storage Driver. This approach addresses the problems with Ceilometer, while still supporting a fully compatible Ceilometer v2 API and leveraging the excellent work in the Ceilometer OpenStack data collection pipeline, which is not addresed in Monasca.


What we will present:
We are working toward the implementation of Ceilosca and we will make available all the code as part of the Monasca code base in Stackforge. We are also planning to demo the solution and provide some performance evaluations of Ceilometer vs. Ceilosca (*) and illustrate the penalties introduced by the cross services overhead and format conversions.

Fluentd and Logstash are popular log collecters.  They both provide unified and pluggable logging layers for OpenStack administrators.

In this presentation, we will provide the comparison between Fluentd and Logstash in several aspects.  This session includes the following topics about both of Fluentd and Logstash:

* How to introduce log collectors in your OpenStack
* Useful plugins and configurations for OpenStack administrators
* Best practices for highly available log collection

Some Neutron plugins like OpenContrail not only implement the APIs specified by Neutron but also extend the API set to provide additional functionality. Some examples include the ability to specify security policies and the ability to define sequences of network functions to be applied to selected tenant traffic.

OpenStack Heat however does not support those extensions out of box. We will therefore see how OpenStack Heat can be extended and support for Neutron extensions can be added.

We will then see examples of using OpenStack Heat to orchestrate chains of network functions to demonstrate such extensions to OpenStack Heat in action.

OpenStack is a mash of technologies and in production its often distributed across multiple servers. This is really challenging to deliver while training OpenStack. OpenStack training labs is a tool which will deploy a multinode cluster running OpenStack services on top of it. This cluster can be spun up with just a simple command on any modern laptop/desktop across multiple OSs. This enables free community training and also self-paced training of OpenStack. Also this could be a potential tool for commercial and internal trainings. We plan to discuss and explain our design decisions made by our team and discuss more about this tool and how to improve it.

About OpenStack Training Labs:

OpenStack Training Labs is a tool for deploying a lean, multinode cluster of OpenStack using VM's on common hardware (laptops/desktops). This tool has been developed as a part of training guides and is really useful for training's, proof-of-concepts, trying out new features/projects of OpenStack as it simplifies the deployment (one command setup) and emulates production grade deployments (with loss of performance ofcourse). The existing code base can be found here:

http://git.openstack.org/cgit/openstack/training-guides/tree/labs 

Recent Developments:

Our team decided to move this project out of training guides and re-brand the name to "OpenStack Labs". This is recent development and the blueprint is almost merged, waiting for the PTL to merge it, although the initial discussions have been done and the project has been approved.

https://blueprints.launchpad.net/openstack-manuals/+spec/openstack-labs

We have the following challenges/to-do list:

Initially our team decided to use BASH but as the code base increased massively over the past few months, we plan to port to Python. This tool just supports Ubuntu guests, we plan to add OpenSUSE and CentOS to provide more options.  We also want to support KVM hypervisor for better performance on Linux based OS.  Re-use this project as a way to test install-guides in the OpenStack CI system.

'Service Sentinel’ - Mobile application envisioned to manage OpenStack related services on the go.

In this paper, we present this application detailed over the following points:

• Current pain points


• Our proposition


• Architecture


• Solution Features


• Demo


• Key Benefits

Current pain points :

• No mobile application to handle all the OpenStack related activities


• No single portal with specific customizable dashboards for 'All users' of OpenStack community
  
  
  
  • Currently, we have different login screens for different activities (log bugs, submit bugs, test them, updates, chats, help, wiki)
  
  
  • Tools (Launchpad, Gerrit, OpenStack Wiki, IRC, Jenkins)
  
  
  • Users (Service Providers, Infrastructure Team, Developers/Reviewers, Lab users)
  
  
  
  


• No reminders/notifications on IRC chats, OpenStack work, release updates, sharing options, product catalogue of service providers, etc.

 Our proposition: 
A one-stop OpenStack services mobile application (Service Sentinel) for enabling potential users to seamlessly operate, monitor, collaborate and contribute to OpenStack while on-the-go. 

Service Sentinel will facilitate the following services:

• Service Providers/Enterprise users: Automate/approve OpenStack business processes remotely for multiple distributed processing steps


• DevOps users: Get involved in deployments, testing and work with collaboration tools for the day to day activities of continuous integration/continuous development (CI/CD)


• Lab Management users: Track the usage of resources, remotely execute 3rd party tools related activities, monitoring the configured OpenStack environment.


• Contributors: can collaborate/review the churn/work in OpenStack projects : Nova, Keystone, Neutron, Horizon, Glance, etc.


• Enabling users to access OpenStack information anytime and anywhere

Architecture: 
Service Sentinel is an android application being developed by TCS that encompasses the core modules to enable inter-working with OpenStack / tools as well as extend user-based roles and applicable profile specific dashboards. We aim to release the application to open-source community. 

All Service Sentinel dashboards ingest information via various Openstack related portals' APIs and inter-works with third party tools. The data is then processed, analysed and presented on the user-specific dashboards of the applications. 

A few examples of viable usage extended by Services Sentinel are:

• Service Providers/ Enterprise users: App dashboard integrated with vendor-specific portals for OpenStack products and services. Additionally single-page dashboard for OpenStack Wiki, Ask OpenStack, IRC and Stackalytics.


• DevOps users: Application dashboard integrated with continuous integration/continuous development tools – CI/CD (example: Launchpad, Jenkins, Zuul, Nodepool, Puppet, Storyboard, OpenStack Wiki, Gerrit, Ask OpenStack, IRC, Stackalytics, etc.). The Service Sentinel application can be extended to infrastructure, build management and professional services users.


• Lab management users: Enable extract of environment information through some vendor-specific third-party tools and leverage CI/CD. In general can also access Openstack Wiki, Gerrit, Ask Openstack, IRC, Stackalytics, etc through the application dashboard.


• Contributors: App dashboard integrated with are Launchpad, OpenStack Wiki, Gerrit, Ask OpenStack, IRC, Stackalytics, etc. 

Solution Features: 
To discuss about 'Service Sentinel' solution features, below are some applicable operational use-cases where Services Sentinel can be used by the main end-users:

• Service Providers - Maintain service catalogue, monitor competitor(s) and comparative study, market products, capacity planning, etc.


• DevOps Team - Automated testing and deployment updates related to Gerrit, Git, Jenkins, zuul, puppet, storyboard, etc.


• Lab Management - Template handling for workspace and network environment, manage resources and user groups.


• Contributors - Bugs and blueprint management mapped with Gerrit IDs, monitor individual and company contributions.


• Common features - OpenStack Wiki, join IRC, release and summit updates, providing feedbacks, filtered results, RSS feeds, share options, receive notifications for tasks, single place to access all profiles (Launchpad, Gerrit, Storyboard, GIT, etc), ask forum and help section.

What we present and demo:
As part of the presentation we cover the ideation behind Service Sentinel, the architecture and the applicable use-cases that will enable user to leverage Service Sentinel.

In the Service Sentinel demo, we showcase user-role specific dashboards integrated with tools and interfaced with open-APIs that can be directly consumed for information.

The demo consists of a-day-with-Service-Sentinel:

• Service Providers/Enterprise users: Market and compare their Openstack products and services anytime, to make a direct channel with the customers, monitor the timelines and have direct reporting


• Devops users: Track bugs related to deployments, operations and integration on storyboard as part of end-to-end CI/CD process


• Lab users: Handling Lab resources, managing scale-up/scale-down


• Developer: Clone/change/submit changes pertaining to launchpad project. Collaborate with other developers/contributors on the go.

Key Benefits: 

• Unified Openstack services management


• Effective business decisions


• Improved customer engagement


• Efficient Resource Management


• Build direct Marketing channel


• Complex Reporting


• IRC chats on the go


• Openstack Wiki updates


• Monitor automated tests, studies and services


• No more vendor dependency


• Personalize and Interact


• Fast results and increased productivity