Developers are already hard at work on OpenStack Pike, due out on Aug. 30. We caught up with OpenStack storage project leaders to ask them about the most requested features and to gain insight into the long-term future of their technologies.
John Dickinson, the project technical lead (PTL) of the OpenStack Swift object storage service, said the most sought-after Swift capabilities fall into two categories: effective and efficient management of clustered storage nodes and enterprise-grade features related to data policy migration, encryption and erasure codes.
Dickinson, who is also director of technology at SwiftStack, said it's hard to predict which features will make it into OpenStack Pike. However, he said Swift project contributors are prioritizing container sharding, to enable users to put potentially billions of objects into a single storage container; global erasure codes, to allow a storage policy to be distributed across a wide geographic area; and improvements to cluster rebalancing.
John Dickinsonproject technical lead, OpenStack Swift
Sean McGinnis, the PTL of the Cinder block storage service and principal architect at Huawei, said one of the most requested features in the project he is leading is the ability to attach a Cinder volume to more than one compute instance to enable clustered applications. He said Manila developers are working closely with the OpenStack Nova compute team on the new volume attachment APIs to try to ready the multiattach capability for the OpenStack Pike release.
Another feature McGinnis said the Cinder team is targeting for the OpenStack Pike release is group replication to give cloud users more control over managing data protection.
"The caveat there, as far as having something to deliver in Pike, is there will be additional work for the volume back-end drivers to be able to use that functionality of Cinder and actually set up the replication," McGinnis said.
Ben Swartzlander, the PTL of the OpenStack Manila shared file systems service and an architect at NetApp, said the feedback he hears most often is related to stability and bug fixes. Also on Manila users' wish lists are data mobility capabilities across clouds, additional forms of data replication and storage mount automation, he said.
Swartzlander said the biggest features targeted for OpenStack Pike include IPv6 support for share access, integration of the OpenStack Telemetry project's Ceilometer data collection service, shared usage monitoring and more granular quota control on resources.
You can read the transcripts of the interviews with the PTLs for the OpenStack Swift, Cinder and Manila storage projects below.
Transcript - What new storage capabilities are coming in OpenStack Pike?
Editor's note: The following are the transcripts of the interviews with OpenStack Swift, Cinder and Manila storage project leaders. The transcript has been edited for clarity.
What are the most requested features for the OpenStack Swift object storage service?
John Dickinson, PTL, OpenStack Swift: The most requested features for Swift have to do with two big categories. The first is figuring out how to effectively and efficiently manage the back-end storage on the storage nodes of the clusters, improving efficiency, lowering latency [and] improving performance.
The other big category of things that we're looking at have to do with what I would classify as enterprise-grade features -- things that have to do with data policy migration, moving data from one policy to the other, improving our encryption support and supporting globally distributed erasure code storage policies.
What features will be added to Swift in the next OpenStack release, called Pike?
Dickinson: I think it's hard to say specifically which features are going to make it into the Pike release.
What I can say is the kind of things that we are absolutely working on, and [we] try to make as much progress as we can on a particular OpenStack release.
One of the things that I'm excited about that we're working hard on and is nearing completion is support for global erasure codes. This is allowing you to have an erasure code storage policy that's distributed across a wide geographic area. The basic idea here is that we can erasure code a particular set of data into a set of fragments and then replicate those fragments to a different region. And then both regions can participate in rebuilding in case of data loss.
Other things that we're working on right now include container sharding, which is a way to distribute some of the data that is on the back-end storage system and insure that it can improve performance.
Also, we're working on improving the rebalance capabilities of the cluster itself as you add capacity to a Swift cluster.
What's in store for Swift in the upcoming OpenStack Queens release and beyond?
Dickinson: In the long-term future for Swift, I'm really excited about what is happening inside of the project and in the broader ecosystem around it.
In addition to the performance improvements, efficiency improvements that we're making in Swift itself, and some of these enterprise features about data tiering and policy migrations, I'm really excited to see additional ecosystem work about different access protocols, including continuing to improve the S3 [Simple Storage Service] access adapter, people in the ecosystem ... adding file system access onto a Swift cluster, and being able to do metadata indexing and searching of all of the data that you're storing inside of Swift.
I'm excited to see that Swift itself and the features that it has, plus these extra ecosystem projects that have sprung up around it, have enabled so many new use cases and so many new deployments of Swift. I see it used in media and entertainment, video games [and] for content on the internet. And I think that is what's really going to be exciting to see in the future.
What are the latest Swift adoption trends?
Dickinson: We're looking at anything that is generating more data.
These days, especially inside of the OpenStack community, there's been a lot of people talking about containers and the container ecosystem. I truly believe that, as applications are containerized and deployed onto containers, these applications still need a very scalable way to store their data. Not only in a way like a Docker Registry sort of thing, where you can pool your container images from Swift, but also storing the data that the application is generating inside of Swift itself. And as the applications grow up and are scaled out, the storage system itself is also able to be scaled out very efficiently and effectively to meet the storage needs. So, these kinds of use cases are fascinating.
There's lots of other buzzwords in the ecosystem, with internet of things and any other kind of scale-out storage needs that people have. I think the future's bright for storage, and there's a lot more data, and we're going to effectively solve that with Swift.
What are the most requested features for the OpenStack Cinder block storage service?
Sean McGinnis, PTL, OpenStack Cinder: We've had a few features that we've been working on for some time. The biggest one right now that's really the most commonly heard one [by me] is the ability to support multiattach of volumes.
So what that is, basically, is having one volume that's exposed to multiple instances, so that they can all access it, and you can run things within your OpenStack cloud, like Oracle RAC [Real Application Clusters] or some type of clustered application that needs access to the same block storage at the same time. That's been an ongoing effort.
We made some changes on the Cinder side, but in order to get that working as a usable solution, there are some changes that need to be done on the Nova side. So, we're working together closely with that team. We have weekly meetings. We're trying to refine the way that Nova and Cinder communicate, and once we get that in place, I'm hoping that we can actually deliver an entire solution there.
The second one -- maybe not as commonly asked for, but there certainly are people that are interested in it -- is the ability to do group replication. [For example], having an end user [who is] able to group a set of volumes, again, maybe something like a database, where they can say, 'These three volumes are part of my database workload,' and have the ability for them to say, 'I need that data protected, so I want this data replicated to another Cinder back end.' In that case, if the primary back end that that data resides on blows up or something happens, even though it won't seamlessly switch over to the copy of the data, there is at least a copy of that data out there. And they can recover it and get back to running again.
What features will be added to Cinder in the OpenStack Pike release?
McGinnis: In Pike, I'm hoping we can get core Cinder services support for both multiattach and group replication. Most of it is there for multiattach. We may need to make additional changes depending on what happens once Nova starts trying to use our APIs. We might need to change something else.
And replication is the biggest thing, I guess, right now. We need to get that core functionality for group replication into Cinder. The caveat there, as far as having something to deliver in Pike, is there will be additional work for the volume back-end drivers to be able to use that functionality of Cinder and actually set up the replication. That part I'm not quite as sure about as far as Pike goes. But we'll see how far we can get.
What's in store for Cinder in the upcoming OpenStack Queens release and beyond?
McGinnis: [With] Queens, I'm hoping to wrap up some of these. I'm hoping that we can maybe clean up some of our legacy code.
We have our v1 and v2 APIs. That's a lot of code in there that we've been trying to get rid of, clean up. We have a lot of code that needs to handle things in certain ways, depending on which of those APIs gets called. So, if we can clean that out, we can simplify some of our code. And a smaller code base is hopefully a code with [fewer] bugs.
What are the most requested features for the OpenStack Manila shared file systems service?
Ben Swartzlander, PTL, OpenStack Manila: Manila is a fairly mature project. It's been around for five years. And, oddly enough, the request we hear most often is that people really want the project to be stable and do bug fixes, which is sort of an anti-feature. But it is the one that we hear the most often, believe it or not.
Other features we get requests for would be data mobility features. Manila already has the ability to do share migration, as well as share replication, across availability zones. But there are a lot of other things users would like to be able to do, including moving their data across clouds and other forms of data replication and data mobility.
I would say the third biggest feature we get asked for -- and this is a very old one -- is mount automation. This goes back to the early days of Manila where people who are accustomed to the Cinder model, where you just tell Cinder to attach your volume to your instance and then your storage appears, don't like the fact that, in Manila, you actually have to manually mount the storage because Manila allows the instances to connect directly to the storage controller without any involvement from the hypervisor. So we've been looking for a long time for ways to automate that process.
There's some work going on from Red Hat to provide a mechanism for doing mount automation, but it relies on changes to upstream projects, like the Linux kernel and KVM [kernel-based virtual machine], and it's taken a very long time for us to get there. So, some day, we hope to have mount automation working, as well.
What features will be added to Manila in the OpenStack Pike release?
Swartzlander: I'd say the biggest feature for Pike is going to be IPv6 support for share access. IPv6 support is something we have for API access and its communication between the Manila services, but not between the client and the storage controller. And that's something we definitely need to have so that IPv6-only environments can access their shares. So that's something we plan on getting done in Pike.
We're doing some work with Ceilometer integration. Ceilometer is the telemetry-as-a-service project within OpenStack. And there's some telemetry information that Manila generates that would be useful for something like Ceilometer to collect, including another new feature that's being worked on, which is share usage monitoring. [Share usage monitoring allows you to] get information about how much space is consumed within a share for billing or generic metrics gathering.
We're adding support for share type quotas, which is a useful feature if you have different classes of storage and the administrator wants to give users a larger quantity of one type and a smaller quantity of another type. You need separate quotas for each share type, and that's something that is coming in Manila in Pike, as well.
What's in store for Manila in the upcoming OpenStack Queens release and beyond?
Swartzlander: We don't have an official plan for Queens yet.
Traditionally, in the Manila project, there's a lot of carryover work at the end of every release. We always have a bigger appetite than [what] we're able to get ... done. And so I anticipate that there will be some stuff that carries over from Pike into Queens.
What are the latest Manila adoption trends?
Swartzlander: Red Hat officially started supporting Manila in their OSP [OpenStack Platform] 10 release, and that is really helping drive the adoption. Ubuntu now includes Manila in the Zesty release as a standard package. SUSE has supported Manila for a long time. But I would say the addition of support in Red Hat is driving a lot of adoption.
Adoption is widest amongst maybe more traditional, larger, more sophisticated companies that already have a significant investment in shared file systems. They're the ones who are more likely to gravitate toward Manila as opposed to the other storage solutions within OpenStack. And we do see a steady growth trend there. It's not explosive, but constant new interest over time.