Watcher

PTG Etherpad

Overview

The 2025.2 PTG was the Second PTG i have attended for watcher. The last PTG had one session to cover all the technical debt required to revive the project. This time we extended the ptg to 2 days.

Day 1

Tech Debt

Croniter

The PTG started with a disscussion about technical debt that emerged during the epxoy cycle. This focused mainly on croniter, which is used by watcher to schedule continuous audits based on cron strings. The Croniter libaray maintainer has decided to stop maintaining this library and it is transitioning to a new maintainer. While that gives watcher some breathing space, our usage is minimal, so we agreed to drop the dependency by leveraging the existing functionality of apscheduler.

Support status of datasources

The second topic of the ptg focused on the support status of datasources and features of watcher in general.

As part of this discussion we touched on the idea of multiple levels of support:

• Core features (supported, tested in ci and ready for production use)
• Experimental features (supported but not tested in CI or may not be suitable for production use)
• Deprecated features (features that are no longer maintained and not suitable for production use)
• Unsupported features (features that are not supported by the project, but may work)

During this session we agreed to deprecate the support for the following features:

• Monasca
• Grafana
• Noisy Neighbour Strategy (l3 cache)

Monasca support was dropped because it is no longer maintained and is officially being retired.

Grafana support was dropped because it is no longer maintained and has no existing ci coverage under our new support levels it could be considered experimental as it may work under certain conditions however since there is no ci and no active development it was agreed that we should drop support for this feature.

The Current Noisy Neighbour Strategy (l3 cache) was deprected because it depends on metrics that are nolnger avialable for several years. a dedicated session on how to replace it was held later in the ptg to discuss how to replace it going forward.

Finally we wrapped up this topic by agreeing that we should create a support and testing matrix for watcher takeing inspiration from https://docs.openstack.org/cinder/latest/reference/support-matrix.html or https://docs.openstack.org/nova/latest/user/support-matrix.html The intent is to provide a way for users to see which features are supported, tested and suitable for production use cases.

Eventlet Removal

The final technical debt topic was Eventlet removal. Watcher like many other OpenStack projects have been using eventlet since its inception, and it has become clear that eventlet is no longer sustainable. While no concrete proposal was made for this topic, to start evaluating the removal this cycle with some initial POCs and aim to remove eventlet in 2026.1

Workflow/API Improvements

SKIPPED status for actions

This discusstion focused on the introduction of a new Skipped status for actions. Today actions can be in one of the following states:

• pending
• ongoing
• succeeded
• failed

When constructing and executing an action plan it would be useful to extend the action states to introduced a skipped state, which would allow users to mark an action as skipped, preventing it from being executed by the engine. The idea is that this can be used to skip actions that are known to fail or be undesirable to execute. Additionally if an actions preconditions cannot be met, the action can be marked as skipped instead of failed.

This lead to a larger discussion about the meaning of SUCCEEDED and FAILED for action plans in general. https://etherpad.opendev.org/p/r.9c40a9e71e93a4be96ebd3e0ad2d7bc4#L108 We discussed that the current behavior where an action plan is considered succeeded when all actions are attempted regardless of the success of the action is unintuitive and not in line with the state machine docs.

it was agree that if any action fails the overall action plan should be reported as failed. Additionally in this context we agreed that if the SKIPPED state is added it should be considered as Succeeded, not Failed.

watcher-dashboard

This sessions was relatively short and focused on 2 areas testing and missing functionality

Testing

We opened the session discussing the fact that the horizon plugin as almost no testing. the net effect of this is every change need to be manually verified by the reviewer. We have two options with regard to testing we can build on django’s unittests integration to validate the core logic, or we could build out selenium testing. no general conclusion was arrived at but we resolved to follow up with the horizon core team in the future for guidance.

Exposing parameter in audits

This was simple and non controversial, today when creating a audit its not possible to set the parameters filed via the horizon ui. we agreed this should be addresses as a speechless blueprint.

SKIPPED status

finally we discussed that if we extend the action status to model the new SKIPPED state we will need to enhance the action plan dashboard to allow skipping an action and to display the skipped state when a precondition fails.

Watcher cluster model collector improvement ideas

This was arguably another tech debt discussion in the guise of an operator pain point. For performance reasons watcher is build with a cached data model that periodically refreshes, combined with integration with notification for near real-time updates.

During the session we reviewed https://bugs.launchpad.net/watcher/+bug/2104220 and resolved to investigate if we are properly consuming the notifications form nova. While we are consuming the notifications bug #2104220 asserts that we are perhaps using the wrong fields to update the instance host on live migration resulting in the source host being updated instead of the destination host.

other mitigation were discussed such as adding the ability to force refresh the model when executing an audit, creating a new audit type to just run the refresh, or adjusting the collector interval.

Day 1 Summary

Day one was pretty packed but we resolved to finish early when we reached the end of the agenda.

We resolved to proceed with 2 new specs for the SKIPPED sate and model collector plugins and to treat the parameter enhancement to watcher-dashboard as a specless blueprint. croniter will be replaced as a bug fix and we need to review the impact of bug: #2104220

Day 2

Watcher and Nova’s visible constraints

Day 2 kicked off where day one ended on the topic of how watcher models nova instances and the visibility of scheduler constraints. In the epoxy cycle we added a number of attributes to the server show response that may be of interest to watcher, namely the scheduler hint and image properties. additionally i noted that in a prior release we added the pinned AZ as a separate filed. while extending the data model to provide this info would not be an api change we discussed that a spec could be nice to have to document the changes but a specless blueprint could also be valid.

Noisy neighbour strategy

This discussion happend as a sperate session on the second day but we touched on it on day 1 as part of the tech debt session. The TL;dr is since the exisitng noisy neighbour policy is non functional and since we want to aovid upgrade impacts a new SLA goal with 1 or multiple strateies shoudl be created. A spec to define the semantics should be created but at a high level the open questions are

• should we have one stagey per metric or a single parmaterised stagey
• which metrics shoudl we use as the key performance indicators to comptue if the sla is treshold is exceeded
  • cpu_steal
  • io_wait
  • cache_misses
  • others?

finally we agreed to deprecate the existing strategy this cycle for removal in 2026.2.

Host Maintenance strategy new use case

The last feature session discussed the limitation with the current host maintenance strategy the main limitations resolved around how to express if a instance should be stopped, cold migrated or live migrated. we agreed to continue with the spec https://review.opendev.org/c/openstack/watcher-specs/+/943873 and that it was ok to add new parameters to allow specifying which actions are allowed for the audit.

This topic was reprised in the nova room on day 4 TL;DR in the nova session we agreed to try to use server metadata to encode policies like live migratable. this can be incorporated into the spec design as a follow up if there is time. i.e. instance can be annotated by adding lifecycle:livemigratable=True|False and similar lifeccyle: metadata keys to express which actions may be taken by an external orchestrator like watcher.

PTG wrappup

contributor docs

The second to last session was a quick discussion of some of the document that watcher is missing namely the scope of the project doc and the chronological release guide. these are both important document for use to create going forward, as is ensuring our existing contributor docs are up to date regarding when to file a bug vs blueprint vs spec.

retro and adhoc planning

To wrap up the ptg we had a short retro on how the last cycle went and any last minute topics. traditionally one start the PTG with the retro rather then finishes but its better late then never. we realised that one topic had not been raised which is the future of the core team we briefly discussed that going forward we will review the core team member ship at the start of each slurp release (i.e 2026.1) and remove inactive cores that have not participated in review, contribution or ptgs since the prior slurp. As such no removal form the core team will be made for 2025.2 although new addition may be made based on review activity over the cycle.

With that we we wrapped up the ptg with a reminder of our two cross project sessions with telemetry/horizon on day 3 and nova on day 4

Telemetry/Horizon

Context

This ptg unlike many others i attended a combined telemetry/horizon cross project session lead by Victoria Martinez de la Cruz (vkmc). There we discussed a number of topics related to observability and how to provide tenant and admin static dashboards to visualise metrics.

Initially we started by discussion why observability and metrics are distinct form the existing views provided by the admin hypervisors dashboard and how they are distinct form showback/chargeback provide by cloud-kitty

showback/change-back focus on billing and the ratings are base not on the utilisation of the provisioned resource but rather then allocation. i.e. you are billed equally for an idle vm that uses 4 cpus and 16G of ram vs a fully loaded vm. cloud kitty can help tell you how large your bill will be but it will not tell you if your application can run with half the resources. A observability view based on near real-time metrics aims to solve the latter problem.

The hypervisor dashboard is also related but distinct. the hypervisor view is intended to provide a semi real-time view of the capacity of a could for admin to be able to plan if they need more capacity. Again this view looks more at the available resources and the allocated resource but does not provide an overview of the utilisation.

POC

An initial poc of what the dashboard could looks like is available here https://github.com/SeanMooney/grian-horizon-plugin

proposal

• create a new horizon plugin as an official deliverable of the telemetry project in the governance repos
• this will be hosted in the https://opendev.org/openstack namespace and will be released as an official deliverable via the release repo
• we need to spike on d3.js vs rickshaw vs chart.js for charting and assess the version compatible and tech debt
• initially the new metrics dashboard will integrate with Prometheus as a metrics store.
• a fake backend may be created for testing, other backends are out of scope but could be added later.
• a combination of new panels and tabs in existing panels will be added.
• initially all dashboards will be static and direct quires to the backend will not be supported
• this will allow the plugin to provide multi tenancy as required in the absence of multi tenancy support in Prometheus.
• the plugin will be stateless with the time span that can be viewed determined by Prometheus.
• the design goal would be to support short term metrics of up to 7 days.
• the python-observability client will be used to query Prometheus.

Nova

The nova ptg was busy as always, i was also sick all week and had conflict with watcher/horizon so i was not able to attended all the sessions so ill focus on some of important topics form my perspective instead of trying to do a full summery. for those looking for a full summery here is a excellent write up Rene’s Official PTG Summary, i will also not cover the eventlet removal topic in detail as gibi has a write up of that here Eventlet removal - Flamingo PTG

DAY 1

day 1 was a short day to allow for some cross project sessions to happen before nova officially started. as is normal we started the ptg with a retrospective on the past cycle, what worked well and how we wanted to proceed.

project planning

In general not much has changed for this cycle but ill just highlight some important dates Timeline:

• Soft spec freeze (no new specs): June 1st
• Hard spec freeze (M2): July 3rd
• Feature Freeze (FF): August 28th
• Final release: late September / early October

For those not familiar with the difference between the soft and hard spec freeze the soft freeze is the last date we ask contributors to propose large pieces of work. it serves as a signal that if your topic is complex and has not already been socialised and reviewed by that point it likely wont be mergeable by the hard freeze. This basically creates a buffer before people tend to disappear for PTO or others factors that might make getting a quorum difficult.

This worked well for us in Epoxy when we set the soft spec freeze in early December to acknowledge that we would loose quorum between late December and early January.

python 3.13 and eventlet

This are related but distinct topics. This cycle we would like to start early testing with python 3.13 as Ubuntu 25.04 has now moved to 3.13 and i believe Debian 13 may also. unfortunately monkey_patching the tread lib breaks on 3.13 so eventlet based service cant actually run on 3.13 under devstack unit test jobs seam to pass so that a start. we agreed to add a functional-py313 job if we can get it to pass however we do rely on eventlet there so that may or may not be possible.

Eventlet removal will be one of nova’s priories for the 2025.2 cycle as we acknowledged we are behind were we should be at this point in time. for those wanting to follow this progress we are going to track it as a recurring slot in our IRC meeting and gibi is planning to document the progress in there blog, the first post is already live!

Day 2

day 2 kicked off with some security related topics and finished up with live migration discussions

SEV TDX and arm CCA

Confidential computing is not new by any means even in nova with support for SEV being available for many years. Last cycle tkajinam had started to make improvement in SEV-ES support in advance of the enablement of SEV-SNP, unfortunately we did not have review bandwidth to land there enhancements. We resolved to prioritise the review of there work and also discussed the new development that are happening on intel and arm platforms. TL;DR the kernel, qemu and libvirt work is not complete for both TDX and CCA at this time and in the arm case there still isn’t hardware in production. effectively its too early to actually add support for arm CCA support and its likely to be a 2027.1 topic based on Ubuntu 26.04+ intel TDX is closer but also likely to early to supprot in 2025.2

PQOS (platform quality of service)

This is and old one. so like 7 years ago when i was still at intel there was an effort to support intel cache and memory bandwidth both statically in nova (pushed by myself) and using an external resource management daemon call RDT. both effort die out but in the last month or two people have started asking about it again.

i was not able to attend this session but it seam like the new contributors are going to pickup my old spec and re-propose it for 2025.2 with updates. Since i wrote that in 2019 the way intel CMT and MBT work has been simplified to some degree so they will update the spec to reflect that and we will review.

While this may be useful for some real-time workloads that are latency sensitive, its non trivial to add, complex to configure and likely not a generically useful outside a select number of workloads. This will be on to monitor but likely not a feature that will be used in many clouds. for those that need it however it could be a big improvement.

live migration topics

vtpm

TL;DR vtpm live migration is our largest gap in this area we briefly discussed later in the ptg that the pattern for vTPM secret handling will also inform how we address the same challenges for cinder encrypted volume live migration and local nova storage encryption when we eventually get back to that. The big change in our plan for vTPM is that this work will be taken over by another contributor and artom will work with them to hand off this work early in the cycle.

pci/vGPU live migration follow ups

In epoxy nova added support for vfio variant drivers and with that support for pci devices that

• Enable VFIO devices with kernel variant drivers
• Live migrate VFIO devices using kernel variant drivers

During the ptg we discussed what could not be completed and the divergence of the code paths between flavor based pci pass-though live migration and neutron SR-IOV port live migration.

The big takeaway form this session was we will address https://bugs.launchpad.net/nova/+bug/2103631 in two steps. First we will move the check before the conductor calls the scheduler. this is a backporable fix and will provide most of the optimization benefit by entirely eliminating the placement and scheduler work. Second we will move the check to the api and return a 409. this will fully optimize the early exit but it will not be as backportable. While the api can already return a 409 https://docs.openstack.org/api-ref/compute/#live-migrate-server-os-migratelive-action chaning a 202 to a 409 is not always considered valid for backports.

DAY 3

day 3 was perhaps the most eclitic of topics with everything form confirming we will complete previous work started in epoxy to discussion of eventlet removal and cross project session with glance/cinder .

VDI enhancements, os-traits and one-time-use devices

VDI and one time use devices were short as the first was a carry over form work that was approved form epoxy but didn’t merge in time and one time use devices merged the week before the ptg :) VDI in openstack is a somewhat complex topic. on one hand we know that there are a number of gaming companies that host there server side infrastructure on openstack but more recently with the rise fo cloud game streaming companies there are now a new type of nova user that use both GPU pass-though and spice fundamentally to delver high performance vdi solutions to there customers. Spice and to a similar degree RDP are both better positioned to meet the semi real-time demands of cloud gaming then VNC ever will be but its also an area of nova that until recently has not had much development in a very long time.

both however hit a common pain point, how we use and release os-traits so we spent a while discussing possible path forward to streamline the work flow. fundamentally we did not resolve to actually make those changes but absorbing os-traits and os-resource-class back into placement but we generally agreed that we could do that if we happen to find the time.

one time use devices also sparked a discussion on a related placement bug and its fix this could be seen as controversial but we all agreed that the existing behaviour is wrong and that we shoudl fix it. Dan presented there solution and we generally agreed that it seam like a reasonable approach and that we should proceed with the review. For operators resolving the current limitation that placement Allocations cannot be adjusted when provider is over capacity will transitively fix 3 separate nova bugs all rooted in this underlying limitation.

This session was also very useful as John Garbutt reminded us that we used the set reserved=total trick to fix a race condition with cleaning in ironic. i.e. for ironic nodes we set reserved=total when we provision a node and reserved is only reset after cleaning is complete. So while this is undocumented behaviour its not undefined and we now have 2 users of it so we probably shoudl test this in placement properly and update the api ref to document it.

tech debt

oenstack client support

last cycle was a departure form previous cycles when we actually had multiple api changes. unfortunately all 4 api change merged late and none of them landed the sdk and osc change required to fully complete the features. manila share support spice direct Show Scheduler Hints in Server Details image properties in server show

While this is a low-light all of the above have patches in flight and should land early this cycle. This topic was actually split between this session on Wednesday with the main topic discussion happening on Thursday with the client/sdk teams and QA teams for the tempest impact.

glance/nova/cinder corss project

unfortunately i could not attend this due to a conflict but the main topics covered were captured in the glance ptg summery the two most impoant topics in the nova context were

new location API (Cinder & Nova)

Glance has introduced two new Location APIs in the Dalmatian cycle. We can use these APIs to address OSSN-0090 and OSSN-0065. Patches for Nova and Cinder must still be merged, hopefully during the Flamingo cycle.

freeze glance client development

Cinder and Nova will need to use the OpenStack SDK instead of the glanceclient. There is no need to complete this work during this cycle, but we should at least have a good idea of what APIs are currently being used, so that we can have a plan for the next cycles.

How nova store image properties in our db

you may wonder why this is not relevant to the glance discussion it is but this is really internal to nova usage of image properties specifically not storing them if we don’t need them. There is a lot of history to this topic but the clips note version is nova stop supporting custom image properties more than a decade ago but our persistence of them in our db was a little leaky see: https://bugs.launchpad.net/nova/+bug/2098384 for more details. We agreed that moving forward we should not store image metadata keys in our db that are not part of the standard set and we should plan to provide a way to clean up the stale entries that already exist.

Finalizing the secure RBAC goal

the TL;DR of this is the implementing of the manager role and service roles in our policies started a while ago but unfortunately gmann did not have time to work on it in Epoxy until late in the cycle and we didn’t have time to review it when they got unblocked. so in 2025.2 we would like to prioritise completing this work which is the final work remaining for nova to complete the SRBAC community goal.

Eventlet removal

i wont go into this in detail because gibi already wrote up a better summery but This was the second session on eventlet in the context of nova (the first being the community one lead by oslo). Here we discussed the concrete next steps for nova both in terms fo what to do for 2025.2 and beyond. The important take always are we will track the progress of this goal weekly in our meetings, gibi and kamil will work to incrementally remove our direct eventlet imports throughout the cycle and try to re-implement scatter gather in the api so that it can be the first nova service to run without eventlet. This is probably the largest change in novas implementation since we added python3 support or the initial move form twisted to eventlet in the very early days. As a community we have 12-18 months to complete this transition and the nova core team committed to prioritise this work going forward.

Day 4

as with day 3 there were many different topic raised mainly in the context of new features combined with cross project session with neutron, tempest and client/sdk teams.

A QOS api for nova

This was and is likely the most contoverial topic that was raised during the entire ptg bar providing hooks for the XML which was a straight -3 as that is never going to happen. during this session we discussed a proposal that is documented here

this topic not only involved nova but also cinder and neutron changes. in essence this requires fundamentally changes to how nova neutron and cinder work internally and together.

Nova does not supprot changing the QOS applied to neutron Port or cinder volume while the resource is attached to nova instance. on the neutron size if the QoS policy is a min bandwidth or PPS policy, changing the QOS on a port can invalidate the placement of the VMs breaking how those features work.

From a nova point of view if you allow the qos policy on a neutron port to change in neutron while its attached to a nova instance it is therefor a bug.

Cinder and nova have a similar impedance mismatch in that cinder defines it qos policies on the volume type not on the volume. that means that changing the qos on a volume type would impact many volumes nova has not way to support that today. To make matters worse operators already abuse the fact that the fronted qos is updated on live migration to apply these changes live to existing instance even though nova has no idea this is happening and that might fail.

This was a somewhat challenging session as the topic was brought by an operator that already designed how they wanted it to work and were asking for use to just accept the changes even though there design is not really compatible with how self service multi tenant cloud should work.

we spend a lot of the session discussing a possible future nova api for QoS similar to neutrons i.e. where a operator can define a set of QoS policies (effectively a sub set fo flavor extra specs) which an end user could select form and that an operator could apply to a project by default.

while that may be a valid evolution of novas api it touch on previous landmines like composable flavors. we will also need to carefully think about the RBAC implication of who should be able to set a QoS policy on a Project or instance level and if it shoudl be settable via a Flavor. For example the manager role likely shoudl be able to set it on a project level if and only if an admin has not set it. admin likely should be able to set a qos policy in a flavor but when not set a person with the member role should be able to select form any of the QoS policies defined by an admin as they can in neutron today.

while we tired to convey some of this feedback i feel like they did not internalise that well. we also provide context to them on previous discussion form 2019 https://etherpad.opendev.org/p/r.23a243a5860c7406917f86f48cfb4491#L389 during the shanghai ptg.

the proposer resolved to follow up with each team separately and file a spec for the works but we will have to wait and see if took on board our feedback or not.

OVN live migration

who would have guessed that its still a pain… we discussed https://bugs.launchpad.net/nova/+bug/2073254 in Thursday and then again on Friday the TL;DR is neutron now send the event at the right time so we can remove the hacks we added 5 years ago so i have proposed https://review.opendev.org/c/openstack/nova/+/946950 to address that.

We also discussed that we may want to move the creation of the tap to os-vif instead of libvirt. https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d#L1174 This likely should only be done if neutron asked us too but we can make progress even without this.

Watcher topics

This was the main topic i added this PTG to discuss how we could evolve nova to better support watcher use-cases. In this session we discussed how we can annotate instance with metadata to inform policy. i.e. can this instance be live migrated. the second topic was how can nova tell us if a vm can be migrated or if other lifecycle operations are valid like cold migration or shelve. the third topic was slightly different on the usefulness of weighing host by traits with two different use cases, preferred/avoided traits via flavor and automatically avoiding host with *_pressure traits.

annotating instances with metadata

The proposal to add a new api for instance annotation was deferred/rejected in favor of using the existing instance metadata for this use-cases. the proposed set of metadata pairs are

• lifecyle:evacuatable=true|false
• lifecyle:cold-migratable=true|false
• lifecyle:shelvable=true|false
• lifecycle:premetable=true|false
• ha:maintance-stragey:in_place|power_off|migrate
• ha:role=primary|secondary
• ha:priority:[string|number]

these can be documented by the service that uses them but we may want to maintain a list of useful metadata keys like the glance useful image properties doc in nova if they are used. We can/should document that nova considers this a valid use-case for metadata provided the user applies the metadata themselves.

note: on Friday we had related discussion about admin metadata which ill cover later.

instance capabilities

we might work on the name because i will never spell that properly :) this we said we should do and i should write a spec during the session artom mention that in addition to the lifecycle annotations above another capability could be resume on host reboot. after the discussion i realized there are some other lifecycle operation like suspend, pause and snapshot that might make sense to report.

so the full set would be

• lifecyle:evacuatable=true|false
• lifecyle:cold-migratable=true|false
• lifecyle:shelvable=true|false
• lifecycle:premetable=true|false
• lifecyle:resume_on_host_reboot=Ture|False
• lifecyle:pause=true|false
• lifecyle:suspend=true|false
• lifecyle:rescue=true|false
• lifecyle:snapshot=true|false

the idea of instance capabilitiesis that nova should know what operations are valid to perform on an instance based on the virt driver, flavor, image and other project resources like manilla shares, cinder volumes, neutron ports, and cyborg accelerators.

This will need a spec to proceed but the general direction is seen as positive.

*_PRESSURE traits and trait based weighing

Firstly we started this conversation by asserting that if the Trait is custom nova will not touch it. So any human or service could annotate nova resource providers with CUSTOM_ traits

second i asserted i would like to have Standard traits for CPU RAM and DISK pressure

• PRESSURE_CPU
• PRESSURE_MEMORY
• PRESSURE_DISK

network pressure could be added late btu its not one of the one reported by the kernel today https://docs.kernel.org/accounting/psi.html#pressure-interface

one of the open question i had for this session is would nova be ok with the libvirt driver reporting these traits or should watcher annotate host with them based on a periodic audit. the core team was generally ok with the libvirt driver doing this but we will confirm that as part of a spec.

The second half of this topic was focused on how these newly reported traits when placing vms

The proposal create 1-2 new weightier to way based on traits.

for the *pressure traits we likely want to aovid them by default however there may be other traits that an operator may want to avoid or prefer. so that implies that there should be a config option. we decide to punt to the spec/implementation if we wat to include the pressure traits by default or not.

the second weigher would be a flavor based weigher that would extend the existing required/forbidden syntax to supprot preferred and avoided

both weigher will rely on a common change to the host state object to pass the provider summaries this will allow filters and weighers consider traits and available resource classes.

note i did not discuss this in the session but this would also allow me to introduce what i have called in the past a boring host weigher. the concept of the boring host weigher is simple prefer host with fewer traits and resource class or generally a simpler provider tree. The logic, simpler the provider tree the less fancy the hardware available on the host and there for its less likely that placing a random vm on the host will prevent a future vm that need one of the special aspect of the host form fitting.

We agreed that while these are all nice to have and in scope of nova we will propose concrete next steps in a spec if/when we work on them.

OpenApi

Stephen made massive progress last cycle in nova but we ran out of review bandwidth they summerised there progress in general across openstack in a recent blog post An Update on OpenAPI in Openstack during the ptg session we resolved to try and continue making progress on this as nova is in a good position to complete this work. we also discussed that once this work is complete and nova has both request and response schema for all apis we could remove the tempest schema validation in a later release. This would be a nice reduction in technical debt for tempest but will need to wait until all stable/* release of nova support open API validation.

client changes

we covered this earlier but Stephen also lead a discussion on how can we work better between the nova and client/sdk teams to make sure we complete the client work when we make api changes other then not merging them really really late which we should avoid anyway we mainly just reiterated that direct pings are ok and actually welcome when we thing a change is ready to merge. in other word reach out when we are close and use depend-on ectra to call out the deps on the api changes.

Testing, testing and more testing

i wont spend much time on this but interleaved with other topics we discussed how to improve testing in tempest, and can we finish the efforts to []emulate hardware in ci](https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d#L859) to test OTU devices/pci passthough, ndd generic mdevs as well as maybe test sr-iov via IGB down the road.

the answer is yes, we can and should update the spec template to note that at a minimum we shoudl update the tempest schema tests when we add a new api. dan has started a effort to test pci passhtough as part fo testing OTU device using the virtio-RNG. sylvain Will revive there mtty series to create a proxy for generic mdevs.

power management next steps

the final session of day 4 was a deep dive on where power management is, the original design goals and what shoudl we do to fix it. Actually that a lie we did this out of order and talked about this before the OpenAPI and client topics but I’m going to bottom on the etherpad so we will pretend this was the last sessions :)

the over all outcome fo this session was that we shoudl not rush to power off the cores on startup and should wait for the resource tracker to be populated with instance so we can ensure we never power off cores related to an instance that is running. second while we could modify the virt driver interface to allow the comptue manager to defer the power off we should explore if we can just internalise this entirely in the libvirt driver first. if we cant then virt driver change it is. in other words don’t rush to power off the core until we are sure we dont need to power them back up and that they are not in use.

Day 5

live migration with cinder encrypted volumes

This does not work today for exactly the same reason as vtpm migration. admins do not have access to secrets owned by users. The fix, learn lessons form how we fix this for vtpm and apply the same to cinder volumes. specifically the only way we can fix this is the host policy https://specs.openstack.org/openstack/nova-specs/specs/2025.1/approved/vtpm-live-migration.html#id2

cinder volume secrets unlike vtpm are not delete form libvirt when we boot the vm. so we could copy them between hosts once we implement that for vtpm. we already depend on the fact the secret is available in libvirt for resuming guest with encrypted volumes on host reboot.

Manila cross project session

now that you can attach a share to a nova instance we took time in this ptg to reflect on the next steps. There are many but in priority order they are the following:

• finish the osc support and backport to epoxy
• testing (tempest, functional testing elsewhere)
• memfd memory type in nova (possible by default)
• online attach/detach
• attaching a share during server create
• live migration (still need to have this fully supproted in libvirt/qemu)

that is a lot of future work but each is incrementally doable and we resolved to have specs as need for each item

provider.yaml enhancments

The main topic of this session is simple. today you can add custom traits via provider.yaml if you later remove them from the file now will not remove it form the resource provider… it should do that.

so we discuss a few ways to address that and i suggested that nova could copy the file to its state Dir .i.e. /var/lib/nova/last_applied_provider.yaml on the next restart nova could then compare the traits in the last_applied_provider.yaml to the current provider.yaml and compute if any trait shoudl be removed. nova would then up copy the new provider.yaml to last_appled_provider.yaml if it successfully updated placement.

the proposer will do a poc and write up a spec or specless blueprint for this. in the future we could support other changes like the ability to remove resource provider created via provider.yaml but that will initially be out of scope.

instance metadata protection

This was another very productive session, the use case that was presented is as follows, I as a cloud operator have some knowledge of my infrasture that i want to apply to an instance, i would like to make this query able via the rest api, to that end i would like to annotate the instance with this metadata but i would like to be able to prevent a normal user form removing this metadata.

the discussion started with there proposal to have a config option to define metadata protection sort of like how you can define protect image properties that only an admin can set in glance.

the main problem with this approach is interoperability between clouds. we then discussed the idea of defining an admin prefix and perhaps a manager too allow user with those roles to set keys the member or reader users cannot modify. while this would work it woudl require any exiting metadata that an operator might be setting to be updated.

finally we discussed how the lock api and the fact that we record the role of who locked the instance. our general recommendation was to create a spec to allow recording the role of the user that set metadata and to prevent lower privileged roles from removing keys set by higher privileged roles. that would define the following prescience relationship admin > manager > member

we said we woudl be happy to review a spec to that effect if they write one.

update/set delete on terminate for neuton port

This was short, we should supprot it but we just have not had a contributor to work on it so patches are welcome. we suggested starting with a poc and filing a spec once they are happy they have something working

[Graceful shutdown] (https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d#L1089)

this was another rapid fire topic. we explained the previous attempt and the fact we currently don’t have capacity to work on this.

john raised an interesting alternative which would be to add a maintenance mode concept. this could be implemented as a block in the api of all write operations alternatively we could decorate all rpc functions that start an instance action or move op with a decorator that would reject the rpc with a 409 if the compute service was set to maintenance mode either as a new sate for disabled or as a new filed.

we all agree that a more graceful shutdown would be nice to have and a poc of different approaches might be a good next step currently the core team does not have time to work on it but we welcome other to propose solutions. The main call to action is to report any bugs for problem we know of when there is not a graceful shutdown while we may or may not be able to fix them that will provide a list of edge cases that need to be solved for.

iothread and block device queue

Again this is a topic that was started on in the past and stalled out. there has been previous resarch done by redhat and other on how virt-blk scales with io vs iothread vs queues https://developers.redhat.com/articles/2024/09/05/scaling-virtio-blk-disk-io-iothread-virtqueue-mapping

this session basily was quick and after some discussion fo the history and previous recommendations of our virt team to always have at least 1 iothread we agreed that the propose should write a spec to introduce 2 new flavor extra specs/image properties to be able to request the number of iothread and block device queues. .e.g hw:io_threads=4 hw:blk_multiqueue:2

the detail will be covered in the the spec but when unset we should likely always allocate 1 iothread and preserve the libvirt default of creating one virtio-blk queue per vcpu.

filtering server list by neutron network

the final large topic of the ptg was a proposal to filter server list by neutron network (possibly also subnet in the future)

the general problem to day is while you can make a single query to get all port device-id values (the nova instance the port is attached too) for all ports on a neutron network, you cant then call nova with 100s of uuids in one query to list the server details. the proposer uses this information to determine the owner of the vm so that when they need to do maintenance or a particular network runs out of ips they can reach out to there customer and ask them to move a port or release an ip ectra.

most of this session revolved around is this in scope of nova to do or not.

in the end we resolved that if the data is in the nova db and easy to filter on or we can make a efficient query to another service that a user cant replicate because they cant pass the resultant uuid to nova for list, then its ok to extend server list to do this.

we will only add filter on a case by case basis. as a result we agreed that if the proposer writes a spec and works on a poc we will review them.

PTG summary.

a lot happened in nova and watcher this PTG and more happened in the TC and other project session that i have not covered at all. hopefully someone will find this useful. i have tried to summarize my understanding of all the sessions i attended and provide links to the sources where possible.

we got a lot done in what was a total of maybe 20 hour of real-time meetings but we equally now have alot of work to do to achieve all we discussed including many specs to read and write.

I woudl personally like to thank rene for running the nova sessions and all the rest that contributed there time. it was a long, tiring and stressful week for many but it was also good to see us disucss these topics as a community in the open.