Network Extension: Orchestrate external Network devices#13032
Network Extension: Orchestrate external Network devices#13032weizhouapache wants to merge 4 commits intoapache:mainfrom
Conversation
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13032 +/- ##
============================================
- Coverage 17.95% 17.95% -0.01%
- Complexity 16522 16630 +108
============================================
Files 6022 6025 +3
Lines 541387 543813 +2426
Branches 66346 66837 +491
============================================
+ Hits 97211 97630 +419
- Misses 433210 435093 +1883
- Partials 10966 11090 +124
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17500 |
There was a problem hiding this comment.
Pull request overview
Introduces a new NetworkOrchestrator extension type to integrate external network orchestration into CloudStack, expanding extension registration to PhysicalNetwork, adding an API to update registered extension details, and wiring dynamic provider/offering behavior in UI and backend.
Changes:
- Added PhysicalNetwork-scoped extension registration + update flow (API, DB mappings, UI).
- Added dynamic provider resolution for extension-backed providers (transient Provider support, element dispatch, capability plumbing).
- Added/updated tests and schema updates to support the new lifecycle and larger extension detail values.
Reviewed changes
Copilot reviewed 73 out of 75 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| ui/src/views/offering/AddVpcOffering.vue | Adds extension-backed providers to VPC offering creation UI and builds services from extension details. |
| ui/src/views/offering/AddNetworkOffering.vue | Adds extension-backed providers to network offering UI and derives supported services from extension details. |
| ui/src/views/infra/network/providers/ProviderListView.vue | Adjusts row key and renders provider “details” as tags. |
| ui/src/views/infra/network/ServiceProvidersTab.vue | Adds dynamic tabs for registered NetworkOrchestrator extensions and a modal to register them on a physical network. |
| ui/src/views/extension/UpdateRegisteredExtension.vue | New UI form to update details of an extension-resource registration. |
| ui/src/views/extension/RegisterExtension.vue | Allows registering an extension against PhysicalNetwork (in addition to Cluster). |
| ui/src/views/extension/ExtensionResourcesTab.vue | Adds “Update extension resource details” action and fixes unregister loading state handling. |
| ui/src/views/extension/CreateExtension.vue | Adds NetworkOrchestrator as a creatable extension type in UI. |
| ui/src/views/extension/AddCustomAction.vue | Adds resourcetype selection and constrains it based on extension type. |
| ui/src/config/section/network.js | Adds “Run Action” entry point for networks. |
| ui/src/config/section/infra/phynetworks.js | Extends physical network update args to include externaldetails. |
| ui/src/config/section/extension.js | Adds filtering for networkorchestrator. |
| ui/public/locales/en.json | Adds UI strings for external network provider and extension update flows. |
| server/src/test/java/com/cloud/vpc/MockNetworkModelImpl.java | Updates mock to support new NetworkModel methods (resolveProvider, etc.). |
| server/src/test/java/com/cloud/vpc/MockNetworkManagerImpl.java | Updates mock signature for updatePhysicalNetwork to include externalDetails. |
| server/src/test/java/com/cloud/network/vpc/VpcManagerImplTest.java | Adds tests for extension-backed VPC elements and provider resolution. |
| server/src/test/java/com/cloud/network/vpc/NetworkACLManagerTest.java | Injects NetworkServiceMapDao for ACL provider checks in tests. |
| server/src/test/java/com/cloud/network/firewall/FirewallManagerImplTest.java | Adds tests for extension-backed Firewall/PortForwarding dispatch. |
| server/src/test/java/com/cloud/network/element/ConfigDriveNetworkElementTest.java | Wires ExtensionHelper into NetworkModel for extension-provider detection in tests. |
| server/src/test/java/com/cloud/network/as/AutoScaleManagerImplTest.java | Updates test to mock resolveProvider. |
| server/src/test/java/com/cloud/network/UpdatePhysicalNetworkTest.java | Updates call site for new updatePhysicalNetwork signature. |
| server/src/test/java/com/cloud/network/NetworkServiceImplTest.java | Adds tests for physical-network external details propagation to extension-resource maps. |
| server/src/test/java/com/cloud/network/NetworkModelImplTest.java | Adds tests for extension-backed provider resolution and element lookup. |
| server/src/test/java/com/cloud/network/MockNetworkModelImpl.java | Updates mock to implement new NetworkModel methods. |
| server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java | Uses resolveProvider, supports extension-backed VPC providers/elements, and adjusts provider checks. |
| server/src/main/java/com/cloud/network/vpc/NetworkACLManagerImpl.java | Adds fallback dispatch to provider name via service map for NetworkACL. |
| server/src/main/java/com/cloud/network/lb/LoadBalancingRulesManagerImpl.java | Adds fallback dispatch to provider name via service map for LB. |
| server/src/main/java/com/cloud/network/firewall/FirewallManagerImpl.java | Adds fallback dispatch to provider name via service map for Firewall/PF and strengthens capability validation. |
| server/src/main/java/com/cloud/network/as/AutoScaleManagerImpl.java | Uses resolveProvider for provider validation. |
| server/src/main/java/com/cloud/network/NetworkServiceImpl.java | Extends updatePhysicalNetwork to accept externalDetails and propagate to extension-resource mappings; uses resolveProvider. |
| server/src/main/java/com/cloud/network/NetworkModelImpl.java | Adds extension-provider dispatch, transient provider resolution, and includes extension-backed providers in supported providers list. |
| server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java | Uses resolveProvider during offering creation/validation. |
| server/src/main/java/com/cloud/api/ApiResponseHelper.java | Uses resolveProvider when building NSP responses. |
| server/src/main/java/com/cloud/api/ApiDBUtils.java | Adds canElementEnableIndividualServicesByName helper. |
| plugins/network-elements/juniper-contrail/src/test/java/org/apache/cloudstack/network/contrail/management/ManagementServerMock.java | Updates call site for new updatePhysicalNetwork signature. |
| framework/extensions/src/test/java/org/apache/cloudstack/framework/extensions/manager/ExtensionsManagerImplTest.java | Adds extensive tests for PhysicalNetwork registration, updateRegisteredExtension, network custom actions, and provider capability parsing. |
| framework/extensions/src/test/java/org/apache/cloudstack/framework/extensions/dao/ExtensionResourceMapDaoImplTest.java | Adds tests for new DAO methods. |
| framework/extensions/src/test/java/org/apache/cloudstack/framework/extensions/api/UpdateRegisteredExtensionCmdTest.java | Adds tests for the new updateRegisteredExtension API command. |
| framework/extensions/src/test/java/org/apache/cloudstack/framework/extensions/api/ListExtensionsCmdTest.java | Adds tests for new ListExtensionsCmd getters. |
| framework/extensions/src/main/resources/META-INF/cloudstack/core/spring-framework-extensions-core-context.xml | Registers the NetworkExtensionElement bean. |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/vo/ExtensionResourceMapDetailsVO.java | Increases value column length to 4096 (entity). |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/vo/ExtensionDetailsVO.java | Increases value column length to 4096 (entity). |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/manager/ExtensionsManagerImpl.java | Implements PhysicalNetwork registration/unregistration, updateRegisteredExtension, network custom actions, and capability/service parsing. |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/manager/ExtensionsManager.java | Adds updateRegisteredExtensionWithResource to interface. |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/dao/ExtensionResourceMapDaoImpl.java | Adds listByResourceIdAndType and listResourceIdsByType. |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/dao/ExtensionResourceMapDao.java | Declares new DAO methods. |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/dao/ExtensionDaoImpl.java | Adds listByType implementation. |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/dao/ExtensionDao.java | Declares listByType. |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/api/UpdateRegisteredExtensionCmd.java | Adds new API command updateRegisteredExtension. |
| framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/api/ListExtensionsCmd.java | Adds filters for type, resourceId, resourceType. |
| engine/schema/src/main/resources/META-INF/db/schema-42210to42300.sql | Expands extension details value columns to VARCHAR(4096). |
| engine/schema/src/main/java/com/cloud/network/vpc/dao/VpcDaoImpl.java | Persists VPC service providers using string provider names (supports dynamic providers). |
| engine/schema/src/main/java/com/cloud/network/vpc/VpcServiceMapVO.java | Updates ctor to accept service/provider names as strings. |
| engine/schema/src/main/java/com/cloud/network/dao/NetworkServiceMapVO.java | Adds ctor accepting service/provider as strings. |
| engine/schema/src/main/java/com/cloud/network/dao/NetworkDaoImpl.java | Persists network service providers using string provider names (supports dynamic providers). |
| engine/orchestration/src/test/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestratorTest.java | Adds tests for including extension-backed network elements. |
| engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java | Extends orchestration to include transient extension-backed NetworkElements and uses resolveProvider. |
| api/src/test/java/org/apache/cloudstack/extension/ExtensionCustomActionTest.java | Adds test for Network resource type. |
| api/src/test/java/org/apache/cloudstack/extension/CustomActionResultResponseTest.java | Adds tests for new getters/isSuccess. |
| api/src/test/java/com/cloud/network/NetworkTest.java | Adds tests for Provider equality/transient provider behavior. |
| api/src/main/java/org/apache/cloudstack/extension/NetworkCustomActionProvider.java | Introduces interface for network-scoped custom actions. |
| api/src/main/java/org/apache/cloudstack/extension/ExtensionResourceMap.java | Adds PhysicalNetwork resource type. |
| api/src/main/java/org/apache/cloudstack/extension/ExtensionHelper.java | Adds PhysicalNetwork helper methods and network capability helpers. |
| api/src/main/java/org/apache/cloudstack/extension/ExtensionCustomAction.java | Adds Network as a custom action resource type. |
| api/src/main/java/org/apache/cloudstack/extension/Extension.java | Adds NetworkOrchestrator as an extension type. |
| api/src/main/java/org/apache/cloudstack/extension/CustomActionResultResponse.java | Adds getResult() and isSuccess(). |
| api/src/main/java/org/apache/cloudstack/api/command/admin/network/UpdatePhysicalNetworkCmd.java | Adds externaldetails map param and passes through to service layer. |
| api/src/main/java/com/cloud/network/element/NetworkElement.java | Adds rollingRestartSupported() default method. |
| api/src/main/java/com/cloud/network/NetworkService.java | Extends updatePhysicalNetwork signature with externalDetails. |
| api/src/main/java/com/cloud/network/NetworkModel.java | Adds resolveProvider and canElementEnableIndividualServicesByName. |
| api/src/main/java/com/cloud/network/Network.java | Adds NetworkExtension provider, transient provider factory, and Provider equality by name. |
| api/src/main/java/com/cloud/event/EventTypes.java | Adds EVENT_EXTENSION_RESOURCE_UPDATE. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (provider == null) { | ||
| provider = _networkModel.getDefaultUniqueProviderForService(service).getName(); | ||
| } else { | ||
| provider = _networkModel.resolveProvider(provider).getName(); |
There was a problem hiding this comment.
Possible NullPointerException: _networkModel.resolveProvider(provider) may return null, but the code immediately calls .getName(). Guard against null (and return a clear Unsupported/Invalid provider error) to avoid failures when an invalid provider string is stored in the offering service map.
| provider = _networkModel.resolveProvider(provider).getName(); | |
| final Provider resolvedProvider = _networkModel.resolveProvider(provider); | |
| if (resolvedProvider == null) { | |
| throw new InvalidParameterValueException("Invalid provider " + provider + " configured for service " + service); | |
| } | |
| provider = resolvedProvider.getName(); |
| final List<Provider> providers = new ArrayList<>(); | ||
| for (final String providerName : providerNames) { | ||
| providers.add(Network.Provider.getProvider(providerName)); | ||
| providers.add(_networkModel.resolveProvider(providerName)); |
There was a problem hiding this comment.
getNetworkProviders adds the result of resolveProvider without filtering nulls. If any provider name is not resolvable, this list will contain null and can cause downstream NPEs if the providers list is iterated (or logged) elsewhere. Consider skipping null providers (and optionally logging a warning).
| providers.add(_networkModel.resolveProvider(providerName)); | |
| final Provider provider = _networkModel.resolveProvider(providerName); | |
| if (provider != null) { | |
| providers.add(provider); | |
| } |
| for (NetworkOfferingServiceMapVO instance : map) { | ||
| String service = instance.getService(); | ||
| Set<Provider> providers; | ||
| providers = serviceProviderMap.get(Service.getService(service)); | ||
| if (providers == null) { | ||
| providers = new HashSet<Provider>(); | ||
| } | ||
| providers.add(Provider.getProvider(instance.getProvider())); | ||
| providers.add(resolveProvider(instance.getProvider())); | ||
| serviceProviderMap.put(Service.getService(service), providers); |
There was a problem hiding this comment.
Possible NPE/data corruption: resolveProvider(instance.getProvider()) can return null, but the null value is added to the providers set. This can later trigger NPEs when code assumes every provider in the set is non-null. Skip null providers (and/or throw an exception if the DB contains an invalid provider name).
| if (!handled) { | ||
| // Get provider name and get the element by provider name (it could be an external provider) | ||
| String pfProviderName = networkServiceMapDao.getProviderForServiceInNetwork(network.getId(), Service.PortForwarding); | ||
| if (pfProviderName != null) { | ||
| NetworkElement element = _networkModel.getElementImplementingProvider(pfProviderName); | ||
| handled = ((PortForwardingServiceProvider) element).applyPFRules(network, (List<PortForwardingRule>) rules); | ||
| } |
There was a problem hiding this comment.
The extension-provider fallback casts element to PortForwardingServiceProvider without checking for null or instanceof. If the provider name isn't backed by a PF-capable element, this will throw at runtime. Add null/instanceof checks and handle the unsupported case gracefully.
| Provider serviceProvider = networkModel.resolveProvider(result.getProviderName()); | ||
| boolean canEnableIndividualServices = ApiDBUtils.canElementEnableIndividualServices(serviceProvider); | ||
| response.setCanEnableIndividualServices(canEnableIndividualServices); | ||
|
|
There was a problem hiding this comment.
networkModel.resolveProvider(...) can return null, but the result is passed into ApiDBUtils.canElementEnableIndividualServices(...) which will NPE on null. Consider calling the new ApiDBUtils.canElementEnableIndividualServicesByName(result.getProviderName()), or guard against null and default canEnableIndividualServices to false.
| if (!foundProvider) { | ||
| // Get provider name and get the element by provider name (it could be an external provider) | ||
| String aclProviderName = networkServiceMapDao.getProviderForServiceInNetwork(network.getId(), Service.NetworkACL); | ||
| if (aclProviderName != null) { | ||
| foundProvider = true; | ||
| NetworkElement element = _networkModel.getElementImplementingProvider(aclProviderName); | ||
| handled = ((NetworkACLServiceProvider) element).applyNetworkACLs(network, rules); | ||
| } |
There was a problem hiding this comment.
The external-provider fallback path casts element without checking for null or interface compatibility. If getElementImplementingProvider returns null (or a NetworkElement that doesn't implement NetworkACLServiceProvider), this will throw at runtime. Add null/instanceof checks and return false (or throw a controlled exception) when the provider can't handle NetworkACL.
| if (!handled) { | ||
| // Get provider name and get the element by provider name (it could be an external provider) | ||
| String lbProviderName = _ntwkSrvcDao.getProviderForServiceInNetwork(network.getId(), Service.Lb); | ||
| if (lbProviderName != null) { | ||
| NetworkElement element = _networkModel.getElementImplementingProvider(lbProviderName); | ||
| handled = ((LoadBalancingServiceProvider) element).applyLBRules(network, rules); | ||
| } |
There was a problem hiding this comment.
The external-provider fallback path casts element without checking for null or instanceof LoadBalancingServiceProvider. If the provider name is present in the service map but no element implements it (or it doesn't support LB), this will throw a ClassCastException/NPE. Add null/instanceof checks and handle the unsupported case gracefully.
| if (!handled) { | ||
| // Get provider name and get the element by provider name (it could be an external provider) | ||
| String fwProviderName = networkServiceMapDao.getProviderForServiceInNetwork(network.getId(), Service.Firewall); | ||
| if (fwProviderName != null) { | ||
| NetworkElement element = _networkModel.getElementImplementingProvider(fwProviderName); | ||
| handled = ((FirewallServiceProvider) element).applyFWRules(network, rules); | ||
| } |
There was a problem hiding this comment.
The extension-provider fallback casts element to FirewallServiceProvider without checking for null or instanceof. If networkServiceMapDao returns a provider name with no matching element (or one that doesn't implement FirewallServiceProvider), this will throw at runtime. Add null/instanceof checks and treat it as "not handled" (or throw a controlled exception).
| { | ||
| api: 'runCustomAction', | ||
| icon: 'thunderbolt-outlined', | ||
| label: 'label.run.action', | ||
| dataView: true, | ||
| show: (record) => { | ||
| return 'runCustomAction' in store.getters.apis && | ||
| 'listCustomActions' in store.getters.apis && | ||
| record.service && record.service.some(s => | ||
| s.provider && s.provider.some(p => p.name === 'ExternalNetwork')) | ||
| }, |
There was a problem hiding this comment.
The runCustomAction action visibility check hard-codes provider name ExternalNetwork, but extension-backed providers are selected by extension/NSP name (and the server-side constant introduced is NetworkExtension). As written, the action will likely never appear for extension-backed networks. Update the predicate to detect extension-backed providers (e.g., via provider name NetworkExtension or by matching any provider name that is a NetworkOrchestrator extension).
| if (provider == null) { | ||
| // Default to VPCVirtualRouter | ||
| provider = Provider.VPCVirtualRouter.getName(); | ||
| } else { | ||
| provider = _ntwkModel.resolveProvider(provider).getName(); | ||
| } |
There was a problem hiding this comment.
Possible NullPointerException: _ntwkModel.resolveProvider(provider) may return null for unknown provider names, but the code immediately calls .getName(). Add a null check and throw an InvalidParameterValueException (or keep the original provider string) when resolution fails.
Overview
This PR introduces and wires a new extension model for external network orchestration in CloudStack, centered on a new extension type:
NetworkOrchestrator.It extends the extension lifecycle from cluster-only registration to physical-network registration, adds API support for updating registered extension metadata, and enables automatic offering creation (network and VPC) based on provider-declared supported services and capabilities.
It also adds smoke coverage (KVM-only) using a Linux network namespace based implementation.
What's new
1) New extension type:
NetworkOrchestratorNetworkOrchestrator.2) Register extension with PhysicalNetwork (in addition to Cluster)
PhysicalNetworkresource.3) Physical network registration details support
PhysicalNetworkare handled similarly to cluster registration details.4) New API: update registered extension
5) Offering automation from external provider capabilities
supportedservicesservice capabilities6) Network support via generated offerings
Using offerings backed by the external provider, networks can be created and operated with supported services/capabilities.
Supported operations include (based on provider capabilities):
7) VPC support via generated offerings
Using VPC offerings backed by the external provider, VPCs and tiers can be created and operated with supported services/capabilities.
Supported operations include (based on provider capabilities):
8) Linux network namespace based external implementation
https://github.com/apache/cloudstack-extensions/tree/network-namespace/Network-Namespace9) Smoke test coverage (KVM-only)
API / behavior changes
NetworkOrchestratorPhysicalNetworkresource targetsWhy this change
This enables CloudStack to integrate external network orchestrators as first-class providers with:
Testing
Added/updated tests
Validation intent
Notes / limitations
Types of changes