Transcription
White paperCisco public5G Automation ArchitectureWhite paperAuthors:Milan StolicJulie Ann ConnaryWei YanArghya MukherjeeMohammad SalaheldinRob Piasecki 2020 Cisco and/or its affiliates. All rights reserved.Page 1 of 29
White paperCisco publicContents1. Abstract 32. Scope 33. Introduction 33.1 Standards view 3.2 5G domains 3.3 Network slicing 3.4 Necessity of automation 3.5 Challenges of moving to the cloud 4. Solution overview 4.1 Customer-facing service and resource-facing service 4.2 DevOps support functions 5. Cross-domain orchestration 5.1 Cross-domain orchestration mapping to the standards 5.2 Cross-domain orchestrator requirements 5.3 Northbound and southbound integration 6. Domain-level orchestration 6.1 RAN domain orchestration 6.2 Transport domain orchestration 6.3 Packet core domain orchestration 6.4 Data center domain orchestration 6.5 Application domain orchestration 7. Service assurance 7.1 Cross-domain SA 7.2 Domain-level SA 7.3 Deployment workflow with closed loop 8. Operational challenges 8.1 Importance of CI/CD and DevOps 467889910111111121313141617182020212225269. Conclusion 2610. References and credits 2020Ciscoitsand/orits affiliates.rights reserved. 2020 Ciscoand/oraffiliates.All rights Allreserved.27Page 2 of 29
White paperCisco publicAbstractMobile service providers are facing increased customer demand with stagnant average revenue per user (ARPU),forcing the need for new revenue-generating services. Network transformation is needed in all network domains to meetthis demand and enable new revenue streams for service providers. This transformation remains daunting for manyoperators. [1]5G by its nature spans many domains beyond mobility and requires a transformation unseen so far. This paper willprovide a simplified view of 5G automation architecture by focusing on per-domain and cross-domain automation andorchestration, while taking into consideration 5G components such as Cloud-RAN, Control User-Plane Separation(CUPS), Multiaccess Edge Computing (MEC), network slicing, and xHaul that have to work in tight synergy.ScopeThis document provides a reference for 5G network automation and orchestration in general terms, applicable to mostuse-case categories. It is an architecture guidance that glues the different components of 5G automation together. Thispaper can be shared internally within Cisco and externally.This document is targeting managers, program managers, and planning, implementation, and operations engineers onboth SP and vendors’ sides involved in 5G automation planning and implementation. It provides an at-a-glance view ofoptions and considerations for orchestrating different aspects and domains of a 5G network.Introduction5G is a fundamental transformation of mobile technology that will lay the path for many future technologies. 5G positionsmobile technology as the core platform for advancing the innovations in future defining technologies [1].5G enables service providers to be an integral part of various businesses, driving growth by providing customizedservices beyond connectivity. This is very important for SPs constantly struggling with identifying new revenueopportunities.At a high level, the 5G network architecture is depicted in the figure below. Even at a high level there are significantadditional complexities compared to the previous generations: it is access agnostic and has disaggregated Radio AccessNetwork (RAN); fronthaul and/or midhaul are introduced; Mobile Core has new elements; Control and User planes areseparated; and edge computing is introduced, to name a few. Understanding evolving requirements, stitching togetherthe pieces of the new network, and transitioning services and traffic from a very different existing network can be acomplicated task.Figure 1. 5G architecture 2020 Cisco and/or its affiliates. All rights reserved.Page 3 of 29
White paperCisco publicAs with many other major changes in the industry, 5G is expected to arrive in phases with a significant impact on existinginfrastructure. This document will not focus on deployment phases including 5G NSA (non-standalone); its focus will be5G automation and orchestration architecture in terms of functionality, and with respect to different network domains andcross-domain, end to end. It will also provide a view and mapping to the 3GPP and ETSI standards, as they apply to theAutomation Architecture.3.1 Standards viewSeveral standard bodies and working groups are actively identifying key requirements and interfaces required for 5Gnetwork automation. These different standard bodies and working groups are complementing each other in order toidentify an end-to-end orchestration strategy.The key standard bodies and working groups are: 3GPP TMForum ETSI NFV ETSI ZSM ONAPIn the next chapter, we define the key roles and outcomes from each of the standard bodies.3.1.1 3GPP3GPP is concerned with identifying a management and orchestration architecture for 3GPP domains, namely 5G coreand RAN, in order to achieve the main use case for automation, which is 5G slicing. 3GPP has defined the architecture in3GPP standards TS 28.530 [4], TS 28.531 [5], TS 28.532 [6], and TS28.533 [7].3GPP identified the following main network functions to aid the 5G automation: CSMF: Communication Service Management Function NSMF: Network Slice Management Function NSSMF: Network Sub-Slice Management Function NFMF: Network Function Management Function MDAF: Management and Data Analytics Function NWDAF: Network Data Analytics Function3.1.2 TMForumThe TMForum provides an end-to-end reference framework for automation and orchestration across multiple domainsthat includes standard interfaces for integration with Operations and Billing Support Systems (OSS/BSS).In general, the TMForum framework provides technology standard body information models when they are needed. For5G network slicing, it provides alternative specifications to the 3GPP for interfaces between technology domains.The TMForum OpenAPI work identifies the following key standard interfaces that can be used for OSS/BSS integration toa network domain: TMF640/641 Service Activation and Configuration API [9][10] TMF638 Service Inventory API [11] TMF633 Service Catalogue API [12] TMF639 Resource Inventory Management API [23] 2020 Cisco and/or its affiliates. All rights reserved.Page 4 of 29
White paperCisco public3.1.3 ETSI NFVETSI NFV (Network Function Virtualization) identifies the NF (Network Function) lifecycle management procedures in avirtualized environment.Our focus in this whitepaper is a part of the ETSI standard specifying the procedure needed to integrate with the 5Gmanagement and orchestration framework (especially at the NSSMF–NFVO connection and NFMF–VNFM level, asdefined in 3GPP standards) for slice instantiation and lifecycle management of VNFs and CNFs.ETSI specifies interfaces in ETSI NFV standards, namely the SOL005 [8] interface, where the NSSMF can be used tocreate, scale, or delete a virtualized network function.3.1.4 ETSI ZSM (Zero-Touch Network and Service Management)The reference architecture—defined in ETSI GS ZSM 002 V1.1.1 [13]—employs a set of architectural principles and aservice-centric architectural model to define, at a high level, a set of management services for zero-touch networkand service management. It also defines a means of management service integration, communication, interoperation,and organization at a functional level. Procedures and detailed information models are beyond the scope of the presentdocument. The reference architecture also defines normative provisions for externally visible management services,defined as part of the reference architecture, as well as recommendations for their organization. It is assumed that thearchitectural patterns introduced in the present document can be used not only for the ZSM framework, but also forarchitecture and design of individual management services.ETSI ZSM has 12 design principles: Modularity Extensibility Scalability Model-driven open interfaces Closed-loop management automation Stateless management functions Resilience Separation of concerns in management Service composability Intent-based interfaces Function abstraction SimplicityTo achieve the purpose of the design principles, the architecture was developed with the following building blocks: Management services Management functions Management domains E2E service management domain Integration fabric Data services 2020 Cisco and/or its affiliates. All rights reserved.Page 5 of 29
White paperCisco public3.1.5 The Linux Foundation—ONAP ProjectONAP[14] defined network slicing as a key use case in the required service provider automation. ONAP recognizes thiseffort as a multi-release effort to achieve the automation use cases.At the time of writing this document, ONAP’s next release is Frankfurt release [15][16], which specifies implementingonly CSMF and NSMF functions as part of ONAP, and integrating with a third-party NSSMF using standard APIs.From the Network Slice Instance (NSI) lifecycle management point of view, in this release, ONAP will implementfunctions in the red box below, which includes NSI design and preprovisioning, NSI instantiation and configuration, andNSI activation and deactivation.Figure 2. Scope for ONAP Frankfurt release3.2 5G domainsWhile there is no strict definition of a “domain,” a service provider’s networks can be loosely divided into the followingdomains: RAN, Transport, Packet Core, Data Center (DC), and Application. This division is not strictly physical; it looks atdifferent functional blocks used to build a network, and these blocks may physically overlap. However, logical division isimportant from the automation and orchestration perspective. Details and configuration of each domain’s elements willvary depending on the specific use case [2]. A high-level diagram of network domains, as described, is shown below.Figure 3. 5G network domains 2020 Cisco and/or its affiliates. All rights reserved.Page 6 of 29
White paperCisco public3.3 Network slicingAmong the 3GPP specifications for the 5G core system are three key types of network services driving the need fornetwork slicing, also viewed as use-case categories: Enhanced Mobile Broadband (eMBB): Requirements focused on greater bandwidth and moderate improvements tolatency for 4G LTE and 5G NR deployments. Ultra-Reliable Low-Latency Communications (URLLC): Provides increased bandwidth for 5G core deployment with afocus on end-to-end latency reduction. Massive Machine-Type Communications (mMTC): Developed to provide connectivity to a larger set of devices (forexample, Internet of Things [IoT] sensors) that typically transmit small blocks of data via low-bandwidth paths.The network slicing concept consists of three layers [3]:1. Service Instance Layer,2. Network Slice Instance Layer, and3. Resource LayerEach service is represented by a Service Instance. Typically, services can be provided by the network operator or bythird parties. A Service Instance is an instance of an end-user service or a business service that is realized within or by aNetwork Slice.A Network Slice Instance (NSI) is a managed entity in the operator’s network with a lifecycle independent of the lifecycleof the service instance(s). In particular, service instances are not necessarily active through the whole duration of therun-time phase of the supporting NSI [3].Figure 4. NSI lifecyclePreperationDesignInstallation, Configuration,and twork erminationThe following phases describe the network slice lifecycle [3]: Preparation phase: Creation and verification of network slice template(s), onboarding, preparing the necessarynetwork environment used to support the lifecycle of NSIs, and any other preparations needed in the network. Instantiation, Configuration, and Activation phase: All resources shared/dedicated to the NSI have been createdand are configured (such as, to a state where the NSI is ready for operation). Run-Time phase: NSI is capable of traffic handling to support communication services of a certain type(s). Thisphase includes monitoring, as well as activities related to modification. Modification could map to several workflowsrelated to run-time tasks (for example, upgrade, reconfiguration, NSI scaling, changes of NSI capacity, changes ofNSI topology, and association and disassociation of network functions with NSI). Decommissioning phase: Deactivation, the reclamation of dedicated resources, and configuration of shared/dependent resources. 2020 Cisco and/or its affiliates. All rights reserved.Page 7 of 29
White paperCisco public3.4 Necessity of automationAlthough at this point 5G is predominantly in initial testing and deployment phases, most MNOs have not rushed toimplement automation; however, they mostly understand the associated challenges [18]. The number of networkelements needed to run a 5G network in all domains means that any cost-effective 5G deployment will requireautomation to deploy efficiently, and keep things running. The reason for hesitation is a perceived lack of maturity andslow progress in automation technology.In an automated deployment scenario, all or most of the heavy preplanning manual work can be eliminated. ArtificialIntelligence (AI) systems, based on Machine Learning (ML), can model how network functions would perform under normaland high-stress conditions. Using run-time performance data, the system can ensure automatic deployment of newelements as needed in a Continual Integration/Continual Deployment (CI/CD) mode. For ongoing optimization and ServiceAssurance, systems can collect and analyze equipment feeds of all types and examine their performance, determining if itmatches the parameters that SPs require and expect, as well as the customer experience of the end users.3.5 Challenges of moving to the cloud5G networks, including 5G core and edge, can be hosted in a public or private cloud, in a centralized or decentralizedmanner. A 5G network can be hosted entirely on a customer premise, or in an edge DC providing improvements inlatency, availability, security, etc. [19].However, hosting the application in the cloud comes with its own challenges: New operations management and security solutions are required Finding use cases and business models behind the cloud edge Clouds must support the required high throughput Operations, processes, security, and availability must meet the expectations of SPs and their customersCloud providers offer their own solutions to ease the design of moving services to the cloud [24][25].Disaggregation of functionality is typical in future networks. Some parts of the infrastructure (for example, User PlaneFunction [UPF] or Virtual Centralized Unit [vCU]) may need to be co-located with the Application Function (AF),and applications served from the Cloud. With this new disaggregation comes additional parties to be involved, withcorresponding clarifications about ownership, responsibilities, and management, that need to be made in a timely manner. 2020 Cisco and/or its affiliates. All rights reserved.Page 8 of 29
White paperCisco publicSolution overviewThe broad scope of 5G automation requires a product-agnostic approach to a 5G system architecture that distills the keyrecommendations and principles of the various extant standards into a deployable operational framework. The solutionshown below breaks down the various functions in the architecture in a way that is prescriptive enough to clearly definethe necessary building blocks, while retaining the flexibility to work with the operational models and needs of all types ofmobile service providers.Figure 5. 5G automation architectureThe figure above shows a high-level overall 5G automation architecture that can be divided into the following areas: A cross-domain layer that acts across multiple domains while also presenting the majority of customer-facingservices Domain-specific layers that translate the intent of the cross-domain layer into the domain intelligence that managesthe appropriate resource-facing services Managed resources, which include virtualized, cloud-native, and physical network functions specific to a domain DevOps support functions essential for automation and operationFunctional elements of each layer are shown in the diagram above.4.1 Customer-facing service and resource-facing serviceThe fundamental goals of a successful 5G automation architecture are to manage complexity and increase operationalefficiency. A layered architecture that separates the specifics of the resource-facing services at the domain level fromthe customer-facing services at the cross-domain level is critical in fulfilling these goals. Key considerations for thisseparation are: 2020 Cisco and/or its affiliates. All rights reserved.Page 9 of 29
White paperCisco public Consistent abstractions Standardized interfaces Idempotent/reversible operations Clear distribution of control between domainsThe intent is to allow the resource-facing functions greater control of fine-grained operations at the domain level, whilepresenting abstracted operations for the customer-facing functions to compose slice-level operations from multipledomains.4.1.1 Functions in CFS and RFSThe following functions are common in the RFS layer in all the domains: NFVO and G-VNFM: Manage the lifecycle of virtualized or containerized workloads. NFMF: Manages configuration, fault, and performance of one or more individual network functions. NSSMF: Manages workflows and orchestrates and performs service lifecycle management at a domain level. It mayalso present some visualization capability such as topology. MDAF: Provides analytics and assurance capabilities at the domain level. Active inventory—the domain view of thestate of all managed devices in order to provide correlation and running data—shall also be maintained while beingsynchronized with the overall static and physical inventory maintained by the OSS/BSS. It may also provide somedomain-level optimization functions.The following functions are present at the cross-domain/CFS layer: NSMF: Performs cross-domain slice management functions, utilizing a workflow engine and a cross-domainorchestrator. Performs policy enforcement and resource enforcement for all slice operations. NWDAF (Network Data Analytics Function): Provides analytics and assurance capabilities at the cross-domain(slice and service) level, while maintaining cross-domain active inventory. It will enable service-level, closed-loopoperations via E-W integration with the NSMF. CSMF: Acts as a presentation layer—presents the service catalog, workflow visualization, and analytics/serviceassurance dashboard.4.2 DevOps support functionsDevOps methodologies and their aligned support functions will be essential to deploy and operate the automationframework described above. These functions are the glue that hold the framework together to provide the logisticsand command-and-control functions that enable all disparate components and humans in the system to interact in aconsistent, secure manner. CI/CD: Supports the ability to rapidly test and deploy software and other network artifacts through the system. NF Onboarding: Supports the ability to rapidly add an NF or application to the service catalog. AAA: Access, authorization, and accounting framework that the entire automation architecture will use—includescertificate servers, RBAC, LDAP integration, audit mechanisms, and SOC integration. 2020 Cisco and/or its affiliates. All rights reserved.Page 10 of 29
White paperCisco publicCross-domain orchestrationTo achieve the goals of 5G automation, a cross-domain orchestration is needed to connect the parts together betweendifferent domains composing the network.The cross-domain orchestrator, residing in the CFS layer of the solution, can receive service instantiation requests fromthe OSS/BSS, or a self-service portal (such as CSMF), and it works on fulfilling the service request in addition to otherfunctions.The cross-domain orchestrator is expected to interface with different domain orchestrators (DOs) residing in thedomain-specific RFS layer in order to achieve the above task.5.1 Cross-domain orchestration mapping to the standardsThe following is the map where cross-domain orchestrator fits in the architecture provided by the different standardbodies: TMF: Cross-domain orchestrator fits in the E2E service orchestrator layer 3GPP: Cross-domain orchestrator covers the functional requirements of CSMF, NSMF, and NWDAF ETSI ZSM: E2E service management domain ONAP deployment: ONAP intends to cover the cross-domain orchestration layer in its next architecture release.5.2 Cross-domain orchestrator requirementsThe cross-domain orchestrator provides multiple functions that are crucial for E2E service delivery and E2E closed-loopautomation (zero-touch network).The cross-domain orchestrator provides the following functions: Integrates with the OSS/BSS layer, receives order management requests (TMF 640/641) and exports availableservices on the service catalog to the BSS digital marketplace (TMF 633)—in case TMF interfaces are not ready onOSS/BSS and the REST API integration is provided. The service fulfilment module translates the order request into fulfillment logic, including pre-checks, servicebreakdowns, policies, and post-checks. Integrates with the RFS layer and possibly different domain orchestrators to fulfill the request on the resource level Service inventory to list all the instantiated services Service topology view Service design studio is provided to build the service creation logic using GUI. Service analytics module receiving telemetry and performance information from various RFS layers to consolidate KPImanagement, perform advanced analytics, and apply logic for closed-loop automation Closed-loop automation for service remediationNote: The self-service portal in this solution is not a replacement for the self-service portal on the BSS layer (digitalmarketplace), which provides integration with the billing system, CRM, or any other required business support system. 2020 Cisco and/or its affiliates. All rights reserved.Page 11 of 29
White paperCisco public5.3 Northbound and southbound integrationAs illustrated in the diagram below, several integrations are needed to fulfill the cross-domain orchestration functionality.Figure 6. Cross-domain orchestrator integration points5.3.1 Northbound integrationAn interface is required to communicate with: Service activation and configuration Service catalog Service inventoryEither general REST APIs are used with the BSS/marketplace or TMF standard APIs can be used, namely: TMF640/641 Service Activation and Configuration API [9][10] TMF638 Service Inventory API [11] TMF633 Service Catalogue API [12]5.3.2 Southbound integrationIntegration with different domain controllers is usually achieved using NETCONF or REST APIs.If a direct integration with NFVO is needed, the SOL005 interface shall be used.It is worth mentioning that 3GPP is in the process of defining specific standard APIs for integration with different RANNSSMF and 5G core NSSMF in releases 16 and 17. 2020 Cisco and/or its affiliates. All rights reserved.Page 12 of 29
White paperCisco publicDomain-level orchestration6.1 RAN domain orchestrationITU R-M.2083 [27] describes the following use cases for RAN in 5G: Enhanced Mobile Broadband (eMBB): Addressing human-centric use cases for access to multimedia contentrequiring seamless coverage and medium to high mobility with much improved user data rates compared to existing(4G) data rates Ultra-Reliable Low-Latency Communications (URLLC): Addressing applications with stringent requirements ofthroughput, latency, and availability (such as industrial manufacturing and remote medical surgery) Massive Machine-Type Communications (mMTC): Addressing applications characterized by a very large number ofconnected devices transmitting a low volume of non–delay-sensitive dataThese use cases require a high degree of automation at a scale that can be challenging. In addition, disaggregation intoedge data centers and virtualization of the RAN are also critical themes that 5G brings to the fore. The NSSMF in thiscase will be required to support both virtualized and containerized workloads via NFVO and VNFM, while also managingsome physical network functions (PNFs) such as cell site routers, microwave radio links, cell site switches, and radiointerface units (RIUs).This architecture will be aligned as well with O-RAN specifications [20] for non–real-time control and other serviceprovisioning activities.Figure 7. RAN domain automation architectureOne example of a vRAN automation call flow is shown in the diagram below. It follows the key ETSI ZSM principles listedin section 3.1.4 of this document. 2020 Cisco and/or its affiliates. All rights reserved.Page 13 of 29
White paperCisco publicFigure 8. RAN domain automation call flowIn this real-life example, applicable to both 4G and 5G deployments in a virtualized environment, a new cell site is beingactivated, with the radio interface unit (RIU), vDU, and vCU configured automatically.In addition to the site initialization, specific slice scheduling configuration needs to be managed by the RAN NSSMF inorder to achieve the strict requirements of each and every slice.In order to create a new slice, NSSMF performs the following: Upon receiving a Network Slice Subnet Instance (NSSI) instantiation request, pre-checks are done to make sure thecurrent NSSI will not be able to support the new request. The available RAN resources are checked for availability of provisioning the new scheduling rules. The required scheduling information is configured for each of the RAN slices from existing templates (NSMT) The configuration requirements are sent to RAN NFMF for configuration management and further for serviceassurance.6.2 Transport domain orchestrationTransport infrastructure plays an important role for 5G services spanning multiple network domains. The end-toend service lifecycle should be seamless, and hence, an approach toward building a unified transport architecture isessential.At a classical top-level view, the mobility transport network is typically comprised of xHaul (fronthaul, midhaul, andbackhaul) and Core domains. Both of these network domains, consisting of routers, either virtualized or physicaldepending on function, will participate in network slicing. Leveraging the latest Transport SDN (T-SDN) programmabilityand automation capabilities on top of a unified Segment Routing (SR)–based transport, device-level configurationchanges related to new slice introduction or removal can be minimized, or in certain cases completely eliminated. 2020 Cisco and/or its affiliates. All rights reserved.Page 14 of 29
White paperCisco publicBefore discussing specifics around network slicing for transport, we first need to explore how 5G is impacting the xHaularchitecture. With the concept of RAN split, xHaul evolves to include fronthaul, midhaul, and backhaul networks. RANsplit enforces very strict requirements on the xHaul for bandwidth, latency, and high availability to support precise clocksynchronization between disaggregated radio functions, additional capacity due to radio densification, and mmWavedeployment. Depending on the UPF placement decision, the transport network for the backhaul (N3 interface) can beeither minimal (UPF placed closer to RAN edge) or extensive (UPF placed centrally or even further for certain nonlatencyenterprise application use cases) [2].Figure 9. Transport domain automationIn addition to the three service types (eMBB, URLLC, and mMTC), network slicing must also meet the followingrequirements within the scope of the network transport infrastructure: Transport Slice Management: Ability to create, modify, and delete a 3GPP network slice, including any actionsrequired on the transport layer. The slice application owner and the operator must be able to monitor the health andperformance of the slice through operations, administration, and maintenance (OAM) capabilities. Slice Isolation: Each transport slice must be isolated from all other transport slices in order to meet stringent SLAs.This means that the independent slice must meet proper QoS, performance, security, operation, and reliability levels. Resource Reservation: The ability to reserve transport resources for a particular transport slice in order to meetQoS requirements. Abstraction: Capability to utilize resources required to model and build a transport infrastructure to meet thedemands of a network slice.In terms of network slicing, there are two types: hard and soft. In each case, they must meet the requirements notedabove, with the difference being the level of resource sharing between the slices. A hard slice has dedicated resources,most notably for transport being bandwidth, that is not shared with other slices. Soft slicing, on the other hand, is a moreagile and flexible approach in which resources can be shared between slices while still maintaining SLA requirementswith those resources being able to return to the network when they are no longer needed.One of the important aspects of transport layer is that it is carrying the traffic connecting all other domains for allcustomers, which requires: 2020 Cisco and/or its affiliates. All rights reserved.Page 15 of 29
White paperCisco public1. Continuous optimization of resourcesTraffic optimization and management require different levels of network automation depending on the applicableuse case.Offline planning: Offers a complete network model taking into consideration the current network status and loads,whi
White paper Cisco public ico anor it aiiate A riht reere Page 1 of 29 Authors: Milan Stolic Julie Ann Connary Wei Yan Arghya Mukherjee Mohammad Salaheldin Rob Piasecki 5G Automation Architecture White paper. White paper Cisco
