LIAN Meng,GU Rentao,JI Yuefeng,WANG Dajiang,LI Hongbiao

(1.Beijing Laboratory of Advanced Information Network,Beijing University of Posts and Telecommunications,Beijing 100876,China;2.Wireline Product Planning Department,ZTE Corporation,Shenzhen 518057,China)

Abstract:The technological development of smart devices and Internet of Things (IoT)has brought ever-larger bandwidth and fluctuating traffic to existing networks.The analy?sis of network capital expenditure (CAPEX)is extremely important and plays a fundamen?tal role in further network optimizing.In this paper,an adaptability analysis is raised for IP switching and optical transport network (OTN) switching in CAPEX when the service bandwidth is fluctuating violently.This paper establishes a multi-layer network architec?ture through Clos network model and discusses impacts of maximum allowable blocking rate and service bandwidth standard deviation on CAPEX of IP network and OTN network to find CAPEX demarcation point in different situations.As simulation results show,when the bandwidth deviation mean rate is 0.3 and the maximum allowable blocking rate is 0.01,the hardware cost of OTN switching will exceed IP switching as the average band?width is greater than 6 100 Mbit/s.When the service bandwidth fluctuation is severe,the hardware cost of OTN switching will increase and exceed IP switching as the single port rate is allowed in optical switching.The increasing of maximum allowable blocking rate can decrease hardware cost of OTN switching.Finally,it is found that Flex Ethernet(FlexE)can be used to decrease CAPEX of OTN switching greatly at this time.

Keywords:bandwidth fluctuating;CAPEX;OTN switching;IP switching;FlexE

1 Introduction

Network data are growing dramatically due to the devel?opment of smart devices and multimedia application technologies.By 2021,global mobile traffic will reach 6.7 times that in 2016,exceeding 48.3 Exa?bytes per month[1],as a result,existing networks will not be able to afford such huge business growth.To address this is?sue,5G cellular networks are developed[2].

With the advent of the 5G era,the development and appli?cation of various technologies are becoming possible.The 4G long term evolution (LTE) cellular system is difficult to sup?port the transmission of such a large number of services brought by those applications,such as Internet of Things (IoT)and e-health,and the next generation datacenter needs to solve the problem of switching and transmission of massive services[3–4].In order to correspond to the exponential rise in user and traffic capacity,datacenter capital expenditure (CA?PEX) will increase significantly.The current datacenter inter?network is required to evolve into a high capacity,low-cost and low-latency network platform.

Packet services based on IP switching are emerging,which raises new requirements for transport networks.Optical net?works can provide high-capacity end-to-end communication pipes for upper-layer services like IP services.Therefore,ser?vice transmission needs multiple electro-optic (E/O) and opti?cal-electronic (O/E) conversions.Among the total services re?quired processed by IP nodes,55%–85% of them only need to transit through these nodes.

Researchers have made significant research and proposed several methods to reduce CAPEX.In general,the problem of network cost is considered as integer linear programming(ILP)optimization problems[5–7].Heuristic approaches are usu?ally applied to solve cost problems,including increasing re?structure capacity[8],adopting optical bypass to avoid traffic in the optical domain from contacting IP routers[9–10],using soft?ware define network (SDN)[11]for business grouping and multilayer joint optimization to reduce costs[12–13].But increasing the number and capacity of infrastructure will lead to the ris?ing cost of network construction as well[8,13].The method of adding optical bypass is not significant,and usually only saves 10%–15% of the CAPEX cost[13].In the joint optimiza?tion process,multiple links are affected in the optimization process of a certain link,so multiple optimizations should be considered[13–15].

Edge micro datacenters (mDC) are often used to handle small bandwidth and low latency service requests[16–17].On the other hand,service requests that interact between multiple datacenters (DC) typically have high bandwidth characteris?tics.The optical layer switching is considered to have large ca?pacity,high bandwidth,and low latency.Therefore,optical transmission technologies[18–19],like optical transport networks(OTN)[20–21],are used extensively in datacenter networks to save cost and reduce latency while enabling high-speed data transmission.

However,OTN switching is very inflexible due to the large switching granularity.This inflexibility is more prominent when the bandwidth of the service fluctuates greatly.In order to ensure the traffic transmission blocking rate,the OTN switching must meet the largest bandwidth service require?ment in the network,which will result in a large amount of cost waste,and therefore OTN switching is not necessarily the optimal choice.As a result,the cost analysis of large volatility bandwidth services needs further research.

Flex Ethernet (FlexE) is a good solution to this problem.FlexE supports the binding of multiple links and the link ag?gregation.For example,a 50 Gbit/s service can be transmitted by one 40 Gbit/s Ethernet port and one 10 Gbit/s Ethernet port instead of one 100 Gbit/s Ethernet port.Link binding us?ing FlexE can greatly reduce resource waste of OTN switching.

In this paper,by calculating CAPEX,we perform adaptive analysis on IP switching and optical switching.We hope to ob?tain the bandwidth adaptation range of IP switching and opti?cal switching when the average bandwidth and the standard deviation of the service are both large.Next,by abstracting the switching behavior of the IP router and the OTN device,a network connection switching model is established to perform CAPEX.Then,we introduce the adaptive assessment process to determine the bandwidth cross-point.After that,modifying the average bandwidth and the service standard of the service,we analyze hardware costs to accommodate the range of two switching methods.We analyze the impact of network service quality and service bandwidth fluctuations on network hard?ware costs by changing the maximum allowable blocking rate and the bandwidth standard deviation.Finally,we use FlexE technology in the model to explore the impact of port bonding on optical switching CAPEX.

The remaining parts of this paper are organized as follows.Section 2 introduces the structure and problems of the data?center network.In Section 3,a mathematical model is de?signed for calculating the cost of the switching network.Sec?tion 4 performs a performance evaluation based on simulation results.Finally,in Section 5,we summarize this paper.

2 Distribution of Datacenter Network Archi?tecture and Cost

With the continuous development of cloud computing and 5G technologies,the granularity of service traffic increases.Under this trend,optical layer switching may have advantages in cost and energy consumption compared with IP switching.

In the existing network,most of the transport layers select OTN and wavelength division multiplexing (WDM)[22]system as the infrastructure.In this paper,we assume that the OTN is used as the network infrastructure[20]technology.

2.1 Datacenter Network Architecture

The datacenter network architecture is mainly implemented through the coordination of the three major planes of applica?tion management,network control and service forwarding.As shown in Fig.1,used for forwarding plane,DCs are intercon?nected with hosting services through a multi-domain network request based on the strategy provided by the controller plane.The SDN control technology is adopted in the controller plane,while centralized management and cross-layer joint network planning and optimization techniques are adopted in the appli?cation plane.In the forwarding plane,differentiated delivery of service with different quality of service (QoS) requirements is mainly achieved through the coordination of IP network and OTN network controllers,ensuring the efficiency of the entire network resources and management.In the controller plane,SDN technology is mainly used to realize the Openflow proto?col.The smooth transition of the network will accommodate the flexible scheduling of multi-layer multi-domain distribut?ed networks in the future.The main feature of the manage?ment plane is the unified and centralized management mode,which greatly simplifies network management processes,re?duces network maintenance costs and improves network man?agement and maintenance efficiency,thereby saving the total cost of the network.

The IP services in the datacenter network are diversified,and optical channels use wavelength switching.With large ca?pacity,there is a fundamental mismatch between service traf?fic and optical flow capacity.Usually,the solution is using the traffic aggregation function of the core router in the IP layer to provide services to convergence processing,and then through the optical layer to achieve large-capacity long-distance trans?mission.This solution offers a high-speed information trans?mission channel for the IP layer.

However,service transmission requires times of E/O and O/E conversions,which greatly increases network CAPEX.Ap?propriately reducing IP routers processed can significantly re?duce the cost of the network between datacenters.The optical layer switching has larger capacity,higher bandwidth,and lower latency compared with the IP layer switching.However,if the granularity of service requests is small,the large granu?larity of optical switching service will reduce resource utiliza?tion and further increase network CAPEX.Therefore,this ar?chitecture proposes the following strategy:When the service bandwidth is in the interval where the IP switching is more ad?vantageous,the service transmission route selects the tradi?tional transmission solution and transmits through both the IP equipment and the OTN equipment.Conversely,when the ser?vice bandwidth is in the interval where the optical switching is more advantageous,the service transmission route selection is directly transmitted through the OTN equipment.The equip?ment provides optical channel data unit (ODU)k-class hard pipes directly,and then transmits high-speed services through ROADM devices in the optical layer,thereby reducing equip?ment investment,saving energy consumption and promoting sustainable development of the transmission network.

This paper considers the direct transmission of services through OTN equipment,avoiding multiple E/O and O/E con?versions.OTN equipment is divided into edge-based aggre?gate OTN equipment and core switching OTN equipment.The aggregate OTN equipment uses the cross-point switching ma?trix to perform optical mixing processing of the service,and the service performs E/O and O/E conversion at the edge.At the core layer,the OTN equipment performs all-optical pro?cessing of the service through the ROADM to ensure reliable and high-quality service (yellow path in Fig.1).Correspond?ingly,the IP network is also divided into two layers:the edge and the core.The edge aggregation router supports aggrega?tion services,and the core layer uses the core router to pro?vide a higher level of switching services (red path in Fig.1).We want to get the range of adaptability of the two switching methods and evaluate the capital expenditures.

2.2 Composition of Network Equipment Cost

In order to facilitate comparison and calculation,this paper calculates the equipment cost after normalization.The cost pa?rameters of the equipment are shown in Tables 1 and 2,and the cost is expressed as a standardized value.OTN equipment is divided into common units and service units.Accordingly,IP equipment includes base systems,processor cards,line cards,and optical modules.The common unit and the basic system are independent of the business service,and they are used to build the basic platform.The service unit is related to the specific service.Line cards and processor cards need to be used together.A mother card can carry two daughter cards.For example,if we need an IP router with two 100 G ports,we choose a base system as the platform,an A-type network proces?sor card,two 100 G line cards and two 100 G optical modules.Therefore,the total cost is calculated as:10+21.14+2 ×58.57+2 × 53.97=256.22.

3 Problem Formalization

In this section,a mathematical model is proposed to obtain the network CAPEX for adaptive analysis.

3.1 Mathematical Model

Due to the large switching granularity of optical switching,the key is how to ensure connectivity in the network.In this paper,a strict non-blocking Clos network structure is used to ensure full network connectivity.

The Clos network was proposed by Charles CLOS in 1952[23].The Clos network refers to a switching network that tries to reduce the number of intermediate cross points as much as possible in order to reduce the cost of a multi-level switching network.The ba?sic idea is to use multiple smaller-scaleswitching units according to a certain connection method.The ad?vantage of Clos network is that its crossbar architec?ture and the Clos network can provide a non-blocking network.

Under the condition of non-blocking,the three-lev?el Clos network is split into switching matrixes and con?stitutes a multi-level Clos network,each switching ma?trix representing a piece of equipment.As shown in Figs.2a and 2b,on the premise of ensuring strict non-blocking,datacenter network can be mapped to Clos network,and a threelayer symmetric switching network is established according to the Clos model.Then,as shown in Fig.2c,the switching network is divided into multi?ple layers based on the service parameters and network param?eters.Then the minimum number of IP switching or OTN switching in the network is determined,and the number of links required by the network and by the number of devices and the network topology is calculated.The OTN equipment or IP equipment is mapped to the switching matrix (SM).The connections between them correspond to the optical links in the real network.

▼Table 2.Cost parameters for normalized IP router

▲Figure 2.Multi-level Clos network:(a) datacenter network;(b) datacenter network mapped to Clos network model;(c)middle layer of Clos network for splitting

We assume that the service bandwidth obeys the normal dis?tributionNwhere the mean value isμsd,the standard deviation isand the probability density function isf(x).

The blocking rateα=dx is allowed,and the valuethe maximum bandwidth of the service request,can be ob?tained according to the blocking rateα.In order to make the model realistic,we assume that the bandwidth mean and the standard deviation of the service are proportional,which means that larger service has a larger standard deviation.

For the IP switching,with the dynamic bandwidth alloca?tion function,bandwidth should be reserved for dynamic ser?vice.For OTN switching,it is necessary to reserve bandwidth resources for the service according to the maximum bandwidth to prevent random service.

For the OTN switching,the service request needs to be mapped to electrical cross-connection and ODUkbefore being multiplexed.Assuming that in ODUk,k=0,1,2,3,4,the mul?tiplexing relationship between the ODUks can be expressed as:2U0=U1,4U1=U2,4U2=U3,2U3=U4.

In Table 3,We define the required symbols,costs and vari?ables.In this paper,a piece of core equipment can carry eight mother cards.

After ensuring the connectivity through the Clos network struc?ture,the objective function can obtain the total hardware cost ofthe network convergence layer and the core layer.And then the minimum cost for different speed card configurations is found.

▼Table 3.Notations,costs and variables

The adaptation constraints of IP layer and OTN layer are evaluated as:

When the FlexE technology is applied,OTN switching al?lows multiple port bindings to be used.At this time,theUkis equal toUk-1+U1.

Strict non-blocking constraints are shown below.Under the premise of strict non-blocking,the network can establish a ser?vice request at any time,if both the source server and the des?tination server are free.

According to the requirements of the equipment,the first three types of line cards need to be matched with the type A network processor card,and the fourth type of the line card needs to be equipped with the type B network processor card.The cost of a single IP equipmentCipand that of a single OTN equipmentCotnare calculated as follows:

After the switching network is split,the number of equip?ment in the convergence layer can be calculated:

The middle tiernitershould be split untilr(i) ≤m.For IP switching or OTN switching,the number of iterations after splitting the middle tierniteris expressed as follows:

wheremandnare Clos network parameters.

3.2 Adaptive Assessment Procedure

For IP switching,since the service bandwidth is normally distributed and according to the symmetry of the normal distri?bution,the large service can always be combined with a small service for transmission.So the maximum capacity of the IP equipment in the network is the average bandwidth.The hard?ware cost of the IP switching is only related to the average bandwidth of the service in the network and is independent of the standard deviation of the service bandwidth in the net?work.The OTN switching focuses on the maximum service bandwidth in the network.In order to transmit the largest bandwidth services,the network must use OTN equipment with higher capacity,which will result in a waste of transmis?sion resource.The hardware cost of OTN switching is related to not only the average bandwidth of network services,but also the standard deviation of service bandwidth.When the servic?es with large bandwidth and large bandwidth standard devia?tion are transmitted,the hardware cost of OTN switching may be greater than the IP switching.

In order to describe the results of cost-adaptive evaluation more accurately,we introduce the concept of bandwidth crosspoint,which is the cost demarcation point for service band?width between IP switching and OTN switching.We divide the cross-points into first bandwidth cross-point and second bandwidth cross-point.When the IP switching cost exceeds the OTN switching,this demarcation point is the first band?width cross-point.When the OTN switching cost exceeds the IP switching,this demarcation point is the second bandwidth cross-point.Since the first bandwidth cross-point is usually small and we are more concerned with the large service band?width,the second bandwidth cross-points are mainly dis?cussed and analyzed.

In the adaptive evaluation process,the minimum cost value of the total hardware of the network is calculated under the premise of ensuring the network blocking rate.For IP switch?ing,we allocate port resources based on the average band?width value for the service request.After that,we set the ser?vice standard deviation to be proportional to the business mean value.When the blocking rate is guaranteed toα,the service granularity is changed,and the total hardware cost can be calculated.

4 Simulation Results and Performance Analysis

In this section,IP equipment and OTN equipment support mixed-port rate transmission(2.5/10/40/100 Gbit/s)in the sim?ulation,and get the lowest cost by calculating the lowest cost connection topology in multiple hybrid modes.In this section,we first explore the relationship between the number of busi?nesses and the cost of network hardware.Then we study the impact of the maximum allowable blocking rate and the ratio of the standard deviation to the service mean of bandwidth(de?viation-mean rate) to the hardware cost of IP switching and OTN switching.Finally,we apply the FlexE technology in the network to explore whether the port binding can reduce the OTN switching hardware cost.

As the average bandwidth is 6 300 Mbit/s,the deviationmean rate is 0.3,and the maximum allowable blocking rate of the network is 0.01.The hardware cost of IP switching and OTN switching is shown in Fig.3a.The deviation-mean rate is 0.5,and the maximum allowable blocking rate of the net?work is 0.1.The hardware cost of IP switching and OTN switching is shown in Fig.3b.When the number of services is greater than 1 250,the OTN switching hardware cost is higher than the IP switching.Therefore,when the services with large bandwidth and large bandwidth standard deviation are trans?mitted,the hardware cost of OTN switching will exceed the IP switching.

▲Figure 3.Hardware cost of IP switching and OTN switching:(a)with 6 300 Mbit/s mean bandwidth,0.3 deviation-mean rate and 0.01 block?ing rate;(b) with 6 300 Mbit/s mean bandwidth,0.5 deviation-mean rate and 0.1 blocking rate

After that,we change the average bandwidth of the ser?vice,the deviation-mean rate,and the maximum blocking rate of the network when when the number of services circu?lated in the network is 1 800.As shown in Fig.4a,when the deviation-mean rate is 0.3 and the maximum allowable blocking rate of the network is 0.01,as the average band?width of the service increases,if the service bandwidth is less than 6 100 Mbit/s,the hardware cost for OTN switching will be less than IP switching.However,as the average band?width is greater than 6 100 Mbit/s,the hardware cost of OTN switching will exceed the IP switching.

As shown in Fig.4b,the maximum allowable blocking rate of the network is 0.01 and the deviation-mean rate is increased to 0.5.As the average bandwidth of the service increases,and the average bandwidth is greater than 4 800 Mbit/s,the hard?ware cost of OTN switching will exceed IP switching.The sec?ond bandwidth cross-point is smaller than the second band?width cross-point when the deviation-mean rate is 0.3.

As shown in Fig.4c,the deviation-mean rate is 0.3 and the maximum allowable blocking rate of the network increases to 0.1.As the average bandwidth of the service increases,and the service bandwidth is greater than 7 400 Mbit/s,the hard?ware cost of OTN switching will exceed IP switching.The sec?ond bandwidth cross-point is larger than the second band?width cross-point at the maximum blocking rate of 0.01.

An important breakthrough in FlexE technology is physical channel bonding.For example,it supports 200 Gbit/s media access control (MAC) based on two bonded 100 Gbit/s physi?cal layer channels.In this model,FlexE technology can be im?plemented by the combination binding of different ODUks and a bonded ODUkcan be used as a new port with a different granularity.Using the FlexE technology to bind OTN equip?ment ports can reduce the hardware cost of OTN switching ef?fectively.Because the multiplexing relationship of the ODUks is 2U0=U1,4U1=U2,4U2=U3,2U3=U4,where the three ports bindings can meet the vast majority of granularity.As shown in Fig.4d,when we allow three-port bindings,the hard?ware cost of OTN switching is significantly reduced and the second bandwidth cross-point is no longer present.

Since the maximum blocking rate of the network should be guaranteed,in the model we assume that the services with larger bandwidth are blocked,which is also practical.There?fore,the smaller the maximum allowable blocking rate of the network is,the larger the maximum bandwidth of the service which can actually pass through the network is.Similarly,in?creasing the standard deviation of service bandwidth can also increase the maximum bandwidth of services in the network.At this time,the OTN switching needs to transmit the largest bandwidth service in the network,so the hardware cost of OTN switching will increase significantly，which leads to re?sources waste in the increased hardware cost.

▲Figure 4.Total hardware cost when service number is 1 800 with:(a)0.3 deviation-mean-rate and 0.01 blocking rate;(b)0.5 deviation-mean-rate and 0.01 blocking rate;(c)0.3 deviation-mean-rate and 0.1 blocking rate;(d)cost of IP switching and OTN switching after using FlexE

FlexE technology can be used to bind OTN equipment ports.When OTN equipment needs to transmit large band?width services,other untagged port resources can be also used.In this way,the resource waste can be reduced and the OTN switching hardware cost can be decreased.

5 Conclusions

This paper analyzes the hardware cost of datacenter net?works.In most solutions,optical switching is considered for large-bandwidth service transmission.This paper not only con?siders the service bandwidth,but also the fluctuation of the service bandwidth.The hardware cost of IP switching and OTN switching when transmitting large bandwidth and large fluctuation services are discussed.The simulation results show that although OTN switching is more suitable for trans?mitting large bandwidth services in principle,the hardware cost of OTN switching will exceed IP switching when large bandwidth services have large bandwidth fluctuations.After that,this paper analyzes the impact of maximum allowable blocking rate of the network and the standard deviation of ser?vice bandwidth on OTN switching hardware cost.

When the deviation-mean rate is 0.3 and the maximum al?lowable blocking rate of the network is 0.01,as the average bandwidth is greater than 6 100 Mbit/s,the hardware cost of OTN switching will exceed the IP switching.As the deviationmean rate increases,the OTN switching hardware cost increas?es;as the maximum allowable blocking rate of the network in?creases,the OTN switching hardware cost decreases.There?fore,the fluctuation of service bandwidth will increase the hardware cost of OTN switching and eventually exceed IP switching.The increase of maximum allowable blocking rate of the network can decrease the OTN switching cost.Finally,the simulation results show using FlexE can effectively reduce the OTN hardware cost.