Abstract：
- 问题：AI在路由协议上的应用仅适用于真实设备，尤其是无线传感器节点
The inclusion of artificial intelligence (AI) can improve the performance of routing protocols. Nowadays the application of AI over routing protocols is only applied to real devices, especially in wireless sensor nodes.
- 解决：提出了基于强化学习智能路由协议，它能根据最佳标准、网络状态选择最佳数据传输路径
In this paper, we present a new proposal to implement an intelligent routing protocol in a SDN topology. The intelligent routing protocol is based on the reinforcement learning process that allows choosing the best data transmission paths according to the best criteria and based on the network status.
强化学习：
根据维基百科的描述，强化学习定义如下:
强化学习是机器学习中的一个领域，强调如何基于环境而行动，以取得最大化的预期利益。其灵感来源于心理学中的行为主义理论，即有机体如何在环境给予的奖励或惩罚的刺激下，逐步形成对刺激的预期，产生能获得最大利益的习惯性行为。
在强化学习的世界里， 算法称之为Agent， 它与环境发生交互，Agent从环境中获取状态（state），并决定自己要做出的动作（action）.环境会根据自身的逻辑给Agent予以奖励（reward）。奖励有正向和反向之分。比如在游戏中，每击中一个敌人就是正向的奖励，掉血或者游戏结束就是反向的奖励。
IDEA
- Quagga : 开源软件，可以修改路由协议
it is open-source so it can be modified to add the AI-based improvements to the routing protocol. However, Quagga has the disadvantage that only allows having one router per PC.
- 图2所示的算法提供了一种强化学习方法，用于建立源和目的地之间的路径。
- 给定一组可能的路径I和一组称为M的网络测量值（延迟，丢失率和带宽），
- 为参数分配不同的权重w1，w2和w3，从而计算每个可用路径的成本ci。
- 数据传输在较低成本的路径发送。
The algorithm showed in Fig. 2 provides a reinforcement learning method to establish a path between a source and a destination. Given a set of possible paths I and a set of network measurements (delay, loss rate and bandwidth) called M, the agent calculate for every available path i a cost ci by assigning different weights w1, w2 and w3 to the parameters. The data transmission is sent by the path with less cost.
- 一段时间，路径节点提供奖励d，其使用关于通过路径的传输的网络参数和传递函数来计算。
- 利用来自路径的反馈，代理再次调整权重，从更好的路径获得更大的回报**。
After spending some time, the following node in the path gives to the agent a reward d which is calculated using network parameters about the transmission through the path and a transfer function. This reward is the parameter that the agent (the router) wants to increase as much as possible. With that feedback from the paths, the agent adjusts again the weights trying to obtain a greater reward from a better path. This is the learning process in which the agent learns to choose the most important parameters to take account in the routing process.
To execute this algorithm, the different elements present in the network need to exchange a set of messages. Fig. 3 shows the message exchange.
做法
- 提出通过SDN实现的新的分布式路由提议，设计了一种基于强化学习的智能算法。
we have presented a new distributed routing proposal implemented over SDN. We have analyzed the way of building the SDN topology that runs routing protocol in a distributed way. Moreover, we have designed an intelligent algorithm based on reinforcement learning to improve some aspects of routing.
- 使用Quagga，将算法添加到OSPF路由协议**中。
This algorithm is added to the OSPF routing protocol using Quagga which allows modifying the routing algorithms.
效果：
- 更稳定，损失率更低，延迟更低的路线。 抖动优化。
- The results showthat the routing proposal works properly and it reaches better QoS features than the traditional one. Our proposal clearly achieves a more stable route with less loss rate that implies to have lower delay. In addition, the jitter obtained with the proposal is significantly better than the values offered by the traditional routing.