Abstract
The Boomerang family of wireless monitors all utilize a cellular modem to provide a remote data connection. This makes remote monitoring very convenient but it does require the installer to understand and adjust to the differences between a traditional wired (or Wi-Fi) network and the cellular data network. This paper presents a recently encountered real world issue and the final solution to the problem.
Real World Example
Recently, a customer was attempting to install multiple Form 2S meter socket Boomerangs (Figure 1) for remote voltage monitoring. These units were to be scanned by a SCADA system via DNP3 to obtain voltage readings multiple times per hour. For more on SCADA and DNP3 download the white paper An Overview of DNP3 and the Boomerang Point Map. In short, DNP3 is a protocol that can utilize various types of physical communication links based on the specific devices in use. In this case, the devices are Boomerangs and the communication link available is IP via the cellular data network. A typical SCADA configuration with multiple outstations is shown in Figure 2.
IP (Internet Protocol) is coupled with one of either TCP (Transmission Control Protocol) or UDP (User Datagram Protocol). For more on UDP vs. TCP, download the white paper DNP3 – UDP vs. TCP Configuration. For this example it is only important to note that TCP/IP and UDP/IP have tradeoffs that are more pronounced when used with the cellular data network.
TCP/IP is perhaps the “go to” choice for most network communications due to its inherent reliability and guaranteed delivery of data and was the initial choice for this customer. The SCADA system was programmed with the appropriate device addresses and TCP/IP was the chosen protocol. The initial results were mixed. Many of the Boomerangs were seen as being offline due to the SCADA system either not connecting to the devices or not receiving responses to its data requests. A review of the Boomerangs themselves showed they were in a normal operating state waiting for the SCADA system to make requests. What then, was the issue?
TCP/IP Wired (Including Wi-Fi) vs. Cellular Wireless
TCP/IP is known in the computer networking world as a connection oriented protocol. This means that two devices (the SCADA system and a Boomerang in our example) connected via TCP/IP maintain an open connection link to the other. The SCADA system requests a connection to the Boomerang and then uses this connection for all of its data requests. This logical connection allows the TCP protocol to not only guarantee the delivery of the data but also to guarantee its integrity. This comes at the cost of additional overhead in the protocol and potential retries of the transmission. On a wired Ethernet connection all of this happens at computer speeds typically measured in tens of milliseconds with occasional delays that might stretch to 1 or 2 seconds depending on network load or device readiness. When using a cellular wireless link for the same TCP/IP connection the times involved vary from under one second to more than 30 seconds. This delay is due to a combination of factors including signal strength, signal noise, cell tower utilization, time since last transmission and the individual service provider. These delays are longer when the network link is mostly idle (e.g. when polling only every few minutes or longer). Faster polling will tend to prevent the cell portion of the link from going idle, and decrease the latency. In this case the desired poll cycle was too slow to keep the cell latency low.
This huge disparity of time values must be taken into account by both the SCADA system and the Boomerang. Typical default timeout values for TCP connections and data responses will usually be geared toward the more common wired Ethernet speeds which are much too short for working with the cellular data network. This then was part of the problem with our real world example. The SCADA system was not programmed to wait long enough for connections to the Boomerang nor for responses to data requests. Once these timeout values were extended (via SCADA system configuration) to 40 seconds the results improved but did not achieve 100% success rates. Further investigation was required.
Protocol Timing
The timing changes for connections and responses improved the SCADA system’s perception of connection availability but there were still a percentage of requests that did not receive a response. The SCADA system did allow for a certain percentage of dropped requests to be ignored and as the percentage of no response requests was increased the success rate improved. However, this was merely masking an underlying issue by ignoring failures. Further investigation finally revealed that a lower level retry mechanism (possibly within the TCP protocol itself) was still only waiting 10 seconds. The SCADA system did not seem to have a programmable value that accounted for this period. While there was likely some way to adjust the SCADA system to account for this short timeout period it was finally decided to step back and reevaluate the entire setup.
DNP3 and TCP
A feature of the TCP protocol is guaranteed delivery of data packets implemented as acknowledgements from each side and a series of retries if the first attempt fails. However, the DNP3 protocol itself also includes acknowledgements and a retry strategy. Individually, each is useful, but if both are present the result can be a confused mix of multiple retries and delays at each layer – e.g. the DNP3 session layer sending a second poll request, while the underlying TCP transport layer is still resending the first one. These overlapping retries add to cell traffic congestion and decrease the likelihood of recovering cleanly from the original error.
Adding to the problem, the DNP3 protocol specification requires that slave devices reject new connection attempts from a master until the previous connection is explicitly terminated. If the DNP3 master sends a poll request to the slave device on an existing TCP connection and doesn’t get a response then it may decide to drop that connection and initiate a new one (possibly before the device has a chance to respond due to timeout values that are too short for cell connections). On the outstation side, the Boomerang will see a new connection attempt from the master, but is forced to follow the DNP3 specification and reject it. The Boomerang then tests the original TCP connection with a DNP3 Link Status Request and discovers that it’s down since the master has already closed it. The Boomerang must then close its side of the connection and begin listening for a new connection. At best, this process itself involves multiple retries on both sides before the master and slave find each other with a new TCP connection. In the meantime, some SCADA systems will count each DNP3 retry as a failure and mark the device as offline. In situations like this, the DNP3 over TCP specification itself is not well suited for a cellular network. The logs from these Boomerangs showed many instances where a single delayed or dropped packet caused several retries and new connection attempts each taking more time and cell data usage.
Requirements
So, what were the actual requirements? The use case in our example was for monitoring voltage levels multiple times per hour. The actual number of times each hour was not a critical item. Once every two minutes (30 samples per hour) was good for this application. Further, the system could easily tolerate up to 15 percent loss of samples within any given hour. It was therefore determined that at least 25 of 30 voltage samples per hour indicated success and the Boomerang would remain in an online condition from the perspective of the SCADA system.
Enter UDP/IP
UDP/IP is a transaction oriented protocol without the concept of a persistent connection between two devices. A message is simply created and sent to an IP address. UDP/IP is simpler with less overhead (good for cellular data plans) but unlike TCP/IP it forgoes the guaranteed delivery mechanism at the transport layer. It is said to be a “send and forget” protocol because once a message is sent the delivery is assumed and the protocol doesn’t know if the message arrives at the destination. For our example of the SCADA system and Boomerangs the send and forget concept completely eliminates the need for a connection timeout and the lack of underlying guaranteed delivery eliminates the TCP/IP transmission retries. The site requirements allow for a small percentage of samples to be ignored which is well within any potential loss of data due to UDP/IP. DNP3 does its own retry of data requests when it doesn’t receive expected responses so even the tiny potential loss due to UDP/IP is mitigated further by the DNP3 protocol. With just the DNP3 acknowledgements and retries at the session layer, duplicate overlapping retries with the network layer are avoided. In addition, since there is no concept of a UDP “connection”, the complication of the outstation being forced to reject new master connections per the DNP3 specification is also eliminated.
The same cellular data network delays exist with UDP/IP. These now go practically unnoticed by both the SCADA system and the Boomerang due to the simplicity of UDP/IP. A UDP/IP message arrives at the Boomerang but there is no need for a network level acknowledgement. The SCADA system has sent the UDP/IP request but is not waiting for the network to report that the data has been received. The DNP3 protocol is keeping track of requests and responses. It will reissue a request if an expected response is not received. This may actually result in two back to back requests from the Boomerang’s perspective if the original request was delayed versus lost. The SCADA system will either utilize the additional response or ignore it and all is well.
The Final Result
In our real world example, the SCADA system and Boomerangs were reconfigured to use UDP/IP as the underlying protocol. The SCADA system was set to wait 30 seconds for a response to a DNP3 request for voltage data. The frequency of the requests was set to once every 2 minutes and the requirement for the Boomerangs to be seen as online was set to 25 out of 30 per hour. These changes resulted in 100% online time for all devices from the perspective of the SCADA system. The observation after monitoring the system for a couple of days was that no requests or responses were lost. All devices were showing 30 of 30 expected data points per hour. UDP/IP was the superior choice for this site considering their requirements especially when coupled with their particular brand of SCADA system.
Conclusion
Setting up a network application requires consideration of site requirements, protocols, bandwidth, reliability, fault tolerance and device availability among other things. Most of these items are well known or well understood when working with traditional wired or Wi-Fi networks. The cellular data network has different characteristics to be taken into account but it allows for unparalleled access for remote monitoring of devices. In particular, UDP is recommended over TCP when using DNP3 over a cellular connection if supported by the SCADA software. The Boomerang family of products have been designed to account for differences in data networks and they are fully capable of navigating the nuances of the cellular system.
