Why Your End of Line Testing of IoT Devices Might Be Failing Without You Knowing

Paul Marshall

Founder & CCO

LinkedIn

The final stage in the manufacturing process is the end of line testing. For IoT devices, there’s the question of how best to test cellular connectivity. It might seem easy. The devices contain SIMs, so surely the simplest (and most realistic) option is to check that they can connect to a live mobile network?

Like many things in IoT, it’s not that easy.

In this article, we discuss the problems that can arise from using a live network for testing IoT device connectivity at the end of the manufacturing line and the potential for unpredictable results this can give.

Why testing is important in IoT

For many IoT deployments, the cost of a failed installation is high. Devices may be taken to a remote location, placed at the top of a telegraph pole, deep underground or on an unmanned oil rig. The last thing you want is for the newly installed device to fail to start or communicate. Even if devices are accessible, the costs of replacing faulty devices could soon threaten ROI.

End of line testing in a manufacturing facility can mitigate this risk by checking that device connectivity works correctly before devices are shipped.

What should end of line testing cover for IoT devices?

As well as instructions for building a device, manufacturers need to know how to test it. For IoT devices, end of line tests might include the following checks:

• Circuit board is assembled correctly with no dry joints, missing items, or broken links
• Correct voltages are applied
• All interfaces – button, screens, lights and so on – function correctly
• Firmware can control the modem
• SIM is fitted correctly and can communicate with the modem
• Device can transmit and receive signals on different frequencies, channels, and bands, and with varying strengths
• Information, such as the SIM ID, can be extracted from the SIM for recording and printing on device housing and packing labels

There are many established ways of testing electronic devices, including visual inspections of circuit boards, bed of nails testers, manual test procedures to perform actions and check responses, and intelligent test systems with automated test applications and sensors to check outputs.

End of line testing of cellular connectivity in IoT devices presents new challenges.

A simple connectivity test for IoT devices

The most obvious way to test connectivity is to connect the device to a network and check that it can transfer data. This demonstrates that the firmware, modem, and SIM are installed correctly and can communicate with each other. It verifies that the device can register onto a network and transfer data to a cloud application over the internet.

Using ‘normal operation’ for testing has the advantage of simplicity – minimal investment in test development and no requirement for specialist test equipment. And it’s proof that the device will connect when it’s installed in the field – isn’t it?

Unfortunately, testing that your device can connect to a network that’s local to your production line doesn’t prove that the device will work anywhere else – unless, possibly, all your devices are guaranteed to have access to the same network in the same conditions as the production plant.

It’s also possible that your planned connectivity testing won’t work at all in the manufacturing facility.

So, what are the problems?

Network (un)availability

Two recent LTE-based standards, CAT-M1 and NB-IoT, are designed to support Low Power Wide Area (LPWA) IoT devices. These are attractive for IoT edge devices as they provide energy-efficient connectivity for deployments with low bandwidth and timing requirements. Network operators are gradually rolling out these standards, but coverage is patchy, and choice is restricted. For example, in China, there are no CAT-M1 networks.

In addition, the availability of some established networks is decreasing. Network operators are in the process of sunsetting 2G and 3G networks to free up more space for LTE. There is an increasing number of countries and regions where it’s now difficult or impossible to test 2G radios.

If your devices use radios with more than one Radio Access Technology (RAT) option – for example, radios that support 2G and CAT-M1 – you may be able to connect using one of these from your production plant. But this means you can only partially test your devices.

If your devices only use one RAT, you may find your devices can’t connect at all for testing. China may be a prime location for high volume manufacturing – but not if your devices can only use CAT-M1!

Unpredictable connection times

Connecting to a mobile network is not always straightforward. A device needs to find a network and attempt to connect. If the connection fails, it needs to find another network to try. If a local mast is down for maintenance or a connection fails because the device isn’t authorised to use that network, the time to connect will be much longer than if all goes smoothly.

Unpredictable connection times can cause problems for end of line testing:

• If a device fails to connect promptly, how long should you allow for the device to keep trying before ending the test?
• How do you know if a connection failure is caused by network issues or the device itself?
• How will variable connection times affect the end of line testing process? Can you handle a backlog of devices waiting for testing because some devices are taking longer to connect than expected?

NB-IoT devices have an additional problem. It can take many tens of minutes for a device to connect to an NB-IoT network, which would cause significant delays on a high-volume production line.

Roaming restrictions

In many IoT deployments, some devices will roam, either because they’re mobile devices or because they’re located in areas not covered by the chosen networks.

Permanent roaming restrictions and data processing regulations are complicating the roaming landscape. In addition, mobile network operators are becoming increasingly reluctant to allow IoT devices to roam on their networks because they receive negligible revenue for devices that generate little traffic but can make heavy demands on their infrastructure and support services.

Operators often restrict the number of IoT devices that can roam on their networks and some block them altogether. Even established roaming agreements are at risk, with operators revoking them, sometimes at short notice.

Roaming during the end of line testing is problematic. A connectivity test that works one day might not work the next if circumstances change – for example if the maximum number of roaming devices has been reached or an operator changes the roaming agreement. Although Eseye’s AnyNet connectivity service can localize devices, so they don’t have to roam, this takes time – not ideal for a high-speed production line.

Limited testing scope

The circumstances for your devices in the field may be very different to those at your production plant: they may have to connect to a different network using different frequencies, channels, or bands. Signal strength may vary considerably depending on location and other conditions.

A test verifying that a device connects to a live network at your production plant covers a very small subset of all the possible factors that your devices might encounter in the real world. You may get a false sense of security if your devices are passing the end of line testing with flying colours – for example, a device with a faulty or incorrectly installed antenna system might still pass the test if there’s a strong cellular signal in the factory. On the other hand, if the test facility is at the limit of service coverage or has an intermittent signal loss, your devices may be failed unfairly.

An extreme example of limited scope is using a golden SIM that’s fitted into devices for testing. A golden SIM is a SIM that’s provisioned to connect to a known local network, giving it a high chance that the connection will succeed. However, this provides no guarantee that the devices will connect successfully with other networks under different conditions.

In addition, golden SIM testing doesn’t verify that the real SIM is installed correctly and can communicate with the modem. And there’s the risk that a tester leaves a golden SIM in a device and doesn’t realise their mistake until hundreds of shrink-wrapped devices are loaded on pallets waiting to leave the factory!

Room for error from manufacturers

Your manufacturer might unwittingly contribute to unreliable connectivity testing by seeking to remedy any issues. For example, if devices are failing to connect and testers discover a better signal somewhere else in the factory, they might reorganise their assembly line to enable testing in that location. Or they might use an RF signal booster to magnify the signal and ensure that devices connect.

However, this testing only proves that the devices can connect to the local network with a strong signal. There’s no guarantee they will work when they’re further away from a cellular mast. This can lead to hard to solve problems when the devices are in the field. For example, they might suffer intermittent failures depending on signal strength or network availability.

Early activations

To connect to a mobile network, you need to use a golden SIM or activate the SIMs in your devices. As discussed, using a golden SIM can cause problems. However, it’s also not ideal to activate SIMs at the manufacturing stage. In some cases, your devices may not go live for weeks or months after manufacture while they’re shipped, distributed, installed, or sold. This gives you less control over the devices – you might have to start paying service charges before the devices are in use or (in the wrong hands) devices could run up charges that you weren’t expecting.

What this all means in a nutshell

Relying on devices connecting to a live network for end of line testing in a manufacturing or production plant can restrict flexibility and cause disruptions to the manufacturing process. Increasingly, problems with network availability can cause unreliable and inconsistent test results. In some cases, a production plant may not have access to a suitable network at all.

Coverage and validity of the tests may be inadequate, leading to faulty devices passing or devices that do meet requirements failing because the test facility is at the limit of service coverage. Activating SIMs in order to connect to a local network gives you less control over the devices once they’re shipped and risks unexpected or early connectivity charges.

Solutions like Eseye’s AnyNet are designed for managing connectivity in a live environment. It takes time to steer or localize devices to different networks, which isn’t appropriate for high-speed production lines.

In our next article, we will discuss two options for establishing a reliable and repeatable process for testing IoT devices on a production line. That way you can be sure that failed devices really are faulty and devices that pass are good to go – saving time and cost and paving the way for smooth and successful installations in the field.

Paul Marshall

Founder & CCO

LinkedIn

Paul is one of Eseye’s co-founders. With a background in senior design engineering, Paul’s focus is on ensuring his development, operations and support teams deliver solutions that work faultlessly in the field.

Paul was co-founder of CompXs, with Ian Marsden, and developed the world’s first IEEE 802.15.4 radio. Before CompXs, Paul was in senior radio design at Philips.