What is the general function of our solution?
Our system is designed to simulate common user behavior within an operator's network, using its services.
In fact, our solutions and services are designed to give our customers a complete insight on their service quality from an end-user perspective. Therefore, a network operator won't be able to distinguish between a real user and a test probe.
What are the system components?
- Probe – An exclusively manufactured device which consists of one or more CPU-Cards (channels) that act as an independent industrial standard computer. Each CPU-Card has, at least, one modem board assigned to it. Every channel has a permanently active agent that performs tests according to predefined scenarios and collects the necessary KQIs/KPIs with detailed technical information (traces). The agent uploads the results to the central server OTA (Over the Air) or via a wired connection through a corporate IT network. Every probe device has a modified Linux OS with the XMASS client installed on it.
- Central server - Hosts the XMASS web application, collects, and processes the uploaded data of the probes. All test probes constantly send their system status to the central server and can be remote controlled via XMASS. The central server is also responsible for report generation, alarm generation, KPI/KQI calculation and provides data feeds to the external systems.
- SIM multiplexer - Stores up to 7x32 sim cards which can be remotely used by the test probes. You are free to assign every sim card you want to probes all over the world and even switch sim cards between each test case.
- Sim farm - Is an adapter board which can store up to 20 sim cards. Unlike a sim multiplexer, it doesn't provide its sim cards to more than one device but to the local channel.
The System architecture can be centralized with one central server for data storage/processing/probe control and distributed probes.
What is the task and function of a server?
Once the probe has started a test (the test description comes from the central Application Server) and has measured all the needed KPIs, the results are then transmitted to the application server immediately.
The Application Server stores these incoming results in its database for later analysis.
It is important to know, that all measurements and the entire test logic takes place in the probe and not on any kind of server! The server is responsible for receiving the data and storing it in its database for later analysis.
Besides this, the (Application Server) also provides functionality to create or modify test scenarios, reporting, and Monitoring of test results, etc. In short, the server is responsible for pre- and post-operations.
How many probes are needed to test and measure all KPIs?
Some Scenarios require two probes to be involved (like Call, SMS, MMS, Video Telephony, VoIP, etc.) as if there were actually two persons involved (MO&MT). Some other test scenarios (like HTTP, FTP, DNS, PING, Streaming, etc.) work against the public Internet or reference download servers (private or public). These type of tests do not require more than one probe since they are single sided (MO only).
Note: There are also some rare scenarios using more than two probes (like Call conference, Call forwarding, etc.)
Depending on the kind of test scenario, you might need one or more probes at the same time to work with each other. The Application Server takes care of them, keeping them in sync and providing them with the right test instructions at the given time.
General information and phrases?
Please refer to the system description documentation to read more about the structure of test scenarios in our system. You will see, that our system is highly flexible due to an open scenario structure, which provides you with the capability to combine all kinds of services as single steps in the same scenario. These steps are called "test cases" and run sequentially, one after each other. The same way a human being would use his handset.
What is the corresponding deployment diagram of probes and engineering steps, if we want to start a trial e.g. on HTTP browsing and download, Video streaming, Voice call and IVR?
The trial would be very simple to manage. Our Platform itself and a reference Download Server will be made available to you in (Amazon Cloud) instances, which are also available in Asia. You just need, for example, two probes, placed in two locations with Power Supply and Internet. That's all.
We can then remotely set up the tests and capture all results according to a pre-agreed test scenario. Only with IVR, it is not possible to be tested that quickly, since IVR scenarios have to be created according to the IVR System used by the customer.
This is a very time-consuming task and is normally done in the production system and not in a trial.
You need to keep in mind, that our System is a very flexible and powerful Test Automation system, thus, it is almost impossible to show all features and capabilities in a trial. The more you work with the system, the more features and capabilities are detected.
The trial can only show an indication of how powerful such a system can be.
How long does it take for test results to be transmitted to the server? What are the test results and how long is the upload duration?
The test results are normally uploaded after every scenario run. E.g. if you run a (download) test scenario every 10 minutes with some steps like “Connect -> DNS Check -> Download Test -> Upload Test -> Disconnect”, 5 Test results (one for each step) are uploaded after every run (every 10 minutes).
The size of the uploaded results depends mainly on the Features and Trace capabilities you have turned on in the test itself. The more Traces and information you want to capture, the bigger the size of the results. E.g. if you enable the IP-Tracing with full packet size capturing and then download a 10MB file, your traces will, at least, have the same size.
Summary: Needed time to output test results with an average size of 0.5 – 10 MB through the LAN is about 5-30 seconds and through OTA and with 2G technology is about 3 minutes.
In general you can work with an average of 5MB per test result, these might take a few seconds over LAN or up to two or 3 Minutes over 2G to be uploaded.
If the connectivity is not given at the upload time or if the probe has been set up to avoid uploads, the results will be stored on the internal flash for a later upload, once the connectivity is established. This feature is very handy for some situations like Drive-Test or indoor poor coverage tests, etc.
The internal Flash provides almost 5GB of free space for storage results. Under normal circumstances, this is enough for up to two weeks of testing without an upload.
What is Drive testing?
Drive Testing is just one feature in our system. Every single probe – particularly PUX, MiniPUX, and MicroPUX – can be setup to run in a (Drive Test) mode. The probe then runs the Service Tests as usual. However, at the same time, it collects all GPS information, as well as Network Snapshots every 2 seconds.
Almost all available Drive Test systems on the market collect only the Radio Information and can barely test the service layer (like HTTP, FTP, SMS, MMS, Call, etc. and combinations).
Our system, in fact, does both at the same time. Of course, by default, the depth of collected Radio Parameters in our system is not as deep as pure Radio-Optimization Systems. However, using the additional Layer3 License allows you to collect the entire Layer3 communication between probes and the network, which is the maximum you can get.
In short, our system does a lot more on top, namely the entire service-testing layer.
What is Standalone or on-demand testing?
This is another example of rich features, capabilities, and flexibility of our system. Stand-Alone or on-demand testing is the capability of every channel in every probe to provide a special software, which can be used remotely (using SSH and X-Forwarding in Linux, similar to Remote Desktop) to design a test scenario, run it and look at its results within minutes. This feature is very handy when it comes to special troubleshooting cases, where you need to “try and error” in a few different situations with different test scenarios. What happens, is the following:
You just log on a channel directly (using SSH), stop the standard Test Client Software and run the RSD Client (Rapid Scenario Designer) instead. In RSD, you can easily create a new scenario and then click on different Icons to add test cases to it. Afterward, you can run the whole scenario or step by step and look at the results immediately. Once you have your test scenario ready with all needed settings, you can save it as an XML file, upload it to the server and distribute it to your entire probe landscape.
What is On device testing?
PIXIP.NET also provides complementary products like XSmart to run tests directly on mobile devices. Currently, XSmart supports iOS and Android devices and is capable of testing Download, Upload, and Round-Trip Delay within 2G, 3G, 4G and WLAN networks. It records the download/upload performance and their details (both Average and effective Throughputs) as well as GPS positions, PLMN, CELL-ID and LAC-ID (only on Android). All results are uploaded to the dedicated database in your environment.
XSmart helps end users as well and animates them to run more tests for the operator. Every time a user runs a test, the App shows the current throughputs and also other results around him, to identify a better position with better performance. This way the operator can provide an added value to end users and collect more and more useful performance measurements from its customers.
XSmart App is free for end users. The operator needs Server licenses, which are CPU and CPU-Core based. We also provide the customization to apply the Operators Colors and Look & Feel to the App.
XSmart can also be purchased as a separate product and is independent of the EMQP platform.
About IM (Instant Message) like WeiXin and QQ to be used mostly in China?
The biggest issue with such a Messenger is, that their protocols are proprietary and not always open or public and their source codes are not available. So it is difficult to test their internal functionality. However, all of the IMs use TCP, UDP, VOIP and other standard protocols. According to uncover the network issues with such Messengers, it is normally sufficient to test the base protocols within the given network. This means to test TCP and UDP transmission, VOIP (SIP), PING and DNS, all these protocols are available in our system out of the Box.
Please note, the main approach in an (Active Test) system like ours, is to test the network and not the applications, since the application stability may vary strongly from one version to another.
Does your system support LTE TDD and CDMA, which are used mostly in China?
Yes. The system currently supports both of them. However, there might be some customization and integration work required according to every operator's network.
The integration very often is just a matter of developing some additional software for XMASS environment.
What about physical and virtual SIMcards, SIM-multiplexer and SIM-Array?
SIM cards can be used in 2 forms per probe:
- Physical SIM cards locally up to 9 SIM cards (= 1 basic SIM card + 8 additional Sim cards using “SIM-Farm” Board) per modem.
- Alternatively using SIM-Multiplexing technology along with SIM-Array, where physical SIM-Cards are in it. In this case, all respective SIM-Card information will be transferred to the installed “SIM multiplexer-enabled probe” in different locations, to test and measure.
The SIM multiplexer platform provides easy assignment and transfer of SIM/USIM from the SIM multiplexer to any locally installed (SIM multiplexer-enabled probe). A single SIM Multiplexer array supports simultaneous access to a maximum of 416 SIM/USIM. Multiple SIM Multiplexer arrays can be deployed to expand the required SIM capacity. SIM arrays can be remotely located from the Central Unit so that physical SIM management can be distributed to different groups.
When we use SIM multiplexing, the probe should be connected via LAN to the server and a stable LAN connectivity is mandatory.
What is the exact architecture of your probes?
Each PUX Probe has a 19-inch chassis with up to 8 Channels. Each Channel is made of 1 CPU Card and 1 Interface Card (Periphery Board /Modem Board).
Each Interface Card (= Modem Board) carries 2 Mobile Interfaces (=Modem) with 1 local SIMs and/or 1 Virtual SIM per each. Each Interface Card (= Modem Board) can optionally be equipped with an "SIM-Farm" Board to hold up to 8 additional local SIMs per Interface (18 local SIMs and/or 2 Virtual SIMs per channel in Total)
Note: Per modem we can have 1 real sim card and 1 virtual sim card, but these two cannot be used at the same time and we should switch to a real sim card or on the virtual sim card, according to what our intention is.
How many Periphery Boards (Modem Boards) are in each Probe and in each Channel?
Up to 8 per Probe and 1 per channel. In MiniPUXes, you can have up to 2 Periphery Boards (Modem Boards) and up to 4 modems (in Improved MiniPUX).
How many parallel tests can be run on the same Probe?
Depending on the number of channels, up to 4 parallel tests per channel and up to 16 per PUX Probe
How do you solve the default routing problem and ensure every test is using a dedicated mobile interface (= Modem) without affecting other interfaces
when multiple data tests are running parallel?
We do not use default routing principally, but dedicated routing. We have developed a part of TCP/IP stack from scratch, to have full control over binding a test to a dedicated real interface. This capability is unique on the market and is one advantage of the system.
How do you ensure that the power of the CPU is sufficient to avoid distortion during measurements?
Our testing engine considers this and automatically deducts the calculation drains on the CPU from the total time measurements. So only Network Interaction durations are considered in the measurements.
Are your probes equipped with a “watch-dog” functionality?
Yes. We have a 3 level watchdog in every channel.
How can probes be configured through the central server? Can the probes be configured, based on different operators’ Simcard parameters, when operating in an OTA mode?
Our probes are based on a sophisticated hardware and run with a “Test-Engine” Software. The probe can be configured to use the installed Modems for communication and transmission to the central server. In OTA mode, the probe can be configured remotely using SMS. It doesn't matter where the SMS comes from, however, the central Server needs to be connected to the SMSC (using SMPP) to be able to deal with the massive number of SMS messages.
In this way, the central Server can be connected to any SMSC or (Premium SMS Provider) to send and receive SMS’s. Normally, a dedicated premium number has to be configured in every probe as a trusted SMS sender to avoid security risks. If the provider doesn’t want to allow the SMSC connection, other premium SMS providers like Amazon can be used.
Note: The probes can be configured to use all available Modems with their SIM cards in a pre-configured order and priority.
Refer to KPIs, how do KPIs convert to QoE? What is the general performance and efficiency of your system?
There is no need to convert any value to others. KPIs are calculated according to a certain formula based on detailed measurements and KQIs are a kind of aggregation of KPIs. This may happen according to standards or according to the Users needs.
General performance: Our Test-System (XMASS) runs the measurements and delivers the results to the central database. Our Real-Time Monitoring system (XOS) uses these measurements to build KPIs and KQIs according to the settings made by Administrators/Users of XOS.
XOS is a very open system and has no restrictions according to building up the KPIs and KQIs.
All Measurements made by XMASS are 100% ETSI compliant. This makes the understanding of their meanings and their trigger points much easier. The other way of visualizing and analyzing the QoE data is to use the reports or to create custom made reports, which read the measurement data from a database and analyze them by calculating KPIs and KQIs.
In our system, we do have two different Reporting platforms available. One is our own development, based on Open Source technologies like BIRT and the other one is a seamless integration with SAP/Business Objects Enterprise Reporting system.
The second one is more convenient for Enterprise approaches with complex reporting structures and restrictions.
Video Telephony, Video Quality?
In Video Telephony, we do not look at the content of (Picture Quality) because this only depends on the used CODEC and the CODEC is something fix. More important is the transmission Quality and its BER (Bit Error Rate). This is what our system extensively measures and according to transmission problems, you can always predict the User Experience Quality.
Does EMQP support Portal Testing?
Portal Testing is a special kind of HTTP/WAP testing with recursive “Follow Link” capabilities. A very special thing about portal testing is, that it compares every single page and its content against a given Handset and the handset capabilities. This way the system generates a report about all pages and contents in the whole portal, which are not compliant with that particular handset capability. We have a database of some common handsets, however, the system also provides a “Handset Profile Recorder”, which allows you to create a profile from a new Handset within seconds. Please refer to the document (PortalScanModule.pdf) for more Information.
The End to End analysis concept. Does the PIXIP.NET system obtain the radio network performance by positioning a test terminal (or probe) in eNodeB and the test core network performance through a test terminal in EPC at the same time?
No. PIXIP.NET probes use and require only the air interface. All measurements, KPIs, and traces are collected over an Air-Interface. This is the reason, why our probes do not need any kind of connection to the operators’ core network. The greatest benefit for this a type of End-to-End testing method is, that you can also easily test the competitors' network. The only thing you need is just an SIM card of the competitor operator.
Can you provide a complete final report module (with no contents but only with items) to potential customers, to view what performance analysis PIXIP.NET system can provide? How to match the QoS concept? Can you provide a full report about a trial network?
In the attachment, you can find a few exemplary report files (example_Reports.zip), however, these are just a few examples and cannot reflect the full range of possibilities of the system.
Our system provides an open database, so there is no limit for the type and the contents of the reports.
Therefore, we deliver our own reporting system (all example reports attached are done with this tool), as well as the full Business-Intelligence platform of (SAP-Business Objects), with our system to provide a very flexible enterprise reporting platform.
This system allows the users to create and run any kind of report on the fly and directly in the browser.
Beside these, it is recommended to have a look at the document “EMQP_KPI_Overview.pdf”, where you can find a collection all ETSI standard KPIs available. Of course, the users can create their own KPIs since all detailed measurements are also available in the database.
Is the QoS concept in your system different from the QoS in the 3GPP specification?
Our system is not different from QoS in 3GPP; however, the 3GPP specifications cover many different aspects of QoS and Testing.
One of these aspects is the so-called “End-to-End Testing” or “Active Testing”.
This is what our system is designed to do and the system does this fully compliant to ETSI spec and 3GPP.
I’d like to understand a little more about setting up tests, e.g. whether you provide a GUI (Graphical User Interface) or if there is a scripting language we can use to set up tests?
Our system is based on a “Building Blocks” principle. This means, there is no need for coding or scripting. The system provides a large library of single steps (we call them test cases), which have their Parameters (Inputs) and their Results (outputs).
Using the GUI (Scenario Designer), you can create your scenario within seconds, by clicking the single steps in the order you want, to simulate a subscriber doing some actions. By using the input Parameters of each test case, you can change the behavior of that particular step. Some examples of test cases are: “location Update”, “once to Data Network”, “disconnect”, “Send SMS”, “Receive SMS”, “all a number”, “hang-up”, etc.
Every single test case is equipped with a timeout parameter, which will cause the test case to interrupt if a timeout occurs, regardless of how far the test case is. This way the total duration of a scenario is fully controllable.
Would a development for our specific test case be something that we would do or PIXIP.NET?
Normally there is no need to develop a test case (we already cover all services), but if needed, you can develop your own test case in Java (of course you will need our support for doing this). Developing a test scenario is a matter of just a few minutes if you know what you are looking for.
