Debugging Routing Issues

Registering XRPi/XR2000 Endpoint to Hawkeye Server

If endpoint does not register:

  1. Check routing to Hawkeye server.
  2. Check ports open to Hawkeye server.
  3. Use nmap (linux command) to check ports open.

nmap -d -p 10117,25025-25050,443 <Hawkeye server>

  1. Confirm able to ping Hawkeye server from endpoint.
  2. Confirm on the Hawkeye Server in Probe management hat the new endpoint can be seen.
  3. Confirm in Probe management for the cloud endpoint that a private corporate network IP and a public internet IP can be seen.
  4. Confirm on the Hawkeye server in Probe Health Check that the endpoint is in a state of link up.

Troubleshoot Connectivity between Endpoints

SSH to the XRPi/XR2000 endpoint or Hawkeye server and confirm routes with “ifconfig” and “route -n” commands. The ping and traceroute commands may be helpful.

ifconfig

route -n

ping -I wlan0 <peer endpoint>

ping -I wlan0 www.google.com

ping -I eth0 <peer endpoint>

Issues with Node to Node tests can be caused by connectivity issues. Check routes and no port blocking.

SSH to endpoint and use nmap to verify path between endpoints.

nmap -d -p 10115,10116,10117 <peer endpoint>

nmap -d -sU 10115,10116 <peer endpoint>

The output will identify if a port is open or closed.

The suggested ports to verify are the following:

  • Port 22 - TCP - traffic for ssh. Debugging for Admin users

  • Port 123 - UDP - NTP clock source

  • Port 80 – TCP – HTTP and HTTPS traffic
  • Port 443 - TCP - HTTP/HTTPS traffic. Real service traffic
  • Port 10115-6 - TCP – management. Hawkeye server to Endpoint. Endpoint to Endpoint management
  • Port 10115 - UDP - Endpoint to Endpoint sync timing traffic for RTP traffic like Skype4B, Voice services, Network KPI
  • Port 10117 - TCP - Endpoint auto-registers with Hawkeye
  • Port 25025-25050 - TCP - Endpoint reporting test results
  • Skype4B traffic port ranges are configurable. Vary with test traffic type. Example for Skype4B is given below:
    • UDP ports 49751 – 49999 (default Skype4B audio)
    • UDP ports 59751 – 59999 (default Skype4B video)

Verify traffic ports used in Hawkeye server in Preferences – Traffic Port Management. To verify routes correct run N2N traffic test between two endpoints (UDP/TCP bidirectional tests). First run ipconfig on endpoint, then run test, then run ipconfig after test and this will show packets increasing for RX/TX on interface used.

Troubleshoot Endpoints/Hawkeye Server in the Cloud

In this case, non-streaming flows over UDP and test traffic are supported only if the actions mentioned in “port limitations” are performed.

For the firewall at the entry point into the cloud, make sure to open the following ports:

TCP management ports:

  • 10115 (for the source endpoint)
  • 10116 (for the source endpoint)
  • 10117 (for the Registration Server)

TCP reporting ports (for the Hawkeye Server): 25025-25050

UDP port: 10115

Open into the Cloud firewall the TCP/UDP ports that you plan to use for test traffic, then go to User Preferences and configure the same range there in the Administration > Preferences > Traffic Port Management.

Probe Bouncing IPs in Probe Management

The Hawkeye server appears to track each probe by its name in Probe Management. However, in Preferences > Test Controller table, Probe Management Identification allows the user to track the probe by its management IP or the UID (MAC address of interface on probe). There are cases where probes such as XR2000 VMs have been cloned at Customer sites and in Probe Management it appears that the probe is bouncing as the IP appears to keep changing in Probe Management. This happens when the multiple XR2000 VMs send the keep alive every 2-4 seconds to the Hawkeye server, the Probe Management entry for the probe keeps changing its IP addresses for the probe. As the XR2000 VMs have been cloned the UID (MAC address for eth0) is always the same.

All XR2000 VM images posted contain a blank UID.

Debugging Test Management Routing Issues

Probe overview for Routing

The probe has two separate services running that depend upon routes and ports. The first is the probe process that is used for management with the Hawkeye Server and executes node to node tests. The second process executes real service tests (if probe supports Real Services).

Probe process - Register with Hawkeye Server and Node to Node tests

Configuration script is run to register the probe with the Hawkeye server. Either the Hawkeye Server IP or Hawkeye hostname is provided. If user chooses the Hawkeye hostname, a DNS lookup is performed to obtain the IP address and from this point onwards, the saved Hawkeye Server IP is used for the ‘keep alives’ sent every 2-4 seconds to the Hawkeye Server. The Hawkeye Server will update the management IP for the probe each time a keep alive is received. If NAT or a gateway is involved the IP and port for the probe can change dynamically.

Every time the Real Service communicates to the Hawkeye Server when Hawkeye Hostname is configured a DNS lookup occurs before packets are sent in case Hawkeye Server VM has been moved to a different IP address.

Check NTP clock synchronization process is running on the probe if Node to Node tests fail. This can be done by ssh to the endpoint and executing the following command:

# ntpq -pn

Probe Management – Status Indicator

Key point is probe originates packets to Hawkeye Server, Hawkeye Server does not initiate communication to probe (unless manual probe). The Hawkeye Server stores the IP from the ‘keep alives’ and uses this IP and port for the probe for all management traffic.

Every 2-4 seconds, the probe (random TCP port) communicates to the Hawkeye Server (port 10117/ TCP unencrypted) to update the probe health status web UI. Any change in the IP address of the probe will also be updated in Auto probe management.

Probe originates management (including registration) to Hawkeye Server, Hawkeye Server records this route (IP + port) then uses this for management packets back to probe. Specifically test commands and configuration and additionally reboot requests.

Real Service Command Manager process

Hawkeye Server sends the Real Service test command information to the management IP (‘keep alives’) defined for the probe in probe management (Server port 443/TCP encrypted). The real service command manager process (running on probe) will execute the test and then send the results back to the Hawkeye Server. The test results will be stored in the Results database and any logs from the test will be stored in the Real Service manager log files on the Hawkeye Server.

Every time the Real Service command manager process communicates to the Hawkeye Server when Hawkeye Hostname is configured. A DNS lookup occurs before packets are sent in case Hawkeye Server VM has been moved to a different IP address.

Node to Node test

When the User runs a node to node test, management communication takes place between the Hawkeye Server and the two probes. In the Test Execution UI on the Hawkeye server the user selects the ‘FROM’ probe and the ‘TO’ probe. The user selects the interface (IP) to be used for the test traffic but the actual commands and configuration are sent to the probe management IP (‘keep alives’ are received on).

There are three key concepts to know what ports are used when:

  • All tests fall into the category of either being classified as UDP tests or TCP tests. The test results for UDP tests are sent from ‘TO’ and TCP tests are sent from ‘FROM’ probe.
  • Probe is listed as Private or Public in Probe Management.
  • Bi-directional tests.

The Hawkeye Server sends commands to both probes at the start of a test on port 10117. Between the two probes (port 10115 and possibly port 10116) are opened for exchange of test statistics and management of tests. The test traffic between the probes will use different ports. Once the test is complete the test results (stored in .csv files) are sent to the Hawkeye Server. If the test is a UDP test, the ‘TO’ probe will send the test statistics (.csv file) and log file to the Hawkeye Server. If the test is a TCP test, the ‘FROM’ probe will send the test statistics (.csv file) to the Hawkeye Server. The Hawkeye Server will store these log files (log rotated) and record test results in the Hawkeye Server Results database.

In Probe Management UI, for each probe a field identifies if the probe is ‘private’ or ’public’. If in a Node to Node test both probes are ‘private’ or both are ’public’ only port 10115 is used between the two probes for management. Specifically the ‘FROM’ probe will open port 10115 to the ‘TO probe. If the probes are a mix of ‘private’ and ’public’, the additional port of 10116 is also opened for management. If one probe is Private and the other is Public, the Private probe will open connections for ports 10115 and 10116 to the other probe for Management.

All Bidirectional tests are different in that both probes send statistics and log files back to the Hawkeye Server. This can be viewed as running two separate uni-directional tests, with each probe in both the ‘FROM’ and ‘TO’ logical position. As a result, for example if running a UDP Bi-directional test, each probe will generate traffic for one test and calculate statistics for the other test.

How to Debug Routing issues – Example: Node to Node Bi-directional test on Linux probe

The XR2000, XRPi and XR2000 VM are Linux based probes. The following identifies Linux debugging commands.

When the probe ‘from’ registers with the Hawkeye Server (it sends ‘keep alive’ message every 3-4 seconds to the Hawkeye Server), this IP (referred to as ‘keep alive IP’) will be used by the Hawkeye Server to send the test command information to the ‘probe from’ using port 10117 (this can be different to IP chosen for test). The ‘probe from’ then sends the test configuration to the ‘probe to’ using the IP addresses defined when creating the test. The probe IP sent in the test command will either be the IP selected from the pull-down menu when creating the test, or if set to ‘default’, will be the IP used to communicate management messages to/from the Hawkeye Server.

For the test duration, the probe ‘from’ and the probe ’to’ will send traffic to each other. Once the test is over each probe will process its test results. Based on the rules above one probe will collect statistics and results and send to the Hawkeye Server on port 10117. The probe’ from’ will wait to collect the test results from the probe ‘to’, then the ‘probe from’ will send the combined test results to the Hawkeye server using port 10117. If the ‘probe from’ does not receive the test results from the ‘probe to’ within a defined timeout period (System preferences – Test Engine – Statistics receive timeout) the probe ‘from’ will send its test results to the Hawkeye Server (without results from probe ‘to’). If the Hawkeye server only gets test results for one direction of traffic it will report the test as an ‘Error’, but still report received test results. The test results will reflect the IP used for the ‘probe from’ and ‘probe to’ for the transmission of test results.

If the test results are not received by the Hawkeye Server and the test is reported as an Error, this could be due to a routing problem. Each probe can have multiple addresses assigned to it. If NAT and gateways are involved, a probe can easily have 3 different IPs and more. There can be the local physical IP, the public IP, a NAT IP and even an IPv6 address (per single interface). The user will have to know which IP to use when creating the test. If the ‘default’ is used for the test it is possible that this will not be correct to route traffic between each probe.

Many steps are available to identify where the routing issue is occurring.

If a Real Service license has been purchased, the real service tests of ‘Port Scan’, ‘Path Discovery’, traceroute, TCP Ping and UDP Ping can easily be used to assist in identifying the routing issue. For example, the Real Service test of ‘Port Scan’ could be used where the IP for each probe in the Node to Node test would run the Port Scan test where the destination would be the other probe. It has been found that the ‘Path Discovery’ test is particularly powerful, which includes traceroute output and calculated network delays.

For each Node to Node test ran, the Hawkeye Server has a log file of the test results. The log file (/opt/hawkeye/logs/Chariot/<testID>.log) can be checked to confirm the test results are present. A key tip is to stop/pause all other tests so the log files can easily be identified.

If a Real Service license has not been purchased, a number of commands can be run from the command line interface of each probe. The below commands will capture all management packets between probe and Hawkeye server using port 10117 and all management packets between probe to probe on port 10115, and send output to a file (-w trace_10115.pcap). Either leave out the log to file options and view in realtime or view .pcap file in wireshark.

# yum install tcpdump

# tcpdump -X -e -n -w trace_10115.pcap port 10115

# tcpdump -X -e -n -w trace_10117.pcap port 10117

The above will identify exchange of packets on port 10117 and the IP for the sent/received packets.

Confirm management port for probe to Hawkeye server and management port between probe to probe is open:

# nmap -d -p 10115 <IP other probe>

# nmap -d -p 10117 <IP Hawkeye Server>

Confirm port for management can be used between probes:

# traceroute -p 10115 10.146.4.238

While a Node to Node Bi-directional test is running the following command will constantly repeat and identify the established/transmitting connections for ports 10115 and ports 10117 every 5 seconds:

# netstat -atnc 5

# netstat -tc 1

Protocol Local address remote address state

tcp 0 0 10.36.32.29:43932 10.36.34.191:10117 ESTABLISHED

tcp 0 0 10.36.32.29:10115 10.36.33.18:51796 TIME_WAIT

tcp 0 0 10.36.32.29:10115 10.36.33.18:51798 TIME_WAIT

The remote address contains the IP for the Hawkeye Server and the other probe for the test. The state will alternate between Wait/Listen/Established when sending packets.

In order to confirm what route will be taken for a given destination IP, view the configured routes on each probe for its routing tables:

# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

10.36.32.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0

169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0

0.0.0.0 10.36.32.1 0.0.0.0 UG 0 0 0 eth0

You will see routes for specific IPs such as the Hawkeye Server IP, and a route for default traffic.

To find all probes communicating to on Hawkeye server:

Run command for a minute then end with control-C.

# netstat -tc 4 > /tmp/10117_trace

# grep 10117 /tmp/10117_trace > /tmp/10117_trace_a

Same netstat command on probe should match entry on Hawkeye server file above.