The following information is provided to help with basic troubleshooting steps for the ComXchange PBX. There are tools in the ComXchange Web Gui, Command line tools, as well as other software that will be helpful in diagnosing configuration and other issues.
Web Gui Modules and Reports
ComXchange Reports
The ComXchange Reports Tab on the Navigation Menu provides links to modules that can provide helpful information from the Asterisk server.
Asterisk Info
Provides reports on:
Registries – Status of Trunk Registrations
Channels - Active Calls
Peers – Extension registration status and IP information
Conferences – Active Conference Bridge Information
Subscriptions – Registered Asterisk Dialplan Hints (Park, BLF,Queues, etc)
Queues – Information on Queues and Agent Status
Asterisk Log Files
Provides access to various system log files including the asterisk “Full”log file that will provide asterisk call detail events, pms log, license log, fail2ban logs, etc
Call Event Logging
Logs of system calls with links to a web table of the asterisk logs for the call
CDR Reports
A reports listing Call Detail Records which has a summary of call information as well as a link in the systems column to the call form with call data from top to bottom
Call Event Logging
A simpler view of call information on the server with a link that provides more detailed, step-by-step records of events within a call's lifetime
Print Extensions
The Print Extensions report can be used to easily verify staff and guest extensions, feature codes, Inbound routes, and other call destinations.
Troubleshooting with Other Web GUI Modules
Sip Settings
The wrong External IP address in the General Settings tab - Nat Settings can cause issues with one way audio or failed outbound calls over a trunk. The IP address entered in this field in most cases should match the WAN IP address of the ComXchange Server. The SIP Local Networks, found in "Asterisk SIP Settings," define the internal IP ranges where local phones and devices reside, allowing the PBX to correctly handle NAT.
You may come across devices that are not sending nat such as some paging devices where calls prematurely hang up you can set the rtp timeout from the default 30 seconds to something like 5 minutes to alleviate the issue
License
In the License Module you can verify that the License application is running and the license is Active as well as the station count is within the limit of the license. Issues with Guest Features such as PMS, Call Accounting, Wakeup calls, or the inability to apply configuration changes (getting a reload error) could be attributed to an inactive license. You can verify the license is active in the Admin > License module
System Settings > DHCP -Leases
The DHCP leases tab will provide the ip address associated with a devices mac address. This can be helpful when trying to add a phone or gateway to verify they are on the network and can communicate with the ComXchange Server
System Settings > Intrusion Detection
Devices that try to register to the ComXchange server with bad credentials will be banned from trying to register again for a time specified in the Intrusion detection settings. If you are having registration issues you can check this module to see if the device has been banned.
System Setting > NTP
If your devices are not getting the correct time you verfy the server time and allowed client networks in the NTP Module. Other causes for the wrong time on devices could be the NTP settings in the device configuration as well.
System Settings > Services
You can verify that system services are running and restart them in the Service Module
Hotel Dashboard
The hotel dashboard at 192.168.101.2:8080 (be default) is useful to verify the room status, names, and restrictions of guest phones. It is also helps verify the PMS connection is functioning and populating information automatically
Interface Testing
There are a few tools that can be used to help in testing the PMS interface and the Call Accounting Server interface that connect to the PMS from the ComXchange Server. You can see interface date the interface logs and data monitor for Call Accounting, run a room status from the PMS interface web gui or run a test from the Call Accounting web gui, and perform a loopback test. For more detailed information see this article https://kb.clearlyip.com/comxchange/Interface-Testing.html
Terminal Emulator and CLI Commands
There are many terminal programs available to remote into servers and devices. Putty is a very popular program that is free to use. With a terminal emulator you can use SSH to console into the ComXchange server. This can be useful to see data in the Asterisk server, restarting services, viewing log files and monitoring log files and monitoring call events in realtime.
Connect to the ComXchange over VPN or the local network then use SSH to log into the ComXchange Server. You can use Linux commands or you can enter the Asterisk CLI to monitor and interact with the asterisk server.
Common Commands for linux:
Below are some common commands that you may use in the linux command line as well as some of the log files that you may want to monitor.
- ifconfig - shows status of your Ethernet interfaces (including assigned IP address and MAC address).
- ip addr (display IP information for all devices)
- ping - always useful for checking network connectivity and/or connectivity to specific remote hosts and networks.
- dig / nslookup - useful for troubleshooting DNS errors. Ie. dig www.google.com or nslookup www.google.com.
- df -h - check free disk space
- top / htop - live system resource utilization
- ps aux - show all running processes
- grep - extract specific information from a command. Ie. ps aux | grep
- pwd - ‘print working directory’ will print the path to your current directory.
- ls - gives you a list of names of files and directories. Can vary the output with flags such as ‘ls -la’ which shows all files and directories (even hidden ones) in ‘long listing’ format.
- man - stands for ‘manual’ - shows usage instructions for any command. Try ‘man ls’ and see what the various flags are.
- cd <name of directory> - change to a different directory. You can use ‘cd ..’ to go back (or ‘up’) a directory.
- cat <filename> - prints the contents of a file - you can go page by page by using ‘cat <filename> | more’ or you can simply do ‘more <filename>’
- tail <filename> - shows the last 10 lines of a file (such as a log file). To show a custom number of lines do ‘tail -50 <filename>’ where -50 is the number of lines you want to see. You can also do ‘tail -f <filename>’ to show the last 10 lines of a file, and continuously display new lines of the file as they are written - very handy for troubleshooting
Navigating log files
tail - tail is very useful for viewing log files. tail -X [/path/to/logfile] will display the last X number of lines of that logfile. tail -f [/path/to/logfile] will continuously display the logfile as entries are written.
Common log files for troubleshooting:
/var/log/messages - the main Linux logfile.
/var/log/maillog - SMTP log
/var/log/asterisk/full - the main Asterisk logfile
Useful Fwconsole/Asterisk commands:
Fwconsole Commands
- fwconsole restart - restart FreePBX and all related services
- fwconsole reload - reload FreePBX configuration (same as pressing ‘Apply Config’)
- fwconsole chown - set ownership of all FreePBX files
- fwconsole pm2 --list - See a list of pm2 services running
- fwconsole pm2 --log <service name> - see a running log of the pm2 service
Asterisk Commands
Asterisk - the Asterisk CLI is incredibly useful for watching PBX activity and running commands in Asterisk that can show you phone/trunk status, set debugging levels, and enabling/disabling services.
- asterisk -rvvv - enter the Asterisk command line. You can add more ‘v’s’ for additional verbosity.
- sip show peers (display SIP peers and IP)
- sip show peer <extension> (display information for a specific extension)
- sip show registry (display SIP registries)
- sip set debug on/off (turn sip debug on and off) note: remember to shut off
- Sip set debug peer 7001 (debug a specific peer)
- Sip set debug ip <IP Address> (debug a specific ip address)
- core/pjsip set debug on/off - turn debugging on/off
- pjsip show registrations - show pjsip registrations (pjsip SIP trunking status)
- pjsip show endpoints - summary of pjsip endpoints (pjsip extension information)
- pjsip show endpoint <extension number> - detail of a single pjsip endpoint
- pri show span <span number> - shows the status of a PRI
Tcpdump
Tcpdump is a powerful tool for analyzing network traffic going to and from your server. With the proper filter settings it is possible to isolate SIP and RTP packets.
Here are some practical examples of commands we use. tcpdump has some parameters that can be used if desired which can be found in the tcpdump man page. Examples
- -n[nn]: Disables DNS/service name resolution. -nn also disables port name resolution (recommended for speed).
- -q: Quick output. Print less protocol information so output lines are shorter.
- -t:Don't print a timestamp on each dump line
Tcpdump Command examples
- View all SIP traffic in realtime
- tcpdump -nqt -s 0 -A port 5060
- View SIP traffic by IP And port
- tcpdump -nqt -s 0 -A host xx.xx.xx.xx and port 5060
- View SIP activity from a source IP and a destination port
- tcpdump -nqt -s 0 -A "src host xx.xx.xx.xx and dst port 5060"
- View asterisk RTP Traffic (VOICE)
- tcpdump -nqt portrange 10000-20000
Writing to a file with TCPdump
Note: these can take up alot of space so be aware of disk space and remove. The captures when done and can be downloaded from the server to a PC with an application like WinSCP
- To Save SIP activity to a file
- tcpdump -n -s 0 port 5060 -vvv -w /tmp/capture_file_name
- To Save SIP AND RTP activity to a file
- tcpdump -n -s 0 port 5060 or udp portrange 10000-20000 -vvv -w /tmp/capture_file_name
- To save all network activity from a specific port such as eth0
- tcpdump -i eth0 -w /tmp/capture.pcap
- To save traffic from any interface
- tcpdump -i any -w /tmp/capture.pcap
Diagnosing SIP issues with Wireshark and Tcpdump Captures
Diagnosing with WireShark involves capturing packet traffic on the network interface to a file and using that file for filtering for sip, and using Telephony > VoIP Calls to analyze call flows and identify failed messages (e.g., \(4xx-6xx\) errors). Key techniques include filtering by Call-ID, checking SDP for media negotiation, and analyzing RTP streams for voice quality issues like jitter or packet loss.
Key Steps for SIP Troubleshooting Capture Traffic
- Use Wireshark to open a packet capture that was done to reproduce the call issue (dropped, one-way, no audio).
- Locate traffic from a server or an endpoint by IP
- Enter ip.addr == x.x.x.x in the filter where x.x.x.x is the IP address of the device
- Locate SIP Traffic: Use the display filter sip to isolate signaling.
- Analyze Call Flow: Navigate to Telephony > VoIP Calls, select the target call, and click Flow Sequence to visualize the exchange between the User Agent Client (UAC) and User Agent Server (UAS).
- Identify Errors: Look for SIP response codes \(\ge 400\). Common errors include:401/407 Unauthorized: Authentication issues.
- 404 Not Found: Incorrect routing or dial plan.
- 486 Busy Here: Destination is busy.
- 487 Request Terminated:
- Call was cancelled (often by a 200 OK after a CANCEL).
- Inspect SIP Headers/SDP: Examine the INVITE request for correct From, To, Contact, and Content-Type: application/sdp to ensure proper media negotiation (codecs, IP addresses).
- Analyzing Audio (RTP) Issues If signaling is successful but audio fails:
- Filter for RTP Traffic:
- RTP Stream Analysis: Go to Telephony > RTP > RTP Streams to check for high jitter, excessive packet loss, or incorrect IP addresses.
- Hear the Audio: Use Telephony > RTP > RTP Streams, select the stream, and click Analyze > Play Streams to hear the audio, confirming one-way or no-audio issues.
Uploading ....
