Uncategorized
/

I worked with the Broadworks platform for over 10 years. One of my favorite elements of their solution is the Network Server, or NS. The NS processes all INVITES from Back-to-Back User Agents to perform translations, routing lookups and more. Once the INVITE has finished processing, the NS will return a 302 Redirect with the Contacts that should be used, or another final response (such as 404 Not found).

One rainy Saturday in Amsterdam (during COVID-19) I decided to build a basic 302 Redirect Server with Kamailio the mimic some of the functionality of the Broadworks NS.

I started off with the minimal-proxy-config file as most Kamailio based projects do. You can find the complete code HERE, including a docker-compose file that you can use to launch the entire environment. The docker-compose file will build the following topology.


Loading Modules and Setting Modparams

First, we need to load the modules used in this tutorial. Before you add these into your config file, be aware that there is an order that modules should be loaded. Check the full file in my gitlab repo for a better understanding.

loadmodule "dmq.so"
loadmodule "db_mysql.so"
loadmodule "htable.so"

I’m also pulling in environmental variables to use in the config. This is performed via #!substdef in combination with the $env psudovariable.

####### Other Args and Env Vars #########
#!substdef "/DEFAULT_DEST/$env(DEFAULT_DEST):5060/"
#!substdef "/MYIP/$env(MYIP)/"
#!substdef "/LOCAL_SOCKET/$env(LOCAL_SOCKET)/"
#!substdef "/PEERNAME/$env(PEERNAME)/"

Next, we need to configure a few modparams. For htable, we are enabling dmq, defining the database url, and defining which tables needs to be replicated via DMQ. Because we are using a database with htable, Kamailio will obtain the contents of the respective database table upon reload of the service.

# ---- htable params ----
modparam("htable", "enable_dmq", 1)
modparam("htable", "db_url", "mysql://user:password@172.16.10.254/db")
modparam("htable", "htable", "dnis=>size=14;dmqreplicate=1;dbtable=dnis;")
modparam("htable", "htable", "deny_list=>size=5;dmqreplicate=1;")
modparam("htable", "htable", "active_calls=>size=5;dmqreplicate=1;")
modparam("htable", "htable", "admission_limits=>size=5;dmqreplicate=1;")

We also need to enable DMQ. The server_address value is where this server is listening for DMQ messages. The notification_address is where all DMQ messages will be sent for another peer to process.

modparam("dmq", "server_address", "sip:MYIP:5060")
modparam("dmq", "notification_address", "sip:PEERNAME:5060")

We are setting flags based upon values being true while the routing logic is taking place. You can learn more about flag operations here.

#!define FLT_A_ON_NET 20
#!define FLT_B_ON_NET 22
#!define FLT_DENY_ADMISSION 23
#!define FLT_DENY_CALL 24

Building the Routing Configuration

Now that we have a few basic settings in place, we can begin looking at the routing logic. The request_route block is simple, and the logic is really handled within other route blocks.

request_route {
  route(HANDLE_DMQ);  // This is the entry point for handling all DMQ messages
  route(HANDLE_OPTIONS); // Responds with a 200 OK for all OPTIONS

  if (is_method("INVITE")) {
    route(IS_A_ON_NET); // Looks up if the originator is listed in htables
    route(IS_B_ON_NET); // Looks up if the term # is in htables
    route(IS_DENIED); // Performs a deny function for blocked B numbers.
    route(REPLY_302); // Sends the 302 redirect 
  } else {
    xlog("L_INFO", "MAIN | $ci | Non INVITE received.");
  }
}

Handling the specific DMQ traffic is performed by the dmq_handle_message() function. You can see that this is only for requests where the method is KDMQ.

route[HANDLE_DMQ]{
  if(is_method("KDMQ"))
   {
           dmq_handle_message();
   }
}

Handling of OPTIONS is much like the DMQ block. This route block checks if the method is OPTIONS, and replies with a 200 OK. This is configured as the “SBC” is sending OPTION pings to determine if the 302 Redirect Servers are online.

route[HANDLE_OPTIONS]{
  if(is_method("OPTIONS"))
   {
           sl_send_reply(200, "OK");
           exit;
   }
}

The routing block IS_A_ON_NET determines if the A numbers is on-net. It looks into the hash table named dnis, and queries with the from user value ($fu). If you aren’t familiar with the pseudovariables, you should spend some time learning the basics. If $fu exists in htable, set the flag bit, otherwise, reset it.

route[IS_A_ON_NET] {
  # This route block tests the originating number to determine if it is linked the platform
  xlog("L_INFO", "IS_A_ON_NET | $sht(dnis=>$fU) \n");
  if ($sht(dnis=>$fu)) {
    setflag(FLT_A_ON_NET);
    return 1;
  } else {
    resetflag(FLT_A_ON_NET);
    return 0;
  }
}

The logic is similar on the B number, with the exception that I am searching for the request user ($rU) value, and that a different flag is set.

route[IS_B_ON_NET] {
  # Tests the B number to determine if this call is to an on-net number
  xlog("L_INFO", "IS_B_ON_NET | $sht(dnis=>$rU) \n");
  if ($sht(dnis=>$rU) != $null) {
    xlog("L_INFO", "IS_B_ON_NET | Trying to set flag \n");
    setflag(FLT_B_ON_NET);
    return 1;
  } else {
    resetflag(FLT_B_ON_NET);
    return 0;
  }
}

In order to determine if a B number is blocked, IS_DENIED will lookup the $rU value in the deny_list htable. If the $rU matches an entry in that field, the DENY_CALL routing block is called which adds an X-header and replies with a 403.

route[IS_DENIED] {
  # Looks up the B number to determine if it is known "bad" destination
  xlog("L_INFO", "IS_DENIED| $sht(deny_list=>$rU) \n");
  if ($sht(deny_list=>$rU)) != $null {
    route(DENY_CALL);
  } else {
    return 0;
  }
}

route[DENY_CALL] {
  # Denies a call because the B number is in the naughty list.
  xlog("L_INFO", "DENY_CALL | $ci $rU \n");
  append_to_reply("X-Deny-Reason: Calls to this destination number are not allowed\n");
  sl_send_reply("403", "Calls Cannot be made to this number");
  exit;
}


Finally, we make it to the point where a 302 is returned. Assuming that the flag FLT_B_ON_NET is set, we add a Contact header and set the host portion of the URI to what was defined as the value in htable. Here’s a peak.

root@522c7d38f9d2:/# kamcmd htable.dump dnis
{
	entry: 12426
	size: 1
	slot: {
		{
			name: 8675309
			value: 172.16.10.50
			type: str
		}
	}
}

I have the number “8675309” (yes.. like the song), listed as a KEY and the VALUE is 172.16.10.50, which is the IP of an internal endpoint. So the Contact header will look like "Contact: sip:8675309@172.16.10.50". You can add additional entries that point to other internal devices. This will allow you to route certain numbers to the platform where they are provisioned, or easily re-route when you need to migrate a number to a different system.

If the B number is not on-net, we will set the Contact as the default destination value. Once the Contact header is added, we send a 302 Redirect back to the “SBC”. Once the “SBC” receives the 302, it is configured to look at the 302, and relay the INVITE to the internal endpoint.

route[REPLY_302] {
  # Sends a 302 redirect back to the proxy that requested the routing lookup
  xlog("L_INFO", "REPLY_302 \n");
  if isflagset(FLT_B_ON_NET) {
    xlog("L_INFO", "REPLY_302 | B IS ON NET $sht(dnis=>$rU) \n");
    append_to_reply("Contact: sip:$rU@$sht(dnis=>$rU)\r\n");
  } else {
    append_to_reply("Contact: sip:$rU@" + 'DEFAULT_DEST' + "\r\n");
  }
  sl_send_reply("302", "Redirect");
  exit;
}

If you are interested in how the “SBC” is handling the 302 redirect, it’s just a failure route.

// This is on the SBC / Edge Proxy. 
failure_route[REDIRECT_NOACC] {
  xlog("L_INFO", "REDIRECT_NOAC | INTERNAL | $ci | $si \n");
	if(!t_check_status("3[0-9][0-9]")) {
		exit;
	}
  xlog("L_INFO", "REDIRECT_NOACC|  \n");
	get_redirects("3:1");
	route(RELAY);
}

After built this topology, I added 100,000 DNIS entries into my database, and then rebooted the 302 redirect servers. This loaded surprisingly fast considering that everything was running on my under powered MacBook Air. Anyways, I hope that this article provided you with some ideas about how Kamailio can be used as a 302 Redirect Server in your projects.

Uncategorized
/

When you are setting up SIP servers, you always need to run scenario tests to verify that messages flow properly from end to end. While there are commercial tools to perform these tests, such as Sigma from NetAxis, many people choose to test with opensource tools. The most common of these tools is SIPp. While SIPp is very powerful, building scenario files can be extremely time-consuming.

Recently, I was frustrated with SIPp and found another great tool for basic scenario tests. This tool doesn’t work for load testing, but general scenario testing, it is amazing. This tool is called bbs or “black box sip“. The one challenge with bbs is that you need to download PJSIP and build it from source on your machine. This too can be a bit of a hassle.

Docker Images Save the Day

… Of course you can save a lot of time and frustration by pulling the docker image that I created. This image has bbs installed and will support testing with TLS. UDP, or TCP.

To obtain the image, you, of course, need docker installed. Once you have docker, run “docker pull voipengineertraining/sip-items:bbs”. This will download the image.

To run the container and use an interactive shell:

docker run -p  5060:5060 -it voipengineertraining/sip-items:bbs 

Once in the shell, you can run:

“bbs -vvvvvv -c <scenario> ” to run a scenario file

You can also build scenarios locally on your machine, and then map a volume to docker:

docker run -p 5060:5060 -v /my_scenario/directory:/bbs -it voipengineertraining/sip-items:bbs  bbs -c /bbs/scenario.yaml

Learn more about Black Box SIP here: https://github.com/irontec/bbs

Uncategorized
/

The telecom industry is changing rapidly.

Ok, maybe not that rapid as telecom is often years behind other IT disciplines, but it’s safe to say that it’s starting to catch up. Because of this, the skills required by voice engineers are also experiencing a change.

I’ve mentioned the need to know a programming language a few times over the past three years, but that’s’ not enough. You need to also understand concepts such as Infrastructure as Code, DevOps, and Scrum/ Kanban. Trust me, these are all slowly sneaking in your world.

The books in the following paragraphs are a great starting point to learn these concepts.

Automate The Boring Stuff With Python

You really can’t go wrong with a book from No Starch Press. I’m a big fan of the style that all of their books follow, and “Automate the Boring Stuff” is a great book to start digging into Python. It goes far beyond the boring “Hello World” scripts and has you start searching for files, and slicing up excel files.

You can find the second edition of this book on Amazon for about $36. Click here to check it out.

Network Programmability and Automation

This book covers a lot of material. I ended up reading it cover to cover during my flight from Minneapolis to Amsterdam when I moved. This goes into programming but also talks about tools such as git, Salt Stack, and NetMiko. If you happen to manage a lot of systems, you really can’t go wrong with this book.

You can find this on Amazon for usually $25ish.

Infrastructure as Code: Managing Servers in The Cloud

While it’s less likely to see a large amount of telephony application servers sitting in the public cloud, this is a book that you should still read. As the title says, it covers Infra as code, which is something that every engineer needs to understand in 2020.

The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

This book doesn’t dig far into the hard/technical skills related to your job. It focuses on concepts such as the organizational culture processes and the tools that are needed to build a successful DevOps organization.

Check this book out on Amazon for about $30.

Kanban: Successful Evolutionary Change For Your Technology Business

While I am a Scrum Master for my department it doesn’t change the fact that I find Kanban a better methodology. Specifically for those working in an operationally focused department.

This book is written by the individual that really introduced the concepts of Kanban to the IT world. It’s a few years old, but still one of the best (if not THE best) book about the topic.

Find it on Amazon for roughly $40.

Uncategorized
/

Those that know me in the real world, know about my love for automation. I enjoy writing code that connects to APIs consumes data, processes it, then sends it off to somewhere else for safekeeping.

I also get a kick out of writing code that can send commands and automate parts of my job. This allows me to do more in less time.

As an engineer, you should be doing the same thing.

Yet, it’s not always that easy. Especially to those in the telecom world. Many of us are still managing systems that are 40 years old (Nortel DMS anyone?) or that lack a REST or SOAP API.

So what’s an engineer to do without an API?

We write code. We automate, and we conquer without it.

Trust me, I’ve been working on far too many systems lately that should be retired, but for some reason, they are still in service. This lead me to think “Well, If I have this problem, maybe someone else does too. As such, I should make a course about automating with Python 3 and Pexpect”.

So that’s what’s I’ve been working on lately. Well, that and a course about Kanban, but that’s for a very different audience!

Want to see a sample video from the Python course? Look below!




Uncategorized
/

About a year ago and came across a posting on Upwork where someone needed an easy installation plan to get SIPp installed on Ubuntu. I was too late to submit for the project, but, I decided to make something anyway.

If you haven’t installed SIPp before, it can be a big hassle to get it up and running. There are a number of dependencies that must be met, but if you watch the video, you will see the link to a github script to help you out.

Cheers! 😉

Acme Packet · SIP · VOIP
/

Symptoms of the Issue:

While setting up an SBC in my lab today I came across an error that I’ve encountered a few times in the past. The error message is “no Redundancy Protocol Process Running” which appears when running a “show health” command.

Command : Show health

Acme packet redundancy troubleshooting

What’s frustrating about this error is that the redundancy config is enabled when you review the redundancy-config section ( show configuration redundancy-config ). 

Command : Show configuration redundancy-config

Troubleshooting:

When troubleshooting anything related to redundancy on the Acme Packet / Oracle SBCs, I’ve found the best approach is to go straight to the logs. Specifically “log.berpd”.

Command : Show logfile log.berpd

acme-packet-solving-redundancy-issues

After reviewing the output I can see that this SBC doesn’t have a hostname that matches what I have built as peer-name within the redundancy config.  There are two locations to create a hostname on the Acme Packet SBCs. One option is to create the name under “config t > system > system-config > hostname”. If you enter the hostname here followed by a save / activate / reboot the error will persist.

 

Resolution

Where the hostname needs to be entered with within the bootparam settings.

troubleshooting-acme-packet-redundancy-issues

  1. Navigate to Enable > Conf t > boot
  2. You will be presented with a line asking for the “Boot File”. Press enter to the skip to the next line.
  3. Keep pressing Enter until you arrive on the Target Name line.
  4. Enter the hostname that you used when building out the redundancy peer-name.
  5. Keep pressing Enter until  you are out of the boot param menu.
  6. Perform a Save / Active.
  7. Reboot the SBC

After rebooting you should be able to run a “show health” and see that redundancy is running.

 

 

VOIP
/

I’ve come across many scenarios where people assume that VOIP calls consume 64kbps per call. They instantly assume that because the payload of a codec such as G.711u is 64k. While the payload IS 64k for a G.711u call there are other factors in play. If you want to properly calculate the VOIP bandwidth of a single call, follow these steps.

1. Determine the bandwidth for the audio codec.

For the sake of simplicity, let’s continue using G.711u as the codec. We do not send one massive 64k packet once per seconds in a call. That would cause some serious issues with audio quality. Instead each packet is going to contain a small amount of audio. The default for most codecs and deployment scenarios is 20ms. This can be adjusted, however it’s limited by the SIP stack and /or DSPs on your system

To find out how big each packet is we do the following equation.

Bytes Per Packet = (Sample Size * Codec Bandwidth) / 8

So we have (.20 * 64,000) /8

This equals 160 bytes.

2. Determine the datalink, network and transport overhead.

Depending on the layer 2 network (datalink) we are going to have different amounts of overhead.

Ethernet = 20 byes
Frame Relay = 6 bytes
PPP = 6 bytes

In terms of the network and transport overhead we can assume that IP, UDP, and RTP will always be used.
IP = 20 bytes
UDP = 8 bytes
RTP = 12 bytes

We will make the assumption that layer 2 on this call is ethernet.

160 + 20 + 40 = 220 bytes

3. Add optional overhead.

Packets often traverse a VPN of some sort. This adds additional bandwidth.

GRE = 24 bytes
MPLS = 4 bytes
IPSEC = 57 bytes

Let’s assume that we have an MPLS connection in place.

220 + 4 = 224 bytes

4. Add it up.

We know that each packet consumes 224 bytes. Remember how we used 20ms as the amount of audio per packet? If we use 20ms, how many packets per second are being sent. If 1 seconds == 1000ms, then 1000 /20 = 50.

We will send 50 packets per second.

224 * 50 = 11200 bytes per second.

We normally don’t look at bandwidth as bytes per second. We need to convert this to bits by simply multiplying by 8.

11200 * 8 = 89,600 bits.

With this scenario, each call will consume almost 90kbps. Had this calculation been made as 64k per call we could quickly run into capacity issues.

Want to run your own calculations? Here are the payload sizes of a few common codecs:

G711 = 64,000
ILBC = 15,200
G729 = 8,000
G.726 = 32,000
G.729a = 8,000
G.728 = 16,000
G.722 = 64,000 (Note – this can also be 56,000 or 48,000 depending on the set bitrate)

Here’s a scenario for you to run through.

Codec :G.729a (8,000)
Sample Size : Each packet has 20 ms of audio
Layer 2 : Ethernet circuit between device A and Device B
Extra Overhead: MPLS and IPSEC are used across the link

Write your answer in the below comment box.

Polycom · Uncategorized · VOIP · VOIP Security
/

Anyone that has worked in telecom long enough has been hit by a SIP scanner at some point in time.

For people in NOC / Support teams, they often have a customer open a ticket where the call goes like this:

Caller : “My phone is randomly ringing and will not stop.”

Tech Support : “Let me look at the logs.”

Tech Support reviews the logs, while the customer enjoys hold music that isn’t even fit for an elevator. 

10 minutes later

Tech Support : “I’m happy to report that we are not sending any calls to your device during that time. Your ticket number is 222444555, have a great day”

A few days later the caller is back on the phone. They are furious as the issue is still happening. The support team doesn’t see the call in the logs, as it didn’t traverse through the VoIP providers core.

Thankfully for those with Polycom phones, you can make a change to the device’s XML file to block unsolicited INVITES.

All you need is this:

< voIpProt.SIP.requestValidation.1.request=”INVITE” voIpProt.SIP.requestValidation.1.method=”source”/>

Once applied, if the phone receives an INVITE from a device other than the IP of reg.1.server.address, it will reject the request with a SIP 400 BAD REQUEST.

Now instead of the user experiencing a phone that rings non-stop, they see and hear nothing.

Bad Guy – Blocked!

 

Research: I tested this with a VVX 500 (5.2 software)  and a SIP testing tool know as SIPP. I sent over 50 calls per second to the device while it was on an active call. During the active call I did not experience any audio issues.

When the call was disconnected, I did experience a phone that acted “slower” than normal to return to the home screen. Going off hook also introduced about a 1 second delay before presenting dial tone.

 

 

 

 

 

 

 

 

Broadworks
/

I had an issue that drove me insane today. While attempting to push a few changes to the Broadworks Call Center App. I was adamant that I had entered the correct values on the “Deployment” tab, yet I kept receiving this error:
(all IPs, domains and usernames have been changed)

 

Timeout occurred waiting for response from server. Please contact your service provider if this problem persists.
Timeout occurred waiting for response from server. Please contact your service provider if this problem persists.

 

This error didn’t seem like it was telling the truth, so I went to the source : The OCSLog file on the server hosting OCI.

 

2015.10.26 16:29:06:340 MST | Info | OpenClientServer

Received message from client on unsecure socket 13143 | Host:Port 1.1.1.1:62253/1.1.1.1 | Local Port 2208

2015.10.26 16:29:06:340 MST | Info | OpenClientServer

Received message from client on unsecure socket 13143 | Host:Port 1.1.1.1:62253/1.1.1.1 | Local Port 2208:


192.168.154.75--4009030666984229966

 adminusername

2015.10.26 16:29:06:348 MST | Info | OpenClientServer

Getting PROV serving AS info from NS for user adminusername.

2015.10.26 16:29:06:348 MST | Info | OpenClientServer

http://10.10.10.71:80/servlet/LocateUser?url=adminusername%40my.default.domain&returnCompatibleXSP=false&callPRequest=false

2015.10.26 16:29:06:348 MST | FieldDebug | Generic

HTTP connected...

2015.10.26 16:29:06:351 MST | Info | OpenClientServer

Getting response from NS (10.10.10.71):


 

2015.10.26 20:51:26:696 MST | Info | OpenClientServer

Sending OCI message to client using connection unsecure socket 13167 | Host:Port 1.1.1.1:55681/1.1.1.1 | Local Port 2208
192.168.16.118--6954723538114549563
 [Error 5401] User is not found on Network Server.

[Error 5401] User is not found on Network Server.

 

Ok… no big deal, I can add the username in question. I logged into the NS and added the user to “NS_CLI/SubscriberMgmt/Administrator/User>”.

NS_CLI/SubscriberMgmt/Administrator/User> get

Administrator ID  =  adminusername@my.default.domain
  Name            =  Name, Admin
  Access Level    =  System Provider

Tried it again, the same error punched me in the face.

I took a wild guess and add the user to ” NS_CLI/SubscriberMgmt/Administrator/HostingNEUser>”

NS_CLI/SubscriberMgmt/Administrator/HostingNEUser> get bwas userID adminusername@my.default.domain
          Administrator ID  Subscriber Maintenance Partition
============================================================
  adminusername@my.default.domain

1 entry found.

SUCCESS! This finally let me in. Hopefully this saves someone else a ridiculous amount of troubleshooting time.