Monitoring IPv6 vs IPv4 traffic on Ubiquiti USG with SNMP

So. This has been painful. I decided I wanted to view how much traffic is flowing in and out of my USG now that I have IPv6!!!! It must be crazy, so let me go look. And then the painful truth, the USG controller has no visuals or commands to find the information. Bummer.

After a lot of web browsing, researching, additional hair loss, crying, and help, I have found the solution. I am going to turn on SNMP on the Ubiquiti USG, install telegraf, influxdb, and grafana on my Ubuntu 18.04 server.

This should be easy, right? No.
What could go wrong? A whole whole lot.

Setting up SNMP

I’m going to skip most the heartburn I had of trying to find the correct OIDs, getting the names correct, and ultimately realizing that Ubuntu does not load the MIBs in by default, so you have to do it by hand. Here’s what I did though:

# apt-get install snmp-mibs-downloader libsnmp-base libsnmp-dev snmp
# cd /usr/share/snmp/mibs
# wget http://dl.ubnt-ut.com/snmp/UBNT-MIB
# wget http://dl.ubnt-ut.com/snmp/UBNT-UniFi-MIB

Now, I did something bad, but since this is pretty limited, I loaded all of the MIBs in by default. My /etc/snmp/snmp.conf file looks like this:

# As the snmp packages come without MIB files due to license reasons, loading
# of MIBs is disabled by default. If you added the MIBs you can reenable
# loading them by commenting out the following line.
# mibs :
mibs +ALL

Setting up InfluxDB

# apt-get install influxdb
# influx
>create database telegraf
>quit

Setting up Telegraf

You will need to go to the InfluxData website and download the deb from there, as it’s not part of the repository. Following their directions to install it.

Now for configurations, this part was challenging and where a lot of time was spent looking for the values in the MIBs to chart the data I was looking for. At the end of the day, these are the OIDs that I homed in on:

# snmpwalk -OT -v 2c -c "public" 192.168.1.1 1.3.6.1.2.1.4

IP-MIB::ipSystemStatsHCInReceives.ipv4 = Counter64: 98316983
IP-MIB::ipSystemStatsHCInReceives.ipv6 = Counter64: 2642391

IP-MIB::ipSystemStatsHCOutTransmits.ipv4 = Counter64: 150775782
IP-MIB::ipSystemStatsHCOutTransmits.ipv6 = Counter64: 2041111

Armed with that, here is my configuration files that I ended up using:

/etc/telegraf/telegraf.d/10-influxdb-output.conf

[[outputs.influxdb]]
urls = ["http://127.0.0.1:8086"]
database = "telegraf"
retention_policy = ""
write_consistency = "any"
timeout = "5s"
username = "admin"
password = "admin"
user_agent = "telegraf"

/etc/telegraf/telegraf.d/20-inputs-snmp-usg.conf

[[inputs.snmp]]
agents = [ "192.168.1.1" ]
interval = "60s"
timeout = "10s"
retries = 3
version = 2
community = "public"
max_repetitions = 10
name = "snmp.USG"

# Get some system information
[[inputs.snmp.field]]
name = "sysUpTime"
oid = "RFC1213-MIB::sysUpTime.0"
# Load averages
[[inputs.snmp.field]]
name = "loadAverage1"
oid = "UCD-SNMP-MIB::laLoad.1"
[[inputs.snmp.field]]
name = "loadAverage5"
oid = "UCD-SNMP-MIB::laLoad.2"
[[inputs.snmp.field]]
name = "loadAverage15"
oid = "UCD-SNMP-MIB::laLoad.3"


##
## CPU utilization
##
# Number of 'ticks' spent on user-level
[[inputs.snmp.field]]
name = "ssCpuRawUser"
oid = "UCD-SNMP-MIB::ssCpuRawUser.0"
# Number of 'ticks' spent on reduced-priority
[[inputs.snmp.field]]
name = "ssCpuRawNice"
oid = "UCD-SNMP-MIB::ssCpuRawNice.0"
# Number of 'ticks' spent on system-level
[[inputs.snmp.field]]
name = "ssCpuRawSystem"
oid = "UCD-SNMP-MIB::ssCpuRawSystem.0"
# Number of 'ticks' spent idle
[[inputs.snmp.field]]
name = "ssCpuRawIdle"
oid = "UCD-SNMP-MIB::ssCpuRawIdle.0"
# Number of 'ticks' spent waiting on I/O
[[inputs.snmp.field]]
name = "ssCpuRawWait"
oid = "UCD-SNMP-MIB::ssCpuRawWait.0"
# Number of 'ticks' spent in kernel
[[inputs.snmp.field]]
name = "ssCpuRawKernel"
oid = "UCD-SNMP-MIB::ssCpuRawKernel.0"
# Number of 'ticks' spent on hardware interrupts
[[inputs.snmp.field]]
name = "ssCpuRawInterrupt"
oid = "UCD-SNMP-MIB::ssCpuRawInterrupt.0"
# Number of 'ticks' spent on software interrupts
[[inputs.snmp.field]]
name = "ssCpuRawSoftIRQ"
oid = "UCD-SNMP-MIB::ssCpuRawSoftIRQ.0"

#
## Interface Details & Metrics
##
# Per-interface traffic, errors, drops
[[inputs.snmp.table]]
oid = "IF-MIB::ifTable"
[[inputs.snmp.table.field]]
is_tag = true
oid = "IF-MIB::ifName"
# Per-interface high-capacity (HC) counters
[[inputs.snmp.table]]
oid = "IF-MIB::ifXTable"
[[inputs.snmp.table.field]]
is_tag = true
oid = "IF-MIB::ifName"

## IPv4 Metrics
[[inputs.snmp.field]]
name = "ipSystemStatsHCInReceives.ipv4"
oid = "IP-MIB::ipSystemStatsHCInReceives.ipv4"
[[inputs.snmp.field]]
name = "ipSystemStatsHCInReceives.ipv6"
oid = "IP-MIB::ipSystemStatsHCInReceives.ipv6"

## IPv6 Metrics
[[inputs.snmp.field]]
name = "ipSystemStatsHCOutTransmits.ipv4"
oid = "IP-MIB::ipSystemStatsHCOutTransmits.ipv4"
[[inputs.snmp.field]]
name = "ipSystemStatsHCOutTransmits.ipv6"
oid = "IP-MIB::ipSystemStatsHCOutTransmits.ipv6"

Now that it’s done, I like to test to make sure everything is working correctly.

# /usr/bin/telegraf --test -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d
2019-03-21T17:23:51Z I! Starting Telegraf 1.10.0
> snmp.USG,agent_host=192.168.1.1,host=calvin ipSystemStatsHCInReceives.ipv4=98359144i,ipSystemStatsHCInReceives.ipv6=2651618i,ipSystemStatsHCOutTransmits.ipv4=150812783i,ipSystemStatsHCOutTransmits.ipv6=2048048i,loadAverage1="0.11",loadAverage15="0.15",loadAverage5="0.15",sysUpTime=11436i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=2,ifName=eth0 ifAdminStatus=1i,ifDescr="eth0",ifInDiscards=58270i,ifInErrors=0i,ifInNUcastPkts=62472i,ifInOctets=49122117i,ifInUcastPkts=1383653866i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=1528176458i,ifOutQLen=0i,ifOutUcastPkts=960392178i,ifPhysAddress="12:34:56:78:9a:8a",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=3,ifName=eth1 ifAdminStatus=1i,ifDescr="eth1",ifInDiscards=848i,ifInErrors=0i,ifInNUcastPkts=21384088i,ifInOctets=2332558116i,ifInUcastPkts=980856782i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=1302311535i,ifOutQLen=0i,ifOutUcastPkts=1412274665i,ifPhysAddress="12:34:56:78:9a:8b",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=4,ifName=eth2 ifAdminStatus=2i,ifDescr="eth2",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=0i,ifInOctets=0i,ifInUcastPkts=0i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=2i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=0i,ifOutQLen=0i,ifOutUcastPkts=0i,ifPhysAddress="12:34:56:78:9a:8c",ifSpecific=".0.0",ifSpeed=10000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=5,ifName=imq0 ifAdminStatus=1i,ifDescr="imq0",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=0i,ifInOctets=0i,ifInUcastPkts=0i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=16000i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=0i,ifOutQLen=0i,ifOutUcastPkts=0i,ifSpecific=".0.0",ifSpeed=0i,ifType=1i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=6,ifName=eth1.3 ifAdminStatus=1i,ifDescr="eth1.3",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=586i,ifInOctets=2292449640i,ifInUcastPkts=9799282i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=1994451227i,ifOutQLen=0i,ifOutUcastPkts=8321717i,ifPhysAddress="12:34:56:78:9a:8b",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=7,ifName=eth1.2 ifAdminStatus=1i,ifDescr="eth1.2",ifInDiscards=31i,ifInErrors=0i,ifInNUcastPkts=16300673i,ifInOctets=4024960817i,ifInUcastPkts=40494388i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=2101662885i,ifOutQLen=0i,ifOutUcastPkts=42088420i,ifPhysAddress="12:34:56:78:9a:8b",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=8,ifName=eth1.4 ifAdminStatus=1i,ifDescr="eth1.4",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=0i,ifInOctets=0i,ifInUcastPkts=0i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=1572754i,ifOutQLen=0i,ifOutUcastPkts=13703i,ifPhysAddress="12:34:56:78:9a:8b",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=1,ifName=lo ifAdminStatus=1i,ifDescr="lo",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=0i,ifInOctets=280913i,ifInUcastPkts=3359i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=65536i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=280913i,ifOutQLen=0i,ifOutUcastPkts=3359i,ifSpecific=".0.0",ifSpeed=10000000i,ifType=24i 1553189047000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=imq0 ifAlias="imq0",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInOctets=0i,ifHCInUcastPkts=0i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=0i,ifHCOutUcastPkts=0i,ifHighSpeed=0i,ifInBroadcastPkts=0i,ifInMulticastPkts=0i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth1.3 ifAlias="eth1.3@eth1",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=586i,ifHCInOctets=2292449640i,ifHCInUcastPkts=9799282i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=1994451227i,ifHCOutUcastPkts=8321717i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=586i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth1.2 ifAlias="eth1.2@eth1",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=16300673i,ifHCInOctets=8319928113i,ifHCInUcastPkts=40494388i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=10691597477i,ifHCOutUcastPkts=42088420i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=16300673i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth1.4 ifAlias="eth1.4@eth1",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInOctets=0i,ifHCInUcastPkts=0i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=1572754i,ifHCOutUcastPkts=13703i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=0i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=lo ifAlias="lo",ifConnectorPresent=2i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInOctets=280913i,ifHCInUcastPkts=3359i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=280913i,ifHCOutUcastPkts=3359i,ifHighSpeed=10i,ifInBroadcastPkts=0i,ifInMulticastPkts=0i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth0 ifAlias="WAN",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=62472i,ifHCInOctets=1752395778885i,ifHCInUcastPkts=1383653866i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=723082682186i,ifHCOutUcastPkts=960392178i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=62472i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth1 ifAlias="LAN",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=21384088i,ifHCInOctets=736771965732i,ifHCInUcastPkts=980856782i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=1766533870191i,ifHCOutUcastPkts=1412274665i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=21384088i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth2 ifAlias="eth2",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInOctets=0i,ifHCInUcastPkts=0i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=0i,ifHCOutUcastPkts=0i,ifHighSpeed=10i,ifInBroadcastPkts=0i,ifInMulticastPkts=0i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000

Now start it up and let’s move on to making things pretty.

# service telegraf start

Setting up Grafana

Go to the grafana website and following the download and installation guide for Debian / Ubuntu.

The graph that I’m really interested is the percentage of traffic that is IPv6 each minute. I built two graphs based on this information.

Percentage of traffic coming in each minute that is IPv6:

SELECT derivative(mean("ipSystemStatsHCInReceives.ipv6"), 1m) / (derivative(mean("ipSystemStatsHCInReceives.ipv4"), 1m) + derivative(mean("ipSystemStatsHCInReceives.ipv6"), 1m)) FROM "snmp.USG" WHERE $timeFilter GROUP BY time($__interval) fill(null)

Percentage of traffic going out each minute that is IPv6:

SELECT derivative(mean("ipSystemStatsHCOutTransmits.ipv6"), 1m) / (derivative(mean("ipSystemStatsHCOutTransmits.ipv4"), 1m) + derivative(mean("ipSystemStatsHCOutTransmits.ipv6"), 1m)) FROM "snmp.USG" WHERE $timeFilter GROUP BY time($__interval) fill(null)

Eventually, I’ll do some more research so that I can create graphs that will show the throughput each minute. Here’s a look at where I am at this point:

IPv6 Grafana Dashboard

Final Thoughts

Security! If you are going to be doing this at the enterprise level, please add additional security items that I have completely skipped over. This was my build system, in my home, completely firewalled off. If you do it for something outside, please follow standard security practices such as:

  • Change the community string (or even use snmp v3)
  • Only load the MIBs you need
  • User manage and change the admin password on influxdb
  • Lock down grafana
  • Probably more, leave thoughts in the comments if you think of others

It has been a decade since I’ve touched or thought about SNMP. After doing this, it brought back the nightmares of trying to get translations working and snmp issues on linux in general. The good news now is, I can get a better understand of how my traffic shifts over time to IPv6 as more sites start using IPv6. Almost all my devices in the home are now dual stack, even my printer. I think they only things that are not are the TVs, Rokus, one printer, the Yamaha receiver, and the switches maybe. Even the Brother laser printer has IPv6 and is enabled.

As of this writing, 20% of my traffic is IPv6 after two days. Seeing that the kids have been grounded from electronics that I know use IPv6 (Netflix, YouTube, etc) for two weeks, I only expect that to increase over the next month.

References

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Trackbacks and Pingbacks: