So. This has been painful. I decided I wanted to view how much traffic is flowing in and out of my USG now that I have IPv6!!!! It must be crazy, so let me go look. And then the painful truth, the USG controller has no visuals or commands to find the information. Bummer.
After a lot of web browsing, researching, additional hair loss, crying, and help, I have found the solution. I am going to turn on SNMP on the Ubiquiti USG, install telegraf, influxdb, and grafana on my Ubuntu 18.04 server.
This should be easy, right? No.
What could go wrong? A whole whole lot.
Setting up SNMP
I’m going to skip most the heartburn I had of trying to find the correct OIDs, getting the names correct, and ultimately realizing that Ubuntu does not load the MIBs in by default, so you have to do it by hand. Here’s what I did though:
# apt-get install snmp-mibs-downloader libsnmp-base libsnmp-dev snmp
# cd /usr/share/snmp/mibs
# wget http://dl.ubnt-ut.com/snmp/UBNT-MIB
# wget http://dl.ubnt-ut.com/snmp/UBNT-UniFi-MIB
Now, I did something bad, but since this is pretty limited, I loaded all of the MIBs in by default. My /etc/snmp/snmp.conf file looks like this:
# As the snmp packages come without MIB files due to license reasons, loading
# of MIBs is disabled by default. If you added the MIBs you can reenable
# loading them by commenting out the following line.
# mibs :
mibs +ALL
Setting up InfluxDB
# apt-get install influxdb
# influx
>create database telegraf
>quit
Setting up Telegraf
You will need to go to the InfluxData website and download the deb from there, as it’s not part of the repository. Following their directions to install it.
Now for configurations, this part was challenging and where a lot of time was spent looking for the values in the MIBs to chart the data I was looking for. At the end of the day, these are the OIDs that I homed in on:
# snmpwalk -OT -v 2c -c "public" 192.168.1.1 ifName
IF-MIB::ifName.1 = STRING: lo
IF-MIB::ifName.2 = STRING: eth0
IF-MIB::ifName.3 = STRING: eth1
IF-MIB::ifName.4 = STRING: eth2
IF-MIB::ifName.5 = STRING: imq0
IF-MIB::ifName.6 = STRING: eth1.3
IF-MIB::ifName.7 = STRING: eth1.2
IF-MIB::ifName.8 = STRING: eth1.4
IF-MIB::ifName.10 = STRING: eth1.10
# snmpwalk -OT -v 2c -c "public" 192.168.1.1 1.3.6.1.2.1.4
IP-MIB::ipSystemStatsHCInReceives.ipv4 = Counter64: 98316983
IP-MIB::ipSystemStatsHCInReceives.ipv6 = Counter64: 2642391
IP-MIB::ipSystemStatsHCOutTransmits.ipv4 = Counter64: 150775782
IP-MIB::ipSystemStatsHCOutTransmits.ipv6 = Counter64: 2041111
IP-MIB::ipIfStatsInReceives.ipv6.6 = Counter32: 4429
IP-MIB::ipIfStatsInReceives.ipv6.7 = Counter32: 2919708
IP-MIB::ipIfStatsInReceives.ipv6.8 = Counter32: 4322
IP-MIB::ipIfStatsHCInReceives.ipv6.6 = Counter64: 4429
IP-MIB::ipIfStatsHCInReceives.ipv6.7 = Counter64: 2919708
IP-MIB::ipIfStatsHCInReceives.ipv6.8 = Counter64: 4322
IP-MIB::ipIfStatsInOctets.ipv6.6 = Counter32: 482728
IP-MIB::ipIfStatsInOctets.ipv6.7 = Counter32: 610174314
IP-MIB::ipIfStatsInOctets.ipv6.8 = Counter32: 474384
IP-MIB::ipIfStatsHCInOctets.ipv6.6 = Counter64: 482728
IP-MIB::ipIfStatsHCInOctets.ipv6.7 = Counter64: 610174314
IP-MIB::ipIfStatsHCInOctets.ipv6.8 = Counter64: 474384
With this information, I was able to determine the traffic for the USG, along with the traffic statics for VLAN 2 (eth1.2 / .7), 3 (eth1.3/.6), and 4 (eth1.4/.8).
Here is my configuration files that I ended up using:
/etc/telegraf/telegraf.d/10-influxdb-output.conf
[[outputs.influxdb]]
urls = ["http://127.0.0.1:8086"]
database = "telegraf"
retention_policy = ""
write_consistency = "any"
timeout = "5s"
username = "admin"
password = "admin"
user_agent = "telegraf"
/etc/telegraf/telegraf.d/20-inputs-snmp-usg.conf
[[inputs.snmp]]
agents = [ "192.168.1.1" ]
interval = "60s"
timeout = "10s"
retries = 3
version = 2
community = "public"
max_repetitions = 10
name = "snmp.USG"
# Get some system information
[[inputs.snmp.field]]
name = "sysUpTime"
oid = "RFC1213-MIB::sysUpTime.0"
# Load averages
[[inputs.snmp.field]]
name = "loadAverage1"
oid = "UCD-SNMP-MIB::laLoad.1"
[[inputs.snmp.field]]
name = "loadAverage5"
oid = "UCD-SNMP-MIB::laLoad.2"
[[inputs.snmp.field]]
name = "loadAverage15"
oid = "UCD-SNMP-MIB::laLoad.3"
##
## CPU utilization
##
# Number of 'ticks' spent on user-level
[[inputs.snmp.field]]
name = "ssCpuRawUser"
oid = "UCD-SNMP-MIB::ssCpuRawUser.0"
# Number of 'ticks' spent on reduced-priority
[[inputs.snmp.field]]
name = "ssCpuRawNice"
oid = "UCD-SNMP-MIB::ssCpuRawNice.0"
# Number of 'ticks' spent on system-level
[[inputs.snmp.field]]
name = "ssCpuRawSystem"
oid = "UCD-SNMP-MIB::ssCpuRawSystem.0"
# Number of 'ticks' spent idle
[[inputs.snmp.field]]
name = "ssCpuRawIdle"
oid = "UCD-SNMP-MIB::ssCpuRawIdle.0"
# Number of 'ticks' spent waiting on I/O
[[inputs.snmp.field]]
name = "ssCpuRawWait"
oid = "UCD-SNMP-MIB::ssCpuRawWait.0"
# Number of 'ticks' spent in kernel
[[inputs.snmp.field]]
name = "ssCpuRawKernel"
oid = "UCD-SNMP-MIB::ssCpuRawKernel.0"
# Number of 'ticks' spent on hardware interrupts
[[inputs.snmp.field]]
name = "ssCpuRawInterrupt"
oid = "UCD-SNMP-MIB::ssCpuRawInterrupt.0"
# Number of 'ticks' spent on software interrupts
[[inputs.snmp.field]]
name = "ssCpuRawSoftIRQ"
oid = "UCD-SNMP-MIB::ssCpuRawSoftIRQ.0"
#
## Interface Details & Metrics
##
# Per-interface traffic, errors, drops
[[inputs.snmp.table]]
oid = "IF-MIB::ifTable"
[[inputs.snmp.table.field]]
is_tag = true
oid = "IF-MIB::ifName"
# Per-interface high-capacity (HC) counters
[[inputs.snmp.table]]
oid = "IF-MIB::ifXTable"
[[inputs.snmp.table.field]]
is_tag = true
oid = "IF-MIB::ifName"
## IPv4 Metrics
[[inputs.snmp.field]]
name = "ipSystemStatsHCInReceives.ipv4"
oid = "IP-MIB::ipSystemStatsHCInReceives.ipv4"
[[inputs.snmp.field]]
name = "ipSystemStatsHCInReceives.ipv6"
oid = "IP-MIB::ipSystemStatsHCInReceives.ipv6"
## IPv6 Metrics
[[inputs.snmp.field]]
name = "ipSystemStatsHCOutTransmits.ipv4"
oid = "IP-MIB::ipSystemStatsHCOutTransmits.ipv4"
[[inputs.snmp.field]]
name = "ipSystemStatsHCOutTransmits.ipv6"
oid = "IP-MIB::ipSystemStatsHCOutTransmits.ipv6"
Now that it’s done, I like to test to make sure everything is working correctly.
# /usr/bin/telegraf --test -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d
2019-03-21T17:23:51Z I! Starting Telegraf 1.10.0
> snmp.USG,agent_host=192.168.1.1,host=calvin ipSystemStatsHCInReceives.ipv4=98359144i,ipSystemStatsHCInReceives.ipv6=2651618i,ipSystemStatsHCOutTransmits.ipv4=150812783i,ipSystemStatsHCOutTransmits.ipv6=2048048i,loadAverage1="0.11",loadAverage15="0.15",loadAverage5="0.15",sysUpTime=11436i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=2,ifName=eth0 ifAdminStatus=1i,ifDescr="eth0",ifInDiscards=58270i,ifInErrors=0i,ifInNUcastPkts=62472i,ifInOctets=49122117i,ifInUcastPkts=1383653866i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=1528176458i,ifOutQLen=0i,ifOutUcastPkts=960392178i,ifPhysAddress="12:34:56:78:9a:8a",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=3,ifName=eth1 ifAdminStatus=1i,ifDescr="eth1",ifInDiscards=848i,ifInErrors=0i,ifInNUcastPkts=21384088i,ifInOctets=2332558116i,ifInUcastPkts=980856782i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=1302311535i,ifOutQLen=0i,ifOutUcastPkts=1412274665i,ifPhysAddress="12:34:56:78:9a:8b",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=4,ifName=eth2 ifAdminStatus=2i,ifDescr="eth2",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=0i,ifInOctets=0i,ifInUcastPkts=0i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=2i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=0i,ifOutQLen=0i,ifOutUcastPkts=0i,ifPhysAddress="12:34:56:78:9a:8c",ifSpecific=".0.0",ifSpeed=10000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=5,ifName=imq0 ifAdminStatus=1i,ifDescr="imq0",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=0i,ifInOctets=0i,ifInUcastPkts=0i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=16000i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=0i,ifOutQLen=0i,ifOutUcastPkts=0i,ifSpecific=".0.0",ifSpeed=0i,ifType=1i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=6,ifName=eth1.3 ifAdminStatus=1i,ifDescr="eth1.3",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=586i,ifInOctets=2292449640i,ifInUcastPkts=9799282i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=1994451227i,ifOutQLen=0i,ifOutUcastPkts=8321717i,ifPhysAddress="12:34:56:78:9a:8b",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=7,ifName=eth1.2 ifAdminStatus=1i,ifDescr="eth1.2",ifInDiscards=31i,ifInErrors=0i,ifInNUcastPkts=16300673i,ifInOctets=4024960817i,ifInUcastPkts=40494388i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=2101662885i,ifOutQLen=0i,ifOutUcastPkts=42088420i,ifPhysAddress="12:34:56:78:9a:8b",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=8,ifName=eth1.4 ifAdminStatus=1i,ifDescr="eth1.4",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=0i,ifInOctets=0i,ifInUcastPkts=0i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=1500i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=1572754i,ifOutQLen=0i,ifOutUcastPkts=13703i,ifPhysAddress="12:34:56:78:9a:8b",ifSpecific=".0.0",ifSpeed=1000000000i,ifType=6i 1553189047000000000
> ifTable,agent_host=192.168.1.1,host=calvin,ifIndex=1,ifName=lo ifAdminStatus=1i,ifDescr="lo",ifInDiscards=0i,ifInErrors=0i,ifInNUcastPkts=0i,ifInOctets=280913i,ifInUcastPkts=3359i,ifInUnknownProtos=0i,ifLastChange=0i,ifMtu=65536i,ifOperStatus=1i,ifOutDiscards=0i,ifOutErrors=0i,ifOutNUcastPkts=0i,ifOutOctets=280913i,ifOutQLen=0i,ifOutUcastPkts=3359i,ifSpecific=".0.0",ifSpeed=10000000i,ifType=24i 1553189047000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=imq0 ifAlias="imq0",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInOctets=0i,ifHCInUcastPkts=0i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=0i,ifHCOutUcastPkts=0i,ifHighSpeed=0i,ifInBroadcastPkts=0i,ifInMulticastPkts=0i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth1.3 ifAlias="eth1.3@eth1",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=586i,ifHCInOctets=2292449640i,ifHCInUcastPkts=9799282i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=1994451227i,ifHCOutUcastPkts=8321717i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=586i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth1.2 ifAlias="eth1.2@eth1",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=16300673i,ifHCInOctets=8319928113i,ifHCInUcastPkts=40494388i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=10691597477i,ifHCOutUcastPkts=42088420i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=16300673i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth1.4 ifAlias="eth1.4@eth1",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInOctets=0i,ifHCInUcastPkts=0i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=1572754i,ifHCOutUcastPkts=13703i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=0i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=lo ifAlias="lo",ifConnectorPresent=2i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInOctets=280913i,ifHCInUcastPkts=3359i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=280913i,ifHCOutUcastPkts=3359i,ifHighSpeed=10i,ifInBroadcastPkts=0i,ifInMulticastPkts=0i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth0 ifAlias="WAN",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=62472i,ifHCInOctets=1752395778885i,ifHCInUcastPkts=1383653866i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=723082682186i,ifHCOutUcastPkts=960392178i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=62472i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth1 ifAlias="LAN",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=21384088i,ifHCInOctets=736771965732i,ifHCInUcastPkts=980856782i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=1766533870191i,ifHCOutUcastPkts=1412274665i,ifHighSpeed=1000i,ifInBroadcastPkts=0i,ifInMulticastPkts=21384088i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
> ifXTable,agent_host=192.168.1.1,host=calvin,ifName=eth2 ifAlias="eth2",ifConnectorPresent=1i,ifCounterDiscontinuityTime=0i,ifHCInBroadcastPkts=0i,ifHCInMulticastPkts=0i,ifHCInOctets=0i,ifHCInUcastPkts=0i,ifHCOutBroadcastPkts=0i,ifHCOutMulticastPkts=0i,ifHCOutOctets=0i,ifHCOutUcastPkts=0i,ifHighSpeed=10i,ifInBroadcastPkts=0i,ifInMulticastPkts=0i,ifOutBroadcastPkts=0i,ifOutMulticastPkts=0i,ifPromiscuousMode=2i 1553189048000000000
Now start it up and let’s move on to making things pretty.
# service telegraf start
Setting up Grafana
Go to the grafana website and following the download and installation guide for Debian / Ubuntu.
The graph that I’m really interested is the percentage of traffic that is IPv6 each minute. I built two graphs based on this information.
Percentage of traffic coming in each minute that is IPv6:
SELECT derivative(mean("ipSystemStatsHCInReceives.ipv6"), 1m) / (derivative(mean("ipSystemStatsHCInReceives.ipv4"), 1m) + derivative(mean("ipSystemStatsHCInReceives.ipv6"), 1m)) FROM "snmp.USG" WHERE $timeFilter GROUP BY time($__interval) fill(null)
Percentage of traffic going out each minute that is IPv6:
SELECT derivative(mean("ipSystemStatsHCOutTransmits.ipv6"), 1m) / (derivative(mean("ipSystemStatsHCOutTransmits.ipv4"), 1m) + derivative(mean("ipSystemStatsHCOutTransmits.ipv6"), 1m)) FROM "snmp.USG" WHERE $timeFilter GROUP BY time($__interval) fill(null)
Eventually, I’ll do some more research so that I can create graphs that will show the throughput each minute. Here’s a look at where I am at this point:
Final Thoughts
Security! If you are going to be doing this at the enterprise level, please add additional security items that I have completely skipped over. This was my build system, in my home, completely firewalled off. If you do it for something outside, please follow standard security practices such as:
- Change the community string (or even use snmp v3)
- Only load the MIBs you need
- User manage and change the admin password on influxdb
- Lock down grafana
- Probably more, leave thoughts in the comments if you think of others
It has been a decade since I’ve touched or thought about SNMP. After doing this, it brought back the nightmares of trying to get translations working and snmp issues on linux in general. The good news now is, I can get a better understand of how my traffic shifts over time to IPv6 as more sites start using IPv6. Almost all my devices in the home are now dual stack, even my printer. I think they only things that are not are the TVs, Rokus, one printer, the Yamaha receiver, and the switches maybe. Even the Brother laser printer has IPv6 and is enabled.
As of this writing, 20% of my traffic is IPv6 after two days. Seeing that the kids have been grounded from electronics that I know use IPv6 (Netflix, YouTube, etc) for two weeks, I only expect that to increase over the next month.
References
- Net-SNMP page for IP MIB: http://www.net-snmp.org/docs/mibs/ip.html
- Forum post talking about the MIBs for Ubiquiti: https://community.ubnt.com/t5/UniFi-Wireless/MIBs/td-p/1905619
- Dump of the interesting values found with a walk (Found way after I spent way too much time doing the same): https://gist.github.com/nbrownus/b6a5b1e16256f5ba035b5c0dcbae7532
- Installing common SNMP MIBs on Ubuntu: https://medium.com/@CameronSparr/downloading-installing-common-snmp-mibs-on-ubuntu-af5d02f85425
- Forum thread of someone doing the same setup on the UniFi APs: https://community.ubnt.com/t5/UniFi-Wireless/Grafana-dashboard-for-UniFi-APs-now-available/m-p/1833532/highlight/true#M210601
- Grafana: http://docs.grafana.org/installation/debian/
hello James
very cool post …as an IPv6 fan …and recent user of grafana …I m trying to the same thing in my home where I have several dual-stacked router and switches …currently my grafana is working well with IPv4 but not with IPv6 . I have activated IPv4 and IPv6 addresses in telegraf …add your ipv4 and IPv6 metrics lines in my telegraf config file but got no ipv6 info and neither percentage of traffic in my dashboard …Can you help me to the same as you ? many thanks