forked from manuelkasper/AS-Stats
-
Notifications
You must be signed in to change notification settings - Fork 0
will-h/AS-Stats
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
AS-Stats v1.41 (2013-03-17) a simple tool to generate per-AS traffic graphs from NetFlow/sFlow records by Manuel Kasper <[email protected]> for Monzoon Networks AG -------------------------------------------------------------------- Changes ------- vx.xx Try harder to determine IP version for sFlow samples. Add IPFIX support (contributed by Daniel Piekacz) Add experimental NetFlow v5 support (contributed by Charlie Allom) Fix handling of 64-bit counters for NetFlow v9 v1.41 Generate v6 RRD DS in contrib/sync_ds.pl too (spotted by Peter Hansen) sFlow: fix creation of new RRDs when multiple entries with the same tag are present in the known links file (spotted by Michal Buchtik) Add startup scripts for FreeBSD (contrib/freebsd) (contributed by Michal Buchtik) Add support for setting the sampling rate per link in the knownlinks file. v1.40 Add support for NetFlow v9 to netflow-asstatd.pl (sponsored by Xplornet Communications Inc.) Add support for IPv6 (for NetFlow v9 and sFlow). Note: existing RRDs need to be upgraded (new data sources added for v6) for this to work. Enable $showv6 in www/config.inc to see separate graphs for IPv6. Add support for 4-byte ASNs (NetFlow v9 and sFlow). Needs testing. Add RRA for 1 month at 4 hour resolution to newly created RRD files. Add links to PeeringDB and robtex (suggested by Steve Glendinning) Fix AS-SET lookup on systems where the whois command returns additional lines. v1.36 Fix creation of new RRDs when multiple entries with the same tag are present in the known links file (spotted by Michel Moriniaux) Add feature to inspect all ASes in an AS-SET (automatic whois lookup). (contributed by Nicolas Debrigode) Updated asinfo.txt with data from whois.cymru.com. v1.35 Allow hyphens in link names. (contributed by Gareth Campling) Smooth I/O burstiness and reduce overall IOPS requirements by flushing only 10% of the cache every 25 seconds (instead of the entire cache at once every 60 seconds). (contributed by James A. T. Rice) v1.34 Fix for NaN detection in rrd-extractstats.pl for 64-bit Perl versions (contributed by Benjamin Schlageter) Skip missing data sources in rrd-extractstats.pl to avoid abort if new data sources are added but the RRDs are not updated. v1.33 Fix for multiple entries with the same tag in the knownlinks file (e.g. for LACP) (contributed by Michal Buchtik) Added sync_ds.pl script to contrib directory, which can synchronize the data sources of RRD files with the tags defined in the knownlinks file (contributed by Michal Buchtik) sflow-asstatd.pl now uses the agent IP instead of the UDP source address - this makes it behave properly when a proxy is being used (contributed by Michel Moriniaux) v1.32 Fix add_ds.sh to support new directory structure (contributed by Sergei Veltistov) Fix PHP warnings and move $ntop to config.inc (contributed by Michal Buchtik) v1.31 Set memory_limit to 256 MB in PHP pages (suggested by Steve Colam <[email protected]>). Allow NetFlow aggregation version 0 as well as 2 to make NetFlow exports from Juniper routers work (suggested by Thomas Mangin <[email protected]>). Updated asinfo.txt with data from whois.cymru.com. v1.3 Changes by Steve Colam <[email protected]>: - ...-asstatd.pl now accepts parameters (UDP listen port, sampling rate etc.) on the command line **************************************************** * Mind the new command line syntax when upgrading! * * Change your start script accordingly... * **************************************************** - hierarchical RRD structure for more efficient storage (one directory per low byte of AS number) ********************************************* * Use tools/migraterrdfiles.pl to move your * * RRD files when upgrading! * ********************************************* - ...-asstatd.pl now re-reads known links file upon SIGHUP Added contrib/generate-asinfo.py script to generate AS list from WHOIS data (contributed by Thomas Mangin <[email protected]>). Moved site-specific parameters of www frontend to config.inc. New flag images from famfamfam.com. Updated asinfo.txt. v1.2 Support for sFlow (through sflow-asstatd.pl); fix for link names with upper-case characters Allow inbound/outbound in graphs to be swapped (via option in www/config.inc) v1.1 Fix for a potential race condition surrounding $childrunning (reported by Yann Gauteron; experienced on a Linux system) v1 Initial release How it works ------------ A Perl script (netflow-asstatd.pl) collects NetFlow v8/v9 AS aggregation records or sFlow v5 samples from one or more routers. It caches them for about a minute (to prevent excessive writes to RRD files), identifies the link that each record refers to (by means of the SNMP in/out interface index), maps it to a corresponding "known link" and RRD data source, and then runs RRDtool. To avoid losing new records while the RRD files are updated, the update task is run in a separate process. For each AS, a separate RRD file is created as needed. It contains two data sources for each link - one for inbound and one for outbound traffic. In generated per-AS traffic graphs, inbound traffic is shown as positive, while outbound traffic is shown as negative values. Another Perl script, rrd-extractstats.pl, is meant to run about once per hour. It sums up per-AS and link traffic during the last 24 hours, sorts the ASes by total traffic (descending) and writes the results to a text file. This is then used to display the "top N AS" and other stats by the provided PHP scripts. Prerequisites ------------- - Perl 5.8 - RRDtool 1.2 (with Perl "RRDs" library) - if using sFlow: the Net::sFlow module (CPAN) - web server with PHP 5 - one or more routers than can generate NetFlow v8/v9 AS aggregation records or sFlow samples Installation ------------ In the instructions below, "xx-asstatd.pl" refers to either netflow-asstatd.pl or sflow-asstatd.pl, depending on whether your routers generate NetFlow or sFlow data. - Copy the perl scripts xx-asstatd.pl and rrd-extractstats.pl to the machine that will collect NetFlow/sFlow records - Create a "known links" file with the following information about each link that you want to appear in your AS stats: - IP address of router (= source IP of NetFlow datagrams) - SNMP interface index of interface (use "show snmp mib ifmib ifindex" to find out) - a short "tag" (12 chars max., alphanumerics only) that will be used internally (e.g. for RRD DS names) - a human-readable description (will appear in the generated graphs) - a color code for the graphs (HTML style, 6 hex digits) See the example file provided (knownlinks) for the format. - Create a directory to hold per-AS RRD files. For each AS, about 128 KB of storage are required, and there could be (in theory) up to 64511 ASes. AS-Stats automatically creates 256 subdirectories in this directory for more efficient storage of RRD files (one directory per lower byte of AS number, in hex). - Start xx-asstatd.pl in the background (or, better yet, write a startup script for your operating system to automatically start xx-asstatd.pl on boot): nohup xx-asstatd.pl -r /path/to/rrd/dir -k /path/to/knownlinks & By default, netflow-asstatd.pl will listen on port 9000 (UDP) for NetFlow datagrams, and sflow-asstatd.pl will listen on port 6343 (UDP) for sFlow datagrams. Use the -p option if you want to change that. If you use sampled NetFlow or sFlow, set the sampling rate with the -s option. sflow-asstatd.pl also needs you to specify your own AS number with the -a option for accurate classification of inbound and outbound traffic. It's a good idea to make sure only UDP datagrams from your trusted routers will reach the machine running xx-asstatd.pl (firewall etc.). - NetFlow only: Have your router(s) send NetFlow v8 or v9 AS aggregation records to your machine. This is typically done with commands like the following (Cisco IOS): ip flow-cache timeout active 5 int Gi0/x.y ip flow ingress ip flow-export source <source interface> ip flow-export version 5 origin-as ip flow-aggregation cache as cache timeout active 5 cache entries 16384 export destination <IP address of server running AS stats> 9000 enabled Adjust the number of cache entries if necessary (i.e. if you get messages like "Netflow as aggregation cache is almost full" in the logs). Note that the version has to be specified as 5, even though the AS aggregation records will actually be v8. Also, setting the global flow cache timeout to 5 minutes is necessary to get "smooth" traffic graphs (default is 30 minutes), as a flow is only counted when it expires from the cache. Decreasing the flow-cache timeout may result in a slight increase in CPU usage (and NetFlow AS aggregation takes its fair share of CPU as well, of course). Routers with MLS (Multi-Layer Switching, e.g. Cisco 7600 series) require additional commands like the following in order to enable NetFlow processing/aggregation for packets processed in hardware: mls aging fast time 4 threshold 2 mls aging long 128 mls aging normal 64 mls flow ip interface-full For IOS XR, the configuration looks as follows: flow exporter-map FEM version v9 ! transport udp 9000 source <source interface> destination <IP address of server running AS stats> vrf default flow monitor-map IPV4-FMM record ipv4 exporter FEM cache entries 16384 cache timeout active 5 ! flow monitor-map IPV6-FMM record ipv6 exporter FEM cache entries 16384 cache timeout active 5 ! sampler-map SM random 1 out-of 10000 router bgp 100 address-family ipv4 unicast bgp attribute-download address-family ipv6 unicast bgp attribute-download For JunOS, the configuration looks as follows: forwarding-options { sampling { input { rate 2048; max-packets-per-second 4096; } family inet { output { flow-active-timeout 60; flow-server x.x.x.x { port 9000; autonomous-system-type origin; aggregation { autonomous-system; } version 8; } } } } } JunOS IPFIX configuration: chassis { tfeb { slot 0 { sampling-instance flow-ipfix; } } } interfaces { ge-1/0/0 { unit 0 { family inet { sampling { input; output; } } } } } forwarding-options { sampling { instance { flow-ipfix { input { rate 1; } family inet { output { flow-server 192.0.2.10 { port 9000; autonomous-system-type origin; no-local-dump; version-ipfix { template { ipv4; } } } inline-jflow { source-address 192.0.2.1; } } } } } } } services { flow-monitoring { version-ipfix { template ipv4 { flow-active-timeout 60; flow-inactive-timeout 60; template-refresh-rate { packets 1000; seconds 10; } option-refresh-rate { packets 1000; seconds 10; } ipv4-template; } } } } - sFlow only: Have your router(s) send sFlow samples to your machine. Your routers may need a software upgrade to make them include AS path information for both inbound and outbound packets (this is a good thing to check if your graphs only show traffic on one direction). - Wait 1-2 minutes. You should then see new RRD files popping up in the directory that you defined/created earlier on. If not, make sure that xx-asstatd.pl is running, not spewing out any error messages, and that the NetFlow/sFlow datagrams are actually reaching your machine (tcpdump...). - Add a cronjob to run the following command every hour: rrd-extractstats.pl /path/to/rrd/dir /path/to/knownlinks \ /path/to/asstats_day.txt That script will go through all RRD files and collect per-link summary stats for each AS, sort them by total traffic (descending), and write them to a text file. The "top N AS" page uses this to determine which ASes to show. - Copy the contents of the "www" directory to somewhere within your web server's document root and change file paths in config.inc as necessary. - Make the directory "asset" within www writable by the web server (this is used to cache AS-SETs and avoid having to query whois for every request). - Wait a few hours for data to accumulate. :) - Access the provided PHP scripts via your web server and marvel at the (hopefully) beautiful graphs. Adding a new link ----------------- Adding a new link involves adding two new data sources to all RRD files. This is a bit of a PITA since RRDtool itself doesn't provide a command to do that. A simple (but slow) Perl script that is meant to be used with RRDtool's XML dump/restore feature is provided (add_ds_proc.pl, add_ds.sh). Note that netflow-asstatd.pl should be stopped while modifying RRD files, to avoid breaking them with concurrent modifications. Changing the RRAs ----------------- By default, the created RRDs keep data as follows: * 48 hours at 5 minute resolution * 1 week at 1 hour resolution * 1 month at 4 hour resolution * 1 year at 1 day resolution If you want to change that, modify the getrrdfile() function in xx-asstatd.pl and delete any old RRD files. To do ----- - rrd-extractstats.pl uses a lot of memory and could probably use some optimization.
About
A simple tool to generate per-AS traffic graphs from NetFlow/sFlow records
Resources
Stars
Watchers
Forks
Packages 0
No packages published
Languages
- Perl 53.0%
- PHP 30.1%
- Shell 12.9%
- CSS 2.9%
- Python 1.1%