Reverse DNS IPv4 scan

July 30th, 2010: I'M DONE!!! After nearly 5 months, over 3.8 billion records scanned and recorded, several hundred gigabytes of data generated (Its 78GB compressed) I'm finally done. Nobody can ever tell me I didn't finish anything. Its going to take me a few weeks to organize the data into something useful. Loading the data into a database alone could take a day or more.

It won't be long before IPv4 addresses are abandoned on the general Internet in favor of IPv6. However when that happens, it will be impossible for anybody even large entities with a huge Internet presence to scan it. Even if you scanned IPv6 at a rate of 4,294,967,296 IPs a second (total number of IPs in IPv4), IPv6 is so big that it would take 200 billion times longer than the age of the universe to scan it all. So I thought that now (2010) is a good time to take a historic snapshot of all of reverse and possibly forward DNS before its too late.

x.x.x.1/8<

Instead of scanning records sequentially and in order to hit the widest part of the Internet possible in a single scan, I reversed the sequence of the scan so that the class A part of the dotted quad was scanned first, then incremented the class B part, then Class C. So I basically hit x.x.x.1 on all class C subnets like this:

1.1.1.1, 2.1.1.1, ..., 1.54.3.1, 2.54.3.1, ..., 223.254.254.1

This eased the load put on name servers as well as helped prevented being detected by firewall analyzers or affected by network honey pots. Most of the hosts using the first ip in their class C subnet are most likely gateway routers, but some may put their gateway elsewhere.

Start date/time of scan: 2010-03-13 03:01:46 EST (unix epoch: 1268467306)
Elapsed time to scan: 105.3 hours
Data generated: 240 MB
Subnets scanned: 10.3 million
Unreachable DNS servers: 239,556
Records with no reverse record: 6,942,667 (67.4%)
Records with a result: 3,140,337
Total number of records: 3,169,771 (including multiple results per IP)
Total unique top level domans in records: 2,225 (There are of course only about 300 legitimate gTLDs)
Number of IPs with multiple reverse records: 21,029
Average length of string including IP and its reverse record: 24 characters
Average length of returned reverse record: 33 characters (not including IP)

Rest of scan

April 15th, 2010. I'm still running the scan and now have 3 servers involved in this full time from different spots on the net. Its a lot of computation, network connections and recursive DNS, but it'll be worth it.

class As being scanned

It would be wasteful to scan every A.x.x.x block completely. IPv4 has holes in it and is not completely used. All the IPs after 223.x.x.x aren't used at all or at least don't have resolvable DNS. There are about 199 blocks that do have DNS and of those, I'm scanning 155 of them that are interesting.

Here is a list of the class A networks I hit in my scan.

8 12 13 15 16 17 18 20 24 27 31 32 35 38 40 41 43 44 46 47 50 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 107 108 109 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 178 180 182 183 184 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 216 217 218 219 220 221 222

Perhaps once I'm done with these I will go back and hit the other 45 or so that I skipped, but large portions of those are empty or very repetitive.

x.x.x.D Blocks scanned

IPs scanned: ~ 3,439,329,280 (99.6% done (based on 155 blocks and 256³ IPs per block))
Data generated: ~ 78 GB (mostly compressed, uncompressed probably around 700GB)
Elapsed time: 139.2 days
Hosts used to scan: as high as 8, but usually just 4 at once.
Current Average scan speed: ~ 2192 records/sec (189.4 million per day)
Estimated completion date of 155 class A blocks: July 17th, 2010
Estimated completion date of all 200 blocks: July 30st, 2010 (just in time for System Administrator Day)
Blocks scanned: 256 of 256 (100%)

x.x.x.0 - Finished
x.x.x.1 - Finished
x.x.x.2 - Finished
x.x.x.3 - Finished
x.x.x.4 - Finished
x.x.x.5 - Finished
x.x.x.6 - Finished
x.x.x.7 - Finished
x.x.x.8 - Finished
x.x.x.9 - Finished
x.x.x.10 - Finished
x.x.x.11 - Finished
x.x.x.12 - Finished
x.x.x.13 - Finished
x.x.x.14 - Finished
x.x.x.15 - Finished
x.x.x.16 - Finished
x.x.x.17 - Finished
x.x.x.18 - Finished
x.x.x.19 - Finished
x.x.x.20 - Finished
x.x.x.21 - Finished
x.x.x.22 - Finished
x.x.x.23 - Finished
x.x.x.24 - Finished
x.x.x.25 - Finished
x.x.x.26 - Finished
x.x.x.27 - Finished
x.x.x.28 - Finished
x.x.x.29 - Finished
x.x.x.30 - Finished
x.x.x.31 - Finished
x.x.x.32 - Finished
x.x.x.33 - Finished
x.x.x.34 - Finished
x.x.x.35 - Finished
x.x.x.36 - Finished
x.x.x.37 - Finished
x.x.x.38 - Finished
x.x.x.39 - Finished
x.x.x.40 - Finished
x.x.x.41 - Finished
x.x.x.42 - Finished
x.x.x.43 - Finished
x.x.x.44 - Finished
x.x.x.45 - Finished
x.x.x.46 - Finished
x.x.x.47 - Finished
x.x.x.48 - Finished
x.x.x.49 - Finished
x.x.x.50 - Finished
x.x.x.51 - Finished
x.x.x.52 - Finished
x.x.x.53 - Finished
x.x.x.54 - Finished
x.x.x.55 - Finished
x.x.x.56 - Finished
x.x.x.57 - Finished
x.x.x.58 - Finished
x.x.x.59 - Finished
x.x.x.60 - Finished
x.x.x.61 - Finished
x.x.x.62 - Finished
x.x.x.63 - Finished
x.x.x.64 - Finished
x.x.x.65 - Finished
x.x.x.66 - Finished
x.x.x.67 - Finished
x.x.x.68 - Finished
x.x.x.69 - Finished
x.x.x.70 - Finished
x.x.x.71 - Finished
x.x.x.72 - Finished
x.x.x.73 - Finished
x.x.x.74 - Finished
x.x.x.75 - Finished
x.x.x.76 - Finished
x.x.x.77 - Finished
x.x.x.78 - Finished
x.x.x.79 - Finished
x.x.x.80 - Finished
x.x.x.81 - Finished
x.x.x.82 - Finished
x.x.x.83 - Finished
x.x.x.84 - Finished
x.x.x.85 - Finished
x.x.x.86 - Finished
x.x.x.87 - Finished
x.x.x.88 - Finished
x.x.x.89 - Finished
x.x.x.90 - Finished
x.x.x.91 - Finished
x.x.x.92 - Finished
x.x.x.93 - Finished
x.x.x.94 - Finished
x.x.x.95 - Finished
x.x.x.96 - Finished
x.x.x.97 - Finished
x.x.x.98 - Finished
x.x.x.99 - Finished
x.x.x.100 - Finished
x.x.x.101 - Finished
x.x.x.102 - Finished
x.x.x.103 - Finished
x.x.x.104 - Finished
x.x.x.105 - Finished
x.x.x.106 - Finished
x.x.x.107 - Finished
x.x.x.108 - Finished
x.x.x.109 - Finished
x.x.x.110 - Finished
x.x.x.111 - Finished
x.x.x.112 - Finished
x.x.x.113 - Finished
x.x.x.114 - Finished
x.x.x.115 - Finished
x.x.x.116 - Finished
x.x.x.117 - Finished
x.x.x.118 - Finished
x.x.x.119 - Finished
x.x.x.120 - Finished
x.x.x.121 - Finished
x.x.x.122 - Finished
x.x.x.123 - Finished
x.x.x.124 - Finished
x.x.x.125 - Finished
x.x.x.126 - Finished
x.x.x.127 - Finished
x.x.x.128 - Finished
x.x.x.129 - Finished
x.x.x.130 - Finished
x.x.x.131 - Finished
x.x.x.132 - Finished
x.x.x.133 - Finished
x.x.x.134 - Finished
x.x.x.135 - Finished
x.x.x.136 - Finished
x.x.x.137 - Finished
x.x.x.138 - Finished
x.x.x.139 - Finished
x.x.x.140 - Finished
x.x.x.141 - Finished
x.x.x.142 - Finished
x.x.x.143 - Finished
x.x.x.144 - Finished
x.x.x.145 - Finished
x.x.x.146 - Finished
x.x.x.147 - Finished
x.x.x.148 - Finished
x.x.x.149 - Finished
x.x.x.150 - Finished
x.x.x.151 - Finished
x.x.x.152 - Finished
x.x.x.153 - Finished
x.x.x.154 - Finished
x.x.x.155 - Finished
x.x.x.156 - Finished
x.x.x.157 - Finished
x.x.x.158 - Finished
x.x.x.159 - Finished
x.x.x.160 - Finished
x.x.x.161 - Finished
x.x.x.162 - Finished
x.x.x.163 - Finished
x.x.x.164 - Finished
x.x.x.165 - Finished
x.x.x.166 - Finished
x.x.x.167 - Finished
x.x.x.168 - Finished
x.x.x.169 - Finished
x.x.x.170 - Finished
x.x.x.171 - Finished
x.x.x.172 - Finished
x.x.x.173 - Finished
x.x.x.174 - Finished
x.x.x.175 - Finished
x.x.x.176 - Finished
x.x.x.177 - Finished
x.x.x.178 - Finished
x.x.x.179 - Finished
x.x.x.180 - Finished
x.x.x.181 - Finished
x.x.x.182 - Finished
x.x.x.183 - Finished
x.x.x.184 - Finished
x.x.x.185 - Finished
x.x.x.186 - Finished
x.x.x.187 - Finished
x.x.x.188 - Finished
x.x.x.189 - Finished
x.x.x.190 - Finished
x.x.x.191 - Finished
x.x.x.192 - Finished
x.x.x.193 - Finished
x.x.x.194 - Finished
x.x.x.195 - Finished
x.x.x.196 - Finished
x.x.x.197 - Finished
x.x.x.198 - Finished
x.x.x.199 - Finished
x.x.x.200 - Finished
x.x.x.201 - Finished
x.x.x.202 - Finished
x.x.x.203 - Finished
x.x.x.204 - Finished
x.x.x.205 - Finished
x.x.x.206 - Finished
x.x.x.207 - Finished
x.x.x.208 - Finished
x.x.x.209 - Finished
x.x.x.210 - Finished
x.x.x.211 - Finished
x.x.x.212 - Finished
x.x.x.213 - Finished
x.x.x.214 - Finished
x.x.x.215 - Finished
x.x.x.216 - Finished
x.x.x.217 - Finished
x.x.x.218 - Finished
x.x.x.219 - Finished
x.x.x.220 - Finished
x.x.x.221 - Finished
x.x.x.222 - Finished
x.x.x.223 - Finished
x.x.x.224 - Finished
x.x.x.225 - Finished
x.x.x.226 - Finished
x.x.x.227 - Finished
x.x.x.228 - Finished
x.x.x.229 - Finished
x.x.x.230 - Finished
x.x.x.231 - Finished
x.x.x.232 - Finished
x.x.x.233 - Finished
x.x.x.234 - Finished
x.x.x.235 - Finished
x.x.x.236 - Finished
x.x.x.237 - Finished
x.x.x.238 - Finished
x.x.x.239 - Finished
x.x.x.240 - Finished
x.x.x.241 - Finished
x.x.x.242 - Finished
x.x.x.243 - Finished
x.x.x.244 - Finished
x.x.x.245 - Finished
x.x.x.246 - Finished
x.x.x.247 - Finished
x.x.x.248 - Finished
x.x.x.249 - Finished
x.x.x.250 - Finished
x.x.x.251 - Finished
x.x.x.252 - Finished
x.x.x.253 - Finished
x.x.x.254 - Finished
x.x.x.255 - Finished

Remaining A.x.x.x blocks

Besides the 155 class A blocks that I did scans against above, there are about 45 remaining ones that I could scan, some of them I won't for my own good (not a good idea to scan military networks). Below are some notes.

1 - APNIC (scan)
2 - RIPE NCC (scan)
3 - GE (scan)
4 - Level 3 (scan)
6 - Army Information Systems Center (do not scan)
7 - ARIN (scan)
9 - IBM (scan)
11 - DoD Intel Information Systems (do not scan)
14 - APNIC (scan)
19 - Ford Motor Company (scan)
21 - DDN-RVN ???
22 - Defense Information Systems Agency (do not scan)
25 - UK Ministry of Defence (do not scan)
26 - Defense Information Systems Agency (do not scan)
28 - DSI-North (do not scan)
29 - Defense Information Systems Agency (do not scan)
30 - Defense Information Systems Agency (do not scan)
33 - DLA Systems Automation Center
34 - Halliburton Company (do not scan ??)
45 - Interop Show Network (scan)
48 - Prudential Securities Inc. (scan)
51 - UK Government Department for Work and Pensions (scan)
52 - E.I. duPont de Nemours and Co., Inc. (scan)
53 - Cap Debis CCS (Daimler AG) (scan)
54 - Merck and Co., Inc. (scan)
55 - DoD Network Information Center (do not scan)
56 - US Postal Service (scan)
110 - APNIC (scan)
111 - APNIC(scan)
112 - APNIC (scan)
113 - APNIC (scan)
114 - APNIC (scan)
115 - APNIC (scan)
116 - APNIC (scan)
117 - APNIC (scan)
118 - APNIC (scan)
119 - APNIC (scan)
120 - APNIC (scan)
121 - APNIC (scan)
122 - APNIC (scan)
123 - APNIC (scan)
124 - APNIC (scan)
125 - APNIC (scan)
126 - APNIC (scan)
176 - RIPE NCC (allocated 2010-05) (scan later)
177 - LACNIC (allocated 2010-06) (scan later)
181 - LACNIC (allocated 2010-06) (scan later)
214 - US-DOD (do not scan)
215 - US-DOD (do not scan)
223 - APNIC (allocated 2010-04) (scan)
224-239 - Multicast, no reverse DNS
225-255 - Not in use, cannot be used due to routers that filter these blocks by policy.

5,23,36,37,39,42,49,100-106,179 and 185 are unallocated currently.

Class A.x.x.x scan schedule

1 - Finished
2 - Finished
3 - Finished
4 - Finished
7 - Finished
9 - Finished
14 - Finished
19 - Finished
45 - Finished
48 - Finished
51 - Finished
52 - Finished
53 - Finished
54 - Finished
56 - Finished
110 - Finished
111 - Finished
112 - Finished
113 - Finished
114 - Finished
115 - Finished
116 - Finished
117 - Finished
118 - Finished
119 - Finished
120 - Finished
121 - Finished
122 - Finished
123 - Finished
124 - Finished
125 - Finished
126 - Finished
176 - Finished
177 - Finished
181 - Finished
223 - Finished

After some hard thinking, I went ahead and scanned the military allocated class A blocks. Since I'm not specifically scanning those in this research, it shouldn't be a big deal. I doubt I'm the first one to scan them in this way either and I haven't heard any other cases of people getting in trouble for it unless they were being malicious, which is not my intent.

Those blocks include 6/8, 11/8, 22/8, 25/8, 26/8, 28/8, 29/8, 30/8, 34/8, 55/8, 214/8, and 215/8. At this point all of them have been scanned except I'm still finishing up the last half of 215.

Host information

This is mostly for my own reference. Initially I had 3 or 4 hosts running queries against their own nameservers on localhost. Then I decided to try using a single nameserver (deanna) and then running queries against that from 3 or 4 different hosts. This ended up being more efficient.

Then, pressed for time and resources, I came up with a new way of doing the scan that used dig's -f option for batch files. By breaking things down into 256 queries at a time, I could strike a good balance between dealing with slow DNS servers (which is the reason I choose the process per IP method) and being efficient with process forking. I can still run hundreds of digs at once, but now I'm only forking them off every second or so, which doesn't generate nearly as much load of course and saves memory, etc. Now from 1 host, I can scan as much as 700 IPs per second. On average, I've been seeing about 400-500 per second. So now with 3 hosts I can scan all of IPv4 in a month.

The numbers in parenthesis are the x.x.x.N blocks scanned from this host.

Master storage - leo
Host1 - dnsscan-vm1 (scan speed: ~483 records/sec, 107-109, 112-119, 123-129,146-149,156-159, 250)
Host2 - linode1 (scan speed: ~192 records/sec over 7 blocks, 160-179)
Host3 - slicehost1 (scan speed: ~197 records/sec 8 blocks, 180-199)
Host4 - nyc2 (scan speed: ~1320 records/sec over 49 blocks!, 201-249, 251-252, 199-191, 179-173)

Overall scan speed: ~2192 records/sec (189,388,800 records/day)

Also, with this new method, I'm not just saving the PTR record, but also the timestamp for each lookup and anything in the AUTHORITY section. I'll probably end up running a second scan to get this information for the blocks I've already scanned.

I ended up bringing nyc2 into the mix. Its a physical host with 4GB of RAM, two processing cores and its a beast to be running this. I can set its max parallel digs to 700+ so that it can blow through slow areas of DNS much faster than the others. The fastest scanning speed I've seen it reach is over 1500 records/sec, but its average is around 800 or 900. It can process about 6 blocks a day so I'm giving it a large piece of the last section of the scan, 201-249.

Old method hosts

Master storage - leo
Host0 - Home machine (2)
Host1 - lists (avg 44 records/sec) (1,5,10,100,200,253,254,16-19,30-39,60-69)
Host2 - linode1 (avg 65.9 records/sec) (40-49, 110-119)
Host3 - slicehost1 (avg 67.9 records/sec) (3,4,6,7,8,9,11,12,13,14,15,20-29,50-59, 52, 0, 255, 130-139)
Host4 - deanna (offline)
Host5 - yanni (queries linode1, avg. 63 records/sec) (70-79)
Host6 - ludwig (queries linode1, avg. 86.1 records/sec) (80-89, 140-149)
Host7 - nyc (queries linode1, avg. 87.8 records/sec) (90-99, 150-159)
Host8 - nyc2 (queries linode1, avg. 65.3 records/sec) (101-109)
Host9 - felix (queries lists, with linode1 as backup, avg. 22.4 rec/sec) (120-129)

I changed the lookup scripts so that they use linode1, but lists as a backup, except for felix.