Project Idea: Make whois great again!

Hackerspace
#1

Currently GPDR has killed whois. But the main registries (IANA, ARIN, RIPE, etc.) still offer a rest api for each of their databases while none of them correlate between each other anymore.

To resolve this this project would setup a api crawler using logstash, elastic search, and python (where needed) to aggregate each database into a central point. Then provide a client interface for searching via the command line.

First release would use surfraw(1) then second release would provide a gateway from Gophernet and a raw tls connection client (ie akin to openssl s_client -connect whois.dapla.net:4343) that preforms in the same manner as whois did before. This client would be build in Go and provide a standardized output based on standard ascii text (UTF-8 encoding).

2 Likes
#2

This is an excellent idea. While I should know these things I’ll ask anyway:

Of the main registries have any of them become gpdr compliant?

Is there any information which would be stored which would expose DMS or ourselves to GDPR?

If yes above, might we consider a blockchain db to lessen that exposure?

In any case I’d love to help on this.

#3

So we have a really cool version below:

DNS

curl cli.fyi/google.com | jq -r '.data.dns[]'

google.com.  581 IN MX 20 alt1.aspmx.l.google.com.
google.com.  581 IN MX 10 aspmx.l.google.com.
google.com.  581 IN MX 50 alt4.aspmx.l.google.com.
google.com.  581 IN MX 40 alt3.aspmx.l.google.com.
google.com.  581 IN MX 30 alt2.aspmx.l.google.com.
google.com.  295 IN A 216.58.213.110
google.com.  48 IN AAAA 2a00:1450:4009:816::200e
google.com.  294894 IN NS ns3.google.com.
google.com.  294894 IN NS ns2.google.com.
google.com.  294894 IN NS ns1.google.com.
google.com.  294894 IN NS ns4.google.com.
google.com.  60 IN SOA ns1.google.com. dns-admin.google.com. (
    264570408  ; serial
    900        ; refresh (15 minutes)
    900        ; retry (15 minutes)
    1800       ; expire (30 minutes)
    60         ; minimum (1 minute)
    )
google.com.  3581 IN TXT "v=spf1 include:_spf.google.com ~all"
google.com.  3581 IN TXT "facebook-domain-verification=22rm551cu4k0ab0bxsw536tlds4h95"
google.com.  3581 IN TXT "globalsign-smime-dv=CDYX+XFHUw2wml6/Gb8+59BsH31KzUr6c1l2BPvqKX8="
google.com.  281 IN TXT "docusign=1b0a6754-49b1-4db5-8540-d2c12664b289"
google.com.  281 IN TXT "docusign=05958488-4752-4ef2-95eb-aa7ba8a3bd0e"

Whois

curl cli.fyi/isup.me | jq -r '.data.whois[]'

Domain Name: ISUP.ME
Registry Domain ID: D108500000001724664-AGRS
Registrar WHOIS Server:
Registrar URL: www.uniregistrar.com
Updated Date: 2019-05-18T01:45:15Z
Creation Date: 2010-12-05T05:33:12Z
Registry Expiry Date: 2021-12-05T05:33:12Z
Registrar Registration Expiration Date:
Registrar: Uniregistrar Corp
Registrar IANA ID: 1659
Registrar Abuse Contact Email:
Registrar Abuse Contact Phone:
Reseller:
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Registrant Organization:
Registrant State/Province: Grand Cayman
Registrant Country: KY
Name Server: GRACE.NS.CLOUDFLARE.COM
Name Server: ADAM.NS.CLOUDFLARE.COM
DNSSEC: unsigned
URL of the ICANN Whois Inaccuracy Complaint Form https://www.icann.org/wicf/)
>>> Last update of WHOIS database: 2019-08-22T15:49:19Z <<<

For more information on Whois status codes, please visit https://icann.org/epp

Access to WHOIS information is provided to assist persons in determining the contents of a domain name registration record in the registry database. The data in this record is provided by The Registry Operator for informational purposes only, and accuracy is not guaranteed. This service is intended only for query-based access. You agree that you will use this data only for lawful purposes and that, under no circumstances will you use this data to (a) allow, enable, or otherwise support the transmission by e-mail, telephone, or facsimile of mass unsolicited, commercial advertising or solicitations to entities other than the data recipient's own existing customers; or (b) enable high volume, automated, electronic processes that send queries or data to the systems of Registry Operator, a Registrar, or Afilias except as reasonably necessary to register domain names or modify existing registrations. All rights reserved. Registry Operator reserves the right to modify these terms at any time. By submitting this query, you agree to abide by this policy.

The Registrar of Record identified in this output may have an RDDS service that can be queried for additional information on how to contact the Registrant, Admin, or Tech contact of the queried domain name.

Doesn’t work with internet protocol (IP) addresses. Autonomous System Number (AS/ASN), MIBS/OIDS, or MAC addresses just yet.

For now there’s a few resources that do exist which help

Subject endpoint authentication?
MAC api.macvendors.com false
OID https://www.oidview.com/api/api.html true
ASN https://stat.ripe.net/docs/data_api false
IP to ASN https://api.iptoasn.com/v1/as/ip/ false
mxtoolbox https://mxtoolbox.com/c/products/mxtoolboxapi false
#4

Can’t you just download the whole TLD Zone file.
I think that’s what companies do for their whois databases.

Get it from Verisign

I used to do this back in the day to see when domain drops where and have my computer fight to register them.

#5

Might be an option but the idea here is having something that both a PWA, serverless scripts, or even shell scripts could parse and act upon.

For example, say one has system logs and packet captures going to an indexer (elasticsearch / splunk) then wants to compare some events from an ip that has “random macs” show up when it hits a wordpress site. Have a json parsing script works best for this. The script would look something like:

## pseudo code [ bash ]
## surfraw elvi [ validateMac.sh ]

events = curl elasticsearch.local:9200/test/_search?q=time:today | jq '.hits.hits[] | with_entites(select(match('source')))[]'

for mac in $events; do
  curl -f cli.fyi/$mac 2>&1 >/dev/null
 [ $? == 22 ] && echo "Invalid MAC detected" || echo "MAC is valid."
done

The end result would verify a mac address found from logs against what is in known without downloading any large text files or having extra tools running.

On can then take this and turn it into a surfraw elvi script so one just calles sr validateMac 10.1.1.1 DE:AD:BE:EF

#6

sql database.
import change zone file once a day to it that verisign will feed you.
have PWA use js IndexDB if you want it to run offline or local copy
run sql select query to find your culrpits

#7

nothing wrong with that. Still requires a server and infrastructure built for that.

A user facing tool is not the only use case here.

Automation and administration tools run best with rest.

#8

No it doesn’t
It’s free for ever
https://cloud.google.com/free/docs/gcp-free-tier
Make a python service worker or something

#9

Thanks for that tip. I always knew about the 12 month / $300 trial. I never realized they also offered the “Always free” tier.

1 Like