How to archive Github in minutes

Hackerspace Computer Science
#1

Step 1, install curl, git, and make (or run linux)
Step 2. Use this makefile:

NAME            := $(shell whoami)
TARGET          := users
PAGE            := 1
MAX             := 1000

all: $(TARGET)

users: CTNX=users
users: clone wikis

orgs: CTNX=orgs
orgs: clone wikis

$(NAME):
        @mkdir $@

clone: $(NAME)
        -$(shell cd $(NAME); curl -sSLk "https://api.github.com/$(CTNX)/$(NAME)/repos?page=$(PAGE)&per_page=$(MAX)" | grep -e 'git_url*' | cut -d\" -f4 | xargs -L1 -P3 git clone -q)

wikis: $(NAME)
        -$(shell cd $(NAME); curl -sSLk "https://api.github.com/$(CTNX)/$(NAME)/repos?page=$(PAGE)&per_page=$(MAX)" | grep -e 'git_url*' | sed 's/\.git/.wiki.git/'| cut -d\" -f4 | xargs -L1 -P3 git clone -q)

Step 3. Execute make NAME=<your github username || orgname > TARGET=<users || orgs>
Step 4. Store all the code somewhere’s safe. Archive.org is a nice choice.

2 Likes
#2

Results after an hour on slow wifi:

denzuko@desktop ~/recovery
$ ls -al
total 213
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 13 01:20 ./
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 13 01:24 ../
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 21:06 betterthanmilk/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 20:46 CompuTEK-Industries/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 20:48 Dallas-Makerspace/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 21:04 Daplanet/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 13 01:17 denzuko/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 21:07 denzuko-ansible-roles/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 21:25 denzuko-archive/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 21:15 denzuko-devops/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 13 01:03 denzuko-forked/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 21:34 denzuko-tests-code/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 20:41 DFW2600/
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 21:01 Garterney/
-rw-r--r-- 1 Dwight Spencer 197121 576 Jan 12 20:49 Makefile
drwxr-xr-x 1 Dwight Spencer 197121   0 Jan 12 21:10 Zero1-Nation/

4.27 GB

Sadly it’s going to longer to store it on long term storage media than to retrieve from GitHub. Plus this version doesn’t include issues and comments. That’s about two/three more web calls though and pull requests are welcome.