Intro
During my OSCP studies, I realized I needed a more efficient system for cracking password hashes. The screaming CPU fans and high CPU usage became a problem. I first tried using hashcat and the GPU on my MacBook Pro in OS X. There are some bugs and problems with hashcat on OS X that would make it crash in the middle of cracking a hash. Also, I was not interested in investing a server with a bunch of GPUs, the high costs to do this would outweigh the amount of time I need the system. All of this lead me to do a little research and found the instructions in the following link to build an AWS instance for password cracking.
Since that post was created there have been some changes to the offerings in AWS EC2 leading me write this post.
If you wish to skip ahead I have created scripts to automate the processes in the rest of this post. They are both in my
github and can be downloaded at the following links.https://github.com/suidroot/AWSScripts/blob/master/aws-ec2-create-kracker.sh
https://github.com/suidroot/AWSScripts/blob/master/configure-kracker.sh
For the rest of the article I will cover some of the instance options in EC2, installation of the needed Linux packages, the basic setup of Hashcat, running Hashcat, and finally monitoring and benchmarks of an EC2 instance.
AWS EC2 Options
There are many options for EC2 instances, they have a huge range
- G3 - https://aws.amazon.com/ec2/instance-types/g3/
- P2 - https://aws.amazon.com/ec2/instance-types/p2/
I found the g3 instances to be the more cost effective tier. For my testing I opted to use the g3.4xlarge tier. Next to choose the AMI image, appropriate the appropriate operating system.
AMI images
There are two options that are I tested hashcat on they are both Ubuntu based. I’m sure there are many other available options that will work too, but I am familiar with Ubuntu systems. The first option is a standard Ubuntu image, there is nothing special about this image and it requires configuration to add the GPU drivers and a little more work.
The next option is a Deep Learning image, this image is preconfigured with the GPU drivers and was originally designed for machine learning applications. I found the the pre-configuration allowed for me skip a few steps in building out a new system.
Instance Build and config
Once you have the instance deployed there are a few steps to get the Instance prepared for hashcat, the steps are a little bit different between a Standard and a Deep Learning Ubuntu instance.
An apt cronjob may already be running and you will have to wait it out.
Prepare Machine (Standard Ubuntu)
This script will install all the required packages and the Nvidia GPU drivers on a vanilla Ubuntu installation.
#!/bin/bash # mostly copied from: https://medium.com/@iraklis/running-hashcat-v4-0-0-in-amazons-aws-new-p3-16xlarge-instance-e8fab4541e9b # sudo apt-get update -yq sudo apt-get install -yq build-essential linux-headers-$(uname -r) unzip p7zip-full linux-image-extra-virtual sudo apt-get install -yq ocl-icd-libopencl1 opencl-headers clinfo #sudo apt-get install -yq libhwloc-plugins libhwloc5 libltdl7 libpciaccess0 libpocl2 libpocl2-common ocl-icd-libopencl1 pocl-opencl-icd sudo apt-get install -yq python3-pip pip3 install psutil sudo touch /etc/modprobe.d/blacklist-nouveau.conf sudo bash -c "echo 'blacklist nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf" sudo bash -c "echo 'blacklist lbm-nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf" sudo bash -c "echo 'options nouveau modeset=0' >> /etc/modprobe.d/blacklist-nouveau.conf" sudo bash -c "echo 'alias nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf" sudo bash -c "echo 'alias lbm-nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf" sudo touch /etc/modprobe.d/nouveau-kms.conf sudo bash -c "echo 'options nouveau modeset=0' >> /etc/modprobe.d/nouveau-kms.conf" sudo update-initramfs -u sudo reboot ### Install nVidia Drivers wget http://us.download.nvidia.com/tesla/410.104/NVIDIA-Linux-x86_64-410.104.run sudo /bin/bash NVIDIA-Linux-x86_64-410.104.run --ui=none --no-questions --silent -X
Prepare Machine (Deep Learning Ubuntu)
In comparison the previous script there is a much simpler script to prepare the Deep Learning instance. The main focus is installing the needed archive extraction tools.
#!/bin/bash sudo apt update sudo apt upgrade sudo apt install clinfo unzip p7zip-full sudo apt install build-essential linux-headers-$(uname -r) # Optional sudo apt-get install -yq python3-pip pip3 install psutil
Hashcat Setup
Now we need to download and extract the star of the show Hashcat. The link in the wget below points to the the most recent version as of writing however you might want to check to see if there is a more recent version at the main site: https://hashcat.net/hashcat/
wget https://hashcat.net/files/hashcat-5.1.0.7z 7z x hashcat-5.1.0.7z
Download wordlists
You will need some wordlists for hashcat to use to crack passwords, he commands listed are for some wordlists I like to use when cracking. You should however add whichever lists are your favories.
mkdir ~/wordlists git clone https://github.com/danielmiessler/SecLists.git ~/wordlists/seclists wget -nH http://downloads.skullsecurity.org/passwords/rockyou.txt.bz2 -O ~/wordlists/rockyou.txt.bz2 cd ~/wordlists bunzip2 ./rockyou.txt.bz2 cd ~
Running hashcat
Now it is time to run hashcat and crack some passwords. When running hashcat I had the best performance with the arguments-O -w 3. Below is an example command line I've used
./hashcat-5.1.0/hashcat64.bin --username -m 1800 ./megashadow256.txt wordlists/rockyou.txt -r hashcat-5.1.0/rules/best64.rule -O -w 3Monitoring the Nvidia GPU
nvidia-smi
ubuntu@ip-172-31-17-6:~$ sudo nvidia-smi Fri Apr 26 14:43:49 2019 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | | N/A 37C P0 42W / 150W | 0MiB / 7618MiB | 97% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+
This example shows a GPU being used by hashcat.
ubuntu@ip-172-31-17-6:~$ sudo nvidia-smi Fri Apr 26 14:44:44 2019 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | | N/A 46C P0 141W / 150W | 828MiB / 7618MiB | 100% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 11739 C ./hashcat-5.1.0/hashcat64.bin 817MiB | +-----------------------------------------------------------------------------+
Conclusion and Benchmarks
Finally here is a benchmark I ran on a g3.4xlarge instance. This instance type contains 1 GPU. These results give an idea of performance for this AWS EC2 instance type.
ubuntu@ip-172-31-17-6:~$ ./hashcat-5.1.0/hashcat64.bin -O -w 3 -b
hashcat (v5.1.0) starting in benchmark mode...
* Device #2: Not a native Intel OpenCL runtime. Expect massive speed loss.
You can use --force to override, but do not report related errors.
nvmlDeviceGetFanSpeed(): Not Supported
OpenCL Platform #1: NVIDIA Corporation
======================================
* Device #1: Tesla M60, 1904/7618 MB allocatable, 16MCU
OpenCL Platform #2: The pocl project
====================================
* Device #2: pthread-Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz, skipped.
Benchmark relevant options:
===========================
* --optimized-kernel-enable
* --workload-profile=3
Hashmode: 0 - MD5
Speed.#1.........: 11611.6 MH/s (90.74ms) @ Accel:512 Loops:512 Thr:256 Vec:4
Hashmode: 100 - SHA1
Speed.#1.........: 4050.2 MH/s (65.01ms) @ Accel:512 Loops:128 Thr:256 Vec:2
Hashmode: 1400 - SHA2-256
Speed.#1.........: 1444.5 MH/s (91.98ms) @ Accel:256 Loops:128 Thr:256 Vec:1
Hashmode: 1700 - SHA2-512
Speed.#1.........: 499.4 MH/s (66.78ms) @ Accel:128 Loops:64 Thr:256 Vec:1
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)
Speed.#1.........: 189.8 kH/s (42.76ms) @ Accel:128 Loops:64 Thr:256 Vec:1
Hashmode: 1000 - NTLM
Speed.#1.........: 18678.1 MH/s (56.58ms) @ Accel:512 Loops:512 Thr:256 Vec:2
Hashmode: 3000 - LM
Speed.#1.........: 10529.6 MH/s (50.60ms) @ Accel:128 Loops:1024 Thr:256 Vec:1
Hashmode: 5500 - NetNTLMv1 / NetNTLMv1+ESS
Speed.#1.........: 10650.8 MH/s (49.60ms) @ Accel:512 Loops:256 Thr:256 Vec:1
Hashmode: 5600 - NetNTLMv2
Speed.#1.........: 829.3 MH/s (80.24ms) @ Accel:256 Loops:64 Thr:256 Vec:1
Hashmode: 1500 - descrypt, DES (Unix), Traditional DES
Speed.#1.........: 442.0 MH/s (37.81ms) @ Accel:4 Loops:1024 Thr:256 Vec:1
Hashmode: 500 - md5crypt, MD5 (Unix), Cisco-IOS $1$ (MD5) (Iterations: 1000)
Speed.#1.........: 4209.1 kH/s (51.39ms) @ Accel:1024 Loops:500 Thr:32 Vec:1
Hashmode: 3200 - bcrypt $2*$, Blowfish (Unix) (Iterations: 32)
Speed.#1.........: 7572 H/s (33.02ms) @ Accel:16 Loops:4 Thr:8 Vec:1
Hashmode: 1800 - sha512crypt $6$, SHA512 (Unix) (Iterations: 5000)
Speed.#1.........: 76958 H/s (83.99ms) @ Accel:512 Loops:128 Thr:32 Vec:1
Hashmode: 7500 - Kerberos 5 AS-REQ Pre-Auth etype 23
Speed.#1.........: 149.4 MH/s (56.00ms) @ Accel:128 Loops:64 Thr:64 Vec:1
Hashmode: 13100 - Kerberos 5 TGS-REP etype 23
Speed.#1.........: 152.1 MH/s (55.00ms) @ Accel:128 Loops:64 Thr:64 Vec:1
Hashmode: 15300 - DPAPI masterkey file v1 (Iterations: 23999)
Speed.#1.........: 32703 H/s (84.02ms) @ Accel:256 Loops:64 Thr:256 Vec:1
Hashmode: 15900 - DPAPI masterkey file v2 (Iterations: 7999)
Speed.#1.........: 21692 H/s (96.24ms) @ Accel:256 Loops:128 Thr:32 Vec:1
Hashmode: 7100 - macOS v10.8+ (PBKDF2-SHA512) (Iterations: 35000)
Speed.#1.........: 5940 H/s (40.09ms) @ Accel:64 Loops:32 Thr:256 Vec:1
Hashmode: 11600 - 7-Zip (Iterations: 524288)
Speed.#1.........: 4522 H/s (55.87ms) @ Accel:256 Loops:128 Thr:256 Vec:1
Hashmode: 12500 - RAR3-hp (Iterations: 262144)
Speed.#1.........: 18001 H/s (56.74ms) @ Accel:4 Loops:16384 Thr:256 Vec:1
Hashmode: 13000 - RAR5 (Iterations: 32767)
Speed.#1.........: 18135 H/s (55.93ms) @ Accel:128 Loops:64 Thr:256 Vec:1
Hashmode: 6211 - TrueCrypt PBKDF2-HMAC-RIPEMD160 + XTS 512 bit (Iterations: 2000)
Speed.#1.........: 121.7 kH/s (59.39ms) @ Accel:128 Loops:32 Thr:256 Vec:1
Hashmode: 13400 - KeePass 1 (AES/Twofish) and KeePass 2 (AES) (Iterations: 6000)
Speed.#1.........: 68380 H/s (158.89ms) @ Accel:512 Loops:256 Thr:32 Vec:1
Hashmode: 6800 - LastPass + LastPass sniffed (Iterations: 500)
Speed.#1.........: 1088.7 kH/s (48.51ms) @ Accel:128 Loops:62 Thr:256 Vec:1
Hashmode: 11300 - Bitcoin/Litecoin wallet.dat (Iterations: 199999)
Speed.#1.........: 2107 H/s (78.97ms) @ Accel:128 Loops:64 Thr:256 Vec:1
Started: Fri Apr 26 14:36:56 2019
Stopped: Fri Apr 26 14:42:03 2019
If you've made it this far congratulation and happy cracking!