Update README.md
Browse files
README.md
CHANGED
@@ -1,146 +1,8 @@
|
|
1 |
-
|
2 |
-
|
3 |
-
|
4 |
-
|
5 |
-
|
6 |
-
|
7 |
-
|
8 |
-
|
9 |
-
|
10 |
-
<!-- dynamic-count-start -->
|
11 |
-
## Current Proxy Count: 2800+ π
|
12 |
-
<!-- dynamic-count-end -->
|
13 |
-
|
14 |
-
|
15 |
-
## Quick Access - By Format
|
16 |
-
|
17 |
-
|
18 |
-
TXT
|
19 |
-
|
20 |
-
curl "https://sunny9577.github.io/proxy-scraper/proxies.txt" > proxies.txt
|
21 |
-
|
22 |
-
JSON
|
23 |
-
|
24 |
-
curl "https://sunny9577.github.io/proxy-scraper/proxies.json" > proxies.json
|
25 |
-
|
26 |
-
CSV
|
27 |
-
|
28 |
-
curl "https://sunny9577.github.io/proxy-scraper/proxies.csv" > proxies.csv
|
29 |
-
|
30 |
-
YAML
|
31 |
-
|
32 |
-
|
33 |
-
curl "https://sunny9577.github.io/proxy-scraper/proxies.yaml" > proxies.yaml
|
34 |
-
|
35 |
-
XML
|
36 |
-
|
37 |
-
|
38 |
-
curl "https://sunny9577.github.io/proxy-scraper/proxies.xml" > proxies.xml
|
39 |
-
|
40 |
-
## Quick Access - By Type
|
41 |
-
###### Supported formats: txt | json | csv | yaml | xml
|
42 |
-
|
43 |
-
SOCKS5
|
44 |
-
|
45 |
-
curl "https://sunny9577.github.io/proxy-scraper/generated/socks5_proxies.txt" > socks5_proxies.txt
|
46 |
-
|
47 |
-
|
48 |
-
SOCKS4
|
49 |
-
|
50 |
-
curl "https://sunny9577.github.io/proxy-scraper/generated/socks4_proxies.txt" > socks4_proxies.txt
|
51 |
-
|
52 |
-
HTTP(S)
|
53 |
-
|
54 |
-
curl "https://sunny9577.github.io/proxy-scraper/generated/http_proxies.txt" > http_proxies.txt
|
55 |
-
|
56 |
-
|
57 |
-
## Setup Requirements
|
58 |
-
1 - A Windows, Mac or Linux Computer
|
59 |
-
|
60 |
-
2 - Google Chrome installed
|
61 |
-
|
62 |
-
3 - Nodejs and NPM installed
|
63 |
-
|
64 |
-
## Setup
|
65 |
-
This bot is based on [protractor](https://www.protractortest.org/#/) automated browser testing software.
|
66 |
-
### 1 - Open a terminal and make sure you are in the project directory
|
67 |
-
cd proxy-scraper
|
68 |
-
|
69 |
-
### 2 - Make sure you have protractor installed
|
70 |
-
To check if you have protractor installed run:
|
71 |
-
|
72 |
-
protractor --version
|
73 |
-
|
74 |
-
If the output is something like this:
|
75 |
-
|
76 |
-
Version 7.0.0
|
77 |
-
|
78 |
-
This means protractor is already installed, otherwise run the command:
|
79 |
-
|
80 |
-
npm install -g protractor
|
81 |
-
|
82 |
-
### 3 - Install Selenium Webdriver
|
83 |
-
This is where your program will run, to install it simply execute the command:
|
84 |
-
|
85 |
-
webdriver-manager
|
86 |
-
|
87 |
-
### 4 - Database Connectivity ( Only if u want to save output to database)
|
88 |
-
You can skip this step just set SAVE_TO_DB to false in config.js
|
89 |
-
|
90 |
-
Go to the file config.js and change with your login information
|
91 |
-
|
92 |
-
npm install mysql2
|
93 |
-
|
94 |
-
Table Structure (Table Name: proxies_tb)
|
95 |
-
|
96 |
-
|Name|Type|Collation|Attributes|Null|Default|Comments|Extra|
|
97 |
-
|----|----|---------|----------|----|-------|--------|-----|
|
98 |
-
|1 |idPrimary |int(11) | |No |None |AUTO_INCREMENT
|
99 |
-
|2 |proxyIndex |varchar(50) |latin1_swedish_ci |No |None
|
100 |
-
|3 |port |int(10) | |No |None
|
101 |
-
|4 |country |varchar(100) |latin1_swedish_ci |No |None
|
102 |
-
|5 |type |varchar(100) |latin1_swedish_ci |No |None
|
103 |
-
|6 |anonymity |varchar(100) |latin1_swedish_ci |No |None
|
104 |
-
|
105 |
-
## Running the bot
|
106 |
-
|
107 |
-
### 1 - Update the Selenium webserver and install dependencies
|
108 |
-
Open a terminal and make sure you are in the project directory:
|
109 |
-
|
110 |
-
cd proxy-scraper
|
111 |
-
|
112 |
-
Update webdrivers:
|
113 |
-
|
114 |
-
webdriver-manager update
|
115 |
-
|
116 |
-
Install dependencies:
|
117 |
-
|
118 |
-
npm install
|
119 |
-
|
120 |
-
### 2 - Run the bot
|
121 |
-
Its time to start retrieving the proxies, make sure you are in the project directory:
|
122 |
-
|
123 |
-
cd proxy-scraper
|
124 |
-
|
125 |
-
Run the bot:
|
126 |
-
|
127 |
-
protractor conf.js
|
128 |
-
|
129 |
-
## Additional Tools
|
130 |
-
|
131 |
-
- #### socks5_test.py
|
132 |
-
|
133 |
-
This python script tests the SOCKS5 proxies against https://icanhazip.com and https://eset.com. Contributed by [@techroy23](https://www.github.com/techroy23)
|
134 |
-
|
135 |
-
Steps to run:
|
136 |
-
- pip3 install tabulate pysocks
|
137 |
-
- python3 socks5_test.py
|
138 |
-
|
139 |
-
## Disclaimer
|
140 |
-
This repository is intended solely for educational purposes. I neither endorse nor encourage any illegal activities or behavior.
|
141 |
-
|
142 |
-
## License
|
143 |
-
|
144 |
-
[MIT](LICENSE)
|
145 |
-
|
146 |
-
[](https://starchart.cc/Naereen/badges)
|
|
|
1 |
+
---
|
2 |
+
license: mit
|
3 |
+
title: Proxy-Scraper
|
4 |
+
sdk: static
|
5 |
+
emoji: π
|
6 |
+
colorFrom: green
|
7 |
+
colorTo: yellow
|
8 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|