Some more tips and tricks
I've had the WL-500gP for about a week now, using Oleg's firmware, with an USB HDD rack and it works really great. I wanted to point out some stuff I've learned the hard way.
Bittorrent client:
I've read this a lot of times and I didn't believe it until I tried and convinced myself: Oleg's torrent.cgi or transmission.cgi is really the best all around torrent client out there at this time.
As far as console clients go, I've tried transmission, ctorrent, enhanced-ctorrent, tornado and rtorrent. Most of them are single-torrent. tornado is multi-torrent but is not interactive. Really, the best console client out there is rtorrent, but after Oleg managed to port it I suddenly realised it lacks automated queue control and that I much more enjoy a web interface, since it's more likely to have a web browser handy wherever I go than a ssh client.
I've also tried various web interfaces. I've managed to make torrentflux-b4rt work with lighttpd, PHP4 and sqlite2, using tornado as a client (see this tutorial) and then managed to compile the special transmission version that comes with b4rt. With tornado it's a real hog. The combination of PHP, lighttpd, sqlite and Python is simply too much. Simply reloading a PHP page works the HDD a lot. With only one torrent going the load stays above 1.4 and I've heard that once you go to about 4 torrents at once the router is pretty much dedicated to that. I've tried and the load shot to 6 or 7. With transmission it's quite alright, but when I stood back and looked at the thing I realized that it didn't do anything that Oleg's cgi doesn't essentially do already.
And compare it to Oleg's cgi, which gives me average load of 0.3 or 0.4 with 6 torrents going. The only gripe I have with the transmission.cgi is that it doesn't move torrents around the queues by itself (LE yes it does, I just hadn't understood about the watchdog at the time). Well, overall is really a simple and very good torrent client and if I have the time I may hack it one day to clean up the code in the watchdog and cgi and add some more features. (I'd like it to know how to queue more than one torrent at once, or to change the speed limits from the web interface.)
How to allow SSH and other stuff from outside:
If you run iptables -L INPUT on your router, you'll probably see something like this in Oleg's latest firmware (1.9.2.7-7f):
Code:
logdrop all -- anywhere anywhere state INVALID
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere state NEW
ACCEPT all -- anywhere anywhere state NEW
SECURITY all -- anywhere anywhere state NEW
ACCEPT udp -- anywhere anywhere udp spt:bootps dpt:bootpc
ACCEPT tcp -- anywhere anywhere tcp dpt:ftp
logdrop all -- anywhere anywhere
The reason many of you haven't been able to open up your outside ports (for SSH and other stuff) is because of that first rule. I don't know what the INVALID chain is for and where it comes from, but that's where the packets stop.
So this is how my /usr/local/sbin/post-firewall looks like:
Code:
#!/bin/sh
## FIREWALL
## set default policy
iptables -P INPUT DROP
## deny ftp access from WAN
iptables -I INPUT 1 -p tcp -i "$1" --syn --dport 21 -j DROP
## Allow access to various router services from WAN
for P in 22 65534 8008; do
iptables -I INPUT 1 -p tcp --syn -i "$1" --dport $P -j ACCEPT
done
What I do with the firewall is:
A. I change the default policy from ACCEPT to DROP, which is saner, security-wise.
B. I insert some rules before the INVALID rule, thus bypassing it: first I deny FTP SYN packets on the outside interface, which cuts access to FTP. Second, I make a list of ports (ssh, torrent and transmission.cgi) and let SYN packages to pass through to them. That's it.
Some more security ideas:
1. For SSH: it would be a good idea to change the default port from 22 to something else, like 24. It cuts down on automatic scanners figuring out you have SSH open. It is NOT a foolproof security measure, but it's good nonetheless. You do this by finding where you start dropbear (probably /usr/local/sbin/post-boot) and adding -p 24 as parameter. Don't forget to change the firewall rule above too.
2. For torrent, opening up 65534 has really helped me. I get a LOT more peers now, about 2-4 times as many as before. I get so many it's ridiculous, it makes some instances of transmissiond have a hard time keeping up with them and use up CPU. (A configurable connection limit in transmissiond would be really sweet BTW.)
3. If you open up web access to transmission.cgi from outside be sure to protect it with a password. See the /opt/etc/httpd.conf help or /opt/etc/init.d/S80busybox_httpd for instructions on how to do that. EVEN SO, it still not good enough! The user and password pass around in cleartext! It will stop someone who accidentally stumbles across your web interface from using it, but if you access it from work for instance almost anyone (your admin definitely) can eavesdrop and pick up the user and password without a problem. So be warned. What would really help here is SSL, but I've yet to figure how to make busybox make a self-signed SSL website.
Wondershaper:
The place to add this is also in /usr/local/bin/post-firewall, at the bottom:
Code:
## WONDERSHAPER
/sbin/wshaper start "$1" 400 190
Replace 400 and 190 with appropriate numbers for your connection.
How to determine them? First, you take the maximum figures for your connection. If you know what they're supposed to be from your ISP (like mine are supposed to be 512 and 192) than start from there. Even better, do a test. I personally love the Speakeasy test. It will give you real figures (remember to stop all other traffic, such as torrents, while you do the test!)
So, you got 512 and 192. But for wshaper to work best you actually need to lower them somewhat. How much? To see how much, you need to ping an outside host, preferably from the router console, but you can do it from a LAN computer too. You can ping anything, such as www.google.com.
But the best would be to ping your gateway. To find out what it is, do this command: cat /proc/net/arp. You'll see something like this:
Code:
IP address HW type Flags HW address Mask Device
89.xxx.xxx.xxx 0x1 0x2 XX:XX:XX:XX:F5:4B * vlan1
That IP on the vlan1 line is the gateway, ping that.
So now you have one console where a ping runs, and you watch the times. It would be best to start some heavy traffic, like torrents, too, to see how it does under real heavy conditions. My ping would go to around 300ms under these circumstances. Now, in another router console, run that wshaper command:
Code:
/sbin/wshaper start vlan1 512 192
(Note I use vlan1, not "$1", since we're in the console not post-firewall). Then look at the ping and see if it goes lower. It probably won't, not much, so start lowering both the download and the upload numbers slightly and run the command again, then look at the ping again. The download number needs to go down a lot more than the upload. I finally ended up with 400 and 190, and at that point my ping went between 10-60ms.
Note that this WILL NOT LIMIT your download speed! So don't be worried. All it does is to tweak the wondershaper for better latency, it has no connection to the actual download speed you get.
That's it for now, will get back if I find out more worthy stuff.
How Oleg's torrent Web interface works
A lot of people, myself included, are probably having some trouble when they first run into Oleg's web interface for torrent download (transmission.cgi). That's because it's a little different from the torrent programs they're used to. So I'm going to describe what's going on under the hood.
There are 3 pieces of software working on the torrents:
1) transmissiond. This is the actual workhorse that's downloading and uploading a torrent. If you run "ps" in a console you will see several of them working (if you have torrents running). Every time a torrent starts, one of these comes along and takes charge of it.
2) transmission_watchdog. This one runs periodically, usually every 30 minutes, and is in charge of moving torrents around the queues (from "queued" to "active" to "done").
3) transmission.cgi this is the actual web interface and is the thing you use to see your torrents and control them.
How to use the torrents
You start by putting a torrent file under the source/ directory. You can do this in many ways: you can download it on a desktop computer and FTP or SCP it to the router; you can SSH to the router, go into that directory and wget it from a website; you can use the "Fetch" button in the web interface to give the URL and the cgi will wget it and put it under source/.
All torrents found under source/ are automatically seen as part of the "Queued" set of torrents. If there are no torrents in the "Active" set, the watchdog will fetch the first one in the Queued set the next time it runs and make it Active. Or, you can manually make as many torrents as you want Active by selecting them and using the Push button in the interface.
Everytime the watchdog runs it will also check for completed Active torrents (the ones seeding) and will move them to the Completed set, where they stop moving and wait for you to come get the files.
You can manually pause an Active torrent by selecting it and using the Push button. This will stop the torrent and put it in the Suspended set. It will not leave this set anymore unless you select it and use the Push button again. (That's right, Push does two different things; when the torrent is first Queued it moves it to Active, and then it moves it between Active and Suspended; there's no way to move it back to Queued anymore).
Active torrents (and their files) are found under the work/ dir. Queued torrents are under source/. Completed torrents are under target/. Suspended torrents are under work/, but are temporarily renamed to ".torrent.suspended".
What other buttons do:
List: this one gives the complete list of all the torrents in all the queues.
Update: this one updates the Active information with actual data. Since this update is an expensive operation for the router, it is usually cached and you need to press Update to refresh it. Otherwise, using List will only show the same old info.
Watchdog: this will force the watchdog to run now instead of waiting for the next normal time. This is useful if you see that a torrent has finished downloading and is seeding, but you don't want to wait for it to be moved to Completed the next time the watchdog would normally run. (Please note that you should let torrents seed so other people can benefit too. Instead of stopping that torrent, consider Push-ing a Queued one instead.)
Pause: this will stop all torrents temporarily. It does this by telling all transmissiond programs to die. It doesn't move torrents around the queues at all, there simply aren't downloads or uploads going. This will be marked clearly in big bold letters saying "torrent processing paused". Use Pause again to resume the Active torrents.
Info will show some information about the torrent (tracker, what files are in it and so on).
Log will try to create a graph showing the state of the downloads. Depends on whether you have gnuplot installed. (I personally prefer rrdtool, look around the forum for the rrdtool tutorial.)
Remove will mark a torrent for removal, and Purge will actually DELETE BOTH THE TORRENT AND THE FILES FOR IT. So be careful with it.
Scrape will attempt to ask the tracker for the scrape info, and Best will attempt to suggest the best seed torrent from the completed ones.
Note will add your personal note to the torrent, in case you want to jot down something about it.
How to protect your webserver with a password
User+password access to your webserver is very very useful because, together with SSL encryption of the connection, it allows you to access your router fairly safely even when not at home.
Warnings:
* SSL by itself is useful, but not much use if anybody can access the router webpages without being asked for user+password.
* This is NOT ABOUT PROTECTING THE SETUP interface. I'm only talking about protecting websites you set up on the router by yourself, such as the transmission cgi interface. The web setup of the router is another breed of webserver who runs separately.
* Even when using SSL+user+pass, you ARE NOT COMPLETELY SAFE. The owner or admin of the computer you're using may have installed keyloggers or network transparent proxies which will intercept your user+pass or decode SSL. They protect against casual eavesdropping, but will not stop a determined and knowledgeable person with access to your immediate network setup or to the computer you're using. Nothing can do that.
These being said, here's how to add user+pass to lighttpd:
NOTE: I've chosen digest authentication because it is a bit more secure than basic authentication. If you want basic change "digest" to "basic" and "htdigest" to "htpasswd" in the instructions below. Be WARNED that dumber browsers (like lynx) cannot use digest authentication!
1) Install lighttpd: ipkg install lighttpd.
2) Copy the default config file (/opt/etc/lighttpd/lighttpd.conf) to another file. This is so you don't risk it being overwritten by mistake during upgrades.
NOTE: if you took the SSL tutorial above and already have an SSL-enabled config, very good, modify that to add user+pass!
3) Edit the config file. Under server_modules, make sure to uncomment "mod_auth".
4) Scroll down and look for "auth.". Set auth.backend to "htdigest". Set auth.backend.htdigest.userfile to the a file where you want to keep users and passwords. Example: /opt/etc/lighttpd/.passwd.
5) Modify the auth.require section. Make sure it looks something like this:
Code:
auth.require = ( "/" =>
(
"method" => "digest",
"realm" => "ABC",
"require" => "valid-user"
)
)
Make sure to close all the brackets, and watch out for those commas!
The "/" is the directory you want to protect, relative to the webserver root dir. Here I say I want to protect all the files.
The method is "digest", and because I chose digest I have called the auth. directives above accordingly! Have I called it "basic", for example, I'd have changed the auth. stuff too.
Realm is the name of the authentication set. Remember what you write here, we'll need it in a moment below. (Every bracket set needs its own name, because other things may be the same across more than one set and can't be used to refer to them.)
Finally, require means who you want to be able to authenticate. "Valid-user" means any user that supplies a valid user+pass. You could also say "user=john" and only allow john in that set, even though mary also has a valid user+pass.
6) You're done with the config, save it and restart lighttpd whenever you want.
7) Now you need to generate the passwd file. You can't just edit it by hand. You need the "htdigest" program, which is found in the apache package. You can "ipkg install apache", of course, but I prefer to go to a temp dir (NOT /tmp!), do "ipkg download apache" and then look inside for what I want (the ipk files are actually tar.gz files and have another "data.tar.gz" file in them).
8) One way or the other, you got htdigest. Use it whenever you want to generate it or change a password or add a user:
Code:
#to generate it:
htdigest -c /opt/etc/lighttpd/.passwd ABC john
# change john's password for the ABC realm
htdigest /opt/etc/lighttpd/.passwd ABC john
# add mary on the XYZ realm
htdigest /opt/etc/lighttpd/.passwd XYZ mary
When you want to delete users edit the files with a text editor and delete lines you don't want.
9) Test your server. If you can't log in, some common errors are:
* wrong user and/or password
* you didn't put the same realm in the conf and the password file
* the password file can't be read (check permissions)
* your browser can't do digest authentication
Check your lighttpd error log (probably /opt/var/log/lighttpd/error.log) for clues.
Small transmission.cgi hack
I've modified transmission_watchdog and transmission.cgi to accomplish a small feature I wasn't able to live without anymore.
Sometimes I have these torrents that are extremely slow for some reason (no seeds ATM, slow peers etc.) I'd like them to keep on doing their job no matter how slowly, but I don't want them to be counted as running by the watchdog, because they keep the queue occupied and other queued torrents don't get a chance to run.
So what I've done is a 2 step hack:
Step 1: hacked transmission.cgi to offer a button that toggles an "ignored" flag on running torrents.
To do this, edit transmission.cgi (tipically found in /opt/share/www/cgi-bin/).
a) Add the following function somewhere:
Code:
# Toggle ignore flag
_toggle_ignore()
{
if [ -z "$ID" ] ; then
echo "<b>Please select torrent first!</b>"
return
fi
_find
if [ -f "${WORK}${TORRENT#${WORK}}" ]; then
if [ -f "${TORRENT%.torrent}.torrent" ]; then
DIR=$(dirname "$TORRENT")
[ -f "${DIR}/ignored" ] && rm -f "${DIR}/ignored" || touch "${DIR}/igno
echo "<b>$TORRENT toggled</b>"
fi
fi
}
b) Find the buttons, near the end of the file, and add this one:
Code:
<input type=submit name=ACTION value="Ignore">
c) Find the actions, almost all the way to the bottom, and add this one:
Code:
"Ignore") _toggle_ignore ; _list ;;
d) To show the ignored flag, find a place that looks like this:
Code:
if [ -n "${URL}" ]; then
echo "<td><a href=\"${URL}\" target=_blank>${DUMMY}</a></td>"
else
echo "<td>${DUMMY}</td>"
fi
And change it to look like this:
Code:
if [ -n "${URL}" ]; then
echo "<td><a href=\"${URL}\" target=_blank>${DUMMY}</a>"
else
echo "<td>${DUMMY}"
fi
[ -f "$P/ignored" ] && echo "<b style="color:red">[I]</b>"
echo "</td>"
Step 2: make the watchdog actually respect the flag.
Edit /opt/sbin/transmission_watchdog. Find a place that looks like this:
Code:
# Check if any torrent file exists in run env.
TORRENT=`ls -1 $WORK/*/*.torrent 2>/dev/null | head -n 1`
# Start up new torrent if work is empty
if [ -z "$TORRENT" ]; then
max_active_torrents=10
if [ ${ACTIVE_TORRENTS} = ${max_active_torrents} ]; then
And change it to look like this:
Code:
WERKIN=$ACTIVE_TORRENTS
IGNEUR=$(ls "$WORK"/*/ignored 2>/dev/null | wc -l)
DIFFER=$((WERKIN-IGNEUR))
echo "<p>${WERKIN} active torrents, ${IGNEUR} ignored, ${DIFFER} acknowledged.</p>
if [ "$DIFFER" -lt 1 ]; then
if [ "$DIFFER" -gt 9 ]; then
How to use:
When in the cgi web interface, select an Active torrent then click on the "Ignore" button. You should see a bold I appear next to it. Do it again to clear the flag.
You can only toggle the flag for Active torrents. As long as they have the flag, they are not counted towards active torrents when the watchdog determines if it needs to promote more torrents from the Queued ones.
Use this to mark slow torrents which you want to keep chugging along slowly and not keep back the other torrents.
Warning! ipkg upgrade may override the modified files!
Everytime the transmission package is upgraded you may lose the files you've just modified! So you are advised to make copies for personal use and name them something else. To do this:
a) Rename transmission.cgi to transmission2.cgi.
b) Rename transmission_watchdog to transmission_watchdog2.
c) Edit your cgi (transmission2.cgi), look at the bottom, find transmission_watchdog among the actions for the buttons, and replace it with transmission_watchdog2.
d) Start using /cgi-bin/transmission2.cgi instead of the original.
How to hack wshaper to prioritize ports
I've hacked wshaper to take a 9th parameter, which specifies ports that are to be given priority over other kinds of traffic.
To hack wshaper yourself, edit it and look for the following two snippets of code. The bold parts are to be added by you, the non-bold parts are what you should look for:
Code:
# low priority destination ports
NOPRIOPORTDST="$8"
# high priority destination ports
HIPRIOPORTDST="$9"
Code:
match u8 0x10 0xff at 33 \
flowid 1:10
# some traffic suffers a better fate
for a in $HIPRIOPORTDST
do
tc filter add dev $DEV parent 1: protocol ip prio 14 32 \
match ip dport $a 0xffff flowid 1:10
done
# some traffic however suffers a worse fate
for a in $NOPRIOPORTDST
do
I think I did it properly. :) I've also tested it: after I made use of the new parameter, I've used a bandwidth measurement site while I had a couple of torrents going. I got the full speed of my line in the test, even though the torrents were using about half of it. This is because now port 80 (http) takes priority.
To use this you need to run wshaper at least once on your router, and re-run it after every boot. It's best to edit /usr/local/sbin/post-firewall and add the call to wshaper in there. Remember to save the modifications to flash!
Here's how I run it:
Code:
wshaper start $1 400 190 "" "" "" "" "20 21 22 25 110 143 80 443 587 995"
If you run wshaper from post-firewall you need to use $1. If you run it from the command line you need to use "vlan1".
To determine the proper values for the 3rd and 4th parameters (400 190) you need to read the first post in this thread.
The ports I've included in the example are: FTP, SSH, SMTP (outgoing email), POP3 and IMAP (incoming email), regular web, secure web, plus a couple of other ports I'm using.
REMEMBER! This setup refers to ports on remote sites or machines, opened with programs on the computers in your LAN (or on the router).
2 Attachment(s)
torrentflux-b4rt: full featured torrent client
torrentflux-b4rt is a fork of torrentflux. It's a full features bittorrent web interface, but it's somewhat of a hog on resources. If you can live with a load of 1.5-2.0 when running 2-3 torrents, each limited to about 40-50 maximum connections, then here's how to set it up.
1. Install the requirements:
Code:
ipkg install lighttpd php php-fcgi sqlite2 python
2. Get the attached torrent-flux.conf. Put it under /opt/etc/lighttpd/. Examine it and create the appropriate directories (such as /opt/share/www/torrent.flux) or log dirs.
3. Download torrentflux-b4rt. Copy the contents of the html/ dir in the package under /opt/share/www/torrent.flux.
4. Start lighttpd:
Code:
/opt/sbin/lighttpd -f /opt/etc/lighttpd/torrent-flux.conf
If you get error messages examine them and fix them, most likely you need some dirs created.
5. Use a browser and go to your router on port 8081, and load /setup.php. You should get the installation wizard. Tip: when asked for an sqlite database, the first field (the file) is best if it's empty or "localhost".
That's it. The rest is up to how you set it up, because it has a very complex configuration. Don't be scared of how slow the interface is, that's how it works I'm afraid, the Asus is a small and slow machine. :(
If you want queueing you'll need to start fluxd from the interface.
I advise you to use bittornado as the client, because it's included with the package and works well out of the box. If you want transmission it won't work with Oleg's transmissioncli! You need to compile the hacked version that comes with torrentflux-b4rt and put it somewhere on your HDD and point the configuration to it. I'm attaching a compiled binary taken from b4rt alpha6, try to use it, YMMV.
Save CPU, pause transmission when not in use
Oleg's transmission wiggles around even when there are no torrents in work. Enough for the load to go around 0.3. So when you're not using it, go in the interface and hit "Pause". All transmission clients will stop soon after that, and you'll get a nice load of 0.0. Or, you can use that CPU power to do something else.
"Can only remove suspended torrents!"
Hi folks,
I also implemented transmission and I am very much excited. But shame on me, I am not able to purge these damned completed torrents. I tried everything:
First I see, the torrent is completed but is seeding as hell. Then I mark it in Transmission WI and press "remove" I get a very unfriendly "Can only remove suspended torrents!".
Ok, now I tried every button... I thougt, "pause" would do the trick. But I then realized, that only my active (downloading that is) torrents go to suspended mode. These torrents I surely DON'T want to purge.
Could you please have mercy and give a newbie a little hint?
What is the correct manner to purge a completed and copied torrent?:confused:
Thanks in advance
Sandman
