FEWGETS, File Manager and wget based downloader AJAX WebUI for Oleg firmwares

[ecaddict]

Updated on 07.10.2011, the most important changes:

• Unified support for 2.4 and 2.6 kernel routers (shutdown script)
• Added right click download from router via fewgets.cgi
• Enhanced the rtorrent starter script /opt/etc/init.d/S99rtorrent
• Enhanced GUI to show commands in non URL encoded form

So what fewgets does for you?

• It contains a simple downloader that uses screen and wget in the background (kindof simplified ADOS) - plugins e.g. for rapid had to be reduced as they don't work anymore
• It has a file manager similar to Filemanager (www.gerd-tentler.de/tools/filemanager/) or eXtplorer (http://extplorer.sourceforge.net/)
• It has a traffic statistics similar to major internet exchanges

What makes fewgets unique?

• It's entirely based on bash script (+javascript generated by bash script) and does not need php
• It aims efficiency over fancy GUI, and it's faster then probably any other similar SW
• UTF-8 support of different alphabets and strong support for special characters
• Specifically designed for the router environment and tries to place load to the client side as much possible

The following tools (packages) have to be installed with ipkg install before you can take into use fewgets (install script automatically takes care of that):
screen, find (findutils), wget, ps (procps), lighttpd, wc (coreutils)

For rtorrent.cgi: gnuplot readline

For the traffic tab cron has to be installed and set up, that is
/opt/etc/crontab should contain (among others):

*/5 * * * * admin run-parts /opt/etc/cron.5mins

The graphs are similar to the ones generated by MRTG or used by major internet exchange sites (daily/weekly/monthly/yearly traffic graphs are shown for each interfaces).
You have to either run traffic.sh directly from /opt/etc/crontab or if you've set up cron to run scripts from directories (recommended way) copy it to /opt/etc/cron.5mins or to /opt/etc/cron.1min (the script has to run nx1 minutes).
If your traffic does not exceed on (5 min) average the 55 Mbit/s it's sufficient to run traffic.sh in every 5 minutes. If it exceeds it (as it can happen with RT-N16) the script has to run in every minute (up to 275 Mbit/s can be handled in this way; this is due to the 32bit counters and signed integers used by the shell).
The traffic log files are stored in /opt/var/log. They are rotated by traffic.sh, thus no extra handling is required.
Note: If you use cron.1min, you have to change LOGLN in traffic.sh to:

LOGLN=1442

It takes 5-10 (1-2) minutes the first plot to appear. The graphs are accurate only if the traffic.sh script is always run when the router routes traffic.

Download:
fewgets.tar.gz

It contains the following files (you can extract it manually also but there will be install as well):

/opt/etc/logrotate.conf
/opt/etc/init.d/rc.unslungstop
/opt/etc/init.d/S99rtorrent
/opt/etc/cron.5mins/traffic.sh
/opt/etc/cron.hourly/
/opt/etc/rtorrlog
/mnt/cgi-bin/plugins/data.hu.sh
/mnt/cgi-bin/plugins/addat.hu.sh
/mnt/cgi-bin/rtorrent.cgi
/mnt/cgi-bin/qwget.sh
/mnt/cgi-bin/fewgets.cgi
/usr/local/sbin/shutdown

Make the .sh files (also in plugin directory) executable via chmod u+x and also /usr/local/sbin/shutdown and /opt/etc/init.d/rc.unslungstop

If you manually extract save shutdown to flash via
flashfs save && flashfs commit && flashfs enable

install.tar.gz
Single line install (note: it overwrites the above files):

cd /tmp && wget -O install.tar.gz "http://wl500g.info/attachment.php?attachmentid=8352&d=1318017040" && tar -xvzf install.tar.gz && ./install.sh

Open https://192.168.1.1/cgi-bin/fewgets.cgi

If it does not work check your /opt/etc/lighttpd/lighttpd.conf, it should have something like the following lines:

server.modules              = (
#...
                                "mod_scgi",
                                "mod_fastcgi",
                                "mod_cgi",
#...
                               "mod_accesslog" )

#...

##                                                                         
# which extensions should not be handle via static-file transfer           
#                                                                          
# .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi        
static-file.exclude-extensions = ( ".fcgi", ".php", ".pl", ".py", ".rb",   
".cgi" )  

#### fastcgi module                                                                                                                                          
## read fastcgi.txt for more info                                                                                                                            
fastcgi.server             = ( ".php" =>                                                                                                                     
                               ((                                                                                                                            
                                 "host" => "192.168.1.1",                                                                                                    
                                 "port" => 1026,                                                                                                             
                               ))                                                                                                                            
                            )                                                                                                                                
                                                                                                                                                             
#### CGI module                                                                                                                                              
cgi.assign                 = ( ".cgi"  => "/bin/sh",                                                                                                         
                               ".pl"   => "/opt/bin/perl" )                                                                                                  
                                                                                                                                                             
scgi.server                = ("/RPC2" =>                                                                                                                     
                               ( "127.0.0.1" =>                                                                                                              
                                (                                                                                                                            
                                  "host" => "127.0.0.1",                                                                                                     
                                  "port" => 5000,                                                                                    
                                  "check-local" => "disable"                                                                           
                                )                                                                                                                            
                               )                                                                                                                             
                             )

The program to work correctly requires canvas/SVG capable browser javascript enabled (best works with Firefox/Chrome).
If icons/lines start to disappear (the program triggers the browser using significant amount of memory), disable the leading lines in fewgets.cgi:
treeul=true -> treeul=false
It also makes displaying a bit faster.
Nevertheless this is the javascript that I can crash any browser effortlessly...

You can use right-click menus (Ctrl+left-clik in Opera) or for file/directory operations Ctr+C|Ctr+X,Ctrl+V.
Shortcut keys are context sensitive and override browser shortcut keys in the given context.

A few screenshots:


Screenshot of the file manager:


The tgz file also includes rtorrent.cgi that can create rtorrent traffic graph from logs.
For that you have to include the following line to your /opt/etc/rtorrent.conf:

schedule = log,180,180,"execute=/opt/etc/rtorrlog,$get_down_rate=,$get_up_rate="

Open: https://192.168.1.1/cgi-bin/rtorrent.cgi
The rtorrent.cgi assumes that your system log is in /opt/var/log and it's either messages or syslog.log.
If that's not the case you have to adjust rtorrent.cgi accordingly.

The install script also creates dyndns updater /opt/etc/cron.hourly/dyndns.sh. I needed it to force daily update even if IP address does not change for months. It starts the update if you set your dyndns account name and password.

Warning: The file manager never asks for confirmation and there is no undelete in Linux so be very cautious with delete.

Also I did a reasonable amount of testing but I'm in no event shall be liable for any lost or corrupted files or any other damage caused by the program. Use it only at your own risk.

[wpte]

Looks really good!
didn't test it yet, but I think this is a better replacement for the download master than ADOS, since ADOS keeps crashing due to (probably) an error in mysqli.

but if you have a public site it's better to protect it with a password from lighttpd

[ecaddict]

This is what I use:

$SERVER["socket"] == ":443" {
#### SSL engine
ssl.engine = "enable"
ssl.pemfile = "/opt/etc/lighttpd/lighttpd.pem"
server.document-root = "/mnt/"
auth.require = ( "" =>
(
"method" => "digest",
"realm" => "Password protected area",
"require" => "user=admin"
)
)
}

require could be

"require" => "valid-user"

as well and then I guess authentication data from
/opt/etc/lighttpd/.lighttpdpassword

I've considered this issue out of scope so that's why not touched earlier.

[kemo]

Very good
But my internet provider allow me downloading only between 00:00 and 6:00 hours. Any solution for this problem???? :/
Thanx....
(sorry for my bad english)

[ecaddict]

If you don't need any sophisticated solution then it's pretty easy to add time constraint to downloads. Probably you've also notices that download is controlled via simple scripts. The qwget.sh controls the queue handling. It can be changed so that it starts download only if actual time is within allowed download time. Otherwise wait (go to sleep) till allowed time comes.
Of course this won't stop ongoing downloads if time limit is reached.

Here is a modified qwget.sh for time control (I have not tested it, later on when tested I'll include the package).
Btw, English is also not my native language.

#!/bin/sh
# written by ecaddict, distributed (conveyed) under GPL version 3 or any later version
SHBASE=/mnt/cgi-bin/
LIMIT_RATE="128k"
EXMT=/opt/bin/mktemp
EXBN=/opt/bin/basename
EXDN=/opt/bin/dirname
DLSTART=00:00
DLSTOP=06:00

IN=qwget.list
SD=`$EXDN $0`

TMPFILE=`$EXMT -q -t qwget_dl.XXXXXXXX`
DLDIR=`pwd`
[ -n "$EWGETDIR" ] && SHBASE="$EWGETDIR"

getlintm() {
  HT="$1" && HT=${HT//:/ }
  FACT=3600; SEC=0
  for F in $HT ;do
    F=${F#0}
    let "SEC+=$F*$FACT"; let "FACT=$FACT/60"
  done
  echo "$SEC"
}
LDLST=`getlintm "$DLSTART"`
LDLSP=`getlintm "$DLSTOP"`

while true
do
  NOW=`uptime` && NOW=`getlintm "${NOW:1:8}"`
  SLEEPS=0
  if [ "$LDLST" -gt "$LDLSP" ];then
    if [ "$NOW" -gt "$LDLSP" ] && [ "$NOW" -lt "$LDLST" ];then
      let "SLEEPS=$LDLST-$NOW"
    fi
  else
    if [ "$NOW" -lt "$LDLST" ];then
      let "SLEEPS=$LDLST-$NOW"
    elif [ "$NOW" -gt "$LDLSP" ];then
      let "SLEEPS=$LDLST+86400-$NOW"
    fi
  fi
  if [ "$SLEEPS" -ne 0 ];then
    sleep "$SLEEPS"
  fi

  read DF < "$SD/$IN"
  if [ -n "$DF" ] ;then
    if [ -n "`echo $DF | sed -ne '/rapidshare.com/p' -e '/data.hu/p'`" ] ;then
      echo "$DF" > "$TMPFILE"
      (
        trap - INT QUIT TSTP
        $SHBASE/rget.sh "$TMPFILE" "$DLDIR"
      )
    else
      wget -c --limit-rate="$LIMIT_RATE" "$DF"
    fi
    DF=`$EXBN "$DF"`
    cat "$SD/$IN" | sed '/'''$DF'''/d' > "$SD/$IN"
  fi
  sleep 15
  clear
done

[kemo]

Thanks man. I try it tomorow...

[ecaddict]

The details about the new fewgets version supporting traffic graphs are in the opening post.

[ecaddict]

The new version fixes the typo in /opt/etc/cron.5mins/traffic.sh that caused that only 4/5 of the traffic is shown in monthly/yearly statistics.

In addition, switched to plugin files for downloads (e.g. rapidshare) that normally need various user actions (like waiting, following links etc.)

If you know some scripts that could also be used, please share it.

[Rebel]

the lighttpd give me "Internal server error" and show me this error:
/opt/share/www/lighttpd/cgi-bin/fewgets.cgi: 119: Syntax error: Bad substitution

any ideas?

[ecaddict]

I have no idea, do you use latest Oleg and have this in lighttpd.conf?

#### CGI module                                                                                                                                              
cgi.assign                 = ( ".cgi"  => "/bin/sh",                                                                                                         
                               ".pl"   => "/opt/bin/perl" )

The page to come up has only this requirement (new Oleg firm. is required to have more bash like function in /bin/sh that helps performance).

[Rebel]

I think the problem is that i don't have this version of oleg firmware.
I have to update it first.
Thanks.

[bigphantom]

Download:
Attachment 6410

(updated on 15.05.2010, fixed UID issue)

Take the following files from the archive under your webroot (e.g in my case /mnt/cgi-bin):
/mnt/cgi-bin/fewgets.cgi
/mnt/cgi-bin/qwget.sh
/mnt/cgi-bin/plugins/...

(older versions used, rget.sh, you can delet it, downloader plugins are used instead)
Make the .sh files (also in plugin directory) executable via chmod +x

Place the shutdown file to /usr/local/sbin
/usr/local/sbin/shutdown
make it executable via chmod +x and save it to flash via
flashfs save && flashfs commit && flashfs enable

and finally copy
/opt/etc/init.d/rc.unslungstop
/opt/etc/cron.5mins/traffic.sh
as well and also make them executable via chmod +x
(shutdown and rc.unslungstop is needed only if you'd like to use the stop tab, it's not needed for the download or for the filemanager tab)

Open https://your_router's_address/cgi-bin/fewgets.cgi

hi there, and thank you for your work till now.
I have a rather strange/stupid question, and i can't figure it out myself.
I have installes on RT-N16 - r2411 the HeadStart II image, using your indication and also installed rtorrent+rutorrent + FEWGETS.

question: how do I figure out which version do I have installed ?

[ecaddict]

question: how do I figure out which version do I have installed ?

There ain't so many versions, so probably you have the latest one (for sure if you've installed it after the summer). If it works enjoy, if not I might help but nowadays I have really very limited time for this.

[ecaddict]

I've updated fewgets, details are in the first post.