Results 1 to 14 of 14

Thread: FEWGETS, File Manager and wget based downloader AJAX WebUI for Oleg firmwares

  1. #1

    [How To]FEWGETS, File Manager and wget based downloader AJAX WebUI for Oleg firmwares

    Updated on 07.10.2011, the most important changes:
    • Unified support for 2.4 and 2.6 kernel routers (shutdown script)
    • Added right click download from router via fewgets.cgi
    • Enhanced the rtorrent starter script /opt/etc/init.d/S99rtorrent
    • Enhanced GUI to show commands in non URL encoded form


    So what fewgets does for you?
    • It contains a simple downloader that uses screen and wget in the background (kindof simplified ADOS) - plugins e.g. for rapid had to be reduced as they don't work anymore
    • It has a file manager similar to Filemanager (www.gerd-tentler.de/tools/filemanager/) or eXtplorer (http://extplorer.sourceforge.net/)
    • It has a traffic statistics similar to major internet exchanges


    What makes fewgets unique?
    • It's entirely based on bash script (+javascript generated by bash script) and does not need php
    • It aims efficiency over fancy GUI, and it's faster then probably any other similar SW
    • UTF-8 support of different alphabets and strong support for special characters
    • Specifically designed for the router environment and tries to place load to the client side as much possible

    The following tools (packages) have to be installed with ipkg install before you can take into use fewgets (install script automatically takes care of that):
    screen, find (findutils), wget, ps (procps), lighttpd, wc (coreutils)

    For rtorrent.cgi: gnuplot readline

    For the traffic tab cron has to be installed and set up, that is
    /opt/etc/crontab should contain (among others):
    */5 * * * * admin run-parts /opt/etc/cron.5mins
    The graphs are similar to the ones generated by MRTG or used by major internet exchange sites (daily/weekly/monthly/yearly traffic graphs are shown for each interfaces).
    You have to either run traffic.sh directly from /opt/etc/crontab or if you've set up cron to run scripts from directories (recommended way) copy it to /opt/etc/cron.5mins or to /opt/etc/cron.1min (the script has to run nx1 minutes).
    If your traffic does not exceed on (5 min) average the 55 Mbit/s it's sufficient to run traffic.sh in every 5 minutes. If it exceeds it (as it can happen with RT-N16) the script has to run in every minute (up to 275 Mbit/s can be handled in this way; this is due to the 32bit counters and signed integers used by the shell).
    The traffic log files are stored in /opt/var/log. They are rotated by traffic.sh, thus no extra handling is required.
    Note: If you use cron.1min, you have to change LOGLN in traffic.sh to:
    LOGLN=1442
    It takes 5-10 (1-2) minutes the first plot to appear. The graphs are accurate only if the traffic.sh script is always run when the router routes traffic.

    Download:
    fewgets.tar.gz

    It contains the following files (you can extract it manually also but there will be install as well):
    Code:
    /opt/etc/logrotate.conf
    /opt/etc/init.d/rc.unslungstop
    /opt/etc/init.d/S99rtorrent
    /opt/etc/cron.5mins/traffic.sh
    /opt/etc/cron.hourly/
    /opt/etc/rtorrlog
    /mnt/cgi-bin/plugins/data.hu.sh
    /mnt/cgi-bin/plugins/addat.hu.sh
    /mnt/cgi-bin/rtorrent.cgi
    /mnt/cgi-bin/qwget.sh
    /mnt/cgi-bin/fewgets.cgi
    /usr/local/sbin/shutdown
    Make the .sh files (also in plugin directory) executable via chmod u+x and also /usr/local/sbin/shutdown and /opt/etc/init.d/rc.unslungstop

    If you manually extract save shutdown to flash via
    flashfs save && flashfs commit && flashfs enable

    install.tar.gz
    Single line install (note: it overwrites the above files):
    Code:
    cd /tmp && wget -O install.tar.gz "http://wl500g.info/attachment.php?attachmentid=8352&d=1318017040" && tar -xvzf install.tar.gz && ./install.sh

    Open https://192.168.1.1/cgi-bin/fewgets.cgi

    If it does not work check your /opt/etc/lighttpd/lighttpd.conf, it should have something like the following lines:
    Code:
    server.modules              = (
    #...
                                    "mod_scgi",
                                    "mod_fastcgi",
                                    "mod_cgi",
    #...
                                   "mod_accesslog" )
    
    #...
    
    ##                                                                         
    # which extensions should not be handle via static-file transfer           
    #                                                                          
    # .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi        
    static-file.exclude-extensions = ( ".fcgi", ".php", ".pl", ".py", ".rb",   
    ".cgi" )  
    
    #### fastcgi module                                                                                                                                          
    ## read fastcgi.txt for more info                                                                                                                            
    fastcgi.server             = ( ".php" =>                                                                                                                     
                                   ((                                                                                                                            
                                     "host" => "192.168.1.1",                                                                                                    
                                     "port" => 1026,                                                                                                             
                                   ))                                                                                                                            
                                )                                                                                                                                
                                                                                                                                                                 
    #### CGI module                                                                                                                                              
    cgi.assign                 = ( ".cgi"  => "/bin/sh",                                                                                                         
                                   ".pl"   => "/opt/bin/perl" )                                                                                                  
                                                                                                                                                                 
    scgi.server                = ("/RPC2" =>                                                                                                                     
                                   ( "127.0.0.1" =>                                                                                                              
                                    (                                                                                                                            
                                      "host" => "127.0.0.1",                                                                                                     
                                      "port" => 5000,                                                                                    
                                      "check-local" => "disable"                                                                           
                                    )                                                                                                                            
                                   )                                                                                                                             
                                 )
    The program to work correctly requires canvas/SVG capable browser javascript enabled (best works with Firefox/Chrome).
    If icons/lines start to disappear (the program triggers the browser using significant amount of memory), disable the leading lines in fewgets.cgi:
    treeul=true -> treeul=false
    It also makes displaying a bit faster.
    Nevertheless this is the javascript that I can crash any browser effortlessly...

    You can use right-click menus (Ctrl+left-clik in Opera) or for file/directory operations Ctr+C|Ctr+X,Ctrl+V.
    Shortcut keys are context sensitive and override browser shortcut keys in the given context.

    A few screenshots:
    Name:  tra_vlan1.jpg
Views: 3926
Size:  277.6 KB

    Screenshot of the file manager:
    Name:  fewgetsFM.jpg
Views: 4044
Size:  172.3 KB

    The tgz file also includes rtorrent.cgi that can create rtorrent traffic graph from logs.
    For that you have to include the following line to your /opt/etc/rtorrent.conf:
    Code:
    schedule = log,180,180,"execute=/opt/etc/rtorrlog,$get_down_rate=,$get_up_rate="
    Open: https://192.168.1.1/cgi-bin/rtorrent.cgi
    The rtorrent.cgi assumes that your system log is in /opt/var/log and it's either messages or syslog.log.
    If that's not the case you have to adjust rtorrent.cgi accordingly.

    The install script also creates dyndns updater /opt/etc/cron.hourly/dyndns.sh. I needed it to force daily update even if IP address does not change for months. It starts the update if you set your dyndns account name and password.

    Warning: The file manager never asks for confirmation and there is no undelete in Linux so be very cautious with delete.

    Also I did a reasonable amount of testing but I'm in no event shall be liable for any lost or corrupted files or any other damage caused by the program. Use it only at your own risk.
    Last edited by ecaddict; 20-10-2011 at 13:28. Reason: extended logrotate.conf to openvpn.log and transmission

  2. #2
    Join Date
    Dec 2007
    Location
    The Netherlands - Eindhoven
    Posts
    1,767
    Looks really good!
    didn't test it yet, but I think this is a better replacement for the download master than ADOS, since ADOS keeps crashing due to (probably) an error in mysqli.

    but if you have a public site it's better to protect it with a password from lighttpd

  3. #3
    This is what I use:
    $SERVER["socket"] == ":443" {
    #### SSL engine
    ssl.engine = "enable"
    ssl.pemfile = "/opt/etc/lighttpd/lighttpd.pem"
    server.document-root = "/mnt/"
    auth.require = ( "" =>
    (
    "method" => "digest",
    "realm" => "Password protected area",
    "require" => "user=admin"
    )
    )
    }
    require could be
    "require" => "valid-user"
    as well and then I guess authentication data from
    /opt/etc/lighttpd/.lighttpdpassword

    I've considered this issue out of scope so that's why not touched earlier.
    Last edited by ecaddict; 09-09-2009 at 09:35.

  4. #4
    Join Date
    Jan 2009
    Location
    Liberec
    Posts
    8
    Very good
    But my internet provider allow me downloading only between 00:00 and 6:00 hours. Any solution for this problem???? :/
    Thanx....
    (sorry for my bad english)
    WL-500gP - oleg 1.9.2.7-d-r740. IPKG,SAMBA,FTP,PHP,ADOS...
    + WD HDD 250GB SATA

  5. #5
    If you don't need any sophisticated solution then it's pretty easy to add time constraint to downloads. Probably you've also notices that download is controlled via simple scripts. The qwget.sh controls the queue handling. It can be changed so that it starts download only if actual time is within allowed download time. Otherwise wait (go to sleep) till allowed time comes.
    Of course this won't stop ongoing downloads if time limit is reached.

    Here is a modified qwget.sh for time control (I have not tested it, later on when tested I'll include the package).
    Btw, English is also not my native language.

    Code:
    #!/bin/sh
    # written by ecaddict, distributed (conveyed) under GPL version 3 or any later version
    SHBASE=/mnt/cgi-bin/
    LIMIT_RATE="128k"
    EXMT=/opt/bin/mktemp
    EXBN=/opt/bin/basename
    EXDN=/opt/bin/dirname
    DLSTART=00:00
    DLSTOP=06:00
    
    IN=qwget.list
    SD=`$EXDN $0`
    
    TMPFILE=`$EXMT -q -t qwget_dl.XXXXXXXX`
    DLDIR=`pwd`
    [ -n "$EWGETDIR" ] && SHBASE="$EWGETDIR"
    
    getlintm() {
      HT="$1" && HT=${HT//:/ }
      FACT=3600; SEC=0
      for F in $HT ;do
        F=${F#0}
        let "SEC+=$F*$FACT"; let "FACT=$FACT/60"
      done
      echo "$SEC"
    }
    LDLST=`getlintm "$DLSTART"`
    LDLSP=`getlintm "$DLSTOP"`
    
    while true
    do
      NOW=`uptime` && NOW=`getlintm "${NOW:1:8}"`
      SLEEPS=0
      if [ "$LDLST" -gt "$LDLSP" ];then
        if [ "$NOW" -gt "$LDLSP" ] && [ "$NOW" -lt "$LDLST" ];then
          let "SLEEPS=$LDLST-$NOW"
        fi
      else
        if [ "$NOW" -lt "$LDLST" ];then
          let "SLEEPS=$LDLST-$NOW"
        elif [ "$NOW" -gt "$LDLSP" ];then
          let "SLEEPS=$LDLST+86400-$NOW"
        fi
      fi
      if [ "$SLEEPS" -ne 0 ];then
        sleep "$SLEEPS"
      fi
    
      read DF < "$SD/$IN"
      if [ -n "$DF" ] ;then
        if [ -n "`echo $DF | sed -ne '/rapidshare.com/p' -e '/data.hu/p'`" ] ;then
          echo "$DF" > "$TMPFILE"
          (
            trap - INT QUIT TSTP
            $SHBASE/rget.sh "$TMPFILE" "$DLDIR"
          )
        else
          wget -c --limit-rate="$LIMIT_RATE" "$DF"
        fi
        DF=`$EXBN "$DF"`
        cat "$SD/$IN" | sed '/'''$DF'''/d' > "$SD/$IN"
      fi
      sleep 15
      clear
    done
    Last edited by ecaddict; 29-11-2009 at 09:14.

  6. #6
    Join Date
    Jan 2009
    Location
    Liberec
    Posts
    8
    Thanks man. I try it tomorow...
    WL-500gP - oleg 1.9.2.7-d-r740. IPKG,SAMBA,FTP,PHP,ADOS...
    + WD HDD 250GB SATA

  7. #7

    new fewgets version supporting traffic graphs

    The details about the new fewgets version supporting traffic graphs are in the opening post.

  8. #8

    New fewgets version (v0.1)

    The new version fixes the typo in /opt/etc/cron.5mins/traffic.sh that caused that only 4/5 of the traffic is shown in monthly/yearly statistics.

    In addition, switched to plugin files for downloads (e.g. rapidshare) that normally need various user actions (like waiting, following links etc.)

    If you know some scripts that could also be used, please share it.
    Last edited by ecaddict; 24-03-2010 at 09:15.

  9. #9
    Join Date
    Feb 2008
    Location
    Portugal
    Posts
    4

    Bad substitution

    the lighttpd give me "Internal server error" and show me this error:
    /opt/share/www/lighttpd/cgi-bin/fewgets.cgi: 119: Syntax error: Bad substitution

    any ideas?

  10. #10
    I have no idea, do you use latest Oleg and have this in lighttpd.conf?

    Code:
    #### CGI module                                                                                                                                              
    cgi.assign                 = ( ".cgi"  => "/bin/sh",                                                                                                         
                                   ".pl"   => "/opt/bin/perl" )
    The page to come up has only this requirement (new Oleg firm. is required to have more bash like function in /bin/sh that helps performance).

  11. #11
    Join Date
    Feb 2008
    Location
    Portugal
    Posts
    4
    I think the problem is that i don't have this version of oleg firmware.
    I have to update it first.
    Thanks.

  12. #12
    Quote Originally Posted by ecaddict View Post

    Download:
    Attachment 6410

    (updated on 15.05.2010, fixed UID issue)

    Take the following files from the archive under your webroot (e.g in my case /mnt/cgi-bin):
    /mnt/cgi-bin/fewgets.cgi
    /mnt/cgi-bin/qwget.sh
    /mnt/cgi-bin/plugins/...

    (older versions used, rget.sh, you can delet it, downloader plugins are used instead)
    Make the .sh files (also in plugin directory) executable via chmod +x

    Place the shutdown file to /usr/local/sbin
    /usr/local/sbin/shutdown
    make it executable via chmod +x and save it to flash via
    flashfs save && flashfs commit && flashfs enable

    and finally copy
    /opt/etc/init.d/rc.unslungstop
    /opt/etc/cron.5mins/traffic.sh
    as well and also make them executable via chmod +x
    (shutdown and rc.unslungstop is needed only if you'd like to use the stop tab, it's not needed for the download or for the filemanager tab)

    Open https://your_router's_address/cgi-bin/fewgets.cgi
    hi there, and thank you for your work till now.
    I have a rather strange/stupid question, and i can't figure it out myself.
    I have installes on RT-N16 - r2411 the HeadStart II image, using your indication and also installed rtorrent+rutorrent + FEWGETS.

    question: how do I figure out which version do I have installed ?

  13. #13
    Quote Originally Posted by bigphantom View Post
    question: how do I figure out which version do I have installed ?
    There ain't so many versions, so probably you have the latest one (for sure if you've installed it after the summer). If it works enjoy, if not I might help but nowadays I have really very limited time for this.

  14. #14
    I've updated fewgets, details are in the first post.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •