nzbget - binary newsgrabber

[hugbug]

Hi DrChair,

In my opinion, in case of broken files, the duplicates of those files should also be verified, to see if they are better (just like Quickpar does). Or is this also a limitation in libpar2 ??

Normally libpar2 (like par2cmdline) scans only files, listed in par2-file. But it is possible to load additional files (via special function) after the scan is completed. I'll see what I can do in your case. I will need to parse filenames to not load all files in a directory (this costs time), but only duplicate-files. Could you please send me your nzb-file for tests.

One last thing that is bothering me: If I add a lot of nzb's at once, the memory usage gets quite big (about 8 MB per dvd5).

It is strange, it should free the memory. Anyway I have already addressed this issue in a current (development) version. Now nzbget keeps articles' info on disk and loads it just before the download of a file starts. In my test I have added few big-nzbs (10-15GB data in each, about 60 GB all) and the memory consumption was as little as 2-3 MB for the whole program (Virt.mem usage reported by "top") and the usage of swap-file was 4 MB.

Finally, it might be useful the have a command to empty the whole queue (nzbget -E D -I * isn't working)

In current (development) version I have implemented new commands for group-editing (delete/move/pause/unpause all files belonging to the same nzb) with one simple command: e.g. "nzbget -E G D 1" deletes all files from the nzb-group, which the file with ID=1 belongs. You can do this also via curses-frontend (use G-key to switch group-view on/off).

If you have compile environment for optware, you can get the mentioned version of nzbget from svn using command:

svn co --revision 52 http://nzbget.svn.sourceforge.net/svnroot/nzbget/trunk nzbget-0.3.1-testing

Then run "configure" and "make dist", copy created file "nzbget-0.3.1-testing.tar.gz" to "optware/downloads", edit version number in "optware/make/nzbget.mk" to "NZBGET_VERSION=0.3.1-testing" and compile the package.

Or I can send you the compiled ipk-package instead
Let me know if you want to test this version.
ChangLog for revision 28, not all new features listed, see svn change log for changes after revision 28.
Note: the communication protocol was changed in version 0.3.1, you will not be able to use windows-version 0.3.0 with server 0.3.1. I could send you the updated compiled windows-version too.
I'm going to release the version 0.3.1 in a few next weeks.

[DrChair]

nevermind the memory issues...

I just found the svn repository and compiled 0.3.1+r52
The result: still using only 10 MB with 9 nzbs in queue

Just a cosmetic tip: add a \n at line 721 (and perhaps line 722) in Options.cpp

And I just found a nzb that gets parsed wrong. When in queue, it displays only a part of the subject, instead of the filenames. Also it can't see the difference between rar's and par's, so it doesn't pause the par's.

I'll pm you that nzb.

[norberto]

Hi all,

I installed and configured nzbget on my Synology DS-106 and I'm having the following error message when trying to launch it (in server or client mode) :

DiskStation> nzbget -s
FATAL ERROR: Invalid option "DaemonUserName"

I tried to change the DaemonUserName in the configuration file
DaemonUserName=root
with "admin" or any user name from my Syno but it doesn't change anything.

I'm sure I'm missing something simple...

Can you help ?

Thanks.

[hugbug]

DiskStation> nzbget -s
FATAL ERROR: Invalid option "DaemonUserName"

Looks like you are trying to use a configuration file from a different (newer) program version, as installed.
You probably installed the program from a package-repository (version 0.3.0), but took the configuration file from svn/trunk (current development version).

You should have an appropriate configuration file example installed in "/opt/share/doc/nzbget/nzbget.conf.exampe" (it comes with optware-package). It can be also downloaded from svn/tags/0.3.0.

Or you can just comment out (with "#"-character) all (new) options the program complains about.

[norberto]

You're right : I followed the instructions found on the Synology forum and downloaded a wrong configuration file.

It seems it's working fine now.

Thanks a lot for your prompt answer !

[hugbug]

norberto, great.

DrChair

And I just found a nzb that gets parsed wrong. When in queue, it displays only a part of the subject, instead of the filenames. Also it can't see the difference between rar's and par's, so it doesn't pause the par's.

The parsing of artcile's subject was improved for better extracting of filename part from it. This fixes the error with parsing of the mentioned file.

I was downloading a collection which I got from exporting to nzb in Newsleecher. This collection contains a few broken files and the complete reposted versions of those broken files.
The result: a broken file.part17.rar and a complete file.part17.rar_duplicate1

There is now a dupe check for files from the same nzb-request to detect reposted files. Now the program chooses the best (biggest) file from them. This works only if there were exactly two files with the same name. (if there were more, the program may decide, it has inccorectly parsed filenames from subjects and may try to use full subjects as filenames). You need to activate the option "dupecheck" in config-file.

Please take the new version from svn.

[DrChair]

Thanx... I was already using newest svn version and I saw you fixed it, but I didn't know the option DupeCheck needed to be enabled.

I thought (as explained in nzbget.conf.example) that that option would only check for dupes on disk, but now you made it also check for dupes in the nzb.

Perhaps you need to update the nzbget.conf.example with this info?

[hugbug]

Perhaps you need to update the nzbget.conf.example with this info?

Thanks for the tip, I forgot about it. I'll update the description.

[DrChair]

Found a few bugs:

Like i already pmed you, the new decoding in svn r57 only works when DirectWrite is disabled.

Second bug: when I cancel a download (I choose Group -> Edit -> select the one to cancel -> Delete), it logs the following:

INFO Collection XXXXXXXX completely downloaded
INFO Queueing XXXXXXXX for par-check
INFO Checking pars for XXXXXXXXXXX

Also it doesn't delete the already downloaded articles from tmp, atleast with DirectWrite=no, I don't know what happens with DirectWrite=yes, cause thats broken atm

This is only the case for downloads that have been started (i.e., already downloaded atleast one article)

[hugbug]

DrChair, many thanks for testing and bug reports!

the new decoding in svn r57 only works when DirectWrite is disabled.

Fixed.

Second bug: when I cancel a download (I choose Group -> Edit -> select the one to cancel -> Delete), it logs the following:
INFO Collection XXXXXXXX completely downloaded
INFO Queueing XXXXXXXX for par-check
INFO Checking pars for XXXXXXXXXXX

I would say it is not a bug but "behaviour by design", but the design seems to be not so good now
PostProcessor reacts to event "last unpaused file from collection was downloaded OR deleted = there no more unpaused files for this collection"). Then it starts par-checker and/or executes postprocess script.
I can change it to react to only completed files before starting par-check.

But should the postprocess-script also not be executed if the last file was deleted?

[hugbug]

when I cancel a download (I choose Group -> Edit -> select the one to cancel -> Delete), it logs the following:
INFO Collection XXXXXXXX completely downloaded
INFO Queueing XXXXXXXX for par-check
INFO Checking pars for XXXXXXXXXXX

Fixed in r59. Now par-checking is not started by deleting of files or collections.

Also it doesn't delete the already downloaded articles from tmp

I'll look into it. But this is not a big problem, because on next start server will delete orphaned files from temp dir.

[allochtheun]

I've installed this great package. The install is going good. but when I start nzbget -D and put an NZB file into the right dir.
The logfile is telling me that it cannot create the queue en nzbget.log file.
I find that rather strange because I've fully access to the dir (/tmp/harddisk/download). and I can create en modify files/dir's through ftp
I double checked if the dir's exist.
Other programs like lighttpd en vsftpd also writing to /tmp/harddisk and there i've no issues with

At the moment I'm not capable of posting some examples ,like (df , mount or logfile's) I'will post them tonight if nessecary.
Hope someone could help me with this one.

Greetings

Allochtheun

[hugbug]

IThe logfile is telling me that it cannot create the queue en nzbget.log file.

Please check if all necessary directories exist, expecially the main directory (option "$MAINDIR" in config file). The program does not create the main dir automatically (this will be fixed in next version).

[allochtheun]

Please check if all necessary directories exist, expecially the main directory (option "$MAINDIR" in config file). The program does not create the main dir automatically (this will be fixed in next version).

I've checked that. They exists I created "$MAINDIR":/tmp/harddisk/downloads
I've manualy added the dir's such as queue etc.
I found it rather strange that he tels me that hij cannot write to /tmp/harddisk/downloads/queue/queue. Why is there twice the queue name.?
I'wil post some screendumps tonight so you can see.

Thnx anyway for your quick reply

Allochtheun

[hugbug]

I found it rather strange that he tels me that hij cannot write to /tmp/harddisk/downloads/queue/queue. Why is there twice the queue name.?

It creates a file named "queue" in dir "queue" So it's OK to have this name twice in the error message. Strange is why it cannot write to the directory.
You better trying to start the program in console mode (not daemon) to see error messages - use "nzbget -s".

To check all things again:
1) create completely new directory, for example "/tmp/harddisk/downloads/test";
2) copy nzbget.conf.example to "/tmp/harddisk/downloads/test/nzbget.conf";
3) edit "/tmp/harddisk/downloads/test/nzbget.conf" and change ONLY "$MAINDIR" to "/tmp/harddisk/downloads/test";
4) start the program:

nzbget -c /tmp/harddisk/downloads/test/nzbget.conf -s

What it says now?