PDA

Bekijk de volledige versie : Tool: Auto-copy images from USB-device and index-image creation



fredlcore
24-10-2006, 00:23
Hi everybody,

I liked the function of Asus' original software to automagically copy the contents of a USB-device (SD-card reader for example) to an incremental directory on the harddisk of my WL-HDD.
Of course, Oleg's firmware outweighed Asus' original, but still I kind of wanted this feature.
Through some help of this forum, I managed to come up with a set of scripts that helps me do so.

Advantages:

1. It works also with multi-card-readers or multi-partition devices that are not detected properly by the automounter
2. It copies the contents of the storage device to an incremental directory on the harddisk that is easy to recognize (date-time format)
3. It also creates an jpeg-image which contains the contents of the folder of the harddisk where the files have been backup-ed to and writes it back to the storage device.

Why no. 3?
Imagine you are out in the wild and want to backup your SD-card of your camera because it's full and only brought your WL-HDD because it's so small. You just plug your card-reader in and you see the lights flashing, but how can you be sure that everything went well during the copying process?
The image that is created and written back to the drive contains the directory output and can (or should) be viewable in your camera for example. When you see that all files have proper sizes and are complete, you can safely format your SD-Card and don't have to worry until the end of your trip to see if everything REALLY went fine :)

Enough talk, here are the scripts, they all assume that your internal hard-drive is mounted to /opt:

Insert the following lines into /tmp/local/sbin/post-boot:


insmod /lib/modules/2.4.20/kernel/drivers/scsi/scsi_mod.o
insmod /lib/modules/2.4.20/kernel/drivers/scsi/sd_mod.o
insmod /lib/modules/2.4.20/kernel/drivers/usb/storage/usb-storage.o


and


echo /opt/utils/hotplug > /proc/sys/kernel/hotplug

The first three lines are needed only if you turned off the ftp-server in the web-interface. If your system already automagically mounts your storage device to /tmp/harddisk, you can omit them.

Create the file /opt/utils/hotplug:


#!/bin/sh

# check for need to mount usb-drive

if [ $ACTION = "add" ]; then
devs=`find /dev -name "part?"`;
# e.g. /dev/scsi/host0/bus0/target0/lun3/part1
for dev in $devs; do
part=`echo $dev | sed -e "s/.*part\(.*\)/\1/"`;
# e.g. 1 if part1
dev=`echo $dev | sed -e "s/.*dev\(.*\)\/part.*/\1/"`;
# e.g. /scsi/host0/bus0/target0/lun3
disc=`ls -l /dev/discs | grep $dev | sed -e 's/.* disc\(.*\) ->.*/\1/`;
# e.g. 4 if $dev maps /dev/discs/disc4

# check if already in mount-tab
result=`mount | grep "/dev/discs/disc$disc/"`;
if [ $? = 1 ]; then
echo "Not mounted: $disc";
mounted=0;
while [ $mounted -lt 1 ]; do
dest="/tmp/harddisk$count";
result=`mount | grep "$dest"`;
# if directory exists, increase counter
if [ $? = 0 ]; then
let count=count+1;
else
mounted=1;
fi
done
mkdir -p "/tmp/harddisk$count";
mount /dev/discs/disc$disc/part$part /tmp/harddisk$count
else
echo "disc$disc already mounted!"
fi
done
fi

# unmount "dead" mountpoints

if [ $ACTION = "remove" ]; then
devs=`mount | grep /part | sed -e "s/\(.*\) on .*/\1/"`;
# e.g. /dev/discs/disc4/part1
for dev in $devs; do
if [ \! -e $dev ]; then
dir=`mount | grep $dev | sed -e "s/.*on \(.*\) type.*/\1/"`;
# e.g. /tmp/harddisk
umount $dir;
fi
done
fi

/sbin/hotplug $*


This file checks for devices in the system and mounts them to /tmp/harddisk (or /tmp/harddisk1[2, 3, 4, ...] if already in use), if this hasn't already happened. Use this if your storage device doesn't come up mounted automagically.

For image-backup, create first /tmp/local/sbin/post-mount:


#!/bin/sh

dirs=`ls /tmp | grep harddisk`;
for dir in $dirs; do
if [ -e /tmp/$dir/dcim ]; then
folder=`date +%Y%m%d_%H%M%S`;
mkdir -p /opt/backup/$folder;
cp -ar /tmp/$dir/dcim/* /opt/backup/$folder;
# this php script requires ipkg-packages gdlib, php, php-gd and cyrus-sasl
ls -lR /opt/backup/$folder | /opt/utils/reportjpg.php;
# modify here according to your camera
cp /opt/backup/result.jpg /tmp/$dir/dcim/100nikon/dscn0001.jpg
umount /tmp/$dir
fi
done


This checks for a directory called "DCIM" (almost all digital cameras use this), and if so, copy all files to /opt/backup/[timestamp], create a report-image in jpg-format and write it back to the camera as dscn0001.jpg (change this according to your camera model). Then unmount the drive.
Unfortunately, my Nikon camera doesn't display this image, it says "this image contains no data", but in Windows Explorer it can be seen, so I assume my Nikon checks some other file attributes. If someone has a solution, please let me know.

To create the image with the directory output, I had to resort to a php-script because I can't code in C to use libgd and libjpeg from there. Would be much faster and more convenient, but that's up to someone else :)
You need the following ipkg-packages: gdlib, php, php-gd, cyrus-sasl (because of some stupid dependencies).

Put this code under /opt/utils/reportjpg.php:

#!/opt/bin/php
<?
while (!feof(STDIN)) {
$input = trim(fgets(STDIN));
$res[] = $input;
}
$im=imagecreate(2048,1536);
imagecolorallocate($im,0,0,0);
$col=imagecolorallocate($im,255,255,255);
$y=0; $x=0;
foreach ($res as &$line) {
imagestring($im,1,$x,$y,$line,$col);
$y=$y+12;
if ($y > 1500) {
$x=$x+400;
$y=0;
}
}
imagejpeg($im,"/opt/backup/result.jpg",100);
imagedestroy($im);
?>


Don't forget to chmod 755 all files and save configuration with
flashfs save; flashfs commit; flashfs enable;
Then reboot to activate hotplug changes.

Hope you enjoy this piece of code and if someone could come up with a more ressourceful way of writing the jpg-report-image, please let me know :)!

CU!

Frederik.

juergen
30-10-2006, 00:02
Hi Frederik,

great work, thank you.

I dont have some experience in scripting (worse luck). I would like copy pictures from some different cameras. This cameras are have different folder names after dcim/. For that there is a problem with the automaticly created content picture. Your script copy it to dcim/100nikon. Is it possible to find out the name(s) from the folder(s) in dcim/ which includ the pictures and where the script can copy the content picture?

Because the WL-HDD USB 1.1 transmission is very slow it would be nice to have an optical indicator whether the copy process is still running. I'm looking for a script or executable or code snippet which let me controll one of the LED from the WL-HDD ( the USB-LED would be fine) but I don't find some one.
If one of you have a solution please let me know.

Thanks again (also to Oleg)
Juergen

fredlcore
30-10-2006, 13:25
Hi Frederik,
great work, thank you.


Thanks.


Is it possible to find out the name(s) from the folder(s) in dcim/ which includ the pictures and where the script can copy the content picture?

Unfortunately, not really. My Nikon for example has at least two directories, 100nikon and misc. If I take panorama pictures, they will be put in 100p1nikon (or something like that). Other cameras have probably similar multitudes of folder names, so it would really blow up the script.
You could do something semi-automatic like defining a variable at the beginning which is set according to the dcim contents.
I just write it from memory, so no guarantee it'll work:


#!/bin/sh

dirs=`ls /tmp | grep harddisk`;
for dir in $dirs; do
if [ -e /tmp/$dir/dcim ]; then
folder=`date +%Y%m%d_%H%M%S`;
mkdir -p /opt/backup/$folder;
if [ -e /tmp/$dir/dcim/100nikon ]; then
destination = "100nikon";
filename = "dscn0001.jpg";
fi
if [ -e /tmp/$dir/dcim/canon ]; then
destination = "100canon";
filename = "p00001.jpg";
fi
cp -ar /tmp/$dir/dcim/* /opt/backup/$folder;
# this php script requires ipkg-packages gdlib, php, php-gd and cyrus-sasl
ls -lR /opt/backup/$folder | /opt/utils/reportjpg.php;
# modify here according to your camera
cp /opt/backup/result.jpg /tmp/$dir/dcim/$destination/$filename
umount /tmp/$dir
fi
done


Please give feedback here if it works as desired.



Because the WL-HDD USB 1.1 transmission is very slow it would be nice to have an optical indicator whether the copy process is still running. I'm looking for a script or executable or code snippet which let me controll one of the LED from the WL-HDD ( the USB-LED would be fine) but I don't find some one.
If one of you have a solution please let me know.


I'd just use a card-reader with a built-in LED (mine cost less than 5 Euros). If it stops flashing, I can plug it off.

But you could also do something like that in the script:
Line 3:
Turn off Power-LED: echo 1 > /dev/gpio/out
Last line:
Turn on Power-LED: echo 0 > /dev/gpio/out

So you'll know if the Power-LED is back on again, the process is finished.

CU!

Frederik.

juergen
31-10-2006, 01:09
Hi Frederik,

it works fine.

I change /tmp/local/sbin/post-mount as following:



#!/bin/sh
#Auto-copy images from USB-device and index-image creation
#Found in thread http://wl500g.info/showthread.php?t=6826
dirs=`ls /tmp | grep harddisk`;
for dir in $dirs; do
if [ -e /tmp/$dir/dcim ]; then

#Start LED flashing
/opt/utils/WL-HDDPowerLedFlash.sh&
folder=`date +%Y%m%d_%H%M%S`;
mkdir -p /opt/backup/$folder;
# for my Nikon
if [ -e /tmp/$dir/dcim/100nikon ]; then
destination="100nikon";
filename="dscn0001.jpg";
fi
# for my Sony
if [ -e /tmp/$dir/dcim/101msdcf ]; then
destination="101msdcf";
filename="dscn0001.jpg";
fi
cp -ar /tmp/$dir/dcim/* /opt/backup/$folder;
# this php script requires ipkg-packages gdlib, php, php-gd and cyrus-sasl
ls -lR /opt/backup/$folder | /opt/utils/reportjpg.php;
# modify here according to your camera
cp /opt/backup/result.jpg /tmp/$dir/dcim/$destination/$filename
umount /tmp/$dir

#Stop LED flashing
killall WL-HDDPowerLedFlash.sh
fi
done


and create a little script /opt/utils/WL-HDDPowerLedFlash.sh for LED flashing:



#!/bin/sh

# Signal-Handle
cleanup()
{
#switch on PWR LED
echo 0 > /dev/gpio/out;
exit
}

# Signal-Handle "cleanup" link with EXIT TERM and INT
trap cleanup EXIT TERM INT

while :
do

#switch off PWR LED
echo 1 > /dev/gpio/out;
sleep 1

#switch on PWR LED
echo 0 > /dev/gpio/out;
sleep 1

done


This lets the Power-LED flash during the copy process.


I know that a shell script is very nice but I don't like it:

destination="100nikon"; are not
destination = "100nikon"; :confused:


Thanks a lot
Juergen



Don't forget to chmod 755 all files and save configuration with
flashfs save; flashfs commit; flashfs enable;

fredlcore
31-10-2006, 07:08
create a little script /opt/utils/WL-HDDPowerLedFlash.sh for LED flashing:
This lets the Power-LED flash during the copy process.


Nice, but how come you can write a script like that if you -as you say - don't have experience in scripting ;)?

In fact it seems quite necessesary to me now to have some optical signal because the process to create the result.jpg takes quite some time and then the USB-Stick's LED stops flashing, so it's not a really safe indicator (although one could wait until it starts flashing again shortly when the jpg is written to the SD-Card and then unplug after flashing finally stops).



I know that a shell script is very nice but I don't like it:
destination="100nikon"; are not
destination = "100nikon"; :confused:


Well, as I said, I wrote it from memory without testing it. I also realized it when putting it into action back home. Sorry.

Frederik.

gdraanen
01-02-2007, 22:22
Interesting mails, but some problems to further test and evaluate this.
- The ipkg packages as referred do not seem to be available (they are not in ipkg list ). Neither are /opt/utils/....

- I would like to manually 'start' the copy script by shortly pressing the (only) button on the wl-hdd (more than half a second, less than 3 seconds ...).
There seems to be an /dev/gpio/in which changes its output in relation to that button.
But I'm not at all comfortable with writing a script around that button.

- I see there's a /sbin/hotplug which is in binary format. It has no man page, and no -h or -? option. So I'm absolutely not sure if e.g. you/Oleg decided to support the 'hotplug' in the firmware distribution ??

Gerard.

martin700
14-05-2007, 16:45
Unfortunately, my Nikon camera doesn't display this image, it says "this image contains no data", but in Windows Explorer it can be seen, so I assume my Nikon checks some other file attributes. If someone has a solution, please let me know.

I have a Nikon too, with the same problem. After some experimenting, I found that the problem is the jpeg sampling.
Libjpeg produces 2x2,1x1,1x1 samples by default. The Nikon reads only 2x1,1x1,1x1 pictures. Although libjpeg has a function to change the sampling, gdlib has no way to adjust this !

So I patched the libjpeg source and changed the default to 2x1....!
Then compiled libjpeg and exchanged it on my WL-HDD

And now my Nikon shows the picture in full screen mode!

Cheers Martin

You have to edit the file jcparam.c line 437 and line 452 from


SET_COMP(0, 1, 2,2, 0, 0,0);
to
SET_COMP(0, 1, 2,1, 0, 0,0);

fredlcore
14-05-2007, 16:52
Hi Martin,



So I patched the libjpeg source and changed the default to 2x1....!
Then compiled libjpeg and exchanged it on my WL-HDD


thanks for the info, this really sounds great. Do you think you could give a step-by-step instruction on how to do this directly on the WL-HDD so one doesn't have to do a cross-compiling on another machine?

Or simply upload the patched version (or a .diff-file) so others can just apply that?

Would help a lot : )!

Thanks,

Frederik.

martin700
16-05-2007, 19:04
Hi Martin,



thanks for the info, this really sounds great. Do you think you could give a step-by-step instruction on how to do this directly on the WL-HDD so one doesn't have to do a cross-compiling on another machine?

Or simply upload the patched version (or a .diff-file) so others can just apply that?

Would help a lot : )!

Thanks,

Frederik.

The simplest way to do it, is to exchange the library. Of course mine is compiled for Olegs firmware and I doubt that it works in any other setting. You can download it here and exchange it with yours in :
/opt/lib/. (A backup of the original would be a good idea)

cheers Martin

fredlcore
16-05-2007, 22:40
The simplest way to do it, is to exchange the library. Of course mine is compiled for Olegs firmware and I doubt that it works in any other setting. You can download it here and exchange it with yours in :
/opt/lib/. (A backup of the original would be a good idea)

Thanks for the file, but although I also have Oleg's firmware (1.9.2.7-7f) on my WL-HDD, it gives me the following errors:


PHP Warning: PHP Startup: Unable to load dynamic library '/opt/lib/php/extensions/gd.so' - File not found in Unknown on line 0
PHP Fatal error: Call to undefined function imagecreate() in /opt/utils/reportjpg.php on line 7

When I copy back the original libjpeg.so.62.0.0, all runs fine...
Do you use a different hardware and/or firmware?

Any ideas?

Frederik.

wwe4lyfe
20-08-2007, 08:50
Hi and thanks for your script, however, before I use it there are a few things I would like to know...

Would it be possible to copy ANY and ALL files from a usb disc (be it memory card or USB stick) as I have a memory card for my mobile phone, and it uses different directories to store files, and also I would quite like it to backup from any USB card / stick I insert.

If this could be done, you would be my saviour and hero :)

tobitobsen
02-10-2007, 13:54
I have a problem getting this up and running
Can't store the hook....

echo /opt/utils/hotplug > /proc/sys/kernel/hotplug
So if i execute that line it looks fine.
But if i take a look at it with vi

vi /proc/sys/kernel/hotplug
the file is still empty. I can edit it according to the command above - but after the next reboot it's gone.
And yes - I used flashfs save && flashfs commit && flashfs enable ... ;)


Any ideas?

- Tobias -

mistraller
02-10-2007, 21:46
I have a problem getting this up and running
Can't store the hook....

echo /opt/utils/hotplug > /proc/sys/kernel/hotplug
So if i execute that line it looks fine.
But if i take a look at it with vi

vi /proc/sys/kernel/hotplug
the file is still empty. I can edit it according to the command above - but after the next reboot it's gone.
And yes - I used flashfs save && flashfs commit && flashfs enable ... ;)


Any ideas?

- Tobias -

That's why you should add this to post-boot.

tobitobsen
03-10-2007, 13:18
ok - so the router would try to copy the files on reboot and not on plug-in?

I tried without the hotplug-script since the rooter does some 'automount' itself. But the even if I write the copy script into !!post-mount!! it only runs once until next reboot.

So no real "plug and copy" for both ways? :(

fredlcore
03-11-2007, 10:42
ok - so the router would try to copy the files on reboot and not on plug-in?

No, you need to put the command
echo /opt/utils/hotplug > /proc/sys/kernel/hotplug
in your /tmp/local/sbin/post-boot (as it is written above) and then save the flash with flashfs save; flashfs commit; flashfs enable;
Then reboot to activate hotplug changes.
After that, you should be able to see the command in /proc/sys/kernel/hotplug and the script should run when a device is plugged in.

F.

fredlcore
03-11-2007, 10:56
Would it be possible to copy ANY and ALL files from a usb disc (be it memory card or USB stick) as I have a memory card for my mobile phone, and it uses different directories to store files, and also I would quite like it to backup from any USB card / stick I insert.

Sorry for the long delay, I was overseas and did not have time to look at this.
Just replace this script with the one above that goes into post-mount:


#!/bin/sh

dirs=`ls /tmp | grep harddisk`;
for dir in $dirs; do
folder=`date +%Y%m%d_%H%M%S`;
mkdir -p /opt/backup/$folder;
cp -ar /tmp/$dir/* /opt/backup/$folder;
umount /tmp/$dir
done

Please report back if it works according to your needs, I have no way of testing it now...

F.

tobitobsen
15-12-2007, 08:13
No, you need to put the command
echo /opt/utils/hotplug > /proc/sys/kernel/hotplug
in your /tmp/local/sbin/post-boot (as it is written above) and then save the flash with flashfs save; flashfs commit; flashfs enable;
Then reboot to activate hotplug changes.
After that, you should be able to see the command in /proc/sys/kernel/hotplug and the script should run when a device is plugged in.

F.

Sorry for the late posting. I was quite busy th last months.

if I run "echo /opt/utils/hotplug > /proc/sys/kernel/hotplug" from the cmd and than have a look with "vi /proc/sys/kernel/hotplug". I can only see a blank file. :confused: