New oleg firmware version

Printable View

Show 40 post(s) from this thread on one page

08-11-2012, 13:43
lly

Quote:

Originally Posted by wpte

I've transferred over 400GB already, but I did noticed some hiccups.
EXT4 seems to work well overall, good performance etc.
But after a while (seems almost random) I get a segmentation fault in cp, the drive becomes inaccessible and the router is not able to umount it, even when the drive is removed.
I don't see any oops, maybe I should turn extra debugging on?
The router is also not able to reboot after the segfault.

In case of you can't reboot, it seems to be kernel problem.
It can be either ext4(jbd2) specific or common writeback/bdi interfaces problem. I never hear that somebody did such stress-test on router, 400Gb is really huge amount of data!

Best of all, if you able to prepare reproducible minimal test-case.
If problem is ext4 specific, than switching to ext3 will help. To enable ext4 debugging you have to recompile ext4.ko, jbd2.ko kernel modules with CONFIG_EXT4_DEBUG=y & CONFIG_JBD2_DEBUG=y

Also, please try r4652 - it is a small chance that I miss something in latest backports.

Anyway, "cp" shouldn't crash silently - please try to use gdb to get type of fault.
08-11-2012, 19:15
wpte

Quote:

Originally Posted by lly

In case of you can't reboot, it seems to be kernel problem.
It can be either ext4(jbd2) specific or common writeback/bdi interfaces problem. I never hear that somebody did such stress-test on router, 400Gb is really huge amount of data!

Best of all, if you able to prepare reproducible minimal test-case.
If problem is ext4 specific, than switching to ext3 will help. To enable ext4 debugging you have to recompile ext4.ko, jbd2.ko kernel modules with CONFIG_EXT4_DEBUG=y & CONFIG_JBD2_DEBUG=y

Also, please try r4652 - it is a small chance that I miss something in latest backports.

Anyway, "cp" shouldn't crash silently - please try to use gdb to get type of fault.

pff 400gig is nothing:p
But that's a total I copied. First I copied about 70GB when cp crashed, then after a hard-reboot I copied the rest without any problem.
I didn't found any corruption of the filesystem btw.

I remember that I had an incident with an other drive as well (a test ext4 drive). It was just spun down for a long time, it worked for days but then I wasn't able to umount or access it. I'm not sure if this is a stress problem.

I will run the cp command with gdb this time, and try the latest revision;)
08-11-2012, 23:44
wpte

Quote:

Originally Posted by lly

To enable ext4 debugging you have to recompile ext4.ko, jbd2.ko kernel modules with CONFIG_EXT4_DEBUG=y & CONFIG_JBD2_DEBUG=y

I get an error when I try that:

Code:

fs/jbd2/journal.c: In function 'journal_free_journal_head': fs/jbd2/journal.c:1880:13: error: 'JBD2_POISON_FREE' undeclared (first use in this function) fs/jbd2/journal.c:1880:13: note: each undeclared identifier is reported only once for each function it appears in

I've just set it with make menuconfig in the kernel after all the patches where applied (after gateway directory was created)
09-11-2012, 06:48
lly

Quote:

Originally Posted by wpte

I get an error when I try that:

Sorry, I must check it myself before to give advice to you. Will fix it ASAP.

Updated: Fixed in r4703
10-11-2012, 16:46
wpte

1 Attachment(s)

Quote:

Originally Posted by lly

Sorry, I must check it myself before to give advice to you. Will fix it ASAP.

Updated: Fixed in r4703

allright, I've loaded up the modules with debugging, and ran cp in gdb.
With gdb this is all I've got:

Code:

Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists.

I've attached the dmesg output. not sure if the debugging option helped that much:
Attachment 9330
2 oops's are logged.

I just noticed I also get quite a bit of IPv6 ICMP spoofing attempts :p
10-11-2012, 17:59
lly

Quote:

Originally Posted by wpte

allright, I've loaded up the modules with debugging, and ran cp in gdb.
With gdb this is all I've got:

Code:

Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists.

I've attached the dmesg output. not sure if the debugging option helped that much:

2 oops's are logged.

Thanks for info, looks like unaligned access cause wrong destination address calculation. Very strange that stack trace seems to be incomplete - memdup_user() can't be called from userspace directly. I will try to dig more deeper...

Could you provide output of

Code:

grep unaligned /proc/cpuinfo

?
10-11-2012, 23:29
wpte

Quote:

Originally Posted by lly

Thanks for info, looks like unaligned access cause wrong destination address calculation. Very strange that stack trace seems to be incomplete - memdup_user() can't be called from userspace directly. I will try to dig more deeper...

Could you provide output of

Code:

grep unaligned /proc/cpuinfo

?

Code:

unaligned_instructions : 687

Sorry, I rebooted my router, so I reproduced the problem.
Each time I execute that grep the unaligned instructions increase by 3 btw.
11-11-2012, 06:43
lly

Quote:

Originally Posted by wpte

Code:

unaligned_instructions : 687

Sorry, I rebooted my router, so I reproduced the problem.
Each time I execute that grep the unaligned instructions increase by 3 btw.

Strangely, when I use built-in /bin/grep - it not increases. Value of unaligned instructions on my router less than 10...
11-11-2012, 12:16
wpte

Quote:

Originally Posted by lly

Strangely, when I use built-in /bin/grep - it not increases. Value of unaligned instructions on my router less than 10...

Well on the RT-AC66U with stock firmware it's: 2873311
On there I've no disk attached or anything:p

I just checked the build-in grep and noticed that it didn't increase.
When I used the optware grep it increased again.
Right now it's >900 but the drive is still working.
When I rmmod the ext4, jdb2 and crc16 module it still behaves with a +3 increase.

Is the optware software compromised maybe?
11-11-2012, 12:23
staticroute

on r3323 unaligned instructions are increasing all the time:

Code:

Every 1.0s: cat /proc/cpuinfo |grep unalign Sun Nov 11 16:24:53 2012 unaligned_instructions : 88652442 unaligned_instructions : 88653829 unaligned_instructions : 88654494

i have l2tp connection and openvpn daemons running
11-11-2012, 12:43
theMIROn

Quote:

Originally Posted by staticroute

on r3323 unaligned instructions are increasing all the time:

Code:

Every 1.0s: cat /proc/cpuinfo |grep unalign Sun Nov 11 16:24:53 2012 unaligned_instructions : 88652442 unaligned_instructions : 88653829 unaligned_instructions : 88654494

i have l2tp connection and openvpn daemons running

in short, l2tp unaligned access was fixed long time ago, but after r3323.
13-11-2012, 23:58
wpte

no crashes so far ever since I updated to r4716
2 days without a crash, and I copied maybe 300GB.
Anyway, let's hope it'll stay that way ;)
14-11-2012, 06:35
lly

Quote:

Originally Posted by wpte

no crashes so far ever since I updated to r4716
2 days without a crash, and I copied maybe 300GB.
Anyway, let's hope it'll stay that way ;)

Thanks for good news. It is due to r4715,r4716 I hope.
23-11-2012, 00:01
wpte

1 Attachment(s)

Quote:

Originally Posted by lly

Thanks for good news. It is due to r4715,r4716 I hope.

Still going strong. I've been using my 4TB EXT4 disk as primary storage location for a few days now.
Soon I'll make a switch for /opt as well to an ssd so I'll have less wear on my big harddrive:D

I've no complaints but I'd like to give some feedback performance-wise.
I still use the debug-enabled modules maybe that's what causing extra load, but this is my load graph from the last week:
Attachment 9343
You can clearly see a load increase while I didn't do anything different than usual.
Again no complaints from my side, I'm happy with EXT4, but it's interesting for people who may want to switch.

In the weekend I'll start using the debug-less modules again I guess, since everything has been running smooth.
23-11-2012, 06:38
lly

Quote:

Originally Posted by wpte

You can clearly see a load increase while I didn't do anything different than usual.

My guess that is is due to our old bdi(backing device interface) kernel subsystem. Modern kernels has per-device bdi, our still use single kthread. Hope, I able to backport per-device bdi in futer, but it requires a lot of time & efforts.

Show 40 post(s) from this thread on one page