Îøèáêà "page allocation failure" íà ïðîøèâêàõ RT-N16-1.9.2.7-rtn

[ryzhov_al]

Êîëëåãè, âðåìÿ îò âðåìåíè íà ôîðóìå âñïëûâàþò ñîîáùåíèÿ îá îøèáêå kernel: swapper: page allocation failure.

Oct  2 23:10:21 kernel: swapper: page allocation failure. order:3, mode:0x4020
Oct  2 23:10:21 kernel: Call Trace:
Oct  2 23:10:21 kernel: [<8000de20>] dump_stack+0x8/0x34
Oct  2 23:10:21 kernel: [<8005914c>] __alloc_pages+0x294/0x344
Oct  2 23:10:21 kernel: [<80077438>] __slab_alloc+0x1d8/0x4b8
Oct  2 23:10:21 kernel: [<80077a34>] __kmalloc_track_caller+0x144/0x184
Oct  2 23:10:21 kernel: [<801a8a30>] __alloc_skb+0x70/0x140
Oct  2 23:10:21 kernel: [<8020badc>] tcp_collapse+0x214/0x468
Oct  2 23:10:21 kernel: [<8020be80>] tcp_prune_queue+0x150/0x340
Oct  2 23:10:21 kernel: [<8020c778>] tcp_data_queue+0x708/0xe10
Oct  2 23:10:21 kernel: [<8020e594>] tcp_rcv_established+0x224/0xc00
Oct  2 23:10:21 kernel: [<802167dc>] tcp_v4_do_rcv+0xe8/0x450
Oct  2 23:10:21 kernel: [<80217598>] tcp_v4_rcv+0xa54/0xb38
Oct  2 23:10:21 kernel: [<801f5604>] ip_local_deliver+0x1e8/0x318
Oct  2 23:10:21 kernel: [<801f4810>] ip_rcv_finish+0x1d0/0x464
Oct  2 23:10:21 kernel: [<801db934>] nf_reinject+0x11c/0x2b8
Oct  2 23:10:21 kernel: [<c014e1ec>] imq_dev_xmit+0x50/0x60 [imq]
Oct  2 23:10:21 kernel: [<801c1958>] qdisc_restart1+0xb4/0x21c
Oct  2 23:10:21 kernel: [<c014e360>] imq_nf_queue+0x164/0x264 [imq]
Oct  2 23:10:21 kernel: [<801db608>] __nf_queue+0x17c/0x38c
Oct  2 23:10:21 kernel: [<801dab20>] nf_hook_slow+0xe0/0x160
Oct  2 23:10:21 kernel: [<801f4d54>] ip_rcv+0x2b0/0x720
Oct  2 23:10:21 kernel: [<801afaa4>] process_backlog+0xf0/0x220
Oct  2 23:10:21 kernel: [<801ad1b0>] net_rx_action+0xb4/0x21c
Oct  2 23:10:21 kernel: [<8002d28c>] __do_softirq+0x88/0x11c
Oct  2 23:10:21 kernel: [<8002d38c>] do_softirq+0x6c/0x74
Oct  2 23:10:21 kernel: [<80007614>] plat_irq_dispatch+0x14c/0x1b8
Oct  2 23:10:21 kernel: [<80007d6c>] ret_from_irq+0x0/0x4
Oct  2 23:10:21 kernel: [<80009d1c>] cpu_idle+0x1c/0x50
Oct  2 23:10:21 kernel: [<80328c00>] start_kernel+0x2e8/0x344
Oct  2 23:10:21 kernel: Mem-info:
Oct  2 23:10:21 kernel: Normal per-cpu:
Oct  2 23:10:21 kernel: CPU    0: Hot: hi:   42, btch:   7 usd:   3   Cold: hi:   14, btch:   3 usd:   6
Oct  2 23:10:21 kernel: Active:14580 inactive:10447 dirty:1524 writeback:0 unstable:0
Oct  2 23:10:21 kernel:  free:2544 slab:1628 mapped:2695 pagetables:128 bounce:0
Oct  2 23:10:21 kernel: Normal free:10176kB min:1440kB low:1800kB high:2160kB active:58320kB inactive:41788kB present:130048kB pages_scanned:0 all_unreclaimable? no
Oct  2 23:10:21 kernel: lowmem_reserve[]: 0
Oct  2 23:10:21 kernel: Normal: 0*4kB 654*8kB 305*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 10176kB
Oct  2 23:10:21 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  2 23:10:21 kernel: Free swap  = 0kB
Oct  2 23:10:21 kernel: Total swap = 0kB
Oct  2 23:10:21 kernel: Free swap:            0kB
Oct  2 23:10:21 kernel: 32768 pages of RAM
Oct  2 23:10:21 kernel: 0 pages of HIGHMEM
Oct  2 23:10:21 kernel: 1132 reserved pages
Oct  2 23:10:21 kernel: 22161 pages shared
Oct  2 23:10:21 kernel: 0 pages swap cached

Óñëîâèÿ âîçíèêíîâåíèÿ îøèáêè:

• ïðîøèâêà RT-N16-1.9.2.7-rtn-rXXXX,
• èñïîëüçàíèå ðîóòåðà äëÿ ñêà÷èâàíèÿ òîððåíòîâ,
• ñêîðîñòü ñêà÷èâàíèÿ >~200KB/s.

Èñïîëüçîâàíèå ñâîïà îò îøèáêè íå èçáàâàëÿåò. Ïðè âîçíèêíîâåíèè îøèáêè ðÿä ñêà÷àííûõ ôàéëîâ îêàçûâàþòñÿ "áèòûìè". Ïðè÷¸ì îøèáêè âîçíèêàþò êàê ïðè èñïîëüçîâàíèè rtorrent, òàê è ïðè èñïîëüçîâàíèè transmission.  ìîìåíò âîñïðîèçâåäåíèÿ ïðîáëåìû ðîóòåð íå òîðìîçèò - íè â WEB-èíòåðôåéñå, íè â êîíñîëè. Åäèíñòâåííîå, ÷òî ìîæíî çàôèêñèðîâàòü - ýòî æóòêî äîëãîå âûïîëíåíèå ëþáîé îïåðàöèè, ñâÿçàííîé ñ ÷òåíèåì/çàïèñüþ íà äèñê ïðè ñêà÷èâàíèè òîððåíòîâ.

×èñëî îòêðûòûõ äèñêðèïòîðîâ äëÿ ôàéëîâîé ñèñòåìû äèñêà:

$ lsof +D /tmp/harddisk/ | wc -l
287

Èç íèõ îòêðûòî íà çàïèñü:

$ lsof +D /tmp/harddisk/ | grep "w " | wc -l
2

Äåðåâî ïðîöåññîâ:

$ ps afx
  PID TTY      STAT   TIME COMMAND
    2 ?        S<     0:00 [kthreadd]
    5 ?        S<     0:00  \_ [khelper]
    4 ?        S<     0:00  \_ [events/0]
    3 ?        RN     0:00  \_ [ksoftirqd/0]
   24 ?        S<     0:00  \_ [kblockd/0]
   52 ?        S<     0:00  \_ [kswapd0]
   53 ?        S<     0:00  \_ [aio/0]
   51 ?        S      0:00  \_ [pdflush]
   50 ?        S      0:00  \_ [pdflush]
  105 ?        S<     0:01  \_ [mtdblockd]
  238 ?        S<     0:00  \_ [khubd]
  297 ?        S<     0:00  \_ [scsi_eh_0]
  298 ?        D<     0:00  \_ [usb-storage]
  336 ?        S<     0:05  \_ [kl2tpd/0]
  385 ?        D<     0:00  \_ [kjournald]
    1 ttyS0    Ss+    0:00 /sbin/init
  199 ?        Ss     0:00 syslogd -m 0 -O /tmp/syslog.log -S -D -l 7 -b 1
  202 ?        Ss     0:00 klogd
  205 ?        S      0:00 nas
  209 ?        Ss     0:00 telnetd
  213 ?        S      0:00 dropbear -p 22 -4
  594 ?        Ss     0:00  \_ dropbear -p 22 -4
  595 pts/0    Ss     0:00  |   \_ -sh
  596 pts/0    S+     0:07  |       \_ top
 1270 ?        Ss     0:00  \_ dropbear -p 22 -4
 1362 pts/1    Ss     0:00      \_ -sh
 1660 pts/1    R+     0:00          \_ ps afx
  216 ?        S      0:00 httpd vlan2
  221 ?        S      0:00 dnsmasq
  245 ?        S      0:00 lld2d br0 eth1
  327 ?        Ss     0:00 udhcpc -i vlan2 -p /var/run/udhcpc0.pid -s /tmp/udhcp
  330 ?        S      0:00 l2tpd
  337 ?        S      0:00  \_ pppd sync nodetach noaccomp nobsdcomp nodeflate n
  334 ?        Ss     0:00 watchdog
  366 ?        S      0:00 upnp -D -L br0 -W ppp0
  388 ?        Ss     0:00 /usr/sbin/vsftpd
  390 ?        Ss     0:00 /usr/sbin/nmbd -D
  392 ?        Ss     0:00 /usr/sbin/smbd -D
  404 ?        Ss     0:00 cron
  548 ?        S      0:00 /opt/bin/bash /opt/bin/dlengine
 1662 ?        S      0:00  \_ sleep 30
  559 ?        S<     0:00 lighttpd -f /opt/etc/lighttpd/lighttpd.conf
  560 ?        S<s    0:00  \_ /opt/bin/php-fcgi
  573 ?        S<     0:00      \_ /opt/bin/php-fcgi
  574 ?        S<     0:00      \_ /opt/bin/php-fcgi
  571 ?        S<s    0:00 darkstat -i ppp0 -p 667 --chroot /opt/var/log
  572 ?        S<s    0:00  \_ darkstat -i ppp0 -p 667 --chroot /opt/var/log
  585 ?        S      0:00 transmission-daemon -g /opt/etc/transmission
  586 ?        S      0:00  \_ transmission-daemon -g /opt/etc/transmission
  587 ?        D      0:25      \_ transmission-daemon -g /opt/etc/transmission
  588 ?        S      0:00      \_ transmission-daemon -g /opt/etc/transmission

Ìîè äîãàäêè: â êàêîé-òî ìîìåíò ïàìÿòü RAM ëàâèíîîáðàçíî çàïîëíÿåòñÿ, à swapper íå óñïåâàåò âûäåëèòü íóæíûé îáú¸ì ïàìÿòè íà äèñêå. Ýòî âèäíî ïî ïðèëîæåííîìó ãðàôèêó: â 21:40 áûëî ñíÿòî îãðàíè÷åíèå íà ñêîðîñòü ñêà÷èâàíèÿ òîððåíòà, à â 21:50 ïîñûïàëèñü îøèáêè â syslog'å.

Åñëè îöåíèòü ñêîðîñòü ââîäà-âûâîäà íà äèñê:

$ dd if=/dev/discs/disca/part1  of=/dev/null count=512K
524288+0 records in
524288+0 records out
268435456 bytes (268 MB) copied, 34,892 s, 7,7 MB/s

$ dd if=/dev/zero  of=/tmp/harddisk/backup/nonsense.swp  count=512K
524288+0 records in
524288+0 records out
268435456 bytes (268 MB) copied, 116,384 s, 2,3 MB/s

òî îíà îêàæåòñÿ ìåíüøå, ÷åì ïèêîâàÿ ñêîðîñòü ñêà÷èâàíèÿ òîððåíòà ïî ãðàôèêàì rrd.

Ïðîáîâàë âîñïðîèçâåñòè ñèòóàöèþ ñ ïîìîùüþ wget, ñêà÷èâàÿ ñ ÏÊ áîëüøèå ôàéëû, íî íå ñìîã ïîëó÷èòü wget'îì ñêîðîñòü ñêà÷èâàíèÿ áîëüøóþ, ÷åì ìàêñèìàëüíàÿ ñêîðîñòü çàïèñè íà äèñê. wget îêàçàëñÿ óìíåå ìåíÿ?

Ëèíóêñîèäû, åñòü ìíåíèå, ÷òî wget íå ñêà÷èâàåò áûñòðåå, ÷åì ìîæåò çàïèñàòü íà äèñê, èñïîëüçóÿ àëãîðèòì "ñêà÷àë áëîê - çàïèñàë áëîê". À âîò òîððåíò-êëèåíòû òàêîé ëîãèêè ëèøåíû. Êàê ìíå ïîäòâåðäèòü ñâîè äîãàäêè?

[sglyk]

Òàê æå âñïëûâàþò è îòâåòû

Íàäî óâåëè÷èòü /proc/sys/vm/min_free_kbytes...
Ó ìåíÿ 16384, è îøèáîê íåò.

[ryzhov_al]

Ñïàñèáî çà ñîâåò. Íàñòðîéêè ïî óìîë÷àíèþ:

$ cat /proc/sys/vm/swappiness
60
$ cat /proc/sys/vm/min_free_kbytes
1442

Óâåëè÷èë min_free_kbytes äî 16384, ïðîâåðþ â ðàáîòå.

Îäíàêî äâà âîïðîñà âñ¸ ðàâíî îñòàëèñü.

Íàñêîëüêî ÿ äëÿ ñåáÿ âûÿñíèë, rtorrent èñïîëüçóåò memory mapping. Ïîëó÷àåòñÿ, ôàéëû íàïðÿìóþ îòîáðàæàþòñÿ â ïàìÿòü, à çà ââîä-âûâîä íà äèñê îòâå÷àþò èñêëþ÷èòåëüíî ôóíêöèè\àëãîðèòìû ÿäðà. Êàê áû íå ñòàðàëñÿ Linux VM, åñëè ñêîðîñòü ñêà÷èâàíèÿ çàìåòíî ïðåâûøàåò ñêîðîñòü çàïèñè íà äèñê, òî kswapd\pdflush ïðîñòî íå óñïåþò íè÷åãî âûãðóçèòü íà äèñê. Linux VM ìû ïàðàìåòðîì min_free_kbytes ïîäêðóòèëè, íî â ñëó÷àå, îïèñàííîì ïîñòîì âûøå çàñàäà âñ¸ ðàâíî âîçíèêíåò?

Ïî÷åìó îøèáêà page allocation failure ó ìåíÿ íèêîãäà íå âîçíèêàëà íà ÿäðàõ 2.4?

[Pablo Escobar]

Ïî÷åìó îøèáêà page allocation failure ó ìåíÿ íèêîãäà íå âîçíèêàëà íà ÿäðàõ 2.4?

ðåãóëÿðíî íà 2,4. Ïðàâäà, íà wl500gp1

[ryzhov_al]

ðåãóëÿðíî íà 2,4. Ïðàâäà, íà wl500gp1

Óãó. Ïðè óïîìèíàíèè ÿäðà 2.4 èìåëîñü â âèäó óñòðîéñòâî wl-500g premium v1 128mb.

[Pablo Escobar]

Óãó. Ïðè óïîìèíàíèè ÿäðà 2.4 èìåëîñü â âèäó óñòðîéñòâî wl-500g premium v1 128mb.

ó ìåíÿ ïî ñèìïòîìàòèêå áûâàåò äâà âàðèàíòà:
- îòâàëèâàåòñÿ dropbear, web-ìîðäà è èíòåðíåò;
- îòâàëèâàåòñÿ òîëüêî dropbear è web-ìîðäà.

Ñóäÿ ïî òîìó, ÷òî â òðè ÷àñà íî÷è òóäà ëüåòñÿ áýêàï (NFS), è ëîãè òàêèå æå, à ñ óòðà â ïîäàâëÿþùåì ïîëøèíñòâå ñëó÷àåâ âñå ðàáîòàåò (áèòûå òîððåíòû èíîãäà ïðèñóòñòâóþò), âàøà èäåÿ ram-swapper ñèëüíî ïðàâäîïîäîáíà.

[Vitaly_k]

Òîæå ðåãóëÿðíî âûâàëèâàåòñÿ îøèáêà ÿäðà.
Ðîóåòåð RT-N16, transmission.
Ñâîï íå âêëþ÷àë, ðàçâå 128 ìá ïàìÿòè ìàëî, ÷òîáû îáîéòèñü áåç ñâîïà?

[nexby]

ïîõîæàÿ îøèáêà âîçíèêàåò ïðè äîáàâëåíèè òîððåíòîâ íà çàêà÷êó.
òîððåíò-êëèåíò - rTorrent
íó è ïðîáëåì è òîðìîçîâ ïðè ýòîì, êàê áû, íèêàêèõ íå çàìå÷åíî.
ìîæ òàê çàäóìàíî?

[nexby]

ïîïðàâî÷êà:
îøèáêà âîçíèêàåò íå ïðè äîáàâëåíèè òîððåíòîâ, à ïðè îòêðûòèè âåá-èíòåðôåéñà. è äàëåå, ïðè îòêðûòîì èíòåðôåéñå âîçíèêíîâåíèå ýòîé îøèáêè íîñèò ïîñòîÿííûé õàðàêòåð âïëîòü äî çàêðûòèÿ âåá-èíòåðôåéñà.

[Pablo Escobar]

ïîïðàâî÷êà:
îøèáêà âîçíèêàåò íå ïðè äîáàâëåíèè òîððåíòîâ, à ïðè îòêðûòèè âåá-èíòåðôåéñà. è äàëåå, ïðè îòêðûòîì èíòåðôåéñå âîçíèêíîâåíèå ýòîé îøèáêè íîñèò ïîñòîÿííûé õàðàêòåð âïëîòü äî çàêðûòèÿ âåá-èíòåðôåéñà.

ïîääåðæèâàþ. ïðè÷åì çàãðóçêà 30-70%

[igor77777]

Óâåëè÷èë min_free_kbytes äî 16384, ïðîâåðþ â ðàáîòå.

À êàê ýòî äåëàåòñÿ?
( ñìûñëå óâåëè÷åíèå min_free_kbytes)

[ryzhov_al]

Ëèáî ãäå-íèáóäü â post-boot

echo 16384 > /proc/sys/vm/min_free_kbytes

ëèáî ÷åðåç sysctl â ñòàðòîâûõ ñêðèïòàõ.

[igor77777]

Ëèáî ãäå-íèáóäü â post-boot

cat 16384 > /proc/sys/vm/min_free_kbytes

ïîñëå ïåðåãðóçêè ïàðàìåòð íå èçìåíèëñÿ.
À åñëè âûïîëíèòü ýòî â putty òî ïîëó÷àþ:

cat 16384 > /proc/sys/vm/min_free_kbytes
16384: No such file or directory

Ãäå ñîáàêà çàðûòà?

[igor77777]

ïîñëå ïåðåãðóçêè ïàðàìåòð íå èçìåíèëñÿ.
À åñëè âûïîëíèòü ýòî â putty òî ïîëó÷àþ:

cat 16384 > /proc/sys/vm/min_free_kbytes
16384: No such file or directory

Ãäå ñîáàêà çàðûòà?

àãà, ñàì æå îòâå÷àþ.
Íóæíî âñòàâëÿòü â post-boot:

echo 16384 > /proc/sys/vm/min_free_kbytes

ïàðàìåòð âðîäå ïîìåíÿëñÿ.
ïîñìîòðþ, ÷òî áóäåò ñ îøèáêàìè.

[ryzhov_al]

Ïîìîãëî, óêàçàííîé îøèáêè áîëüøå íåò.
Ïðè ñêà÷èâàíèè íà ñêîðîñòè 2,5ÌÁ/c íàáëþäàåì êàðòèíó:

top - 00:00:14 up 1 day,  1:32,  0 users,  load average: 3.33, 3.45, 3.53
Tasks:  52 total,   1 running,  51 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  0.7%sy,  0.0%ni,  0.0%id, 97.7%wa,  0.0%hi,  1.0%si,  0.0%st
Mem:    126544k total,   101812k used,    24732k free,      868k buffers
Swap:        0k total,        0k used,        0k free,    69620k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
21805 god       16   0  1648  648  500 R  1.3  0.5   0:02.30 top
  337 god       10  -5     0    0    0 S  0.3  0.0   5:26.75 kl2tpd/0
 5008 god       18   0  9024 6536 1756 D  0.3  5.2  17:13.04 transmission-da
    2 god       10  -5     0    0    0 S  0.0  0.0   0:00.00 kthreadd
    3 god       34  19     0    0    0 S  0.0  0.0   0:12.55 ksoftirqd/0
   24 god       10  -5     0    0    0 S  0.0  0.0   0:00.42 kblockd/0
   51 god       15   0     0    0    0 S  0.0  0.0   0:01.41 pdflush
    4 god       10  -5     0    0    0 S  0.0  0.0   0:00.07 events/0
   53 god       20  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  105 god       10  -5     0    0    0 S  0.0  0.0   0:02.80 mtdblockd
  196 god       15   0  1544  284  232 S  0.0  0.2   0:00.25 syslogd
   52 god       10  -5     0    0    0 S  0.0  0.0   0:02.90 kswapd0
  202 god       15   0  1128  188   96 S  0.0  0.1   0:00.74 nas
    1 god       23   0  1568  616  516 S  0.0  0.5   0:00.58 init
  210 god       15   0  1172  348  276 S  0.0  0.3   0:00.00 dropbear
  213 god       15   0  1196  428  324 S  0.0  0.3   0:01.34 httpd
  218 nobody    15   0   932  420  312 S  0.0  0.3   0:02.91 dnsmasq

Âèäíî, ÷òî 97% âðåìåíè ïðîöåññîð îæèäàåò çàâåðøåíèÿ I/O-îïåðàöèé. Â äàëüíåéøåì ìîæíî áóäåò ïîèãðàòüñÿ ïàðàìåòðàìè /proc/sys/vm/* äëÿ ìèíèìèçàöèè wait-âðåìåíè (ïðèìåð).
RAM áîëüøå íå ïåðåïîëíÿåòñÿ: