Important Announcement for the TrueNAS Community.

You are using an old browser. It has the ability to falsely display this website and other websites. In your case, the shortcut is to update or apply a different browser.

NFS dies under load

• Thread starter Ideanaatwi
• Start of thread on August 14, 2013

Status is not closed for further replies.

ideanatewi

Cadet

Toet member from May 13, 2013 Notices 8

I am using the latest version of Freenas 9. 1 as motivation for CrashPla n-reserv e-cop y-system. Under load, it likewise fails to meet the NFS requirement. Previously, pausing and restarting NFS solved it, but the latest version requires an absolute reload of Freenas.

Has it happened to me too?

How can I debug it?

Jason Hamilton

Contributor

Added on July 4, 2013 Notices 141

I also use CrashPlan for copying backups. But I didn't realize that it would cause NFS to die completely. When I move something big from my desktop to the NAS, the NFS is pretty slow for all users at home while the move is in progress. But as soon as the move is over, everything starts working again. I found this, among other things, on my previous NAS (which was remote) that I installed via Ubuntu. I would like to know why it would be slower during a huge crossover for example, and I cannot figure it out right now. The only thing I believe is that my switch has no chance of overcoming this. But really, I have no doubt. It's a pretty big switch and it needs to overcome this. Dell C2100 Assembly Freenas-Corral Pereon Intel(R) Xeon(R) CPU E5620 @ 2, 40 GHz X2 Memory 98264MB Sachness 12 CDS 2 TB Radz2 2 groups with 6 disks 6 disks all types all types

Added 8 posts from May 13, 2013

ideanatewi

Cadet

Toet member from May 13, 2013 Notices 8

In such a case, no NFS export is developed, except for a reload of the Freenas server, because you can restart the Freena s-server. I can rebuild the NFS disk without touching anything on the copper.

This happens regularly. Between the beta and merciless versions it just got the worst (in the beta version the NFS-service was stopped and further started, the buyer had to change the disk, now this requires an absolute reboot).

In this state Freenas became unsuitable for use.

I also use Crashplan for backup. However, NFS has never stopped. If you move something big from the desktop on the NAS, NFS will be very slow to all users in the house. But as soon as the movement is over, everything starts again. This was the same for the NAS I used before via Ubuntu (remotely). I want to know why the operation is slow during a large overflow, but I don't understand yet. The only thing you can think of is that my switch cannot handle it? I can't say that. This is a pretty big switch, so you should be able to respond.

I also use CrashPlan for copying backups. But I didn't realize that it would cause NFS to die completely. When I move something big from my desktop to the NAS, the NFS is pretty slow for all users at home while the move is in progress. But as soon as the move is over, everything starts working again. I found this, among other things, on my previous NAS (which was remote) that I installed via Ubuntu. I would like to know why it would be slower during a huge crossover for example, and I cannot figure it out right now. The only thing I believe is that my switch has no chance of overcoming this. But really, I have no doubt. It's a pretty big switch and it needs to overcome this. Dell C2100 Assembly Freenas-Corral Pereon Intel(R) Xeon(R) CPU E5620 @ 2, 40 GHz X2 Memory 98264MB Sachness 12 CDS 2 TB Radz2 2 groups with 6 disks 6 disks all types all types

dvc9

Explorer

I think this is the same problem as a network card driver.

Instead of restarting the server, try "IFCONFIG (NIC) Down" and "IFCONFIG (NIC) Up".

May 8, 2013 Postscript 8

ideanatewi

Cadet

Toet member from May 13, 2013 Notices 8

Post from May 2, 2012 72

dvc9

Explorer

I think this is the same problem as a network card driver.

If you are looking for another distribution, look for Smart OS (http: // smartOS. Org/) and Omni OS with Nap p-i t-Plugin. (http: // omnios. Omniti. com/) (http: // napp-ite. Org/).

You may miss the simplicity of FreeNAS, but you can understand it yourself;)

By the way, there is no problem with this NFS in FreeNAS 8. 3. 1. We use freeNAS as a media storage for . dpx sequence in video production. Resourc e-NFS V4. FreeNAS 8. 3. 1, many RAM,+ 10Gbe NIC.

19 posted on March 25, 2012 was added.

cyberjock

Inactive Account

Your omission. As described in the forum rules, you can get more answers if you describe your hardware and other everything.

Hardwar e-Recommende d-RAID5/RAIDZ1 DEAD READ MINE

management ! I answered frequently asked questions for beginners. You are the only one who can prevent the land mining war! Read FreeNA S-Leadership. :) My main system

FreeNAS-9. 10. 2-U1 | 2X Intel E5-2670 | SUPERMICRO X9DR4-LN4F 128GB DDR3 ECC 1600 RAM | 32GB SATA DOM | ES (RAIDZ2, 40. 9TB USABLE Space) | 10GBE (Chelsio Adapter)

Posted on May 8, 2013 added 8

ideanatewi

Cadet

Toet member from May 13, 2013 Notices 8

Instead of restarting the server, try "IFCONFIG (NIC) Down" and "IFCONFIG (NIC) Up".

May 8, 2013 Postscript 8

I just tried to run the commands "ifconfig ix1; sleep 5; ifconfig ix1 up" in one command so that I would not lose the only connection. Then suddenly the DF team quit!

Funny. What is this about?

lol) I didn't give up completely, I was just upset for a while and had to complete the client backup before trying anything else.

As for Freenas8, I panicked immediately when it failed to start. The support service said they would not fix it, we could only wait for V9.

Is there an easy way to get device details from the web interface of Freenas? I published details about my Dell (R720XD) model, the drives are higher, but I need to get all the details about which PER C-card, which exact etherne t-card, drives, etc. You're right. I have experience with Linux, but not so much with BSD. The DMIDECODE output seems too detailed for a forum, but I'll enable it as an attachment for the sake of completeness of the image.

dmidecode.

Attachments

32. 5 kb - Views: 322 May 2012 2 Days Added Posts 72

dvc9

Explorer

I think this is the same problem as a network card driver.

I made a little script to reset the network card. Are you using Intel 10GBE?

I think there is a problem with the IXGBE driver in release 9. 1. I have an X520 card and I use one port for high-end video and I have to restart this port in OPCONFIG 3-4 times a day.

The other port is "low" traffic, just editing sound from protools. )

Joined March 25, 2012 Posts 19.

cyberjock

Inactive Account

Your omission. As described in the forum rules, you can get more answers if you describe your hardware and other everything.

I made a little script to reset the network card. Are you using Intel 10GBE?

Hardwar e-Advic e-RAID5/RAIDZ1 IS DEAD TO READ MINE

management ! I answered frequently asked questions for beginners. You are the only one who can prevent the land mining war! Read FreeNA S-Leadership. :) My main system

FreeNAS-9. 10. 2-U1 | 2X Intel E5-2670 | SUPERMICRO X9DR4-LN4F 128GB DDR3 ECC 1600 RAM | 32GB SATA DOM | ES (RAIDZ2, 40. 9TB USABLE Space) | 10GBE (Chelsio Adapter)

Posted on May 8, 2013 added 8

ideanatewi

Cadet

Toet member from May 13, 2013 Notices 8

Look at Post#3.

errorPlease see.If the correctly set thing does not work as it is, I call it an error and do not call it a techno help request.

And now we have two users alone, what is the task?

errorPlease see.Yes, my Dell R720XD has Intel X520 DP 10GB DA/SFP+server adapter installed, apparently with this card.

It's really unusual for the Web GUI to keep answering what the interface, including when NFS stops working!

8 posted on May 13, 2013

ideanatewi

Cadet

Toet member from May 13, 2013 Notices 8

Do you know if there is a function to force this tap again?

May 2, 2012 Postscript Report 72

dvc9

Explorer

I think this is the same problem as a network card driver.

I would like to share it with you and everyone at the Forum FreeNAS if there is a way to make a detour, and if no one will publish the amendment. )

TOET members notification from May 2, 2012 72

dvc9

Explorer

I think this is the same problem as a network card driver.

This may be the difference between UDP and TCP because NFS V3 uses UDP. I will organize it during the day =).

The actual intelligence (https: // www. Gugl. No/search? Q = Ixgb. Thinking You can be done.

It's great if you get the instruction to exchange drivers or have the opportunity to cross over a more fresh IXGBE driver!

September 18, 2013 Participation Notification 8

freefan

Cadet

Hello, I also feel that the loading/ box does not work with a new FreeNAS (9. 1.).

FreeNAS BOX info 9. 1. 1. 1 ================================================== I'm, I , d, FreeBSD Freebsd (9. 1.) ============================================================================== doesn't ======== (BOX 1.) FreeNAS Box Info 9.

# 1-Stable FreeBSD 9. 1-Stable # 0 R+16F6355: 27 AUGUST 00:38:40 PDT 2013 ROOT@Build. Com:/Tank/Home/JKH/SRC/BASE/BASE- 64/ Tank/Home/JKH/SRC/FreeBSD/SRC/SYS/FREENAS.~# zpoolist Myvol00 5, 44T 1, 46T 3, 98T 26% 1, 00x Online /mnt~Entering a mistake: Even if this box explains the overview, there is no mistake in/var/log/messages and dmesg, but I think the load on FreeNAS is also increasing.

(BOX 2.) FreeNAS Client Informatio n-KVM environment (tested both Unbuntu and CentOS) ================================================================ and =========================================================== was =========================================================== was ===== [Root@Client] # Unam e-a Linux mybox

#Unam e-a Linux myboxa01 2. 32-358. x86_64 # 1 SMP Fri Feb 22 00:31:26 UTC 2013 X86_64 GNU/Linux ======= Memory statistics (8 GB memory) ========== TO P-16:35:58 To 15:06, 2 Users, load average: 0, 06, 04, 08, 08 Question: Total 147, 1 Working, 145 Sleeping, 1 STOPPED, 0 Zombie CPU (S): 0, 3%US, 0, 2%SY, 0, 0%NI, 98, 6%ID, 0, 9%WA ” K Fluent, 2039944K Cache ====== ============== Comment with NFS Freenas sharing withetc /fstab:~I tried various mount methods, but this problem did not solve. This is the latest mount method,

========== 10. 95. 1. 1. 40:/mnt/Myvol00/MyDir/mnt/MyDirnfs soft, timeo = 900, retrans = 3, TCP

Description of my task: (first part - rsync process crashes, but starts again when user issues command in shell) As a result, I won with this (you can relate this first part with "dies under load") Finally, it all started when I first installed Freenas 9. 1. 1 from the previous 8.* branch. Then I installed the latest version and everything was fine, but when I started rsyncing data from an external volume where I made a backup copy of an externally powered device (i. e. external hard drive or LSI card is a separate volume) from the main Freenas zpool to Freenas, Freenas "frozen" during the rsync. It wasn't completely frozen, it wasn't stuck, it just "stuck". I started the rsync, came back a few hours later and ran d f-ah in shell. I thought it was unusual. Looking inside the box, the rsync process seemed to still be alive, but its load was pretty low, almost like it was asleep. So I went back to the shell a few more times and did a d f-ah and noticed that rsync had grown to 5GB and the process load was growing again. I thought that was unusual. I'll give it some more time.

After it was all finished and my data was transferred to the new freenas 9. 1. 1 I dismissed the rsync file transfer issue as an accident or a freak oversight of rsync that had nothing to do with freenas. Either way, I finally transferred all my data back and across the space with the latest version of freenas:

Description of my task: (Second share - Buyer core toxins with correct messages about mistakes and core dumps) I finally managed to eliminate all kinds of problems with the operating system of the buyer (first is Ubuntu 12, later, as I will explain, Center OS 6) of the buyer! Almost no first time task. "Nice" I believe ... I set up a personal website, everything happens, everything is fine! All the data on the site! And then I leave, thinking "All is well and good, everything is cool." Meanwhile, I did not come back for about 2 hours. When I load the website, everything freezes and does not load. Enter the customer, lift the box - the load is pretty high and it increased. But I do not see even the slightest process that is causing this high load. During this time, I looked in the journal of the web server, nothing. Nothing. Checking the LO G-magazine of Unbuntu and the Journal of Notifications of Cent OS, I found the correct:

Client Freenas, under high and easy load, the correct

16 Sep 02:18:46 myboxA0 kernel: [ 3600. 648109] INFO: wget:2373 stat op 120 pagina's. 16 Sep 02:18:46 myboxA0 kernel: [ 3600. 648250] 「echo 0 & amp; amp; gt; /proc/sys/kernel/hung_task_timeout_secs」 is beschikbaar. 16 sep 02:18:46 myboxA0-kernel: [ 3600. 648347] wget D ffff88021fd14580 0 2373 2372 0x00000000 16 sep 02:18:46 myboxA0-kernel: [ 3600. 648353] ffff880210509b98 00000000000002 ffff880210509fd8 0000000000014580 16 sep 02:18:46 myboxA0-kernel: [ 3600. 648357] ffff880210509fd8 00000000000014580 ffff880211b85dc0 ffff88021fd14e30 16 Sep 02:18:46 myboxA0 kernel: [ 3600. 648360] ffff88021ffb72e8 0000000000002 ffffff8113f130 ffff880210509c10 16 Sep 02:18:46 myboxA0 kernel: [ 3600. 648364] Oproep tracering: 16 sep 02:18:46 myboxA0 kernel: [ 3600. 648375] [] ? wait_on_page_read+0x60/0x60 16 sep 02:18:46 myboxA0 kernel: [ 3600. 648381] [] io_schedule+0x9d/0x130 16 Sep 02:18:46 myboxA0 kernel: [ 3600. 648384] [] sleep_on_page+0xe/0x20 16 Sep 02:18:46 myboxA0 kernel: [ 3600. 648387] [] __wait_on_bit+0x60/0x90 16 Sep 02:18:46 myboxA0 kernel: [ 3600. 648390] [] wait_on_page_bit+0x7f/0x90 16 Sep 02:18:46 myboxA0 kernel: [ 3600. 648394] [] ? wake_atomic_t_function+0x40/0x40 16 Sep 02:18:46 myboxA0-kernel: [ 3600. 648399] [] ? pagevec_lookup_tag+0x21/0x30 16 Sep 02:18:46 myboxA0-kernel: [ 3600. 648402] [] filemap_fdatawait_range+0x101/0x190 16 Sep 02:18:46 myboxA0-kernel: [ 3600. 648485] [] ? do_writepages+0x1e/0x40 16 Sep 02:18:46 myboxA0-kernel: [ 3600. 648488] [] ?

Doesn't there be a French? I thought. Interestingly, when I connected to Freenas by SSH, everything came back as soon as I started SSH connection to FreeNAS! The web page is loaded immediately, and the load on the client returns to the normal level. Now, I'm starting to think about the D F-Ah shade that caused RSync to keep working. Can I repeat this? Of course, it is a continuous loop, a process that regularly accesses Freenas resources (I've seen crashing with WGET, NGINX, or other processes, and searching to confirm that it's not a specific process. (I did a lot), but since the NFS server disappears, I keep hanging. However, if you try to connect via the network using SSH, everything will come back immediately. Literally, the web browser can be loaded immediately from the standby state.

Summarized

I don't know if these two problems are related, but Freenas seems to be "sleepy" when a high load is high. Both look the same behavior that must be DF to continue the file transfer, and that the NFS server must be SSH to return from "no response". I tried Ubuntu and Cent OS to exclude the problem of the operating system, but occurred with both clients with the same settings.

Also, if you google with "Blocked for More Than 120 Seconds", the result will come out, but it was old.

Why is Freenas fall into such a strange scenario? Freenas is great, so I want to keep using it as it is, but it's hard to return to 8.*brunch.

Login date: July 25, 2013 Posted on: 374

aufalien

Patron

It's late, but other distributions are: SmartOS is a hypervisor, and Omnios supports the outdated hardware. So don't run ahead.

Nappit is incredibly ugly and written in Perl. Freenas user interface is much better.

DVC9 is really ugly.

Logon posted on May 2, 2012 72

dvc9

Explorer

I think this is the same problem as a network card driver.

Isn't that a little different?

The NFS dies when the load is still applied, and the Freenas 9. 1. 1 IXGBE driver is still strange.

But I found a solution: by setting the network cards in LAG, you can prevent them from being killed by NFS.

I don't know why. If you want to discuss other NAS operating systems, go ahead and start your own topic.

Added on 18 Sep 2013 Post 8

freefan

Cadet

Hello, I also feel that the loading/ box does not work with a new FreeNAS (9. 1.).

1. Has there already been an official bug reported about the ixgbe driver crashing in freenas9. 1. 1? If freenas is used in a production system, obviously others will encounter this problem and there will need to be an update to fix it.

2. Did previous versions of freenas have the same bug?

I wanted to use a LAG install as a quick solution, but my switch doesn't support it.

As I said, I love freenas, but something breaks every time I'm in production (for 6-12 hours now). Any tips or ideas? Thanks!

Logon May 2, 2012 Post 72

dvc9

Explorer

I think this is the same problem as a network card driver.

I wonder if it's worth installing 9. 1. 2 alpha to test it :p

I also noticed a performance drop from 8. 3 to 9. 1. 1, so there seems to be something fishy here too.

Added May 2, 2012 Post 72

dvc9

Explorer

I think this is the same problem as a network card driver.

Current ixgbe version is 2. 15, now 2. 7

[root@FRAG-SERVER]

# SYSCTL dev. ix dev. ix. 0.%Desc: Intel (R) Pro/10GBE PC I-Express ネットワークドライバ、バージョン - 2. 0.%Locatie: slot = 0 functie = 0 dev. 0.%Pnpinfo: leverancier = 0x8086 Device = 0x151c Subvendor = 0x8086 Subdevice = 0xa03c /& amp; amp; gt; dev. 0.%ouder: pci8 dev. ix. 0. fc: 3 dev. ix. 0. enable_aim: 1 dev. ix. 0. advertise_speed: 0 dev. ix. 0. Dropped: 0 dev. ix. 0. MBUF_DEFRAG_FAILED: 0 dev. ix. 0. watchdog_events: 0 dev. ix. 0. link_irq: 23 dev. ix. 0. queue0. Interrupt_Rate: 100000 dev. txd_head: 354 dev. ix. 0 queue0. txd_tail: 354 dev. ix. 0 queue0. tso_tx: 151567 dev. ix. 0 queue0. no_tx_dma_setup: 0 dev. ix. 0 queue0. rx_packets: 1360637 dev. ix. 0. queue0. rx_bytes: 1973685017 dev. ix. 0. queue0. lro_flushed: 1214009 dev. ix. 0. queue1. interrupt_rate: 125000 dev . ix. 0. queue1. irqs: 4476143 dev. ix. 0. queue1. txd_head: 1262 dev. ix. 0. queue1. txd_tail: 1262 dev. ix. 0. queue1. tso_tx: 2727 dev. ix. 0. queue1 . no_tx_dma_setup: 0 dev. ix. 0. queue1. no_desc_avail: 0 dev. ix. 0. queue1. tx_packets: 4400923 dev. ix. 0. queue1. rxd_head: 1944 dev. ix. 0. queue1. rxd_tail: 1943 dev. ix. 0. queue1. rx_packets: 1378200 dev. ix. 0. queue1.~ix. 0. queue4. txd_tail: 225 dev. ix. 0 queue4. tso_tx: 169871 dev. ix. 0 queue4. NO_TX_DMA_SETUP: 0 dev. ix. 0 queue4. no_desc_avail: 0 dev. ix. 0 . queue4. tx_packets: 2099209 dev. ix. 0 queue4. rxd_head: 683 dev. ix. 0 queue4. rxd_tail: 682 dev. rx_packets: 1475243 dev. ix. 0 queue4. rx_bytes: 2155923782 dev. ix. 0 queue4. rx_copies: 38942 dev. ix. 0 queue4. lro_queuuatie: 1443537 dev. lro_flushedule4: 1311422 dev. ix. 0. queue5. 割り込みレート: 125000 dev. txd_head: 1141 dev. ix. 0. queue5. TSO_TX: 235165 dev. ix. 0. queue5. no_tx_dma_setup: 0 dev. ix. 0. queue5. no_desc_avail: 0 dev. ix. 0. queue5. tx_packets: 32595976 dev. ix. 0 1043 dev. rx_packets: 39197715 dev. ix. 0. queue5. rx_bytes: 55548278168 dev. ix. 0. queue5. queue5. Lro_Queuded: 39056159 dev. ix. 0. queue5. lro_flushed: 4864118 dev. ix. 0. queue6. Interrupt_Rate: 100000 dev. ix. 0. queue6. txd_he_hoofd: 1913 dev. ix. 0. queue6. txd_tail: 1913 dev. ix. 0 . queue6. tso_tx: 476 dev. ix. 0. queue6. no_tx_dma_setup: 0 dev. ix. 0. queue6. tx_packets: 3462373 dev. ix. 0 queue6. rxd_head: 495 dev. ix. 0 queue6. rxd_tail: 494 dev. ix. 0 . queue6. rx_packets: 1368559 dev.