Forum Discussion
11.4.0 unable to import 3M-record external data-file
f5 BIG-IP LTM VE v11.4.0
ESXi VM allocated with 200 gb hdd , 16 gb ram
System > File Management > Data Group File List > Import > point to 3M-record data-file on disk ( 216 MB )
receive error : " File upload failed, please check log file for details "
I review all logs but don't see any entries related to data-file load.
System > File Management > iFile List > Import > point to the same 3M-record file
I receive the same error.
Examining
/var/log/ltmerr diskmonitor: 011d0004: Disk partition _root_ has only 0% freebut the timestamp of this error does not coincide with my attempted file uploads.
I assume
_root_In
/var/log- Three million records is a pretty decent number of rows. What kind of content is it? How large is the actual file? What is it you're trying to accomplish with this data once it's uploaded? With a little bit more information we may be able to more accurately assist. ;) Colin
hooleylistCirrostratus
Also, what error is listed in /var/log/ltm when the import fails?
And how much RAM does the VE have allocated?
And how many TMMs are running?
Thanks, Aaron
- Thanks for the response Colin. This is a string type file of key-value pairs of url redirects/rewrites; size is about 220 MB.
uniAltostratus
According to the TMOS concepts manuals, up to v11.3, the maximum size when importing an iFile is 4MB. Strangely, the line has been removed from the v11.4 manual, but there is no mention of a change in the release notes.
I know that doesn't help much. Perhaps you could log a support request and ask them what the maximum size is for 11.4.
- Ok but I have successfully imported Data Group External File of size 120MB. Does anyone know how an iFile differs internally from a Data Group External File ?
- uniI don't know about data group files - I have only imported very small ones. On the import screen it say "Free space on disk". What does this say for you? On one I looked at, this corresponded to the amount of space free on the root (linux) partition, even though the file is stored in the /config partition, and which had very much more space available.
- iFiles are meant for web pages and other markup/script -- they allow one to avoid the TCL encodings necessary when the code is stored directly in an iRule. For me it also lists "Free space on disk" as a very small value, such as ~170MB. But why would the Linux `_root_` partition be so small ?
uni_87886According to the TMOS concepts manuals, up to v11.3, the maximum size when importing an iFile is 4MB. Strangely, the line has been removed from the v11.4 manual, but there is no mention of a change in the release notes.
I know that doesn't help much. Perhaps you could log a support request and ask them what the maximum size is for 11.4.
- Ok but I have successfully imported Data Group External File of size 120MB. Does anyone know how an iFile differs internally from a Data Group External File ?
- uni_87886I don't know about data group files - I have only imported very small ones. On the import screen it say "Free space on disk". What does this say for you? On one I looked at, this corresponded to the amount of space free on the root (linux) partition, even though the file is stored in the /config partition, and which had very much more space available.
- iFiles are meant for web pages and other markup/script -- they allow one to avoid the TCL encodings necessary when the code is stored directly in an iRule. For me it also lists "Free space on disk" as a very small value, such as ~170MB. But why would the Linux `_root_` partition be so small ?
- uni
I think I have it. When you import a file through the Configuration Utility, the file is uploaded to /tmp (this seems to be a function of the Apache web server), which is in the "/" (root) partition. Thus the size of the upload is limited to the amount of free space in the root partition. Once the file is imported to the local file system, it is moved to the /config partition, which may have more space available.
Curiously, if you use tmsh to load a file (tmsh create sys file ifile source-path ), the temporary file goes to /var/tmp, which should have a lot of space available.
I suggest you try the tmsh method
- Very interesting stuff uni. But shouldn't it be possible to expand the size of the "/" (root) partition ? How to do this ?
- hooleylistThat looks like the best option Uni. Thanks for suggesting it. Balto, it would be great if you could raise this issue with F5 Support as I think the GUI operation should be fixed to support very large data group files. Thanks, Aaron
- uniI wouldn't pin my hopes on expanding partition sizes. The BigIP is an appliance not a server. If you start messing with the partition sizes, you will have to do it every time you update or apply a hotfix. In my opinion, F5 should link /tmp to somewhere in a larger partition, eg to /shared/tmp, or change where the httpd temporary files go
- uni_87886
I think I have it. When you import a file through the Configuration Utility, the file is uploaded to /tmp (this seems to be a function of the Apache web server), which is in the "/" (root) partition. Thus the size of the upload is limited to the amount of free space in the root partition. Once the file is imported to the local file system, it is moved to the /config partition, which may have more space available.
Curiously, if you use tmsh to load a file (tmsh create sys file ifile source-path ), the temporary file goes to /var/tmp, which should have a lot of space available.
I suggest you try the tmsh method
- Very interesting stuff uni. But shouldn't it be possible to expand the size of the "/" (root) partition ? How to do this ?
- hooleylistThat looks like the best option Uni. Thanks for suggesting it. Balto, it would be great if you could raise this issue with F5 Support as I think the GUI operation should be fixed to support very large data group files. Thanks, Aaron
- uni_87886I wouldn't pin my hopes on expanding partition sizes. The BigIP is an appliance not a server. If you start messing with the partition sizes, you will have to do it every time you update or apply a hotfix. In my opinion, F5 should link /tmp to somewhere in a larger partition, eg to /shared/tmp, or change where the httpd temporary files go
- uni_87886
Having said don't mess with the file system, I just did, and the following worked. The Configuration Utility tells me there is as much space available as in the /shared partition. I doubt F5 would approve of this though.
mkdir /shared/tmp/tmp chgrp root /shared/tmp/tmp chmod 1777 /shared/tmp/tmp mv /tmp /tmp.save ln -s /shared/tmp/tmp /tmpYou should probably do a bigstart restart (or at least bigstart restart tomcat).
- uni_87886Oops. I left out where I save the old tmp directory before the ln line: mv /tmp /tmp.save
- Now I'm running out of space in the BIG-IP Common partition ( "/config" ) as I add more data-group-files. Does f5 provide a way to expand the Common partition ?
- uni
Having said don't mess with the file system, I just did, and the following worked. The Configuration Utility tells me there is as much space available as in the /shared partition. I doubt F5 would approve of this though.
mkdir /shared/tmp/tmp chgrp root /shared/tmp/tmp chmod 1777 /shared/tmp/tmp mv /tmp /tmp.save ln -s /shared/tmp/tmp /tmpYou should probably do a bigstart restart (or at least bigstart restart tomcat).
- uniOops. I left out where I save the old tmp directory before the ln line: mv /tmp /tmp.save
- Now I'm running out of space in the BIG-IP Common partition ( "/config" ) as I add more data-group-files. Does f5 provide a way to expand the Common partition ?
