I am working on extending the functionality of PostgreSQL on Linux.
I need to know what's the largest filesize for a single file that I can have
under Linux?

Regards,
Partho



Tue, 11 Mar 2003 17:31:54 GMT
"Partho Bhowmick" <pbhowmick@sbcglobal.net>
Largest filesize under Linux
In article <e4pba.794$yy2.89177201@newssvr13.news.prodigy.com>,
Partho Bhowmick <pbhowmick@sbcglobal.net> wrote:
-I am working on extending the functionality of PostgreSQL on Linux.
-I need to know what's the largest filesize for a single file that I can have
-under Linux?

The answer is definitive: it depends...

There are multiple layers that have file size restrictions (from bottom to top):

* The filesystem itself on the disk.
* The Virtual File System implementation in the kernel.
* The filesystem library interface.
* The application.

The max file size is the minimum of the allowable file sizes for each 
component. Generally there are two available sizes:

* 2.1GB imposed by a +/- 31 bit file offset. Until recently the VFS, library,
  and applications imposed this limit.
* 9 million TB imposed by a +/- 63 bit file offset. Most recent distributions
  have all 4 elements implementing this filesize.

Here's a simple test. On a filesystem with more than 5GB available try the
following:

# dd if=/dev/zero of=testfilesize bs=1024K count=3000

This will attempt to create a 3GB file which exceeds the 2.1GB max imposed by
31 bit offsets. If the file creation is successful, you can pretty much
create a file of any size.

BAJ

11 Mar 2003 13:41:31 -0500
byron@cc.gatech.edu (Byron A Jeff)
Re: Largest filesize under Linux
Quoth "Partho Bhowmick" <pbhowmick@sbcglobal.net>:
> I am working on extending the functionality of PostgreSQL on Linux.
> I need to know what's the largest filesize for a single file that I can have
> under Linux?

That depends.

It depends on the size supported by the filesystem.  Usually that's at
least 2TB, these days.  (For ext2)  Any of the interesting alternative
FSes (xfs, jfs, reiserfs) offer Way Big file sizes, >2TB.

But you also need for the file size to be supported by:
 a) GLIBC, and
 b) The kernel VFS.

I believe that recent kernels and GLIBCs allow arbitrary large files,
but you will have to check on your favorite kernel and libraries, as
milage may vary.  And it's disputable whether or not that is
necessarily going to be a "Linux" question...
-- 
output = reverse("moc.enworbbc@" "enworbbc")
http://www3.sympatico.ca/cbbrowne/fs.html
"One often contradicts  an opinion when what is  uncongenial is really
the tone in which it was conveyed." -- Nietzsche

Tue, 11 Mar 2003 15:01:44 -0500
Christopher Browne <cbbrowne@acm.org>
Re: Largest filesize under Linux
Well, I am running kernel-2.4.18-26. What's the limit on this baby?

"Christopher Browne" <cbbrowne@acm.org> wrote in message
news:m3bs0hvfiv.fsf@chvatal.cbbrowne.com...
> Quoth "Partho Bhowmick" <pbhowmick@sbcglobal.net>:
> > I am working on extending the functionality of PostgreSQL on Linux.
> > I need to know what's the largest filesize for a single file that I can
have
> > under Linux?
>
> That depends.
>
> It depends on the size supported by the filesystem.  Usually that's at
> least 2TB, these days.  (For ext2)  Any of the interesting alternative
> FSes (xfs, jfs, reiserfs) offer Way Big file sizes, >2TB.
>
> But you also need for the file size to be supported by:
>  a) GLIBC, and
>  b) The kernel VFS.
>
> I believe that recent kernels and GLIBCs allow arbitrary large files,
> but you will have to check on your favorite kernel and libraries, as
> milage may vary.  And it's disputable whether or not that is
> necessarily going to be a "Linux" question...
> --
> output = reverse("moc.enworbbc@" "enworbbc")
> http://www3.sympatico.ca/cbbrowne/fs.html
> "One often contradicts  an opinion when what is  uncongenial is really
> the tone in which it was conveyed." -- Nietzsche



Tue, 11 Mar 2003 21:05:11 GMT
"Partho Bhowmick" <pbhowmick@sbcglobal.net>
Re: Largest filesize under Linux
In article <bcsba.1331$%s6.89002224@newssvr21.news.prodigy.com>,
Partho Bhowmick <pbhowmick@sbcglobal.net> wrote:
-Well, I am running kernel-2.4.18-26. What's the limit on this baby?

I strongly suggest that you go read my response to you on this subject. As 
Christopher pointed out the kernel is only on aspect of determination.

I also gave you a quick test.

Go take a read. It will really help you.

BAJ
-
-"Christopher Browne" <cbbrowne@acm.org> wrote in message
-news:m3bs0hvfiv.fsf@chvatal.cbbrowne.com...
-> Quoth "Partho Bhowmick" <pbhowmick@sbcglobal.net>:
-> > I am working on extending the functionality of PostgreSQL on Linux.
-> > I need to know what's the largest filesize for a single file that I can
-have
-> > under Linux?
->
-> That depends.
->
-> It depends on the size supported by the filesystem.  Usually that's at
-> least 2TB, these days.  (For ext2)  Any of the interesting alternative
-> FSes (xfs, jfs, reiserfs) offer Way Big file sizes, >2TB.
->
-> But you also need for the file size to be supported by:
->  a) GLIBC, and
->  b) The kernel VFS.
->
-> I believe that recent kernels and GLIBCs allow arbitrary large files,
-> but you will have to check on your favorite kernel and libraries, as
-> milage may vary.  And it's disputable whether or not that is
-> necessarily going to be a "Linux" question...


11 Mar 2003 16:32:38 -0500
byron@cc.gatech.edu (Byron A Jeff)
Re: Largest filesize under Linux
After a long battle with technology,"Partho Bhowmick" <pbhowmick@sbcglobal.net>, an earthling, wrote:
> Well, I am running kernel-2.4.18-26. What's the limit on this baby?

Sounds like you didn't read what I wrote.  Which filesystem?  Which
version of GLIBC?  By the way, have your applications been compiled
with LFS support versus without?

/Those/ are the questions.  Read the URL below; it's got the standard
response from Alex Viro, who is one of the authors of VFS...
-- 
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://www.ntlug.org/~cbbrowne/fs.html
Pound for pound, the amoeba is the most vicious animal on earth.

11 Mar 2003 22:54:52 GMT
Christopher Browne <cbbrowne@acm.org>
Re: Largest filesize under Linux
The following info is from the SuSE 8.1 Administration Guide, page 467
The table lists both maximum file size and maximum file system size:

	file system		file size		file system
	ext2 & 3 (1kB blocks)	~16 GB		2 TB
	ext2/3 (2kB blocks)	256 GB		8 TB
	ext2/3 (4kB blocks)	2 TB		16 TB

	resiser 3.5		4 GB		16 TB
	resiser 3.6		2^60 bytes	16 TB
	  (used in 2.4 kernel)	(1 EB)

	XFS			2^63 bytes	2 TB (kernel limit)
				(8 EB)

	JFS (512B blocks)	41943904 GB	512 TB
				(4 PB)
	JFS (4kB blocks)	32 PB

Hope this helps.
73 de Leigh W3NLB

On Tue, 11 Mar 2003 17:31:54 GMT, "Partho Bhowmick"
<pbhowmick@sbcglobal.net> wrote:

>I am working on extending the functionality of PostgreSQL on Linux.
>I need to know what's the largest filesize for a single file that I can have
>under Linux?
>
>Regards,
>Partho
>


Wed, 12 Mar 2003 12:03:40 -0500
Leigh W3NLB <w3nlbREMOVETHIS@amsat.org>
Re: Largest filesize under Linux