I am working with several groups getting the Win32 port ready for 7.4
and I have a few questions:

What is the standard workaround for the fact that rename() isn't atomic
on Win32?  Do we need to create our own locking around the
reading/writing of files that are normally updated in place using
rename()?

Second, when you unlink() a file on Win32, do applications continue
accessing the old file contents if they had the file open before the
unlink?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Wed, 18 Sep 2002 20:01:42 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Win32 rename()/unlink() questions
On Wed, Sep 18, 2002 at 08:01:42PM -0400, Bruce Momjian wrote:
 
> Second, when you unlink() a file on Win32, do applications continue
> accessing the old file contents if they had the file open before the
> unlink?

I'm pretty sure it errors with 'file in use'. Pretty ugly, huh?

Ross


Thu, 19 Sep 2002 00:07:22 -0500
"Ross J. Reedstrom" <reedstrm@rice.edu>
Re: Win32 rename()/unlink() questions
Bruce Momjian wrote:
> I am working with several groups getting the Win32 port ready for 7.4
> and I have a few questions:
> 
> What is the standard workaround for the fact that rename() isn't atomic
> on Win32?  Do we need to create our own locking around the
> reading/writing of files that are normally updated in place using
> rename()?

Visual C++ comes with the source to Microsoft's C library:

rename() calls MoveFile() which will error if:

1. The target file exists
2. The source file is in use

MoveFileEx() (not available on 95/98) can overwrite the target 
file if it exists. The Apache APR portability library uses 
MoveFileEx() to rename files if under NT/XP/2K vs. a sequence of :

1. CreateFile() to test for target file existence
2. DeleteFile() to remove the target file
3. MoveFile() to rename the old file to new

under Windows 95/98. Of course, some other process could create 
the target file between 2 and 3, so their rename() would just 
error out in that situation. I haven't tested it, but I recall 
reading somewhere that MoveFileEx() has the ability to rename an 
opened file. I'm 99% sure MoveFile() will fail if the source 
file is open.

> 
> Second, when you unlink() a file on Win32, do applications continue
> accessing the old file contents if they had the file open before the
> unlink?
> 

unlink() just calls DeleteFile() which will error if:

1. The target file is in use

CreateFile() has the option:

FILE_FLAG_DELETE_ON_CLOSE

which might be able to be used to simulate traditional unlink() 
behavior.

Hope that helps,

Mike Mascari
mascarm@mascari.com










Thu, 19 Sep 2002 01:23:45 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
> On Wed, Sep 18, 2002 at 08:01:42PM -0400, Bruce Momjian wrote:
>
> > Second, when you unlink() a file on Win32, do applications continue
> > accessing the old file contents if they had the file open before the
> > unlink?
>
> I'm pretty sure it errors with 'file in use'. Pretty ugly, huh?

Yeah - the windows filesystem is pretty poor when it comes to multiuser
access.  That's why even as administrator I cannot delete borked files and
people's profiles and stuff off our NT server - the files are always 'in
use'.  Even if you kick all users off, reboot the machine, do whatever.
It's terrible.

Chris


Thu, 19 Sep 2002 13:24:01 +0800
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au>
Re: Win32 rename()/unlink() questions
Christopher Kings-Lynne wrote:
>>On Wed, Sep 18, 2002 at 08:01:42PM -0400, Bruce Momjian wrote:
>>
>>
>>>Second, when you unlink() a file on Win32, do applications continue
>>>accessing the old file contents if they had the file open before the
>>>unlink?
>>
>>I'm pretty sure it errors with 'file in use'. Pretty ugly, huh?
> 
> 
> Yeah - the windows filesystem is pretty poor when it comes to multiuser
> access.  That's why even as administrator I cannot delete borked files and
> people's profiles and stuff off our NT server - the files are always 'in
> use'.  Even if you kick all users off, reboot the machine, do whatever.
> It's terrible.
 >
 > Chris
 >

Yep. That's why often it requires rebooting to uninstall 
software. How can the installer remove itself? Under Windows 
95/98/ME, you have to manually add entries to WININIT.INI. With 
Windows NT/XP/2K, MoveFileEx() with a NULL target and the 
MOVEFILE_DELAY_UNTIL_REBOOT flag will add the appropriate 
entries into the system registry so that the next time the 
machine reboots it will remove the files specified. Its a real 
pain and a real hack of an OS.

Mike Mascari
mascarm@mascari.com



Thu, 19 Sep 2002 01:31:19 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
Mike Mascari wrote:
> Bruce Momjian wrote:
> > I am working with several groups getting the Win32 port ready for 7.4
> > and I have a few questions:
> > 
> > What is the standard workaround for the fact that rename() isn't atomic
> > on Win32?  Do we need to create our own locking around the
> > reading/writing of files that are normally updated in place using
> > rename()?
> 
> Visual C++ comes with the source to Microsoft's C library:
> 
> rename() calls MoveFile() which will error if:
> 
> 1. The target file exists
> 2. The source file is in use
> 
> MoveFileEx() (not available on 95/98) can overwrite the target 
> file if it exists. The Apache APR portability library uses 
> MoveFileEx() to rename files if under NT/XP/2K vs. a sequence of :
> 
> 1. CreateFile() to test for target file existence
> 2. DeleteFile() to remove the target file
> 3. MoveFile() to rename the old file to new
> 
> under Windows 95/98. Of course, some other process could create 
> the target file between 2 and 3, so their rename() would just 
> error out in that situation. I haven't tested it, but I recall 
> reading somewhere that MoveFileEx() has the ability to rename an 
> opened file. I'm 99% sure MoveFile() will fail if the source 
> file is open.

OK, I downloaded APR and see in apr_file_rename():

        if (MoveFileEx(frompath, topath, MOVEFILE_REPLACE_EXISTING |
                                         MOVEFILE_COPY_ALLOWED))


Looking at the entire APR function, they have lots of tests so it works
on Win9X and wide characters.  I think we will just use the APR as a
guide in implementing the things we need.  I think MoveFileEx() is the
proper way to go;  any other solution requires loop tests for rename.

I see the MoveFileEx manual page at:

	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/movefile.asp

> > Second, when you unlink() a file on Win32, do applications continue
> > accessing the old file contents if they had the file open before the
> > unlink?
> > 
> 
> unlink() just calls DeleteFile() which will error if:
> 
> 1. The target file is in use
> 
> CreateFile() has the option:
> 
> FILE_FLAG_DELETE_ON_CLOSE
> 
> which might be able to be used to simulate traditional unlink() 
> behavior.

No, that flag isn't going to help us.  I wonder what MoveFileEx does if
the target file exists _and_ is open by another user?  I don't see any
loop in that Win32 rename() routine, and I looked at the Unix version of
apr_file_rename and its just a straight rename() call.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 19 Sep 2002 16:24:01 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questions
Bruce Momjian wrote:
> > > Second, when you unlink() a file on Win32, do applications continue
> > > accessing the old file contents if they had the file open before the
> > > unlink?
> > > 
> > 
> > unlink() just calls DeleteFile() which will error if:
> > 
> > 1. The target file is in use
> > 
> > CreateFile() has the option:
> > 
> > FILE_FLAG_DELETE_ON_CLOSE
> > 
> > which might be able to be used to simulate traditional unlink() 
> > behavior.
> 
> No, that flag isn't going to help us.  I wonder what MoveFileEx does if
> the target file exists _and_ is open by another user?  I don't see any
> loop in that Win32 rename() routine, and I looked at the Unix version of
> apr_file_rename and its just a straight rename() call.

This says that if the target is in use, it is overwritten:

	http://support.microsoft.com/default.aspx?scid=KB;EN-US;q140570&

While I think that is good news, does it open the problem of other
readers reading partial updates to the file and therefore seeing
garbage.  Not sure how to handle that, nor am I even sure how I would
test it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Thu, 19 Sep 2002 22:50:41 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questions
Bruce Momjian wrote:
> Bruce Momjian wrote:
>>>
>>>unlink() just calls DeleteFile() which will error if:
>>>
>>>1. The target file is in use
>>>
>>>CreateFile() has the option:
>>>
>>>FILE_FLAG_DELETE_ON_CLOSE
>>>
>>>which might be able to be used to simulate traditional unlink() 
>>>behavior.
>>
>>No, that flag isn't going to help us.  I wonder what MoveFileEx does if
>>the target file exists _and_ is open by another user?  I don't see any
>>loop in that Win32 rename() routine, and I looked at the Unix version of
>>apr_file_rename and its just a straight rename() call.
> 
> 
> This says that if the target is in use, it is overwritten:
> 
> 	http://support.microsoft.com/default.aspx?scid=KB;EN-US;q140570&

I read the article and did not come away with that conclusion. 
The article describes using the MOVEFILE_DELAY_UNTIL_REBOOT 
flag, which was created for the express purpose of allowing a 
SETUP.EXE to remove itself, or rather tell Windows to remove it 
on the next reboot. Also, if you want the Win32 port to run in 
95/98/ME, you can't rely on MoveFileEx(), you have to use 
MoveFile().

I will do some testing with concurrency and let you know. But 
don't get your hopes up. This is one of the many advantages that 
TABLESPACEs have when more than one relation is stored in a 
single DATAFILE. There was Oracle for MS-DOS, after all..

Mike Mascari
mascarm@mascari.com





Fri, 20 Sep 2002 00:01:49 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
Mike Mascari wrote:
> I read the article and did not come away with that conclusion. 
> The article describes using the MOVEFILE_DELAY_UNTIL_REBOOT 
> flag, which was created for the express purpose of allowing a 
> SETUP.EXE to remove itself, or rather tell Windows to remove it 
> on the next reboot. Also, if you want the Win32 port to run in 
> 95/98/ME, you can't rely on MoveFileEx(), you have to use 
> MoveFile().
> 
> I will do some testing with concurrency and let you know. But 
> don't get your hopes up. This is one of the many advantages that 
> TABLESPACEs have when more than one relation is stored in a 
> single DATAFILE. There was Oracle for MS-DOS, after all..

I was focusing on handling of pg_pwd and other config file that are
written by various backend while other backends are reading them.  The
actual data files should be OK because we have an exclusive lock when we
are adding/removing them.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 20 Sep 2002 00:05:27 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questions
Bruce Momjian wrote:
> Mike Mascari wrote:
> 
>>I will do some testing with concurrency and let you know. But 
>>don't get your hopes up. This is one of the many advantages that 
>>TABLESPACEs have when more than one relation is stored in a 
>>single DATAFILE. There was Oracle for MS-DOS, after all..
> 
> 
> I was focusing on handling of pg_pwd and other config file that are
> written by various backend while other backends are reading them.  The
> actual data files should be OK because we have an exclusive lock when we
> are adding/removing them.
> 

OK. So you want to test:

1. Process 1 opens "foo"
2. Process 2 opens "foo"
3. Process 1 renames "foo" to "bar"
4. Process 2 can safely read from its open file handle

Is that what you want tested? I have a small Win32 app ready to 
test. Just let me know the scenarios...

Mike Mascari
mascarm@mascari.com





Fri, 20 Sep 2002 00:31:46 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
Mike Mascari wrote:
> Bruce Momjian wrote:
> 
>> Mike Mascari wrote:
>>
>>> I will do some testing with concurrency and let you know. But don't 
>>> get your hopes up. This is one of the many advantages that 
>>> TABLESPACEs have when more than one relation is stored in a single 
>>> DATAFILE. There was Oracle for MS-DOS, after all..
>>
>>
>>
>> I was focusing on handling of pg_pwd and other config file that are
>> written by various backend while other backends are reading them.  The
>> actual data files should be OK because we have an exclusive lock when we
>> are adding/removing them.
>>
> 
> OK. So you want to test:
> 
> 1. Process 1 opens "foo"
> 2. Process 2 opens "foo"
> 3. Process 1 renames "foo" to "bar"
> 4. Process 2 can safely read from its open file handle

Actually, looking at the pg_pwd code, you want to determine a 
way for:

1. Process 1 opens "foo"
2. Process 2 opens "foo"
3. Process 1 creates "bar"
4. Process 1 renames "bar" to "foo"
5. Process 2 can continue to read data from the open file handle 
and get the original "foo" data.

Is that correct?

Mike Mascari
mascarm@mascari.com


Fri, 20 Sep 2002 01:01:03 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
Mike Mascari wrote:
> Actually, looking at the pg_pwd code, you want to determine a 
> way for:
> 
> 1. Process 1 opens "foo"
> 2. Process 2 opens "foo"
> 3. Process 1 creates "bar"
> 4. Process 1 renames "bar" to "foo"
> 5. Process 2 can continue to read data from the open file handle 
> and get the original "foo" data.

Yep, that's it.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 20 Sep 2002 01:29:33 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questions
Bruce Momjian wrote:
> Mike Mascari wrote:
> 
>>Actually, looking at the pg_pwd code, you want to determine a 
>>way for:
>>
>>1. Process 1 opens "foo"
>>2. Process 2 opens "foo"
>>3. Process 1 creates "bar"
>>4. Process 1 renames "bar" to "foo"
>>5. Process 2 can continue to read data from the open file handle 
>>and get the original "foo" data.
> 
> 
> Yep, that's it.
> 

So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING) 
returns "Access Denied" when Process 1 attempts the rename. But 
I'm continuing to investigate the possibilities...

Mike Mascari
mascarm@mascari.com





Fri, 20 Sep 2002 01:35:23 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions

On Fri, 20 Sep 2002, Mike Mascari wrote:

> Bruce Momjian wrote:
> > Mike Mascari wrote:
> >
> >>Actually, looking at the pg_pwd code, you want to determine a
> >>way for:
> >>
> >>1. Process 1 opens "foo"
> >>2. Process 2 opens "foo"
> >>3. Process 1 creates "bar"
> >>4. Process 1 renames "bar" to "foo"
> >>5. Process 2 can continue to read data from the open file handle
> >>and get the original "foo" data.
> >
> >
> > Yep, that's it.
> >
>
> So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
> returns "Access Denied" when Process 1 attempts the rename. But
> I'm continuing to investigate the possibilities...

Does a sequence like
Process 1 opens "foo"
Process 2 opens "foo"
Process 1 creates "bar"
Process 1 renames "foo" to <something>
 - where something is generated to not overlap an existing file
Process 1 renames "bar" to "foo"
Process 2 continues reading
let you do the replace and keep reading (at the penalty that
you've now got to have a way to know when to remove the
various <something>s)



Thu, 19 Sep 2002 22:50:36 -0700 (PDT)
Stephan Szabo <sszabo@megazone23.bigpanda.com>
Re: Win32 rename()/unlink() questions
Stephan Szabo wrote:
> On Fri, 20 Sep 2002, Mike Mascari wrote:
>>Bruce Momjian wrote:
>>>Mike Mascari wrote:
>>>>Actually, looking at the pg_pwd code, you want to determine a
>>>>way for:
>>>>
>>>>1. Process 1 opens "foo"
>>>>2. Process 2 opens "foo"
>>>>3. Process 1 creates "bar"
>>>>4. Process 1 renames "bar" to "foo"
>>>>5. Process 2 can continue to read data from the open file handle
>>>>and get the original "foo" data.
>>>
>>>
>>>Yep, that's it.
>>>
>>
>>So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
>>returns "Access Denied" when Process 1 attempts the rename. But
>>I'm continuing to investigate the possibilities...
> 
> 
> Does a sequence like
> Process 1 opens "foo"
> Process 2 opens "foo"
> Process 1 creates "bar"
> Process 1 renames "foo" to <something>
>  - where something is generated to not overlap an existing file
> Process 1 renames "bar" to "foo"
> Process 2 continues reading
> let you do the replace and keep reading (at the penalty that
> you've now got to have a way to know when to remove the
> various <something>s)

Yes! Indeed that does work.

Mike Mascari
mascarm@mascari.com


Fri, 20 Sep 2002 02:03:43 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
On Fri, 20 Sep 2002, Mike Mascari wrote:

> Stephan Szabo wrote:
> > On Fri, 20 Sep 2002, Mike Mascari wrote:
> >>So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
> >>returns "Access Denied" when Process 1 attempts the rename. But
> >>I'm continuing to investigate the possibilities...
> >
> >
> > Does a sequence like
> > Process 1 opens "foo"
> > Process 2 opens "foo"
> > Process 1 creates "bar"
> > Process 1 renames "foo" to <something>
> >  - where something is generated to not overlap an existing file
> > Process 1 renames "bar" to "foo"
> > Process 2 continues reading
> > let you do the replace and keep reading (at the penalty that
> > you've now got to have a way to know when to remove the
> > various <something>s)
>
> Yes! Indeed that does work.

Thinking back, I think that may still fail on Win95 (using MoveFile).
Once in the past I had to work on (un)installers for Win* and I
vaguely remember Win95 being more strict than Win98 but that may just
have been with moving the executable you're currently running.


Thu, 19 Sep 2002 23:14:14 -0700 (PDT)
Stephan Szabo <sszabo@megazone23.bigpanda.com>
Re: Win32 rename()/unlink() questions
Stephan Szabo wrote:
> On Fri, 20 Sep 2002, Mike Mascari wrote:
>>
>>Yes! Indeed that does work.
> 
> 
> Thinking back, I think that may still fail on Win95 (using MoveFile).
> Once in the past I had to work on (un)installers for Win* and I
> vaguely remember Win95 being more strict than Win98 but that may just
> have been with moving the executable you're currently running.

Well, here's the test:

foo.txt contains "This is FOO!"
bar.txt contains "This is BAR!"

Process 1 opens foo.txt
Process 2 opens foo.txt
Process 1 sleeps 7.5 seconds
Process 2 sleeps 15 seconds
Process 1 uses MoveFile() to rename "foo.txt" to "foo2.txt"
Process 1 uses MoveFile() to rename "bar.txt" to "foo.txt"
Process 1 uses DeleteFile() to remove "foo2.txt"
Process 2 awakens and displays "This is FOO!"

On the filesystem, we then have:

foo.txt containing "This is BAR!"

The good news is that this works fine under NT 4 using just 
MoveFile(). The bad news is that it requires the files be opened 
using CreateFile() with the FILE_SHARE_DELETE flag set. The C 
library which ships with Visual C++ 6 ultimately calls 
CreateFile() via fopen() but with no opportunity through the 
standard C library routines to use the FILE_SHARE_DELETE flag. 
And the FILE_SHARE_DELETE flag cannot be used under Windows 
95/98 (Bad Parameter). Which means, on those platforms, there 
still doesn't appear to be a solution. Under NT/XP/2K, 
AllocateFile() will have to modified to call CreateFile() 
instead of fopen(). I'm not sure about ME, but I suspect it 
behaves similarly to 95/98.

Mike Mascari
mascarm@mascari.com





Fri, 20 Sep 2002 03:13:26 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
Stephan Szabo <sszabo@megazone23.bigpanda.com> writes:
> ... let you do the replace and keep reading (at the penalty that
> you've now got to have a way to know when to remove the
> various <something>s)

That is the hard part.  Mike's description omitted one crucial step:

6. The old "foo" goes away when the last open file handle for it is
closed.

I doubt there is any practical way for Postgres to cause that to happen
if the OS itself does not have any support for it.

			regards, tom lane

Fri, 20 Sep 2002 10:27:52 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Win32 rename()/unlink() questions 

I don't think we are not going to be supporting Win9X so there isn't an
issue there.  We will be supporting Win2000/NT/XP.

I don't understand FILE_SHARE_DELETE.  I read the description at:

	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp

but I don't understand it:

	FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on
	the object will succeed only if delete access is requested. 

---------------------------------------------------------------------------

Mike Mascari wrote:
> Stephan Szabo wrote:
> > On Fri, 20 Sep 2002, Mike Mascari wrote:
> >>
> >>Yes! Indeed that does work.
> > 
> > 
> > Thinking back, I think that may still fail on Win95 (using MoveFile).
> > Once in the past I had to work on (un)installers for Win* and I
> > vaguely remember Win95 being more strict than Win98 but that may just
> > have been with moving the executable you're currently running.
> 
> Well, here's the test:
> 
> foo.txt contains "This is FOO!"
> bar.txt contains "This is BAR!"
> 
> Process 1 opens foo.txt
> Process 2 opens foo.txt
> Process 1 sleeps 7.5 seconds
> Process 2 sleeps 15 seconds
> Process 1 uses MoveFile() to rename "foo.txt" to "foo2.txt"
> Process 1 uses MoveFile() to rename "bar.txt" to "foo.txt"
> Process 1 uses DeleteFile() to remove "foo2.txt"
> Process 2 awakens and displays "This is FOO!"
> 
> On the filesystem, we then have:
> 
> foo.txt containing "This is BAR!"
> 
> The good news is that this works fine under NT 4 using just 
> MoveFile(). The bad news is that it requires the files be opened 
> using CreateFile() with the FILE_SHARE_DELETE flag set. The C 
> library which ships with Visual C++ 6 ultimately calls 
> CreateFile() via fopen() but with no opportunity through the 
> standard C library routines to use the FILE_SHARE_DELETE flag. 
> And the FILE_SHARE_DELETE flag cannot be used under Windows 
> 95/98 (Bad Parameter). Which means, on those platforms, there 
> still doesn't appear to be a solution. Under NT/XP/2K, 
> AllocateFile() will have to modified to call CreateFile() 
> instead of fopen(). I'm not sure about ME, but I suspect it 
> behaves similarly to 95/98.
> 
> Mike Mascari
> mascarm@mascari.com
> 
> 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 20 Sep 2002 10:31:22 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questions
Bruce Momjian wrote:
> I don't think we are not going to be supporting Win9X so there isn't an
> issue there.  We will be supporting Win2000/NT/XP.
> 
> I don't understand FILE_SHARE_DELETE.  I read the description at:
> 
> 	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp
> 
> but I don't understand it:
> 
> 	FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on
> 	the object will succeed only if delete access is requested.

I think that's a rather poor description. I think it just means 
that if the file is opened once via CreateFile() with 
FILE_SHARE_DELETE, then any subsequent CreateFile() calls will 
fail unless they too have FILE_SHARE_DELETE. In other words, if 
one of us can delete this file while its open, any of us can.

Mike Mascari
mascarm@mascari.com





Fri, 20 Sep 2002 10:57:00 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions

It is good that moving the file out of the way works, but it doesn't
completely solve the problem.

What we have now with Unix rename is ideal:

	1) old opens continue seeing the old contents
	2) new opens see the new contents
	3) the file always exists under the fixed name

We have that with MoveFileEx(), but we have to loop over the routine
until is succeeds.  If we move the old file out of the way, we loose the
ability to know the file always exists and then we have to loop over
open() until is succeeds.

I think we may be best just looping on MoveFileEx() until is succeeds. 
We do the pg_pwd writes while holding an exclusive lock on pg_shadow so
that will guarantee that no one else will slip an old version of the
file in after we have written it.  However, it also prevents pg_shadow
access while we are doing the looping.  Yuck.

---------------------------------------------------------------------------

Mike Mascari wrote:
> Stephan Szabo wrote:
> > On Fri, 20 Sep 2002, Mike Mascari wrote:
> >>Bruce Momjian wrote:
> >>>Mike Mascari wrote:
> >>>>Actually, looking at the pg_pwd code, you want to determine a
> >>>>way for:
> >>>>
> >>>>1. Process 1 opens "foo"
> >>>>2. Process 2 opens "foo"
> >>>>3. Process 1 creates "bar"
> >>>>4. Process 1 renames "bar" to "foo"
> >>>>5. Process 2 can continue to read data from the open file handle
> >>>>and get the original "foo" data.
> >>>
> >>>
> >>>Yep, that's it.
> >>>
> >>
> >>So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
> >>returns "Access Denied" when Process 1 attempts the rename. But
> >>I'm continuing to investigate the possibilities...
> > 
> > 
> > Does a sequence like
> > Process 1 opens "foo"
> > Process 2 opens "foo"
> > Process 1 creates "bar"
> > Process 1 renames "foo" to <something>
> >  - where something is generated to not overlap an existing file
> > Process 1 renames "bar" to "foo"
> > Process 2 continues reading
> > let you do the replace and keep reading (at the penalty that
> > you've now got to have a way to know when to remove the
> > various <something>s)
> 
> Yes! Indeed that does work.
> 
> Mike Mascari
> mascarm@mascari.com
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 20 Sep 2002 11:04:53 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questionst
Mike Mascari wrote:
> Bruce Momjian wrote:
> > I don't think we are not going to be supporting Win9X so there isn't an
> > issue there.  We will be supporting Win2000/NT/XP.
> > 
> > I don't understand FILE_SHARE_DELETE.  I read the description at:
> > 
> > 	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp
> > 
> > but I don't understand it:
> > 
> > 	FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on
> > 	the object will succeed only if delete access is requested.
> 
> I think that's a rather poor description. I think it just means 
> that if the file is opened once via CreateFile() with 
> FILE_SHARE_DELETE, then any subsequent CreateFile() calls will 
> fail unless they too have FILE_SHARE_DELETE. In other words, if 
> one of us can delete this file while its open, any of us can.

I don't understand what that gets us.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 20 Sep 2002 11:05:38 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questions
On Fri, 20 Sep 2002, Mike Mascari wrote:

> Bruce Momjian wrote:
> > I don't think we are not going to be supporting Win9X so there isn't an
> > issue there.  We will be supporting Win2000/NT/XP.
> >
> > I don't understand FILE_SHARE_DELETE.  I read the description at:
> >
> > 	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp
> >
> > but I don't understand it:
> >
> > 	FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on
> > 	the object will succeed only if delete access is requested.
>
> I think that's a rather poor description. I think it just means
> that if the file is opened once via CreateFile() with
> FILE_SHARE_DELETE, then any subsequent CreateFile() calls will
> fail unless they too have FILE_SHARE_DELETE. In other words, if
> one of us can delete this file while its open, any of us can.

The question is, what happens if two people have the file open
and one goes and tries to delete it?  Can the other still read
from it?


Fri, 20 Sep 2002 08:10:19 -0700 (PDT)
Stephan Szabo <sszabo@megazone23.bigpanda.com>
Re: Win32 rename()/unlink() questions
Mike Mascari wrote:

> instead of fopen(). I'm not sure about ME, but I suspect it
> behaves similarly to 95/98.

I just checked with Katie and the good news (tm) is that the Win32 port
we did here at PeerDirect doesn't support 95/98 and ME anyway. It does
support NT4, 2000 and XP. So don't bother.


Jan

-- 

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #

Fri, 20 Sep 2002 11:36:39 -0400
Jan Wieck <JanWieck@Yahoo.com>
Re: Win32 rename()/unlink() questions
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I think we may be best just looping on MoveFileEx() until is succeeds. 
> We do the pg_pwd writes while holding an exclusive lock on pg_shadow so
> that will guarantee that no one else will slip an old version of the
> file in after we have written it.  However, it also prevents pg_shadow
> access while we are doing the looping.  Yuck.

Surely you're not evaluating this on the assumption that the pg_shadow
triggers are the only places that use rename() ?

I see other places in pgstat and relcache that expect rename() to work
per Unix spec.

			regards, tom lane

Fri, 20 Sep 2002 11:50:17 -0400
Tom Lane <tgl@sss.pgh.pa.us>
Re: Win32 rename()/unlink() questionst 
Stephan Szabo wrote:
> On Fri, 20 Sep 2002, Mike Mascari wrote:
> 
> 
>>I think that's a rather poor description. I think it just means
>>that if the file is opened once via CreateFile() with
>>FILE_SHARE_DELETE, then any subsequent CreateFile() calls will
>>fail unless they too have FILE_SHARE_DELETE. In other words, if
>>one of us can delete this file while its open, any of us can.
> 
> 
> The question is, what happens if two people have the file open
> and one goes and tries to delete it?  Can the other still read
> from it?

Yes. I just tested it and it worked. I'll test Bruce's scenario 
as well:

foo contains: "FOO"
bar contains: "BAR"

1. Process 1 opens "foo"
2. Process 2 opens "foo"
3. Process 1 calls MoveFile("foo", "foo2");
4. Process 3 opens "foo" <- Successful?
5. Process 1 calls MoveFile("bar", "foo");
6. Process 4 opens "foo" <- Successful?
7. Process 1 calls DeleteFile("foo2");
8. Process 1, 2, 3, 4 all read from their respective handles.

I think the thing to worry about is a race condition between the 
two MoveFile() attempts. A very ugly hack would be to loop in a 
CreateFile() in an attempt to open "foo", giving up if the error 
is not a NOT EXISTS error code.

Mike Mascari
mascarm@mascari.com


Fri, 20 Sep 2002 11:54:52 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I think we may be best just looping on MoveFileEx() until is succeeds. 
> > We do the pg_pwd writes while holding an exclusive lock on pg_shadow so
> > that will guarantee that no one else will slip an old version of the
> > file in after we have written it.  However, it also prevents pg_shadow
> > access while we are doing the looping.  Yuck.
> 
> Surely you're not evaluating this on the assumption that the pg_shadow
> triggers are the only places that use rename() ?
> 
> I see other places in pgstat and relcache that expect rename() to work
> per Unix spec.

Yes, I know there are others but I think we will need _a_ rename that
works 100% and then replace that in all Win32 rename cases.

Given what I have seen, I think a single rename with a loop that uses
MoveFileEx() may be our best bet.  It is localized, doesn't affect the
open() code, and should work well.  The only downside is that under
heavy read activity the loop will loop around a few times but I just
don't see another solution.

I was initially concerned that the loop in rename could let old renames
update the file overwriting newer contents but I realize now that
rename() itself has the same issue (an old rename could hit in the code
after a newer rename) so in all cases we must already have the proper
locking in place.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 20 Sep 2002 11:56:02 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questionst
I wrote:
> Stephan Szabo wrote:
 >>
>> The question is, what happens if two people have the file open
>> and one goes and tries to delete it?  Can the other still read
>> from it?
> 
> Yes. I just tested it and it worked. I'll test Bruce's scenario as well:
> 
> foo contains: "FOO"
> bar contains: "BAR"
> 
> 1. Process 1 opens "foo"
> 2. Process 2 opens "foo"
> 3. Process 1 calls MoveFile("foo", "foo2");
> 4. Process 3 opens "foo" <- Successful?
> 5. Process 1 calls MoveFile("bar", "foo");
> 6. Process 4 opens "foo" <- Successful?
> 7. Process 1 calls DeleteFile("foo2");
> 8. Process 1, 2, 3, 4 all read from their respective handles.

Process 1: "FOO"
Process 2: "FOO"
Process 3: Error - File does not exist
Process 4: "BAR"

Its interesting in that it allows for Unix-style rename() and 
unlink() behavior, but with a race condition. Without Stephan's 
two MoveFile() trick and the FILE_SHARE_DELETE flag, however, 
the result would be Access Denied. Are the places in the backend 
that use rename() and unlink() renaming and unlinking files that 
are only opened for a brief moment by other backends?

Mike Mascari
mascarm@mascari.com


Fri, 20 Sep 2002 12:27:32 -0400
Mike Mascari <mascarm@mascari.com>
Re: Win32 rename()/unlink() questions
Mike Mascari wrote:
> Its interesting in that it allows for Unix-style rename() and 
> unlink() behavior, but with a race condition. Without Stephan's 
> two MoveFile() trick and the FILE_SHARE_DELETE flag, however, 
> the result would be Access Denied. Are the places in the backend 
> that use rename() and unlink() renaming and unlinking files that 
> are only opened for a brief moment by other backends?

Yes, those files are only opened for a brief moment.  They are not held
open.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 20 Sep 2002 13:31:27 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questions
Mike Mascari wrote:
> > foo contains: "FOO"
> > bar contains: "BAR"
> > 
> > 1. Process 1 opens "foo"
> > 2. Process 2 opens "foo"
> > 3. Process 1 calls MoveFile("foo", "foo2");
> > 4. Process 3 opens "foo" <- Successful?
> > 5. Process 1 calls MoveFile("bar", "foo");
> > 6. Process 4 opens "foo" <- Successful?
> > 7. Process 1 calls DeleteFile("foo2");
> > 8. Process 1, 2, 3, 4 all read from their respective handles.
> 
> Process 1: "FOO"
> Process 2: "FOO"
> Process 3: Error - File does not exist
> Process 4: "BAR"
> 
> Its interesting in that it allows for Unix-style rename() and 
> unlink() behavior, but with a race condition. Without Stephan's 
> two MoveFile() trick and the FILE_SHARE_DELETE flag, however, 
> the result would be Access Denied. Are the places in the backend 
> that use rename() and unlink() renaming and unlinking files that 
> are only opened for a brief moment by other backends?

I think we are better off looping over
MoveFileEx(MOVEFILE_REPLACE_EXISTING) until the file isn't opened by
anyone.  That localizes the changes to rename only and not out to all
the opens.

The open failure loops when the file isn't there seem much worse.

I am a little concerned about starving the rename when there is a lot of
activity but I don't see a better solution.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Fri, 20 Sep 2002 13:53:49 -0400 (EDT)
Bruce Momjian <pgman@candle.pha.pa.us>
Re: Win32 rename()/unlink() questions