17:01 < JaffaCake> hi folks, welcome to the #ghc meeting 17:02 < JaffaCake> one thing we could talk about is this: http://hackage.haskell.org/trac/ghc/wiki/Design/BuildSystem 17:02 < lambdabot> Title: Design/BuildSystem - GHC - Trac 17:02 < JaffaCake> our plan for making the build system a bit easier to manage 17:02 < tibbe> hi all 17:02 < tibbe> what's the topic? 17:02 < JaffaCake> hi tibbe 17:03 < JaffaCake> the build system, if anyone is interested (link above) 17:03 < JaffaCake> the other suggestion was to talk about contributing to GHC 17:04 < tibbe> ok 17:04 tibbe would like to fix GHC so it uses epoll but doesn't know where to start 17:04 < tibbe> I'll read the build doc 17:04 < malcolmw> the plan for rejigging the build system looks good to me 17:04 < ndmitchel> JaffaCake: the build system ideas all sound very good, much more solid 17:05 < ndmitchel> i'd also suggest, as far as possible, freezing at a cabal version before a release starts 17:05 < malcolmw> using the cabal file as the declarative meta-data is a good idea - as you know, it is one of the reasons nhc98 switched from pure Make to Make + cabal meta-data 17:06 < JaffaCake> ok, the reactions seem to be mostly positive so far 17:06 < malcolmw> it is easier to keep multiple build-systems in sync if there is only one copy of the essential meta-data 17:07 < JaffaCake> right, that's the problem we had before the Cabal switchover, GHC had a separate copy of the metadata 17:07 < JaffaCake> I think we'll probably do this after the 6.10 fork 17:08 < JaffaCake> to try to do it for 6.10 would probably delay 6.10 17:08 < malcolmw> seems sensible 17:09 < gbeshers> Q: is there a consistent way to add libraries? Ideally so that hugs and jhc can use the same mechanism? 17:09 < JaffaCake> tibbe: the place to start for epoll() would be in GHC.Conc 17:09 < JaffaCake> tibbe: that's where we call select() (in the -threaded RTS) 17:09 < tibbe> JaffaCake: what does GHC currently do for filedescriptors it can't select on? 17:09 < ndmitchel> gbeshers: not sure what you mean? add libraries to what? 17:09 < tibbe> JaffaCake: so select is not used in the non-threaded RTS? 17:10 < gbeshers> ndmitchell: third party libraries (hackage). 17:10 < tibbe> JaffaCake: btw, the build doc seems reasonable to me 17:10 < JaffaCake> tibbe: re your first question.. are there any? 17:10 < tibbe> JaffaCake: I thought so, let me check 17:10 < Igloo> gbeshers: Add third party libraries to what? 17:10 < JaffaCake> tibbe: for the second question, yes the non-threaded RTS uses select(), but the machinery is inside the RTS instead of in Haskell 17:11 < gbeshers> ndmitchell: and have dependencies make sense as say debian packages would... 17:11 < tibbe> JaffaCake: OK 17:11 < gbeshers> Igloo: Suppose I want to add something (e.g., space usage monitor) which depends on hopengl. 17:12 < Igloo> gbeshers: OK, so adding new core libraries? 17:12 >>> andyjgill!n=andy@dhcp207.ittc.ku.edu 17:12 < Igloo> "boot libraries", I should say 17:12 < malcolmw> gbeshers: the referenced page about the ghc build system is _only_ talking about a small number of core libs necessary for building ghc itself. other 3rd party libs remain on hackage, with cabal as a build system, just as before. (and even the core libs have copies there too) 17:13 < gbeshers> Yes: maybe this comes under "Improvements for later". 17:13 < tibbe> JaffaCake: "On Windows, the underlying select() function is provided by the WinSock library, and does not handle file descriptors that don't originate from WinSock." says the Python documentation 17:13 < ndmitchel> gbeshers: i'm not sure these things will ever be done inside GHC, they are more hackage/cabal issues 17:13 < JaffaCake> tibbe: ah, on Windows we don't use select() 17:13 < tibbe> JaffaCake: :) what do you use then? 17:13 < JaffaCake> separate OS threads 17:13 < Igloo> gbeshers: For GHC you just add them to SUBDIRS in libraries/Makefile; I don't think there's really anything to be shared there 17:13 < tibbe> JaffaCake: that's what I thought, that's how it's usually emulated 17:13 < gbeshers> malcolmw, ndmitchell, Igloo: OK 17:14 < tibbe> JaffaCake: a small thread pool? 17:14 < JaffaCake> tibbe: OS threads are created on demand, so there will be as many as are needed 17:14 >>> waern!i=53915df2@gateway/web/ajax/mibbit.com/x-d03e78c5c680e1df 17:15 < JaffaCake> we don't currently free them later, we probably should 17:15 < JaffaCake> but anyway on Windows we ought to be using completion ports or whatever they're called 17:15 < tibbe> yes 17:15 < BSP_> JaffaCake: i'm only vaguely familiar with what select() does, but does WaitForMultipleObjectsEx do what you need on windows? 17:15 < tibbe> completion ports 17:15 < tibbe> JaffaCake: do you use edge or leveled triggered notifications? 17:16 >>> tengvall!i=53915df2@gateway/web/ajax/mibbit.com/x-d1fea08b88fa675d 17:16 < JaffaCake> select() only gives you edge, doesn't it? 17:17 < tibbe> let me check 17:17 < JaffaCake> BSP_: not really, you can't use all kinds of Handles with WFMO 17:17 < BSP_> ah, ok 17:17 < JaffaCake> BSP_: it's a mess on Windows, but it's not much better on Linux 17:18 < tibbe> JaffaCake: select is level-triggered 17:19 < JaffaCake> oh, perhaps I don't understand the difference then 17:19 < JaffaCake> select tells you when there's non-zero data available to read, or non-zero buffer space to write 17:19 < tibbe> JaffaCake: if you call select on a file descriptor that has data it will return. 17:19 < tibbe> JaffaCake: edge triggered only causes epoll() to return when the data becomes available 17:19 < JaffaCake> ah, I see - so edge only tells you once, level tells you every time 17:20 < tibbe> yes 17:20 < tibbe> apparently there's an efficiency difference 17:20 < tibbe> we can use epoll as a level-triggered API with somewhat lower performance 17:20 < tibbe> but it should still be better than select and poll 17:20 < tibbe> on bsd we qould use kqueue 17:21 < JaffaCake> yep 17:21 < tibbe> JaffaCake: then there is aio which is true async I/O (i.e. it uses callbacks) 17:22 < Igloo> So if there's nothing available on fd n, then something becomes available, and then you call epoll in edge-only mode, does it return? 17:22 < quicksilv> I was wondering that. 17:22 < quicksilv> tibbe's description sounded like it had a race condition 17:22 < tibbe> Igloo: yes presumably 17:23 < JaffaCake> edge does sound harder to use 17:23 < tibbe> hmm 17:23 < tibbe> JaffaCake: yes it is 17:23 < BSP_> are you meant to time out the epoll then and recheck periodically? 17:23 < pejo> tibbe, there probably needs to be a general fallback to select() since not all systems have kqeue/epoll/etc. Also, how robust is AIO implementations currently? 17:23 < tibbe> BSP_: I guess your supposed to read all the data that become available 17:24 < tibbe> pejo: yes, a fallback is needed, I'm not sure about AIO 17:24 < tibbe> pejo: what we could do is to write a System.Event wrapper 17:24 < tibbe> I started wrapping kqueue but its interface is much more annoying than epoll 17:26 < quicksilv> ideally epoll would check for 'edges since I last called epoll' 17:26 < quicksilv> that wouldn't have a race condition. 17:26 < tibbe> then System.Poll could emulate using threads 17:26 < tibbe> quicksilver: isn't that just level triggered? 17:26 < quicksilv> I don't know. 17:27 < JaffaCake> no, because you don't get told twice about a single transition 17:27 quicksilver reads epoll manual pages. 17:27 < JaffaCake> tibbe: so if you're interested in this, do you want to get a GHC build going and start poking around? 17:27 < tibbe> quicksilver: I think so 17:28 < quicksilv> meta-question: why is this all so hard? 17:28 < quicksilv> everybody wants asynchronous IO and always has. 17:28 < JaffaCake> I wish I knew 17:28 < tibbe> JaffaCake: yes, I have a few projects on my plate at the same time but I would like to get around to it 17:28 < quicksilv> why are ther 64 different incompatible APIs, and whichever one you choose someone will tell you it is poor performing. 17:28 < JaffaCake> there ought to be a general "wait for events" API, but neither Windows nor Linux seems to have one 17:29 < tibbe> quicksilver: some of the top linux kernel guys expressed it as: you either get hard (AIO) and good performance or easy (threads) and not so good performance 17:29 < tibbe> quicksilver: dunno if I agree though 17:29 < JaffaCake> in Haskell you should get both :) 17:29 < tibbe> JaffaCake: libevent tries to unify a bunch, there are also similar things in c++ 17:29 < JaffaCake> threads using AIO under the hood 17:29 < tibbe> JaffaCake: yes that would be nice :) 17:30 < tibbe> JaffaCake: yes, that's green threads 17:30 < JaffaCake> which we have, the only problem is they use select() 17:31 < malcolmw!n=malcolm@venice.cs.york.ac.uk ["got to go..."] 17:31 < JaffaCake> Igloo: btw, I think the problem is that the RTS is throwing BlockedOnDeadMVar without the appropriate wrapper 17:31 < JaffaCake> also BlockedIndefinitely 17:31 < Igloo> Hmm, OK 17:32 < Igloo> I'll do some grepping for all the constructors and fix any problematic ones 17:33 < JaffaCake> those are the only two I see, looking in rts/Prelude.h 17:33 >>> byorgey_!n=byorgey@pool-98-114-77-36.phlapa.east.verizon.net 17:33 ~ byorgey_ is now byorgey 17:35 < tibbe> JaffaCake: there's also the problem that in the cases you want to emulate async I/O you might want to tweak the thread pool parameters, I assume you can't do that now 17:35 >>> simonpj!n=simonpj@nat/microsoft/x-49d75a13ef026dc5 17:35 < tibbe> JaffaCake: how big a problem this is in reality I don't know 17:35 < JaffaCake> why would you need to emulate async I/O? 17:38 < SamB> hmm, sync-all doesn't seem able to play nice with tools like StGIT 17:38 < JaffaCake> StGIT? 17:39 < tibbe> JaffaCake: like you do on windows for file system I/O 17:39 < SamB> it's a tool for local patch management 17:39 < LarstiQ> Stacked git, similar to quilt. 17:39 < tibbe> JaffaCake: all APIs do not support all kinds of file descriptors 17:40 < BSP_> SamB: if you know how to fix it please do. i've never heard of StGIT 17:40 < JaffaCake> tibbe: right, but we want to hide those details in the implementation of IO 17:40 < JaffaCake> SamB: isn't *git* a tool for local patch management? 17:40 < tibbe> JaffaCake: sure, what I'm saying is that applications with special performance needs might need to tweak those parameters 17:40 < dolio!n=dolio@ [Read error: 104 (Connection reset by peer)] 17:40 < SamB> JaffaCake: no, git is the stupid content tracker 17:41 < JaffaCake> oh, ok :) 17:41 < SamB> (lest anyone think I be insulting git, they should look at the top of the manpage and see if I didn't quote it!) 17:42 < tibbe> it's Git design filosophy 17:42 < tibbe> Git's* 17:42 < JaffaCake> tibbe: I'd take a wait and see approach to that - find an example that has bad performance first 17:42 < tibbe> JaffaCake: well, it might also be the case that people look at GHC and says, this doesn't allow me to control X so I won't try 17:43 < tibbe> JaffaCake: from experience with working with high performance systems I'd say that having some knobs to tweak is important 17:43 < tibbe> but maybe we could make it an RTS flag 17:43 < JaffaCake> SamB: yes, but right under that it says " Git is a fast, scalable, distributed revision control system with an unusually rich command set..." 17:44 < SamB> JaffaCake: yes, but git does NOT store patches ... 17:44 >>> dolio!n=dolio@ 17:44 < JaffaCake> tibbe: ok, but since the IO system is all in Haskell it'd be an API call 17:44 < SamB> it stores directory trees 17:44 < JaffaCake> well, it stores patches implicitly as the difference between directory trees 17:45 < tibbe> JaffaCake: sure, I haven't yet digested all the implications of having the RTS run the event system rather than the programmer 17:45 < JaffaCake> we don't *want* the RTS to run it, I'm sure 17:45 < tibbe> JaffaCake: have you measured whether having an I/O manager thread hurts performance? 17:45 < JaffaCake> it improved performance, when I tried it 17:45 < tibbe> JaffaCake: sorry, I meant built in in the threading library 17:46 >>> claus!i=d57a2b7a@gateway/web/ajax/mibbit.com/x-0f14969d195dce2f 17:47 < JaffaCake> I'm sure you could do better by hand-coding everything and tweaking for a particular application, yes 17:47 < tibbe> JaffaCake: Maybe a compomise would be to have a System.Event and have the threading library use that 17:48 < tibbe> if you then don't want the threading library to do that for you you could use System.Event directly 17:48 < JaffaCake> tibbe: I like the sound of that, especially if it could be made platform-independent 17:48 < tibbe> JaffaCake: that would be the idea, like libevent or whatever the name of the C++ equivalent 17:48 < JaffaCake> right 17:48 < tibbe> The C++ library is called Proactor I think 17:49 < tibbe> it uses callbacks 17:49 < tibbe> so it has a thread internally that calls epoll or uses aio when available 17:50 < JaffaCake> our IO manager thread also handles dispatching signal handlers, btw 17:50 < tibbe> I had an idea where you could call read with some kind of callback e.g. a left fold and have that fold be called every time new data is available thereby creating efficient enumerator style I/O 17:50 < tibbe> JaffaCake: ok, I don't know much about that 17:51 < tibbe> JaffaCake: does the I/O manager hand over execution to another OS thread in the threaded RTS e.g. if the handler performs extensive processing 17:51 < JaffaCake> each handler runs in a new thread 17:52 < tibbe> hmm 17:53 < tibbe> JaffaCake: so the IO manager requires NOINLINE global IORefs? 17:53 < JaffaCake> yep 17:53 < tibbe> hmm 17:54 < JaffaCake> there has to be a way to communicate with the IO manager thread 17:55 < JaffaCake> ok, I have to go, bye folks 17:55 < tibbe> I guess that's a pro with an explicit event service you have to register with 17:55 < tibbe> it doesn't required global IORefs 17:55 < tibbe> JaffaCake: bye 17:56 < quicksilv> I think it's an implementation detail that JaffaCake's IO manager uses global IO Refs 17:56 < quicksilv> morally, the RTS subsystem "owns" the IO monad 17:56 < tibbe> quicksilver: what else could you do? 17:56 < quicksilv> so it woudl be allowed to add state directly to the IO monad 17:57 < quicksilv> for its own 'global variables' 17:57 < JaffaCake> quicksilver: that's my thinking, yes 17:57 < tibbe> quicksilver: but it adds complexity if I want to add other functions that uses it 17:57 < ghcbot> Build x86 Windows head fast #2593 finished: Failure (failed stage2) 17:57 < ghcbot> Build details are at http://darcs.haskell.org/buildbot/all/builders/x86%20Windows%20head%20fast/builds/2593 17:57 < tibbe> so if I want FFI to add some more functions that uses async I/O then I have to integrate them with that system 17:57 < tibbe> instead of registering them with an object 17:58 < tibbe> hmm 17:58 < tibbe> I need to think more about this 18:00 < tibbe> oh well, I'm off, bye 18:00 < quicksilv> at some stage you have to admit that the RTS isn't pure haskell. 18:00 < quicksilv> for convenience, it's nice to write as much of it as you can in haskell 18:01 < jcpetruzza_!i=98510b2f@gateway/web/ajax/mibbit.com/x-a0a5f8cdf18f214e [] 18:01 < quicksilv> because haskell is the nicer language :) 18:01 < quicksilv> but there have to be primitives somewhere. 18:01 < ndmitchel> also, global IORef's aren't that bad, they are well accepted 18:01 < quicksilv> global IORef are extraordinarily bad. 18:01 < quicksilv> probably worse than the sack of Carthage. 18:01 < quicksilv> I definitely don't accept them :) 18:02 < LarstiQ> quicksilver: what makes you feel that strongly? 18:02 < tibbe> quicksilver: but it's more primitive than C on this point! C doesn 18:02 < tibbe> C, well Linux, doesn't use a global variable for its event system 18:02 < quicksilv> LarstiQ: they are badly non-compositional. 18:03 < tibbe> LarstiQ: they rely on not being inlined not to break! 18:03 < tibbe> quicksilver: and what quicksilver said 18:03 < LarstiQ> ok :) 18:03 < quicksilv> I forgive the RTS, tho', because the RTS is what defines the IO monad and its allowed to add stuff to it. 18:03 < quicksilv> however, using them in pure haskell libraries is allowing libraries to "patch things onto" the IO monad. 18:03 < tibbe> quicksilver: sure, but there are better an worse ways to do it. 18:03 < quicksilv> which is non-compositional. 18:04 < quicksilv> because the two patches might not commute! 18:04 < quicksilv> (forgive the darcs pun :P)