From boud w astro.uni.torun.pl Sat Jun 1 20:21:58 2002 From: boud w astro.uni.torun.pl (Boud Roukema) Date: Sat, 1 Jun 2002 20:21:58 +0200 (MET DST) Subject: w_Q \approx -1.5 ??? (DE-V0.04) In-Reply-To: Message-ID: Cze�� wszystkim, On Fri, 31 May 2002, Boud Roukema wrote: ... > Now, with a more "neutral" z distribution for the "randoms", the > signal should in principle be stronger - if it is real. > > Well, the resulting signal is spectacular! :) :) :) :) > > We still have a lot of work to do (I have to get plot_cf working), but > it seems clear that (0.3,0.7,-1.0) gives a bad fit, > and that (0.3,0.7,w_Q) with > > w_Q \approx -1.5 > > is likely to give a better fit and signal, which would greatly upset > all the theorists! Well, after the initial excitement, it looks like it's going to be less trivial than I thought to get an analysis which is totally clear and as little dependent on "arbitrary" decisions as possible... Gary, as far as your talks are concerned, my feeling right at the moment (after having spent Friday night and the whole of Saturday working on this :( ) is that it would be better *not* to show the new plots to your seminar audiences, as they are "hot-off-the-press" results which have not yet been carefully enough thought through and checked, and they would distract from the careful, detailed analysis/discussion in RMB02. You can certainly say that we're continuing to work on this and that we got some exciting stuff last week. Anyone who seriously would like to contribute to the project is welcome to subscribe to the mailing list (info below) and to help develop the software package DE. People in Toru�: the people at Piwnice yesterday decided to have another meeting next Friday (14:00 as usual), even though officially the academic year is over. So it's an unofficial wyk�ad :). But I'm no longer sure that we can have a paper ready by the end of June - at least my guess right now is that it's less easy than I thought... Na ra�e Boud From gam w iap.fr Thu Jun 13 21:45:57 2002 From: gam w iap.fr (Gary Mamon) Date: Thu, 13 Jun 2002 21:45:57 +0200 (MET DST) Subject: xi(r) from P(k) Message-ID: Hi everyone, For my talk at the IAP colloquium on July 2, it would be nice if I could show the xi(r) expected from the 2dFGRS and 2QZ P(k)'s (that are quite similar). I want to see how strong and where will the wiggles be in the xi(r) derived from the P(k). Does anyone have time to do this? By the way, I don't understand the formulation given in Peebles's book (1993, PoPC), eq. 21-40 (p. 509). If P(k) = <|delta(k)|^2>, then shouldn't the right-hand-side of the 2nd line in eq. 21-40 be sin(kr) instead of sin(kr/(kr)? cheers Gary P.S. I agree with Boud that doing ML estimation or chi2 estimation on xi(r)s is dangerous because the separation bins are not truly independent. From gam w iap.fr Sun Jun 30 21:25:39 2002 From: gam w iap.fr (Gary Mamon) Date: Sun, 30 Jun 2002 21:25:39 +0200 (MET DST) Subject: xi(r) from Hoyle's P(k) Message-ID: Dear all, I finally computed the correlation function extpected from Hoyle etal.'s P(k) obtained from their analysis of the 2QZ-10k quasar sample. I used the final part of eq. (21.40) of Peebles (1993) book to convert P(k) to xi(r). I now understand that last equation as resulting from the integral over angles of the previsou integral (which is in fact triple: dk dtheta dphi). I extended the P(k) beyond the values of Hoyle et al in two ways: 1) I used the theoretical BBKS (Bardeen et al. 84, appendix) P(k) (black & white plot). Notice that Hoyle's points are about a factor of 10000 too high so the P(k) is highly discontinuous. 2) I moved up the BBKS P(k) on each side of the Hoyle range so as to make the whole P(k) continuous. 3) Same as 2) with a step in log k decreased from 0.075 (Hoyle et al.'s step) to 0.01. (see 4th attached file). The three xi(r) plots are attached. The calculation were done with Omega = 0.3, h = 0.7, Omegab = 0.04. Even though Hoyle's peak is at 2 pi h / 89 Mpc, the 1st plot shows xi(r) with peaks at 67, 127 and 255 h-1 Mpc, close to what we (Boud really) gave in RMB02. This gives us some confidence that Boud's calculations are not completely wrong :-) and that there is no simple relation such as peak in xi at 2 pi / k_peak, where k_peak is the peak in P(k). However, in the 2nd plot, the peaks are very narrow and marginally significant at 90.6, 107.6, 109.3, 127.8, 180.6 and 255.0 h-1 Mpc. Finally, in the 3rd plot, it is hard to distinguish anything! However, with some imagination, one can guess excess broad power around 140 h-1 Mpc and a finer excess around 255 h-1 Mpc... Let me know what you think. all the best Gary -------------- następna część --------- Binarny załącznik wiadomości został usunięty... Nazwa: xihoyle.ps Typ: application/postscript Rozmiar: 43421 bytes Opis: Adres: -------------- następna część --------- Binarny załącznik wiadomości został usunięty... Nazwa: xihoylenew.ps Typ: application/postscript Rozmiar: 45149 bytes Opis: Adres: -------------- następna część --------- Binarny załącznik wiadomości został usunięty... Nazwa: xihoylenext.ps Typ: application/postscript Rozmiar: 42155 bytes Opis: Adres: -------------- następna część --------- Binarny załącznik wiadomości został usunięty... Nazwa: Pofkadopt.ps Typ: application/postscript Rozmiar: 24012 bytes Opis: Adres: From gam w iap.fr Sun Jun 30 22:15:59 2002 From: gam w iap.fr (Gary Mamon) Date: Sun, 30 Jun 2002 22:15:59 +0200 (MET DST) Subject: smoothed xi(r) Message-ID: Hi again, Attached you will find the same xi(r) as case 3 of the previous e-mail with, in red, it's smoothing by a 15 h-1 Mpc gaussian. I must admit that I worked avery quickly and there may be errors... cheers Gary -------------- następna część --------- Binarny załącznik wiadomości został usunięty... Nazwa: xihoylesm.ps Typ: application/postscript Rozmiar: 41952 bytes Opis: Adres: From gam w iap.fr Mon Jun 3 01:35:30 2002 From: gam w iap.fr (Gary Mamon) Date: Mon, 3 Jun 2002 01:35:30 +0200 (MET DST) Subject: w_Q \approx -1.5 ??? (DE-V0.04) In-Reply-To: Message-ID: Hi Boud, > On Fri, 31 May 2002, Boud Roukema wrote: > ... > > Now, with a more "neutral" z distribution for the "randoms", the > > signal should in principle be stronger - if it is real. > > > > Well, the resulting signal is spectacular! :) :) :) :) > > > > We still have a lot of work to do (I have to get plot_cf working), but > > it seems clear that (0.3,0.7,-1.0) gives a bad fit, > > and that (0.3,0.7,w_Q) with > > > > w_Q \approx -1.5 > > > > is likely to give a better fit and signal, which would greatly upset > > all the theorists! Wow! > Well, after the initial excitement, it looks like it's going to > be less trivial than I thought to get an analysis which is totally > clear and as little dependent on "arbitrary" decisions as possible... > > Gary, as far as your talks are concerned, my feeling right at the > moment (after having spent Friday night and the whole of Saturday > working on this :( ) is that it would be better *not* to show the new > plots to your seminar audiences, as they are "hot-off-the-press" > results which have not yet been carefully enough thought through and > checked, and they would distract from the careful, detailed > analysis/discussion in RMB02. Thank you for working on this. I will stick to my old guns and mention that new more discriminating statistical analyses are in progress. > You can certainly say that we're continuing to work on this and that > we got some exciting stuff last week. Anyone who seriously would like > to contribute to the project is welcome to subscribe to the mailing > list (info below) and to help develop the software package DE. Will do. Do let me know how things come along. Since the signal was weak in RMB02, you could also be bold and use all 10k quasars. Will query today (I am partly at the AAO today), about the next data release from 2dF-2QZ. best regards and do ckorovo (does that work in Polish?) Gary From gam w iap.fr Mon Jun 3 08:44:46 2002 From: gam w iap.fr (Gary Mamon) Date: Mon, 3 Jun 2002 08:44:46 +0200 (MET DST) Subject: comments from Brian Boyle Message-ID: Dear all, Brian Boyle (PI of 2dF quasar survey) came to my talk at Macquarie University. He seemed to like our analysis. His main comment was: "Have you tried repeating your correlation function analysis by replacing the scambled redshifts with a smoothed redshift distribution?" This is what I gathered you (Boud) have done these last few days (extrapolating from Boud's often cryptic messages). I answered that we are in the process of doing just that, and our *** preliminary *** result is that the 125 h-1 Mpc feature comes out strongly in the 1st z-bin, but that we did not have yet any conclusion on Omega,Lambda. cheers Gary From boud w astro.uni.torun.pl Mon Jun 3 15:37:45 2002 From: boud w astro.uni.torun.pl (Boud Roukema) Date: Mon, 3 Jun 2002 15:37:45 +0200 (MET DST) Subject: comments from Brian Boyle In-Reply-To: Message-ID: On Mon, 3 Jun 2002, Gary Mamon wrote: > Brian Boyle (PI of 2dF quasar survey) came to my talk at Macquarie > University. He seemed to like our analysis. His main comment was: "Have > you tried repeating your correlation function analysis by replacing the > scambled redshifts with a smoothed redshift distribution?" This is what I > gathered you (Boud) have done these last few days (extrapolating from > Boud's often cryptic messages). I answered that we are in the process of That's more or less correct, except that we've used the full catalogue observational dN/dz rather than a smoothed distribution. Remember that we analyse individual "fields" and then average them, so this ought to be better than z-scrambling. We should also test a smoothed dN/dz, I agree, although the interpretation will be tricky because of the z selection effects... > doing just that, and our *** preliminary *** result is that the 125 h-1 > Mpc feature comes out strongly in the 1st z-bin, but that we did not have > yet any conclusion on Omega,Lambda. This is a correct statement. Hopefully by the end of the week we'll be able to say some less "cryptic" ;) statements and each of us will be able to compile DE-V0.05 (or DE-V0.06) and check the results... Czesc Boud From gam w iap.fr Tue Jun 4 10:25:29 2002 From: gam w iap.fr (Gary Mamon) Date: Tue, 4 Jun 2002 10:25:29 +0200 (MET DST) Subject: seminar at Mt Stromlo Message-ID: Dear all, My talk at Mt Stromlo went very well. Brian Schmidt of supernovae fame made a good remark. Instead of looking for peaks, which could simply require that the xi(r)'s in different z bins are similar (he thought of using a maximum likelihood technique). He was referring to Alcock & Paczynski... Matthew Colless mentioned that both 2dF galaxies (2dFGRS) and quasars (2QZ) show a feature in P(k) at 2pi/89 h Mpc-1. Matthew's postdoc, Roberto de Propris, showed me a plot of the correlation function of the superposition of many pencil beams drawn through the 2dFGRS. He has access to the full 250k sample, and avoided computing a complete xi(r) using all 30 billion separations. There is a feature around 250 Mpc, but there are also stronger features elsewhere, but not at 130 h-1 Mpc. It thus appears that Broadhurst et al. were lucky back in 1990. cheers Gary From boud w astro.uni.torun.pl Tue Jun 4 13:08:29 2002 From: boud w astro.uni.torun.pl (Boud Roukema) Date: Tue, 4 Jun 2002 13:08:29 +0200 (MET DST) Subject: seminar at Mt Stromlo In-Reply-To: Message-ID: Cze�� wszystkim, On Tue, 4 Jun 2002, Gary Mamon wrote: > Dear all, > > My talk at Mt Stromlo went very well. > > Brian Schmidt of supernovae fame made a good remark. Instead of looking I met Brian on my last visit to Stromlo - and hopefully convinced him to drop the word "global" from his description of his work on *local* cosmological parameters... > for peaks, which could simply require that the xi(r)'s in different z bins > are similar (he thought of using a maximum likelihood technique). He was Micha� Fr�ckowiak had the same idea last Friday :) - he suggested multiplying xi_1 xi_2 and dividing by the uncertainties, also abs(xi_1-xi_2). Given the different amplitudes at different redshifts, it seems that xi_1 xi_2 is the most reasonable. > referring to Alcock & Paczynski... > > Matthew Colless mentioned that both 2dF galaxies (2dFGRS) and quasars > (2QZ) show a feature in P(k) at 2pi/89 h Mpc-1. > > Matthew's postdoc, Roberto de Propris, showed me a plot of the correlation > function of the superposition of many pencil beams drawn through the > 2dFGRS. He has access to the full 250k sample, and avoided computing a > complete xi(r) using all 30 billion separations. There is a feature > around 250 Mpc, but there are also stronger features elsewhere, but not at > 130 h-1 Mpc. It thus appears that Broadhurst et al. were lucky back in > 1990. Well, I still think the 2dFGRS has much too small a volume to be able to say anything as significant as the 2QZ, which covers a much larger volume. Na ra�e Boud From michalf w ncac.torun.pl Mon Jun 3 18:16:56 2002 From: michalf w ncac.torun.pl (Michal Frackowiak) Date: 03 Jun 2002 18:16:56 +0200 Subject: Maximum likehood unleashed! In-Reply-To: References: Message-ID: <1023121017.889.24.camel@chimera> Hi! I am sending a short theory behind max. likehood method and its adaptation in comparing curves with given errors. I believe this will be helpfull - it seems to me as the best way that does the trick: - it takes particular errors into account - lets you estimate errors of the fit - as well as contours of confidence when making a grid. This is well tested in my own soft so I hope should work in this case as well. In case of questions - I would be even more than glad to help. regards Michal -------------- następna część --------- Binarny załącznik wiadomości został usunięty... Nazwa: idea.ps Typ: application/postscript Rozmiar: 45799 bytes Opis: nie znany Adres: -------------- następna część --------- Binarny załącznik wiadomości został usunięty... Nazwa: signature.asc Typ: application/pgp-signature Rozmiar: 232 bytes Opis: This is a digitally signed message part Adres: From boud w astro.uni.torun.pl Wed Jun 5 13:08:14 2002 From: boud w astro.uni.torun.pl (Boud Roukema) Date: Wed, 5 Jun 2002 13:08:14 +0200 (MET DST) Subject: Maximum likehood unleashed! In-Reply-To: <1023121017.889.24.camel@chimera> Message-ID: Thanks Micha�, On 3 Jun 2002, Michal Frackowiak wrote: > I am sending a short theory behind max. likehood method and its > adaptation in comparing curves with given errors. I believe this will be > helpfull - it seems to me as the best way that does the trick: > - it takes particular errors into account > - lets you estimate errors of the fit > - as well as contours of confidence when making a grid. It's a nice explanation. > This is well tested in my own soft so I hope should work in this case as > well. > In case of questions - I would be even more than glad to help. Well, although it's clear the method can give a result, for it to give a correct result, the different r values would need to be independent from one another. So I see three problems, 2 easily solvable, 1 more fundamental: - solvable: (1) If we combine all three: L_{12} L_{23} L_{13} then one of these three is dependent on the other two. So it seems to me that we have to (arbitrarily) remove one of the three, even though it's clear that this is an arbitrary choice. (2) There is some smoothing in the curves output by DEplotcorrnall. This can be removed (just set ismoo=0 on line 283 of DEplotcorrnall in DE-V0.04, I think this should be OK), but then my worry is that the result will be extremely noisy. An alternative solution would be to to only test one out of every (ismoo+1) values of r . - fundamental: (3) Different bins in a correlation function depend on one another. A single quasar is a member of many pairs, and different pairs fall into different bins. So the different r values in a single function zeta(r) depend on one another. So for this reason I find it hard to believe that L (rescaled) would be a true probability density function. It's certainly a good idea, so I'll put it as one of the DEplot_cf tests, but I don't think it'll give true error bars. Cze�� Boud From michalf w ncac.torun.pl Tue Jun 4 13:32:49 2002 From: michalf w ncac.torun.pl (Michal Frackowiak) Date: 04 Jun 2002 13:32:49 +0200 Subject: Maximum likehood unleashed! In-Reply-To: References: Message-ID: <1023190370.2579.18.camel@chimera> On Wed, 2002-06-05 at 13:08, Boud Roukema wrote: > Thanks Michał, > > On 3 Jun 2002, Michal Frackowiak wrote: > > > I am sending a short theory behind max. likehood method and its > > adaptation in comparing curves with given errors. I believe this will be > > helpfull - it seems to me as the best way that does the trick: > > - it takes particular errors into account > > - lets you estimate errors of the fit > > - as well as contours of confidence when making a grid. > > It's a nice explanation. > > > This is well tested in my own soft so I hope should work in this case as > > well. > > In case of questions - I would be even more than glad to help. > > Well, although it's clear the method can give a result, for it to > give a correct result, the different r values would need to be > independent from one another. > > So I see three problems, 2 easily solvable, 1 more fundamental: > > - solvable: > (1) If we combine all three: L_{12} L_{23} L_{13} then one of these > three is dependent on the other two. So it seems to me that we have to > (arbitrarily) remove one of the three, even though it's clear that > this is an arbitrary choice. Not exactly. If you remove one of them, imagine the situation: you have 2 curves (A and B) almost identical and 1 (C) very different. Now if you calculate L wita A-B and A-C you get nothing from the first pair - they are identical - but much from B-C. BUT if you chose to comare A-C (much difference) and B-C (also much difference) you get useless info. That is why you have to compare each witch each. > > (2) There is some smoothing in the curves output by DEplotcorrnall. > This can be removed (just set ismoo=0 on line 283 of DEplotcorrnall in > DE-V0.04, I think this should be OK), but then my worry is that > the result will be extremely noisy. An alternative solution would > be to to only test one out of every (ismoo+1) values of r . > In eqs. (4) and (5) we should then replace the sum with an integral - that would be of course more natural. I have ommited it. > - fundamental: > (3) Different bins in a correlation function depend on one another. > A single quasar is a member of many pairs, and different pairs fall > into different bins. So the different r values in a single function > zeta(r) depend on one another. > > So for this reason I find it hard to believe that L (rescaled) would > be a true probability density function. > I do not think it willd be a problem - since you rescale it properly. I will think about it, but the fact that the values are somehow corelated should not affect the method - you can estimate errors for any r as I understand and the method is just to compare curves! > It's certainly a good idea, so I'll put it as one of the DEplot_cf > tests, but I don't think it'll give true error bars. > I think it will. For error bars for the best fit you can calculate the second derivative matrix for l({parameters}), than inverse it - and you get covariance matrix! From that point elements \sqrt{c_{ii}} will give you errors of 1-sigma for i-th parameter. That's a method for estimating errors of minimising method. It works for me. And agrees with plotted contours. regards Michal -------------- następna część --------- Binarny załącznik wiadomości został usunięty... Nazwa: signature.asc Typ: application/pgp-signature Rozmiar: 232 bytes Opis: This is a digitally signed message part Adres: