deep sky images with perl/how i built an internet telescope in an afternoon

Open Source Deep Sky Images Using PerlEver since I was a child, I've loved looking at the night sky. Rural Ireland has the benefit of being rather dark, although rural Cambridgeshire also has the benefit of being dark. I couldnt ever live in a city, being unable to see the stars at night would drive me mad.

And my formative years with filled with film, a cable release and too much hope, never yielding anything like the images I saw in my astronomy books. Ah well.

Later, In another previous life, I was somewhat of a stellar astrophysicist and looked at the sky in a different way. With Fortran and assembly, as it happened, and arrays of CCDs that covered several tables. But then something else happened, and perl took over and I left all that behind.

Sort of. While I haven't quite finished building my latest telescope, grinding mirrors is slow, and driving motors in an equatorial fashion with a Raspberry Pi is a chore (and abandoned, nice PR job, not up for what I want), there must be another way of capturing the heavens,

I found another way.

OREver since I was a child, I've loved looking at the night sky. Rural Ireland has the benefit of being rather dark, although rural Cambridgeshire also has the benefit of being dark. I couldnt ever live in a city, being unable to see the stars at night would drive me mad.





How I built an Internet Telescope in an AfternoonEver since I was a child, I've loved looking at the night sky. Rural Ireland has the benefit of being rather dark, although rural Cambridgeshire also has the benefit of being dark. I couldnt ever live in a city, being unable to see the stars at night would drive me mad.





Jupiter and the PleiadesEveryone loves pictures of the sky, no? And what better than to capture it yourself. This image was take with a ten-year old digital camera on a tripod, not even through a telescope. Use what you have, I suppose.

A decent telescope (with a decent mount) will set you back a good grand and a half, while a decent CCD another grand at least. Quite an expensive barrier to entry to amateur astrophotography, all the same. High initial setup costs, but little after that, until you get an eyepiece addiction.

You will also fight against the English weather, by my reckoning I get about 90 days worth of decent observing a year. And this time of year, it can get cold.

And that image is a bitnaked eye-esque.

Lame

Horsehead Nebula in OrionGiven the audience in this room, I would doubt there is anyone who hasnt seen at least one image from one of the famous space telescopes, like Hubble, Spitzer or even COBE. To get such images takes time, and billions of dollars. Neither of which, alas, I have.

Nor do you even have to send something into orbit, adaptive optics are wonderful things, if you are ever passing through the Institute of Astronomy in Cambridge, seek out the ALMA people and get them to show you some of their research images, the stuff that doesnt get to the general public.

Ohhhh

Rosette NebulaI know it can seem a bit of a cheat, all these false-colour images, of something that you can never see with the naked eye, even given time and space, but that doesnt stop them being awesome. I love them, and unlike some, studying this stuff in fine detail never made me lose my initial wonder at the heavens.

But realistically, we have no chance of capturing those sorts of images ourselves from our back garden observatories.

Ahhhh

The North America Nebula

So I made myself content to look at APOD every day, and wistfully ponder on giving up a lucrative career in The City and go back to working the big 'scopes. Although not sure my wife would like that, but once the children have left, Chile might be appealing again...

Crikey

IC 1848But there are other ways of creating these images.

Recently I discovered that a lot of the raw data from various space telescopes is available online. For anyone to use. Now I got to thinking, sure, I don't need to sit out and record 36 hours of RGB channel data from my back garden, just to get some faint smudge of a distance nebulosity, I can grab what is there and process it myself. Find portions of the sky that are unloved, and bring them to life. Produce vistas different from the standard fantastic images. Get off the universes beaten track, and maybe even produce something no one on earth has ever seen before.

Hold on!OK, first admission. While the first picture of the Pleiades and Jupiter was taken from my back garden, the last few pictures you have been looking at were taken from the Digital Sky Survey (the second), and processed on an eight year old MacBook on my sofa. Using perl.

And how cool is that? So let me tell you how I did it, perl as glue, CPAN ignored, new code thumped together in fifteen minutes to do some simple celestialgeometry, and then on to bigger and better things.

Well, the initial script took fifteen minutes, the whole system took a bit longer. But lets make that journey, with the mistakes and bloopers laid out bare for you to see.

Come with me on a journey, not only of sight and sound, but of the mind. Actually, no sound, unless you could me swearing at my own stupidity during the development process.

OK, first admission. While the first picture of the Pleiades and Jupiter was taken from my back garden, the last few pictures you have been looking at were taken from the Digital Sky Survey (the second), and processed on an eight year old MacBook on my sofa. Using perl.












God bless the intertubesOK, first admission. While the first picture of the Pleiades and Jupiter was taken from my back garden, the last few pictures you have been looking at were taken from the Digital Sky Survey (the second), and processed on an eight year old MacBook on my sofa. Using perl.




http://skyview.gsfc.nasa.gov/current/cgi/query.plSo here is the lovely web 1.0 interface that NASA make available for researchers to grab data from a vast array of satellites. There are also a myriad of options, geometries and more below out of shot, but I just want to show you the datasets you can get.

Too many options, and I could go in to all of them for hours on end. I dont have hours on end. If I am running to schedule, I might have another fifteen minutes.

So, Ill take a single route through this. I am going to use the Digital Sky Surveys second output, and just the red and blue channels.

Notice you can get infrared, ultraviolet, gamma, all sorts from there. And including some of those to generate the false colour images is fun.

But lets grab some data and see what we can do.

When this site was first brought to my attention, I did this by hand a time or two, to see what I could make of it, and if it lived up to what I imagined, and whether the data was beyond my talent.

FlexibleImageTransportSystemOh, I have plenty of star atlases at home, YMMV. I can recommend some if you want, or use your favourite search engine to look for well-worn parts of thesky to view, then move out beyond the usual catalogues.

Either way, you need some coordinates. Plug them in, and click submit. That is a small pixel image, illustrative of what you will get from the site.

But the important bit is the download FITS link. As that is what we want. A FITS file per colour channel. Red and blue, that there showing you the blue of the centre of the cresent nebula in Cygnus.

So, FITS files seem to be the format of choice for astromoners. It has metadata in it, and is really rather handy for astronomical image data. But a bit of an arse of the beret-wearing artists, as you cant process them the same way you would your photographs. At least I couldnt.

NASA helpfully also provide a tool for messing with FITS images, called FITS Liberator. It is a bit memory hungry, too. But, you can load up the image, push sliders and tweak numbers, alter geometry and save it out as a tiff file, for processing in your favourite image manipulation tool. Which in no way is the GIMP.

But at this stage, we are still doing this all by hand. And that cant be right, as this all seems like what computers were made for.

Lame

So may as well cobble a quick script to grab the data, and avoid all that using-a-browser nonsense. Even writing a grab-a-750px-preview script meant I could avoid the browser. I hate browsers.

But a bit of LWP, submit the form, get the FITS link and download the file. For both the red and the blue channel. A word of warning, though. A full 6000x6000 pixel FITS file is big. Then multiply that by the number of channels you want.

Even so. I could now give a script some coordinates, and have it fetch me the files. Which I then could stuff into FITS Liberator and photoshop and be poke at it by hand. Some automation is better than none.

Lame

Sharpless 105So it seems I dont need FITS Liberator to turn the FITS files into tiffs, and I dont need Photoshop to turn the grayscale tiffs to RGB, nor do I need Photoshop to layer and colourise the image.

I can script all that too. And there we have it, one of the first images I created by hand re-created automatically using my script. Not perfect, as a human can tweak the image to make it more to their personal preferences, but that isnt the point. I now had a virtual telescope that I could give coordinates and it would show me the glory of the universe.

I was reasonably pleased with this.

And yet

it was only the startOnce I had this going, I fiddled with the colour settings. Most space images use what is called the Hubble Palette, but dont feel you need to stick to that. Especially when you add in other satellite data. It is tough adding in the gamma data, and some of the images look really freaky, but it is still fun to play with.

A mindless bit of fun for half an hour. With results that ranged between pleasing fantastic and pleasingly quickly removed. Some things humans are just better at.

I need more seconds of arcThen I started to think bigger. And bigger means more seconds of arc across the sky.

Then I could stitch them together by hand, to get something...even more majestic. And even more...crippling on my machine.

One of the parameters available is the angular size. For single plates I was varying between 0.5 and 5 degrees. Obviously the wider the request, keeping the image size the same is like using a shorter lens on the same viewpoint.

I decided to keep the angular size the same, and with a bit of maths I could get the coordinates around the centre, then get all those individual plates.

And how do I get those extra coordinates?

Well, what does every lazy perl coder do when they want something? Look at CPAN. And I looked. And looked. But damned if I could find anything that would do what I wanted. Simple celestial geometry. Give it RA/dec, add a bit, subtract a bit, give me new coordinates.

Seriously, nothing?

At this point, given it is a simple problem, I decided I'd hack my fetch script a litte. Not too hard, really.

OK, so I lie, there are some spherical geometry modules, and even Astro::Coords (which is ace, and in the end when I had written some stuff I could replace little bits and pieces with those objects.)

I keep mentioning the coordinates, and I dont want to give a lecture on that, but we use what is called right ascension and declination, although other systems are available. I still think in RA and dec, all the same.

Just think rectangular polar coordinates and youll not be far off.

I think I have more seconds of arcNo doubt someone will tell me where the module is, and what I should have used, but I'll add writing spherical celestial coordinate systems to my what-coders-all-do-as-a-rite-of-passage, along with a templating system and an ORM.

Now, at this point, I am still sitting on the sofa, and not really caring. And I should have done, had I thought I'd be sharing here. But whatever don't be that guy in the second row sagely telling me what I should have done, and how I should have done it. Or, dont be that girl in the second row. How I'd love to see that phrase change. Don't be that person in the second row. Or do, I am quite happy to clash wits (and fists if you are that way inclined).

Anyhow. I wrote some quick code, naive and dull, but it worked.

And my total time working on this code has risen from fifteen minutes to about an hour. I was getting distracted, Im not as sharp as I used to be, it was raining.there was an earthquake. And I have no shades to remove in this cultural reference joke.

Horsehead nebula and surroundingsNow I have a script that will fetch data from a given space telescope, shift across the sky (and down), to give a bigger viewpoint. Space is big. Youwon't believe how vastly, mind-boggling big it is.

But, err, that isnt quite there yet. Something has gone wrong. But hey, not far off, right?

Useful for huge emission nebulae, mostly. Some of those are fantastic. At this point, Id definitely recommend you look into some of the gamma/ultraviolet data, and incorporate that. Although by hand, for now. Until you decide how you want that added in.

Whoops

but even soLets check my maths (at this point I have textbooks on the sofa beside me, I much prefer that to having to open a browser again).

Fire it up, grab the plates, convert and tweak colourization, then stitch them together side by side.

North America nebula

not quite there yetDont skim read your text books, youll miss a step or two, and your pen will slip while scribbling on your notepad

Go big or go home (and get your textbooks)

Take is easy, deep breath, and we get there. Seamless stitching, huge seconds of sky. And well, a happy me. Come on, that is seriously cool, no?

one more tweakCant resist it, really, always try to make it a bit better, just enough so you can stop for a rest, and see what works your hand has wrought.

and choose your target wiselyBut a word of caution. While lots of the sky has been mapped, not all is to the same precision.

There are degrees of sky that are dusty (and not the good interstellar dust), break, flaws, and bad data.

But hey. That only happens in the most off the beaten track of the off the beaten track places, which are the ones that interest me, I like the open clusters and diffuse clouds.

This took ten and a half minutes to fetch, colourise, convert and stitch on this crumbly old MacBookM51So here is the fifty-first entry in Messiers catalogue, 3 x 3 degrees of sky, generated by a script, the only human intervention to give it the central coordinates of the interacting pair.

As an aside, most reputable cosmologists will cite this as a pair of interacting galaxies. Obviously. But they tend to say that it is a collision.

But they would be wrong, along with all the hot Big Bang peddlars. That is ejection, and Hoyle was right. But that is a whole other can of nucleosynthesis I am not going to get into here. Just to say if you believe the line you are fed about the Big Bang, well, you certainly arent thinking for yourself, and are being conned by those who ignore observational data all around them.

there is always work to be doneMy maths isnt perfect, my stitching is sometimes obvious, yeah yeah, whatever

There are lots of image algorithms lying around, so I got some adaptive smoothing ones, some contrast setting ones, some better stitching ones so I didnt have to precisely line up the images. I even dug out an old graphics maths textbook from my writing video games days, which helped.

And after I had done all that, another throw-away comment about Hugin made me think that I neednt do all that in pure perl. Ah well.

but even so

Monoceros

aint it cool?Seriously. Isnt it? It isnt just me. I hope.

Yes, I am a riot at parties. And yes, I have used the line would you like to come to my observatory and see Andromeda?

Deneb 2 degree

Deneb 1 degree

So a quick comparison of one of my fave summer sights, as it means open cluster season is upon us. Deneb. A one-degree single image, and a two degree magically stitched star field.

There are endless tweaks and configuration options I could make, but havent. Suffice to say that was a start, and just shows what can be done from sitting on the sofa pondering a throw-away line I was fed about the telescope data.

And perl makes it all so easy. Sure, it isnt all perl, I used Imagemagick, but I wouldnt have like to have done that in java. Fortran, for old times sake maybe, but only under heavy drinking conditions. Perl gets the job done. I dont even need to write the core maths in C, even this old machine can handle it. I am not handling Big Data, although I have plans to make an entire image of the sky above my house, some kind of gigapixel mosaic. Maybe.

And my work was done.

Dance to the music of the SpheresSeriously. Isnt it? It isnt just me. I hope.


Seriously. Isnt it? It isnt just me. I hope.


Seriously. Isnt it? It isnt just me. I hope.


Can I squeeze a live demo in?Seriously. Isnt it? It isnt just me. I hope.


Coming to a cpan near you soonSeriously. Isnt it? It isnt just me. I hope.


References and help

There are plenty of libraries to do lots of this stuff, like the NAG stuff, or StarLink, or just look around the academic papers and grab the algorithms and code it yourself, which aside from the converting to tiff, is what I did.

There are plenty of books, too. Aside from general astronomy textbooks, I heartily recommend the Cambridge Star Atlas, and Turn Left At Orion, if you are wanting to get in to actual physical observations yourself.

The Digitized Sky Surveys were produced at the Space Telescope Science Institute under U.S. Government grant NAG W-2166. The images of these surveys are based on photographic data obtained using the Oschin Schmidt Telescope on Palomar Mountain and the UK Schmidt Telescope. The plates were processed into the present compressed digital form with the permission of these institutions. The National Geographic Society - Palomar Observatory Sky Atlas (POSS-I) was made by the California Institute of Technology with grants from the National Geographic Society. The Second Palomar Observatory Sky Survey (POSS-II) was made by the California Institute of Technology with funds from the National Science Foundation, the National Geographic Society, the Sloan Foundation, the Samuel Oschin Foundation, and the Eastman Kodak Corporation. The Oschin Schmidt Telescope is operated by the California Institute of Technology and Palomar Observatory. The UK Schmidt Telescope was operated by the Royal Observatory Edinburgh, with funding from the UK Science and Engineering Research Council (later the UK Particle Physics and Astronomy Research Council), until 1988 June, and thereafter by the Anglo-Australian Observatory. The blue plates of the southern Sky Atlas and its Equatorial Extension (together known as the SERC-J), as well as the Equatorial Red (ER), and the Second Epoch [red] Survey (SES) were all taken with the UK Schmidt. All data are subject to the copyright given in the copyright summary. Copyright information specific to individual plates is provided in the downloaded FITS headers. Supplemental funding for sky-survey work at the ST ScI is provided by the European Southern Observatory. Yeah, read the licence. It is meant for research purposes, and dont misuse it, as Id hate for them to pull the plug on it.

but even after all that

Jupiter and the PleiadesAfter all that, *this* is still my favourite image. Why? As I collected the pixels myself, in my back garden. Sure, it isnt as impressive, but that isnt the point. This is *my* image. Humans are strange.

Thank you!

(and thanks NASA)

Click to edit the title text formatClick to edit Master title style

16/08/14