Today, after the 8 hour “Industrial Control Systems Security for IT Professionals class”, I wanted to make something pretty. And code. And work on a protocol problem.  I’ve needed to look a little at the new Stimulus bill for work lately, so I thought I’d try and at least say I’d written  Python today, dissect the text of the bill into parsable chunks, then throw it into some visualizations.  I can’t easily capture the interesting avenues of analysis I was pursuing visually (and I dont feel like writing it up), but I did manage to make some kind of pretty pictures. Hopefully someone feels inspired from them and goes down a similar path. (I already have some ideas at further stats I want to parse from the bill to be able to look at it more meaningfully. Perhaps Ill do it this weekend – this was just the first cut at setting it up.)

First, I grabbed the full text of the bill from HERE. Then, I wrote some (stupidly) simple python (again, I’m never sure if it’s -good- python) to parse the bill and turn it into a new file with five columns: Word Number, Word Length, Line Number, Work Position in Line, and the actual Word itself. This essentially turned the bill into a a text file with every word in the bill on its own line (in the order it showed up), but with  machine readable meta-data I could use to visually represent it.

stimulus = open(‘/Users/sintixerr/Documents/stimulus.txt’, ‘r’)
finalfile = open(‘/Users/sintixerr/Documents/sdump.txt’, ‘w’)
linenum=0
wordnum=0
lineposition=0
gstruct=[]
for line in stimulus:

lineposition=0
linenum+=1
word=line.split(‘ ‘)
word=word[:len(word)-1]
for w in word:

lineposition+=1
wordnum+=1
gstruct=str(wordnum)+’\t’+str(linenum)+’\t’+str(lineposition)+’\t’+w.upper()+’\t’+str(len(w))+’\n’
finalfile.write(gstruct)

stimulus.close()
finalfile.close()

Then, I opened up the new tab delimited bill in my visualizer of choice and ran it through a few different ways of representing the bill.

First, the raw text – without any real manipulation – looked cool in and of itself and I noticed some interesting, if obvious in hindsight, features. (I did clean out some obviously bad data first with a little  sed action, but that mostly just involved removing punctuation that caused the same words to show up as different ones. )

Stimulus Bill Visualized in its Entirety. In this image, the Y axis represents every word (ASCII characters with spaces or carriage returns on either side) in the bill and the X axis represents the Line Numbers those words appeared on.

First, if you look about a fourth of the way from the left, and then again closer to halfway, you see a vertical “break” in the scatterplot where it looks like the density is much lower.  That is probably a major section break in the original document (I honestly haven’t actually read it in english yet).  That possibility is supported by the second observation which is: Even in human written documents, you can still discern protocol visually. (Again, obvious, but it’s neat.).  If you look at the bottom third of the image, it looks nothing like the top 2/3.  Much more curving paths, fewer horizontal lines, less density, etc.  If you look at those “words”, they’re all document structure words (like section numbers, headings, etc.). …and monetary figures.  If you look closely, there appear at first glance to be two or more incompatible or unrelated document content structures there.  Above that section is where the more obvious “free form” english exists in the set.

Moving on from there, I wanted to see if I could get anything intellectually or aesthetically interesting by using a scatterplot to draw out the shape of the bill.  To do that, I plotted “Line Number” on the X axis and “Position of Word in the Line” on the Y axis.  (Actually, originally those two were swapped, but the resulting image “looked better” when I swapped the X and Y).   I colored everything by Word on a categorical scale so things wouldn’t blend together too much and then ratcheted up the size scale to reduce empty space. I was looking for a visual representation of the literal structure of the document, not an analysis tool or I wouldn’t have done that last bit.

The resulting image looks like this:

Shape of the Stimulus Bill on its side. If you were to compress the actual text of the whole bill into one page and rotate it 90 degrees counter-clockwise, it would probably have the same shape as this, only with text.

Finally, I was curious if I could do a little manual clustering work. I tried to narrow down the words into the data set to those that might have some intrinsic meaning in the context of the stimulus bill. This means I got rid of prepositions, repeated filler words, etc.  I did this by knocking out every word under 4 letters and all of those over 17 chars (over 17 were all artifacts of turning the bill into something parsable, not actual real words).  Then I created a bar chart of words and sorted it by how often words appeared in the document and removed about the bottom 70% of words. I made an assumption (which is almost definitely so broad that the data will have to be sliced again a different way for meaningful analysis) that any words that weren’t repeated that often just werent a real “theme” to the people writing the document. Interestingly, things like “security” and “health” and some others were left in the set, but “cyber” was removed. Hmm.   After that, I went manually through the remaining set of words and removed those that seemed to not have any cluster value (both through intuition as well as by visually watching the scatterplot of the whole set while I highlighted individual words t see what lit up.) Finally, and lastly, since I originally wanted to make visually interesting things more than do real analysis, I used some blurring, resharpening, and layering to give a more cloudy, vibrant feeling to it.  Interestingly, that created “clouds” around many of the clusters and made them easier to make out for analysis.  That supports my whole theory that what the eyes and mind like to look at is what the mind and eyes are better able to make intelligent use of.

The final result is here:

Words of substance that might be indicative of topics or subjects within the bill. X axis, like the first picture, is line number and Y axis is Word.

Color updates on the mosaic generator have now been made. It’s a bit slow – since it operates on all three color bands independently of each other – but the effects are kind of cool. The questions the code asks you make more sense, now, and you no longer have to specify the base image on the command line.

I’ll release Windows and Mac binaries in the next 2 days or so. I also threw a GPL open-source license on it.

You can find the code here (and I know it’s still not completely “python-like”, that’s next):

http://sintixerr.wordpress.com/tone-altering-mosaic-generator-tamogen-in-python/

For sample images, I used two old conte-crayon pieces I’d done and combined them. The first image is of the “whole” mosaic. The second is zoomed in. I love the color variations the code creates when it makes the mosaic – I think the second image could be a standalone piece of art in its own right:

First, I finished the python code I was working on that will allow two -color- images to be merged into one color mosaic. The color transformations it has to make to fit in the smaller picture to the larger one seem to result in some pretty wild effects – I’m digging it. I’ll clean up the code and post it here tomorrow.

As far as social stuff goes: Angela Kleis’s blogger night at Artomatic was pretty cool. I don’t want to post a lot of thoughts on that yet (I will tomorrow), but it did reinforce the fact that a lot of event management will have to be done at the June 6th ArtDC Artist’s tour dinner. Unfortunately, people have short attention spans and the time each artist speaks will have to be managed and expectations set ahead of time. 5 minutes seems to be about the “max”. We’ll have to bring a timer or something. It’s going to be a -really- interesting night, though, and a lot of fun.

More info on the upcoming dinner can be found in this thread:

http://artdc.org/forum/index.php?topic=8997.0

Pictures of Blogger’s Night can be found here in a set:

http://www.flickr.com/photos/sintixerr/sets/72157605133432629/

Finally, Erin Antognoli took a couple of great shots of my space while I talked about it to what was left of the crowd:

Erf. I just got done saying I wasn’t going to do one of these (Artomatic Experience Posts), but I just saw the base image assembled as a whole for the first time and was excited enough that I wanted to post pics of the image at least. Just getting that out of the way first – this is a blatant excuse to post pics..(yet somehow this is long-winded…I have nights like this).

Artomatic Piece

Originally uploaded by sintixerr

So -

As mentioned earlier, when I first really started thinking about AOM this year, I thought I was going to be in Arkansas for the majority of the show and wanted to put together something “simple” (yeah, right, me? simple?). I was so strapped for time, in fact, that Angela and Paivi had to grab me some space for the second year in a row while I was out of town. ( My space is on the 8th floor at SE D6 btw.)

Ok. Great. Have space. Project? Art? No so much.  Ultimately I decided to do a huge (6′x6′) self portrait installation (covered in earlier blog posts – http://sintixerr.wordpress.com/2008/04/27/python-photography-digital-art-artomatic/ )

So, uhm, how do you print a photo that’s 6′x6′??? I ended up deciding to print the picture in 9 sections, 24″x24″ each. I originally wanted to do an 8′x8′ image, but at 24×24″ sections I could go one price category lower at Chrome than 32″x32″. Still, lots of cash. Hrm. Where else will print this? For a much more reasonable sum?

It turns out that -Ritz Camera- of all places will print really large images for less than half the price of Chrome. Supposedly archival. I test printed one 24×24″ section and it came out perfect. Sweet. I had a printing solution.

For framing I went with Angela’s suggestion of American Framing and picked the smallest frame borders I could – with no matting. The ultimate effect would hopefully look like the face was peering in through…something (window-esque?). That was an easy process and everything showed up toot-sweet.

I went with overnightprints.com for the business cards. They did a good job but, in hindsight, I didnt. I think my cards are a little bit juvenile and cheesy, but, we all make mistakes sometimes I mean, who puts “Artist” on their cards? I guess I do. Meh.

While I was still playing with my final image, I met Angela and Paivi at our spaces last Sunday to help them paint, do lights, and take a look at my space.

I got there first and spent an hour or so mocking up what I wanted things to look like with rope and gorilla tape (it made sense then). The final result was looking pretty good and I was feeling very satisfied. Until Ang and Paivi showed up.

“Thats not your space, Jack”

“Whu?”

“Thats. Not. Your. Space. Its the one next to it.”

–Insert vulgar words here–

Luckily I hadn’t done anything permanent and I sheepishly took down all my stuff. At any rate, I knew what it was going to look like and everything was good.

Angela and Paivi that day managed to paint their space, buy lights, put up lights. I managed to…put down duct tape and put up one wall of the shadowbox (seen in pics later). Really, I hate this part. Im slow as HELL at painting walls and usually a mess. People always regret asking me to help them paint their house/interior. I suck at it.

I came in the next Thursday after work by myself to try my luck alone and got a lot further. I managed to put up the second shadowbox wall, put down a layer of paint, and not end up with blood or paint all over my clothes. It was a good day in that regard. Still, I had forgotten paint rollers and had to use a brush. Those partitions, even with primer, suck. up. the paint. I was despairing a little bit about how things were looking, but put that aside knowing how much was left to do.

So far, I’d run into a couple of people here and there that I knew, but the AOM space is -so- large this year that unless youre actively wandering around looking for people or just outright lucky, it’s entirely possible to work with a dozen people you know there at the same time and miss them completely. (Except for Melissa, whom I run into every single time Im in the elevator.)

This past Saturday night I had my first volunteer shift with Angela and Paivi, so I came in ahead of time to get some work done. I met my “partition-mate”, who’s name I can’t recall at the moment, but who’s work I had been familiar with and am really psyched to be next to. If I had space in my apartment, Id buy some of it honestly. More painting ensued and things were good.

Queue the volunteer shift.

I have a gripe here. We were given a 3 minute introduction to our responsibilities by the GM on duty. The instructions did not mention two big things that we were asked about -all- night:

1. Lights. Anything about them. Where to get them? What to do if any were broken? Etc. I know this was a GM responsibility, but by Saturday you’d have figured that there were enough questions being asked that the volunteers should be given some sort of heads-up.

2. Brightest Young Things:

Anyone who was at AOM that night figured out very quickly that there was some shindig on the first floor. The only instructions of info we were given by anyone about the first floor party was “oh yeah, tell people bathrooms are on the 10th and 12th floors if they start bitching about the first floor bathrooms”.

What about the first floor bathrooms? Huh? And who’s making all that racket next door? Apparently Philipa Hughs’ Pinkline and Artomatic collaborated to help throw a relaunch party for a website called BrightestYoungThings.com ( http://www.brightestyoungthings.com/events/byt-relaunch-party-1200-1st-street-ne/ )

So, in addition to AOM people, there were tons of the trendiest, coolest, hippest, artsiest, brightest young things wandering in and out of the front door. Nothing against them (and it was cool something else was going on there), but it was really a pain in the ass to do “lobby” duty while there were bands playing, a plastic fence between us and them, etc.

I was asked to “keep people from there going up to AOM”…but really…how was anyone supposed to tell the difference? There were also a lot of people bringing in artist materials through the lobby that I didnt catch because I couldn’t always pick them out from the crowd milling about in the lobby.

This all would’ve been cool if we had been given heads-up….but we weren’t. We were left to piece it together ourselves…

I also wish I’d NOT worn the -skankiest- shirt I own to paint in. Felt like I was bringin the place down just sitting next to the party Wink

The monotony was broken up a bit by a trip to Sketchies (aka Wendy’s) to bring us back some much-needed diet cokes (Angela and Paivi were covering Loading Dock duty that night) and also by Jim, who gestured for another volunteer and myself (I think he meant me too) to come out front where he proudly showed us the new lighting for the Artomatic signage in the windows. The “ART” in “ARTOMATIC” was glowing red! Cool!

By the end of the shift, we were all -done-. Id gone running that morning for the first time since the Cherry Blossom 10 miler and between that, working on the space, and the volunteer shift, I was no longer human. Paivi wanted to take some pics of the graffiti kids up on our floor (Cool doing their thing (didnt work out), but we did run into Halo and Arty4ever putting down some finishing touches on Michael’s space (which looks great!).

I also met a couple of cool volunteers, but unfortunately was too brain-fried to ask for the business card of one of them and now forget her name. Ill need to check the volunteer list later if I can…

That brings me to today. Lights are up, painting is finished, etc. I just need to get some fabric to frame the shadowbox, hang the piece, and add some other artistic touches to the installation (and it will be an installation) that I dont want to describe here. I managed to frame all 9 sections (woohoo – Im really slow at that, so Im proud of myself here) today as well!

Ok, whew. Finally, this behemoth of a post is at it’s end.

(As a side note, the lighting in the first pic is provided by a couple of White Lighting 10,000 strobe, which we’ve now started using as generic apartment lamps lately. Heh.)

In the last post I alluded to the fact that what I was creating for Artomatic was going to be a little bit more holistic and effect focused than in the recent past. This year, I’d like to get into blending media, rather than focusing on it.

I’m still mulling over what I want to say about the whole thing, but tonight I’d like to offer a prelude by way of a small technical glimpse into the core of the piece.

Essentially, I’ve been wanting to work on the idea of self for some time. The Second Life work had addressed some of that, but -primarily- from a perspective that was fairly extreme and lacked a lot of emotional resonance (not least with me).

So this year, I’m going to be mixing up my newfound interest in photography with some traditional sketching technique, adding a dash of emotional investment, and finally tying it together with some custom computer scripting.

It’s this last component I want to talk about. There are a lot of “themes” Im starting to address with this piece (later post) which made me feel like I needed to use some sort of mosaic. Initially, I thought I was going to do a self-portrait of myself….composed the very same self portrait. This would constitute the anchor concept of the piece.

The image was originally supposed to look something like this (right image is zoomed in on the left eye where you can see the main image is composed of many copies of itself)

But how does one go about creating this kind of mosaic? Doing it by hand would take so long I’d never finish!!!

Maybe there are programs “on the internet” that will do it? Maybe (it turns out there -sortof- are), but what’s the fun in that?

Instead, I decided to write my own program to do it. This has a lot to do with the fact that computers are a huge part of who I am and writing code to help me generate a self-portrait struck a chord with me

I had played around with the Python scripting language back in my NetSec days (for rapidly setting up data analysis) and while I had never been particularly proficient in Python and it had been some years since I’d worked with it, I -had- enjoyed it’s way of dealing with the world of bits and bytes. it seemed like a great language to try this out in.

So, I grabbed my new Macbook (heh. Microsoft forced me into -that- with Vista) – which had python already on it – and sat down to write a self-referential mosaic generator. Away we go!
Hrm. Or not. How does one actually go about editing images with a scripting language often used in web pages? It beat me! I’d never done it before. In any language.

More Hrm’s. Google kindly suggested I give the Python Imaging Library a try, so I checked out the tutorial online. “Wow. This might be doable”, I thought to myself. This looks like a really simple library.

And it was….the code below took about 3 days to write starting from “Uh, how do for/next loops work in Python again?”. So while it’s not a LONG program and it’s NOT elegant, and it CAN be done a lot better, it does the trick.

In fact, not only does it do the trick, but the code now lets you specify which image to use to recreate the base image of the mosaic. (And the concept of the art piece has followed with that, Im now using two self-portraits.)

The program I wrote is a little different from web-clients for Mosaic creation I’ve seen. Those go and grab images (often at random) from a repository and create a mosaic of another image out of them based on which filler-images already best-fit which piece of the base image. The filler images, themselves, arent altered.

In my code, the user specifies one filler image and one base image. The code then goes through and checks tone averages and alters the filler image to fit into a given section of the base image. If the filler’s average tone is higher than the current section of the base image being converted, the program darkens the filler image and then pastes it in.

The ultimate effect resulted in this image created from one base self-portrait and one (different) filler self-portrait (Click it so see the smaller faces):

The code which produces this image  can be found here:

http://sintixerr.wordpress.com/tone-altering-mosaic-generator-tamogen-in-python/#comment-2512

Please be kind…it really isn’t pretty yet and I know it

Finally, a snapshot of what it looked like put together at Artomatic: