Noise Derived

My Live Image Processing final ended up being a bit different from the idea I recently posted. I simplified it to underline my grasp of the fundamentals of Cinder and GLSL. I’m happy with the results, given that I developed both of these in a 15 hour marathon that came on the end of two weeks of finally digging in to the language and the framework in tandem. Most of my classmates’ work was developed in Max/MSP.

There are two versions of the idea. They both are remixes of my favorite visual art piece, _Grau by Robert Seidel. I misappropriated the video from vimeo.com and offer my apologies to Robert.

The first remix creates a kaleidoscope of the moving image using one texture object in an OpenGL shader. The mouse movements control spread and rotation as do some of the number keys.

Grau – Kaleidoscope

The logical next step of this project is to offset the layers in a stereoscopic projection and output them to a headset. I think that given the right perspective lines, you would end up with a tunneling effect that is similar to what Patrick Hebron is working on. It was also inspired by a piece Don Miller did for the spring show at ITP.

Also, the transitions become pretty useless with more than 20 frames running, although I seem to recall from Exposé that I should be able to run many more movies at the same time if I could streamline the pipeline. That would probably require bypassing Cinder all together and finding the right codec, and at this point, it’s just a thought experiment.

I could stand to practice the performance aspect of it, increase the resolution, cut out the dithering, and combine the kaleidoscope with some multitextured compositing. I attempted to use Framebuffers to maneuver around these issues, but the geometry got screwed up and I ran out of time.

The second remix is a simple compositor based on Desaturation, Screening, Color Dodging, and Multiplication. This was intended to get through the concept of multitexturing using shaders, but in the end I wasn’t able to go where I wanted because I’m not sure how to best leverage the 8 multitexture units into a block of, say, 512.

Grau – Composite

Making a music video is kind of a coming-of-age thing to do. It’s technically rich in terms of the kinds of things you have to know to make algorithmic magic. People respond to it. It’s can be a good reel for your work. So, it’s a good summer project.

Luke DuBois’ Live Image Processing class is mostly an extended demonstration of Max/MSP. Max is interesting but not my cup of tea, at least not right now. I’ve been trying to scale the fortress walls of Cinder, and I think I’ve for the right project for it.

Cinder has its own quirks. Perhaps they’re just advanced programming techniques I’ve never encountered, but there seems to be a leverage point between Surfaces and Textures that’s allowed me to dig into the code a bit. For example, the way colors are embedded in Surfaces are strange to me, but trying to figure them out has been a noble pursuit.

Cinder doesn’t have well formed audio support at this point (as of version 0.8.1), I’ve been digging through the internetz for explanations of Fourier Transforms (which in and of itself is incredibly useful). Probably the best explanations I’ve found are here and the Stanford class in iTunes University. This coupled with the CoreAudio documentation has me fairly convinced that I can get the signal out and turn it into something useful. The CoreAudio Sonogram example gives some good visualization starting points, and even gives you a rough idea of how your track might end up looking.

Finding music was a night long project. I searched for tracks that had a tonal variety that might translate well into a range of visuals, but this might be a fool’s errand given that beauty might also come out of the rhythm in a way that isn’t obvious by listening to it. Anyway, I landed on the first track to be released from the Scissor Sisters album for a number of reasons. I’m a little concerned that by the time I finish the exercise, Invisible Light will have worn out a bit, but it doesn’t seem as if the band is going to put one together, and maybe it won’t matter.

Of course, the intended visuals always change as soon as you find a new way to do something.

Originally, I thought about taking an OBJ model of a guy and running things across the surface of it with most of the work in the camera movement.

Then I though I’d like to paint the band members’ faces across the model, so I picked out some youtube footage that would work well.

Now, I’m not so sure I’m going to use the model (maybe as a depth map). I spent a night learning keying in After Effects and did a very dirty matte. I’m not too concerned about the quality because it was really to just deemphasize the background, and at one point I thought the faces were going to be particle systems sliding across time anyway (they still might be). The resulting test looks like this:

testJason

The concept as it is right now (Sunday at 2-ish) is to remix this footage into some perlin noise clouds.

• Load the movie as a texture or surface (?)
• Create a grayscale channel for the data
• Posterize it into 5 levels
• Create 5 transparent Perlin noise spaces, incrementing octaves and frequencies
• Slave the Perlin space to the posterized colors
• Run a test by putting the frame of video in the middle of the space and activating the opacity of the noise in the layers both towards and away from the fourth wall, increasing the radius of the opacity in each layers, but also decreasing the opacity itself according to a decay.
• Add in the FFT data to activate areas according the audio response, also change the depth of the activation layer according to the amplitude of the frequency.
• Maybe add particle systems, emissive lighting, and backlighting created from the original footage

Ultimately it sounds doable but will also take a lot of rendering. And, this is just the first test of one layer of visuals, so, probably this will take a while to finish. More to come, hopefully.

I knew going into Device Independence for the Mobile Web that it would be a survey of web technology. What I didn’t really wrap my head around was that it’s next to impossible to find time to explore new technologies on your own during a summer class. Things move too quickly.

Rather than revisit the midterm I made of Dan Shiffman’s flocking code rewritten in Javascript, I landed upon redesigning the ITP People Directory for a more mobile/social experience.

The ITP People Directory has a lot of potential when it comes to strengthening the Alumni body, but its functionality is rather limited at present. I want to create an umbrella under which some of the major activities of student and alumni interaction sit, such as archeology, payment transfers, presentations, and S.O.S. requests.

The ITP Mobile Directory has six functional areas that operate off a simple interaction model:

1. Feed
2. Threads
3. Pile
4. Community
5. Archeology
6. Settings

A dashboard drops down, much like the Mobile Safari chrome, to navigate through the application.

The model centers around a searchable “Pile,” in which the user adds any of the elements she wants to keep handy, kind of a like the Newton soup. To add to the pile, one would swipe left on the div. To delete the item, one swipes to the right. Drilling down or causing an action to occur is triggered by a tap. Pinching a list in will apply a filter that restricts the information to the user’s class or favorite users list. Pinching out will remove the filter.

The feed is a place for a news cycle of five types of information:

• Twitter
• Blog Blender
• Calendar
• Requests
• Media Links

The intention is to provide a continuously updating and brief summary of current activity. Twitter statuses (and perhaps Foursquare) put you in touch with your friends. The blog blender showcases the current projects documented by students. The calendar is an outreach tool for the staff. A student can request help or compatriots for food. Finally, media links are provided for entertainment.

The student and alumni lists are explicitly sectioned off to provide some regulation on the information flow. In this case, they would be threaded as well as provide some ability to post A/V items. Tapping on a thread count opens the thread. Tapping on a content item allows you to reply. Shake to undo. Buttons are provided for undo and creating a new thread.

The pile is simply a blend of all of the items that are saved by swiping left in the Feed, Threads, or Community Profiles. The pile is generated by a local storage list. The content could be stored in an offline database to facilitate faster searching.

The community is explored through constellations. Each user creates a profile that is then augmented by picking out a core group of associates as well as a natural language search of blog linking and perhaps class linking to suggest additional relationships. A sorted scroll might also be provided, but the point is really to provide a web-like linking of individuals.

An additional rating system might be implemented to categorize the types of work students and alumni are involved in. Generative keywords could be visualized based on the content of blogs, media links, and associations with other community members.

When a user is selected, their personal feed of links, blogs etc. will show up in a cordoned section at the bottom.

Finally, a link through the Paypal API is provided to facilitate shared ordering of parts, food, etc. by tapping on the user’s contact information.

The archeology section, not pictured here, is a gateway to past work, the itpedia, the physical computing reference, and other resources. I hope that this section would give more substance to the collective memory of the student body and provide platforms for truly innovative work.

The final section is a list of settings that would customize the program.

The backend of the application is a collection of HTML5, CSS3, PHP, Javascript, JSON, and RSS. A simple, but central database acts as a blender and filter of information served based on the login. Preliminary diagrams are provided below.

An excerpt from my new blog garden.neocyde.net, which is the documentation of my thesis at NYU’s Interactive Telecommunication Program.

In my opinion, a few important characteristics can augment a data set to a new breed

1. A data set is collected indefinitely without significantly changing the survey. (ie. US Census)
2. The data are either continuous or have several to many categories for each variable.
3. The data reflect changing behavior by dynamic subjects.

The net result of any combination of these characteristics may result in data that appears to grow, breathe, and die, much like the patterns seen in life.

“Organic” data calls for a novel approach to analysis. Traditional methods of employing size, position, and color to denote the context and comparisons between data may fail to offer intuitive insight into complex interactions encoded within billions of recorded data points. While these traditions may continue to add value to new visualization, I believe that designers have access to additional, undiscovered methods applicable to difficult presentations.

Behaviors may represent the lowest hanging fruit when looking for novel approaches to visualizing data. Behaviors can quickly create wide varieties of extremely individualistic and identifiable comparisons between isolated or groups of “citizens”. They do not require a complex set of rules to create fresh patterns. They are also relatively easy to implement, even if they are somewhat difficult to balance.

In Daniel Shiffman’s Nature of Code, we use flocking to demonstrate abstract methods of creating complex behavior. I hope the translate these and similar methods into a grammar with which to read an n-dimensional data space such as the US Census. My first investigation involved processing the US Census Bureau’s 2008 American Community Survey Public Use Microdata Sample in C++ and applying a rudimentary front end to debug the kinds of behaviors that might be useful.

I had been putting off posting the final project from my Visualizing Data class in the fall semester because it was one of those things where I could really see moving forward with it. This is, in fact, the ridiculous problem that I have with participating in ITPs shows, because I typically am satisfied with something a few days after the opportunity has passed. Anyway, I’ve come to the realization that it’s good to document it the way it was presented in class. It took about two weeks to code. If I had my druthers, I would layer out things in After Effects rather than compositing in Processing. The original file is 1080p and too large to post.

The visualization is an attempt to throw most of the human-created data from Wikileak’s 9/11 Text Message database onto a timeline. I curated the text messages from more than 500,000 lines down to 3,100 lines, highlighted the ones I thought were poignant, and added some narration from official investigations. I hoped that the overwhelming emotion of the data would emerge and obfuscate any particular detail.

nineEleven

I applied for an interesting workshop at Eyebeam this afternoon. Recording the responses here.
Summer School Masterclass: Visual Music Collaborative

Link to your website or a relevant work sample (1)
http://neocyde.net/derive

Link to your website or a relevant work sample (2):
http://neocyde.net/portfolio

Please briefly describe your experience with your preferred working tools, such as OpenFrameworks, Processing or equiv.:
I have over two years of autodidactic experience with Processing. As a data analyst, I created a sketch in Processing used by the Philadelphia School District to visualize the relationship between choices in academic programming for high school students and current jobs opportunities in the region. I also undertook several mini projects in Stewart Smith’s Visualizing Data Class at NYU’s ITP, including a presentation of the timeline of the Wikileaks 9/11 text message data. Recently, I have combined ideas from Dan Shiffman’s Nature of Code with Nick Bilton’s Telling Stories with Data to experiment with US Census data in OpenGL implemented in Python, OF, and C++. I walked through the tutorials for Cinder this week.

What is your motivation for participating in this course?:
The vision for my career is built around decision making grounded in ambient visualization of complex information. I believe that the increasing ability to measure phenomena and the declining cost of storage have led to an disconnect between the traditional tools used to disseminate information and bodies upon which they operate. Beyond color, size, and position lies relationships that may be intuitively understood through observation of behavior. My goal is to identify the methods through which complex data may be cultivated into an ecosystem that uncovers abstract but intimate knowledge of the system it represents.

What inspires your work?:
There is a pattern found in movement of wind through grass or light through water that is also evident in decision making of pedestrians crossing an intersection and the subtleties of surface in Michelangelo’s Pieta. This pattern, which may just be my own touch of consciousness, is what I’m chasing after. The greatest inspiration I’ve had so far is in the work of my peers at ITP, each finding vastly different ways to extend the grasp of experience. I take intermittent but deep inspiration from getting purposefully lost in landscapes. The ultimate expression of my life is probably in digital gardening, itself just an extension of the creativity of my mother, who keeps very wild gardens.

Ideally, what would you like to take away from this course?:
When it was just myself teaching myself, I really got going in Photoshop by mashing up NASA photos with some of my own photographs. Losing track of a night rocking out to Orbital and finding a new way to look at things is something I miss as I head down the path of poking through languages and IDE and databases for the next step in my game. While octrees and Xcode and Friendly Fires have their own brand of magic, I need to get back to seeing the detail on the screen jump beyond technical accomplishment into the fire of transcendence.

I came into ITP with the idea that you could garden data as you would garden wisteria. The courses this semester have been geared towards building a platform on which to build that idea. In the last month, my assignments have begun to converge into fleshing out a prototype.

My final project for Learning Bit by Bit, GLArt, Telling Stories with Data, and the Nature of Code will be an attempt to use behaviors of citizen objects to visualize selected variables from the 2008 US Census American Community Survey of New York City. Traditionally this information is presented as a series of comparisons defined by color, size, and position in charts and tables.

Several recent projects contain elements of the ideas that may be used. Daytum, created for Telling Stories with Data, began to build an environment that represented the data points collected in a one week survey of my computing. The visualization was a simple exercise in path following along a 3-space Perlin Noise path.

The Processing visualization did not calculate the position change fast enough for interaction or complex behavior. Ideally, the visualization would be responsive to interaction and robust in terms of backend computation. Significant power may be gained by moving calculations to the graphics processor using OpenGL and OpenCL capabilities. A project for GLArt recontextualized the Perlin Noise path as a Vertex Buffer Object in OpenGL.

In this case, the addition of noise was accelerated by iterating through a static array of noise values rather than calculating noise in every frame.

To move forward, I must assemble a collection of code in areas in which I have little experience. For quite a while, I struggled to find the cleanest environment for OpenGL and OpenCL by comparing languages and IDEs. I set up similar code bases in Processing, Python, and C++ in XCode. Despite the steeper learning curve with arrays, pointers, etc., I settled on C++.

Most of my time so far has been spent setting up the problem and exploring the right containers to put the data into. My goal is to keep as much of the data as possible in memory and minimize the number of transfers between the OpenGL server and client. I hope to use OpenCL kernels to run calculations on vertex buffer objects, but I am unclear on how this might be accomplished. In fact, there are many points of the process that remain unclear.

I outlined the steps to produce a visualization of simple attraction and repulsion based on the vectors inferred by the census data.

Setup

1. Parse the geography CSV file
2. Put geography into an object well suited for calculation
3. (Possible) Create custom flow fields leading to each Public Use Microdata Area
4. Parse the American Community Survey Data
5. Put the ACS data into an object that can be quickly looked up
6. (Possible) Compress the ACS data by some method – investigating support vector machines, self organizing maps
7. Create the citizen display list
8. Create an object to hold the citizen characteristics: position, alignment, color, size(?) – the size characteristic depends on whether or not I can achieve a 1:1 ratio of citizen objects to persons
9. Randomly distribute the citizen objects

Update For Each Citizen

1. Keep track of the visible citizens within the local subdivision of space.
2. Create acceleration vectors towards the citizens with like characteristics and away from citizens with unlike characteristics- perhaps weight these vectors with eigenvectors
3. Adjust acceleration in some kind of orbit that reflects the NYC geography
4. (Possible) Create a generational flow through which behaviors are encoded in genetic algorithms

There are certain challenges that I’m already aware of:

After parsing a CSV file containing roughly 63,000 rows and 25 columns of census data, I realized that I need to find a more efficient way of reading in the data.

The ACS data is missing values in many cells. Python requires masking these cells to create eigenvalues. I haven’t tested any method to create eigenvalues in OpenCL.

I have not written a kernel in OpenCL, nor have I passed this calculation on to be drawn without returning to the CPU.

I must find a way to create a kd-tree and keep track of the citizens local to each one.

I must also find a way to quickly move through the calculations for all of the citizens. This may not be possible. With a 1:1 visualization, there would be roughly 8.3 million citizens running calculations on each other.

The possibilities for this projects imply that there are many chances to improve the efficiency and sophistication of the algorithms which determine the behaviors. Many learning opportunities and probably many frustrations.

Daytum Visualization

Foursquare describes human behavior from their visits to crowdsourced geocoded locations. Our assignment in Telling Stories was to collect our own Foursquare visits for a week and then hand them off to be visualized. I got Eun Joo Lee’s data.

My initial reaction to her visits was that she spent most of her time in the East Village and Chinatown. Indeed, this was the reaction of most of my classmates to much of the data. My take on that reaction was to follow up on the fantasy that I have that there are layers in New York where people lead almost identical lives in different neighborhoods.

In this case, I wanted to turn the foursquare data on its head. Foursquare wants to sell the idea of a person using their configuration of paths back to the users and the locations. What if you took those same paths, in essence, the same arrangement of latitude and longitude differences and overlaid them around a different locus? What if you walked the same path in a different neighborhood? What if you lived somewhere else?

I took the top five locations from Eun Joo’s Foursquare checkins in the East Village and calculated their geographic relationship to each other. Then, I took these paths into Google Earth and did flythroughs around various famous locations in the city. I recorded the places where each of the new points fell and wrote a short blurb about the type of person that might visit those locations. It’s interesting to see how a “personality” emerges from the visits, even though they’re almost entirely arbitrary.

Although I kind of like the idea that any one of these stories could be the truth (and in all the possibilities of spacetime they in fact exist in some way), I separated out the real data at the suggestion of the class.

Due for Week 6

• Reading: “Particle animation and rendering using data parallel computation”, Karl Sims (available via NYU network/proxy)
• Reading: “Particle Systems, a Technique for Modeling a Class of Fuzzy Objects”, Reeves (available via NYU network/proxy)
• Reading: How my Dog learned Polymorphism
• Other suggested reading: Particle Systems (Siggraph), Particle System API, by David K. McAllister, Particle Systems by Allen Martin, Physically Based Modeling, Particle System Dynamics by Andrew Witkin
  Flight404′s Particle example: http://www.flight404.com/blog/?p=113
• Assignment: Develop a proposal and a prototype for a “midterm” project. The scope of the project can be quite large (trial idea for a final, for example), however, you will not be expected to implement the entire project. For the proposal, include a description, relevant links, and a quick Processing sketch of the first step towards the idea. Link your proposal from the wiki. Next week, we will look at a selection of proposals and then on Mar 2/3, the results for all midterm projects will be presented.

I presented these images as a stand in for the crash visualization I’m working on using Robert Carlsen’s data. The original proposal is in this post. Most of the time so far has been spent munging the data into container that I can wrap a front end on. Longitude and latitude have been converted into meters and then into pixels. Several times I’ve had to go back and adjust the coefficients to find some semblance of accuracy. I’ve also smoothed the lat/long measurements across the acceleration measurements to remain at 10 Hertz.

The base path is augmented by cylindrical spaces around each measurement that represent the estimated accuracy of the iPhone’s sensors. In addition, each point in time is tagged with acceleration vectors in three dimensions, the meters per second, and heading. I will not end up using all of this data, but I have realized that there is likely a distinction to be made between the local data (acceleration), global data (location and heading), and calculated data (speed). The visualization is not only an exercise in trying to visually compress the data available, but also make comparisons between the utility of the three types of data in generalized situations. Ultimately, it fulfills my desire to begin mapping complex data in three-space by sculpting the data around events.

This learning exercise keeps extending itself with new prospects. For example, Patrick Hebron mentioned that I might be interested in quad trees to do some of the searching, and I have gone on to discover the magic of kd-trees. I also would like to implement the VertexBufferObjects Rune showed me to speed up some of the movements. Finally, I’ve begun to think about a way to use springs to connect the coincidental points that belong to more than one cylindrical space, but I’m not sure how it fits in the conceptual model of points moving in spacetime. I think I like it though. Anyway, I plan to rewrite the raw data object to move things into 3D vectors in the setup and calculate the voxel distances on each update rather than through a system wide search each frame.

Base path

Path with accuracy sketching

Closeup of accuracy measures

Raw acceleration vectors

Smooth acceleration vectors

Path of acceleration vectors corrected for latitude/longitude mapping to meters

Space filled with potential agents

Due for Week 7

• Assignment: Complete midterm project documentation, post link to wiki and be prepared to present next week.