Archive for April, 2011
Surface Automotives
eRideShare.com
eRideShare.com is a website where people can easily connect and share rides. It is widely used throughout the world. Each day, there are about 10,000 ride share requests just in the United States. I’m using data from both short commuter ride requests as well as longer travel ride requests. This sketch is a representation of the rides occurring each day, with the temporal resolution adjusted so that each car particle leaves its home and travels to its destination when clicked on, or at random intervals.
I am pulling the ride requests each day – after visualizing the cars laid out geographically, I noticed a lot of activity in cities, especially in the coastal regions. There are many more short term ride requests, and many overlapping rides. I wanted initially to create an interactive map that the user could zoom into easily.
I decided to decontextualize the rides from their geographical setting in order to perhaps derive some new insight. So I assigned each car a city, and can arrange the cities in a line, in a circle, in order of population…
Technically:
I plotted it on google maps, but wanted to make the car particles “alive” and mobile.
So without an API (google made it difficult for me to get the gps coordinates as strings) I’m using Python and a data set from census.gov of zipcodes converted to GPS lats and longitudes, to parse
“98337,98409| 33063,33442| 95602,95821| 95123,94043| 90815,92618| 90815,92618| 90815,92618| 07060,07054| 92603,90731| P0E1G0,L4N2M3| 94117,94941| 01778,01748| 90813,92626| 20720,20004| 08724,07016| 18360,11050| ”
into:
122.071562 47.895381 SNOHOMISH WA 122.200571 47.988431 EVERETT WA
78.390533 40.50524 ALTOONA PA 78.586068 40.460779 CRESSON PA
78.091224 38.233937 MONTFORD VA 77.093197 38.887103 ARLINGTON VA
71.35741 42.287476 NATICK MA 71.125611 42.377045 CAMBRIDGE MA
83.279236 42.643856 PONTIAC MI 83.282892 42.822272 OXFORD MI
Further:
-Adjust the speed and the graphics (cars, houses, garages)
-web app, temporal accuracy
-Take into account the size of the cities
-Show route information better
-Heat map? other effective visualization tools?
-Interesting information about carpooling and sharing (individual cities, 
per capita, cartogram? different modes of city arrangement)
-design?
–>Biggest thing I got out of it:
What can I learn from data? How can I represent it differently to extract different patterns?
Some iterations I went through:
Ferriferous progress
I made a prototype, based on MakerBot’s Unicorn PenPlotter, of a servo mechanism that will move vertically in linear motion:
Then we modified it to look simpler :
The circuit:
At first we thought we could use the solar kit to charge a 12V solenoid, which would, through arduino, read our data of solar light levels over the past month, and pulse an electromagnetic field through ferrofluid, thereby mapping our data at a condensed temporal resolution. However, although the solar kit has a 12 V battery, we were not getting nearly enough response from the solenoid because, perhaps, the charge controller limits the current? (I need to think through this more)
We also tried using different beam circuits to store voltage in capacitors then to discharge, pulsing a solenoid. Similarly, we could not quite get enough current.
We had a bit of trouble reading through the data from our micro SD card – We switched to the adafruit sd Card shield, and I think we might have been choosing the wrong chip Select pin? we couldn’t figure out why the file DATA.TXT was not opening correctly. After a while, we switched back to the sparkfun shield or the card reader, and it worked fine.
This is the code we practiced with, but later we got the data on the SD card to read directly onto the arduino (see below)
#include <Servo.h>
int changer = 1;
int incomingData;//variable to read incoming data
Servo myservo; // create servo object to control a servo
// a maximum of eight servo objects can be created
int servoState;
int prevState;
const int transistorPin = 9; // connected to the base of the transistor
int pos = 0; // variable to store the servo position
int myData[] = {1,2,3,4,5,6,7,8,9,10,11,12,13, 13,13,13,13,13,13,13,13,13,13,12,11,10,9,8,7,6,5,4,3,2,1};// = new Array[12];//1,myData2;
void setup()
{
Serial.begin(9600);
pinMode(transistorPin, OUTPUT);
myservo.attach(9); // attaches the servo on pin 9 to the servo object
Serial.print("Servo Data Test!!");
}
void loop()
{
for(int i=0;i<29;i=i++){
if(i>12 || i<=1){changer*=-1;}
int intt=map((myData[i]*20),0,20*13, 180,70);
int solenoid = map((myData[i]*20),0,20*13, 700,1023);
analogWrite(5, solenoid);
myservo.write(intt);
delay(50);
}
}
We are using the Sparkfun data logging shield
//CS pin : 8; VCC/GRNd; DO pin 12; DI pin 11
//using the sparkfun microsd data logger, (or the shield); writes millis and time in secs
#include <SdFat.h>
#include <SdFatUtil.h>
#include <ctype.h>
//Create the variables to be used by SdFat Library
Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;
const int transistorPin = 5; // connected to the base of the transistor
int potValue1 = 0; // value returned from the potentiometer
#include <Servo.h>
char name[] = "DATA.TXT"; //Create an array that contains the name of our file.
char contents[256]; //This will be a data buffer for writing contents to the file.
char in_char=0;
String line;
int index=0; //Index will keep track of our position within the contents buffer.
Servo myservo; // create servo object to control a servo
void setup(void)
{
Serial.begin(9600); //Start a serial connection.
pinMode(transistorPin, OUTPUT);
pinMode(8, OUTPUT); //Pin 10 must be set as an output for the SD communication to work.
card.init(); //Initialize the SD card and configure the I/O pins.
myservo.attach(9); // attaches the servo on pin 9 to the servo object
volume.init(card); //Initialize a volume on the SD card.
root.openRoot(volume); //Open the root directory in the volume.
}
void loop(void){
// file.open(root, name, O_CREAT | O_APPEND | O_WRITE); //Open or create the file 'name' in 'root' for writing to the end of the file.
//sprintf(contents, "Millis: %d ", millis()); //Copy the letters 'Millis: ' followed by the integer value of the millis() function into the 'contents' array.
//file.print(contents); //Write the 'contents' array to the end of the file.
// file.close(); //Close the file.
file.open(root, name, O_READ); //Open the file in read mode.
in_char=file.read(); //Get the first byte in the file.
//Keep reading characters from the file until we get an error or reach the end of the file. (This will output the entire contents of the file).
while(in_char >=0){ //If the value of the character is less than 0 we've reached the end of the file.
Serial.print(in_char); //Print the current character
//line = file.read();//read each line?
// Serial.println(line);
in_char=file.read(); //Get the next character
float newFloat = map(in_char, 0,9,0,180);
myservo.write(in_char);
float solin_char= map(in_char, 0,9,500,1023);
analogWrite(5, solin_char);
delay(300);
}
file.close(); //Close the file
//delay(1000); //Wait 1 second before repeating the process.
}
Some more videos of progres and the mechanism:
more with Ferrofluid and Solar Energy
Using the MakerBot Unicorn PenPlotter as a guideline for a mechanism that uses a servo to laterally back and forth along a single axis, I made a mechanism that oscillates the proximity of a rare earth magnet back and forth in the proximity of ferrofluid. We are iterating through our collected data, to visualize and graph energy and weather data we are collecting over the month, in the planar layer of ferrofluid.
We hollowed out lightbulbs in order to cheaply make small terrariums for units that contain ferrofluid and a couple magnets, that will be perched atop solenoids and magnets.
I used a linear slide from a chain lock to keep the arm holding the magnet in place. The pieces are drying over night
The data logging is going well – although every now and then when I check the card, it didn’t read the data (I think I have to unplug the arduino each time. Oh, and in case I forget, it took me two days to realize that with the data logger and RTC to work, I need at least 8V from a wall wort (from the 5V one we were getting 0/0/0 for the date / time. (Also, I switched the code so we it is always writing to data.txt as opposed to a new file each time, so it will be much easier to graph (instead of copying and pasting from different text files, we can just use data.txt in processing).
We also thought that because the solenoid was working with 12V from the bench power supply, we would be able to use the 12V battery powered off the solar kit / charge controller at school – However, when we plugged in the solenoid to the 12 V battery…. turned out, it hardly worked. And the voltage reading was about 0.7… RIGHT….
NOC
(Before I figure out speed I have to reduce the data set)
Ideas? follow a 3d path / steering method; text clarity; binning; experiment with L systems or recursion as I scale, and from day to day? sound?
NOC with names, updating once per day
I would like to : get the cars to follow a 3d path, think more about how they will interact, fix the letters, possibly integrate a better map system to make it more informative?
ferrofluidic motor syringe
continuing with the ferrofluid idea:
so my servos are strong enough to push syringes with liquid:
I got inspired by a radial engine:
and was going to start off by building a piston:
But I have a bunch of servos, so I decided to model some sort of an improvised two-cylinder piston that takes in fluid and spits it back out, that is modelled after MakerBot’s Unicorn Pen plotter – An elbow pushes back and forth laterally
I think that I am going to do some 3d moving model using ferrofluid, perhaps using the data that Genevieve and I are collecting about solar / temperature data
So, using the makerBot wiki, I got a pattern to cut the pieces for the mechanism: I cut it in both masonite and clear plexi (I assume the plexiglass is stronger but I preferred the transparent aesthetic, so I’ll try it!)….
now, to get some nuts and bolts…
logging the data
14.38, 2.52
with the solar panels in, we’re 17, 3
click to see more info
-
- April 10
-
- April 13
logging solar data
Genevieve and I now decided to log data from the sun over the course of the month – most likely, voltage, and light level. We will use both a photocell and two large solar panels (during the day one gave about 20 V). (And of course, we will power the arduino sustainably!)
We are using a RTC (real time clock from adafruit), a micro sd card data shield, and the following code (derived from Tom Igoe’s github)
Here is the circuit:
With a 9 V battery plugged in for a few minutes, this is the result (reading voltage once per minute):
Ferrofluid, and more…
Genevieve and thought about building solar powered ferrofluidic sculpture. So we started to play around with ferrofluid – and made a mess. It is cool stuff and can make some beautiful patterns and behavior.
At first we talked about creating a sound with a photocell , using a “light to sound” circuit – The speaker, powered by a solar panel, would create an electromagnetic field, driving cool patterns in ferrofluid….
well, electromagnets aren’t THAT easy…
a light to sound circuit:
So we have since modified our plan, after many iterations… I think we will log voltage and light input over the course of the month, and do something cool with that data (a ferrofluid-type graph?)
We are still hooked on the electromagnet idea and have played around with some (a nail with a coil, a few solenoids) to see what type of voltage we need to input in order to get some interaction with the ferrofluid
MAGNETIC EXPLOSION:
after the mess:
RWEL Homework – another approach – playing around with email and ngrams
I sometimes have crazy email interactions with my mother, and I decided to take her emails and write some ngrams with them – and see how they fare
When I was just starting out I figured out how to pull in the body text from emails from specific people (in the process I accidentally deleted 12,000 unread messages (it took about 20 minutes) but… I realize my gmail box needs some sorting
import sys
import imaplib, re
conn = imaplib.IMAP4_SSL(&amp;quot;imap.gmail.com&amp;quot;, 993)
conn.login('gabriella.levine', 'qwewqwe1!')
output = conn.select(&amp;quot;INBOX&amp;quot;)
print &amp;quot;new messages&amp;quot;, output
unread=conn.search(None, 'UnSeen')
print unread
unreadCount = re.search(&amp;quot;UNSEEN (\d+)&amp;quot;, conn.status(&amp;quot;INBOX&amp;quot;, &amp;quot;(UNSEEN)&amp;quot;)[1][0]).group(1)
print &amp;quot;unread&amp;quot;, unreadCount
def get_emails_from_ruth():
ruth = conn.search(None, 'FROM', '&amp;quot;Ruth&amp;quot;')
print &amp;quot;ruth&amp;quot;, ruth
#have to write a functioin that outputs ruth then run it in a nother function below ruthmsg
#output ruth
ruthmsg = conn.fetch('149', '(BODY.PEEK[TEXT])')
print ruthmsg
#msgdata = conn.fetch(num, &amp;quot;(BODY.PEEK[HEADER.FIELDS (FROM TO CC DATE SUBJECT MESSAGE-ID)] UID)&amp;quot;)
This gave me each number of her emails in a list, as well as how many I had from her, as well as a large mass of text from one of her emails. Then I wrote it better so I got all of her emails, as a text, and only the body text of her email (so I split the text on “7BIT\r\n\r\nhi” and “\r\n\r\n–Boundary” which surrounded the body text of every email I extracted and printed out.
I wasn’t able to get all of her emails but here is some of what I output:
thought you might like to you can breathe a good time together: i thought you can babble on and on like the one who i\’ve been back in a good time didn\’t pass. we\’re together. what a good time together: in our lives in a good time didn\’t pass. have so much fun when we\’re together. we feel great depths for us.\r\n it\’s wonderful for us.\r\n these are SOOOOO different in our lives in touch with her last night. it\’s like my dinner with her last night. schook – the man she lives now can breathe a good time together: i knew all the man she married, and uncles and brother and have such a different world realistically and aunts and emotionally and intellectually, but we can babble on and i thought you when you when we\’re together. mom
hi gabby,, we feel great depths for each other and brother and aunts and have such a good time together: we have so much fun when we\’re together. i thought you might like the one who i\’ve been back in a relief afterwards, hopefully. it\’s like the same about me. noone in tons of relief afterwards, hopefully. it\’s like my dinner with her parents and have you can breathe a sigh of course, there\’s history: mom
So I have to clean it up a lot, and take out the line break symbols. But It is something I will work with. My first step is to get every email in the list (I was getting an error when I was trying to pass in all the emails from her, as opposed to a few at a time)
RWEL – smtp and IMAP libraries
I used this code to send an email:
import smtplib
fromaddr = 'gabriella.levine@gmail.com'
toaddrs = 'gabriella.levine@gmail.com'
msg = 'There was a terrible error that occured and I wanted you to know!'
# Credentials (if needed)
username = 'gabriella.levine@gmail.com'
password = 'qwewqwe1!'
# The actual mail send
server = smtplib.SMTP('smtp.gmail.com:587')
server.starttls()
server.login(username,password)
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
I would like to be able to monitor a project I’m doing for Sustainable energy (where I will be monitoring the voltage input from a solar panel), and send myself an email if any of the components (the Arduino, the Xbee, the Real time clock, the sd card / shield) stops functioning properly
I thought using this code and the smtp library would send an email but I KEEP getting an error(see below), and I’m not quite sure why yet:
import smtplib
from email.mime.text import MIMEText
sender = 'gabriella.levine@gmail.com'
recipients = 'gl715@nyu.edu'
msg = MIMEText('hi my name is gabby')
msg['Subject'] = 'test'
msg['gabriella levine'] = sender
msg['you'] = recipients
smtpserver = 'smtp.gmail.com'
smtpuser = 'gabriella.levine' # set SMTP username here
smtppass = 'qwewqwe1!' # set SMTP password here
session = smtplib.SMTP(&amp;amp;amp;quot;smtp.gmail.com&amp;amp;amp;quot;, 587)
session.ehlo()
session.starttls()
session.ehlo()
session.login(smtpuser, smtppass)
smtpresult = session.sendmail(sender, [recipients], msg.as_string())
if smtpresult:
errstr = &amp;amp;amp;quot;&amp;amp;amp;quot;
for recip in smtpresult.keys():
errstr = &amp;amp;amp;quot;&amp;amp;amp;quot;&amp;amp;amp;quot;Could not delivery mail to: %s
Server said: %s
%s
%s&amp;amp;amp;quot;&amp;amp;amp;quot;&amp;amp;amp;quot; % (recip, smtpresult[recip][0], smtpresult[recip][1], errstr)
raise smtplib.SMTPException, errstr
session.close()
ERROR:
Traceback (most recent call last):
File “1151.py”, line 1, in
import smtplib
File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/smtplib.py”, line 46, in
import email.utils
File “/Users/administrator/Desktop/pyEmail/email.py”, line 25, in
conn.login( “putemailhere”, “putpasswordhere” )
File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/imaplib.py”, line 501, in login
raise self.error(dat[-1])
imaplib.error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)
I also started trying to sort through email using the IMAP library – I haven’t gotten very far, and have to look at the documentation some more, but this is what I have:
import sys
import imaplib
conn = imaplib.IMAP4_SSL("imap.gmail.com", 993)
conn.login('gabriella.levine', 'qwewqwe1!')
output = conn.select("INBOX")
print "new messages", output
#toaddress = msg['to']
#output1 = conn.select("SENT MAIL")
#print output1
output1=conn.select('Sent')
print output1
almost finished photopopper
See him moving towards light:
slightly blurry, but you get the idea, maybe
and here again (he falls off the computer
RWEL HW- rideshare data (also messing around with sound)
playing around with python’s ability to write .wav file – I’d like to redo my midterm in python, writing two different sin waves, as well as reading through a text’s binary data and having a tone play on 1 and nothing on 0… here’s the start
import sys
import math
import wave
import struct
freq = 440.0
data_size = 40000
fname = &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;WaveTest.wav&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;
frate = 11025.0 # framerate as a float
amp = 64000.0 # multiplier for amplitude
sine_list_x = []
for x in range(data_size):
sine_list_x.append(math.sin(2*math.pi*freq*(x/frate)))
wav_file = wave.open(fname, &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;w&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;)
nchannels = 1
sampwidth = 2
framerate = int(frate)
nframes = data_size
comptype = &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;NONE&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;
compname = &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;not compressed&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;quot;
wav_file.setparams((nchannels, sampwidth, framerate, nframes,
comptype, compname))
for s in sine_list_x:
# write the audio frames to file
wav_file.writeframes(struct.pack('h', int(s*amp/2)))
wav_file.close()
Additionally, I’m doing more with the code to extract zip code data: I had not been printing the city and state in my processing sketch earlier, but now I am-
[‘43068’, ‘43215’] (zip codes, origin and destination of rideshare requests)
82.803454 39.955145 83.004383 39.967106 (data of gps lats / longs from the zip codes – I had been using after cleaning it with python)
80.281567 40.604424 MACARTHUR PA 80.073393 40.438045 CRAFTON PA (data I will now use, using the new code – see below)
where the first zip code is the origin of a rideshare request, and the latter is a destination point: gps lat., gps long., city name, state name, from a zip code
Now I just need to figure out how to do this through processing (in other words, I don’t want to have to run this code to pull the data from online in order to generate the text file that processing can use)
import sys
import urllib
#define a function
def len(array):
i=0
for entry in array:
i=i+1
return i
def clean(line):
line = line.replace("\'", "")
line = line.replace(" ", "")
line = line.replace(",", ".")
line = line.replace("(", "")
line = line.replace(")", "")
line = line.replace("...", ",")
line = line.strip("(),'\"")
return line
def zipToLat(origin, destination, zipCodeMap):
#for each origin and destination zipcode that exists in the zipCodeMap, print it out in the format : (latOrigin, lonOrigin; latDestination, lonDestination)
if zipCodeMap.has_key(origin) and zipCodeMap.has_key(destination):
#print zipCodeMap[origin][0], zipCodeMap[origin][1], zipCodeMap[destination][0], zipCodeMap[destination][1]
print zipCodeMap[origin][0],",", zipCodeMap[origin][1],",", zipCodeMap[origin][2], zipCodeMap[origin][3],",", zipCodeMap[destination][0],",", zipCodeMap[destination][1],",",zipCodeMap[destination][2], zipCodeMap[destination][3]
#print "yes"
#print zipCodeMap[origin]+zipCodeMap[destination]
#print origin
def maxloc(locationsDict,zipmap):
#return string of both values - location and how many times it appears
tmp_max_count = -1
tmp_max_value = ''
zipMapRF={}
for entry in locationsDict:
if len(entry)>1:
zipcode = entry[0]
count = entry[1]
if zipmap.has_key(zipcode):
#print zipmap[zipcode] + " , " , count
#print "%s %s" % (zipmap[entry[0]],entry[1])
zipMapRF[entry[0]] = entry[1]
if tmp_max_count < count:
tmp_max_count = count
tmp_max_value = zipmap[zipcode]
#entry 1 num times apprs, 0 is name of location
#chek beforehand - does zipmap have this key..
# print locationsDict
print "popular location is " + tmp_max_value + " which was visited " , tmp_max_count , " times."
return tmp_max_value
#now get value and the map
zipNames = {}
zipCoordinates = {}
starts = {}
destinations= {}
locations={}
for line in open('zips.txt'):#can i just load it from http://www.census.gov/tiger/tms/gazetteer/zips.txt
tup = line.split(",")
#print tup
if len(tup)>=4:
zip=tup[1][1:-1]
state=tup[2][1:-1]
town=tup[3][1:-1]
lon=tup[4]
lat=tup[5]
point = lon,lat
point = lon,lat, town, state
#print point
#zipmap[zip]=pair
#print zip + " (" + pair[0] + "," + pair[1] + ")"
zipNames[zip] = town + ", " + state
zipCoordinates[zip] = point
#print zipCoordinates #this Now stores most of what I need
#print zipmap
#print zipNames
#print zipCoordinates
#f = urllib.urlopen('http://www.erideshare.com/!!.php')
#s = f.read()
#f.close()
f = urllib.urlopen('http://www.erideshare.com/!!.php')
lines = f.readlines()
f.close()
for line in lines:#open('phpErid.txt'):#figure this out
for pointPair in line.split("|"):
#build concordence
pointPair = pointPair.strip()
points = pointPair.split(",")
#print points
#print "points!"
#print points
if len(points)>=2:
dest = points[1]#the second zip code is the dest
loc = points[0]#first zip code of the pair
destinations[dest]=destinations.get(dest,0) +1
starts[loc] = starts.get(loc,0) +1
locations[dest] = locations.get(dest,0) +1
locations[loc] = locations.get(loc,0) +1
zipToLatValue = zipToLat(loc,dest, zipCoordinates)
l=len(locations)
topZipCode = maxloc(locations,zipNames)
locations = locations.items()
maxloc(locations,zipNames)
locations.sort
#print locations
if len(locations)>1:
locations.sort()
# print locations[0]
# print locations[l-1]
maxloc(locations,zipNames)
#{'M': 1, 's': 4, 'p': 2, 'i': 4}
#print zipNames
#print zipmap
#60647,60120| 60647,60120| L1X2W8,M5S1N5| L1X2W8,M5S1N5| 22601,20041| 22601,20041| 19446,19477| 19446,19477| 19446,19477| 19446,19477| 21224,20019| 90840,92660| 02114,02132| 17047,17128| 94612,94903| 94612,94903| 94612,94903| 62034,62035| 62034,62035| 94110,94403| 94110,94403| 07306,07059|
So here is an image of the processing sketch I have when I write this text file inside of the processing folder
(and here is the sketch in early phases, running sort of slowly)
