RWEL Homework – another approach – playing around with email and ngrams

I sometimes have crazy email interactions with my mother, and I decided to take her emails and write some ngrams with them – and see how they fare

When I was just starting out I figured out how to pull in the body text from emails from specific people (in the process I accidentally deleted 12,000 unread messages (it took about 20 minutes) but… I realize my gmail box needs some sorting

import sys
import imaplib, re

conn = imaplib.IMAP4_SSL("imap.gmail.com", 993)
conn.login('gabriella.levine', 'qwewqwe1!')
output = conn.select("INBOX")
print "new messages", output
unread=conn.search(None, 'UnSeen')
print unread
unreadCount = re.search("UNSEEN (\d+)", conn.status("INBOX", "(UNSEEN)")[1][0]).group(1)
print "unread", unreadCount


def get_emails_from_ruth():

ruth = conn.search(None, 'FROM', '"Ruth"')
print "ruth", ruth
#have to write a functioin that outputs ruth then run it in a nother function below ruthmsg
#output ruth

ruthmsg = conn.fetch('149', '(BODY.PEEK[TEXT])')
print ruthmsg
#msgdata = conn.fetch(num, "(BODY.PEEK[HEADER.FIELDS (FROM TO CC DATE SUBJECT MESSAGE-ID)] UID)")

This gave me each number of her emails in a list, as well as how many I had from her, as well as a large mass of text from one of her emails. Then I wrote it better so I got all of her emails, as a text, and only the body text of her email (so I split the text on “7BIT\r\n\r\nhi” and “\r\n\r\n–Boundary” which surrounded the body text of every email I extracted and printed out.

I wasn’t able to get all of her emails but here is some of what I output:

thought you might like to you can breathe a good time together: i thought you can babble on and on like the one who i\’ve been back in a good time didn\’t pass. we\’re together. what a good time together: in our lives in a good time didn\’t pass. have so much fun when we\’re together. we feel great depths for us.\r\n it\’s wonderful for us.\r\n these are SOOOOO different in our lives in touch with her last night. it\’s like my dinner with her last night. schook – the man she lives now can breathe a good time together: i knew all the man she married, and uncles and brother and have such a different world realistically and aunts and emotionally and intellectually, but we can babble on and i thought you when you when we\’re together. mom
hi gabby,, we feel great depths for each other and brother and aunts and have such a good time together: we have so much fun when we\’re together. i thought you might like the one who i\’ve been back in a relief afterwards, hopefully. it\’s like the same about me. noone in tons of relief afterwards, hopefully. it\’s like my dinner with her parents and have you can breathe a sigh of course, there\’s history: mom

So I have to clean it up a lot, and take out the line break symbols. But It is something I will work with. My first step is to get every email in the list (I was getting an error when I was trying to pass in all the emails from her, as opposed to a few at a time)

RWEL HW- rideshare data (also messing around with sound)

playing around with python’s ability to write .wav file – I’d like to redo my midterm in python, writing two different sin waves, as well as reading through a text’s binary data and having a tone play on 1 and nothing on 0… here’s the start

import sys
import math
import wave
import struct

freq = 440.0
data_size = 40000
fname = "WaveTest.wav"
frate = 11025.0  # framerate as a float
amp = 64000.0     # multiplier for amplitude

sine_list_x = []
for x in range(data_size):
    sine_list_x.append(math.sin(2*math.pi*freq*(x/frate)))

wav_file = wave.open(fname, "w")

nchannels = 1
sampwidth = 2
framerate = int(frate)
nframes = data_size
comptype = "NONE"
compname = "not compressed"

wav_file.setparams((nchannels, sampwidth, framerate, nframes,
    comptype, compname))

for s in sine_list_x:
    # write the audio frames to file
    wav_file.writeframes(struct.pack('h', int(s*amp/2)))

wav_file.close()

Additionally, I’m doing more with the code to extract zip code data: I had not been printing the city and state in my processing sketch earlier, but now I am-
[‘43068’, ‘43215’] (zip codes, origin and destination of rideshare requests)
82.803454 39.955145 83.004383 39.967106 (data of gps lats / longs from the zip codes – I had been using after cleaning it with python)
80.281567 40.604424 MACARTHUR PA 80.073393 40.438045 CRAFTON PA (data I will now use, using the new code – see below)

where the first zip code is the origin of a rideshare request, and the latter is a destination point: gps lat., gps long., city name, state name, from a zip code

Now I just need to figure out how to do this through processing (in other words, I don’t want to have to run this code to pull the data from online in order to generate the text file that processing can use)

import sys
import urllib

#define a function
def len(array):
	i=0
	for entry in array:
		i=i+1
	return i
	
def clean(line):
		line = line.replace("\'", "")
		line = line.replace(" ", "")
		line = line.replace(",", ".")
		line = line.replace("(", "")
		line = line.replace(")", "")
		line = line.replace("...", ",")
		line = line.strip("(),'\"")
		return line
		
def zipToLat(origin, destination, zipCodeMap):
	#for each origin and destination zipcode that exists in the zipCodeMap, print it out in the format : (latOrigin, lonOrigin; latDestination, lonDestination)
	

	if zipCodeMap.has_key(origin) and zipCodeMap.has_key(destination):
		#print zipCodeMap[origin][0], zipCodeMap[origin][1], zipCodeMap[destination][0], zipCodeMap[destination][1]
		print zipCodeMap[origin][0],",", zipCodeMap[origin][1],",", zipCodeMap[origin][2], zipCodeMap[origin][3],",", zipCodeMap[destination][0],",", zipCodeMap[destination][1],",",zipCodeMap[destination][2], zipCodeMap[destination][3]
		
		#print "yes"
		#print zipCodeMap[origin]+zipCodeMap[destination]
		#print origin
			
		
		
def maxloc(locationsDict,zipmap):
#return string of both values - location and how many times it appears
	tmp_max_count = -1
	tmp_max_value = ''
	zipMapRF={}
	for entry in locationsDict:
		if len(entry)>1:
			zipcode = entry[0]
			count = entry[1]
			if zipmap.has_key(zipcode):
				#print zipmap[zipcode] + " , " , count
				#print "%s %s" % (zipmap[entry[0]],entry[1])
				
				zipMapRF[entry[0]] = entry[1]
	
				if tmp_max_count <  count:
					tmp_max_count = count
					tmp_max_value  = zipmap[zipcode]
	
			#entry 1 num times apprs, 0 is name of location
			#chek beforehand - does zipmap have this key..

#	print locationsDict
	print "popular location is " + tmp_max_value + " which was visited " , tmp_max_count , " times."
	return tmp_max_value
	#now get value and the map
	


zipNames = {}
zipCoordinates = {}
starts = {}
destinations= {}
locations={}



for line in open('zips.txt'):#can i just load it from http://www.census.gov/tiger/tms/gazetteer/zips.txt
		tup = line.split(",")
		#print tup              
		if len(tup)>=4:
			zip=tup[1][1:-1]
			state=tup[2][1:-1]
			town=tup[3][1:-1]
			lon=tup[4]
			lat=tup[5]
			point = lon,lat
			point = lon,lat, town, state
			
			#print point
			#zipmap[zip]=pair
			#print zip + " (" + pair[0] + "," + pair[1] + ")" 
			zipNames[zip] = town + ", " + state		
			
			zipCoordinates[zip] = point
#print zipCoordinates #this Now stores most of what I need		


#print zipmap
#print zipNames
#print zipCoordinates
#f = urllib.urlopen('http://www.erideshare.com/!!.php')
#s = f.read()
#f.close()

f = urllib.urlopen('http://www.erideshare.com/!!.php')
lines = f.readlines()
f.close()

for line in lines:#open('phpErid.txt'):#figure this out
	for pointPair in line.split("|"):
		#build concordence
		pointPair = pointPair.strip()
		points = pointPair.split(",")
		#print  points
		#print "points!"
		#print points
		if len(points)>=2:
			
			dest = points[1]#the second zip code is the dest
			loc = points[0]#first zip code of the pair
			
			destinations[dest]=destinations.get(dest,0) +1
			starts[loc] = starts.get(loc,0) +1
			
			
			locations[dest] = locations.get(dest,0) +1
			locations[loc] = locations.get(loc,0) +1
			
			

			zipToLatValue = zipToLat(loc,dest, zipCoordinates)
	
		
l=len(locations)
topZipCode = maxloc(locations,zipNames)
locations = locations.items()
maxloc(locations,zipNames)
locations.sort
#print locations
if len(locations)>1:
	locations.sort()
#	print locations[0]
#	print locations[l-1]
maxloc(locations,zipNames)
#{'M': 1, 's': 4, 'p': 2, 'i': 4}

#print zipNames
#print zipmap



#60647,60120| 60647,60120| L1X2W8,M5S1N5| L1X2W8,M5S1N5| 22601,20041| 22601,20041| 19446,19477| 19446,19477| 19446,19477| 19446,19477| 21224,20019| 90840,92660| 02114,02132| 17047,17128| 94612,94903| 94612,94903| 94612,94903| 62034,62035| 62034,62035| 94110,94403| 94110,94403| 07306,07059|

So here is an image of the processing sketch I have when I write this text file inside of the processing folder
(and here is the sketch in early phases, running sort of slowly)