# HG changeset patch # User DaveM # Date 1526223685 -3600 # Node ID 52ea10196da3471d845e259fc19e87b8494df03e # Parent 5825520de1433b4ef59125425d4ebd24e999e85c implementing vertex pull, and testing diff -r 5825520de143 -r 52ea10196da3 .hgignore --- a/.hgignore Mon Apr 30 18:13:40 2018 +0100 +++ b/.hgignore Sun May 13 16:01:25 2018 +0100 @@ -16,7 +16,7 @@ fullResult.csv scores.csv issues.csv - +_data/* # C extensions *.so diff -r 5825520de143 -r 52ea10196da3 V5/runme.py --- a/V5/runme.py Mon Apr 30 18:13:40 2018 +0100 +++ b/V5/runme.py Sun May 13 16:01:25 2018 +0100 @@ -4,7 +4,7 @@ import dParse as dp import synastry as syn # import requests -# import time +import localRequests as lr import csv import pdb import os @@ -12,24 +12,6 @@ import sys from bs4 import BeautifulSoup -# def parsePage(resp): -# gotLocation = 0 -# horiscope = syn.planetPositions() -# soup = BeautifulSoup(resp.content, 'lxml') -# tcCell = soup.find_all('div', attrs={'class':'right-sedy-banner-svetlejsi'}) -# for cell in tcCell: -# if "Planets in partner's house" in cell.get_text(): -# gotLocation = 1 -# divList = cell.find_all('div') -# for i in range(len(divList)): -# planetName = divList[i].getText().lower().strip().replace(':','').split(' ')[0] -# if planetName in syn.planetPositions.planetNames: -# if gotLocation and not '/' in planetName: -# horiscope.planets[planetName].setHouse(divList[i+2].getText(),divList[i+4].getText()) -# else: -# horiscope.planets[planetName].setLocation(divList[i+2].getText(),divList[i+4].getText()) -# return horiscope - def makePeople(filename): stream = csv.DictReader(open(filename,'rb')) dictList = [] @@ -72,6 +54,15 @@ else: csv_file.write(str(person.id)+','+str(person.issue)+'\n') +def outputDescriptions(filename,people): + with open(filename, "wb") as csv_file: + csv_file.write('Person ID, Description \n') + for person in people: + if person.issue is None: + csv_file.write(str(person.id)+','+str(person.description)+'\n') + else: + csv_file.write(str(person.id)+','+str(person.issue)+'\n') + def outputIssues(filename,people): with open(filename, "wb") as csv_file: csv_file.write('Person ID, Issue \n') @@ -107,8 +98,12 @@ print 'Posting Request for person '+ person.id person.makePayload() rawData[person.id] = person.requestURL() + # print 'parsing person '+ person.id - person.horiscope = person.parsePage() + if person.resp is None: + print 'SKIPPING person '+ person.id + ' error with unknown error' + else: + person.horiscope = person.parsePage() # person.horiscope.printPositions() if saveFile is not None: # savePick('raw_'+saveFile,rawData) @@ -133,10 +128,12 @@ savePick(saveFile,people) def _main(): - # os.remove('outData.pick') - # os.remove('fullResults.pick') - # os.remove('outData_collect.pick') - # os.remove('raw_outData_collect.pick') + resetAll = False + if resetAll: + for item in os.listdir('.'): + if item.endswith(".pick"): + os.remove(item) + os.remove('outData_collect.pick') pickFile = 'outData.pick' if not os.path.exists(pickFile): people = makePeople('individuals.csv') @@ -159,7 +156,7 @@ print person.id person.score = None if person.issue is None: - person.score = comp.calcCompatibility(person.horiscope) + person.score,person.description = comp.describeCompatibility(person.horiscope) if person.score is None: person.issue = 'None Planet Locations' # pdb.set_trace() @@ -171,6 +168,7 @@ people = loadPick('fullResults.pick') outputPeople('fullResult.csv',people) outputScores('scores.csv',people) + outputDescriptions('description.csv',people) outputIssues('issues.csv',people) if __name__ == "__main__": diff -r 5825520de143 -r 52ea10196da3 V5/synastry.py --- a/V5/synastry.py Mon Apr 30 18:13:40 2018 +0100 +++ b/V5/synastry.py Sun May 13 16:01:25 2018 +0100 @@ -8,6 +8,7 @@ from bs4 import BeautifulSoup import pdb +DEFAULT_SOURCE = 'local' class compatibility: def __init__(self): @@ -64,21 +65,41 @@ def calcCompatibility(self, horiscope): score = 0 for rule in sorted(self.rules): - if 'vertex' in rule.planet: - # print 'ERROR - catch Vertex issue' - pass - # elif 'southnode' in rule.planet: - # pdb.set_trace() - # # print 'ERROR - catch South Node issue' - # pass - elif rule.planet[1] == 'house' and rule.planet[0] == 'asc': - # print 'ERROR - catch Ascendant House issue' + if rule.planet[1] == 'house' and rule.planet[0] == 'asc': angle = horiscope.calcAngle('asc','asc') if (angle > 210 and angle < 240) or (angle > 120 and angle < 150): score += rule.score - print rule.planet[0] +' '+ rule.planet[1] +' '+ str(aspect)+' '+str(rule.score) + elif horiscope.planets[rule.planet[0]].angleA is None: + pdb.set_trace() + print 'Error - None Location' + return None + elif rule.planet[1] == 'house': + if horiscope.isInHouse(rule.planet[0],rule.aspect): + score += rule.score + elif rule.planet[0] == rule.planet[1]: + aspect,variance = horiscope.calcAspect(rule.planet[0],rule.planet[1]) + if aspect is not None: + if aspect in rule.aspect: + score += rule.score + else: + for order in [0,1]: + aspect,variance = horiscope.calcAspect(rule.planet[order],rule.planet[not order]) + if aspect is not None: + if aspect in rule.aspect: + score += rule.score + return score + + + def printCompatibility(self, horiscope): + score = 0 + outString = [] + for rule in sorted(self.rules): + if rule.planet[1] == 'house' and rule.planet[0] == 'asc': + angle = horiscope.calcAngle('asc','asc') + if (angle > 210 and angle < 240) or (angle > 120 and angle < 150): + score += rule.score + outString += rule.planet[0] +' '+ rule.planet[1] +' '+ str(aspect)+' '+str(rule.score) + '\n' # aspect,variance = horiscope.calcAspect(rule.planet[order],rule.planet[not order]) - elif horiscope.planets[rule.planet[0]].angleA is None: pdb.set_trace() # print 'Error - None Location' @@ -86,27 +107,23 @@ elif rule.planet[1] == 'house': if horiscope.isInHouse(rule.planet[0],rule.aspect): score += rule.score - print rule.planet[0] +' '+ rule.planet[1] +' '+ str(aspect)+' '+str(rule.score) + outString += rule.planet[0] +' '+ rule.planet[1] +' '+ str(aspect)+' '+str(rule.score)+ '\n' elif rule.planet[0] == rule.planet[1]: aspect,variance = horiscope.calcAspect(rule.planet[0],rule.planet[1]) if aspect is not None: if aspect in rule.aspect: score += rule.score - print rule.planet[0]+' '+aspect +' '+ rule.planet[1] +' '+ str(rule.score) + outString += rule.planet[0]+' '+aspect +' '+ rule.planet[1] +' '+ str(rule.score) +'\n' else: for order in [0,1]: aspect,variance = horiscope.calcAspect(rule.planet[order],rule.planet[not order]) if aspect is not None: - # print aspect - # currentR = r - # pdb.set_trace() if aspect in rule.aspect: score += rule.score - print rule.planet[0] +' '+ aspect +' '+ rule.planet[1] +' '+ str(rule.score) + outString += rule.planet[0] +' '+ aspect +' '+ rule.planet[1] +' '+ str(rule.score)+'\n' else: - print rule.planet[0] +' '+ aspect +' '+ rule.planet[1] +' 0' - print score - return score + outString += rule.planet[0] +' '+ aspect +' '+ rule.planet[1] +' 0'+'\n' + return score,outString @@ -132,6 +149,7 @@ self.p_cob = personDict['pCOB'] self.horiscope = None self.resp = None + self.vertex = None self.issue = 'INCOMPLETE' def identifyIssues(self): @@ -221,13 +239,15 @@ def requestURL(self): # self.resp = requests.get(self.url, params=self.payload) - self.resp = lr.get(self.url, params=self.payload, timeout=5, source='local', verbose=True) + self.resp = lr.get(self.url, params=self.payload, timeout=5, source=DEFAULT_SOURCE, verbose=True) # time.sleep(5) return self.resp def parsePage(self): self.horiscope = None # pdb.set_trace() + if self.resp is None: + pdb.set_trace() if('Please use valid date.' in self.resp['content']): self.issue = 'dob' return self.horiscope @@ -253,9 +273,47 @@ if planetName == 'node': self.horiscope.planets['southnode'].angleA = self.horiscope.planets['node'].angleA+180 self.horiscope.planets['southnode'].angleB = self.horiscope.planets['node'].angleB+180 + vertex = requestVertex(None, self.payload) + self.horiscope.planets['vertex'].setLocation(vertex[0][0],vertex[0][1],0) + self.horiscope.planets['vertex'].setLocation(vertex[1][0],vertex[1][1],1) return self.horiscope +def parseVertex(content): + soup = BeautifulSoup(content, 'lxml') + tcCell = soup.find('div', attrs={'class':'detail-rozbor-obalka'}) + tableList = tcCell.find_all('table')[0] + row = tableList.find_all('tr')[1] + box = row.find_all('td')[2] + num = box.getText() + zone = box.img.attrs['alt'] + return num,zone +def slicedict(d, s): + return {k.replace(s,''):v for k,v in d.iteritems() if k.startswith(s)} + +def setVertexPayload(payload,isPartner): + if isPartner: + head = 'zena_' + else: + head = 'muz_' + + vPayload = slicedict(payload, head) + vPayload['send_calculation'] = 1 + # pdb.set_trace() + return vPayload + +def requestVertex(url,payload): + resp = dict() + vertex = dict() + if url == None: + url = 'https://horoscopes.astro-seek.com/calculate-vertex-sign/' + for i in [0,1]: # 0 is individual, 1 is partner + vPayload = setVertexPayload(payload,i) + resp[i] = lr.get(url, params=vPayload, timeout=5, source=DEFAULT_SOURCE, verbose=True) + # pdb.set_trace() + vertex[i] = parseVertex(resp[i]['content']) + # pdb.set_trace() + return vertex class planetRelation: noHouseList = ['asc','ic','dsc','mc','asc/mc','sun/moon','sNode'] @@ -292,7 +350,7 @@ aspectRange = {'conjunction':10,'semi-square':2,'sextile':4,'square':10,'trine':10,'opposition':10} # zodiacAngle = {'aries':0,'taurus':30,'gemini':60,'cancer':90,'leo':120,'virgo':150,'libra':180,'scorpio':210,'sagittarius':240,'capricorn':270,'aquarius':300,'pisces':330} # Taken from https://en.wikipedia.org/wiki/Astrological_sign, with reference from https://en.wikipedia.org/wiki/Astrological_symbols#Signs_of_the_zodiac - planetNames = ['sun','moon','mercury','venus','mars','jupiter','saturn','uranus','neptune','pluto','node','southnode','lilith','chiron','asc','ic','dsc','mc','asc/mc','sun/moon'] + planetNames = ['sun','moon','mercury','venus','mars','jupiter','saturn','uranus','neptune','pluto','node','southnode','lilith','chiron','asc','ic','dsc','mc','asc/mc','sun/moon','vertex'] def __init__(self): self.planets = {} for p in planetPositions.planetNames: