ps.knnW_from_shapefile returns wrong W ids when idVariable specified

[darribas]

The following code:

import pysal as ps

shp = ps.examples.get_path('Chicago77.shp')
w = ps.weights.knnW_from_shapefile(shp, k=5, idVariable="COMMUNITY")
print "ID order (first 5): ", w.id_order[:5]
print "Neighbors (first 5): ", w.neighbors.keys()[:5]

produces this output:

In [5]: run ~/Desktop/ex.py
ID order (first 5):  ['ROGERS PARK', 'WEST RIDGE', 'EDISON PARK', 'FOREST GLEN', 'OHARE']
Neighbors (first 5):  [0, 1, 2, 3, 4]

There must be some inconsistency in how the ids are (not) replaced in the neighbors/weights dictionaries.

[sjsrey]

import pysal as ps
shp = ps.examples.get_path('Chicago77.shp')
w = ps.weights.knnW_from_shapefile(shp, k=5, idVariable="COMMUNITY")
print "ID order (first 5): ", w.id_order[:5]
print "Neighbors (first 5): ", w.neighbors.keys()[:5]

for i in w.neighbors.keys()[:5]:
     print w.id_order[i]

Generates:

ID order (first 5):  ['ROGERS PARK', 'WEST RIDGE', 'EDISON PARK', 'FOREST GLEN', 'OHARE']
Neighbors (first 5):  [0, 1, 2, 3, 4]
ROGERS PARK
WEST RIDGE
EDISON PARK
FOREST GLEN
OHARE

The ids attribute is not replaced since when it comes time to write out the gal things can go south with strings (West Ridge) might be considered two neighbors instead of one.

For a bridge between the idVariable and the position/row in the weights see:

In [2]: w.id2i
Out[2]:
{'ALBANY PARK': 11,
 'ARCHER HEIGHTS': 41,
 'ARMOUR SQUARE': 35,
 'ASHBURN': 59,
 'AUBURN GRESHAM': 60,
 'AUSTIN': 23,
 'AVALON PARK': 61,
 'AVONDALE': 17,
 'BELMONT CRAGIN': 19,
 'BEVERLY': 67,
 'BRIDGEPORT': 36,
 'BRIGHTON PARK': 39,
 'BURNSIDE': 66,
 'CALUMET HEIGHTS': 64,
 'CHATHAM': 63,
 'CHICAGO LAWN': 54,
 'CLEARING': 56,
 'DOUGLAS': 37,
 'DUNNING': 16,
 'EAST GARFIELD PARK': 29,
 'EAST SIDE': 68,
 'EDGEWATER': 6,
 'EDISON PARK': 2,
 'ENGLEWOOD': 51,
 'FOREST GLEN': 3,
 'FULLER PARK': 43,
 'GAGE PARK': 48,
 'GARFIELD RIDGE': 45,
 'GRAND BOULEVARD': 42,
 'GREATER GRAND CROSSING': 57,
 'HEGEWISCH': 74,
 'HERMOSA': 18,
 'HUMBOLDT PARK': 25,
 'HYDE PARK': 49,
 'IRVING PARK': 13,
 'JEFFERSON PARK': 8,
 'KENWOOD': 46,
 'LAKE VIEW': 14,
 'LINCOLN PARK': 21,
 'LINCOLN SQUARE': 9,
 'LOGAN SQUARE': 22,
 'LOOP': 28,
 'LOWER WEST SIDE': 33,
 'MCKINLEY PARK': 38,
 'MONTCLARE': 20,
 'MORGAN PARK': 73,
 'MOUNT GREENWOOD': 72,
 'NEAR NORTH SIDE': 26,
 'NEAR SOUTH SIDE': 32,
 'NEAR WEST SIDE': 27,
 'NEW CITY': 44,
 'NORTH CENTER': 15,
 'NORTH LAWNDALE': 31,
 'NORTH PARK': 7,
 'NORWOOD PARK': 5,
 'OAKLAND': 40,
 'OHARE': 4,
 'PORTAGE PARK': 12,
 'PULLMAN': 71,
 'RIVERDALE': 75,
 'ROGERS PARK': 0,
 'ROSELAND': 65,
 'SOUTH CHICAGO': 62,
 'SOUTH DEERING': 70,
 'SOUTH LAWNDALE': 34,
 'SOUTH SHORE': 58,
 'UPTOWN': 10,
 'WASHINGTON HEIGHTS': 69,
 'WASHINGTON PARK': 50,
 'WEST ELSDON': 47,
 'WEST ENGLEWOOD': 52,
 'WEST GARFIELD PARK': 30,
 'WEST LAWN': 55,
 'WEST PULLMAN': 76,
 'WEST RIDGE': 1,
 'WEST TOWN': 24,
 'WOODLAWN': 53}

[darribas]

Correct, but if you try to write it out, it fails:

fo = ps.open('test.gwt', 'w')
fo.write(w)
fo.close()

Returns:

In [5]: fo.write(w)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-5-8009ae4bdfd6> in <module>()
----> 1 fo.write(w)

/home/dani/code/pysal/pysal/core/IOHandlers/gwt.pyc in write(self, obj)
    247             header = '%s %i %s %s\n' % ('0', obj.n, self.shpName, self.varName)
    248             self.file.write(header)
--> 249             self._writelines(obj)
    250             #obj.transform = transform
    251 

/home/dani/code/pysal/pysal/core/IOHandlers/gwt.pyc in _writelines(self, obj)
    176         """
    177         for id in obj.id_order:
--> 178             neighbors = zip(obj.neighbors[id], obj.weights[id])
    179             for neighbor, weight in neighbors:
    180                 self.file.write('%s %s %6G\n' % (str(id),

KeyError: 'ROGERS PARK'

In [6]:

A similar approach to, say, ps.threshold_continuousW_from_shapefile would change all the dictionaries:

import pysal as ps
import numpy as np
shp = ps.examples.get_path('Chicago77.shp')
w = ps.threshold_continuousW_from_shapefile(shp, np.inf, idVariable="COMMUNITY")
print "ID order (first 5): ", w.id_order[:5]
print "Neighbors (first 5): ", w.neighbors.keys()[:5]

would return:

ID order (first 5):  ['ROGERS PARK', 'WEST RIDGE', 'EDISON PARK', 'FOREST GLEN', 'OHARE']
Neighbors (first 5):  ['CALUMET HEIGHTS', 'ROGERS PARK', 'WEST TOWN', 'EDISON PARK', 'FOREST GLEN']

(Ignore the order as neighbors is a dictionary and won't preserve it)

[sjsrey]

This is a bug that raises its head in several places. If you write out the gwt for your last example, you will see that it looks something like (first few lines):

0 77 Unknown Unknown
ROGERS PARK ARCHER HEIGHTS 1.35026E-05
ROGERS PARK GARFIELD RIDGE 1.22131E-05
ROGERS PARK WEST ELSDON 1.24567E-05
ROGERS PARK WEST LAWN 1.12682E-05
ROGERS PARK CLEARING 1.13634E-05
ROGERS PARK ASHBURN 1.03377E-05
ROGERS PARK AUBURN GRESHAM 1.03276E-05
ROGERS PARK BEVERLY 9.25702E-06

This is obviously not what we want. The two Unknown should be replaced with the id_variable and name of shapefile
and the i and j in each row should be integers corresponding to the position of each in the order of the shapes.
The "names" associated with each i and j are in the id_variable.

[sjsrey]

on #244 (comment)

there are two issues. first, if i read the example correctly you are generating a knn weight, but then writing to a gwt file extension. the writers are dispatched based on the extension, so this would be a mismatch (knn vs. gwt).

but, the second issue is if you write it to a gal, the same error is triggered (key error). this is the bug we need to fix.

while i started on this, it raises a couple of more issues.

• how are multi-word string ids (ROGERS PARK) to be dealt with in the gwt specification?
• how do we support that with our readers and writers for W objects
• same questions can be asked about the gal format.

we should have a discussion about this as many fingers have been involved in the weights, reader, writers, so there are likely interdependencies that no single person has knowledge of.

[sjsrey]

One options is to follow GeoDa's approach and restrict the type of variable that can be used for the idVariable to be integer. in that case no string variables are allowed and the issue of multi word strings goes away.

[sjsrey]

I think we don't mess with this in 1.6. I started a branch https://github.com/pysal/pysal/tree/weight_ids to look at the refactoring of the idVariable.

[darribas]

I've come across a slightly different but related issue with idVariable on knnW_from_shapefile so, rather than putting it in a separate issue, I'll add it to this thread. For some reason, in some cases, the method fails if an idVariable is passed. Example:

import pysal as ps
l = ps.examples.get_path('10740.shp')
w = ps.knnW_from_shapefile(l, 3) #Works fine
w = ps.knnW_from_shapefile(l, 3, idVariable='STFID') #Fails

The error returned is:

In [9]: w = ps.knnW_from_shapefile(l, 3, idVariable='STFID')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-82b98fb8dab8> in <module>()
----> 1 w = ps.knnW_from_shapefile(l, 3, idVariable='STFID')

/home/dani/code/pysal_darribas/pysal/weights/user.pyc in knnW_from_shapefile(shapefile, k, p, idVariable, radius)
    322     if idVariable:
    323         ids = get_ids(shapefile, idVariable)
--> 324         return knnW(data, k=k, p=p, ids=ids)
    325     return knnW(data, k=k, p=p)
    326 

/home/dani/code/pysal_darribas/pysal/weights/Distance.pyc in knnW(data, k, p, ids, pct_unique)
    146         idset = np.arange(len(info))
    147 
--> 148     neighbors = dict( [ (idset[i],row[row!=idset[i]].tolist()[:k]) for i,row in enumerate(info)  ] )
    149     return pysal.weights.W(neighbors,  id_order=ids)
    150 

TypeError: 'int' object has no attribute '__getitem__'

In [10]:

[sjsrey]

Closing as the multi-word ids is not supported in the GWT spec so it isn't something we can do much about.

Also the related issue @darribas mentioned above with the 10740.shp is now fixed.