Monday, March 17

Get started with Scrapy

As usual I setup the virtual environment and just pip install packages: lxml and Scrapy, lxml being the prerequisite. But this time it failed with following error:


It seems it tried to build lxml with packages libxml2 and libxsit from anaconda. Anaconda is a python distribution I installed to my machine, enabling easy installation of many scientific packages like scipy, matplotlib and pandas. It is strange because in virtual environment the python distribution should be the one within the environment.

After poking around, I found I had following line in my ~/.bash_profile:
# added by Anaconda 1.8.0 installer
export PATH="//anaconda/bin:$PATH"

It looks like to be added automatically when I installed Anaconda, making the python distribution in Anaconda the default one. In my virtual environment, where the supposed python distribution should be the virtual one, when I install packages, it still look for the system default one. By removing the line from bash profile, it seems fine now.

[TODO]


No comments:

Post a Comment