基于Mesos的网页爬虫框架RENDLER
RENDLER是一个基于Apache Mesos的分布式爬虫框架,有很多值得借鉴的东西,大家可关注下。
https://github.com/mesosphere/RENDLER
http://mesosphere.github.io/presentations/hack-week-2014
安装 vagrant ssh
先运行 https://github.com/mesosphere/RENDLER/blob/master/cpp/README.md
分配的CPU和内存
mv result.dot ../
cd /home/vagrant/hostfiles/bin
./make-pdf
Generating ‘/home/vagrant/hostfiles/result.pdf’