Warning: You are using the test version of PyPI. This is a pre-production deployment of Warehouse. Changes made here affect the production instance of TestPyPI (testpypi.python.org).
Help us improve Python packaging - Donate today!

An image crawler with extendible modules and gui

Project Description

Comic Crawler 是用來扒圖的一支 Python Script。擁有簡易的下載管理員、圖書館功能、 與方便的擴充能力。


  • Extendible module design.
  • Easy to use function grabhtml, grabimg.
  • Auto setup referer and other common headers.


  • pyexecjs - to execute javascript.
  • pythreadworker - a small threading library.

Development Dependencies

  • pypandoc - to convert markdown to rst.


Comic Crawler is on PyPI. 安裝完 python 後,可以直接用 pip 指令自動安裝。


你需要 Python 3.4 以上。安裝檔可以從它的 官方網站 下載。

安裝時記得要選「Add python.exe to path」,才能使用 pip 指令。

Comic Crawler

在 cmd 底下輸入以下指令

pip install comiccrawler


pip install --update comiccrawler

Supported domains

chan.sankakucomplex.com comic.acgn.cc comic.ck101.com comic.sfacg.com danbooru.donmai.us deviantart.com exhentai.org g.e-hentai.org konachan.com manhua.dmzj.com tel.dm5.com www.8comic.com www.99comic.com www.comicvip.com www.dm5.com www.manhuadao.com www.pixiv.net yande.re


  comiccrawler domains
  comiccrawler download URL [--dest SAVE_FOLDER]
  comiccrawler gui
  comiccrawler (--help | --version)

  domains             列出支援的網址
  download URL        下載指定的 url
  gui                 啟動主視窗

  --dest SAVE_FOLDER  設定下載目錄(預設為 ".")
  --help              顯示幫助訊息
  --version           顯示版本



  • 在文字欄貼上網址後點「加入連結」或是按 Enter
  • 若是剪貼簿裡有支援的網址,且文字欄同時是空的,程式會自動貼上
  • 對著任務右鍵,可以選擇把任務加入圖書館。圖書館內的任務,在每次程式啟動時,都會檢查是否有更新。


; 設定下載完成後要執行的程式,會傳入下載資料夾的位置
runafterdownload =

; 啟動時自動檢查圖書館更新
libraryautocheck = true

; 下載目的資料夾
savepath = ~/comiccrawler/download

; 開啟 grabber 偵錯
logerror = false

; 每隔 5 分鐘自動存檔
autosave = 5
  • 設定檔位於 %USERPROFILE%\comiccrawler\setting.ini
  • 執行一次 comiccrawler gui 後關閉,設定檔會自動產生

Module example

#! python3
This is an example to show how to write a comiccrawler module.


import re
import comiccrawler.core

# The header used in grabber method
header = {}

# Match domain
domain = ["www.example.com", "comic.example.com"]

# Module name
name = "This Is an Example"

# With noepfolder = True, Comic Crawler won't generate subfolder for each episode.
noepfolder = False

# Wait 5 seconds between each page
rest = 5

# Specific user settings
config = {
    "user": "user-default-value",
    "hash": "hash-default-value"

def loadconfig():
    """This function will be called each time the config reloaded.
    header["Cookie"] = "user={}; hash={}".format(config["user"], config["hash"])

def gettitle(html, url):
    """Return mission title.

    Title will be used in saving filepath, so be sure to avoid duplicate title.
    return re.search("<h1 id='title'>(.+?)</h1>", html).group(1)

def getepisodelist(html, url):
    """Return episode list.

    The episode list should be sorted by date, latest at last, so the
    downloader will download the oldest first.
    base = re.search("(https?://[^/]+)", url).group(1)
    ms = re.findall("<a href='(.+?)'>(.+?)</a>", html)
    s = []
    for m in ms:
        u, title = m
        e = comiccrawler.Episode()
        e.title = title
        e.firstpageurl = base + url
    return s

There are two methods to get images url. If you can get all urls from the
first page, then use getimgurls. If you have to download each pages to get
image url, use getimgurl and nextpage functions.

You should only use one of two methods. Never write getimgurls and getimgurl

def getimgurls(html, url):
    """Return the list of all images"""

    ms = re.findall("<img src='(.+?)'>", html)
    return [m[0] for m in ms]

def getimgurl(html, page, url):
    """Return the url of the image"""

    return re.search("<img id='showimage' src='(.+?)'>", html).group(1)

def getnextpageurl(page, html, url):
    """Return the url of the next page. Return '' if this is the last page.

    r = re.search("<a id='nextpage' href='(.+?)'>next</a>", html)
    if r is None:
        return ""
    return r.group(1)

def errorhandler(er, ep):
    """Downloader will call errorhandler if there is an error happened when
    downloading image. Normally you can just ignore this function.


Release History

Release History

This version
History Node


Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting