SimpleImport¶
The SimpleImport module offers a command line tool to import joboffers from feeds using a fix defined structure. It's usefull, if you plan to use YAWIK as a jobportal and if you would like to import jobs from customers.
The CLI offers the following functionalisites.
imple import operations
yawik simpleimport import [--limit] [--name] [--id] Executes a data import for all registered crawlers
yawik simpleimport add-crawler Adds a new import crawler
--limit=INT Number of crawlers to check per run. Default 3. 0 means no limit
--name=STRING The name of a crawler
--id=STRING The Mongo object id of a crawler
--organization==STRING The ID of an organization
--feed-uri=STRING The URI pointing to a data to import
--runDelay=INT The number of minutes the next import run will be proceeded again
--type=STRING The type of an import (e.g. job)
--jobInitialState=STRING The initial state of an imported job
--jobRecoverState=STRING The state a job gets, if it was deleted, but found again in later runs.
yawik simpleimport info Displays a list of all available crawlers.
yawik simpleimport info [--id] <name> Shows information for a crawler
yawik simpleimport update-crawler [--id] <name> [--rename] [--limit] [--organization] [--feed-uri] [--runDelay] [--type] [--jobInitalState] [--jobRecoverState] Updates configuration for a crawler.
yawik simpleimport delete-crawler [--id] <name> Deletes an import crawler
<name> The name of the crawler to delete.
--id Treat <name> as the MongoID of the crawler
--rename=STRING Set a new name for the crawler.
yawik simpleimport guess-language [--limit] Find jobs without language set and pushes a guess-language job into the queue for each.
--limit=INT Maximum number of jobs to fetch. 0 means fetch all.
yawik simpleimport check-classifications <root> <categories> [<query>] Check job classifications.
<root> Root category ("industrues", "professions" or "employmentTypes")
<categories> Required categories, comma separated. E.g. "Fulltime, Internship"
<query> Search query for selecting jobs.
--force Do not ignore already checked jobs.
Feeds have to be formatted as defined in :ref:the scrapy docs<scrapy:format>.
Example: Add a feed to a an organization
You need an organizationId in order to add a crawler job. So at first you have to create a company in your yawik. The organizationId appears in an URL, if you try to edit the organization.
root@yawik:/var/www/YAWIK# ./vendor/bin/yawik simpleimport add-crawler --name=example-crawler \
--organization=59e4b53e7bb2b553412f9be9 \
--feed-uri=http://ftp.yawik.org/example.json
A new crawler with the ID "59e4b5a87bb2b567468b4567" has been successfully added.
This command created a crawler in the mongo collection simpleimport.crawler and returns the ObjectId.
> db.simpleimport.crawler.find({"_id":ObjectId("59e4b5a87bb2b567468b4567")}).pretty();
{
"_id" : ObjectId("59e4b5a87bb2b567468b4567"),
"name" : "example-crawler",
"organization" : ObjectId("59e4b53e7bb2b553412f9be9"),
"type" : "job",
"feedUri" : "http://ftp.yawik.org/example.json",
"runDelay" : NumberLong(1440),
"dateLastRun" : {
"date" : ISODate("1970-01-01T00:00:00Z"),
"tz" : "+00:00"
},
"options" : {
"initialState" : "active",
"_doctrine_class_name" : "SimpleImport\\Entity\\JobOptions"
}
}
Note
if you execute the command twice, the crawler will be added twice. If you want to remove a crawler, you have to do so on the mongo cli.