Skip to content
Back to search

yacy_grid_mcp

The YaCy Grid Master Connect Program

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/yacy--yacy-grid-mcp/badge.svg)](/listings/yacy--yacy-grid-mcp)

HTML

<a href="/listings/yacy--yacy-grid-mcp"><img src="/api/listings/yacy--yacy-grid-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

At this time, yacy_grid_mcp is not provided in compiled form, you easily build it yourself. It's not difficult and done in one minute! The source code is hosted at https://github.com/yacy/yacy_grid_mcp, you can download it and run loklak with:

> git clone https://github.com/yacy/yacy_grid_mcp.git > cd yacy_grid_mcp > gradle run

Requirements for development

Maven and Gradle tools should be installed. To refresh gradle settings in eclipse, right click on project -> configure -> Add Gradle Nature

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 292 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # YaCy Grid Component: Master Connect Program (MCP)
  2. [![Build Status](https://travis-ci.org/yacy/yacy_grid_mcp.svg?branch=master)](https://travis-ci.org/yacy/yacy_grid_mcp)
  3. The YaCy Grid is the second-generation implementation of YaCy, a peer-to-peer search engine.
  4. A YaCy Grid installation consists of a set of micro-services which communicate with each other
  5. using a common infrastructure for data persistence. The required storage functions of the YaCy Grid are:
  6. - An asset storage, basically a file sharing environment for YaCy components
  7. - A message system providing an Enterprise Integration Framework using a message-oriented middleware
  8. - A database system providing search-engine related retrieval functions.
  9. All YaCy components are microservices which can be deployed i.e. using Docker or other application
  10. hosting methods which are provided as cloud services from many different providers. A YaCy grid will
  11. scale with the number of microservices that connect to a common broker.
  12. That broker is the MCP, this application.
  13. The MCP provides http servlets to access the services mentioned above: assets, messages, databases.
  14. If a YaCy Grid microservice wants to connect to this infrastructure, it connects to the MCP to
  15. use them. The API to access MCP servlets are integrated into the MCP as well, every YaCy Grid service
  16. must integrate the whole MCP as infrastructure to get access to the API. This makes it possible that
  17. the MCP acts also as router to the infrastructure: if a YaCy Grid service has used the MCP once, it
  18. learns from the MCP to connect to the infrastucture itself. For example:
  19. - a YaCy Grid service starts up and connects to the MCP
  20. - the Grid service pushes a message to the message queue using the MCP
  21. - the MCP fullfills the message send operation and respons with the actual address of the message broker
  22. - the YaCy Grid service learns the direct connection information
  23. - whenever the YaCy Grid service wants to connect to the message broker again, it can do so using a direct broker connection.
  24. This process is done transparently, the Grid service does not need to handle such communication details itself. The routing is done automatically.
  25. To use the MCP inside other grid components the git submodule functionality is used.
  26. The asset storage, message distribution and database services provided by the MCP are handled in an
  27. abstract way, that means there is no need to know which system actually performs the operation.
  28. However, the following external services are supposed to be supported by the MCP:
  29. - ftp server for asset storage
  30. - RabbitMQ message queues for the message system
  31. - Elasticsearch for database operations.
  32. For a full enterprise-integration installation, the operator has to start and support a ftp server, a rabbitmq and an elasticsearch. However, all of them can be omitted and the MCP will provide all of these services with built-in functions. For database operations and message storage a MapDB is used, asset files are stored in a local data directory.
  33. ## Port numbers in the YaCy Grid:
  34. Every YaCy Grid service has a default port. However every service can be started several times.
  35. The service is able to detect that another service is running on the same port and will choose an
  36. alternative port. That means we will not have any port number collisions any more, just more microservices for one grid.
  37. The default port number of the MCP is 8100
  38. Other port numbers will be:
  39. - 8200: webloader, a http(s) loader acting as headless browser which is able to enrich http with AJAX content
  40. - 8300: webcrawler, a crawler which loads a lot of documents from web documents
  41. - 8400: warcmanager, a process which combines single WARC files to bigger ones to create archives
  42. - 8500: yacyparser, a parser service which turns WARC into YaCy JSON
  43. - 8600: yacyenricher, a semantic enricher for YaCy JSON objects
  44. - 8700: yacyindexer, a loader which pushes parsed/enriched YaCy JSON content to a search index
  45. - 8800: aggregation, a search front-end which combines different index sources into one
  46. - 8900: moderation, a search front-end which for content moderation, i.e. search index account management
  47. - 10100: successmessages, a service which handles the successful operation messages
  48. - 10200: errormessages, a service which handles failure messages and broken action chains
  49. - 2121: ftp, a FTP server to be used for mass data / file storage
  50. - 5672: rabbitmq, a rabbitmq message queue server to be used for global messages, queues and stacks
  51. - 9300: elastic, an elasticsearch server or main cluster address for global database storage
  52. ## Communication
  53. Please join our forums at https://searchlab.eu
  54. ## How do I install the yacy_grid_mcp: Download, Build, Run
  55. At this time, yacy_grid_mcp is not provided in compiled form, you easily build it yourself. It's not difficult and done in one minute! The source code is hosted at https://github.com/yacy/yacy_grid_mcp, you can download it and run loklak with:
  56. > git clone https://github.com/yacy/yacy_grid_mcp.git
  57. > cd yacy_grid_mcp
  58. > gradle run
  59. ### Requirements for development
  60. Maven and Gradle tools should be installed. To refresh gradle settings in eclipse, right click on project -> configure -> Add Gradle Nature
  61. ## How to install the infrastructure services (message server, ftp server)?
  62. ### install and start apache ftp server
  63. - Download apache-ftpserver-1.1.1.tar.gz (or a later version) from https://mina.apache.org/downloads-ftpserver.html
  64. - decompress the server package with
  65. > tar xfz apache-ftpserver-1.1.1.tar.gz
  66. - modify the write right of the anonymous user that we will use to write assets: edit the file
  67. apache-ftpserver-1.1.1/res/conf/users.properties
  68. - set the write permission for anonymous.. if you don't want to set an user account
  69. ftpserver.user.anonymous.writepermission=true
  70. - to create a user account, clone all entries for the user anonymous, rename the account name and set a password using a md5 encoding. The following command will encode the password 'password4account':
  71. > echo -n password4account | md5sum
  72. - set a home path for the user, i.e.
  73. ftpserver.user.anonymous.homedirectory=/Users/admin/Downloads/ftphome
  74. - set a high number of allowed connections for `maxloginnumber` and `maxloginperip`
  75. The complete configuration may then look like
  76. ```
  77. ftpserver.user.yacygrid.userpassword=<here is the md5sum>
  78. ftpserver.user.yacygrid.homedirectory=./res/home
  79. ftpserver.user.yacygrid.enableflag=true
  80. ftpserver.user.yacygrid.writepermission=true
  81. ftpserver.user.yacygrid.maxloginnumber=2000
  82. ftpserver.user.yacygrid.maxloginperip=2000
  83. ftpserver.user.yacygrid.idletime=30000
  84. ftpserver.user.yacygrid.uploadrate=0
  85. ftpserver.user.yacygrid.downloadrate=0
  86. ```
  87. - edit the file apache-ftpserver-1.1.1/bin/ftpd.sh and set JAVA_HOME according to your system, i.e. on a Mac you set
  88. JAVA_HOME=/usr
  89. - run the server, doing:
  90. > cd apache-ftpserver-1.1.1
  91. > bin/ftpd.sh res/conf/ftpd-typical.xml
  92. This will run the ftp server at port 2121. To test the connection use a standard ftp client and start it with
  93. > ftp -P 2121 anonymous@127.0.0.1
  94. ### install and start rabbitmq:
  95. - Download rabbitmq-server-generic-unix-3.6.6.tar.xz (from https://www.rabbitmq.com/download.html)
  96. - Extract the package with
  97. > tar xf rabbitmq-server-generic-unix-3.6.6.tar.xz
  98. - install the erlang programming language, rabbitmq is written in erlang. I.e. in debian, install:
  99. > apt-get install erlang
  100. - if you own a Mac, install brew and then install erlang with
  101. > brew install erlang
  102. - Run the server with
  103. > rabbitmq_server-3.6.6/sbin/rabbitmq-server
  104. - install the management plugin to be able to use a web interface:
  105. > rabbitmq_server-3.6.6/sbin/rabbitmq-plugins enable rabbitmq_management
  106. - add a rabbitmq user 'anonymous' with password 'yacy':
  107. > rabbitmq_server-3.6.6/sbin/rabbitmqctl add_user anonymous yacy
  108. - set administration rights to that user (without that, it is not possible to open the administration pages)
  109. > rabbitmq_server-3.6.6/sbin/rabbitmqctl set_user_tags anonymous administrator
  110. - set access right to vserver path '/' (makes it possible that queues are written over the api)
  111. > rabbitmq_server-3.6.6/sbin/rabbitmqctl set_permissions -p / anonymous ".*" ".*" ".*"
  112. - to make it possible that the rabbitmq server can be accessed from outside of localhost, add a configuration file to `rabbitmq_server-3.6.6/etc/rabbitmq/rabbitmq.config` with the following content:
  113. ```
  114. [{rabbit, [{loopback_users,[]}]}].
  115. ```
  116. - if you made it possible to set a remote connection to localhost, change the default password of the guest account (or remove the guest account)
  117. - To view the administration pages, open in your web browser: http://127.0.0.1:15672
  118. - open http://127.0.0.1:15672/api/ for an api documentation
  119. - open http://127.0.0.1:15672/cli/ for the rabbitmqadmin
  120. ## Configuration of the MCP
  121. The mcp will create a subdirectory `data/mcp-8100`. There, within a `conf` sub-path you can place a file `config.properties` which has the same structure as the file in `<application-home>/conf/config.properties`. Just copy that file to `data/mcp-8100/conf/config.properties` and replace the default values with your own.
  122. You should set the ftp address and the broker address using url-encoded user-name/pw-settings, like:
  123. ```
  124. grid.ftp.address = <user>:<pw>@<ftp-host>:2121
  125. grid.broker.address = <user>:<pw>@<broker-host>:5672
  126. ```
  127. ## How to use the API
  128. To test the api, try the following example:
  129. ### Writing messages
  130. Call:
  131. curl "127.0.0.1:8100/yacy/grid/mcp/messages/send.json?serviceName=testService&queueName=testQueue&message=hello_world"
  132. This will send a message "hello_world" to the queue 'test' of service 'test'. You can ask for the number of entries in the queue with
  133. curl "http://127.0.0.1:8100/yacy/grid/mcp/messages/available.json?serviceName=testService&queueName=testQueue"
  134. To get an entry from such a queue, call:
  135. curl "http://127.0.0.1:8100/yacy/grid/mcp/messages/receive.json?serviceName=testService&queueName=testQueue"
  136. If you did not run the RabbitMQ message server, the messages are written to
  137. data/mcp-8100/messages/testService
  138. If you started a RabbitMQ message server, please monitor the writing of the messages at the web page http://127.0.0.1:15672
  139. ### Writing assets
  140. Call:
  141. curl "http://127.0.0.1:8100/yacy/grid/mcp/assets/store.json?path=/xx/test.txt&asset=hello_world"
  142. This will write an asset to the path xx/test.txt with the content "hello_world".
  143. curl "http://127.0.0.1:8100/yacy/grid/mcp/assets/load?path=/xx/test.txt"
  144. will load the asset again.
  145. If you started a ftp server, the file(s) will be written relatively to the root path of the ftp home path.
  146. If you did not start a ftp server, you can find the file in data/mcp-8100/assets/xx/test.txt
  147. ### Using a second MCP to use the primary MCP
  148. The MCP organises the connection to the remote RabbitMQ server and a ftp server, but if another Grid Service is started, the MCP tells that service to handle the connection to the RabbitMQ and ftp server itself. The MCP can act as such an external service: just start another MCP and it will run at port 8101. Repeat the curl commands as given in the example above, but now use port 8101 to access the MCP. You will see that the second MCP learns from the first MCP to handle the connection by itself.
  149. ### How do I install yacy_grid_mcp with Docker?
  150. To install yacy_grid_mcp with Docker please refer to the [yacy Docker installation readme](/docs/installation_docker.md).
  151. ### How do I deploy yacy_grid_mcp on Cloud Providers?
  152. To install yacy_grid_mcp on Cloud Providers please look documentations at [yacy_grid_mcp Cloud installation readme](/docs/).
  153. ### Useful debugging URLs
  154. #### RabbitMQ
  155. - http://localhost:15672/
  156. - use user 'anonymous' with password 'yacy'
  157. #### Elasticsearch
  158. - http://localhost:9200/_cat/indices
  159. - http://localhost:9200/crawlstart/_search?q=*:*
  160. - http://localhost:9200/crawler/_search?q=*:*
  161. - http://localhost:9200/web/_search?q=url_s:*tagesschau*
  162. - http://localhost:9200/web/_search?q=_id:d5452c253d5eb541135394f42bfa655f
  163. - http://localhost:9200/web/_doc/d5452c253d5eb541135394f42bfa655f
  164. - http://localhost:9200/crawler/_doc/d5452c253d5eb541135394f42bfa655f
  165. #### MCP
  166. - http://localhost:8100/yacy/grid/mcp/info/threaddump.txt
  167. - http://localhost:8100/yacy/grid/mcp/control/loaderThrottling.json?url=klg.de
  168. - http://localhost:8100/yacy/grid/mcp/index/count.json?index=web&query=tagesthemen
  169. - http://127.0.0.1:8100/yacy/grid/mcp/index/exist.json?index=web&id=31bf58014628ee9e28b5ffb8b91ddf3e
  170. - http://127.0.0.1:8100/yacy/grid/mcp/index/gsasearch.xml?q=*
  171. - http://127.0.0.1:8100/yacy/grid/mcp/index/yacysearch.json?query=*
  172. #### Crawler
  173. - http://localhost:8300/yacy/grid/mcp/info/threaddump.txt
  174. - http://localhost:8300/yacy/grid/crawler/crawlStart.json?crawlingURL=yacy.net&indexmustnotmatch=.*Mitmachen.*&mustmatch=.*yacy.net.*
  175. #### Loader
  176. - http://localhost:8200/yacy/grid/mcp/info/threaddump.txt
  177. - http://localhost:8200/yacy/grid/loader/warcloader.warc.gz?url=http://yacy.net
  178. - curl "http://localhost:8200/yacy/grid/loader/warcprocess.json?collection=test&targetasset=test/yacy.net.warc.gz&url=http://yacy.net"
  179. - ...places the warc file on the asset store
  180. #### Parser
  181. - http://localhost:8500/yacy/grid/mcp/info/threaddump.txt
  182. - http://127.0.0.1:8500/yacy/grid/parser/jsonldvalidator.json?etherpad=05cc1575f55de2dc82f20f9010d71358
  183. - http://127.0.0.1:8500/yacy/grid/parser/jsonldvalidator.json?url=http://ebay.de
  184. - http://127.0.0.1:8500/yacy/grid/parser/jsonldvalidator.json?url=https://release-8-0-x-dev-224m2by-lj6ob4e22x2mc.eu.platform.sh/test
  185. - Flatfile Example: save the json result as flat file to land.nrw.flatjson
  186. 1. produce a WARC file with `wget https://www.land.nrw/ --warc-file=land.nrw`
  187. 2. compress the WARC file `gzip -9 land.nrw`
  188. 3. parse the warc file and write a flat json with `curl -X POST -F "sourcebytes=@land.nrw.warc.gz;type=application/octet-stream" -F "flatfile=true" -o land.nrw.flatjson http://127.0.0.1:8500/yacy/grid/parser/parser.json`
  189. ## Contribute
  190. This is a community project and your contribution is welcome!
  191. 1. Check for [open issues](https://github.com/yacy/yacy_grid_mcp/issues)
  192. or open a fresh one to start a discussion around a feature idea or a bug.
  193. 2. Fork [the repository](https://github.com/yacy/yacy_grid_mcp.git)
  194. on GitHub to start making your changes (branch off of the master branch).
  195. 3. Write a test that shows the bug was fixed or the feature works as expected.
  196. 4. Send a pull request and bug us on Gitter until it gets merged and published. :)
  197. ## What is the software license?
  198. LGPL 2.1 (C) by Michael Peter Christen
  199. Have fun!
  200. @0rb1t3r

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.