blockspider

module
v0.0.0-...-91eafdd Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 14, 2023 License: MIT

README

blockspider

An open source blockchain crawler and kafka producer.

Note: Although fully functional, blockspider is currently considered a WIP. Significant changes will occur before the first official release.

Requires
  • go 1.20.x or greater
  • Apache Kafka 3.6.x or greater
  • a geth-like rpc endpoint
Get the source
git clone https://github.com/iquidus/blockspider blockspider
build
cd blockspider && make blockspiderd
Configure
cp ./config.json.example ./config.json

Make required changes in config.json

{
  "chainId": 8, // chainId of target network
  "crawler": {
    // crawler settings
    "start": 0, // start block
    "interval": "10000ms", // polling interval. e.g 0.5 * target block time
    "routines": 1, // go routines
    "kafka": {
      "events": [
        {
          "broker": "localhost:9092",
          "topic": "events",
          "addresses": [],
          "topics": []
        }
      ],
      "blocks": {
        "broker": "localhost:9092",
        "topic": "blocks"
      }
    }
  },
  "rpc": {
    "type": "http",
    "endpoint": "http://127.0.0.1:8588"
  },
  "state": {
    "path": "~/.blockspider/ubiq-mainnet.json",
    "cache": 128, // number of blocks to keep in local cache. Must be larger than reorgs.
  }
}
Run
./build/bin blockspiderd -c config.json
Kafka

Download the latest Kafka release and extract it

tar -xzf kafka_2.13-3.6.0.tgz
cd kafka_2.13-3.6.0

Run the following commands in order to start all services in the correct order

# Start the ZooKeeper service
bin/zookeeper-server-start.sh config/zookeeper.properties

Open another terminal session and run

# Start the Kafka broker service
bin/kafka-server-start.sh config/server.properties

Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.

Create topics

bin/kafka-topics.sh --create --topic blocks --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic events --bootstrap-server localhost:9092

Directories

Path Synopsis
cmd
blockspiderd command
consumer command
gettestdata command
reorgd command
transmuted command
internal
Package hexutil implements hex encoding with 0x prefix.
Package hexutil implements hex encoding with 0x prefix.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL