InterPlanetary File System (IPFS) on Jupyter

Jovians and Bluemix users rejoice!


Art of the Possible sometimes means mashing up Alpha technology with production level hosted cloud platforms ;-)
Brought to you courtesy of the IBM jStart Team (http://ibm.com/jstart)

In [1]:
# Author:  sanjay joshi  joshi(at)us(dot)ibm(dot)com  @jstartter   http://ibm.biz/sanjay_joshi
# Organization:  IBM jStart - Emerging Technologies and Client Engagement
# License: Apache 2.0
# Copyright 2016 IBM
# 
# To Reset the Notebook Installation State, run the following commands within a cell
#      !rm -rf $ipfsRepoDir
#      !rm -rf $localShare
#      !rm -rf $ipfsHomeDir
#      !rm -rf $goHomeDir
#      !rm -rf data
#      !ps -ef | grep ipfs
#      !kill pid# for the ipfs daemon shown in prior command
In [3]:
%load https://raw.githubusercontent.com/joshisa/jspark/master/jspark/ipfs.py
Prefix proposal accepted
prefix = /gpfs/global_fs01/sym_shared/YPProdSpark/user/sf62-201228d7d2d0db-bf066087ecf5
shareDir = /gpfs/global_fs01/sym_shared/YPProdSpark/user/sf62-201228d7d2d0db-bf066087ecf5/.local/share
Python packageDir = /gpfs/global_fs01/sym_shared/YPProdSpark/user/sf62-201228d7d2d0db-bf066087ecf5/.local
ipfs Dir = /gpfs/global_fs01/sym_shared/YPProdSpark/user/sf62-201228d7d2d0db-bf066087ecf5/.local/share/ipfs
Go Install Dir = /gpfs/global_fs01/sym_shared/YPProdSpark/user/sf62-201228d7d2d0db-bf066087ecf5/.local/share/go-download
IPFS Repo Dir = /gpfs/global_fs01/sym_shared/YPProdSpark/user/sf62-201228d7d2d0db-bf066087ecf5/.local/share/ipfsrepo
Congratulations! IPFS is already installed within your notebook user space

Success

In [4]:
ipfs=ipfs()
ipfs.help()
ipfs version 0.4.2

Error: ipfs daemon is running. please stop it to run this command
Use 'ipfs daemon --help' for information about this command

USAGE
  ipfs - Global p2p merkle-dag filesystem.

  ipfs [<flags>] <command> [<arg>] ...

SUBCOMMANDS
  BASIC COMMANDS
    init          Initialize ipfs local configuration
    add <path>    Add a file to ipfs
    cat <ref>     Show ipfs object data
    get <ref>     Download ipfs objects
    ls <ref>      List links from an object
    refs <ref>    List hashes of links from an object
  
  DATA STRUCTURE COMMANDS
    block         Interact with raw blocks in the datastore
    object        Interact with raw dag nodes
    files         Interact with objects as if they were a unix filesystem
  
  ADVANCED COMMANDS
    daemon        Start a long-running daemon process
    mount         Mount an ipfs read-only mountpoint
    resolve       Resolve any type of name
    name          Publish or resolve IPNS names
    dns           Resolve DNS links
    pin           Pin objects to local storage
    repo          Manipulate the IPFS repository
  
  NETWORK COMMANDS
    id            Show info about ipfs peers
    bootstrap     Add or remove bootstrap peers
    swarm         Manage connections to the p2p network
    dht           Query the DHT for values or peers
    ping          Measure the latency of a connection
    diag          Print diagnostics
  
  TOOL COMMANDS
    config        Manage configuration
    version       Show ipfs version information
    update        Download and apply go-ipfs updates
    commands      List all available commands
  
  Use 'ipfs <command> --help' to learn more about each command.
  
  ipfs uses a repository in the local file system. By default, the repo is located
  at ~/.ipfs. To change the repo location, set the $IPFS_PATH environment variable:
  
    export IPFS_PATH=/path/to/ipfsrepo


In [5]:
ipfs.cat("/ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG/readme")
Hello and Welcome to IPFS!

██╗██████╗ ███████╗███████╗
██║██╔══██╗██╔════╝██╔════╝
██║██████╔╝█████╗  ███████╗
██║██╔═══╝ ██╔══╝  ╚════██║
██║██║     ██║     ███████║
╚═╝╚═╝     ╚═╝     ╚══════╝

If you're seeing this, you have successfully installed
IPFS and are now interfacing with the ipfs merkledag!

 -------------------------------------------------------
| Warning:                                              |
|   This is alpha software. Use at your own discretion! |
|   Much is missing or lacking polish. There are bugs.  |
|   Not yet secure. Read the security notes for more.   |
 -------------------------------------------------------

Check out some of the other files in this directory:

  ./about
  ./help
  ./quick-start     <-- usage examples
  ./readme          <-- this file
  ./security-notes

In [6]:
ipfs.swarm("peers")
/ip4/10.142.18.202/tcp/4001/ipfs/QmfVkShe7UGoq1tdL8F6cY56HotXX5Bm6cMZAygTmQkXrx
/ip4/10.142.18.237/tcp/4001/ipfs/QmS9UxP2z2m9qJxH3Ua388n5cyHF28Yxv4VgQXJsyGyb4H
/ip4/10.143.133.129/tcp/4001/ipfs/QmUzXpnWikgzH9S1mXJ76vZmyFvq5C4r5AGVDLchQfp1sC
/ip4/10.143.133.52/tcp/4001/ipfs/QmWxvTueTm3XccWvbGvqtscB5sD2twZMuBRhrSxxD2oBSu
/ip4/173.49.6.14/udp/4737/utp/ipfs/QmWABRCqWoYCGyGV2PZ2T3D9upV39yMgkmFCprXVsKvAZr
/ip4/198.100.147.192/udp/4001/utp/ipfs/QmQTw94j68Dgakgtfd45bG3TZG6CAfc427UVRH4mugg4q4
/ip4/65.19.134.243/udp/4001/utp/ipfs/QmNfnZ8SCs3jAtNPc8kf3WJqJqSoX7wsX7VqkLdEYMao4u
/ip4/65.19.134.244/udp/4001/utp/ipfs/QmctA1rfDq4ZiaZgCLcRFvjYdVFfH8ebzTe67bfG7cVWsZ
/ip4/67.22.162.186/udp/4001/utp/ipfs/QmXMdzTPjzDGPYTPLX3VGiypvTuAahg7wNShENY31bLEQT
/ip4/71.204.170.241/udp/4001/utp/ipfs/QmPBY5mmBknFVDNybDC4fyk2uiLSs28xnu6gkKXB5bgkXp

In [7]:
ipfs.id()
{
	"ID": "Qmd2frzgukznnSPVRf9CicvkFMzLTnSz8bH6VY5t9ASvNP",
	"PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDm8erigwnTYzSkmjFHe8DafKcaWxv2oEJnKv/PZItyHq/EbKM6NbrvtfDJS+oT2FY2g6pVe7Sq5lLOc3Bqn/52xq0BJANpEdRnWIYIaE7KZphsprwBF7e0YaYIeo2PI1ySJjgY4E1Hez0gi5DGULuVYiNCQc0/5LS4sj/IIe48K8AFo015/xvMaUi+efOs5SMwKwsdQUrBmd66D3gMDyazyQYZUsaGvQgxd2i9uSeo+QuJPmhMs9VpSt9jnfhyDRhGaRf9gkUde9erE0T88JjN/xoF7BpjSRBhVfFPpL559VI8WVKFJlicO1cNJ0222DRWljAlO3R4TPBBOMTZUJinAgMBAAE=",
	"Addresses": [
		"/ip4/127.0.0.1/tcp/4001/ipfs/Qmd2frzgukznnSPVRf9CicvkFMzLTnSz8bH6VY5t9ASvNP",
		"/ip4/10.143.133.24/tcp/4001/ipfs/Qmd2frzgukznnSPVRf9CicvkFMzLTnSz8bH6VY5t9ASvNP",
		"/ip4/169.55.224.140/tcp/4001/ipfs/Qmd2frzgukznnSPVRf9CicvkFMzLTnSz8bH6VY5t9ASvNP",
		"/ip6/::1/tcp/4001/ipfs/Qmd2frzgukznnSPVRf9CicvkFMzLTnSz8bH6VY5t9ASvNP"
	],
	"AgentVersion": "go-libp2p/0.1.0",
	"ProtocolVersion": "ipfs/0.1.0"
}

In [ ]:
!mkdir data 2> /dev/null
ipfs.cmd("get -o ./data/ol_dump_authors_2016-08_24.txt QmVNsuKUVTSBSRyS71mwwvkpq58jAVLr491jLL4zQrC7XX")

Success

In [ ]:
ipfs.cmd("get -o ./data/ol_dump_works_2016-05-31.txt QmNYCJciFhmn1PN9JnbURvNyLf46oxJrraTrdedPswi9Aj")

Success

In [ ]:
ipfs.cmd("get -o ./data/jstart.jpg QmVfFbASG11MjUFuvfAJmjKpxRGYstCAw9GmZEpbA7KE7A")
Image("./data/jstart.jpg")

Success

In [ ]:
ipfs.cmd("add ./data/ol_dump_authors_2016-05-31.txt")
ipfs.cmd("add ./data/ol_dump_works_2016-05-31.txt")
ipfs.cmd("add ./data/jstart.jpg")
In [ ]:
ipfs.cmd("pin ls --type=recursive")
Congratulations. Your InterPlanetary Tour on Jupyter is now over. Hope you enjoyed the show!

InterPlanetary File System (IPFS) on Jupyter

Jovians and Bluemix users rejoice!


Food For Thought

  • IPFS is still very young. Alpha means YMMV, but the project is open and evolving.
  • Data Scientists need data. Can IPFS provide a more frictionless experience to accessing data?
  • Certain JDBC drivers (e.g. Xerial's SQLite, ...) assume/require database files on local paths. Can IPFS help data scientists work with these constrained drivers, either through download or via FUSE mount?
  • Can access to the IBM Analytics Exchange be streamlined by having the platform generate and provide permanent IPFS references?
  • How does IPFS decentralized technology compare to other similar open projects such as DAT, MaidSafe and Storj.io?
  • IPFS is a technology focused on distribution, not storage, of content. It is a peer-to-peer hypermedia protocol.
  • Can this be an easier way of sharing popular code snippets? Imagine running cat on a url and receiving a python code segment for using the Spark Technology Center's Stocator library in your data analytics projects. Rapid access to code examples. Maintaining a list of best of breed approaches via a list of links sounds convenient. Maybe this opens up opportunities for a trusted advisor chat bot that I populate with content link lists that I trust?
  • What use cases does IPFS make better? In what cases is P2P unwise or unnecessary?
  • How can IBM Cloud offerings differentiate themselves by embracing the momentum around distributed web and mesh networks?
  • Did you notice that some of these embedded images are being served via an IPFS resource url hosted @ ipfs.io?
  • Opportunities with real-time analytics, Spark streaming and the streaming capabilities of IPFS streams?
  • Opportunities for triggering "serverless" lambda functions (e.g. OpenWhisk, AWS Lambda, ...) via an IPFS client to emit events, thus causing parallel content movement programmatically (e.g. spin up lambda, fetch content and put it in a blob store)? Would this require new flavors of lambda functions -- compute optimized vs. bandwidth optimized?
  • What next ? ...

Useful References

Useful Tools

In [ ]: