Manfred Touron

What You Should Know About The History of P2P

Peer-to-peer sharing was a feature of the defunct ARPANET of 1969. As technology advanced, so did the government and entertainment industry giants’ efforts to suppress file-sharing.

However, P2P has survived well into the 21st century and it seems that the best is yet to come for the P2P community. Numerous new technologies are springing up and innovations and improvements are constantly being introduced.

Crash Course on the History of P2P

File sharing began back when the first computer networks were introduced. The ARPANET allowed users to send and receive files directly – that was back in 1969. One of the earliest transfer protocols was FTP (file transfer protocol). It was introduced in 1971.

In 1979, Usenet was born. It was primarily made for dial-up technology, but it made its way into the internet more than a decade later. Users could exchange files on bulletin boards. The video game Doom first became popular on bulletin boards in the early 1990s.

Two decades later in 1999, Napster was created, and with it, the modern era of modern P2P file sharing. Napster used a centralized indexing server, which would prove to be its downfall. Almost immediately after its introduction, Napster experienced a meteoric rise in popularity. By 2000, it had more than a million users. The next year, Metallica sued Napster and by the July of the same year, the service was shut down.

One year after Napster’s inception, Gnutella led a new wave. Unlike its predecessors, Gnutella was decentralized and allowed more people to use the platform at the same time. LimeWire is perhaps the most famous Gnutella client.

Current Technologies

Bittorrent

The next big step in the development of P2P file sharing happened in 2001 when Bram Cohen introduced Bittorrent. This platform is still in use today, one of the oldest and most widely used P2P protocols.

Bittorrent introduced a host of innovations. Users could search for files on online sites that contain trackers, while the file sharing happened directly between the users. This significantly improved transfer speeds. Additionally, Bittorrent clients would break a file into small fragments for multiple hosts, thus increasing the download speeds tremendously.

Blockchain

Bitcoin was introduced eight years after Bittorrent and it’s still in prevalent use today. Though it wasn’t designed for P2P file sharing, it brought about a new generation of P2P storage frameworks. It is based on blockchain.

Blockchain is so named for a constantly growing list of connected blocks. Each block or record contains data, a unique hash number, and the previous block’s hash. A blockchain is automatically updated every 10 minutes and uses a decentralized P2P network which anyone can join.

IPFS

IPFS (InterPlanetary File System) network and protocol were introduced in 2015. IPFS is the next step in P2P file sharing that works similarly to Bittorrent and other torrent protocols. Users can download as well as host content. There is no central server and each user has a small portion of a data package.

It is also similar to Blockchain in that it uses connected blocks protected with hash numbers. Also, the data within IPFS blocks can’t be easily manipulated without changing the block’s hash. However, IPFS does support file versioning.

Ether

Ether is another popular P2P sharing platform based on blockchain technology. It is somewhat similar to Bitcoin; Ethereum is the name of the cryptocurrency used on Ether network.

Ether was launched in 2014 as an open-source platform. You can use it to anonymously make transactions and share data with other users.

Similar to some other advanced Blockchain networks, Ether uses Smart Contracts. These are protocols designed to facilitate the execution of transactions by cutting out the middle man.

The Start of P2P

The initial vision of Tim Berners-Lee, regarded as the inventor of the World Wide Web, was for the internet to be similar to a P2P network. He envisioned the internet as a place where all users would and should be active content contributors and editors.

Its early precursor, the ARPANET, allowed two remote computers to send and receive data packets. However, it wasn’t a self-organized nor decentralized file-sharing system. Additionally, it didn’t support content and context-based routing.

Usenet addressed many of those issues, continuing and evolving the idea of a free internet.

The Continued Appeal of P2P

Nowadays, thanks to advanced technology, P2P networks can offer much more than content and context-based file searches. Some of the top reasons for using and improving P2P platforms today include:

  • Anonymity and privacy. P2P networks allow users to remain anonymous and protect their privacy on the network.
  • Cooperation and resource sharing. Many are drawn to P2P networks for the cooperation and sharing of resources.
  • Trust and accountability. Modern P2P networks are largely based on trust and the transactions have to be community approved.
  • Decentralization and lack of censorship. Today’s P2P networks are decentralized, thus preventing almost all forms of censorship. This ensures network neutrality.
  • Data integrity and encryption. Blockchain introduced hash numbers and proof-of-work. The latest innovations include encryption and smart contracts.

BitTorrent’s Struggle

The BitTorrent protocol remains popular even as almost two decades have passed since its introduction. It faced many adversities throughout the years in the form of more modern and advanced P2P platforms, poor business decisions on the part of its creator and his associates, and countless legal problems, even with the US government.

What kept BitTorrent alive all this time is the fact that it’s decentralized, easy to use, and built for easy transfers of huge amounts of data. Other than that, Facebook, Blizzard, and Twitter have openly admitted to using BitTorrent. Most importantly, the values of sharing and cooperation among BitTorrent users kept the flame burning through the dark times.

P2P Hall of Fame

Here’s a list of some of the most important people in the history of P2P sharing.

  • Tim Berners-Lee, inventor of the World Wide Web.
  • Sean Parker and Shawn Fanning, founders of Napster.
  • Bram Cohen, the mastermind behind the BitTorrent protocol.
  • Gottfrid Svartholm, Fredrik Neij, and Peter Sunde, creators of The Pirate Bay.
  • Satoshi Nakamoto, creator of Blockchain technology.

Final Words

P2P is starting to gain traction in the outside world. More and more people are adopting and incorporating the rules and ideas that govern P2P file-sharing technologies into their lives. This is especially true of self-organizing communities that sprung up in recent years.

Self-organizing communities share a number of values and principles characteristic of P2P technologies. They might be appealing to a wide range of individuals and groups, most notably those interested in cooperation and resource sharing, proponents of decentralization, and the occasional anarchistic souls.

Golang project layout

Context

In a previous blogpost, I talked about why I always trash the V1 of my projects. At Berty, we recently decided to start a clean V2, based on everything we learned from the V1.

What works well

  • having a monorepo
  • being protobuf-first and generating a lot of code
  • the codebase was “big-refactor”-friendly, including several refactors that modified 50+ files at once
  • we’ve learned a lot about:
    • our project, the features, the roadmap, the difficulties, etc
    • about our dependencies (IPFS, gomobile, react-native, BLE, etc.)

What needs to be improved

  • The code was too complex to read
  • The codebase was too complex to update safely
  • There were not enough rules about:
    • where to implement something, how to name things
    • how to implement things
  • Makefile rules, and CI can be improved
  • The tests should be more reliable
  • We need to learn more about our future protocol, for now, it’s only in our head, and we will undoubtedly fail to implement the v1 of the protocol, I prefer to fail fast!

Several blogposts, slides, repos, and videos later…

I passed the last three days reading blog posts, slides, repositories, and watching videos about what other people are doing right now.

Then, I looked back on Berty and my other projects and listed a set of rules I like the most.

New rules

As usual, a rule is something that can always have exceptions :)

  • Focus on readability, it’s a very good pattern to check what the godoc looks like to know if the API seems easy to adopt.
  • Avoid magic, no global vars, no func init()
  • Sharing logic / reusable business functionality is most of the time over-engineering
  • Enumerate requirements at function constructors. Use dependency injection (not dependency containers!), make go build your best friend; the logger should also be injected
  • If your project is small enough, put everything at the root of the project -> mono package
  • When you are creating a very powerful and complex library, it can be a good thing to make its little-sister library that will wrap the most permissive one in a light opinionated library
  • Embrace middlewares to lose coupling for timeout handling, retry mechanisms, authentication checks, etc
  • Reduce comments, focus on useful variable and function naming
    • Function and variable names are important to review
  • Limit the number of packages, the number of functions, the number of interfaces
    • Keep things simple and do not split into too many components at the beginning, split only because of a problem, not because of an anticipation
  • Try always to have a minimal indentation level
  • Use short function and variable names
    • Variables can even be one or two letters long (initials) when used near to their initialization
    • Receiver name should always be 1 or 2 letters long
  • Prefer synchronous functions to asynchronous ones, it’s easy to make an asynchronous wrapper over asynchronous function, not the opposite
  • Use named results (return) for documentation
  • Be flat, only use {cmd,pkg,internal}/<package-name>/<files>.go
  • Use pkg/ for packages you want other people to use, and internal/ for the code your implementation details; most of the code should start in internal/ before being moved to pkg/, only after you are sure it can be useful for someone else and after it becomes mature enough, so it has less risk of changing.
  • use feature-flags to configure the app, feature-flags are “documentation”! They also allow you to have (multiple) (unfinished) (long-running) experiments merged more quickly
  • Flags should be taken into account in this order: CLI > config > env
  • use a structured logger, bind it with std logger (https://github.com/go-kit/kit/tree/master/log#interact-with-stdlib-logger)
  • If your repo uses multiple main languages, they should be namespaced in their directory to make everything easier to manipulate for the tools.
  • Put your .proto files in an api/ directory, but you can configure them to generate files in your existing go packages.
  • Go routines
  • Package names should be:
    • the same as the directory name (always)
    • singular, lowercase, alpha-num
    • unique in your project; unique with go core packages too, if possible
  • Use -race when building and testing from the beginning
  • Context
    • context.Value is only used for request-scoped information and only when it can’t be passed in another way
    • Do not hesitate to pass context.Context as the first var of most of your functions (I need to investigate more and have a more strict rule here)
  • Always put a doc.go file in the pkg/* packages to configure the package vanity URLs and put some documentation. When your package has multiple go files, it will be easier to know where to edit those things
  • Avoid having too many interfaces, and when doing some, try to always declare them in the caller package, not the implementer one
  • Tests
    • go test should always work after a fresh clone! If you have unreliable/specific tests, use flags, env vars
    • The tests should be easily readable and explaining, it’s probably the best place to “document” the edge cases of your library
    • Use table-driven tests a lot
    • If you are manipulating test-fixtures often, you can add a test -update flag
    • If you write mock, they should be implemented in the same package than the real implementation, in a testing.go file; a mock should, in general, return a fully started in-memory server.
    • If you need to write tests at runtime, you can use http://github.com/mitchellh/go-testing-interface
    • If you have a complex struct, i.e., a server, do not hesitate to add a Test bool field that configures it to be testing-friendly
    • When testing complex structs, compare a string representation (JSON, or something like that)
    • Only test exported functions; unexported functions are implementation details
    • If you write helpers, they should not return an error but take testing.T as an argument and call t.Fail directly
  • Most of the rules defined here can be skipped entirely in the internal/ directory. This directory is the perfect place for things that changes often.
  • Add a githook that run goimports.

Unanswered questions

  • When is it better to have a ListsUser(query) instead of ListAllUsers() + ListUsersByGroup() + ListActiveUsers()...?
  • What the best way of organizing code that involves multiple languages, i.e., bridges?
  • When does it makes sense to have an iface package?
  • When does it make sense to have a model package vs. a user package?

Suggested project layout for the monorepo of a big project

* api/
  * a.proto
  * a.swagger.json (generated)
  * b.proto
  * b.swagger.json (generated)
* assets/
  * logo.png
* build/
  * ci/
    * script.sh
  * package/
    * script.sh
* configs/
  * prod.json
  * dev.json
* deployments/
  * c/
    * docker-compose.yml
  * d/
    * docker-compose.yml
* docs/
  * files.md
* examples/
  * descriptive-dirname/
    * ...
* githooks/
  * pre-commit
* go/
  * cmd/
    * mybinary/
      * main.go
  * internal
    * e/
      * doc.go
      * e.go
    * f/
      * doc.go
      * f.go
  * pkg/
    * g/
      * doc.go
      * g.go
    * h/
      * doc.go
      * h.go
  * Makefile
  * go.mo
* js/
* test/
  * testdata/
    * blob.json
* tools/
  * docker-protoc/
    * Dockerfile
  * script.sh
* Makefile
* Dockerfile
  • https://blog.chewxy.com/2018/03/18/golang-interfaces/

    I however also run into cases where I end up accidentally writing Java-style interfaces - typically after I come back from a stint of writing code in Python or Java. The desire to overengineer and “class all the things” something is quite strong, especially when writing Go code after writing a lot of object-oriented code.

  • http://www.brendangregg.com/usemethod.html

  • https://en.m.wikipedia.org/wiki/Extreme_programming

  • https://github.com/golang-standards/project-layout

  • https://github.com/golang/go/wiki/CodeReviewComments

  • https://github.com/mishudark/eventhus

  • https://medium.com/@benbjohnson/standard-package-layout-7cdbc8391fc1

  • https://medium.com/@rdsubhas/10-modern-software-engineering-mistakes-bc67fbef4fc8

    TL;DR — The House (Business) Always Wins – In my 15-year involvement with coding, I have never seen a single business “converge” on requirements. They only diverge. It is simply the nature of business and its not the business people’s fault.

    TL;DR - Duplication is better than the wrong abstraction - Designs are always playing catch up to changing real-world requirements. So even if we found a perfect abstraction by a miracle, it comes tagged with an expiry date because #1 — The House wins in the end. The best quality of a Design today is how well it can be undesigned. There is an amazing article on write code that is easy to delete, not easy to extend.

    TL;DR — Wrappers are an exception, not the norm. Don’t wrap good libraries for the sake of wrapping.

    TL;DR — Don’t let <X>-ities go unchallenged. Clearly define and evaluate the Scenario/Story/Need/Usage. Tip: Ask a simple question — “What’s an example story/scenario?” — And then dig deep on that scenario. This exposes flaws in most <X>-ities.

  • https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html

  • https://peter.bourgon.org/blog/2017/06/09/theory-of-modern-go.html

  • https://peter.bourgon.org/go-for-industrial-programming/

    Industrial programming means writing code once and maintaining it into perpetuity. Maintenance is the continuous practice of reading and refactoring. Therefore, industrial programming overwhelmingly favors reads, and on the spectrum of easy to read vs. easy to write, we should bias strongly towards the former.

    Looking at interfaces as a way to classify implementations is the wrong approach; instead, look at interfaces as a way to identify code that expects common sets of behaviors.

  • https://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to

    Instead of making code easy-to-delete, we are trying to keep the hard-to-delete parts as far away as possible from the easy-to-delete parts.

    Write more boilerplate. You are writing more lines of code, but you are writing those lines of code in the easy-to-delete parts.

    I’m not advocating you go out and create a /protocol/ and a /policy/ directory, but you do want to try and keep your util directory free of business logic, and build simpler-to-use libraries on top of simpler-to-implement ones. You don’t have to finish writing one library to start writing another atop.

    Layering is less about writing code we can delete later, but making the hard to remove code pleasant to use (without contaminating it with business logic).

    You’ve copy-pasted, you’ve refactored, you’ve layered, you’ve composed, but the code still has to do something at the end of the day. Sometimes it’s best just to give up and write a substantial amount of trashy code to hold the rest together.

    Business logic is code characterized by a never-ending series of edge cases and quick and dirty hacks. This is fine. I am ok with this. Other styles like ‘game code’, or ‘founder code’ are the same thing: cutting corners to save a considerable amount of time.

    The reason? Sometimes it’s easier to delete one big mistake than try to delete 18 smaller interleaved mistakes. A lot of programming is exploratory, and it’s quicker to get it wrong a few times and iterate than think to get it right first time.

    the whole step 5 is <3

    I’m not suggesting you write the same ball of mud ten times over, perfecting your mistakes. To quote Perlis: “Everything should be built top-down, except the first time”. You should be trying to make new mistakes each time, take new risks, and slowly build up through iteration.

    Instead of breaking code into parts with common functionality, we break code apart by what it does not share with the rest. We isolate the most frustrating parts to write, maintain, or delete away from each other.; We are not building modules around being able to re-use them, but being able to change them.

    When a module does two things, it is usually because changing one part requires changing the other. It is often easier to have one awful component with a simple interface, than two components requiring a careful co-ordination between them.

    The strategies I’ve talked about — layering, isolation, common interfaces, composition — are not about writing good software, but how to build software that can change over time.

  • https://rakyll.org/style-packages/

  • https://www.martinfowler.com/articles/designDead.html

  • https://youtu.be/ZsHMHukIlJY

    A common fallacy is to assume authors of incomprehensible code will somehow be able to express themselves lucidly and clearly in comments.

moul's ascii-art logo (draft)

Playing with ascii-art!

I’m currently trying to make an ascii-art version of my avatar, so I can use it in my source code :)


  
+--------------------------------------------------------------+
| + + + ░░░░░░░░░░░░░░░░░░░░  Hello  ░░░░░░░░░░░░░░░░░░░░░░░░░░|
+--------------------------------------------------------------+
|                                                              |
|                                                              |
|     ++           Lorem ipsum dolor sit amet, consectetur     |
|     ++++         adipiscing elit, sed do eiusmod tempor      |
|      +++         incididunt ut labore et dolore magna        |
|   ++++++++++     aliqua. Ut enim ad minim veniam.            |
|  +++       |                                                 |
|  ++         |    Exercitation ullamco laboris nisi ut        |
|  +  -==   ==|    aliquip ex ea commodo consequat. Duis aute  |
| (   <+>   <+>    irure dolor in reprehenderit in voluptate   |
|  |           |   velit esse cillum dolore eu fugiat nulla    |
|  |         __|   pariatur. Excepteur sint occaecat cupidatat |
|  |      +++      non proident, sunt in culpa qui officia     |
|   \      =+      deserunt mollit anim id est laborum.        |
|    \      +                                  _ _  __    /    |
|     \++++++                                 / / //_//_//     |
|       ++++                                                   |
|                                                              |
|                                                              |
+--------------------------------------------------------------+


           ++                ++               ___                 ++
           ++++              ++++             \  \                ++++
            +++               +++             _\__\__              +++
         ++++++++++        ++++++++++        / ______\          ++++++++++
        +++    \   :      +++       |       / /       \        +++       )
        ++      \  :      ++         |      |/         \       ++         )
        + .===   ==:      +  -==   ==|      |  ---   ---\      +  -==   ==(
       (   <+>   <+ .    (   <+>   <+>     (   (o)   (o)|     (   <+>   <+>)
        .           :     |           |     |           |     (           (
        :   /     ._.     |         __|     |      _____|      )         __)
        :  /  .===        |      +++        |       _|         (      +++
         \      --         \      =+         \       |          (      =+
          \      +          \      +          \______|           (      +
           -++++++           \++++++           \     |            (++++++
             ++++              ++++             \____|              ++++


          ___                                                      ++
          \  \             ___                   ++                ++++
          _\__\__          \  \__               ++++                +++
         / ______\         /_____\               +++             ++++++++++
        / /       \       //      \           ++++++++++        +++       |
        |/         \     //        \         +++    \   +       ++         |
        |  ---   ---\   /   ---   --\        ++      \  |       +  -==   ==|
       (   (o)   (o)|   (_  (.)   (.)\       + .===   ==|      (   <+>   <+>
        |           |    \            \     (   <+>   <+}\      |           |
        |       ____|     \        ___/      .           |      |         __|
        |       -=         \     --\         |   /    .__|      |      +++
         \       |          \       \        |  /  .+++          \      =+
          \______|           \______/         \      --           \      +
           \     |                             \      +           |\++++++
            \____|                              \++++++           |  ++++
                                                  ++++         ___|   |___
                                                              /    ---    \
                                                             /  |       |  \


                                              ______                        __
             ____                 _          / ____ \____ ___  ____  __  __/ /
            / __ \ _ __  ___ _  _| |        / / __ `/ __ `__ \/ __ \/ / / / /
           / / _` | '  \/ _ \ || | |       / / /_/ / / / / / / /_/ / /_/ / /
           \ \__,_|_|_|_\___/\_,_|_|       \ \__,_/_/ /_/ /_/\____/\__,_/_/
            \____/                          \____/


                    _     ,----.                          ,--.
      _____ ___ _ _| |   '   ,  |,--,--,--. ,---. ,--.,--.|  |      _  _    |
     |     | . | | | |   |  |   /|        || .-. ||  ||  ||  |     |||(_)|_||
     |_|_|_|___|___|_|   '  '--'||  |  |  |' '-' ''  ''  '|  |
                          `----' `--`--`--' `---'  `----' `--'

                       .__                                                 _
    _____   ____  __ __|  |          __                                   | |
   /     \ /  _ \|  |  \  |    |\/| /  \ |  | |      _ __ ___   ___  _   _| |
  |  Y Y  (  <_> )  |  /  |__  |  | \__/ \__/ |___  | '_ ` _ \ / _ \| | | | |
  |__|_|  /\____/|____/|____/                       | | | | | | (_) | |_| | |
        \/                                          |_| |_| |_|\___/ \__,_|_|

    _____                  _ .d8888888b.                                  888
   / ___ \__ _  ___  __ __/ d88P"   "Y88b                                 888
  / / _ `/  ' \/ _ \/ // / /888  d8b  888                                 888
  \ \_,_/_/_/_/\___/\_,_/_/ 888  888  888 88888b.d88b.   .d88b.  888  888 888
   \___/                    888  888bd88P 888 "888 "88b d88""88b 888  888 888
                            888  Y8888P"  888  888  888 888  888 888  888 888
                            Y88b.     .d8 888  888  888 Y88..88P Y88b 888 888
                             "Y88888888P" 888  888  888  "Y88P"   "Y88888 888

  • ▌ ▄ ·.       ▄• ▄▌▄▄▌
  ·██ ▐███▪▪     █▪██▌██•                              __
  ▐█ ▌▐▌▐█· ▄█▀▄ █▌▐█▌██▪       ____ ___  ____  __  __/ /                  _
  ██ ██▌▐█▌▐█▌.▐▌▐█▄█▌▐█▌▐▌    / __ `__ \/ __ \/ / / / /    _ __  ___ _  _| |
  ▀▀  █▪▀▀▀ ▀█▄▀▪ ▀▀▀ .▀▀▀    / / / / / / /_/ / /_/ / /    | '  \/ _ \ || | |
                             /_/ /_/ /_/\____/\__,_/_/     |_|_|_\___/\_,_|_|



                                 _      __    __  ______  __  __  __
   ____                         | |    /\ "-./  \/\  __ \/\ \/\ \/\ \
  / __,\  _  _  _    __         | |    \ \ \-./\ \ \ \/\ \ \ \_\ \ \ \____
 | /  | |/ |/ |/ |  /  \_|   |  |/      \ \_\ \ \_\ \_____\ \_____\ \_____\
 | \_/|/   |  |  |_/\__/  \_/|_/|__/     \/_/  \/_/\/_____/\/_____/\/_____/
  \____/

                             ██████╗ ███╗   ███╗ ██████╗ ██╗   ██╗██╗
                     __     ██╔═══██╗████╗ ████║██╔═══██╗██║   ██║██║
    __ _  ___  __ __/ /     ██║██╗██║██╔████╔██║██║   ██║██║   ██║██║
   /  ' \/ _ \/ // / /      ██║██║██║██║╚██╔╝██║██║   ██║██║   ██║██║
  /_/_/_/\___/\_,_/_/       ╚█║████╔╝██║ ╚═╝ ██║╚██████╔╝╚██████╔╝███████╗
                             ╚╝╚═══╝ ╚═╝     ╚═╝ ╚═════╝  ╚═════╝ ╚══════╝


                  ███▄ ▄███▓ ▒█████   █    ██  ██▓
                 ▓██▒▀█▀ ██▒▒██▒  ██▒ ██  ▓██▒▓██▒
                 ▓██    ▓██░▒██░  ██▒▓██  ▒██░▒██░
                 ▒██    ▒██ ▒██   ██░▓▓█  ░██░▒██░
                 ▒██▒   ░██▒░ ████▓▒░▒▒█████▓ ░██████▒
                 ░ ▒░   ░  ░░ ▒░▒░▒░ ░▒▓▒ ▒ ▒ ░ ▒░▓  ░
                 ░  ░      ░  ░ ▒ ▒░ ░░▒░ ░ ░ ░ ░ ▒  ░
                 ░      ░   ░ ░ ░ ▒   ░░░ ░ ░   ░ ░
                        ░       ░ ░     ░         ░  ░

I started a golang library to use this logo -> https://github.com/moul/asciimoul

Comments V1

Initial version of my comment system based on GitHub Issues.

I’ve ideas for a V2 :)