Manfred Touron

Code

50 pages about "Code"

Challenges of Open-Source

Can I – and should I – contribute to this project?

Most projects will be happy to accept any kind of contribution. If, for any reason, the mainteiners don’t want contributions, they can write in README.md or CONTRIBUTING.md files. Though it’s quite an unlikely scenario that people wouldn’t want any contribs to their public repositories – I’ve never seen that.

If they don’t want to maintain a project and don’t want any activity on it, the author can simply archive the repo to make it read-only. For example: https://github.com/boltdb/bolt. In this case, you can clone their repo and continue developing your own branch.

If their CONTRIBUTING.md file says that they dont like contributions, you will have a message the first time you open a PR, saying “you should take a look at the contributing.md file”. But even on repos with strict contribution policies, you wont hurt anyone by opening a PR – they might just ignore it, but they also might reply with a nice message, or let a bot reply for them :)

However, even though people generally don’t refuse contributions, you may stumble upon a project that is not very contribution-friendly. For example:

  • Maintainers are too proud of themselves and will actually refuse to admit that you helped them, saying: “Yes, I managed to do that, you know, I’m so much smarter than you”.
  • People that won’t be respectful of your contribution (sometimes knowingly, but most of the time because the are beginners on GitHub), and will just take your idea and recode it or find a way of pushing your edit with their name as an author.
  • People that are too perfectionist of arrogant; they will accept your help but ask you to be as perfect as they are, “Thank you for your idea (that helps my project a lot), can you please: add a lot of docs, a lot of tests, a lot of …, because I/we have very high quality standard, you know”.

In most cases, those behaviors change over time to become more open and respectful. But if you want to avoid things like that in the first place, you can check the history of the individual or organization that created the project, to see how they act.

Good signs to look for

First of all, check the other contributions to this project – it’s easy to view other accepted/refused contributions, analyze what was done by the contributor, analyze how open was the maintainer to accept the contribution, how long it took to accept or reject it.

If the current repo doesn’t have much activity, you can check the other repos from the same author/organization. One of the main concepts of open source is transparency. GitHub will help you get most of this information easy, you can even have more info by using alternative tools (git log, external websites, etc).

Indicators that a repository is contribution-friendly:

  • Project is “fresh” – the repo has some recent commits, author has recent GitHub activity.
  • Readme or Contributing files have some kind of a “contributions are welcome” message.
  • Website has a ribbon “fork me on GitHub”, or a messgae “See a typo? Propose a change on GitHub”.
  • The project is well-documented so that other developers can easily run it.
  • The author has some “open source lover” in his/her bio.

Browsing for projects

If you don’t have a specific repo in mind and just browse to find a project to contribute to, GitHub has a few standardised ways of explicitly asking for help:

GitHub’s official “help wanted” issue/pr label

This label is a way for authors to openly ask help from the community. Sometimes it will be for a feature idea they have in mind, sometimes for bug fixing, sometimes for feedback/reviewing. GitHub will handle those labels in a custom way, a repo could say “hey we have 3 help wanted issues”. Over 600k issues on the site have this label.

You can also check for a particular author.

GitHub’s official “good first issue” issue/pr label

More info about using labels on your projects .

This label is similar to “help wanted”б but with a specific flavor: available for beginners :) This label will also make GitHub react differently by making the information more visible to you in different places.

“Good first issue” issue something that considered relatively easy by the maintainer, but something he’d greatly appreciate some help with.

My own repos also have issues with this label ;)

GitHub’s unofficial “help wanted” project topic

Project topics were added recently, most of the time you use them to indicate the main areas of a project: “privacy”, “container”, “web”; but some folks are also using some project topics to ask for assistance: * https://github.com/topics/help-wanted * https://github.com/topics/maintainer-wanted * https://github.com/topics/looking-for-maintainer * https://github.com/topics/needs-maintainer

If you’re still not sure if your contribution would be relevant, you can open an issue, saying something like: “Hello, I like your project and was planning to contribute, what do you think of my contribution idea: … ….”

Taking a leap

If you’re hesitant about making your first contribution, the best thing to do is to just go for it. Getting the hang of GitHub might take a bit of time, but and the fastest way to do it is dive in and be active.

Besides, not all contributions require coding. Something like typo fixing might not be very glamorous, but you will see that your contribution is accepted, and that you actually took part in a project to make it a little bit better – it’s an inspiring feeling.

Another example of an easy contribution is this: if, while installing the project, you had some issues that you eventually solved, it would be very appreciated if you update the installation instructions.

If you make a contribution that is rejected, but you still think it should be there, another part of the open source is the right to modify! Just fork the project; read the original license and explain in the readme why you fork the project, how different it will be, etc. Most of the time, the original author that rejected the contribution will be happy to let you link your fork in the original project’s readme or at least in the rejected pull request.

The “worst” that can could happen is that you lose some time on a contribution that is rejected. It’s a bit disappointing, but not life-shattering. And, most importantly, the things that you have learned while working on the contribution would stay with you, no matter if it’s accepted or not.

Fear of exposing your code

Some people are reluctant to publicly share their code because they are concerned with the potential reaction of the future employers or colleagues.

Here, people are assuming that their code isn’t great, or that they aren’t using some cool libraries, etc; and that they would be shunned for it. But most of the time, the code is ok, and people aren’t going to judge anyway. Even if some potential employers were indeed to dismiss your candidacy just because they’re picky about the projects you contribute to or libraries that you use, you’re probably better off not working for them anyway.

And if your code were truly awful, your PRs wouldn’t get accepted and won’t show on your profile in the first place – so, if your contributions are accepted, it means they are ok to be viewed :)

README and documentation – best practices

More often than not, the concern about making a project open-source-ready is more about “checking we don’t let out the code we aren’t proud of” than actually “making the project easy to contribute”.

At a bare minimum, you can just open your project with no changes. But something that is quite helpful and not too time -consuming is creating a good README. This file will be read many more times than any other doc.

README.md should allow someone to quickly discover what your project is about, how to install and use it, which features to look up to.

Readme is an index where you could put all the important and relevant things:

  • Wiki
  • Demo
  • Documentation
  • Installation instructions
  • Embed a screenshot/screencast
  • Changelog (if it’s important enough, you can even place it directly into the Readme)
  • If you use npm, link to npm; if you use GitHub releases, link to the release page

You can specify some things about your approach and philosophy, for example:

  • “This is a side project that I do not plan to maintain, but feel free to fork it or open PRs”
  • “This project was made with a specific goal in mind, sorry but I won’t accept big design changes”
  • “Philosophy of this project: stay simple”

It’s a good idea to explicitly write that all contributions and feedback are appreciated, and people are welcome to open issues if they want to share their ideas.

If the project has a planned future, link to the corresponding issues if you want to receive feedback and ideas on them, or to and external public Trello with the issues.

If you are very cool, you can link to the projects that are the alternatives of yours, and explain what the differences are.

Unfinished README or project

Something you can often use in an open source project is the “TODO” keyword. If you currently don’t have time or inspiration to write a certain Readme section properly, you can add the “Installation” title and whire TODO under it. It will show that you have it in mind for the future and didn’t just forget about including it, and will also show other users that you would really apprecite if someone helps you complete the section.

If your project isn’t finished, it’s not a problem, just write it in the READMe – some people will bookmark the project for later, because they are interested to know how it will evolve over time.

Creating the project documentation

Write the initial doc for yourself, something you would regret not having if you open the project in two years. What are the most important things that you would need to know?

Then, when you have more time and energy, you can expand on this crucial doc. Documentation will also expand as you get explicit or implicit feedback from your first users.

No time for maintaining a project?

Not having time/energy to maintain a project can be a concern – or, rather, an excuse – that prevents from publishing an OSS project.

It’s actually rare that a project takes an overwhelming amount of time. For one, your new project will probably unused or used by just a few people for a long time. And as it grows and gets more users, it’ll be easier to find people who want to maintain it.

Being inactive is not a dealbreaker, especially if you clearly describe the situation.

  • In your project’s README, say something like: “Sorry, I don’t have much time to maintain it”.
  • Add the “help wanted” issue label to get more active help.
  • Set up a Patreon and define a way of receiving cash if someone needs something specific in a short amount of time.
  • If you’re not planning to maintain it at all, suggest your readers to check out some similar projects (bonus points if you give links to some options), or to contact you if they want to take the repo over.

Worst thing to do is to make your project unavailable (delete it) – this may break others’ software. Rather, put the repository into read-only mode (archive it), so that people can use it without any support if they want to, or fork it and continue developing their branch.

More thoughts on time management and OSS

  • Some companies give some time to work on open source (1 day per week)
  • Some people prefer to work part-time (4 days for their employers) and 1 day for themselves by doing open source
  • You can consider open source as an activity you can do on weekends, in the evening

Most of the time, if you remove the constraints of work (forced roadmap, forced topic to work on, delivery pressure etc), you would be coding for pleasure and fun, and you’ll be motivated to do it, just like you do any other thing you enjoy.

You can read my article on the subject, Coding useless stuff, or check out my video from the Paris P2P festival.

SEO and promotion

“SEO” sounds like something evil and corporate, but “search engine optimization” is often used for good. When you type a problem and immediately see a Stackoverflow link discussing and solving this exact problem, it doesn’t feel like you’re being tricked by marketing. And this match occured due to the power of SEO.

People who look for specific things

Project title and description are very important – not only for SEO, but also because they show on a preview when a link is shared.

The next in importance is README. Think what kind of people will need this project, what kind of queries they put into the GitHub search field, and add some introductory paragraphs with new keywords. Of course, keep the text coherent and use the words and their combinations that are most relevant.

People who are just browsing

Use project topics to add the technical ecosystems and also usage ecosystems. You will be listed for people that want to list “everything related to this domain”.

You can also list your project on awesome lists – there are plenty, it’s easy to add your project to a list with a small description that can help people interested in a topic to find it easily.

Register your project in project databases: npm, homebrew, etc. can be considered something like project databases.

More tips for being discovered

List your project in a README of a related project. For example, if you create an SDK for another company, you can be listed in their doc and their repos. It could even be an alternative/competitor project, and your project could go into the “Alternatives” section of their readme file.

Protomote it on Twitter, in community chats. Most of the languages have a Slack or equivalent where you can post your project written in this language.

Make open source contributions to other projects using your own project. If your project is a library that addresses something, you can open pull requests on other projects to add yours as a dependency: “Hello, this PR uses my project, which gives you such and such advantages”.

Answer people’s question on StackOverflow, GitHub issues, etc, and link to your project when you think it can be relevant to the answer. People would click on the link to analyze your project, and then, if your readme is understandable enough, they will point to it by themselves to help other people.

Keep in mind that, in most cases, project recognition takes time. It usually rises in peaks, and sometimes you need to wait until someone famous tweets about it :)

Once again, any kind of search optimization can be done later – you can release your barebones project early, and add the docs, SEO, etc. later.

Is it possible to make money from OSS?

Some ways open source can improve your income, directly or indirectly:

  • You can make OSS for your company and get paid indirectly for doing open source
  • You can consider OSS as your best CV, a proof of your skills and work ethics, and lend good jobs thanks to it
  • Setting up Patreon and GitHub sponsors
  • Taking bug bounties and be paid for doing stuff others asked you
  • You may receive prizes/rewards (as I do from Google), kind of donation
  • Your OSS project can help with or be a part of another project that brings you money

A lot of good brains are acutlaly working on new models to allow people to live from open source. People are looking into ways to democratize bug bounties, etc, in order to allow some companies to have no full-time employees, only the external contributors that are well-ranked thanks to their contributions.

How to Join and Contribute to Open Source Communities (presentation)

Many developers have misconceptions about open source collaboration. Being a bit shy and afraid to make a mistake is a frequent reason for not contributing, and so is just not knowing about how to start. This talk will share lessons learned about open collaboration, both for project leads and independent contributors.

A talk I gave at Paris P2P Festival #0.


Topics:

  • Why I love Open-Source, and should you
  • How to be a good contributor
  • How to open your projects

Video


Slides

Loading...

Link to the presentation

Download slides

P2P & Crypto in Go (presentation)

A talk I gave at Golang Paris.


Topics:

  • P2P
    • General introduction
    • P2P & Go
  • Cryptography
    • General Introduction
    • Cryptography & Go
  • Berty
    • Project introduction
    • Berty & Go
  • Paris P2P

Loading...

Link to the presentation

Golang project layout

Context

In a previous blogpost, I talked about why I always trash the V1 of my projects. At Berty, we recently decided to start a clean V2, based on everything we learned from the V1.

What works well

  • having a monorepo
  • being protobuf-first and generating a lot of code
  • the codebase was “big-refactor”-friendly, including several refactors that modified 50+ files at once
  • we’ve learned a lot about:
    • our project, the features, the roadmap, the difficulties, etc
    • about our dependencies (IPFS, gomobile, react-native, BLE, etc.)

What needs to be improved

  • The code was too complex to read
  • The codebase was too complex to update safely
  • There were not enough rules about:
    • where to implement something, how to name things
    • how to implement things
  • Makefile rules, and CI can be improved
  • The tests should be more reliable
  • We need to learn more about our future protocol, for now, it’s only in our head, and we will undoubtedly fail to implement the v1 of the protocol, I prefer to fail fast!

Several blogposts, slides, repos, and videos later…

I passed the last three days reading blog posts, slides, repositories, and watching videos about what other people are doing right now.

Then, I looked back on Berty and my other projects and listed a set of rules I like the most.

New rules

As usual, a rule is something that can always have exceptions :)

  • Focus on readability, it’s a very good pattern to check what the godoc looks like to know if the API seems easy to adopt.
  • Avoid magic, no global vars, no func init()
  • Sharing logic / reusable business functionality is most of the time over-engineering
  • Enumerate requirements at function constructors. Use dependency injection (not dependency containers!), make go build your best friend; the logger should also be injected
  • If your project is small enough, put everything at the root of the project -> mono package
  • When you are creating a very powerful and complex library, it can be a good thing to make its little-sister library that will wrap the most permissive one in a light opinionated library
  • Embrace middlewares to lose coupling for timeout handling, retry mechanisms, authentication checks, etc
  • Reduce comments, focus on useful variable and function naming
    • Function and variable names are important to review
  • Limit the number of packages, the number of functions, the number of interfaces
    • Keep things simple and do not split into too many components at the beginning, split only because of a problem, not because of an anticipation
  • Try always to have a minimal indentation level
  • Use short function and variable names
    • Variables can even be one or two letters long (initials) when used near to their initialization
    • Receiver name should always be 1 or 2 letters long
  • Prefer synchronous functions to asynchronous ones, it’s easy to make an asynchronous wrapper over asynchronous function, not the opposite
  • Use named results (return) for documentation
  • Be flat, only use {cmd,pkg,internal}/<package-name>/<files>.go
  • Use pkg/ for packages you want other people to use, and internal/ for the code your implementation details; most of the code should start in internal/ before being moved to pkg/, only after you are sure it can be useful for someone else and after it becomes mature enough, so it has less risk of changing.
  • use feature-flags to configure the app, feature-flags are “documentation”! They also allow you to have (multiple) (unfinished) (long-running) experiments merged more quickly
  • Flags should be taken into account in this order: CLI > config > env
  • use a structured logger, bind it with std logger (https://github.com/go-kit/kit/tree/master/log#interact-with-stdlib-logger)
  • If your repo uses multiple main languages, they should be namespaced in their directory to make everything easier to manipulate for the tools.
  • Put your .proto files in an api/ directory, but you can configure them to generate files in your existing go packages.
  • Go routines
  • Package names should be:
    • the same as the directory name (always)
    • singular, lowercase, alpha-num
    • unique in your project; unique with go core packages too, if possible
  • Use -race when building and testing from the beginning
  • Context
    • context.Value is only used for request-scoped information and only when it can’t be passed in another way
    • Do not hesitate to pass context.Context as the first var of most of your functions (I need to investigate more and have a more strict rule here)
  • Always put a doc.go file in the pkg/* packages to configure the package vanity URLs and put some documentation. When your package has multiple go files, it will be easier to know where to edit those things
  • Avoid having too many interfaces, and when doing some, try to always declare them in the caller package, not the implementer one
  • Tests
    • go test should always work after a fresh clone! If you have unreliable/specific tests, use flags, env vars
    • The tests should be easily readable and explaining, it’s probably the best place to “document” the edge cases of your library
    • Use table-driven tests a lot
    • If you are manipulating test-fixtures often, you can add a test -update flag
    • If you write mock, they should be implemented in the same package than the real implementation, in a testing.go file; a mock should, in general, return a fully started in-memory server.
    • If you need to write tests at runtime, you can use http://github.com/mitchellh/go-testing-interface
    • If you have a complex struct, i.e., a server, do not hesitate to add a Test bool field that configures it to be testing-friendly
    • When testing complex structs, compare a string representation (JSON, or something like that)
    • Only test exported functions; unexported functions are implementation details
    • If you write helpers, they should not return an error but take testing.T as an argument and call t.Fail directly
  • Most of the rules defined here can be skipped entirely in the internal/ directory. This directory is the perfect place for things that changes often.
  • Add a githook that run goimports.

Unanswered questions

  • When is it better to have a ListsUser(query) instead of ListAllUsers() + ListUsersByGroup() + ListActiveUsers()...?
  • What the best way of organizing code that involves multiple languages, i.e., bridges?
  • When does it makes sense to have an iface package?
  • When does it make sense to have a model package vs. a user package?

Suggested project layout for the monorepo of a big project

* api/
  * a.proto
  * a.swagger.json (generated)
  * b.proto
  * b.swagger.json (generated)
* assets/
  * logo.png
* build/
  * ci/
    * script.sh
  * package/
    * script.sh
* configs/
  * prod.json
  * dev.json
* deployments/
  * c/
    * docker-compose.yml
  * d/
    * docker-compose.yml
* docs/
  * files.md
* examples/
  * descriptive-dirname/
    * ...
* githooks/
  * pre-commit
* go/
  * cmd/
    * mybinary/
      * main.go
  * internal
    * e/
      * doc.go
      * e.go
    * f/
      * doc.go
      * f.go
  * pkg/
    * g/
      * doc.go
      * g.go
    * h/
      * doc.go
      * h.go
  * Makefile
  * go.mo
* js/
* test/
  * testdata/
    * blob.json
* tools/
  * docker-protoc/
    * Dockerfile
  * script.sh
* Makefile
* Dockerfile
  • https://blog.chewxy.com/2018/03/18/golang-interfaces/

    I however also run into cases where I end up accidentally writing Java-style interfaces - typically after I come back from a stint of writing code in Python or Java. The desire to overengineer and “class all the things” something is quite strong, especially when writing Go code after writing a lot of object-oriented code.

  • http://www.brendangregg.com/usemethod.html

  • https://en.m.wikipedia.org/wiki/Extreme_programming

  • https://github.com/golang-standards/project-layout

  • https://github.com/golang/go/wiki/CodeReviewComments

  • https://github.com/mishudark/eventhus

  • https://medium.com/@benbjohnson/standard-package-layout-7cdbc8391fc1

  • https://medium.com/@rdsubhas/10-modern-software-engineering-mistakes-bc67fbef4fc8

    TL;DR — The House (Business) Always Wins – In my 15-year involvement with coding, I have never seen a single business “converge” on requirements. They only diverge. It is simply the nature of business and its not the business people’s fault.

    TL;DR - Duplication is better than the wrong abstraction - Designs are always playing catch up to changing real-world requirements. So even if we found a perfect abstraction by a miracle, it comes tagged with an expiry date because #1 — The House wins in the end. The best quality of a Design today is how well it can be undesigned. There is an amazing article on write code that is easy to delete, not easy to extend.

    TL;DR — Wrappers are an exception, not the norm. Don’t wrap good libraries for the sake of wrapping.

    TL;DR — Don’t let <X>-ities go unchallenged. Clearly define and evaluate the Scenario/Story/Need/Usage. Tip: Ask a simple question — “What’s an example story/scenario?” — And then dig deep on that scenario. This exposes flaws in most <X>-ities.

  • https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html

  • https://peter.bourgon.org/blog/2017/06/09/theory-of-modern-go.html

  • https://peter.bourgon.org/go-for-industrial-programming/

    Industrial programming means writing code once and maintaining it into perpetuity. Maintenance is the continuous practice of reading and refactoring. Therefore, industrial programming overwhelmingly favors reads, and on the spectrum of easy to read vs. easy to write, we should bias strongly towards the former.

    Looking at interfaces as a way to classify implementations is the wrong approach; instead, look at interfaces as a way to identify code that expects common sets of behaviors.

  • https://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to

    Instead of making code easy-to-delete, we are trying to keep the hard-to-delete parts as far away as possible from the easy-to-delete parts.

    Write more boilerplate. You are writing more lines of code, but you are writing those lines of code in the easy-to-delete parts.

    I’m not advocating you go out and create a /protocol/ and a /policy/ directory, but you do want to try and keep your util directory free of business logic, and build simpler-to-use libraries on top of simpler-to-implement ones. You don’t have to finish writing one library to start writing another atop.

    Layering is less about writing code we can delete later, but making the hard to remove code pleasant to use (without contaminating it with business logic).

    You’ve copy-pasted, you’ve refactored, you’ve layered, you’ve composed, but the code still has to do something at the end of the day. Sometimes it’s best just to give up and write a substantial amount of trashy code to hold the rest together.

    Business logic is code characterized by a never-ending series of edge cases and quick and dirty hacks. This is fine. I am ok with this. Other styles like ‘game code’, or ‘founder code’ are the same thing: cutting corners to save a considerable amount of time.

    The reason? Sometimes it’s easier to delete one big mistake than try to delete 18 smaller interleaved mistakes. A lot of programming is exploratory, and it’s quicker to get it wrong a few times and iterate than think to get it right first time.

    the whole step 5 is <3

    I’m not suggesting you write the same ball of mud ten times over, perfecting your mistakes. To quote Perlis: “Everything should be built top-down, except the first time”. You should be trying to make new mistakes each time, take new risks, and slowly build up through iteration.

    Instead of breaking code into parts with common functionality, we break code apart by what it does not share with the rest. We isolate the most frustrating parts to write, maintain, or delete away from each other.; We are not building modules around being able to re-use them, but being able to change them.

    When a module does two things, it is usually because changing one part requires changing the other. It is often easier to have one awful component with a simple interface, than two components requiring a careful co-ordination between them.

    The strategies I’ve talked about — layering, isolation, common interfaces, composition — are not about writing good software, but how to build software that can change over time.

  • https://rakyll.org/style-packages/

  • https://www.martinfowler.com/articles/designDead.html

  • https://youtu.be/ZsHMHukIlJY

    A common fallacy is to assume authors of incomprehensible code will somehow be able to express themselves lucidly and clearly in comments.