I’m passionate about devtool and data visualization for a long time.
I’m regularly giving trials to new tools in the mission to improve my productivity; I’m also creating some tools by myself (assh, depviz, wookie ADN solver, etc.)
There is a sub-topic where I’m more consuming external tools than producing ones: graph visualization; which contains itself some sub-topics: graph databases, graph optimizations, social graphs, real-time graph, graph UI, etc.
For about a week now, thinking (again) about ”how to represent a complex system”, and more precisely: ”how to make a collaborative tool that allows to defining, viewing and analyzing an unstructured, complex, evolutive, and living system.”
For now, I’m mostly reading articles and PoCing, from what I can see, there are a lot (maybe too much) of different existing solutions that handle every part I would need for the final solution.
Problems to address
- how to store the data
- how to programmatically inject and edit data
- how to manually inject and edit data
- how to generate code based on the model definitions
- how to visualize easily / navigate
- how to perform queries
- how to create real-time dashboards
My plan is now to give a more in-depth look at my favorite options.
In this intention, I listed 10 (actually 11) ideas of usages that are easier to implement than my target.
Usages for myself
- Visualize GitHub issues relationships – dependencies, author, people working on, people commenting out, project & organization hierarchy, milestone grouping, labels tagging, etc.
- Visualize Git code/PRs/Commits – similar to GitHub issues above, but based on code instead of issues: code, files, languages, author, PRs, commits, etc.
- Define and visualize IT architecture (intermediary milestone to my target) – host, container, ports, cluster, dependencies, Datacenter, process, developers, product, etc.
- Personal CRM – maintain a wiki about my relationships (people and company), visualize my social graph, etc.
- Blog content relationship – analyze content based on multiple criteria (keywords, tags, labels, label’s metadata, etc.) to analyze what I talk most, and generate better “related posts” suggestions.
- Cross-service social graph – create aggregators to visualize people, groups, the friends of friends, followers, followed, etc.
- Real-time monitoring – define services hierarchically and then write probs that to monitor the health
- Service/Application comparison based on features and other attributes
- My GitHub stats; repos, organizations, languages, libraries, metadata, CI used, followers, custom flags (more than 1000 commits, edited < 1 year ago, has a Dockerfile, contributors, etc.).
- Log parser to analyze user agent to endpoint (real-time dependency) – https://link.medium.com/rSLv1KGPnU
- P2P network efficiency analysis
Bonus: usages for friends
- PayFit: engine rules visualization
- Doctrine: analysis of legal case relationships
- Zenly: social graph, party recommendation
- Sounds.am: social graph, friend/playlist/artist/song recommendations
- Scaleway: visualize relationships between image / volume / volume layer / server entities
PS: about 1 year ago, I forced myself to list “10 something” every morning for a month.
Writing this blog post motivated me to retry the experience and share some outputs in the form of small articles on this blog.Note: this article is the output of a daily routine, the content of this list won't change over time. It's, however, possible that I create a whole new list on the same subject as a dedicated new post.
There are two main ways of using GraphQL:
- as the core method of defining & accessing APIs
- as an alternative way of accessing existing APIs (a proxy)
The original idea here is not about the technical way of doing it as it is pretty standard.
I focus on a specific context where this idea may become very interesting:
- An heterogenous system,
- made of mixed protocols (Rest, gRPC, GraphQL, …),
- multiple languages,
- standalone/autonomous teams (not working in the same office),
- legacy code,
- third-party services,
- mix of big monoliths and small micro-services,
A Central GraphQL for Big Organizations is a tool that allows to aggregate everyone on a common format while preserving the diversity of the ecosystem, with a very limited risk, and with “free documentation” as a bonus.
It may also be a solution for smaller organizations as the secret is to only involve a small team or even one person to bootstrap the project.
- choose a way of defining contracts
- You can choose to write in the GraphQL definition format directly, in protobuf and then generate the GraphQL code, probably even in swagger, etc
- implement (at least) two different services using this format
- bootstrap a PoC implementing the two services, you may have to develop some “drivers” for common feature: authentication, retry policy etc
- show your PoC to some other key people (architects, team leaders, etc.)
- motivate people to help you write missing API specifications and “drivers”
- you now have three options:
- to try to make everyone fit in the same API formats and everything (your initial plan?) 😉
- or ask them to maintain a specification file for their existing APIs
- you can now use advanced features of GraphQL: cross-services unions, streams, validation, type safety, client code generation, etc…
- it can be tested without risk, if this project fails, you only involed a small team, not the whole organization
Create a tool that allows to colleboratively define, visualize (with graphs) and query dependencies.
a.k.a. “The perfect mix between Neo4J, Wikipedia/Wikidata, and LucidChart“
- can be done manually
- can be done using scripts/APIs to aggregate external sources
- should be typed yet flexible
- Wikibase, the engine behind https://wikidata.org, to store and manually manage the dependencies
- An existing or a custom solution to import/synchronize existing data
- Well-defined base entities to ensure a good overall quality
- A custom frontend to provide useful visualizations
- Some process/guidance to have collaboration