Hive workflow proposal

in #hive3 months ago

Note: this post is primarily intended for Hive developers, but it may be of interest to other readers who are interested in the details of the software development process. I’ll make another post later with ideas about how general community members can become involved in the development process and how to go about submitting bug reports.

I propose we adopt a common workflow for most if not all Hive-related projects. This will make it easier for developers to collaborate on multiple projects with a minimum amount of friction.

I believe the primary goals (in no particular order) of our workflow should be:

  • create an easily findable list of hive-related tasks
  • prevent duplication of work effort
  • allow easy collaboration between multiple developers
  • establish a process for determining task priority
  • setup testing standards for tasks in each project
  • setup review standards for tasks in each project
  • ensure existence of easily findable development code in a functional/compilable state where developers can test their new code against potential interactions with other new code
  • ensure existence of easily findable production code which is safe to be deployed to the public

Below are some of my initial proposals for how to achieve some of the above goals.

Creating issues for new Hive tasks

When a new task comes up, create an issue in the appropriate project inside the Hive group.

Selecting a project for a task (issue)

If it is not clear what project a task belongs to, or if the task belongs to no existing project, create an issue for it in the project called Issues without projects yet. If this issue is later assigned to a specific project, create a corresponding new issue in the correct project and close the issue in Issues without projects yet.

Describing an issue with labels

Choose existing labels to describe the nature of the issue, such as “bug”, “enhancement”, “documentation”. Currently we only have group-level labels. Generally group-level labels are preferred, unless you need labels that really only make sense for a specific project in the group.

Optional assignment of a person to a task

The issue can be assigned at creation time to a specific developer, if you are self-assigning or you are assigning to someone you are confident will want to take on the task (in the latter case, you should probably also contact them directly about the new issue).

Using boards to monitor project issues

Using gitlab, each individual project in the Hive group has a “board” that shows the states of the project’s issues:

  • open: an issue exists, but no decision has yet been made to work on it
  • To Do: decision has been made to do the task and it is awaiting someone to work on it, or is blocked by some other precursor task
  • Doing: one or more developers are currently working on issue
  • Review: work is done, and is awaiting review by an independent developer, after review is completed, the state will be changed to either To Do, Doing, or closed
  • closed: work has been completed and passed review

When an issue is first created it will show up on the “open” list. Setting the appropriate label on the issue moves it from To Do, Doing, and Review. Closing the issues moves it to the closed list.

Using Group-level board to monitor all workflow

In addition to the project-level boards, there is a group-level board that aggregates the status of issues from all projects in the group: https://gitlab.syncad.com/hive/hive/-/boards

Process for setting To Do, Doing, and Review status

The process for setting the status of an issue will likely vary across projects. For example, if the project is a web site hosted by a specific entity, that entity may want to determine which issues should be worked on, who works on them, and how they are reviewed.

Some core projects such as the blockchain code should also have more rigorous criterion for selection of which issues to work on and how those issues are reviewed, because of the potential ripple effects on other projects.

The blockchain code is particular sensitive, as new features can easily have impacts on memory, disk, and CPU utilization that could increase the cost of operating a node.

Similarly, changes to the commonly used libraries such as hive-js and financial-related projects such as wallet apps need close review to ensure no malicious code is injected.

Reference issues when committing associated source code to a repository

It’s very useful to link all code changes to an issue when possible. When you make a code commit related to an issue, you can place a tag in your code commit comment that will link the commit to the issue it is related to. If you forget to do this when you commit the code, you can also manually update the issue afterward by adding a comment with a URL link to the commit.

Testing requirements

Unit tests and image "tests"

One or more tests should generally be included with each code change, to verify that the code works properly, and to be able to detect future regressions (future changes that break functionality). For backend code, such tests are usually not too difficult to create and should be required for the review process.

For user interface (UI) changes, it’s often difficult to create such tests, but in such cases, it’s useful to attach a “before” and “after” set of images to the issue, to make it easier for the reviewer to understand the work that was done.

Continuous integration (CI) should be setup for projects

Another important component of proper testing is to ensure that the code that compiles for one developer still compiles for others. Due to environment differences, it’s easy to commit code that compiles and works properly on one developer’s machine, but then fails on others.

To detect such problems, a project should setup a CI task in Gitlab that will automatically recompile the project and run regression tests each time new code is pushed to a code repository. This will enable rapid detection of commit problems and avoid wasting the time of other developers.

Branch management in repositories

The master branch serves as the branch for testing interactions between new code, and for deployment to production.

Tags are used to indicate the most recent version of the code approved for use in a production environment. Only tagged versions should be deployed to production environments!

The head of the master branch should represent the most recent version of development code that is believed to be compilable and functional.

Note that the above flow will require some cleanup to the existing steemd (now renamed hive) repo.

Public testnet for blockchain code

Preferably, we should always have one or more public testnets available for testing by apps, without requiring them to test on the main blockchain network. Witnesses are encouraged to operate consensus and API nodes to support the testnet(s).

A new public testnet may be deployed at any time when it’s deemed that changes have been made that potentially impact third party apps.

We should tag whatever version of the master branch that is being used for a public testnet.

Sort:  

Very interesting. This does seem like a good way for developers to share problems and solutions.

It could shortcut the development process which would be wonderful for the Hive community.

Question: Is Gitlab the now preferred over Github?

I personally prefer gitlab, because it's easy to run your own instances of it, and I like the available workflow better. And it's even customizable, since it's open source, unlike github.

  • Github was acquired by Microsoft if I am not wrong. That is a big REDFLAG!

This is how you develop software, so most of the developers should be familiar with that kind of workflow. Last weeks were crazy, and everyone worked on time pressure, but now the workflow should definitely change, and we, as a blockchain community, should promote best practices in software development.

CI/CD

Testing is crucial, and I believe we could set up minimal test coverage required before merging changes (should be possible with Gitlab CI/CD afair). Getting this information (coverage) might be quite tricky because different technologies and frameworks are used in various projects, but with Gitlab CI/CD it's doable for most of them. Also, it would be great to have merge requests approvals configured. End-to-end tests where possible.

Screenshot from 20200404 003428.png

Building and pushing docker images automatically on merges/releases is also a must. But that's quite common and easy to configure. The next step is a staging environment for some backend tools and condenser. It's a game-changer if you can review changes in a real environment, just a few minutes after MR was approved. I can help a lot in this field.

Linters

It would be great to see standardized config for eslint / prettier or equivalent for other languages. It should also be a part of the CI/CD workflow.

Git hooks

It's quite tricky to distribute git hooks, but there is an excellent tool for nodejs to achieve that: Husky. We would be able to ensure code is linted and unit tested before being pushed. This prevents many small mistakes that can give you a headache.


I'm glad we've moved to Gitlab but I'm wondering if we should use an instance managed by a private entity. Couldn't just use Gitlab SaaS?

I'm glad we've moved to Gitlab but I'm wondering if we should use an instance managed by a private entity. Couldn't just use Gitlab SaaS?

It would make it easier to get involved with a standard GitLab account. Who is syncad? Is that a trusted and reliable member of the community?

It's a software development company of mine that does all the work for BlockTrades (and other companies).

It's just as trustworthy as the guys who run gitlab saas, in my biased opinion.

Yeah self-hosting shows that you don't have centralization risk on a company. I hear you. gitlab.com also has a nice mirroring feature, so maybe also a good idea to set up official mirrors as well as community mirrors. Also I think GitLab is more likely to end up on Software Heritage.

Mirrors sound like a good idea.

Ideally, these are things that often go in (should be in) each repo as CONTRIBUTING.md (as appropriate). E.g.: https://gitlab.syncad.com/hive/devportal/-/blob/develop/CONTRIBUTING.md

Your all so smart and I dont understand most of this tech talk but I trust in your ability haha.

Not a bad way to develop software

@tipu curate


A huge hug from @amico! 🤗

This is a must. Good idea.

This would be an amazing step forward to reduce inefficiencies in development.There is not enough time or resources for us to duplicate efforts.

This would be an amazing step forward to reduce inefficiencies in development.There is not enough time or resources for us to duplicate efforts.

O'Right. thanks for putting this out!

Sounds great - it would be good if there were ‘easy pickings’ tasks that could be for new developers that want to help out and feel involved like Django does.

A lot of the initial tasks are pretty easy ones. I suppose we could create a tag to make it simple to identify the easy ones.

Yea. Tag would help...

create an easily findable list of hive-related tasks
prevent duplication of work effort

There definitely needs to be some sort of assignable work-order and bug tracking system if there isn't anything currently in place.

It sure makes life a lot easier. Of course it's been about 10 years since I was working in a collaborate environment, so I have no idea what the current best solution out there is.

Gitlab's issue system works pretty good for new work and for bugs, it's mainly just a matter of using a uniform labeling system when creating issues and then standardizing on a method of processing the issues (assignment, testing, review).

Finally.
I've been bitching about Steem not having something like this to encourage open source development since I joined Steem. It's so fucking weird that they ran something they acted like was open source and seemed like they didn't do anything to actually encourage development with.
Good on you for proposing such a well thought out plan for implementation. There are many ways to do it of course.

Interesting way to solve issues

Yup and I'll be hoping there'll be a way to "dumb down" these technical terms so average users and investors like me don't get lost at sea

great , also involve new novice developers , some program to involve those who don't know much programming but want to contribute. Hive is best community blockchain. Hive can partner with gitcoin for this. All in all the the best to awesome developers who will be making best decentralised softwares ☺️☺️👍👍

i gone start now