X Tutup
The Wayback Machine - https://web.archive.org/web/20210623214036/https://github.com/netdata/netdata/pull/5566
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Squash integration #5566

Merged
merged 4 commits into from Sep 16, 2019
Merged

Squash integration #5566

merged 4 commits into from Sep 16, 2019

Conversation

@emiquelito
Copy link
Contributor

@emiquelito emiquelito commented Mar 6, 2019

Summary

This is a patch to integrate Squash.io with NetData. I believe this is going to be handy to the project so any project maintainers and developers can quickly check the branch changes deployed on its own isolated VM. This is for the NetData application itself, not for the docs or just static files.

Example of NetData working with Squash: https://squash-setup-uoie4.squash.io

Squash is free for Open Source, maintainers just need to signup and reach out to the support through the chat window and we will enable a forever free Open Source account. Once this is done, Squash will post a comment on each Pull Request like this:

squash-pr-comment

When you click on the Squash link in a new PR it will deploy the branch of code into a brand new VM, this process takes about 2 minutes.

Let me know what you think please. Thanks.

Component Name

N/A

Additional Information

This is an easy way to preview the branch changes for the NetData application itself, not for the docs or just static files.

@emiquelito emiquelito requested a review from ktsaou as a code owner Mar 6, 2019
@CLAassistant
Copy link

@CLAassistant CLAassistant commented Mar 6, 2019

CLA assistant check
All committers have signed the CLA.

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Mar 27, 2019

Hi @ktsaou just checking to see if you have any thoughts on this patch. Please let me know, thanks.

@cakrit
Copy link
Contributor

@cakrit cakrit commented Mar 28, 2019

Thanks for the reminder @emiquelito and apologies for the late response. We've discussed this with @ktsaou and it is something that would be useful. I was wondering though, if it's possible for the integration to be a bit different. If you see the automated checks running even in this PR, the netlify ones are pretty close to what you're doing. If the build fails we get an error and the Details link shows us the generated documentation. I would expect a similar behavior from any other tool. A comment with a link is not ideal, as it doesn't really tell you for example if it's rebuilding after a new commit on a PR.

Is this something you would consider looking into? We can definitely incorporate it and help you debug the integration if you like. It would just be much cleaner and nicer this way...

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Mar 29, 2019

hey @cakrit thanks for your response and feedback. See answers below

If you see the automated checks running even in this PR, the netlify ones are pretty close to what you're doing. If the build fails we get an error and the Details link shows us the generated documentation. I would expect a similar behavior from any other tool. A comment with a link is not ideal, as it doesn't really tell you for example if it's rebuilding after a new commit on a PR.

I see your point and definitely agree it's a valid use case. We already have this in our product roadmap and looking to release it pretty soon, likely early in April.

Squash has a few nuances compared to Netlify and even traditional CI tools. A Squash deployment can stay up for a longer period so you can test the app without interruptions. That being said, we will have new settings to automatically restart an existing deployment based on new commits so it acts like a CI tool if that's what users want. In our case Squash is about showing your application fully working from a URL endpoint. Our plan for this automatic check within GitHub would be to trigger a build on each commit and give a feedback (green or red) if:

  • the build is successful
  • squash receives a success response, which means the app's launch process also works
  • The deployment URL would then still active for a few hours depending on the shutdown schedule. For open source accounts this is currently defined at 3 hours.

Is this what you had in mind?

Is this something you would consider looking into? We can definitely incorporate it and help you debug the integration if you like. It would just be much cleaner and nicer this way...

Thank you. Absolutely, I will get back to you very soon once we have this feature launched.

@cakrit
Copy link
Contributor

@cakrit cakrit commented Mar 29, 2019

Thanks for the reply. What you described is perfect, except for the 3 hour limit. The author will get to see the result in that period, but the reviewer will generally not work on it immediately. I understand the reasoning behind a time limit, but we'll need to find some way to make it a bit more useful.

How about the following:

  • When I press on Details, I am informed that the URL is no longer accessible.
  • I am able to press a retry deployment button, as I do in Travis and Netlify. The effect is to get a new URL, when I can see it for the next 3 hours.

You could conceivably even combine the above with a limit on concurrently available URLs. Say that the limit for open source projects is 3. It could work in a round-robin way, so that if I go and do 4 PRs within a given time, only the last 3 will have an active link in the Details. I could go and press retry deployment on the first one, which would then deactivate the second one. I believe extending the duration the URL is available (container live) is more important than the number of concurrently available URLs.

Even without a limit, I think that a retry deployment button will be needed, so that we don't do dummy commits just to trigger the build & publish process again.

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Mar 29, 2019

hi @cakrit

Thanks for the reply. What you described is perfect, except for the 3 hour limit. The author will get to see the result in that period, but the reviewer will generally not work on it immediately. I understand the reasoning behind a time limit, but we'll need to find some way to make it a bit more useful.

We can definitely adjust these limits to make it work for you guys. The way we have envisioned this feature is to still allow the Squash comment to appear in PRs (and you can also turn it off in the .squash.yml as you see fit) so you can still easily grab the deployment URL and launch it anytime. Anytime Squash starts a new deployment it will fetch the latest code in the branch by default, this is how it works now.

With the GitHub checks (as with Travis, Netlify, etc) it's the same thing, you would still be able to click on the deploy URL from there and re-launch it in case it's already down. Bear in mind that the deployment process usually takes around 2 min (to build and make the app available for uat testing), Squash also has a deployment cache feature that helps to speed up this process.

When I press on Details, I am informed that the URL is no longer accessible.

Squash creates a unique URL per branch, and that URL is always accessible. However, when you click on it the deployment might be down and when that happens Squash automatically starts it and shows you a loading page right away. As I mentioned above this process usually takes about 2 min depending on the app.

I am able to press a retry deployment button, as I do in Travis and Netlify. The effect is to get a new URL, when I can see it for the next 3 hours.

Correct, same as above. Just clicking on that URL will trigger the deployment to start.

You could conceivably even combine the above with a limit on concurrently available URLs. Say that the limit for open source projects is 3. It could work in a round-robin way, so that if I go and do 4 PRs within a given time, only the last 3 will have an active link in the Details. I could go and press retry deployment on the first one, which would then deactivate the second one. I believe extending the duration the URL is available (container live) is more important than the number of concurrently available URLs.

We have a limit of concurrent deployments for open source and we can also adjust this to fit your needs. For netdata we could start with a limit of 10 and adjust as needed. You do have a dashboard on Squash where you can see all the active deployments and manually shut down the ones you don't need anymore.

Even without a limit, I think that a retry deployment button will be needed, so that we don't do dummy commits just to trigger the build & publish process again.

Thanks, this is pretty much how the new feature will work. We are also planning to add a separate field in the squash YAML file to allow you to specify a short TTL for deployments automatically triggered from commits. This will help to give you more resources for the deployments you want to check yourself from the Squash URL.

Makes sense? please let me know your thoughts and if you have any additional feedback. Thanks

@cakrit
Copy link
Contributor

@cakrit cakrit commented Mar 29, 2019

It's great, no problem. A few minutes waiting to get the URL working is nothing, so there's no need to worry about the number of instances and the 3 hours. The comment won't be necessary with what you described, but it's not a deal breaker.

I signed up and was actually about to install it on netdata/netdata, but I see it asks for administrator write permissions, which of course we can't grant.

Screenshot from 2019-03-29 17-52-50

I can reach out to support for the license later, but please suggest how we should proceed.

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Mar 30, 2019

hey @cakrit thanks for this.

Squash asks for this admin write permissions due to a limitation on GitHub, this is because we need to be able to add a deploy key into the repo (and GitHub doesn't provide a separate permission for this) so the service can automatically pull code from private repos during deployments.

I see you guys are using Travis, it's the same permissions requested by Travis and many other services:

https://docs.travis-ci.com/user/github-oauth-scopes/

Capture

That being said, it makes sense to reduce this permission level for Open Source projects since the repos are public and we don't need to install deploy keys. We are currently reviewing this and will get back to you early next week. Thanks

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Apr 5, 2019

hey @cakrit just a heads up that we have a solution to reduce the level or permissions requested from GitHub and we are planning to launch it very soon, I will keep you posted.

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Apr 12, 2019

Hey @cakrit, just a heads up that we just updated our permission settings and you will no longer be asked for admin permissions when installing Squash. Let me know if you have any questions.

The actual GitHub checks we discussed are currently in QA and will be released soon, I will let you know when this is ready.

@cakrit
Copy link
Contributor

@cakrit cakrit commented May 7, 2019

@emiquelito just let me know as soon as the github integration is ready, since I didn't get the time to try it out with the previous logic. I'm eager to integrate this in our project.

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented May 7, 2019

@cakrit thanks for the heads up and sorry for the delay on getting this ready for you. We had to tackle a few other unexpected requests. This is back to the top of our list and we will be launching it very soon, I will keep you posted.

@emiquelito emiquelito requested a review from cakrit as a code owner May 31, 2019
@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented May 31, 2019

Hi @cakrit this is now ready, sorry for the delay.

This branch has been updated to use our new automated checks feature, more details here:

https://www.squash.io/docs/automated-checks/

and here is an example where the checks are working:

emiquelito#1

Let me know if you have any feedback on this.

The current YAML setup will automatically shutdown the deployment once the check has passed or failed (HTTP response), but you can also adjust it to leave such deployments running for a bit longer. You can also just go to the Squash dashboard and restart specific checks and manually change their spire policy to leave they running longer.

My recommendation is to just rely on the Squash PR comment when you need to check the app using the latest code in the branch.

Deployment Cache

Due to the way your docker image is setup with the code being copied at the beginning of the Docker file we recommend turning off the Deployment Cache feature, see below, you just need to uncheck the "Cache Docker Images" field.. This is to ensure each check/deployment uses the latest code in the branch. Please note that without the cache the total build/deployment process is slightly above 3 min.

If this becomes a problem just let me know and we can come up with a solution without the docker file and move the deployment process entirely to the Squash YAML file, this way we can make the cache work.

cache-docker-images

Account limits

Please let me know when you merge this and signup on Squash (just leave us a note in our chat support here https://www.squash.io/support/) so I can increase your account limits to accommodate all your user activity without issues.

@cakrit
Copy link
Contributor

@cakrit cakrit commented Jun 3, 2019

Thanks @emiquelito, I'l ask @paulkatsoulakis to test this within the next couple of weeks.

@cakrit cakrit requested a review from paulkatsoulakis Jun 3, 2019
@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Aug 13, 2019

Hi @cakrit and @paulkatsoulakis I trust all is well. I just wanted to follow up on this PR and see if you have any thoughts.

Please let me know

Thanks

@emiquelito emiquelito requested a review from cosmix as a code owner Aug 13, 2019
@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Sep 9, 2019

Hey @cakrit and @paulkatsoulakis, I just wanted to follow up and see if you have any questions about this PR. Thanks

@cakrit
cakrit approved these changes Sep 16, 2019
@cakrit cakrit merged commit c18b816 into netdata:master Sep 16, 2019
12 checks passed
12 checks passed
@netlify
Header rules - netdata No header rules processed
Details
@netlify
Pages changed - netdata 4 new files uploaded
Details
@netlify
Redirect rules - netdata No redirect rules processed
Details
Codacy/PR Quality Review Up to standards. A positive pull request.
Details
LGTM analysis: C/C++ No code changes detected
Details
LGTM analysis: JavaScript No code changes detected
Details
LGTM analysis: Python No code changes detected
Details
@netlify
Mixed content - netdata No mixed content detected
Details
@travis-ci
Travis CI - Pull Request Build Passed
Details
@wip
WIP Ready for review
Details
license/cla Contributor License Agreement is signed.
Details
@netlify
netlify/netdata/deploy-preview Deploy preview ready!
Details
@cakrit
Copy link
Contributor

@cakrit cakrit commented Sep 16, 2019

After merging I completed the sign up and did a test deployment. Seems to be working fine, I'll wait to see it in action in a PR as well. I also contacted support to request the limits.

@cakrit
Copy link
Contributor

@cakrit cakrit commented Sep 16, 2019

Note that I see the following, even though .squash.yml has auto_deploy_on_commits: true:

Screenshot from 2019-09-16 16-58-20

I checked a recent PR and it doesn't look like it triggered anything.

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Sep 16, 2019

Hi @cakrit, thanks for letting me know. We are looking into this and will get back to you ASAP.

@emiquelito
Copy link
Contributor Author

@emiquelito emiquelito commented Sep 17, 2019

@cakrit we've checked this issue and everything is in order actually. You just need to create a new PR for a branch that is in sync with master (with the .squash.yml file) and then the GitHub check and Squash comment will work.

I've just added a simple PR for you to check:

#6864

The GitHub check for Squash worked fine and you should now see historical results here:

https://app.squash.io/accounts/checks/

Let me know if you have any other questions.

@cakrit
Copy link
Contributor

@cakrit cakrit commented Sep 17, 2019

Got it. I'll comment on that PR regarding what I see.

Saruspete added a commit to Saruspete/netdata that referenced this pull request Oct 9, 2019
* Create a .squash.yml file

* enable deployment checks and a few small improvements
jackyhuang85 added a commit to jackyhuang85/netdata that referenced this pull request Jan 1, 2020
* Create a .squash.yml file

* enable deployment checks and a few small improvements
Saruspete added a commit to Saruspete/netdata that referenced this pull request May 21, 2020
* Create a .squash.yml file

* enable deployment checks and a few small improvements
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
X Tutup