Commit Graph

132 Commits

Author SHA1 Message Date
Steven Masley 09f00c08df
chore: shutdown provisioner should stop waiting on client (#13118)
* chore: shutdown provisioner should stop waiting on client
* chore: add unit test that replicates failed client conn
2024-05-03 10:15:17 -05:00
Kyle Carberry 895df54051
fix: separate signals for passive, active, and forced shutdown (#12358)
* fix: separate signals for passive, active, and forced shutdown

`SIGTERM`: Passive shutdown stopping provisioner daemons from accepting new
jobs but waiting for existing jobs to successfully complete.

`SIGINT` (old existing behavior): Notify provisioner daemons to cancel in-flight jobs, wait 5s for jobs to be exited, then force quit.

`SIGKILL`: Untouched from before, will force-quit.

* Revert dramatic signal changes

* Rename

* Fix shutdown behavior for provisioner daemons

* Add test for graceful shutdown
2024-03-15 13:16:36 +00:00
Kayla Washburn-Love 475c3650ca
feat: add support for optional external auth providers (#12021) 2024-02-21 11:18:38 -07:00
Cian Johnston 643c3ee54b
refactor(provisionerd): move provisionersdk.VersionCurrent -> provisionerdproto.VersionCurrent (#12225) 2024-02-20 12:44:19 +00:00
Garrett Delfosse 5b122d108e
fix: publish workspace update on quota failure (#11559) 2024-01-11 14:59:40 -05:00
Kyle Carberry 30afe43f8a
fix: create tempdir prior to cleanup (#11394)
See https://github.com/coder/coder/actions/runs/7399827933/job/20132407700

Seems like this happened because the test was being cleaned up
while the tempdir was being made.
2024-01-03 19:18:15 +00:00
Cian Johnston 213b768785
feat(coderd): insert provisioner daemons (#11207)
* Adds UpdateProvisionerDaemonLastSeenAt
* Adds heartbeat to provisioner daemons
* Inserts provisioner daemons to database upon start
* Ensures TagOwner is an empty string and not nil
* Adds COALESCE() in idx_provisioner_daemons_name_owner_key
2023-12-18 16:44:52 +00:00
Spike Curtis 9a4e1100fa
chore: move drpc transport tools to codersdk/drpc (#11224)
Part of #10532

DRPC transport over yamux and in-mem pipes was previously only used on the provisioner APIs, but now will also be used in tailnet.  Moved to subpackage of codersdk to avoid import loops.
2023-12-15 12:41:39 +04:00
Spike Curtis 211718f95a
fix: fix MaliciousTar test case (#10158)
fixes #9895

Problem was that provisionerd tries to acquire the next job, and races with shutdown, triggering the assert in the handler.  Switches this test case to use the more robust handler.
2023-10-10 13:24:43 +04:00
Kyle Carberry 5596fb20b5
chore: move `/gitauth` to `/externalauth` on the frontend (#9954)
* chore: move `/gitauth` to `/externalauth` on the frontend

This actually took a lot more jank than anticipated,
so I wanted to split this up before adding the ability
to embed new providers.

* Rename FE

* Fix em' up

* Fix linting error

* Fix e2e tests

* chore: update helm golden files
2023-09-30 14:30:01 -05:00
Kyle Carberry 8abca9bea7
chore: rename `git_auth` to `external_auth` in our schema (#9935)
* chore: rename `git_auth` to `external_auth` in our schema

We're changing Git auth to be external auth. It will support
any OAuth2 or OIDC provider.

To split up the larger change I want to contribute the schema
changes first, and I'll add the feature itself in another PR.

* Fix names

* Fix outdated view

* Rename some additional places

* Fix sort order

* Fix template versions auth route

* Fix types

* Fix dbauthz
2023-09-29 19:13:20 +00:00
Spike Curtis 375c70d141
feat: integrate Acquirer for provisioner jobs (#9717)
* chore: add Acquirer to provisionerdserver pkg

Signed-off-by: Spike Curtis <spike@coder.com>

* code review improvements & fixes

Signed-off-by: Spike Curtis <spike@coder.com>

* feat: integrate Acquirer for provisioner jobs

Signed-off-by: Spike Curtis <spike@coder.com>

* Fix imports, whitespace

Signed-off-by: Spike Curtis <spike@coder.com>

* provisionerdserver always closes; remove poll interval from playwright

Signed-off-by: Spike Curtis <spike@coder.com>

* post jobs outside transactions

Signed-off-by: Spike Curtis <spike@coder.com>

* graceful shutdown in test

Signed-off-by: Spike Curtis <spike@coder.com>

* Mark AcquireJob deprecated

Signed-off-by: Spike Curtis <spike@coder.com>

* Graceful shutdown on all provisionerd tests

Signed-off-by: Spike Curtis <spike@coder.com>

* Deprecate, not remove CLI flags

Signed-off-by: Spike Curtis <spike@coder.com>

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-09-19 10:25:57 +04:00
Jon Ayers 622442203d
chore: fix test flake in TestProvisionerd (#9709) 2023-09-18 11:23:22 -05:00
Spike Curtis 11b6068112
feat: add support for networked provisioners (#9593)
* Refactor provisionerd to use interface to connect to provisioners

Signed-off-by: Spike Curtis <spike@coder.com>

* feat: add support for networked provisioners

Signed-off-by: Spike Curtis <spike@coder.com>

* fix token length and linting

Signed-off-by: Spike Curtis <spike@coder.com>

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-09-08 09:53:48 +00:00
Mathias Fredriksson b240799f47
refactor(cli): use codersdk for provisioner types (#9508)
This change removes one use of `coderd/database` from the slim binary
and more correctly uses codersdk instead of database or provisionerd
packages.

No size change (yet).

Ref: #9380
2023-09-04 21:42:15 +03:00
Spike Curtis 90acf998bf
fix: fix null pointer on external provisioner daemons with daily_cost (#9401)
* fix: fix null pointer on external provisioner daemons with daily_cost

Signed-off-by: Spike Curtis <spike@coder.com>

* Add logging for debounce and job acquire

Signed-off-by: Spike Curtis <spike@coder.com>

* Return error instead of panic

Signed-off-by: Spike Curtis <spike@coder.com>

* remove debounce on external provisioners to fix test flakes

Signed-off-by: Spike Curtis <spike@coder.com>

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-08-30 10:48:35 +00:00
Spike Curtis 60d5002eb6
refactor: change template archive extraction to be on provisioner (#9264)
* refactor provisionersdk protocol

Signed-off-by: Spike Curtis <spike@coder.com>

* refactor provisioners to use new protocol

Signed-off-by: Spike Curtis <spike@coder.com>

* refactor provisionerd to use new protocol

Signed-off-by: Spike Curtis <spike@coder.com>

* refactor tests & proto renames

* Fixes from self-review

Signed-off-by: Spike Curtis <spike@coder.com>

* appease fmt & link

Signed-off-by: Spike Curtis <spike@coder.com>

* code review fixes & e2e fixes

Signed-off-by: Spike Curtis <spike@coder.com>

* More fmt

Signed-off-by: Spike Curtis <spike@coder.com>

* Code review fixes

Signed-off-by: Spike Curtis <spike@coder.com>

* new gen; use uuid for session workdir

Signed-off-by: Spike Curtis <spike@coder.com>

* Revert nix-based gen CI task until dogfood is on nix

Signed-off-by: Spike Curtis <spike@coder.com>

* revert deleting dogfood Docker stuff

Signed-off-by: Spike Curtis <spike@coder.com>

* Revert "revert deleting dogfood Docker stuff"

This reverts commit 9762158167.

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-08-25 06:10:15 +00:00
Jon Ayers 7ddb216d87
chore: revert nix-related CI changes (#9321)
* chore: revert nix-related CI changes

- Reverts using nix to run CI-dependencies.
- Running 'make gen' in a dogfood workspace resulted in inconsistent
  results for protobuf-related files making it difficult to pass CI.
  This PR imports the minimum changes necessary to make CI compatible
  with dogfood.
2023-08-24 21:26:30 -05:00
Ammar Bandukwala 545a256b57
fix: correctly reject quota-violating builds (#9233)
Due to a logical error in CommitQuota, all workspace Stop->Start operations
were being accepted, regardless of the Quota limit. This issue only
appeared after #9201, so this was a minor regression in main for about
3 days. This PR adds a test to make sure this kind of bug doesn't recur.

To make the new test possible, we give the echo provisioner the ability
to simulate responses to specific transitions.
2023-08-22 02:55:39 +00:00
Ammar Bandukwala 6b8102cf4c
feat(cli): add daily_cost to `coder ls` (#9200) 2023-08-19 12:56:08 -05:00
Kyle Carberry 22e781eced
chore: add /v2 to import module path (#9072)
* chore: add /v2 to import module path

go mod requires semantic versioning with versions greater than 1.x

This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```

Migrate generated files to import /v2

* Fix gen
2023-08-18 18:55:43 +00:00
Kyle Carberry c0a78533bf
chore: reduce build time by converting docker image to build with nix (#9099)
* Add Docker image

* Try building the container in CI

* Uncomment testing lines

* Trim image step

* Hit 'em with a Docker load

* Oopsie!

* Add a tag and push!

* Fix image name 🤦

* Fix sudo

* Fix target name

* Build and push

* Fix login to docker hub

* Revert to Docker push

* Fix PATH

* Fix SHA

* Fix ca certs

* Fix ca certs

* Fix coping files in

* Fix docker init

* Fix Docker group and init

* Add comments to our Nix

* Fix build stage

* Add some more comments

* Remove old dogfood image files

* Tag and push with branch name

* Fix tag passing

* Fix tag passing

* Remove old pull triggers

* Convert gen to use Nix

* Add protobuf to the flake

* Add prettier to the dev shell

* Swap to the faster Nix cache action

* Add the correct yq

* Fix gen

* Add make to the flake

* Update extensions
2023-08-17 21:19:36 -05:00
Ammar Bandukwala be40dc85ab
chore: cleanup extraneous logging (#9156)
* The batchstats warning went out on every Ctrl+C in my development

Rule of silence:

The provisioner and connect messages messages were sent out on every startup
without a corresponding user event, making them annoying and more-so
debug messages.
2023-08-17 21:01:55 +00:00
Spike Curtis 21af020386
feat: add external provisioner daemon helm chart (#8939)
* Refactor helm to extract common templates to libcoder

Signed-off-by: Spike Curtis <spike@coder.com>

* Remove comment from libcoder Chart.yaml

Signed-off-by: Spike Curtis <spike@coder.com>

* Add provisioner helm chart

* Fix prettier, linting, docs

Signed-off-by: Spike Curtis <spike@coder.com>

* Log at INFO when provisionerd connects to coderd

Signed-off-by: Spike Curtis <spike@coder.com>

* remove unnecessary exports in helm tests

Signed-off-by: Spike Curtis <spike@coder.com>

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-08-10 13:59:43 +04:00
Asher 7ed17b2605
fix: add some missing workspace updates (#7790)
* Standardize on function to get workspace channel name

There were two, now there is one.

* Add some missing workspace updates

There are some failure cases where we do not set the type as a workspace
build which causes the workspace update to never be published.

* Make build failures warnings

Otherwise the associated test fails due to the logger fataling on
error messages.
2023-07-14 15:07:48 -08:00
Marcin Tojek c6fcd7ee93
fix: report failed CompletedJob (#8318) 2023-07-06 07:26:33 +00:00
Dean Sheather 98a5ae7f48
feat: add provisioner job hang detector (#7927) 2023-06-25 13:17:00 +00:00
Marcin Tojek 4fb4c9b270
chore: add more rules to ensure logs consistency (#8104) 2023-06-21 12:00:38 +02:00
Colin Adler 7d43f4ceee
chore: update dogfood image (#8114) 2023-06-20 15:08:15 -05:00
Marcin Tojek b1d1b63113
chore: ensure logs consistency across Coder (#8083) 2023-06-20 12:30:45 +02:00
Mathias Fredriksson c12c9f1f4e
chore(go.mod): update cdr.dev/slog (#7994)
* chore(mod): update cdr.dev/slog

* fix: change uses of []slog.Field to []any to match new API
2023-06-13 18:17:04 +00:00
goodspark 0665a6c2f2
feat: add metric for provisioner daemons (#7858) 2023-06-06 16:50:11 -05:00
Marcin Tojek a7366a8b76
feat!: drop support for legacy parameters (#7663) 2023-06-02 11:16:46 +02:00
Marcin Tojek 4eb0baa849
chore: update protoc-gen-go (#7708) 2023-05-30 10:24:34 +02:00
Colin Adler 8bd9f9c351
feat: unified tracing between coderd<->provisionerd (#7370) 2023-05-03 23:02:35 +00:00
Colin Adler 085330ad96
fix(provisionerd): only heartbeat when logs aren't being flushed (#7110) 2023-04-13 14:02:10 -05:00
ElliotG 0069831e8d
fix: use error log when failing provisioner job (#6812)
Co-authored-by: Colin Adler <colin1adler@gmail.com>
2023-04-05 13:30:53 -05:00
Colin Adler a29fc7dd6f
chore: update otel to v1.14.0 (#6963) 2023-04-03 00:31:39 -05:00
Marcin Tojek 0ba200c2a1
feat: Enable workspace debug logging (#6838)
* feat: Enable workspace debug logging

* Fix

* Fix

* Fix

* fix

* fix

* Enable RBAC

* unit tests

* Fix

* fix

* fix

* fix

* more tests

* fix: workspacebuild_test use roles

* fix: swagger comment

* fix: ctx.Done

* fix: address PR comments

* break loop
2023-03-30 16:00:33 +02:00
Kyle Carberry df31636e72
feat: pass `access_token` to `coder_git_auth` resource (#6713)
This allows template authors to leverage git auth to perform
custom actions, like clone repositories.
2023-03-22 19:37:08 +00:00
Marcin Tojek 7587850a1c
feat: import value from legacy variable to build parameter (#6556) 2023-03-14 12:02:44 +00:00
Marcin Tojek 3b87316ad7
feat: propagate job error codes (#6507)
* feat: propagate job error_code

* fix

* Fix

* Fix

* Fix

* add errors to typesGenerated

* Address PR comments

* Fix
2023-03-08 16:32:00 +01:00
Cian Johnston 26a725f86d
fix: provisionerd: add more context to logs emitted, fix log level (#6508)
- Previously, we were logging all provision response logs at level INFO, regardless of the log level of the log streamed from the provisioner. We now log these at the original level (defaulting to INFO).
- Now logging "provision failed" message at level ERROR WARN and including the error field in the message.
2023-03-08 15:12:52 +00:00
Cian Johnston 1483b42259
fix(provisionerd): emit workspace build transition in lower case (#6455) 2023-03-06 11:14:19 +00:00
Kyle Carberry 7f226d4f90
feat: add support for `coder_git_auth` data source (#6334)
* Add git auth providers schema

* Pipe git auth providers to the schema

* Add git auth providers to the API

* Add gitauth endpoint to query authenticated state

* Add endpoint to query git state

* Use BroadcastChannel to automatically authenticate with Git

* Add error validation for submitting the create workspace form

* Fix panic on template dry-run

* Add tests for the template version Git auth endpoint

* Show error if no gitauth is configured

* Add gitauth to cliui

* Fix unused method receiver

* Fix linting errors

* Fix dbauthz querier test

* Fix make gen

* Add JavaScript test for git auth

* Fix bad error message

* Fix provisionerd test race

See https://github.com/coder/coder/actions/runs/4277960646/jobs/7447232814

* Fix requested changes

* Add comment to CreateWorkspacePageView
2023-02-27 10:18:19 -06:00
Cian Johnston 473ab208af
fix: replace owner_email with workspace_owner, change action to workspace_transition (#6337) 2023-02-24 04:38:58 +00:00
Cian Johnston 282507f0fb
fix(provisionerd): add more detailed workspace build logging (#6331) 2023-02-24 00:03:12 +00:00
Marcin Tojek 50db90c33d
fix: rephrase rich parameters error (#6327) 2023-02-23 21:49:40 +00:00
Cian Johnston 43e8ba0811
feat(api): add prometheus metric coderd_workspace_builds_total (#6314)
This PR adds the prometheus metric coderd_workspace_builds_total.
It measures the total number of workspace builds, along with a number of labels intended to be useful for an operator debugging a failed workspace build trying to discover the scope of the issue.
2023-02-23 01:28:10 +00:00
Ammar Bandukwala f05609b4da
chore: format Go more aggressively 2023-02-18 18:32:09 -06:00