Commit Graph

1862 Commits

Author SHA1 Message Date
Kyle Carberry 6612e3c9c7
feat: Add config-ssh command (#735)
* feat: Add config-ssh command

Closes #254 and #499.

* Fix Windows support
2022-03-30 17:59:54 -05:00
Colin Adler 3abb87ddb6
chore: remove usage of ioutil (#642)
It was deprecated as of 1.17.
2022-03-29 14:59:32 -05:00
Kyle Carberry 82dfd6c72f
feat: Add UI for awaiting agent connections (#578)
* feat: Add stage to build logs

This adds a stage property to logs, and refactors the job logs
cliui.

It also adds tests to the cliui for build logs!

* feat: Add stage to build logs

This adds a stage property to logs, and refactors the job logs
cliui.

It also adds tests to the cliui for build logs!

* feat: Add config-ssh and tests for resiliency

* Rename "Echo" test to "ImmediateExit"

* Fix Terraform resource agent association

* Fix logs post-cancel

* Fix select on Windows

* Remove terraform init logs

* Move timer into it's own loop

* Fix race condition in provisioner jobs

* Fix requested changes
2022-03-28 19:19:28 -05:00
Garrett Delfosse bd20d9ee7f
feat: Add datadog tracing to http middleware (#530)
* add datadog tracing to http handlers
2022-03-28 22:11:52 +00:00
Kyle Carberry a502a5fa14
feat: Add AWS instance identity authentication (#570)
* feat: Add AWS instance identity authentication

This allows zero-trust authentication for all AWS instances.

Prior to this, AWS instances could be used by passing `CODER_TOKEN`
as an environment variable to the startup script. AWS explicitly
states that secrets should not be passed in startup scripts because
it's user-readable.

* Fix sha256 verbosity

* Fix HTTP client being exposed on auth
2022-03-28 19:31:03 +00:00
Colin Adler be8389fd74
chore: update to go 1.18 (#628)
* add make lint to Makefile
2022-03-28 19:14:40 +00:00
Kyle Carberry b33dec9d38
feat: Add stage to build logs (#577)
* feat: Add stage to build logs

This adds a stage property to logs, and refactors the job logs
cliui.

It also adds tests to the cliui for build logs!

* Fix comments
2022-03-28 18:43:22 +00:00
Kyle Carberry 3a48e4000e
fix: Race when shutting down and opening WebSockets (#576)
Adding to a WaitGroup while calling wait is a race condition. Surrounding
this in a mutex should solve the problem. Since context is used for
cancellation on all sockets, cleanup should occur properly.

See: https://github.com/coder/coder/runs/5701221057?check_suite_focus=true#step:10:98
2022-03-26 13:53:50 -05:00
Steven Masley 591523a078
chore: Move httpapi, httpmw, & database into `coderd` (#568)
* chore: Move httpmw to /coderd directory
httpmw is specific to coderd and should be scoped under coderd

* chore: Move httpapi to /coderd directory
httpapi is specific to coderd and should be scoped under coderd

* chore: Move database  to /coderd directory
database is specific to coderd and should be scoped under coderd

* chore: Update codecov & gitattributes for generated files
* chore: Update Makefile
2022-03-25 16:07:45 -05:00
Kyle Carberry 6be949a88e
feat: Separate workspace agent for tests (#567)
This adds tests for Google Cloud authentication, and lays
the ground-work for future agent auth types in the future.
2022-03-25 14:48:08 -05:00
Kyle Carberry a06821c103
feat: Update Coder Terraform Provider to v0.2.1 (#563)
This update exposes the workspace name and owner, and changes
authentication methods to be explicit. Implicit authentication
added unnecessary complexity and introduced inconsistency.
2022-03-25 16:34:45 +00:00
Kyle Carberry ddd86ab547
feat: Add systemd service and production deployment (#545)
* feat: Add systemd service and production deployment

This modifies CI to use a dpkg produced from release to update and
run Coder on a tiny VM in GCP.

It's intentionally kept simple, because customers should
be able to get this same easy install experience.

* Update globalSetup.ts

* Update globalSetup.ts

* Update globalSetup.ts

* Update coder.yaml

* Use pinned version of Go
2022-03-24 15:07:33 +00:00
Kyle Carberry d665263d37
fix: Improve coverage by uploading PostgreSQL tests (#532)
This also adds a test for workspace creation via the CLI.
2022-03-23 10:03:28 -05:00
Kyle Carberry 26d24f4508
chore: Improve CI builds by caching Go modules (#528)
* chore: Improve CI builds by caching Go modules

* Skip running with `race` on non-Linux systems

* Fix darwin file descriptor error

* Fix log after close

* Improve PostgreSQL test speeds

* Fix parallel connections with PostgreSQL tests

* Fix CI flake

* Separate test/go into PostgreSQL
2022-03-22 17:09:04 -05:00
Kyle Carberry c451f4e685
feat: Add templates to create working release (#422)
* Add templates

* Move API structs to codersdk

* Back to green tests!

* It all works, but now with tea! 🧋

* It works!

* Add cancellation to provisionerd

* Tests pass!

* Add deletion of workspaces and projects

* Fix agent lock

* Add clog

* Fix linting errors

* Remove unused CLI tests

* Rename daemon to start

* Fix leaking command

* Fix promptui test

* Update agent connection frequency

* Skip login tests on Windows

* Increase tunnel connect timeout

* Fix templater

* Lower test requirements

* Fix embed

* Disable promptui tests for Windows

* Fix write newline

* Fix PTY write newline

* Fix CloseReader

* Fix compilation on Windows

* Fix linting error

* Remove bubbletea

* Cleanup readwriter

* Use embedded templates instead of serving over API

* Move templates to examples

* Improve workspace create flow

* Fix Windows build

* Fix tests

* Fix linting errors

* Fix untar with extracting max size

* Fix newline char
2022-03-22 13:17:50 -06:00
Colin Adler 706e60bb3b
chore: pluralize table names (#463) 2022-03-17 18:10:46 -05:00
Bryan ec077c6191
refactor: Migrate from Next.js to pure webpack config (#360)
Fix for #348 - migrate our NextJS project to a pure webpack project w/ a single bundle

- [x] Switch from `next/link` to `react-router-dom`'s link 

> This part was easy - just change the import to `import { Link } from "react-router-dom"` and `<Link href={...} />` to `<Link to={...} />`

- [x] Switch from `next/router` to `react-router-dom`'s paradigms (`useNavigation`, `useLocation`, and `useParams`)

> `router.push` can be converted to `navigate(...)` (provided by the `useNavigate` hook)
> `router.replace` can be converted `navigate(..., {replace: true})` 
>  Query parameters (`const { query } = useRouter`) can be converted to `const query = useParams()`)

- [x] Implement client-side routing with `react-router-dom`

> Parameterized routes in NextJS like `projects/[organization]/[project]` would look like:
> ```
>               <Route path="projects">
>                    <Route path=":organization/:project">
>                    <Route index element={<ProjectPage />} />
>                  </Route>
>               </Route>
> ```

I've hooked up a `build:analyze` command that spins up a server to show the bundle size:
<img width="1303" alt="image" src="https://user-images.githubusercontent.com/88213859/157496889-87c5fdcd-fad1-4f2e-b7b6-437aebf99641.png">

The bundle looks OK, but there are some opportunities for improvement - the heavy-weight dependencies, like React, ReactDOM, Material-UI, and lodash could be brought in via a CDN: https://stackoverflow.com/questions/50645796/how-to-import-reactjs-material-ui-using-a-cdn-through-webpacks-externals
2022-03-12 12:51:05 -08:00
Kyle Carberry 3eb6fb75d6
feat: Automate releases with goreleaser (#404) 2022-03-07 19:50:37 -06:00
Kyle Carberry e1205a033f
fix: Test flake for resolving instance identity auth (#405) 2022-03-07 18:38:30 -06:00
Kyle Carberry bf0ae8f573
feat: Refactor API routes to use UUIDs instead of friendly names (#401)
* Add client for agent

* Cleanup code

* Fix linting error

* Rename routes to be simpler

* Rename workspace history to workspace build

* Refactor HTTP middlewares to use UUIDs

* Cleanup routes

* Compiles!

* Fix files and organizations

* Fix querying

* Fix agent lock

* Cleanup database abstraction

* Add parameters

* Fix linting errors

* Fix log race

* Lock on close wait

* Fix log cleanup

* Fix e2e tests

* Fix upstream version of opencensus-go

* Update coderdtest.go

* Fix coverpkg

* Fix codecov ignore
2022-03-07 11:40:54 -06:00
Kyle Carberry b6017a7cc5
feat: Add destroy to workspace provision job (#373)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* fix: Convert all jobs to use a common resource and agent type

This enables a consistent API for project import and provisioned resources.

* Add "coder_workspace" data source

* feat: Remove magical parameters from being injected

This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.

* feat: Add graceful exits to provisionerd

Terraform (or other provisioners) may need to cleanup state, or
cancel actions before exit. This adds the ability to gracefully
exit provisionerd.

* Fix cancel error check

* feat: Add destroy to workspace provision job

This enables the full flow of create/update/delete.
2022-02-28 19:20:36 +00:00
Kyle Carberry 9d2803e07a
feat: Add graceful exits to provisionerd (#372)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* fix: Convert all jobs to use a common resource and agent type

This enables a consistent API for project import and provisioned resources.

* Add "coder_workspace" data source

* feat: Remove magical parameters from being injected

This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.

* feat: Add graceful exits to provisionerd

Terraform (or other provisioners) may need to cleanup state, or
cancel actions before exit. This adds the ability to gracefully
exit provisionerd.

* Fix cancel error check
2022-02-28 18:40:49 +00:00
Kyle Carberry e5c95552cd
feat: Remove magical parameters from being injected (#371)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* fix: Convert all jobs to use a common resource and agent type

This enables a consistent API for project import and provisioned resources.

* Add "coder_workspace" data source

* feat: Remove magical parameters from being injected

This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.
2022-02-28 18:26:01 +00:00
Kyle Carberry bd0293aff9
fix: Convert all jobs to use a common resource and agent type (#369)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* fix: Convert all jobs to use a common resource and agent type

This enables a consistent API for project import and provisioned resources.
2022-02-28 18:00:52 +00:00
Kyle Carberry 35ae532f7c
feat: Add Terraform Provider for agent resources (#368)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* Update codersdk/files.go

Co-authored-by: Bryan <bryan@coder.com>

Co-authored-by: Bryan <bryan@coder.com>
2022-02-28 17:16:44 +00:00
Kyle Carberry 8958b641e9
feat: Add agent authentication based on instance ID (#336)
* feat: Add agent authentication based on instance ID

Each cloud has it's own unique instance identity signatures, which
can be used for zero-token authentication. This change adds support
for tracking by "instance_id", and automatically authenticating
with Google Cloud.

* Add test for CLI

* Fix workspace agent request name

* Fix race with adding to wait group

* Fix name of instance identity token
2022-02-21 20:36:29 +00:00
Kyle Carberry 67613da86d
feat: Add "projects list" command to the CLI (#333)
This adds a WorkspaceOwnerCount parameter returned from the
projects API. It's helpful to display the amount of usage
a specific project has.
2022-02-21 12:47:08 -06:00
Kyle Carberry d04570ad29
fix: Use sync.WaitGroup to await hijacked HTTP connections (#337)
WebSockets hijack the HTTP connection from the server, causing
server.Close() to not wait for these connections to fully cleanup.

This adds a global wait-group to the coderd API, which ensures all
WebSocket HTTP handlers have properly exited before returning.
2022-02-20 16:29:16 -06:00
Kyle Carberry 91bf8636fb
feat: Add workspace agent for SSH (#318)
* feat: Add workspace agent for SSH

This adds the initial agent that supports TTY
and execution over SSH. It functions across MacOS,
Windows, and Linux.

This does not handle the coderd interaction yet,
but does setup a simple path forward.

* Fix pty tests on Windows

* Fix log race

* Lock around dial error to fix log output

* Fix context return early

* fix: Leaking yamux session after HTTP handler is closed

Closes #317. We depended on the context canceling the yamux connection,
but this isn't a sync operation. Explicitly calling close ensures the
handler waits for yamux to complete before exit.

* Lock around close return

* Force failure with log

* Fix failed handler

* Upgrade dep

* Fix defer inside loops

* Fix context cancel for HTTP requests

* Fix resize
2022-02-18 23:13:32 -06:00
Kyle Carberry 65de96c8b4
fix: Leaking yamux session after HTTP handler is closed (#329)
* fix: Leaking yamux session after HTTP handler is closed

Closes #317. The httptest server cancels the context after the connection
is closed, but if a connection takes a long time to close, the request
would never end. This applies a context to the entire listener that cancels
on test cleanup.

After discussion with @bryphe-coder, reducing the parallel limit on
Windows is likely to reduce failures as well.

* Switch to windows-2022 to improve decompression

* Invalidate cache on matrix OS
2022-02-18 22:06:56 -06:00
Bryan 3f7781403d
feat: Login via CLI (#298)
Fixes #210 - this isPR implements `coder login` in the case where the default user is already created.

This change adds:
- A prompt in the case where there is not an initial user that opens the server URL + requests a session token
  - This ports over some code from v1 for the `openURL` and `isWSL` functions to support opening the browser
- A `/api/v2/api-keys` endpoint that can be `POST`'d to in order to request a new api key for a user
  - This route was inspired by the v1 functionality
- A `cli-auth` route + page that shows the generated api key
- Tests for the new code + storybook for the new UI

The `/cli-auth` route, like in v1, is very minimal:

<img width="624" alt="Screen Shot 2022-02-16 at 5 05 07 PM" src="https://user-images.githubusercontent.com/88213859/154384627-78ab9841-27bf-490f-9bbe-23f8173c9e97.png">

And the terminal UX looks like this:

![2022-02-16 17 13 29](https://user-images.githubusercontent.com/88213859/154385225-509c78d7-840c-4cab-8f1e-074fede8f97e.gif)
2022-02-17 20:09:33 -08:00
Kyle Carberry 503d09c149
feat: Add support for executing processes with Windows ConPty (#311)
* Initial agent

* fix: Use buffered reader in peer to fix ShortBuffer

This prevents a io.ErrShortBuffer from occurring when the byte
slice being read is smaller than the chunks sent from the opposite
pipe.

This makes sense for unordered connections, where transmission is
not guarunteed, but does not make sense for TCP-like connections.

We use a bufio.Reader when ordered to ensure data isn't lost.

* SSH server works!

* Start Windows support

* Something works

* Refactor pty package to support Windows spawn

* SSH server now works on Windows

* Fix non-Windows

* Fix Linux PTY render

* FIx linux build tests

* Remove agent and wintest

* Add test for Windows resize

* Fix linting errors

* Add Windows environment variables

* Add strings import

* Add comment for attrs

* Add goleak

* Add require import
2022-02-17 10:44:49 -06:00
Kyle Carberry 154b9bce57
feat: Add "coder projects create" command (#246)
* Refactor parameter parsing to return nil values if none computed

* Refactor parameter to allow for hiding redisplay

* Refactor parameters to enable schema matching

* Refactor provisionerd to dynamically update parameter schemas

* Refactor job update for provisionerd

* Handle multiple states correctly when provisioning a project

* Add project import job resource table

* Basic creation flow works!

* Create project fully works!!!

* Only show job status if completed

* Add create workspace support

* Replace Netflix/go-expect with ActiveState

* Fix linting errors

* Use forked chzyer/readline

* Add create workspace CLI

* Add CLI test

* Move jobs to their own APIs

* Remove go-expect

* Fix requested changes

* Skip workspacecreate test on windows
2022-02-12 13:34:04 -06:00
Kyle Carberry 07fe5ced68
feat: Add "coder" CLI (#221)
* feat: Add "coder" CLI

* Add CLI test for login

* Add "bin/coder" target to Makefile

* Update promptui to fix race

* Fix error scope

* Don't run CLI tests on Windows

* Fix requested changes
2022-02-10 08:33:27 -06:00
Bryan 78bf4c6d21
feat: nextrouter pkg to handle nextjs routing rules (#167)
An issue came up last week... our `embed.go` strategy doesn't handle dynamic NextJS-style routes! This is a blocker, because I'm aiming to set up CD on Monday, and the v2 UI makes heavy use of dynamic routing.

As a potential solution, this implements a go pkg `nextrouter` that serves `html` files, but respecting the dynamic routing behavior of NextJS:
- Files that have square brackets - ie `[providers]` provide a single-level dynamic route
- Files that have `[[...` prefix - ie `[[...any]]` - are catch-all routes.
- Files should be preferred over folders (ie, `providers.html` is preferred over `/providers`)
- Fixes the trailing-slash bug we hit in the previous `embed` strategy

This also integrates with `slog.Logger` for tracing, and handles injecting template parameters - a feature we need in v1 and v2 to be able to inject stuff like CSRF tokens.

This implements testing by using an in-memory file-system, so that we can exercise all of these cases.

In addition, this adjust V2's `embed.go` strategy to use `nextrouter`, which simplifies that file considerably. I'm tempted to factor out the `secureheaders` logic into a separate package, too.

If this works OK, it could be used for V1 too (although that scenario is more complex due to our hybrid-routing strategy). Based on our FE variety meeting, there's always a chance we could move away from NextJS in v1 - if that's the case, this router will still work and be more tested than our previous strategy (it just won't make use of dynamic routing). So I figured this was worth doing to make sure we can make forward progress in V2.
2022-02-08 17:01:19 -08:00
Kyle Carberry 7364933e65
refactor: Allow provisioner jobs to be disconnected from projects (#194)
* Nest jobs under an organization

* Rename project parameter to parameter schema

* Update references when computing project parameters

* Add files endpoint

* Allow one-off project import jobs

* Allow variables to be injected that are not defined by the schema

* Update API to use jobs first

* Fix CLI tests

* Fix linting

* Fix hex length for files table

* Reduce memory allocation for windows
2022-02-08 12:00:44 -06:00
Kyle Carberry 795bba2af4
feat: Add dry run for provisioners (#178)
* refactor: Rename ProjectParameter to ProjectVersionParameter

This was confusing with ParameterValue before. It still is a bit,
but this should help distinguish scope.

* Add project version resources table

* Allow project parameters to optionally have user and workspace

* Add dry run for provisioners

* Add resource detection on project import
2022-02-07 19:35:18 -06:00
Kyle Carberry 13360e2e2a
fix: Insert provisioner job logs async (#185)
The context could be cancelled after a log is sent, which resulted
in a failure. This prevent's that from occurring by using the
background context.
2022-02-07 14:14:15 -08:00
Kyle Carberry d55231cc0f
refactor: Rename ProjectParameter to ProjectVersionParameter (#170)
This was confusing with ParameterValue before. It still is a bit,
but this should help distinguish scope.
2022-02-07 21:40:08 +00:00
Kyle Carberry ed705f6af2
refactor: Generalize log ownership to allow for scratch jobs (#182)
* refactor: Generalize log ownership to allow for scratch jobs

Importing may fail when creating a project. We don't want to lose this output,
but we don't want to allow users to create a failing project.

This generalizes logs to soon enable one-off situations where a user can upload
their archive, create a project, and watch the output parse to completion.

* Improve file table schema by using hash

* Fix racey test by allowing logs before

* Add debug logging for PostgreSQL insert
2022-02-07 15:32:37 -06:00
Kyle Carberry 3002a27dd1
ci: Replace DataDog CI with custom upload script (#169)
* ci: Replace DataDog CI with custom upload script

This will reduce CI time by ~6 minutes across all of
our runners. It's a bit janky, but I believe worth
the slight maintainance burden.

* Fix test race when job would complete too early

* Fix job cancelation override

* Fix race where provisioner job is inserted before project version
2022-02-07 17:32:32 +00:00
Kyle Carberry 1796dc6c2f
chore: Add test helpers to improve coverage (#166)
* chore: Rename ProjectHistory to ProjectVersion

Version more accurately represents version storage. This
forks from the WorkspaceHistory name, but I think it's
easier to understand Workspace history.

* Rename files

* Standardize tests a bit more

* Remove Server struct from coderdtest

* Improve test coverage for workspace history

* Fix linting errors

* Fix coderd test leak

* Fix coderd test leak

* Improve workspace history logs

* Standardize test structure for codersdk

* Fix linting errors

* Fix WebSocket compression

* Update coderd/workspaces.go

Co-authored-by: Bryan <bryan@coder.com>

* Add test for listing project parameters

* Cache npm dependencies with setup node

* Remove windows npm cache key

Co-authored-by: Bryan <bryan@coder.com>
2022-02-05 18:24:51 -06:00
Kyle Carberry f19770b2c6
chore: Rename ProjectHistory to ProjectVersion (#165)
* chore: Rename ProjectHistory to ProjectVersion

Version more accurately represents version storage. This
forks from the WorkspaceHistory name, but I think it's
easier to understand Workspace history.

* Rename files
2022-02-04 20:47:58 -06:00
Kyle Carberry 7a7f664727
test: Invalid error scope caused race in WorkspaceHistoryLogs (#164) 2022-02-04 17:20:55 -06:00
Kyle Carberry 682238d384
feat: Add echo provisioner (#162)
This replaces the cdr-basic provisioner type with
"echo". It reads binary data from the directory
and returns the responses in order.

This is used to test project and workspace job logic.
2022-02-04 16:51:54 -06:00
Kyle Carberry 65de6eef9c
feat: Add streaming endpoint for workspace history (#157)
* feat: Add parameter querying to the API

* feat: Add streaming endpoint for workspace history

Enables a buildlog-like flow for reading job output.

* Fix empty parameter source and destination

* Add comment for usage of workspace history logs endpoint
2022-02-04 19:36:43 +00:00
Kyle Carberry c3bae6704b
feat: Add parameter querying to the API (#143) 2022-02-04 12:11:24 -06:00
Kyle Carberry c65850b654
fix: Simplify provisionerd job acquire (#158)
This uses a simple channel to detect whether a
job is running or not, and moves all cancels
to be in goroutines.
2022-02-03 19:13:22 -06:00
Kyle Carberry e75bde4e31
feat: Add provisionerdaemon to coderd (#141)
* feat: Add history middleware parameters

These will be used for streaming logs, checking status,
and other operations related to workspace and project
history.

* refactor: Move all HTTP routes to top-level struct

Nesting all structs behind their respective structures
is leaky, and promotes naming conflicts between handlers.

Our HTTP routes cannot have conflicts, so neither should
function naming.

* Add provisioner daemon routes

* Add periodic updates

* Skip pubsub if short

* Return jobs with WorkspaceHistory

* Add endpoints for extracting singular history

* The full end-to-end operation works

* fix: Disable compression for websocket dRPC transport (#145)

There is a race condition in the interop between the websocket and `dRPC`: https://github.com/coder/coder/runs/5038545709?check_suite_focus=true#step:7:117 - it seems both the websocket and dRPC feel like they own the `byte[]` being sent between them. This can lead to data races, in which both `dRPC` and the websocket are writing.

This is just tracking some experimentation to fix that race condition

## Run results: ##
- Run 1: peer test failure
- Run 2: peer test failure
- Run 3: `TestWorkspaceHistory/CreateHistory`  - https://github.com/coder/coder/runs/5040858460?check_suite_focus=true#step:8:45
```
status code 412: The provided project history is running. Wait for it to complete importing!`
```
- Run 4: `TestWorkspaceHistory/CreateHistory` - https://github.com/coder/coder/runs/5040957999?check_suite_focus=true#step:7:176
```
    workspacehistory_test.go:122: 
        	Error Trace:	workspacehistory_test.go:122
        	Error:      	Condition never satisfied
        	Test:       	TestWorkspaceHistory/CreateHistory
```
- Run 5: peer failure
- Run 6: Pass  
- Run 7: Peer failure

## Open Questions: ##

### Is `dRPC` or `websocket` at fault for the data race?

It looks like this condition is specifically happening when `dRPC` decides to [`SendError`]). This constructs a new byte payload from [`MarshalError`](f6e369438f/drpcwire/error.go (L15)) - so `dRPC` has created this buffer and owns it.

From `dRPC`'s perspective, the callstack looks like this:
- [`sendPacket`](f6e369438f/drpcstream/stream.go (L253))
  - [`writeFrame`](f6e369438f/drpcwire/writer.go (L65))
    - [`AppendFrame`](f6e369438f/drpcwire/packet.go (L128))
      - with finally the data race happening here:
```go
// AppendFrame appends a marshaled form of the frame to the provided buffer.
func AppendFrame(buf []byte, fr Frame) []byte {
...
	out := buf
	out = append(out, control).   // <---------
```

This should be fine, since `dPRC` create this buffer, and is taking the byte buffer constructed from `MarshalError` and tacking a bunch of headers on it to create a proper frame.

Once `dRPC` is done writing, it _hangs onto the buffer and resets it here__: f6e369438f/drpcwire/writer.go (L73)

However... the websocket implementation, once it gets the buffer, it runs a `statelessDeflate` [here](8dee580a7f/write.go (L180)), which compresses the buffer on the fly. This functionality actually [mutates the buffer in place](a1a9cfc821/flate/stateless.go (L94)), which is where get our race.

In the case where the `byte[]` aren't being manipulated anywhere else, this compress-in-place operation would be safe, and that's probably the case for most over-the-wire usages. In this case, though, where we're plumbing `dRPC` -> websocket, they both are manipulating it (`dRPC` is reusing the buffer for the next `write`, and `websocket` is compressing on the fly).

### Why does cloning on `Read` fail?

Get a bunch of errors like:
```
2022/02/02 19:26:10 [WARN] yamux: frame for missing stream: Vsn:0 Type:0 Flags:0 StreamID:0 Length:0
2022/02/02 19:26:25 [ERR] yamux: Failed to read header: unexpected EOF
2022/02/02 19:26:25 [ERR] yamux: Failed to read header: unexpected EOF
2022/02/02 19:26:25 [WARN] yamux: frame for missing stream: Vsn:0 Type:0 Flags:0 StreamID:0 Length:0
```

# UPDATE:

We decided we could disable websocket compression, which would avoid the race because the in-place `deflate` operaton would no longer be run. Trying that out now:

- Run 1:  
- Run 2: https://github.com/coder/coder/runs/5042645522?check_suite_focus=true#step:8:338
- Run 3:  
- Run 4: https://github.com/coder/coder/runs/5042988758?check_suite_focus=true#step:7:168
- Run 5: 

* fix: Remove race condition with acquiredJobDone channel (#148)

Found another data race while running the tests: https://github.com/coder/coder/runs/5044320845?check_suite_focus=true#step:7:83

__Issue:__ There is a race in the p.acquiredJobDone chan - in particular, there can be a case where we're waiting on the channel to finish (in close) with <-p.acquiredJobDone, but in parallel, an acquireJob could've been started, which would create a new channel for p.acquiredJobDone. There is a similar race in `close(..)`ing the channel, which also came up in test runs.

__Fix:__ Instead of recreating the channel everytime, we can use `sync.WaitGroup` to accomplish the same functionality - a semaphore to make close wait for the current job to wrap up.

* fix: Bump up workspace history timeout (#149)

This is an attempted fix for failures like: https://github.com/coder/coder/runs/5043435263?check_suite_focus=true#step:7:32

Looking at the timing of the test:
```
    t.go:56: 2022-02-02 21:33:21.964 [DEBUG]	(terraform-provisioner)	<provision.go:139>	ran apply
    t.go:56: 2022-02-02 21:33:21.991 [DEBUG]	(provisionerd)	<provisionerd.go:162>	skipping acquire; job is already running
    t.go:56: 2022-02-02 21:33:22.050 [DEBUG]	(provisionerd)	<provisionerd.go:162>	skipping acquire; job is already running
    t.go:56: 2022-02-02 21:33:22.090 [DEBUG]	(provisionerd)	<provisionerd.go:162>	skipping acquire; job is already running
    t.go:56: 2022-02-02 21:33:22.140 [DEBUG]	(provisionerd)	<provisionerd.go:162>	skipping acquire; job is already running
    t.go:56: 2022-02-02 21:33:22.195 [DEBUG]	(provisionerd)	<provisionerd.go:162>	skipping acquire; job is already running
    t.go:56: 2022-02-02 21:33:22.240 [DEBUG]	(provisionerd)	<provisionerd.go:162>	skipping acquire; job is already running
    workspacehistory_test.go:122: 
        	Error Trace:	workspacehistory_test.go:122
        	Error:      	Condition never satisfied
        	Test:       	TestWorkspaceHistory/CreateHistory
```

It  appears that the `terraform apply` job had just finished - with less than a second to spare until our `require.Eventually` completes - but there's still work to be done (ie, collecting the state files). So my suspicion is that terraform might, in some cases, exceed our 5s timeout.

Note that in the setup for this test - there is a similar project history wait that waits for 15s, so I borrowed that here.

In the future - we can look at potentially using a simple echo provider to exercise this in the unit test, in a way that is more reliable in terms of timing. I'll log an issue to track that.

Co-authored-by: Bryan <bryan@coder.com>
2022-02-03 20:34:50 +00:00
Kyle Carberry 177eba87b6
refactor: Move all HTTP routes to top-level struct (#130)
* feat: Add history middleware parameters

These will be used for streaming logs, checking status,
and other operations related to workspace and project
history.

* refactor: Move all HTTP routes to top-level struct

Nesting all structs behind their respective structures
is leaky, and promotes naming conflicts between handlers.

Our HTTP routes cannot have conflicts, so neither should
function naming.
2022-02-01 22:15:26 +00:00
Jonathan Yu 58f3e439dd
chore: upgrade from chi 1.5.4 to 5.0.7 (#109) 2022-01-30 19:13:14 -08:00
Kyle Carberry 9db5fb0952
refactor: Improve handshake resiliency of peer (#95)
* fix: Synchronize peer logging with a channel

We were depending on the close mutex to properly
report connection state. This ensures the RTC
connection is properly closed before returning.

* Disable pion logging

* Remove buffer

* Try ICE servers

* Remove flushed

* Add diagram explaining handshake

* Fix candidate accept ordering

* Add debug logging to peerbroker

* Fix send ordering

* Lock adding ICE candidate

* Add test for negotiating out of order

* Reduce connection to a single negotiation channel

* Improve test times by pre-installing Terraform

* Lock remote session description being applied

* Organize conn

* Revert to multi-channel setup

* Properly close ICE gatherer

* Improve comments

* Try removing buffered candidates

* Buffer local and remote messages

* Log dTLS transport state

* Add pion logging
2022-01-30 20:11:18 -06:00
Kyle Carberry b3c5bb3576
feat: Compute project build parameters (#82)
* feat: Add parameter and jobs database schema

This modifies a prior migration which is typically forbidden,
but because we're pre-production deployment I felt grouping
would be helpful to future contributors.

This adds database functions that are required for the provisioner
daemon and job queue logic.

* feat: Compute project build parameters

Adds a projectparameter package to compute build-time project
values for a provided scope.

This package will be used to return which variables are being
used for a build, and can visually indicate the hierarchy to
a user.

* Fix terraform provisioner

* Improve naming, abstract inject to consume scope

* Run CI on all branches
2022-01-29 17:45:42 -06:00
Kyle Carberry 5b01f615eb
feat: Add APIs for querying workspaces (#61)
* Add SQL migration

* Add query functions for workspaces

* Add create routes

* Add tests for codersdk

* Add workspace parameter route

* Add workspace query

* Move workspace function

* Add querying for workspace history

* Fix query

* Fix syntax error

* Move workspace routes

* Fix version

* Add CLI tests

* Fix syntax error

* Remove error

* Fix history error

* Add new user test

* Fix test

* Lower target to 70%

* Improve comments

* Add comment
2022-01-25 19:52:58 +00:00
Bryan 69d88b4a6d
feat: Add Sign-out functionality (#46)
#37 implemented the Sign-_in_ flow, but there wasn't a Sign-_out_ flow as part of that PR (aside from letting the cookie expire... or manually deleting the cookie...), which is obviously not ideal.

This PR implements a basic sign-out flow, along with a very simple user dropdown:
![2022-01-21 18 09 14](https://user-images.githubusercontent.com/88213859/150620847-94e4d22f-1dcf-451e-8b4a-cec24702ea6c.gif)

Bringing in a few pruned down components for the `<UserDropdown />` to integrate into the `<NavBar />`.

In addition, this also implements a simple back-end API for `/logout` which just clears the session token.
2022-01-24 17:09:39 -08:00
Kyle Carberry a44056cff5
feat: Add project API endpoints (#51)
* feat: Add project models

* Add project query functions

* Add organization parameter query

* Add project URL parameter parse

* Add project create and list endpoints

* Add test for organization provided

* Remove unimplemented routes

* Decrease conn timeout

* Add test for UnbiasedModulo32

* Fix expected value

* Add single user endpoint

* Add query for project versions

* Fix linting errors

* Add comments

* Add test for invalid archive

* Check unauthenticated endpoints

* Add check if no change happened

* Ensure context close ends listener

* Fix parallel test run

* Test empty

* Fix organization param comment
2022-01-24 17:07:42 +00:00
Kyle Carberry 8be245616a
feat: Add organizations endpoint for users (#50)
* feat: Add organizations endpoint for users

This moves the /user endpoint to /users/me instead. This
will reduce code duplication.

This adds /users/<name>/organizations to list organizations
a user has access to. It doesn't contain the permissions a
user has over the organizations, but that will come in a future
contribution.

* Fix requested changes

* Fix tests

* Fix timeout

* Add test for UserOrgs

* Add test for userparam getting

* Add test for NoUser
2022-01-22 23:58:10 -06:00
Kyle Carberry 50d8151995
ci: Run tests using PostgreSQL database and mock (#49)
* ci: Run tests using PostgreSQL database and mock

This allows us to use the mock database for quick iterative testing,
and have confidence from CI using a real PostgreSQL database.

PostgreSQL tests are only ran on Linux. They are *really* slow on MacOS
and Windows runners, and don't provide much additional confidence.

* Only run PostgreSQL tests once for speed

* Fix race condition of log after close

Not all resources were cleaned up immediately after a peer connection was
closed. DataChannels could have a goroutine exit after Close() prior to this.

* Fix comment
2022-01-22 21:58:26 +00:00
Kyle Carberry 2654a93132
chore: Fix golangci-lint configuration and patch errors (#34)
* chore: Fix golangci-lint configuration and patch errors

Due to misconfiguration of a linting rules directory, our linter has not been
working properly. This change fixes the configuration issue, and all remaining
linting errors.

* Fix race in peer logging

* Fix race and return

* Lock on bufferred amount low

* Fix mutex lock
2022-01-20 10:00:13 -06:00
Kyle Carberry 6a919aea79
feat: Add authentication and personal user endpoint (#29)
* feat: Add authentication and personal user endpoint

This contribution adds a lot of scaffolding for the database fake
and testability of coderd.

A new endpoint "/user" is added to return the currently authenticated
user to the requester.

* Use TestMain to catch leak instead

* Add userpassword package

* Add WIP

* Add user auth

* Fix test

* Add comments

* Fix login response

* Fix order

* Fix generated code

* Update httpapi/httpapi.go

Co-authored-by: Bryan <bryan@coder.com>

Co-authored-by: Bryan <bryan@coder.com>
2022-01-20 13:46:51 +00:00
Bryan 4dc6e35c24
feat: Bundle UI into coderd; add ./develop.sh script (#28)
This change bundles the static assets like we have for v1 - using the [`embed`](https://pkg.go.dev/embed) go package. Fixes #22 

In addition, it sets up a development script that runs `coderd` locally and serves the front-end, with hot-reloading. The script used is `./develop.sh`:

![2022-01-14 17 30 14](https://user-images.githubusercontent.com/88213859/149603926-f673d3d3-ba12-4eda-bcdd-427252405480.gif)

> NOTE: The UI is still placeholder, of course. Need to start testing out a simple, placeholder flow for the new v2 world as a next step

Summary of changes:
- Add build steps for `go` in the `Makefile`
  - Add a step for production build, in which we use the `embed` tag
  - Add a step for development, which doesn't need the `embed` tag - so we don't need to build the front-end twice
- Add `next export` build step to output front-end artifacts in `out`
- Add a `site` package for `go`
  - Add `embed_static.go` and `embed.go`. This is mostly brought in as-is from v1, except removing some intercom/sentry CSP entries that we aren't using.
- Add a [next development server](https://nextjs.org/docs/advanced-features/custom-server)
- Add a `v2-dev` script, that runs `coderd` and the `next` dev server side-by-side
- Use the `site` package as the fallback handler.
- Add `.gitignore` entries for additional build collateral
2022-01-18 13:13:19 -08:00
Kyle Carberry afc2fa3b62
feat: Add Coder Daemon to serve the API (#18)
* feat: Add v1 schema types

This adds compatibility for sharing data with Coder v1. Since the tables are the same, all CRUD operations should function as expected.

* Add license table

* feat: Add Coder Daemon to serve the API

coderd is a public package which will be consumed by v1 to support running both at the same time. The frontend will need to be compiled and statically served as part of this eventually.

* Fix initial migration

* Move to /api/v2

* Increase peer disconnectedTimeout to reduce flakes on slow machines

* Reduce timeout again

* Fix version for pion/ice
2022-01-13 16:55:28 -06:00