* chore: rename startup logs to agent logs
This also adds a `source` property to every agent log. It
should allow us to group logs and display them nicer in
the UI as they stream in.
* Fix migration order
* Fix naming
* Rename the frontend
* Fix tests
* Fix down migration
* Match enums for workspace agent logs
* Fix inserting log source
* Fix migration order
* Fix logs tests
* Fix psql insert
* fix: remove startup logs eof for streaming
We have external utilities like logstream-kube that may send
logs after an agent shuts down unexpectedly to report additional
information. In a recent change we stopped accepting these logs,
which broke these utilities.
In the future we'll rename startup logs to agent logs or something
more generalized so this is less confusing in the future.
* fix(cli/cliui): handle never ending startup log stream in Agent
---------
Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
This commit reverts some of the changes in #8029 and implements an
alternative method of keeping track of when the startup script has ended
and there will be no more logs.
This is achieved by adding new agent fields for tracking when the agent
enters the "starting" and "ready"/"start_error" lifecycle states. The
timestamps simplify logic since we don't need understand if the current
state is before or after the state we're interested in. They can also be
used to show data like how long the startup script took to execute. This
also allowed us to remove the EOF field from the logs as the
implementation was problematic when we returned the EOF log entry in the
response since requesting _after_ that ID would give no logs and the API
would thus lose track of EOF.
* feat(coderd,agent): send startup log eof at the end
* fix(coderd): fix edge case in startup log pubsub
* fix(coderd): ensure startup logs are closed on lifecycle state change (fallback)
* fix(codersdk): fix startup log channel shared memory bug
* fix(site): remove the EOF log line
* fix(codersdk): wait for subscription in WatchWorkspaceAgentMetadata
* fix(coderd): subscribe before sending initial metadata event
* test(coderd): add retries to TestWorkspaceAgent_Metadata to avoid flake
* feat(api): Add agent shutdown lifecycle states
* feat(agent): Add shutdown_script support
* feat(agent): Add shutdown_script timeout
* feat(site): Support new agent lifecycle states
---
Co-authored-by: Marcin Tojek <marcin@coder.com>
* chore: convert workspace agent stats from json to table
* chore: convert agent stats to use a table
Backwards compatibility becomes hard when all agent stats are in a JSON blob.
We also want to query this table for new agents that are failing health checks
so we can display it in the UI.
* Fix migration using default values
* Add git auth providers schema
* Pipe git auth providers to the schema
* Add git auth providers to the API
* Add gitauth endpoint to query authenticated state
* Add endpoint to query git state
* Use BroadcastChannel to automatically authenticate with Git
* Add error validation for submitting the create workspace form
* Fix panic on template dry-run
* Add tests for the template version Git auth endpoint
* Show error if no gitauth is configured
* Add gitauth to cliui
* Fix unused method receiver
* Fix linting errors
* Fix dbauthz querier test
* Fix make gen
* Add JavaScript test for git auth
* Fix bad error message
* Fix provisionerd test race
See https://github.com/coder/coder/actions/runs/4277960646/jobs/7447232814
* Fix requested changes
* Add comment to CreateWorkspacePageView
* fix(tailnet): Improve start and close to detect connection races
* fix: Prevent agentConn use before ready via AwaitReachable
* fix(tailnet): Ensure connstats are closed on conn close
* fix(codersdk): Use AwaitReachable in DialWorkspaceAgent
* fix(tailnet): Improve logging via slog.Helper()
* chore: rename `AgentConn` to `WorkspaceAgentConn`
The codersdk was becoming bloated with consts for the workspace
agent that made no sense to a reader. `Tailnet*` is an example
of these consts.
* chore: remove `Get` prefix from *Client functions
* chore: remove `BypassRatelimits` option in `codersdk.Client`
It feels wrong to have this as a direct option because it's so infrequently
needed by API callers. It's better to directly modify headers in the two
places that we actually use it.
* Merge `appearance.go` and `buildinfo.go` into `deployment.go`
* Merge `experiments.go` and `features.go` into `deployment.go`
* Fix `make gen` referencing old type names
* Merge `error.go` into `client.go`
`codersdk.Response` lived in `error.go`, which is wrong.
* chore: refactor workspace agent functions into agentsdk
It was odd conflating the codersdk that clients should use
with functions that only the agent should use. This separates
them into two SDKs that are closely coupled, but separate.
* Merge `insights.go` into `deployment.go`
* Merge `organizationmember.go` into `organizations.go`
* Merge `quota.go` into `workspaces.go`
* Rename `sse.go` to `serversentevents.go`
* Rename `codersdk.WorkspaceAppHostResponse` to `codersdk.AppHostResponse`
* Format `.vscode/settings.json`
* Fix outdated naming in `api.ts`
* Fix app host response
* Fix unsupported type
* Fix imported type
* feat: Validate Git tokens before consuming them
This works the exact same way that the Git credential manager does. It ensures the user token is valid before returning it to the client.
It's been manually tested on GitHub, GitLab, and BitBucket.
* Fix requested changes
* fix: Improve tailnet connections by reducing timeouts
This awaits connection ping before running a dial. Before,
we were hitting the TCP retransmission and handshake timeouts,
which could intermittently add 1 or 5 seconds to a connection
being initialized.
* Update Tailscale
* feat: Add connection_timeout and troubleshooting_url to agent
This commit adds the connection timeout and troubleshooting url fields
to coder agents.
If an initial connection cannot be established within connection timeout
seconds, then the agent status will be marked as `"timeout"`.
The troubleshooting URL will be present, if configured in the Terraform
template, it can be presented to the user when the agent state is either
`"timeout"` or `"disconnected"`.
Fixes#4678
This feature is used by the coder agent to exchange a new token. By
protecting the SessionToken via mutex we ensure there are no data races
when accessing it.
* fix: Refactor agent to consume API client
This simplifies a lot of code by creating an interface for
the codersdk client into the agent. It also moves agent
authentication code so instance identity will work between
restarts.
Fixes#3485 and #4082.
* Fix client reconnections
* feat: HA tailnet coordinator
* fixup! feat: HA tailnet coordinator
* fixup! feat: HA tailnet coordinator
* remove printlns
* close all connections on coordinator
* impelement high availability feature
* fixup! impelement high availability feature
* fixup! impelement high availability feature
* fixup! impelement high availability feature
* fixup! impelement high availability feature
* Add replicas
* Add DERP meshing to arbitrary addresses
* Move packages to highavailability folder
* Move coordinator to high availability package
* Add flags for HA
* Rename to replicasync
* Denest packages for replicas
* Add test for multiple replicas
* Fix coordination test
* Add HA to the helm chart
* Rename function pointer
* Add warnings for HA
* Add the ability to block endpoints
* Add flag to disable P2P connections
* Wow, I made the tests pass
* Add replicas endpoint
* Ensure close kills replica
* Update sql
* Add database latency to high availability
* Pipe TLS to DERP mesh
* Fix DERP mesh with TLS
* Add tests for TLS
* Fix replica sync TLS
* Fix RootCA for replica meshing
* Remove ID from replicasync
* Fix getting certificates for meshing
* Remove excessive locking
* Fix linting
* Store mesh key in the database
* Fix replica key for tests
* Fix types gen
* Fix unlocking unlocked
* Fix race in tests
* Update enterprise/derpmesh/derpmesh.go
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* Rename to syncReplicas
* Reuse http client
* Delete old replicas on a CRON
* Fix race condition in connection tests
* Fix linting
* Fix nil type
* Move pubsub to in-memory for twenty test
* Add comment for configuration tweaking
* Fix leak with transport
* Fix close leak in derpmesh
* Fix race when creating server
* Remove handler update
* Skip test on Windows
* Fix DERP mesh test
* Wrap HTTP handler replacement in mutex
* Fix error message for relay
* Fix API handler for normal tests
* Fix speedtest
* Fix replica resend
* Fix derpmesh send
* Ping async
* Increase wait time of template version jobd
* Fix race when closing replica sync
* Add name to client
* Log the derpmap being used
* Don't connect if DERP is empty
* Improve agent coordinator logging
* Fix lock in coordinator
* Fix relay addr
* Fix race when updating durations
* Fix client publish race
* Run pubsub loop in a queue
* Store agent nodes in order
* Fix coordinator locking
* Check for closed pipe
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* feat: show listening ports in port forward popup
* Move fetch logic to a machine
* feat: don't show app ports and common non-HTTP ports
Co-authored-by: Bruno Quaresma <bruno@coder.com>
* fix: Add coder user to docker group on installation
This makes for a simpler setup, and reduces the likelihood
a user runs into a strange issue.
* Add wgnet
* Add ping
* Add listening
* Finish refactor to make this work
* Add interface for swapping
* Fix conncache with interface
* chore: update gvisor
* fix tailscale types
* linting
* more linting
* Add coordinator
* Add coordinator tests
* Fix coordination
* It compiles!
* Move all connection negotiation in-memory
* Migrate coordinator to use net.conn
* Add closed func
* Fix close listener func
* Make reconnecting PTY work
* Fix reconnecting PTY
* Update CI to Go 1.19
* Add CLI flags for DERP mapping
* Fix Tailnet test
* Rename ConnCoordinator to TailnetCoordinator
* Remove print statement from workspace agent test
* Refactor wsconncache to use tailnet
* Remove STUN from unit tests
* Add migrate back to dump
* chore: Upgrade to Go 1.19
This is required as part of #3505.
* Fix reconnecting PTY tests
* fix: update wireguard-go to fix devtunnel
* fix migration numbers
* linting
* Return early for status if endpoints are empty
* Update cli/server.go
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* Update cli/server.go
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* Fix frontend entites
* Fix agent bicopy
* Fix race condition for the last node
* Fix down migration
* Fix connection RBAC
* Fix migration numbers
* Fix forwarding TCP to a local port
* Implement ping for tailnet
* Rename to ForceHTTP
* Add external derpmapping
* Expose DERP region names to the API
* Add global option to enable Tailscale networking for web
* Mark DERP flags hidden while testing
* Update DERP map on reconnect
* Add close func to workspace agents
* Fix race condition in upstream dependency
* Fix feature columns race condition
Co-authored-by: Colin Adler <colin1adler@gmail.com>
* fix: Change uses of t.Cleanup -> defer in test bodies
Mixing t.Cleanup and defer can lead to unexpected order of execution.
* fix: Ensure t.Cleanup is not aborted by require
* chore: Add helper annotations
* feat: Add app support
This adds apps as a property to a workspace agent.
The resource is added to the Terraform provider here:
https://github.com/coder/terraform-provider-coder/pull/17
Apps will be opened in the dashboard or via the CLI
with `coder open <name>`. If `command` is specified, a
terminal will appear locally and in the web. If `target`
is specified, the browser will open to an exposed instance
of that target.
* Compare fields in apps test
* Update Terraform provider to use relative path
* Add some basic structure for routing
* chore: Remove interface from coderd and lift API surface
Abstracting coderd into an interface added misdirection because
the interface was never intended to be fulfilled outside of a single
implementation.
This lifts the abstraction, and attaches all handlers to a root struct
named `*coderd.API`.
* Add basic proxy logic
* Add proxying based on path
* Add app proxying for wildcards
* Add wsconncache
* fix: Race when writing to a closed pipe
This is such an intermittent race it's difficult to track,
but regardless this is an improvement to the code.
* fix: Race when writing to a closed pipe
This is such an intermittent race it's difficult to track,
but regardless this is an improvement to the code.
* fix: Race when writing to a closed pipe
This is such an intermittent race it's difficult to track,
but regardless this is an improvement to the code.
* fix: Race when writing to a closed pipe
This is such an intermittent race it's difficult to track,
but regardless this is an improvement to the code.
* Add workspace route proxying endpoint
- Makes the workspace conn cache concurrency-safe
- Reduces unnecessary open checks in `peer.Channel`
- Fixes the use of a temporary context when dialing a workspace agent
* Add embed errors
* chore: Refactor site to improve testing
It was difficult to develop this package due to the
embed build tag being mandatory on the tests. The logic
to test doesn't require any embedded files.
* Add test for error handler
* Remove unused access url
* Add RBAC tests
* Fix dial agent syntax
* Fix linting errors
* Fix gen
* Fix icon required
* Adjust migration number
* Fix proxy error status code
* Fix empty db lookup
Exponential backoff is only enabled if the websocket fails to open. If
the websocket is opened but immediately killed, the agent will try to
immediately reconnect. This is desireable in cases where coderd is being
replaced or network conditions cause the connection to die, but not for
permanent errors.
Abstracting coderd into an interface added misdirection because
the interface was never intended to be fulfilled outside of a single
implementation.
This lifts the abstraction, and attaches all handlers to a root struct
named `*coderd.API`.
* feat: Add web terminal with reconnecting TTYs
This adds a web terminal that can reconnect to resume sessions!
No more disconnects, and no more bad bufferring!
* Add xstate service
* Add the webpage for accessing a web terminal
* Add terminal page tests
* Use Ticker instead of Timer
* Active Windows mode on Windows
This makes setting up git a bit simpler, and users
can always override these values!
We'll probably add a way to disable our Git integration
anyways, so these could be part of that.
This removes split ownership for workspaces. They are now
a resource of organizations and have a designated owner,
which is a user.
This enables simple administration for commands like:
- `coder stop ben/dev`
- `coder build logs colin/arch`
or if we decide to allow administrators to access workspaces,
they could even SSH using this syntax: `coder ssh colin/dev`.
These values were ignored. Environment variables are applied to
new sessions, and are refreshed on reconnect. This is cool because
a workspace could be updated with new environment variables without
requiring a complete start/stop.
The startup script is only ran once regardless of changes, which
feels like the expected behavior.
This enables a "kubernetes_pod" to attach multiple agents that
could be for multiple services. Each agent is required to have
a unique name, so SSH syntax is:
`coder ssh <workspace>.<agent>`
A resource can have zero agents too, they aren't required.
Dean Sheather
Kyle Carberry
Mathias Fredriksson
Jon Ayers
Ammar Bandukwala
Colin Adler
Garrett Delfosse
Spike Curtis
David Wahler