* build/linters: Bump Go to v1.25 and golangci-lint to v2.4.0
* refactor: Update TODO comments for net.Listen and net.DialTimeout; improve variable naming in websocket and exchange methods
* refactor: Rename massageMissingData to backfillMissingData for clarity and update references in RSI and MFI calculations
* fix: Correct typo in TODO comment for net.Listen in RPC server
* I've made a comprehensive overhaul of the Makefile for your Go development workflow.
This update brings a significant series of enhancements to the project's Makefile, aimed at modernizing and improving your Go development experience. This includes improvements to linting, formatting, dependency management, Docker integration, and the overall Makefile structure, style, and usability.
Here are the key improvements:
1. **Linting (`golangci-lint`):**
* **Version & Path:** I've upgraded `golangci-lint` to use the v2 path (`github.com/golangci/golangci-lint/cmd/golangci-lint/v2`) and pinned it to a specific version (`v2.1.6`) for consistency.
* **Target Renaming:** The primary linting target was renamed from `linter` to `lint` to align with common conventions.
* **Command Simplification:** The `lint` target's command was streamlined to `$(LINTBIN) run --verbose`, relying directly on `golangci-lint`'s exit code for success/failure reporting.
* **Redundancy Removal:** A superfluous `go install $(GCTPKG)` command was removed from the `lint` target.
2. **Dockerized Linting (`lint_docker`):**
* A new `lint_docker` target (renamed from `lint-docker` for consistent naming) was introduced to execute `golangci-lint` within a Docker container (`golangci/golangci-lint:v2.1.6`).
* This ensures a consistent linting environment across different developer setups and CI.
* The target uses `$(CURDIR)` for reliable Docker volume mounting.
* It includes the `--verbose` flag for detailed linter output and the `-t` flag to allocate a pseudo-TTY, enabling colored output from the linter.
* It incorporates a check for Docker's availability before attempting to run.
* Whitespace in the Docker check command was normalized for readability (single space after `||`).
3. **Code Formatting (`gofmt` & `gofumpt`):**
* **Default Formatter:** The main `fmt` target now consistently uses `gofmt` for standard Go formatting.
* **Stricter Formatting Option:** A new `gofumpt` target has been added, allowing you to use the stricter `gofumpt` formatter. This target includes a command to automatically install `gofumpt` if it's not already present.
* **File Exclusion Control (`GO_FILES_TO_FORMAT`):**
- A Make variable `GO_FILES_TO_FORMAT` was introduced to precisely define the set of Go files to be processed by the formatters.
- This variable is configured to exclude:
- Generated ORM code (e.g., `./database/models/*`).
- Vendored dependencies (`./vendor/*`).
- Protobuf generated files (`*.pb.go`).
- gRPC gateway generated files (`*.pb.gw.go`).
- Both the `fmt` and `gofumpt` targets now utilize this variable, preventing accidental reformatting of generated or third-party code.
4. **Makefile Structure & General Enhancements:**
* **Redundant Target Removal:** The `get` target was removed, as its functionality was fully covered by the more optimized `install` target.
* **`.PHONY` Declarations:**
- The default `all` target was explicitly added to the `.PHONY` list for improved clarity and robustness.
- All relevant Makefile targets, including newly added and renamed ones (`lint`, `gofumpt`, `lint_docker`, `fmt`), are explicitly declared in the `.PHONY` list, adhering to best practices.
- The order of targets in the main `.PHONY` declaration was adjusted to match the sequence of their definitions in the Makefile, enhancing readability.
These comprehensive changes aim to provide you with a more robust, flexible, consistent, and maintainable build and development environment.
* Fix: Ensure GOPATH is defined in Makefile
This commit enhances the Makefile by ensuring that the GOPATH
variable is defined before being used by LINTBIN and GOFUMPTBIN.
The line `GOPATH ?= $(shell go env GOPATH)` has been added.
This means:
- If GOPATH is set in the environment, that value will be used.
- Otherwise, GOPATH will be set to the output of `go env GOPATH`,
which is the canonical way to determine the Go path.
This change makes the Makefile more robust and less dependent on
your shell environment having GOPATH explicitly set.
---------
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
* Sonic: Add sonic_on build tag
Thinking about other ways to do this, but they amount to the same thing.
It's messy, but I don't have another idea.
* Okx: Remove redundant useAsIs on slices
Slices are automatically used-as-is, so passing true for these types was
unnecessary.
Removing the field simplifies rationalising inverting the flag
* Okx: Unify timestamp response types
* Okx: Change ResetRFQMMPStatus to return a types.Time
* Okx: Move withdrawData type to types
* Okx: Fix AccountConfiguration slice return
* Okx: Improve SendHTTPRequest documentation
* Okx: Extend tests for UseAsIs non-defaults
* Okx: Fix GetPublicUnderlying with sonic 1.12.9
Using **result for slices with useAsItIs causes sonic to fail.
This might be addressed upstream, but it's also not clear what the
unmarshal behaviour for an untyped reference to a typed reference should
be in the RFC, so we could get a golang.org/encoding/json regression on
this too.
There's no harm in fixing this, for consistency, to match our handling
for non-slice []any wrapping to just use the pointer as is.
Note: As of today this requires sonic:main for this to work, because of
the other bug:
```
M go.mod
- github.com/bytedance/sonic v1.12.9
+ github.com/bytedance/sonic v1.12.10-0.20250224121557-e30ac4f2e4fe
```
* Okx: Remove redundant slice check
This code didn't work, and if I make it look at rv.Elem().Kind() it
errors.
Haven't dug too deeply but right now I think we just remove it.
* Okx: Simplify SendHTTPRequest by removing UseAsIs
Looks to be entirely derivable
* Okx: Remove http check that resps must contain data
GetAnnouncementTyeps failing in US because of empty response.
But also any situation where there really is no data.
e.g. GetCandlesticks might return no candlesticks for a period and instrument, because there aren't any.
That shouldn't be an error.
More saliently if you request orders, or something similar.
So, since that check wasn't working before, and it's causing issues now, I'm going to remove it.
* Okx: Fix TestGetAnnouncementTypes failing in US
announcement-types returns empty in the US, where our github actions
run.
That's kinda okay. Just don't test we get any back
* Sonic: Default to sonic off
We've seen too many fatal panics and races with sonic, both in GCT runs
and being reported in sonic, to default to it being turned on right now.
Whilst we have faith sonic will get through these with time, for now the
sensible thing to do for our users is make sonic opt-in.
This also removes any of the conditions around 386, etc.
If someone wants to run with sonic, they can. Most notably if they're
trying out an experimental sonic branch that supports 386, etc.
* build/ci: Update Go to v1.24, golangci-lint to v1.64.5 and fix issues
* Address shazbert's nitters
* linter/config: Fix new linter issue and use versionSize const
* Address gk's nitters and fix additional linter issue after rebase
* Address glorious nits
* staticcheck: Fix additional linter issues after upgrading to Go 1.24.1 and golangci-lint v1.64.6
Also addresses nits
* Improve testing, assertify usage and use common.ErrParsingWSField
* TestCreateNewStrategy: Replace must > should wording
* tag optional sonic and allow full library conversion
* Add workflow and disallow arm and darwin usage
* Add basic hotswap benchmark
* linter: fix
* use bash
* linter: fix?
* Fix whoopsie, add to make file, also add mention in features list.
* test enforcement
* actually read documentation see if this works
* linter: fix
* linter: fix
* sonic: bump tagged version
* encoding/json: drop build tag arch and os filters
* encoding/json: consolidate tests
* encoding/json: log build tag usage
* rm superfluous builds
* glorious/nits: add template change and regen docs
* glorious/nits: update commentary on nolint directive
* glorious/nits: rm init func and log results in main.go
* Test to actually pull flag in
* linter: fix
* thrasher: nits
* gk: nits 4 goflags goooooooooo!
* gk: nits rn
* make sonic default json implementation
* screen 386
* linter: fix
* Add commentary
* glorious: nits Makefile not working
* gk: nits
* gk: nits whoops
* whoopsirino
* mention 32bit systems won't be sonic
* gk: super-duper nit of extremes
---------
Co-authored-by: Ryan O'Hara-Reid <ryan.oharareid@thrasher.io>
* CI/build: Update Go version, linters and fix minor issues
* linters: Add intrange, copyloopvar, additional go vet linters to match gopls and fix issues
* qa: Enforce JSON lint for GCT configs
* Makefile/GHA: Make output more verbose on success, clean duplicate PHONY and rid excess newlines
* Makefile: Use printf for OS cross-compatibility output
* CI/build: Update Go version, linters and fix minor issues
* Bump golangci-lint to v1.56.1
* BinanceUS: Make uint usage consistent
* Throw blank identifiers into the trash
* CI: Add macOS, Windows and Linux arm64 support
* Modify arm64 job name to be inline with the others
* linter: Prevent cache from causing "Cannot open: File exists" issues
* Use setup-go and setup-node's inbuilt caching
* implements futures functions and GRPC functions on new branch
* lint and test fixes
* Fix uneven split pnl. Adds collateral weight test. docs. New clear func
* Test protection if someone has zero collateral
* Uses string instead of double for accuracy
* Fixes old code panic
* context, match, docs
* Addresses Shazniterinos, var names, expanded tests
* Returns subaccount name, provides USD values when offlinecalc
* Fixes oopsie
* Fixes cool bug which allowed made up subaccount results
* Subaccount override on FTX, subaccount results for collateral
* Strenghten collateral account info checks. Improve FTX test
* English is my first language
* Fixes oopsies
* Adds some conceptual futures order details to track PNL
* Initial design of future order processing in the backtester
* Introduces futures concept for collateral and spot/futures config diffs
* Fixes most tests
* Simple designs for collateral funding pair concept
* Expands interface use so much it hurts
* Implements more collateral interfaces
* Adds liquidation, adds strategy, struggles with Binance
* Attempts at getting FTX to work
* Adds calculatePNL as a wrapper function and adds an `IsFutures` asset check
* Successfully loads backtester with collateral currency
* Fails to really get much going for supporting futures
* Merges master changes
* Fleshes out how FTX processes collateral
* Further FTX collateral workings
* hooks up more ftx collateral and pnl calculations
* more funcs to flesh out handling
* Adds more links, just can't fit the pieces together :(
* Greatly expands futures order processing
* Fleshes out position tracker to also handle asset and exchange +testing
* RM linkedOrderID. rn positioncontroller, unexport
* Successfully tracks futures order positions
* Fails to calculate PNL
* Calculates pnl from orders accurately with exception to flipping orders
* Calculates PNL from orders
* Adds another controller layer to make it ez from orderstore
* Backtester now compiles. Adds test coverage
* labels things add scaling collateral test
* Calculates pnl in line with fees
* Mostly accurate PNL, with exception to appending with diff prices
* Adds locks, adds rpc function
* grpc implementations
* Gracefully handles rpc function
* beautiful tests!
* rejiggles tests to polish
* Finishes FTX testing, adds comments
* Exposes collateral calculations to rpc
* Adds commands and testing for rpcserver.go functions
* Increase testing and fix up backtester code
* Returns cool changes to original branch
* end of day fixes
* Fixing some tests
* Fixing tests 🎉
* Fixes all the tests
* Splits the backtester setup and running into different files
* Merge, minor fixes
* Messing with some strategy updates
* Failed understanding at collateral usage
* Begins the creation of cash and carry strategy
* Adds underlying pair, adds filldependentevent for futures
* Completes fill prerequsite event implementation. Can't short though
* Some bug fixes
* investigating funds
* CAN NOW CREATE A SHORT ORDER
* Minor change in short size
* Fixes for unrealised PNL & collateral rendering
* Fixes lint and tests
* Adds some verbosity
* Updates to pnl calc
* Tracks pnl for short orders, minor update to strategy
* Close and open event based on conditions
* Adds pnl data for currency statistics
* Working through PNL calculation automatically. Now panics
* Adds tracking, is blocked from design
* Work to flesh out closing a position
* vain attempts at tracking zeroing out bugs
* woww, super fun new subloggers 🎉
* Begins attempt at automatically handling contracts and collateral based on direction
* Merge master + fixes
* Investigating issues with pnl and holdings
* Minor pnl fixes
* Fixes future position sizing, needs contract sizing
* Can render pnl results, focussing on funding statistics
* tracking candles for futures, but why not btc
* Improves funding statistics
* Colours and stats
* Fixes collateral and snapshot bugs
* Completes test
* Fixes totals bug
* Fix double buy, expand stats, fixes usd totals, introduce interface
* Begins report formatting and calculations
* Appends pnl to receiving curr. Fixes map[time]. accurate USD
* Improves report output rendering
* PNL stats in report. New tests for futures
* Fixes existing tests before adding new coverage
* Test coverage
* Completes portfolio coverage
* Increase coverage exchange, portfolio. fix size bug. NEW CHART
* WHAT IS GOING ON WITH PNL
* Fixes PNL calculation. Adds ability to skip om futures tracking
* minor commit before merge
* Adds basic liquidation to backtester
* Changes liquidation to order based
* Liquidationnnnnn
* Further fleshes out liquidations
* Completes liquidations in a honorable manner. Adds AppendReasonf
* Beginnings of spot futures gap chart. Needs to link currencies to render difference
* Removes fake liquidation. Adds cool new chart
* Fixes somet tests,allows for zero fee value v nil distinction,New tests
* Some annoying test fixes that took too long
* portfolio coverage
* holding coverage, privatisation funding
* Testwork
* boring tests
* engine coverage
* More backtesting coverage
* Funding, strategy, report test coverage
* Completes coverage of report package
* Documentation, fixes some assumptions on asset errors
* Changes before master merge
* Lint and Tests
* defaults to non-coloured rendering
* Chart rendering
* Fixes surprise non-local-lints
* Niterinos to the extremeos
* Fixes merge problems
* The linter splintered across the glinting plinths
* Many nits addressed. Now sells spot position on final candle
* Adds forgotten coverage
* Adds ability to size futures contracts to match spot positions.
* fixes order sell sizing
* Adds tests to sizing. Fixes charting issue
* clint splintered the linters with flint
* Improves stats, stat rendering
* minifix
* Fixes tests and fee bug
* Merge fixeroos
* Microfixes
* Updates orderPNL on first Correctly utilises fees. Adds committed funds
* New base funcs. New order summary
* Fun test updates
* Fix logo colouring
* Fixes niteroonies
* Fix report
* BAD COMMIT
* Fixes funding issues.Updates default fee rates.Combines cashcarry case
* doc regen
* Now returns err
* Fixes sizing bug issue introduced in PR
* Fixes fun fee/total US value bug
* Fix chart bug. Show log charts with disclaimer
* sellside fee
* fixes fee and slippage view
* Fixed slippage price issue
* Fixes calculation and removes rendering
* Fixes stats and some rendering
* Merge fix
* Fixes merge issues
* go mod tidy, lint updates
* New linter attempt
* Version bump in appveyor and makefile
* Regex filename, config fixes, template h2 fixes
* Removes bad stats.
* neatens config builder. Moves filename generator
* Fixes issue where linter wants to fix my spelling
* Fixes pointers and starts
* Bump CI versions
* Specifically set go version as 1.17.x bumps it to 1.18
* Another
* Adjust AppVeyor
* Part 1 of linter issues
* Part 2
* Fix various linters and improvements
* Part 3
* Finishing touches
* Tests and EqualFold
* Fix nitterinos plus bonus requester jobs bump for exchanges with large number of tests
* Fix nitterinos and bump golangci-lint timeout for AppVeyor
* Address nits, ensure all books are returned on err due to syncer regression
* Fix the wiggins
* Fix duplication
* Fix nitterinos
* golangci-lint/CI: Bump versions
Fix remaining linter issues
* Specifically set AppVeyor version
* Fix the infamous typos 👀
* Add go env cmd to AppVeyor
* Add go version cmd to AppVeyor
* Specify AppVeyor image, adjust linters
* Update go get to go install due to deprecation
* Bump golangci-lint timeout time for AppVeyor
* Change NW contract to NQ
* Address nitters
* GetRandomPair -> Pair{}
* Address nits
* Address time nitterinos plus additional tweaks
* More time inception upgrades!
* Bending time and space
* the generator should use the same path as the database/drivers/sqlite3
* remove unused data directory
* generate sqlboiler.json for model generation
* ignore target folder
* Add 32bit build matrix test
* Adjust travis/Makefile
* Set dist back to xenial
* export CGO_ENABLED=1
* Add gcc-multilib
* Sudo installerino
* make check -> make test
* Migrated to goose & sqlboiler
* create tests with sqlboiler
* code clean up
* Added gct -> sqlboiler config gen
* dropped pgx support
* dropped pgx support because who needs connection pools
* reenable sqlite audit tests
* first pass of migration changes
* stuff is broken :D
* sqlboiler :D
* end of date commit
* Added comments code clean up
* revert go module files back to upstream
* bug fix
* pushed go.mod update to use correc goose version
* renamed sqlite to sqlite3 for consistency across codebase and PR feedback changes
* makefile updates
* things are broken end of day commit
* added postgresql test
* use correct database name
* travis fixes for env vars
* travis fixes for env vars
* test fixes
* run migration on test setup
* test adding postgres support to appveyor
* Skip tests on appveyor due to issues with missing binaries
* oh yeah i have to support windows don't i
* bumped goose version up
* add postgres to osx
* fix travis config as osx does not support services move spin up to before_script
* added PGDATA path fix
* pass PG_DATA to pg_ctl
* added initdb to before install
* fixes to wording and bumps up goose version
* who needs ssl anyway
* moved ssl to correct section :D
* bumped goose version up
* unbreak travis
* unbreak travis
* fix if database is disabled in config
* move strings to consts
* converted more strings to const
* improvements to sqlboiler mmodel gen
* Added contrib\sqlboiler file
* sqlboiler windows contrib fixes
* bumped goose version up
* :D whoops
* further fixes to sql models
* further fixes to sql models
* database type fix for config gen
* README update
* go.mod clean up
* added config details for appveyor
* appveyor ordering fix
* force psql9.6
* appveyor config changes
* all the environmen vars
* model changes for psql
* model changes for psql
* sqlite model fixes
* attempt at osx fix
* added error check for migration
* typos and check against goose error instead of string :D
* updated sqlboiler commit id
* bump sqlboiler version again
* set decimal package to @0bb1631
* readme and makefile updates
* bump goose version update readme and add override flag to config gen
* README typo fix and lowered inserts in test down to 20 as we are only testing that inserts work running 200 was unnecessary
* added gctcli command for audit event
* Added debug output toggle to config added both postgres & sqlite support to gctcli command
* Wording changes on errors
* set sqlite to 1 connection to stop locke database issues
* Usage update for order
* README updates with config examples
* go.mod/sum tidy
* removed lines in import second
* removed lines in imports
* convert local time to utc for database and display output
* go mod clean up and error checking to time
* renamed all packages to sqlite3
* added windows command output for sql model gen
* time conversion fix
* time conversion on gctcli
* added audit manager
* Basic database DOA setup
* Added base config file
* added sqlite support and creation of schema
* added basic tests and config entry
* corrected issues of database is disabled
* fixed path for test
* WIP
* Added tests fixed config checking
* reverted files back to upstream
* reverted go.mod files
* no more test test test
* removed local testing details for psql
* hello
* added comments
* increased ping to 30 seconds
* renamed database table and added additional condition around test
* removed database test details
* goimport ran on all files
* WIP
* first attempt at migration
* fixes for migration system
* Migration system logger interface implemented
* fixes to print functions
* added write pooling pass
* gofmt :D
* formatted imports correctly
* removed old code
* added creation of migration
* gofmt
* :D Hello
* ❌🏎️
* maybe one day i will remember to revert go mod files
* checked err return condition correctly
* first changes for PR feedback
* code clean up
* protect Connected with RWmutex & event with mutex
* : D
* we can just pretend like it never happened
* MOved migrations back to source directory and added README
* readme formatting update
* Addd command line override for datadir
* use correct var when creating a migration and confirm folder is created
* Check if database version is newer than latest migration and also you know make migrations work.....
* uses filepath instead of manual path to use correct path seperator
* Add connection message and lower timeout
* Added support for sslmode for psql
* no longer force Close of database instead allow driver to maage
* Added closer func to test output
* sslmode added to example config
