* WIP
* updated appveyor and increased deadline 5 seconds due to increased linters being added
* revert files to upstream/engine
* WIP
* WIP
* mod file changes
* added script manager
* Added manager/and cli interfaces to scripting
* Added script task handler
* WIP - Added timer/repeat support and fleshed out wrapper further
* autoload support added + WIP
* WIP commit
* added account balance info
* btc markets temp work around
* WIP - merged with upstream for new order package BTC Markets responses broken
* Cancel order wrapper WIP
* order wrapper update
* Added test coverage for VM
* moved to map for VM List shutdown of all VM now handled added gctcli commands for list and stop of running scripts
* added override to load/execute for path
* fixed incorrect channel shutdown added further test coverage and restructured gctcli commands into sub commands
* increased test coverage for packages
* Added docs cleaned up tests and example scripts
* Test coverage increased for module/gct/exchange package
* windows fixes
* merged upstream/engine
* WIP
* logger fixes - removed pointer to bool check removed duplicate test check for logger
* remove unused mutex
* added inital upload support
* fix linter issues for go-fmt
* added zip support for uploading and added base for fund withdrawing
* changed error return types and also log errors, fix zip path issue
* improved error outputs and code flow
* pairs response fix added protobuf defs for stop all and list all
* added stop all running scripts general clean up and moved across to OrderManager
* linter fixes (gofmt)
* added list all command
* rewrote zip handler to be cleaner also fixed file overwrite on upload
* added query command reworked tests
* added further error checking to compileandrun corrected use of pointers for accountinfo
* bumped tengo version
* Removed named returns reworded log messages removed unused falseptr
* WIP
* Added virutal machine limit improved config options
* added model for script event added upload validation
* script_event table has been completed, tests for wrapper functions implemented
* README updates
* reverted changes opened new PR to move withdraw struct outs
* intial work on adding withdraw support after merger of withdraw package
* started work on examples
* Added crypto withdraw support
* fix switch case assignment and gofmt project
* Reworking Fiat withdraw request pending #402
* removed double pointer call
* added withdraw support for fiat currencies
* added tests for withdraw methods increased readme
* removed local tengo require and also fix linter issues
* Added default log size const added basic test for invalid script execution
* First pass at moving wrapper to validator package to allow proper validation of uploaded scripts
* Added script details to README added config test added test for no file extension
* moved tests to const and fixed incorrect pathing
* added test coverage to withdraw package
* corrected file close handling
* point to included configtest.json
* extended validator support when a script is uploaded
* Bug fix on bool logic
* Added mutex
* Don't create autit events on test execution
* reverted common to master
* moved file rename to unix timestamp format
* converted logger enabled back to pointer as i need nilness check also moved scriptid to text over blob
* started work on autoload add/remove support
* First round of PR fixes (mostly commented exports)
* Moved GCTScript load to last, removed unneeded error from cleanup()
* Comment clairty for AuitEventID
* added autoload add/remove command to cli
* added tests for autoload
* Test updates for Exchanges
* linter fixes (gofmt)
* Removed double check of engine pointer
* remove possible nil pointer on GetSpecificTicker
* Fixed not closing file handler on write that causes archive removal to fail
* file handler Close clean ups
* corrected spelling on error return and return invalid name n autoload
* moved strings to cosnt moved bool pointer creation to convert package
* new zip extractor added
* Validation has been added to archive uploads
* removed shadow var on err
* added ok check to conversion
* converted condition check
* basic test for zip extract added
* new zip handler
* reverted back to old atomic loading system
* removed shadow err
* lets add a new line
* added space to error return
* command line toggle for script now works properly
* readme updated
* set configLoaded to true
* check for configLoaded condition
* added mutex to allow for multiple access on virtual machine increased test coverage disable script manager if scripting is disabled
* linked up to enable/disablesubsystem commands
* added start/stop example to readme
* reworked logic on test as check should be done on Load()
* updated to tengo v2
* linters
* lower time on ntp client to stop slippage
* remove all fails if any fail validtion from an archive
* remove vm from list if timer is invalid
* removed shadow on err
* remove config creation from NTPCheck test
* WIP testing DB changes
* add unique constraint
* WIP: created has many model
* linters run
* basic sqlite3 support added for new database format
* linters run
* Added test coverage for script repo
* removed unused print
* updated env vars for CI instances
* updated env vars for CI instances
* Updated test packages
* Test updates for postgresql
* removed invalid tests from postgres
* remove duplication of struct and improved code flow
* general cleanup
* wording changes on log output
* use databasemgr logger and add support for autoload without file extension
* corrected test naming
* return correct error
* return correct error again version 82
* store scriptdata on creation
* Hello
* Errorln -> Errorf
* Removed unused vars
* Read me updates
* testing without parallel
* comment on exported type
* added nil check against VM for test
* add debugging information
* gofmt
* remove verbose and data sent to channel
* Added debug information
* linter fixes (gofmt)
* remove unused CompileAndRun() call
* test sleep to see if issue is timing related
* semi-concurrent map fixes
* one day i will run gofmt or setup precommit hooks
* new line :D
* increased test coverage
* added correct sleep time
* Moved over to sync map
* linter fixes (gofmt)
* goimports
* moved VM related methods to vm.go
* new line at end of file
* trying increased timeout on golangci-lint for appveyor
* add debugging information
* removed timeout
* reworked timeout logic
* linter fixes (gofmt)
* increased test coverage
* increased test coverage
* one day i will run gofmt or setup precommit hooks
* removed unused exchange test
* increased golangci-lint timeout
* Added nil check on shutdown and test coverage for it lowered timeout back to 1:30
* reworked ID system
* removed script hash as it was unused
* added comments on exported methods and read me update
* reorder code
* removed to atomic.value for test execution flag
* increased test coverage
* move add further up execution
* point to correct script file
* Bittrex GetDepositHistory method return new DepositHistory struct type
* Id field of DepositHistory type changed to int64
* check exchange (exist and enabled) before calling methods on, to avoid program crashes
* Update bittrex_types.go
* Renames func. Creates new func to setup pinghander either via gorilla style or our own
* Cleans up all ping pong handlers.......
* Clears up issues, makes naming a bit better
* Adds tests
* Adds ping support to binance
* Cleans up ping pongs and adds a comment
* Cleans up waitgroup stuff.
* DISCREETLY cleans up woeful function
* Fixes Kraken ping message type. Removes unnecessary test property. Adds `if err == websocket.ErrCloseSent {` to ping func
* +1 for +v
* bug fix for websocket orderbook processing
* Fix more panics
* fix linter issue
* kick panic can down the road
* temp fix for issue with a 404 returned error as chainz.cryptoid dropped eth support
* Address nits and fixed orderbook updating
* Fix trade data, rm'd event time from struct
* fix time conversion for huobi
* Actually process kline data and fix time stamps
* btse time conversion fix and RM log, as it seems that the gain is reflecting transaction side. Drop ticker fetching support because there does not seem to be support on docs. And added trade fetching support.
* revert huobi println
* Adressed suggestion
* rm unnecessary assignment
* rm unnecessary check and assign
* fix conversion mishap
* fix currency conversion bug
* update websocket logging
* RM websocket type which stops conversion and copy
* fix linter issue, add in unknown side type
* Start Coinbene SWAP implementation
* Flesh out more API endpoints
* Code cleanup
* Add more endpoints, bug fixes and order validation checks
* More endpoints, tests and bugfixes
* Remove omitempty for enabled/available pairs
* moved to getticekers endpoint
* add support for tickerbatching
* const and things
* moved openOrders to bool
* gofmt and clean up test logs
* revert canManipulateRealOrders to false
* corrected param order Side -> Type, also corrected condition check for IsOrderPlaced
* send open status for GetActiveOrders
* GetActiveOrder() changes to include OrderID and status matching
* BTC Markets batch order limit fixes & SplitStringSliceByLimit method
BTC markets batch end points have limits (20 for cancel 50 for query) adds new method SplitStringSliceByLimit in common to split a slice by limit and return slice of slice
* rm line :D
* Added test for SplitStringSliceByLimit and moved to const
* ntp client reworked to not return error if no valid time servers are found but default to system
* clean up
* new line added
* use TimeMgr sublogger and wording correction on output
* Moved to DialTimeout() & Removed SetDeadline call
* removed line
* added setdeadline fix
* goimport file
* removed unused error from NTPClient as we now default to system time if no server can be reached
* Added checks for number overflows
* converted to uint as you should not be passing a negative number in
* Increased test cases for NTPClient
* Removed Helper call as no longer outputting any data from function
* removed unused param
* Basic concept commit
* Initial changes to support bitfinex v2. Reverts linter changes as they suck. Exports bitfinex ws types
* Adds ticker, trade and orderbook support
* Candles sub that returns no data COMPLETE
* Adds authenticated ws support
* Adds the barebones endpoints to support
* Adds more endpoints
* Even more endpoints
* minicommit to switch and test
* All the interactive types
* Adds support for simultaneous connections. Updates tests. Nothing is working
* Successfully adds place order. Moves all authenticated endpoints to new switch case
* Cancel order and modify order
* Cancel all orders, cancel multi orders
* Finalises implementation. Uses testMain
* Adds WS wrapper support for some funcs
* Fixing rebasing issues
* Replaces use of currency as a variable. Updates a lot of coinut websocket auth endpoint stuff
* Fixes some fun for loops with GetEnabledPairs
* Fixes tests impacted by currency var change
* Adds coinut support for WS functions. Replaces `order` vars with `ord`. Fixes some for loops too. Removes verbose from bitfinex
* So many panics
* I'm fixing a hole, where the panics get in, and stops my mind from wandering, where it will go
* Moves func `CanUseAuthenticatedWebsocketEndpoint` to Websocket package as it fits better. Adds test coverage of `CanUseAuthenticatedWebsocketEndpoint`
* Finishes up all of coinuts ws implementations.
* GateIO implementation
* Adds some helper funcs for types, sides and status. Adds support for huobi. Removes unnecessary type
* Adds forgotten huobi endpoint
* Fixes cancel order endpoint
* go hates my formatting and so do I
* The process to get authenticated kraken websocket to work. Uses testmain. Adds new auth channel, auth subscriptions, auth data handling. Not working yet
* Finishes open orders handling
* Mini update for status only updates
* Fixes some kraken points
* Finishes WS kraken since it doesn't work
* Unfinished commit, cleaning up types
* Finishes the const replacing
* Fixes extra GetNAmes after rebase
* An end to the cleanup. testmain for gateio
* Adds ZB support
* Bitfinex cleanup. Renamed func
* Testmain-47s for everyone!!! yayaaaaaaa
* Adds kraken websocket wrapper support
* Fixes rebase issues
* Fixes tests from rebase
* Adds test for conversion. Fixes for loop. Updates test order pricing. Fixes some poor made tests. Adds proper error handling for ws responses instead of logging them. Fixed issue where commented code ruined kraken ws.
* Fixes secret linting issues. Prioritises bitfinex channelID responses over authorised
* Fixes sloppy error/var declarations
* Fixes crazy bad logic where submit order errors weren't really considered. Parralols alphapoint/alphapoint_test.go. Removes buffer for multi-websocket comms channel.
* Removal of inline string and removal of redundant nil checkerinos
* Fixes err checks. Checks whether float has decimal. Fixes append. Drops omitempties. Parallel to some tests. Moves var declarations
* Replaces my lazy sprintfs with strconv.FormatInt(time.Now().Unix(), 10)
* Adds shiny new FullyMatched bool. Fixes coinbene buy sell consts
* Fixes oopsie with coinbene const replacement
* Fixes currency issue
* Cleans up new places that use JSONDecode
* Fixes huge panic bug from string int conversion. Adds large testtable for strings to order types
* Fixes some more strconversion issues. Fixes table test var usage. Changes mapperino name
* Added some new scenarios for number splitting
* Fixes lint issues
* negative num fix
* Typo fix
* Accuracy warning comment
* drop common uuid v4 func and imported package as needed
* removed common functions regarding json marshal and unmarshal and used the json package directly. WRT unmarshal it was calling reflect and converted to string which is also checked in the JSON package so it was doing a double up, this will be a tiny gain as it was directly used in the requester package for all our outbound requests.
* add in string
* explicitly throw away return error value
* atleast return the error that websocket initialise returns
* return error when not connected
* fix comment
* Adds comments
* move package declarations
* drop append whenever we call supported
* remove unused import
* Change incorrect spelling
* fix tests
* fix go import issue
* Add directories to exclusion list && change default repo string
* Add in .idea folder to directory exclusion list
* cleaned
* Added support for calling the tool outside of its file
* fix formatting
* changed strings
* Adds new file.Move func to address a bug with Golang/Docker volumes when using os.Rename
Also uses TempDir for tests instead of live directories and increases test coverage for file.Write
* Goimport the imports
* Make usage of file package name consistent so it no longer clashes with vars
* Remove outputFile if io.Copy fails
* 1) Update Dockerfile/docker-compose.yml
2) Remove inline strings for buy/sell/test pairs
3) Remove dangerous order submission values
4) Fix consistency with audit_events (all other spec files use
CamelCase)
5) Update web websocket endpoint
6) Fix main param set (and induce dryrun mode on specific command line
params)
* Engine QA
Link up exchange syncer to cmd params, disarm market selling bombs and fix OKEX endpoints
* Fix linter issue after merge
* Engine QA changes
Template updates
Wrapper code cleanup
Disarmed order bombs
Documentation updates
* Daily engine QA
Bitstamp improvements
Spelling mistakes
Add Coinbene exchange to support list
Protect API authenticated calls for Coinbene/LBank
* Engine QA changes
Fix exchange_wrapper_coverage tool
Add SupportsAsset to exchange interface
Fix inline string usage and add BCH withdrawal support
* Engine QA
Fix Bitstamp types
Inform user of errors when parsing time accross the codebase
Change time parsing warnings to errors (as they are)
Update markdown docs [with linter fixes]
* Engine QA changes
1) Add test for dryrunParamInteraction
2) Disarm OKCoin/OKEX bombs if someone accidently sets canManipulateRealOrders to true and runs all package tests
3) Actually check exchange setup errors for BTSE and Coinbene, plus address this in the wrapper template
4) Hardcode missing/non-retrievable contributors and bump the contributors
5) Convert numbers/strings to meaningful types in Bitstamp and OKEX
6) If WS is supported for the exchange wrapper template, preset authWebsocketSupport var
* Fix the shadow people
* Link the SyncContinuously paramerino
* Also show SyncContinuously in engine.PrintSettings
* Address nitterinos and use correct filepath for logs
* Bitstamp: Extract ALL THE APM
* Fix additional nitterinos
* Fix time parsing error for Bittrex
* Improved error message when no config is set on startup
* Change inccorect error wording
* bump Bitfinex websocket orderbook return length to max
* temporary fix of incorrect orderbook updates, limit to bid and ask len of 100, will be extended later if needed
* Fixed issue in binance websocket that appended 0 volume bid/ask items
* Fix panic when unmarshalling an empty pair from config
* Add get pair asset method for exchange base
Fix Bitmex orderbook stream
Unbuffer Bitmex orderbook stream
* force syncer to update ticker instead of fetch, which allows a stream
* Fix websocket last price for coinbasepro
* fix websocket ticker for coinut
* Fix websocket orderbook stream Huobi
* increase orderbook depth REST for Huobi
* Fix websocket support and ensure data integrity
* Fix time parsing issue after error checks
* check error, only process enabled currency pairs, signal websocket data processing
* expanded websocket functionality for okgroup
* Add logic to not process zero length slice for orderbooks
* fix websocket ticker only updating enabled and individual book updates
* ZB fixes to order submission/retrieval/cancellation w/ general fixes
* Quiet unnecessary warning
* updated config entry values for REST and websocket (initial hack until I come up with a better solution for asset types)
* Ch GetName function to field access modifyer & rm useless code
* Add in error I missed
* Nits addressed
* some more fixes
* Turned kraken default websocket to true and some small changes
* fixes linter issues
* Ensured okgroup books and sent update through to datahandler. Zb update as well.
* Add test case to get asset type from pair
* Add test for pairs unmarshal
* Add testing and addressed nits
* FIX linter issue
* Addressed Gees nits
* Thanks glorious spotter
* more nitorinos
* Addres even more nits
* Add stringerino 4000
* Fix for panic cause by sort slice out of range, also nits addressed
* fix linter issues
* Changed from function to field access
* Changed from function to field access
* fix for orderbook update panic, removes quick fix - caused by sync item fetching through same protocol
* Add new test and update random generator
* pass in invalid string to future ob fetching, due to futures contract expire and a http 400 error is returned
* Removes lock unlock timer and instead sets unlocks between getting a nonce and sending a payload. Increases dispatch channel buffer to deal with len(enabledCurrencies) > ~100
* Adds additional comments to help explain the situation
* Fixes bug that could unlock mutex too early
* Fixes LIES where Gemini gets a nonce and then proceeds to declare it doesn't get a nonce causing an unrecoverable lock
* Fun new concept! The creation of a tested timed mutex. Unlocking an unlocked mutex cannot occur and response can be checked to verify whether the mutex was unlocked from timeout or command.
* Adds new cmd parameter "dispatchjobbuffer"
* Expands comments and renames benchmark. Makes `Timer` property private
* Happy little linters
* Renames jobBuffer and all related instances to jobs limit
* Tiny error message update
* Grammatical fix and setting dispatch.Start to use defaults
* Attempts to update orderbook so it doesn't need to sort
* Reverts the ws ob stuff. Gets rid of sorting because it happens later. Adds some exchange features
* update existing feature lists. Expands list definition to match my emotions
* Adds bithumb bitmex and bitstamp. adds a couple more types
* Features for you, features for me, features for bittrex, btcmarkets, btse, coinbasepro, coinut, exmo, gateio and gemini
* Features for hitbtc, huobi, itbit, kraken, lakebtc, lbank, localbitcoins, okcoin, okex, poloniex, yobit, zb
* Who can forget good old alphapoint?
* Adds btcmarksets websocket :glitch_crab: fixes alphapoint features
* Adds extra data not in the documentation :/
* Replaces websocket features by using protocol features. However, it breaks it due to import cycles. I'm not sure what I'll do just yet
* Removes import cycle via duplicate structs.
* Increases coverage of config with `TestCheckCurrencyConfigValues`. Moves all currency pair package types into their own files or places it at the bottom of files if necessary
* Increase coverage in code.go
* One way of determining a test has failed, is when to it fails. Removed redundant explanation
* Increases code coverage of conversion
* Lint fixes
* Fixes orderbook tests
* Re-adds sorting because its important to still have the internal pre-processed orderbook to be representative of a real orderbook
* Secret lints that did not show up via Windows linting
* Adds protocol package to contain exchange features
* Fixes protocol implementation
* Fixes ws tests
* Addresses the following: Removes st-st-stutters in config types, changes GetAvailableForexProviders -> GetSupportedForexProviders, removes errors from tests where error is nil, removes orderbook setup when not necessary, removes import newlines, removes false bools from declaration, changes should of to should have
* imports and casing
* Fixes two more nil error checks
* Greatly increases efficiency of web socket orderbook processing by minimising multiple map lookups. Changes buffer to use pointers. Ensures orderbook benchmarks are on equal footing and set correctly. Removes data race by not setting waitgroup adds inside go routine causing wait and add to happen simultaneously. Updates ticker and orderbook service to use a pointer var instead of map lookups when setting data.
* Removes misguided comment
* Removes waitgroups and goroutines for updating bids and asks for non-id orderbook updates via websocket. Updates benchmarks to be consistent
* Added dispatch service
* Added orderbook streaming capabilities
* Assigned correct orderbook.base exchange name
* Fixed Requested niterinos
Add in cli orderbook QA tool to gctcli
Add exchange orderbook streaming
* Add ticker streaming support through dispatch package
* Added in some more info on error returns for orderbook.go
* fix linter issues
* Fix some issues
* Update
* Fix requested
* move dispatch out of exchanges folder to its own independant folder
* Fix requested
* change orderbook string to tickers
* Limit orderbooks to 50 and made dispatch system more stateless in operation
* lower cases for update/retrieve/sub exchange name
* Adds in asset validation and lower case conversion on cli
* Remove comment
* Moved timer to a higher scope so its not constantly initialised just reset per instance and removed returning unused channel on error
* Rm unused release function in dispatch.go
Reset timer and bleed buffered timer chan if needed in dispatch.go
Added in ticker.Stop() and timer.Stop() functions for worker routine return in dispatch.go
Index aggregated bid and ask functions for orderbook.go
Added in dummy slice for wsorderbook_test.go
* Moved drain to above Reset so potential race would not occur in dispatch.go
Fix various linter issues dispatch.go
* Fix some issues
* change to start/stop service, added in service state change via cli, updated logger
* fix requested
* Add worker amount init spawning
* fix linter issues
* Fix more linter issues
* More fixes
* Fix race issue on releasing pipe channel on a close after shutting down dispatcher system
* Moved all types to dispatch_types.go && remove panic
* Moved types into serperate file && improve test coverage
* RM unnecessary select case for draining channel && fixed error string
* Added orderbook_types file and improved code coverage
* gofmt file
* reinstated select cases on drain because I am silly
* Remove error for drop worker
* Added more test cases
* not checking error issue fix
* remove func causing race in test, this has required protection via an exported function
* set Gemini websocket orderbook exchange name
* Initial commit tearing down the websocket connection management. The purpose is to remove the traffic monitoring and dropping as syncer.go is a better manager
* Adds a readwrite mutex and helper functions to minimise inline lock/unlocks and prevent races
* Creates new WebsocketType struct to contain all parameters required. Deletes WebsocketReset. Utilises ReadMessageErrors channel for all websocket readmessages to analyse when an error returned is due to a disconnect
* Fixes issue with syncer trying to connect while connecting
* Simplifies initialisation function for websocket. Reconnects and resubscribes after disconnection
* Adds WebsocketTimeout config value to dictate when the websocket traffic monitor should die. Default to two minutes of no traffic activity. Increases test coverage and updates existing tests to work with new technologic. RE-ADDS TESTS I ACCIDENTALLY DELETED FROM PREVIOUS PR
* Removes snapshot override as its always necessary when considering reconnections. Increases test coverage. Re-adds tests that were ACCIDENTALLY DELETED. Removes unused websocket channels. Bug fix for traffic monitor to shutdown via goroutine instead of killing itself
* Fixes gateio bug for authentication errors when null. Adds little entry to syncer for when websocket is switched to rest and then back, you get a log notifying of the return. Fixes okgroup bug where ws message is sent on a disconnected ws, causing panic. Renames setConnectionStatus to setConnectedStatus. Puts connection monitor log behind verbose bool
* Fixes lingering races. Fixes bug where websocket was enabled whether you liked it or not. Removes demonstration test
* Fixes log message, renames unc, removes comments
* Fixes data race
* Removes verbosity, ensures shutdown sets connection status appropriately
* Removes go routine causing CPU spike. Stops timers properly and resets timers properly
* Renames `WsEnabled` to `Enabled`. Increases test coverage. Fixes typos. Handles unhandled errors
* The forgotten lint
* With using RWlocks, removes the channel nil check and relies on !w.IsConnected() to prevent a shutdown from recurring
* Removes extra closure step in the defer as it causes all the issues
* Prevents timer channel hangups. Minimises use of websocket Connect(). Expands disconnection error definition. Removes routine disconnection error handling. Ensures only one traffic monitor can ever be run. Renames subscriptionLock to subscriptionMutext for consistency
* Extends timeout to 30 seconds to cover for non-popular exchanges and non-popular currencies
* Updates test from rebase to use new websocket setup function
* Fixes test to ensure it tests what it says it does
