d0zingcat/gocryptotrader
1
0
Fork 0
mirror of https://github.com/d0zingcat/gocryptotrader.git synced 2026-05-13 23:16:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

exchange: binance orderbook fix (#599)

Browse Source 
* port orderbook binance management from draft singular asset (spot) processing add additional updates to buffer management

* integrate port

* shifted burden of proof to exchange and remove repairing techniques that obfuscate issues and could caause artifacts

* WIP

* Update exchanges, update tests, update configuration so we can default off on buffer util.

* Add buffer enabled switching to all exchanges and some that are missing, default to off.

* lbtc set not aggregate books

* Addr linter issues

* EOD wip

* optimization and bug fix pass

* clean before test and benchmarking

* add testing/benchmarks to sorting/reversing functions, dropped pointer to slice as we aren't changing slice len or cap

* Add tests and removed ptr for main book as we just ammend amount

* addr exchange test issues

* ci issues

* addr glorious issues

* Addr MCB nits, fixed funding rate book for bitfinex and fixed potential panic on nil book return

* addr linter issues

* updated mistakes

* Fix more tests

* revert bypass

* Addr mcb nits

* fix zero price bug caused by exchange. Filted out bid result rather then unsubscribing. Updated orderbook to L2 so there is no aggregation.

* Allow for zero bid and ask books to be loaded and warn if found.

* remove authentication subscription conflicts as they do not have a channel ID return

* WIP - Batching outbound requests for kraken as they do not give you the partial if you subscribe to do many things.

* finalised outbound request for kraken

* filter zero value due to invalid returned data from exchange, add in max subscription amount and increased outbound batch limit

* expand to max allowed book length & fix issue where they were sending a zero length ask side when we sent a depth of zero

* Updated function comments and added in more realistic book sizing for sort cases

* change map ordering

* amalgamate maps in buffer

* Rm ln

* fix kraken linter issues

* add in buffer initialisation

* increase timout by 30seconds

* Coinbene: Add websocket orderbook length check.

* Engine: Improve switch statement for orderbook summary dissplay.

* Binance: Added tests, remove deadlock

* Exchanges: Change orderbook field -> IsFundingRate

* Orderbook Buffer: Added method to orderbookHolder

* Kraken: removed superfluous integer for sleep

* Bitmex: fixed error return

* cmd/gctcli: force 8 decimal place usage for orderbook streaming

* Kraken: Add checksum and fix bug where we were dropping returned data which was causing artifacts

* Kraken: As per orderbook documentation added in maxdepth field to update to filter depth that goes beyond current scope

* Bitfinex: Tracking down bug on margin-funding, added sequence and checksum validation websocket config on connect (WIP)

* Bitfinex: Complete implementation of checksum

* Bitfinex: Fix funding book insertion and checksum - Dropped updates and deleting items not on book are continuously occuring from stream

* Bitfinex: Fix linter issues

* Bitfinex: Fix even more linter issues.

* Bitmex: Populate orderbook base identification fields to be passed back when error occurrs

* OkGroup: Populate orderbook base identification fields to be passed back when error occurrs

* BTSE: Change string check to 'connect success' to capture multiple user successful strings

* Bitfinex: Updated handling of funding tickers

* Bitfinex: Fix undocumented alignment bug for funding rates

* Bitfinex: Updated error return with more information

* Bitfinex: Change REST fetching to Raw book to keep it in line with websocket implementation. Fix woopsy.

* Localbitcoins: Had to impose a rate limiter to stop errors, fixed return for easier error identification.

* Exchanges: Update failing tests

* LocalBitcoins: Addr nit and bumped time by 1 second for fetching books

* Kraken: Dynamically scale precision based on str return for checksum calculations

* Kraken: Add pair and asset type to validateCRC32 error reponse

* BTSE: Filter out zero amount orderbook price levels in websocket return

* Exchanges: Update orderbook functions to return orderbook base to differentiate errors.

* BTSE: Fix spelling

* Bitmex: Fix error return string

* BTSE: Add orderbook filtering function

* Coinbene: Change wording

* BTSE: Add test for filtering

* Binance: Addr nits, added in variables for buffers and worker amounts and fixed error log messages

* GolangCI: Remove excess 0

* Binance: Reduces double ups on asset and pair in errors

* Binance: Fix error checking
This commit is contained in:
Ryan O'Hara-Reid
2021-01-04 17:19:55 +11:00
committed by GitHub
parent 59013ea076
commit eb0571cc9b
80 changed files with 11139 additions and 1408 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.golangci.yml
View File

@@ -1,5 +1,5 @@
run:
timeout: 1m30s
timeout: 2m0s
issues-exit-code: 1
tests: true
skip-dirs:

21
cmd/exchange_template/wrapper_file.tmpl
View File

@@ -272,37 +272,32 @@ func ({{.Variable}} *{{.CapitalName}}) FetchOrderbook(currency currency.Pair, as
// UpdateOrderbook updates and returns the orderbook for a currency pair
func ({{.Variable}} *{{.CapitalName}}) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: {{.Variable}}.Name, Pair: p, AssetType: assetType}
// NOTE: UPDATE ORDERBOOK EXAMPLE
/*
orderbookNew, err := {{.Variable}}.GetOrderBook(exchange.FormatExchangeCurrency({{.Variable}}.Name, p).String(), 1000)
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Quantity,
Price: orderbookNew.Bids[x].Price,
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
Amount: orderBook.Asks[x].Quantity,
Price: orderBook.Asks[x].Price,
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderBookNew.Asks[x].Quantity,
Price: orderBookNew.Asks[x].Price,
})
}
*/
orderBook.Pair = p
orderBook.ExchangeName = {{.Variable}}.Name
orderBook.AssetType = assetType
err := orderBook.Process()
err := book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get({{.Variable}}.Name, p, assetType)

2
cmd/gctcli/commands.go
View File

@@ -3152,7 +3152,7 @@ func getOrderbookStream(c *cli.Context) error {
askPrice = resp.Asks[i].Price
}
fmt.Printf("%f %s @ %f %s\t\t%f %s @ %f %s\n",
fmt.Printf("%.8f %s @ %.8f %s\t\t%.8f %s @ %.8f %s\n",
bidAmount,
resp.Pair.Base,
bidPrice,

35
config/config_types.go
View File

@@ -113,23 +113,24 @@ type ConnectionMonitorConfig struct {
// ExchangeConfig holds all the information needed for each enabled Exchange.
type ExchangeConfig struct {
Name string `json:"name"`
Enabled bool `json:"enabled"`
Verbose bool `json:"verbose"`
UseSandbox bool `json:"useSandbox,omitempty"`
HTTPTimeout time.Duration `json:"httpTimeout"`
HTTPUserAgent string `json:"httpUserAgent,omitempty"`
HTTPDebugging bool `json:"httpDebugging,omitempty"`
WebsocketResponseCheckTimeout time.Duration `json:"websocketResponseCheckTimeout"`
WebsocketResponseMaxLimit time.Duration `json:"websocketResponseMaxLimit"`
WebsocketTrafficTimeout time.Duration `json:"websocketTrafficTimeout"`
WebsocketOrderbookBufferLimit int `json:"websocketOrderbookBufferLimit"`
ProxyAddress string `json:"proxyAddress,omitempty"`
BaseCurrencies currency.Currencies `json:"baseCurrencies"`
CurrencyPairs *currency.PairsManager `json:"currencyPairs"`
API APIConfig `json:"api"`
Features *FeaturesConfig `json:"features"`
BankAccounts []banking.Account `json:"bankAccounts,omitempty"`
Name string `json:"name"`
Enabled bool `json:"enabled"`
Verbose bool `json:"verbose"`
UseSandbox bool `json:"useSandbox,omitempty"`
HTTPTimeout time.Duration `json:"httpTimeout"`
HTTPUserAgent string `json:"httpUserAgent,omitempty"`
HTTPDebugging bool `json:"httpDebugging,omitempty"`
WebsocketResponseCheckTimeout time.Duration `json:"websocketResponseCheckTimeout"`
WebsocketResponseMaxLimit time.Duration `json:"websocketResponseMaxLimit"`
WebsocketTrafficTimeout time.Duration `json:"websocketTrafficTimeout"`
WebsocketOrderbookBufferLimit int `json:"websocketOrderbookBufferLimit"`
WebsocketOrderbookBufferEnabled bool `json:"websocketOrderbookBufferEnabled"`
ProxyAddress string `json:"proxyAddress,omitempty"`
BaseCurrencies currency.Currencies `json:"baseCurrencies"`
CurrencyPairs *currency.PairsManager `json:"currencyPairs"`
API APIConfig `json:"api"`
Features *FeaturesConfig `json:"features"`
BankAccounts []banking.Account `json:"bankAccounts,omitempty"`
// Deprecated settings which will be removed in a future update
AvailablePairs *currency.Pairs `json:"availablePairs,omitempty"`

97
engine/routines.go
View File

@@ -3,6 +3,7 @@ package engine
import (
"errors"
"fmt"
"strconv"
"strings"
"sync"
@@ -115,16 +116,32 @@ func printTickerSummary(result *ticker.Price, protocol string, err error) {
}
}
const (
book = "%s %s %s %s: ORDERBOOK: Bids len: %d Amount: %f %s. Total value: %s Asks len: %d Amount: %f %s. Total value: %s\n"
)
func printOrderbookSummary(result *orderbook.Base, protocol string, err error) {
if err != nil {
if err == common.ErrNotYetImplemented {
log.Warnf(log.Ticker, "Failed to get %s ticker. Error: %s\n",
if result == nil {
log.Errorf(log.OrderBook, "Failed to get %s orderbook. Error: %s\n",
protocol,
err)
return
}
log.Errorf(log.OrderBook, "Failed to get %s orderbook. Error: %s\n",
if err == common.ErrNotYetImplemented {
log.Warnf(log.OrderBook, "Failed to get %s orderbook for %s %s %s. Error: %s\n",
protocol,
result.ExchangeName,
result.Pair,
result.AssetType,
err)
return
}
log.Errorf(log.OrderBook, "Failed to get %s orderbook for %s %s %s. Error: %s\n",
protocol,
result.ExchangeName,
result.Pair,
result.AssetType,
err)
return
}
@@ -132,57 +149,33 @@ func printOrderbookSummary(result *orderbook.Base, protocol string, err error) {
bidsAmount, bidsValue := result.TotalBidsAmount()
asksAmount, asksValue := result.TotalAsksAmount()
if result.Pair.Quote.IsFiatCurrency() &&
result.Pair.Quote != Bot.Config.Currency.FiatDisplayCurrency {
var bidValueResult, askValueResult string
switch {
case result.Pair.Quote.IsFiatCurrency() && result.Pair.Quote != Bot.Config.Currency.FiatDisplayCurrency:
origCurrency := result.Pair.Quote.Upper()
log.Infof(log.OrderBook, "%s %s %s %s: ORDERBOOK: Bids len: %d Amount: %f %s. Total value: %s Asks len: %d Amount: %f %s. Total value: %s\n",
result.ExchangeName,
protocol,
FormatCurrency(result.Pair),
strings.ToUpper(result.AssetType.String()),
len(result.Bids),
bidsAmount,
result.Pair.Base,
printConvertCurrencyFormat(origCurrency, bidsValue),
len(result.Asks),
asksAmount,
result.Pair.Base,
printConvertCurrencyFormat(origCurrency, asksValue),
)
} else {
if result.Pair.Quote.IsFiatCurrency() &&
result.Pair.Quote == Bot.Config.Currency.FiatDisplayCurrency {
log.Infof(log.OrderBook, "%s %s %s %s: ORDERBOOK: Bids len: %d Amount: %f %s. Total value: %s Asks len: %d Amount: %f %s. Total value: %s\n",
result.ExchangeName,
protocol,
FormatCurrency(result.Pair),
strings.ToUpper(result.AssetType.String()),
len(result.Bids),
bidsAmount,
result.Pair.Base,
printCurrencyFormat(bidsValue),
len(result.Asks),
asksAmount,
result.Pair.Base,
printCurrencyFormat(asksValue),
)
} else {
log.Infof(log.OrderBook, "%s %s %s %s: ORDERBOOK: Bids len: %d Amount: %f %s. Total value: %f Asks len: %d Amount: %f %s. Total value: %f\n",
result.ExchangeName,
protocol,
FormatCurrency(result.Pair),
strings.ToUpper(result.AssetType.String()),
len(result.Bids),
bidsAmount,
result.Pair.Base,
bidsValue,
len(result.Asks),
asksAmount,
result.Pair.Base,
asksValue,
)
}
bidValueResult = printConvertCurrencyFormat(origCurrency, bidsValue)
askValueResult = printConvertCurrencyFormat(origCurrency, asksValue)
case result.Pair.Quote.IsFiatCurrency() && result.Pair.Quote == Bot.Config.Currency.FiatDisplayCurrency:
bidValueResult = printCurrencyFormat(bidsValue)
askValueResult = printCurrencyFormat(asksValue)
default:
bidValueResult = strconv.FormatFloat(bidsValue, 'f', -1, 64)
askValueResult = strconv.FormatFloat(asksValue, 'f', -1, 64)
}
log.Infof(log.OrderBook, book,
result.ExchangeName,
protocol,
FormatCurrency(result.Pair),
strings.ToUpper(result.AssetType.String()),
len(result.Bids),
bidsAmount,
result.Pair.Base,
bidValueResult,
len(result.Asks),
asksAmount,
result.Pair.Base,
askValueResult,
)
}
func relayWebsocketEvent(result interface{}, event, assetType, exchangeName string) {

2
exchanges/binance/binance.go
View File

@@ -65,6 +65,8 @@ type Binance struct {
// Valid string list that is required by the exchange
validLimits []int
obm *orderbookManager
}
// GetExchangeInfo returns exchange information. Check binance_types for more

1
exchanges/binance/binance_live_test.go
View File

@@ -34,6 +34,7 @@ func TestMain(m *testing.M) {
if err != nil {
log.Fatal("Binance setup error", err)
}
b.setupOrderbookManager()
b.Websocket.DataHandler = sharedtestvalues.GetWebsocketInterfaceChannelOverride()
log.Printf(sharedtestvalues.LiveTesting, b.Name, b.API.Endpoints.URL)
os.Exit(m.Run())

2
exchanges/binance/binance_mock_test.go
View File

@@ -39,6 +39,8 @@ func TestMain(m *testing.M) {
log.Fatal("Binance setup error", err)
}
b.setupOrderbookManager()
serverDetails, newClient, err := mock.NewVCRServer(mockfile)
if err != nil {
log.Fatalf("Mock server error %s", err)

51
exchanges/binance/binance_test.go
View File

@@ -1,6 +1,7 @@
package binance
import (
"encoding/json"
"testing"
"time"
@@ -63,7 +64,7 @@ func TestGetOrderBook(t *testing.T) {
t.Parallel()
_, err := b.GetOrderBook(OrderBookDataRequestParams{
Symbol: currency.NewPair(currency.BTC, currency.USDT),
Limit: 10,
Limit: 1000,
})
if err != nil {
@@ -911,6 +912,7 @@ func TestWsTradeUpdate(t *testing.T) {
}
func TestWsDepthUpdate(t *testing.T) {
b.setupOrderbookManager()
seedLastUpdateID := int64(161)
book := OrderBook{
Asks: []OrderbookItem{
@@ -1184,3 +1186,50 @@ func TestGetRecentTrades(t *testing.T) {
t.Error(err)
}
}
func TestSeedLocalCache(t *testing.T) {
t.Parallel()
err := b.SeedLocalCache(currency.NewPair(currency.BTC, currency.USDT))
if err != nil {
t.Fatal(err)
}
}
func TestGenerateSubscriptions(t *testing.T) {
t.Parallel()
subs, err := b.GenerateSubscriptions()
if err != nil {
t.Fatal(err)
}
if len(subs) != 4 {
t.Fatal("unexpected subscription length")
}
}
var websocketDepthUpdate = []byte(`{"E":1608001030784,"U":7145637266,"a":[["19455.19000000","0.59490200"],["19455.37000000","0.00000000"],["19456.11000000","0.00000000"],["19456.16000000","0.00000000"],["19458.67000000","0.06400000"],["19460.73000000","0.05139800"],["19461.43000000","0.00000000"],["19464.59000000","0.00000000"],["19466.03000000","0.45000000"],["19466.36000000","0.00000000"],["19508.67000000","0.00000000"],["19572.96000000","0.00217200"],["24386.00000000","0.00256600"]],"b":[["19455.18000000","2.94649200"],["19453.15000000","0.01233600"],["19451.18000000","0.00000000"],["19446.85000000","0.11427900"],["19446.74000000","0.00000000"],["19446.73000000","0.00000000"],["19444.45000000","0.14937800"],["19426.75000000","0.00000000"],["19416.36000000","0.36052100"]],"e":"depthUpdate","s":"BTCUSDT","u":7145637297}`)
func TestProcessUpdate(t *testing.T) {
t.Parallel()
p := currency.NewPair(currency.BTC, currency.USDT)
var depth WebsocketDepthStream
err := json.Unmarshal(websocketDepthUpdate, &depth)
if err != nil {
t.Fatal(err)
}
err = b.obm.stageWsUpdate(&depth, p, asset.Spot)
if err != nil {
t.Fatal(err)
}
err = b.obm.fetchBookViaREST(p)
if err != nil {
t.Fatal(err)
}
err = b.obm.cleanup(p)
if err != nil {
t.Fatal(err)
}
}

37
exchanges/binance/binance_types.go
View File

@@ -1,9 +1,11 @@
package binance
import (
"sync"
"time"
"github.com/thrasher-corp/gocryptotrader/currency"
"github.com/thrasher-corp/gocryptotrader/exchanges/asset"
)
// withdrawals status codes description
@@ -107,13 +109,13 @@ type DepthUpdateParams []struct {
// WebsocketDepthStream is the difference for the update depth stream
type WebsocketDepthStream struct {
Event string `json:"e"`
Timestamp time.Time `json:"E"`
Pair string `json:"s"`
FirstUpdateID int64 `json:"U"`
LastUpdateID int64 `json:"u"`
UpdateBids [][]interface{} `json:"b"`
UpdateAsks [][]interface{} `json:"a"`
Event string `json:"e"`
Timestamp time.Time `json:"E"`
Pair string `json:"s"`
FirstUpdateID int64 `json:"U"`
LastUpdateID int64 `json:"u"`
UpdateBids [][2]interface{} `json:"b"`
UpdateAsks [][2]interface{} `json:"a"`
}
// RecentTradeRequestParams represents Klines request data.
@@ -746,3 +748,24 @@ type WsPayload struct {
Params []string `json:"params"`
ID int64 `json:"id"`
}
// orderbookManager defines a way of managing and maintaining synchronisation
// across connections and assets.
type orderbookManager struct {
state map[currency.Code]map[currency.Code]map[asset.Item]*update
sync.Mutex
jobs chan job
}
type update struct {
buffer chan *WebsocketDepthStream
fetchingBook bool
initialSync bool
}
// job defines a synchonisation job that tells a go routine to fetch an
// orderbook via the REST protocol
type job struct {
Pair currency.Pair
}

433
exchanges/binance/binance_websocket.go
View File

@@ -28,6 +28,18 @@ const (
var listenKey string
var (
// maxWSUpdateBuffer defines max websocket updates to apply when an
// orderbook is initially fetched
maxWSUpdateBuffer = 100
// maxWSOrderbookJobs defines max websocket orderbook jobs in queue to fetch
// an orderbook snapshot via REST
maxWSOrderbookJobs = 2000
// maxWSOrderbookWorkers defines a max amount of workers allowed to execute
// jobs from the job channel
maxWSOrderbookWorkers = 10
)
// WsConnect initiates a websocket connection
func (b *Binance) WsConnect() error {
if !b.Websocket.IsEnabled() || !b.IsEnabled() {
@@ -85,10 +97,24 @@ func (b *Binance) WsConnect() error {
}
go b.wsReadData()
b.setupOrderbookManager()
return nil
}
func (b *Binance) setupOrderbookManager() {
if b.obm == nil {
b.obm = &orderbookManager{
state: make(map[currency.Code]map[currency.Code]map[asset.Item]*update),
jobs: make(chan job, maxWSOrderbookJobs),
}
for i := 0; i < maxWSOrderbookWorkers; i++ {
// 10 workers for synchronising book
b.SynchroniseWebsocketOrderbook()
}
}
}
// KeepAuthKeyAlive will continuously send messages to
// keep the WS auth key active
func (b *Binance) KeepAuthKeyAlive() {
@@ -410,7 +436,6 @@ func (b *Binance) SeedLocalCache(p currency.Pair) error {
if err != nil {
return err
}
return b.SeedLocalCacheWithBook(p, &ob)
}
@@ -457,57 +482,33 @@ func (b *Binance) UpdateLocalBuffer(wsdp *WebsocketDepthStream) error {
return err
}
currentBook := b.Websocket.Orderbook.GetOrderbook(currencyPair, asset.Spot)
if currentBook == nil {
// Used when a pair/s is enabled while connected
err = b.SeedLocalCache(currencyPair)
if err != nil {
return err
}
currentBook = b.Websocket.Orderbook.GetOrderbook(currencyPair, asset.Spot)
err = b.obm.stageWsUpdate(wsdp, currencyPair, asset.Spot)
if err != nil {
return err
}
// Drop any event where u is <= lastUpdateId in the snapshot.
// The first processed event should have U <= lastUpdateId+1 AND u >= lastUpdateId+1.
// While listening to the stream, each new event's U should be equal to the previous event's u+1.
if wsdp.LastUpdateID <= currentBook.LastUpdateID {
return nil
err = b.applyBufferUpdate(currencyPair)
if err != nil {
cleanupErr := b.Websocket.Orderbook.FlushOrderbook(currencyPair, asset.Spot)
if cleanupErr != nil {
log.Errorf(log.WebsocketMgr,
"%s flushing websocket error: %v",
b.Name,
cleanupErr)
}
cleanupErr = b.obm.cleanup(currencyPair)
if cleanupErr != nil {
log.Errorf(log.WebsocketMgr,
"%s cleanup websocket orderbook error: %v",
b.Name,
cleanupErr)
}
return err
}
var updateBid, updateAsk []orderbook.Item
for i := range wsdp.UpdateBids {
p, err := strconv.ParseFloat(wsdp.UpdateBids[i][0].(string), 64)
if err != nil {
return err
}
a, err := strconv.ParseFloat(wsdp.UpdateBids[i][1].(string), 64)
if err != nil {
return err
}
updateBid = append(updateBid, orderbook.Item{Price: p, Amount: a})
}
for i := range wsdp.UpdateAsks {
p, err := strconv.ParseFloat(wsdp.UpdateAsks[i][0].(string), 64)
if err != nil {
return err
}
a, err := strconv.ParseFloat(wsdp.UpdateAsks[i][1].(string), 64)
if err != nil {
return err
}
updateAsk = append(updateAsk, orderbook.Item{Price: p, Amount: a})
}
return b.Websocket.Orderbook.Update(&buffer.Update{
Bids: updateBid,
Asks: updateAsk,
Pair: currencyPair,
UpdateID: wsdp.LastUpdateID,
Asset: asset.Spot,
})
return nil
}
// GenerateSubscriptions generates the default subscription set
@@ -568,3 +569,337 @@ func (b *Binance) Unsubscribe(channelsToUnsubscribe []stream.ChannelSubscription
b.Websocket.RemoveSuccessfulUnsubscriptions(channelsToUnsubscribe...)
return nil
}
// ProcessUpdate processes the websocket orderbook update
func (b *Binance) ProcessUpdate(cp currency.Pair, a asset.Item, ws *WebsocketDepthStream) error {
var updateBid []orderbook.Item
for i := range ws.UpdateBids {
price, ok := ws.UpdateBids[i][0].(string)
if !ok {
return errors.New("type assertion failed for bid price")
}
p, err := strconv.ParseFloat(price, 64)
if err != nil {
return err
}
amount, ok := ws.UpdateBids[i][1].(string)
if !ok {
return errors.New("type assertion failed for bid amount")
}
a, err := strconv.ParseFloat(amount, 64)
if err != nil {
return err
}
updateBid = append(updateBid, orderbook.Item{Price: p, Amount: a})
}
var updateAsk []orderbook.Item
for i := range ws.UpdateAsks {
price, ok := ws.UpdateAsks[i][0].(string)
if !ok {
return errors.New("type assertion failed for ask price")
}
p, err := strconv.ParseFloat(price, 64)
if err != nil {
return err
}
amount, ok := ws.UpdateAsks[i][1].(string)
if !ok {
return errors.New("type assertion failed for ask amount")
}
a, err := strconv.ParseFloat(amount, 64)
if err != nil {
return err
}
updateAsk = append(updateAsk, orderbook.Item{Price: p, Amount: a})
}
return b.Websocket.Orderbook.Update(&buffer.Update{
Bids: updateBid,
Asks: updateAsk,
Pair: cp,
UpdateID: ws.LastUpdateID,
Asset: a,
})
}
// applyBufferUpdate applies the buffer to the orderbook or initiates a new
// orderbook sync by the REST protocol which is off handed to go routine.
func (b *Binance) applyBufferUpdate(pair currency.Pair) error {
fetching, err := b.obm.checkIsFetchingBook(pair)
if err != nil {
return err
}
if fetching {
return nil
}
recent := b.Websocket.Orderbook.GetOrderbook(pair, asset.Spot)
if recent == nil {
return b.obm.fetchBookViaREST(pair)
}
return b.obm.checkAndProcessUpdate(b.ProcessUpdate, pair, recent)
}
// SynchroniseWebsocketOrderbook synchronises full orderbook for currency pair
// asset
func (b *Binance) SynchroniseWebsocketOrderbook() {
b.Websocket.Wg.Add(1)
go func() {
defer b.Websocket.Wg.Done()
for {
select {
case j := <-b.obm.jobs:
err := b.processJob(j.Pair)
if err != nil {
log.Errorf(log.WebsocketMgr,
"%s processing websocket orderbook error %v",
b.Name, err)
}
case <-b.Websocket.ShutdownC:
return
}
}
}()
}
// processJob fetches and processes orderbook updates
func (b *Binance) processJob(p currency.Pair) error {
err := b.SeedLocalCache(p)
if err != nil {
return fmt.Errorf("%s %s seeding local cache for orderbook error: %v",
p, asset.Spot, err)
}
err = b.obm.stopFetchingBook(p)
if err != nil {
return err
}
// Immediately apply the buffer updates so we don't wait for a
// new update to initiate this.
err = b.applyBufferUpdate(p)
if err != nil {
errClean := b.Websocket.Orderbook.FlushOrderbook(p, asset.Spot)
if errClean != nil {
log.Errorf(log.WebsocketMgr,
"%s flushing websocket error: %v",
b.Name,
errClean)
}
errClean = b.obm.cleanup(p)
if errClean != nil {
log.Errorf(log.WebsocketMgr, "%s cleanup websocket error: %v",
b.Name,
errClean)
}
return err
}
return nil
}
// stageWsUpdate stages websocket update to roll through updates that need to
// be applied to a fetched orderbook via REST.
func (o *orderbookManager) stageWsUpdate(u *WebsocketDepthStream, pair currency.Pair, a asset.Item) error {
o.Lock()
defer o.Unlock()
m1, ok := o.state[pair.Base]
if !ok {
m1 = make(map[currency.Code]map[asset.Item]*update)
o.state[pair.Base] = m1
}
m2, ok := m1[pair.Quote]
if !ok {
m2 = make(map[asset.Item]*update)
m1[pair.Quote] = m2
}
state, ok := m2[a]
if !ok {
state = &update{
// 100ms update assuming we might have up to a 10 second delay.
// There could be a potential 100 updates for the currency.
buffer: make(chan *WebsocketDepthStream, maxWSUpdateBuffer),
fetchingBook: false,
initialSync: true,
}
m2[a] = state
}
select {
// Put update in the channel buffer to be processed
case state.buffer <- u:
return nil
default:
return fmt.Errorf("channel blockage for %s, asset %s and connection",
pair, a)
}
}
// checkIsFetchingBook checks status if the book is currently being via the REST
// protocol.
func (o *orderbookManager) checkIsFetchingBook(pair currency.Pair) (bool, error) {
o.Lock()
defer o.Unlock()
state, ok := o.state[pair.Base][pair.Quote][asset.Spot]
if !ok {
return false,
fmt.Errorf("check is fetching book cannot match currency pair %s asset type %s",
pair,
asset.Spot)
}
return state.fetchingBook, nil
}
// stopFetchingBook completes the book fetching.
func (o *orderbookManager) stopFetchingBook(pair currency.Pair) error {
o.Lock()
defer o.Unlock()
state, ok := o.state[pair.Base][pair.Quote][asset.Spot]
if !ok {
return fmt.Errorf("could not match pair %s and asset type %s in hash table",
pair,
asset.Spot)
}
if !state.fetchingBook {
return fmt.Errorf("fetching book already set to false for %s %s",
pair,
asset.Spot)
}
state.fetchingBook = false
return nil
}
// completeInitialSync sets if an asset type has completed its initial sync
func (o *orderbookManager) completeInitialSync(pair currency.Pair) error {
o.Lock()
defer o.Unlock()
state, ok := o.state[pair.Base][pair.Quote][asset.Spot]
if !ok {
return fmt.Errorf("complete initial sync cannot match currency pair %s asset type %s",
pair,
asset.Spot)
}
if !state.initialSync {
return fmt.Errorf("initital sync already set to false for %s %s",
pair,
asset.Spot)
}
state.initialSync = false
return nil
}
// fetchBookViaREST pushes a job of fetching the orderbook via the REST protocol
// to get an initial full book that we can apply our buffered updates too.
func (o *orderbookManager) fetchBookViaREST(pair currency.Pair) error {
o.Lock()
defer o.Unlock()
state, ok := o.state[pair.Base][pair.Quote][asset.Spot]
if !ok {
return fmt.Errorf("fetch book via rest cannot match currency pair %s asset type %s",
pair,
asset.Spot)
}
state.initialSync = true
state.fetchingBook = true
select {
case o.jobs <- job{pair}:
return nil
default:
return fmt.Errorf("%s %s book synchronisation channel blocked up",
pair,
asset.Spot)
}
}
func (o *orderbookManager) checkAndProcessUpdate(processor func(currency.Pair, asset.Item, *WebsocketDepthStream) error, pair currency.Pair, recent *orderbook.Base) error {
o.Lock()
defer o.Unlock()
state, ok := o.state[pair.Base][pair.Quote][asset.Spot]
if !ok {
return fmt.Errorf("could not match pair [%s] asset type [%s] in hash table to process websocket orderbook update",
pair, asset.Spot)
}
// This will continuously remove updates from the buffered channel and
// apply them to the current orderbook.
buffer:
for {
select {
case d := <-state.buffer:
process, err := state.validate(d, recent)
if err != nil {
return err
}
if process {
err := processor(pair, asset.Spot, d)
if err != nil {
return fmt.Errorf("%s %s processing update error: %w",
pair, asset.Spot, err)
}
}
default:
break buffer
}
}
return nil
}
// validate checks for correct update alignment
func (u *update) validate(updt *WebsocketDepthStream, recent *orderbook.Base) (bool, error) {
if updt.LastUpdateID <= recent.LastUpdateID {
// Drop any event where u is <= lastUpdateId in the snapshot.
return false, nil
}
id := recent.LastUpdateID + 1
if u.initialSync {
// The first processed event should have U <= lastUpdateId+1 AND
// u >= lastUpdateId+1.
if updt.FirstUpdateID > id && updt.LastUpdateID < id {
return false, fmt.Errorf("initial websocket orderbook sync failure for pair %s and asset %s",
recent.Pair,
asset.Spot)
}
u.initialSync = false
} else if updt.FirstUpdateID != id {
// While listening to the stream, each new event's U should be
// equal to the previous event's u+1.
return false, fmt.Errorf("websocket orderbook synchronisation failure for pair %s and asset %s",
recent.Pair,
asset.Spot)
}
return true, nil
}
// cleanup cleans up buffer and reset fetch and init
func (o *orderbookManager) cleanup(pair currency.Pair) error {
o.Lock()
state, ok := o.state[pair.Base][pair.Quote][asset.Spot]
if !ok {
o.Unlock()
return fmt.Errorf("cleanup cannot match %s %s to hash table",
pair,
asset.Spot)
}
bufferEmpty:
for {
select {
case <-state.buffer:
// bleed and discard buffer
default:
break bufferEmpty
}
}
o.Unlock()
// reset underlying bools
_ = o.stopFetchingBook(pair)
_ = o.completeInitialSync(pair)
return nil
}

31
exchanges/binance/binance_wrapper.go
View File

@@ -183,6 +183,7 @@ func (b *Binance) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: b.GenerateSubscriptions,
Features: &b.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
SortBufferByUpdateIDs: true,
})
@@ -400,37 +401,31 @@ func (b *Binance) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderb
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *Binance) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: b.Name, Pair: p, AssetType: assetType}
orderbookNew, err := b.GetOrderBook(OrderBookDataRequestParams{
Symbol: p,
Limit: 1000})
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids,
orderbook.Item{
Amount: orderbookNew.Bids[x].Quantity,
Price: orderbookNew.Bids[x].Price,
})
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Quantity,
Price: orderbookNew.Bids[x].Price,
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks,
orderbook.Item{
Amount: orderbookNew.Asks[x].Quantity,
Price: orderbookNew.Asks[x].Price,
})
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x].Quantity,
Price: orderbookNew.Asks[x].Price,
})
}
orderBook.Pair = p
orderBook.ExchangeName = b.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(b.Name, p, assetType)

3
exchanges/bitfinex/bitfinex.go
View File

@@ -77,6 +77,9 @@ const (
// activity. Cancelling orders will be possible.
bitfinexMaintenanceMode = 0
bitfinexOperativeMode = 1
bitfinexChecksumFlag = 131072
bitfinexWsSequenceFlag = 65536
)
// Bitfinex is the overarching type across the bitfinex package

81
exchanges/bitfinex/bitfinex_test.go
View File

File diff suppressed because one or more lines are too long

4
exchanges/bitfinex/bitfinex_types.go
View File

@@ -6,8 +6,7 @@ import (
"github.com/thrasher-corp/gocryptotrader/exchanges/order"
)
// AcceptedOrderType defines the accepted market types, exchange strings denote
// non-contract order types.
// AcceptedOrderType defines the accepted market types, exchange strings denote non-contract order types.
var AcceptedOrderType = []string{"market", "limit", "stop", "trailing-stop",
"fill-or-kill", "exchange market", "exchange limit", "exchange stop",
"exchange trailing-stop", "exchange fill-or-kill"}
@@ -377,7 +376,6 @@ type WebsocketBook struct {
ID int64
Price float64
Amount float64
Rate float64
Period int64
}

408
exchanges/bitfinex/bitfinex_websocket.go
View File

@@ -4,9 +4,12 @@ import (
"encoding/json"
"errors"
"fmt"
"hash/crc32"
"net/http"
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/gorilla/websocket"
@@ -26,6 +29,15 @@ import (
var comms = make(chan stream.Response)
type checksum struct {
Token int
Sequence int64
}
// checksumStore quick global for now
var checksumStore = make(map[int]*checksum)
var cMtx sync.Mutex
// WsConnect starts a new websocket connection
func (b *Bitfinex) WsConnect() error {
if !b.Websocket.IsEnabled() || !b.IsEnabled() {
@@ -39,6 +51,7 @@ func (b *Bitfinex) WsConnect() error {
b.Name,
err)
}
go b.wsReadData(b.Websocket.Conn)
if b.Websocket.CanUseAuthenticatedEndpoints() {
@@ -139,14 +152,42 @@ func (b *Bitfinex) wsHandleData(respRaw []byte) error {
}
}
case []interface{}:
if hb, ok := d[1].(string); ok {
chanF, ok := d[0].(float64)
if !ok {
return errors.New("channel ID type assertion failure")
}
chanID := int(chanF)
var datum string
if datum, ok = d[1].(string); ok {
// Capturing heart beat
if hb == "hb" {
if datum == "hb" {
return nil
}
// Capturing checksum and storing value
if datum == "cs" {
var tokenF float64
tokenF, ok = d[2].(float64)
if !ok {
return errors.New("checksum token type assertion failure")
}
var seqNoF float64
seqNoF, ok = d[3].(float64)
if !ok {
return errors.New("sequence number type assertion failure")
}
cMtx.Lock()
checksumStore[chanID] = &checksum{
Token: int(tokenF),
Sequence: int64(seqNoF),
}
cMtx.Unlock()
return nil
}
}
chanID := int(d[0].(float64))
chanInfo, ok := b.WebsocketSubdChannels[chanID]
if !ok && chanID != 0 {
return fmt.Errorf("bitfinex.go error - Unable to locate chanID: %d",
@@ -198,43 +239,81 @@ func (b *Bitfinex) wsHandleData(respRaw []byte) error {
if len(obSnapBundle) == 0 {
return errors.New("no data within orderbook snapshot")
}
sequenceNo, ok := d[2].(float64)
if !ok {
return errors.New("type assertion failure")
}
var fundingRate bool
switch id := obSnapBundle[0].(type) {
case []interface{}:
for i := range obSnapBundle {
data := obSnapBundle[i].([]interface{})
id, okAssert := data[0].(float64)
if !okAssert {
return errors.New("type assertion failed for orderbook item data")
}
pricePeriod, okAssert := data[1].(float64)
if !okAssert {
return errors.New("type assertion failed for orderbook item data")
}
rateAmount, okAssert := data[2].(float64)
if !okAssert {
return errors.New("type assertion failed for orderbook item data")
}
if len(data) == 4 {
fundingRate = true
amount, okFunding := data[3].(float64)
if !okFunding {
return errors.New("type assertion failed for orderbook item data")
}
newOrderbook = append(newOrderbook, WebsocketBook{
ID: int64(data[0].(float64)),
Period: int64(data[1].(float64)),
Rate: data[2].(float64),
Amount: data[3].(float64)})
ID: int64(id),
Period: int64(pricePeriod),
Price: rateAmount,
Amount: amount})
} else {
newOrderbook = append(newOrderbook, WebsocketBook{
ID: int64(data[0].(float64)),
Price: data[1].(float64),
Amount: data[2].(float64)})
ID: int64(id),
Price: pricePeriod,
Amount: rateAmount})
}
}
err := b.WsInsertSnapshot(pair, chanAsset, newOrderbook)
err := b.WsInsertSnapshot(pair, chanAsset, newOrderbook, fundingRate)
if err != nil {
return fmt.Errorf("bitfinex_websocket.go inserting snapshot error: %s",
err)
}
case float64:
pricePeriod, okSnap := obSnapBundle[1].(float64)
if !okSnap {
return errors.New("type assertion failed for orderbook snapshot data")
}
amountRate, okSnap := obSnapBundle[2].(float64)
if !okSnap {
return errors.New("type assertion failed for orderbook snapshot data")
}
if len(obSnapBundle) == 4 {
fundingRate = true
var amount float64
amount, okSnap = obSnapBundle[3].(float64)
if !okSnap {
return errors.New("type assertion failed for orderbook snapshot data")
}
newOrderbook = append(newOrderbook, WebsocketBook{
ID: int64(id),
Period: int64(obSnapBundle[1].(float64)),
Rate: obSnapBundle[2].(float64),
Amount: obSnapBundle[3].(float64)})
Period: int64(pricePeriod),
Price: amountRate,
Amount: amount})
} else {
newOrderbook = append(newOrderbook, WebsocketBook{
ID: int64(id),
Price: obSnapBundle[1].(float64),
Amount: obSnapBundle[2].(float64)})
Price: pricePeriod,
Amount: amountRate})
}
err := b.WsUpdateOrderbook(pair, chanAsset, newOrderbook)
err := b.WsUpdateOrderbook(pair, chanAsset, newOrderbook, chanID, int64(sequenceNo), fundingRate)
if err != nil {
return fmt.Errorf("bitfinex_websocket.go updating orderbook error: %s",
err)
@@ -282,17 +361,38 @@ func (b *Bitfinex) wsHandleData(respRaw []byte) error {
return nil
case wsTicker:
tickerData := d[1].([]interface{})
b.Websocket.DataHandler <- &ticker.Price{
ExchangeName: b.Name,
Bid: tickerData[0].(float64),
Ask: tickerData[2].(float64),
Last: tickerData[6].(float64),
Volume: tickerData[7].(float64),
High: tickerData[8].(float64),
Low: tickerData[9].(float64),
AssetType: chanAsset,
Pair: pair,
if len(tickerData) == 10 {
b.Websocket.DataHandler <- &ticker.Price{
ExchangeName: b.Name,
Bid: tickerData[0].(float64),
Ask: tickerData[2].(float64),
Last: tickerData[6].(float64),
Volume: tickerData[7].(float64),
High: tickerData[8].(float64),
Low: tickerData[9].(float64),
AssetType: chanAsset,
Pair: pair,
}
} else {
b.Websocket.DataHandler <- &ticker.Price{
ExchangeName: b.Name,
FlashReturnRate: tickerData[0].(float64),
Bid: tickerData[1].(float64),
BidPeriod: tickerData[2].(float64),
BidSize: tickerData[3].(float64),
Ask: tickerData[4].(float64),
AskPeriod: tickerData[5].(float64),
AskSize: tickerData[6].(float64),
Last: tickerData[9].(float64),
Volume: tickerData[10].(float64),
High: tickerData[11].(float64),
Low: tickerData[12].(float64),
FlashReturnRateAmount: tickerData[15].(float64),
AssetType: chanAsset,
Pair: pair,
}
}
return nil
case wsTrades:
if !b.IsSaveTradeDataEnabled() {
@@ -833,77 +933,120 @@ func (b *Bitfinex) wsHandleOrder(data []interface{}) {
// WsInsertSnapshot add the initial orderbook snapshot when subscribed to a
// channel
func (b *Bitfinex) WsInsertSnapshot(p currency.Pair, assetType asset.Item, books []WebsocketBook) error {
func (b *Bitfinex) WsInsertSnapshot(p currency.Pair, assetType asset.Item, books []WebsocketBook, fundingRate bool) error {
if len(books) == 0 {
return errors.New("bitfinex.go error - no orderbooks submitted")
}
var bid, ask []orderbook.Item
var book orderbook.Base
for i := range books {
if books[i].Amount > 0 {
bid = append(bid, orderbook.Item{
ID: books[i].ID,
Amount: books[i].Amount,
Price: books[i].Price})
item := orderbook.Item{
ID: books[i].ID,
Amount: books[i].Amount,
Price: books[i].Price,
Period: books[i].Period,
}
if fundingRate {
if item.Amount < 0 {
item.Amount *= -1
book.Bids = append(book.Bids, item)
} else {
book.Asks = append(book.Asks, item)
}
} else {
ask = append(ask, orderbook.Item{
ID: books[i].ID,
Amount: books[i].Amount * -1,
Price: books[i].Price})
if books[i].Amount > 0 {
book.Bids = append(book.Bids, item)
} else {
item.Amount *= -1
book.Asks = append(book.Asks, item)
}
}
}
var newOrderBook orderbook.Base
newOrderBook.Asks = ask
newOrderBook.AssetType = assetType
newOrderBook.Bids = bid
newOrderBook.Pair = p
newOrderBook.ExchangeName = b.Name
return b.Websocket.Orderbook.LoadSnapshot(&newOrderBook)
book.AssetType = assetType
book.Pair = p
book.ExchangeName = b.Name
book.NotAggregated = true
book.IsFundingRate = fundingRate
return b.Websocket.Orderbook.LoadSnapshot(&book)
}
// WsUpdateOrderbook updates the orderbook list, removing and adding to the
// orderbook sides
func (b *Bitfinex) WsUpdateOrderbook(p currency.Pair, assetType asset.Item, book []WebsocketBook) error {
orderbookUpdate := buffer.Update{
Asset: assetType,
Pair: p,
}
func (b *Bitfinex) WsUpdateOrderbook(p currency.Pair, assetType asset.Item, book []WebsocketBook, channelID int, sequenceNo int64, fundingRate bool) error {
orderbookUpdate := buffer.Update{Asset: assetType, Pair: p}
for i := range book {
switch {
case book[i].Price > 0:
orderbookUpdate.Action = "update/insert"
if book[i].Amount > 0 {
// update bid
orderbookUpdate.Bids = append(orderbookUpdate.Bids,
orderbook.Item{
ID: book[i].ID,
Amount: book[i].Amount,
Price: book[i].Price})
} else if book[i].Amount < 0 {
// update ask
orderbookUpdate.Asks = append(orderbookUpdate.Asks,
orderbook.Item{
ID: book[i].ID,
Amount: book[i].Amount * -1,
Price: book[i].Price})
item := orderbook.Item{
ID: book[i].ID,
Amount: book[i].Amount,
Price: book[i].Price,
Period: book[i].Period,
}
if book[i].Price > 0 {
orderbookUpdate.Action = buffer.UpdateInsert
if fundingRate {
if book[i].Amount < 0 {
item.Amount *= -1
orderbookUpdate.Bids = append(orderbookUpdate.Bids, item)
} else {
orderbookUpdate.Asks = append(orderbookUpdate.Asks, item)
}
} else {
if book[i].Amount > 0 {
orderbookUpdate.Bids = append(orderbookUpdate.Bids, item)
} else {
item.Amount *= -1
orderbookUpdate.Asks = append(orderbookUpdate.Asks, item)
}
}
case book[i].Price == 0:
orderbookUpdate.Action = "delete"
if book[i].Amount == 1 {
// delete bid
orderbookUpdate.Bids = append(orderbookUpdate.Bids,
orderbook.Item{
ID: book[i].ID})
} else if book[i].Amount == -1 {
// delete ask
orderbookUpdate.Asks = append(orderbookUpdate.Asks,
orderbook.Item{
ID: book[i].ID})
} else {
orderbookUpdate.Action = buffer.Delete
if fundingRate {
if book[i].Amount == 1 {
// delete bid
orderbookUpdate.Asks = append(orderbookUpdate.Asks, item)
} else {
// delete ask
orderbookUpdate.Bids = append(orderbookUpdate.Bids, item)
}
} else {
if book[i].Amount == 1 {
// delete bid
orderbookUpdate.Bids = append(orderbookUpdate.Bids, item)
} else {
// delete ask
orderbookUpdate.Asks = append(orderbookUpdate.Asks, item)
}
}
}
}
cMtx.Lock()
checkme := checksumStore[channelID]
if checkme == nil {
cMtx.Unlock()
return b.Websocket.Orderbook.Update(&orderbookUpdate)
}
checksumStore[channelID] = nil
cMtx.Unlock()
if checkme.Sequence+1 == sequenceNo {
// Sequence numbers get dropped, if checksum is not in line with
// sequence, do not check.
ob := b.Websocket.Orderbook.GetOrderbook(p, assetType)
if ob == nil {
return fmt.Errorf("cannot calculate websocket checksum: book not found for %s %s",
p,
assetType)
}
err := validateCRC32(ob, checkme.Token)
if err != nil {
return err
}
}
return b.Websocket.Orderbook.Update(&orderbookUpdate)
}
@@ -961,6 +1104,14 @@ func (b *Bitfinex) GenerateDefaultSubscriptions() ([]stream.ChannelSubscription,
// Subscribe sends a websocket message to receive data from the channel
func (b *Bitfinex) Subscribe(channelsToSubscribe []stream.ChannelSubscription) error {
var errs common.Errors
checksum := make(map[string]interface{})
checksum["event"] = "conf"
checksum["flags"] = bitfinexChecksumFlag + bitfinexWsSequenceFlag
err := b.Websocket.Conn.SendJSONMessage(checksum)
if err != nil {
return err
}
for i := range channelsToSubscribe {
req := make(map[string]interface{})
req["event"] = "subscribe"
@@ -1192,3 +1343,100 @@ func (b *Bitfinex) WsCancelOffer(orderID int64) error {
func makeRequestInterface(channelName string, data interface{}) []interface{} {
return []interface{}{0, channelName, nil, data}
}
func validateCRC32(book *orderbook.Base, token int) error {
// Order ID's need to be sub-sorted in ascending order, this needs to be
// done on the main book to ensure that we do not cut price levels out below
reOrderByID(book.Bids)
reOrderByID(book.Asks)
// RO precision calculation is based on order ID's and amount values
var bids, asks []orderbook.Item
for i := 0; i < 25; i++ {
if i < len(book.Bids) {
bids = append(bids, book.Bids[i])
}
if i < len(book.Asks) {
asks = append(asks, book.Asks[i])
}
}
// ensure '-' (negative amount) is passed back to string buffer as
// this is needed for calcs - These get swapped if funding rate
bidmod := float64(1)
if book.IsFundingRate {
bidmod = -1
}
askMod := float64(-1)
if book.IsFundingRate {
askMod = 1
}
var check strings.Builder
for i := 0; i < 25; i++ {
if i < len(bids) {
check.WriteString(strconv.FormatInt(bids[i].ID, 10))
check.WriteString(":")
check.WriteString(strconv.FormatFloat(bidmod*bids[i].Amount, 'f', -1, 64))
check.WriteString(":")
}
if i < len(asks) {
check.WriteString(strconv.FormatInt(asks[i].ID, 10))
check.WriteString(":")
check.WriteString(strconv.FormatFloat(askMod*asks[i].Amount, 'f', -1, 64))
check.WriteString(":")
}
}
checksumStr := strings.TrimSuffix(check.String(), ":")
checksum := crc32.ChecksumIEEE([]byte(checksumStr))
if checksum == uint32(token) {
return nil
}
return fmt.Errorf("invalid checksum for %s %s: calculated [%d] does not match [%d]",
book.AssetType,
book.Pair,
checksum,
uint32(token))
}
// reOrderByID sub sorts orderbook items by its corresponding ID when price
// levels are the same. TODO: Deprecate and shift to buffer level insertion
// based off ascending ID.
func reOrderByID(depth []orderbook.Item) {
subSort:
for x := 0; x < len(depth); {
var subset []orderbook.Item
// Traverse forward elements
for y := x + 1; y < len(depth); y++ {
if depth[x].Price == depth[y].Price &&
// Period matching is for funding rates, this was undocumented
// but these need to be matched with price for the correct ID
// alignment
depth[x].Period == depth[y].Period {
// Append element to subset when price match occurs
subset = append(subset, depth[y])
// Traverse next
continue
}
if len(subset) != 0 {
// Append root element
subset = append(subset, depth[x])
// Sort IDs by ascending
sort.Slice(subset, func(i, j int) bool {
return subset[i].ID < subset[j].ID
})
// Re-align elements with sorted ID subset
for z := range subset {
depth[x+z] = subset[z]
}
}
// When price is not matching change checked element to root
x = y
continue subSort
}
break
}
}

63
exchanges/bitfinex/bitfinex_wrapper.go
View File

@@ -199,6 +199,7 @@ func (b *Bitfinex) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: b.GenerateDefaultSubscriptions,
Features: &b.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
UpdateEntriesByID: true,
})
if err != nil {
@@ -385,9 +386,15 @@ func (b *Bitfinex) FetchOrderbook(p currency.Pair, assetType asset.Item) (*order
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *Bitfinex) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
o := &orderbook.Base{
ExchangeName: b.Name,
Pair: p,
AssetType: assetType,
NotAggregated: true}
fPair, err := b.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return o, err
}
b.appendOptionalDelimiter(&fPair)
var prefix = "t"
@@ -395,30 +402,46 @@ func (b *Bitfinex) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orde
prefix = "f"
}
orderbookNew, err := b.GetOrderbook(prefix+fPair.String(), "P0", 100)
orderbookNew, err := b.GetOrderbook(prefix+fPair.String(), "R0", 100)
if err != nil {
return nil, err
return o, err
}
var o orderbook.Base
for x := range orderbookNew.Asks {
o.Asks = append(o.Asks, orderbook.Item{
Price: orderbookNew.Asks[x].Price,
Amount: orderbookNew.Asks[x].Amount,
})
if assetType == asset.MarginFunding {
o.IsFundingRate = true
for x := range orderbookNew.Asks {
o.Asks = append(o.Asks, orderbook.Item{
ID: orderbookNew.Asks[x].OrderID,
Price: orderbookNew.Asks[x].Rate,
Amount: orderbookNew.Asks[x].Amount,
Period: int64(orderbookNew.Asks[x].Period),
})
}
for x := range orderbookNew.Bids {
o.Bids = append(o.Bids, orderbook.Item{
ID: orderbookNew.Bids[x].OrderID,
Price: orderbookNew.Bids[x].Rate,
Amount: orderbookNew.Bids[x].Amount,
Period: int64(orderbookNew.Bids[x].Period),
})
}
} else {
for x := range orderbookNew.Asks {
o.Asks = append(o.Asks, orderbook.Item{
ID: orderbookNew.Asks[x].OrderID,
Price: orderbookNew.Asks[x].Price,
Amount: orderbookNew.Asks[x].Amount,
})
}
for x := range orderbookNew.Bids {
o.Bids = append(o.Bids, orderbook.Item{
ID: orderbookNew.Bids[x].OrderID,
Price: orderbookNew.Bids[x].Price,
Amount: orderbookNew.Bids[x].Amount,
})
}
}
for x := range orderbookNew.Bids {
o.Bids = append(o.Bids, orderbook.Item{
Price: orderbookNew.Bids[x].Price,
Amount: orderbookNew.Bids[x].Amount,
})
}
o.Pair = fPair
o.ExchangeName = b.Name
o.AssetType = assetType
err = o.Process()
if err != nil {
return nil, err

24
exchanges/bitflyer/bitflyer_wrapper.go
View File

@@ -250,33 +250,33 @@ func (b *Bitflyer) FetchOrderbook(p currency.Pair, assetType asset.Item) (*order
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *Bitflyer) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: b.Name, Pair: p, AssetType: assetType}
fPair, err := b.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
orderbookNew, err := b.GetOrderBook(b.CheckFXString(fPair).String())
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{Price: orderbookNew.Asks[x].Price, Amount: orderbookNew.Asks[x].Size})
book.Asks = append(book.Asks, orderbook.Item{
Price: orderbookNew.Asks[x].Price,
Amount: orderbookNew.Asks[x].Size})
}
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{Price: orderbookNew.Bids[x].Price, Amount: orderbookNew.Bids[x].Size})
book.Bids = append(book.Bids, orderbook.Item{
Price: orderbookNew.Bids[x].Price,
Amount: orderbookNew.Bids[x].Size})
}
orderBook.Pair = fPair
orderBook.ExchangeName = b.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(b.Name, fPair, assetType)

17
exchanges/bithumb/bithumb_wrapper.go
View File

@@ -242,16 +242,16 @@ func (b *Bithumb) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderb
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *Bithumb) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: b.Name, Pair: p, AssetType: assetType}
curr := p.Base.String()
orderbookNew, err := b.GetOrderBook(curr)
if err != nil {
return orderBook, err
return book, err
}
for i := range orderbookNew.Data.Bids {
orderBook.Bids = append(orderBook.Bids,
book.Bids = append(book.Bids,
orderbook.Item{
Amount: orderbookNew.Data.Bids[i].Quantity,
Price: orderbookNew.Data.Bids[i].Price,
@@ -259,22 +259,17 @@ func (b *Bithumb) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*order
}
for i := range orderbookNew.Data.Asks {
orderBook.Asks = append(orderBook.Asks,
book.Asks = append(book.Asks,
orderbook.Item{
Amount: orderbookNew.Data.Asks[i].Quantity,
Price: orderbookNew.Data.Asks[i].Price,
})
}
orderBook.Pair = p
orderBook.ExchangeName = b.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(b.Name, p, assetType)
}

5
exchanges/bitmex/bitmex_test.go
View File

@@ -868,9 +868,12 @@ func TestWSOrderbookHandling(t *testing.T) {
]
}`)
err = b.wsHandleData(pressXToJSON)
if err != nil {
if err != nil && err.Error() != "perpetualcontract ETHUSD update cannot be deleted id: 17999995000 not found" {
t.Error(err)
}
if err == nil {
t.Error("expecting error")
}
}
func TestWSDeleveragePositionUpdateHandling(t *testing.T) {

94
exchanges/bitmex/bitmex_websocket.go
View File

@@ -491,27 +491,29 @@ func (b *Bitmex) processOrderbook(data []OrderBookL2, action string, p currency.
switch action {
case bitmexActionInitialData:
var newOrderBook orderbook.Base
var book orderbook.Base
for i := range data {
if strings.EqualFold(data[i].Side, order.Sell.String()) {
newOrderBook.Asks = append(newOrderBook.Asks, orderbook.Item{
Price: data[i].Price,
Amount: float64(data[i].Size),
ID: data[i].ID,
})
continue
}
newOrderBook.Bids = append(newOrderBook.Bids, orderbook.Item{
item := orderbook.Item{
Price: data[i].Price,
Amount: float64(data[i].Size),
ID: data[i].ID,
})
}
switch {
case strings.EqualFold(data[i].Side, order.Sell.String()):
book.Asks = append(book.Asks, item)
case strings.EqualFold(data[i].Side, order.Buy.String()):
book.Bids = append(book.Bids, item)
default:
return fmt.Errorf("could not process websocket orderbook update, order side could not be matched for %s",
data[i].Side)
}
}
newOrderBook.AssetType = a
newOrderBook.Pair = p
newOrderBook.ExchangeName = b.Name
orderbook.Reverse(book.Asks) // Reverse asks for correct alignment
book.AssetType = a
book.Pair = p
book.ExchangeName = b.Name
err := b.Websocket.Orderbook.LoadSnapshot(&newOrderBook)
err := b.Websocket.Orderbook.LoadSnapshot(&book)
if err != nil {
return fmt.Errorf("bitmex_websocket.go process orderbook error - %s",
err)
@@ -519,17 +521,16 @@ func (b *Bitmex) processOrderbook(data []OrderBookL2, action string, p currency.
default:
var asks, bids []orderbook.Item
for i := range data {
if strings.EqualFold(data[i].Side, "Sell") {
asks = append(asks, orderbook.Item{
Amount: float64(data[i].Size),
ID: data[i].ID,
})
continue
}
bids = append(bids, orderbook.Item{
nItem := orderbook.Item{
Price: data[i].Price,
Amount: float64(data[i].Size),
ID: data[i].ID,
})
}
if strings.EqualFold(data[i].Side, "Sell") {
asks = append(asks, nItem)
continue
}
bids = append(bids, nItem)
}
err := b.Websocket.Orderbook.Update(&buffer.Update{
@@ -537,7 +538,7 @@ func (b *Bitmex) processOrderbook(data []OrderBookL2, action string, p currency.
Asks: asks,
Pair: p,
Asset: a,
Action: action,
Action: buffer.Action(action),
})
if err != nil {
return err
@@ -548,24 +549,6 @@ func (b *Bitmex) processOrderbook(data []OrderBookL2, action string, p currency.
// GenerateDefaultSubscriptions Adds default subscriptions to websocket to be handled by ManageSubscriptions()
func (b *Bitmex) GenerateDefaultSubscriptions() ([]stream.ChannelSubscription, error) {
assets := b.GetAssetTypes()
var allPairs currency.Pairs
var associatedAssets []asset.Item
for x := range assets {
contracts, err := b.GetEnabledPairs(assets[x])
if err != nil {
return nil, err
}
for y := range contracts {
allPairs = allPairs.Add(contracts[y])
associatedAssets = append(associatedAssets, assets[x])
}
}
if len(allPairs) != len(associatedAssets) {
return nil, fmt.Errorf("%s generate default subscriptions: pair and asset type len mismatch", b.Name)
}
channels := []string{bitmexWSOrderbookL2, bitmexWSTrade}
subscriptions := []stream.ChannelSubscription{
{
@@ -573,13 +556,24 @@ func (b *Bitmex) GenerateDefaultSubscriptions() ([]stream.ChannelSubscription, e
},
}
for i := range channels {
for j := range allPairs {
subscriptions = append(subscriptions, stream.ChannelSubscription{
Channel: channels[i] + ":" + allPairs[j].String(),
Currency: allPairs[j],
Asset: associatedAssets[j],
})
assets := b.GetAssetTypes()
for x := range assets {
contracts, err := b.GetEnabledPairs(assets[x])
if err != nil {
return nil, err
}
for y := range contracts {
for z := range channels {
if assets[x] == asset.Index && channels[z] == bitmexWSOrderbookL2 {
// There are no L2 orderbook for index assets
continue
}
subscriptions = append(subscriptions, stream.ChannelSubscription{
Channel: channels[z] + ":" + contracts[y].String(),
Currency: contracts[y],
Asset: assets[x],
})
}
}
}
return subscriptions, nil

39
exchanges/bitmex/bitmex_wrapper.go
View File

@@ -155,6 +155,7 @@ func (b *Bitmex) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: b.GenerateDefaultSubscriptions,
Features: &b.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
UpdateEntriesByID: true,
})
if err != nil {
@@ -332,46 +333,50 @@ func (b *Bitmex) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *Bitmex) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{
ExchangeName: b.Name,
Pair: p,
AssetType: assetType,
}
if assetType == asset.Index {
return nil, common.ErrFunctionNotSupported
return book, common.ErrFunctionNotSupported
}
fpair, err := b.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := b.GetOrderbook(OrderBookGetL2Params{
Symbol: fpair.String(),
Depth: 500})
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
for i := range orderbookNew {
if strings.EqualFold(orderbookNew[i].Side, order.Sell.String()) {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
switch {
case strings.EqualFold(orderbookNew[i].Side, order.Sell.String()):
book.Asks = append(book.Asks, orderbook.Item{
Amount: float64(orderbookNew[i].Size),
Price: orderbookNew[i].Price})
continue
}
if strings.EqualFold(orderbookNew[i].Side, order.Buy.String()) {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
case strings.EqualFold(orderbookNew[i].Side, order.Buy.String()):
book.Bids = append(book.Bids, orderbook.Item{
Amount: float64(orderbookNew[i].Size),
Price: orderbookNew[i].Price})
default:
return book,
fmt.Errorf("could not process orderbook, order side [%s] could not be matched",
orderbookNew[i].Side)
}
}
orderbook.Reverse(book.Asks)
orderBook.Pair = p
orderBook.ExchangeName = b.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(b.Name, p, assetType)
}

2
exchanges/bitstamp/bitstamp_test.go
View File

File diff suppressed because one or more lines are too long

21
exchanges/bitstamp/bitstamp_wrapper.go
View File

@@ -165,6 +165,7 @@ func (b *Bitstamp) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: b.generateDefaultSubscriptions,
Features: &b.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
})
if err != nil {
return err
@@ -315,40 +316,34 @@ func (b *Bitstamp) FetchOrderbook(p currency.Pair, assetType asset.Item) (*order
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *Bitstamp) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: b.Name, Pair: p, AssetType: assetType}
fPair, err := b.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
orderbookNew, err := b.GetOrderbook(fPair.String())
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Amount,
Price: orderbookNew.Bids[x].Price,
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x].Amount,
Price: orderbookNew.Asks[x].Price,
})
}
orderBook.Pair = fPair
orderBook.ExchangeName = b.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(b.Name, fPair, assetType)
}

18
exchanges/bittrex/bittrex_wrapper.go
View File

@@ -328,9 +328,10 @@ func (b *Bittrex) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderb
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *Bittrex) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: b.Name, Pair: p, AssetType: assetType}
fpair, err := b.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := b.GetOrderbook(fpair.String())
@@ -338,9 +339,8 @@ func (b *Bittrex) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*order
return nil, err
}
orderBook := new(orderbook.Base)
for x := range orderbookNew.Result.Buy {
orderBook.Bids = append(orderBook.Bids,
book.Bids = append(book.Bids,
orderbook.Item{
Amount: orderbookNew.Result.Buy[x].Quantity,
Price: orderbookNew.Result.Buy[x].Rate,
@@ -349,23 +349,17 @@ func (b *Bittrex) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*order
}
for x := range orderbookNew.Result.Sell {
orderBook.Asks = append(orderBook.Asks,
book.Asks = append(book.Asks,
orderbook.Item{
Amount: orderbookNew.Result.Sell[x].Quantity,
Price: orderbookNew.Result.Sell[x].Rate,
},
)
}
orderBook.Pair = p
orderBook.ExchangeName = b.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(b.Name, p, assetType)
}

2
exchanges/btcmarkets/btcmarkets.go
View File

@@ -670,7 +670,7 @@ func (b *BTCMarkets) GetBatchTrades(ids []string) (BatchTradeResponse, error) {
request.Auth)
}
// CancelBatchOrders cancels given ids
// CancelBatch cancels given ids
func (b *BTCMarkets) CancelBatch(ids []string) (BatchCancelResponse, error) {
var resp BatchCancelResponse
marketIDs := strings.Join(ids, ",")

2
exchanges/btcmarkets/btcmarkets_websocket.go
View File

@@ -119,7 +119,7 @@ func (b *BTCMarkets) wsHandleData(respRaw []byte) error {
if ob.Snapshot {
err = b.Websocket.Orderbook.LoadSnapshot(&orderbook.Base{
Pair: p,
Bids: bids,
Bids: orderbook.SortBids(bids), // Alignment completely out sort is needed
Asks: asks,
LastUpdated: ob.Timestamp,
AssetType: asset.Spot,

23
exchanges/btcmarkets/btcmarkets_wrapper.go
View File

@@ -156,7 +156,7 @@ func (b *BTCMarkets) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: b.generateDefaultSubscriptions,
Features: &b.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: true,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
})
if err != nil {
@@ -351,33 +351,34 @@ func (b *BTCMarkets) FetchOrderbook(p currency.Pair, assetType asset.Item) (*ord
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *BTCMarkets) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{
ExchangeName: b.Name,
Pair: p,
AssetType: assetType, NotAggregated: true}
fpair, err := b.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
tempResp, err := b.GetOrderbook(fpair.String(), 2)
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
for x := range tempResp.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: tempResp.Bids[x].Volume,
Price: tempResp.Bids[x].Price})
}
for y := range tempResp.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: tempResp.Asks[y].Volume,
Price: tempResp.Asks[y].Price})
}
orderBook.Pair = p
orderBook.ExchangeName = b.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(b.Name, p, assetType)
}

16
exchanges/btse/btse_test.go
View File

@@ -849,3 +849,19 @@ func TestGetHistoricTrades(t *testing.T) {
t.Error("unexpected error")
}
}
func TestOrderbookFilter(t *testing.T) {
t.Parallel()
if !b.orderbookFilter(0, 1) {
t.Fatal("incorrect filtering")
}
if !b.orderbookFilter(1, 0) {
t.Fatal("incorrect filtering")
}
if !b.orderbookFilter(0, 0) {
t.Fatal("incorrect filtering")
}
if b.orderbookFilter(1, 1) {
t.Fatal("incorrect filtering")
}
}

28
exchanges/btse/btse_websocket.go
View File

@@ -111,7 +111,7 @@ func (b *BTSE) wsHandleData(respRaw []byte) error {
var result Result
err := json.Unmarshal(respRaw, &result)
if err != nil {
if strings.Contains(string(respRaw), "UNLOGIN_USER connect success") ||
if strings.Contains(string(respRaw), "connect success") ||
strings.Contains(string(respRaw), "authenticated successfully") {
return nil
} else if strings.Contains(string(respRaw), "AUTHENTICATE ERROR") {
@@ -223,7 +223,7 @@ func (b *BTSE) wsHandleData(respRaw []byte) error {
})
}
return trade.AddTradesToBuffer(b.Name, trades...)
case strings.Contains(result["topic"].(string), "orderBookApi"):
case strings.Contains(result["topic"].(string), "orderBookL2Api"):
var t wsOrderBook
err = json.Unmarshal(respRaw, &t)
if err != nil {
@@ -242,6 +242,9 @@ func (b *BTSE) wsHandleData(respRaw []byte) error {
if err != nil {
return err
}
if b.orderbookFilter(price, amount) {
continue
}
newOB.Asks = append(newOB.Asks, orderbook.Item{
Price: price,
Amount: amount,
@@ -258,6 +261,9 @@ func (b *BTSE) wsHandleData(respRaw []byte) error {
if err != nil {
return err
}
if b.orderbookFilter(price, amount) {
continue
}
newOB.Bids = append(newOB.Bids, orderbook.Item{
Price: price,
Amount: amount,
@@ -275,6 +281,7 @@ func (b *BTSE) wsHandleData(respRaw []byte) error {
newOB.Pair = p
newOB.AssetType = a
newOB.ExchangeName = b.Name
orderbook.Reverse(newOB.Asks) // Reverse asks for correct alignment
err = b.Websocket.Orderbook.LoadSnapshot(&newOB)
if err != nil {
return err
@@ -286,9 +293,24 @@ func (b *BTSE) wsHandleData(respRaw []byte) error {
return nil
}
// orderbookFilter is needed on book levels from this exchange as their data
// is incorrect
func (b *BTSE) orderbookFilter(price, amount float64) bool {
// Amount filtering occurs when the amount exceeds the decimal returned.
// e.g. {"price":"1.37","size":"0.00"} currency: SFI-ETH
// Opted to not round up to 0.01 as this might skew calculations
// more than removing from the books completely.
// Price filtering occurs when we are deep in the bid book and there are
// prices that are less than 4 decimal places
// e.g. {"price":"0.0000","size":"14219"} currency: TRX-PAX
// We cannot load a zero price and this will ruin calculations
return price == 0 || amount == 0
}
// GenerateDefaultSubscriptions Adds default subscriptions to websocket to be handled by ManageSubscriptions()
func (b *BTSE) GenerateDefaultSubscriptions() ([]stream.ChannelSubscription, error) {
var channels = []string{"orderBookApi:%s_0", "tradeHistory:%s"}
var channels = []string{"orderBookL2Api:%s_0", "tradeHistory:%s"}
pairs, err := b.GetEnabledPairs(asset.Spot)
if err != nil {
return nil, err

22
exchanges/btse/btse_wrapper.go
View File

@@ -185,6 +185,7 @@ func (b *BTSE) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: b.GenerateDefaultSubscriptions,
Features: &b.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
})
if err != nil {
return err
@@ -318,32 +319,33 @@ func (b *BTSE) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbook
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (b *BTSE) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: b.Name, Pair: p, AssetType: assetType}
fPair, err := b.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
a, err := b.FetchOrderBook(fPair.String(), 0, 0, 0, assetType == asset.Spot)
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
for x := range a.BuyQuote {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Price: a.BuyQuote[x].Price,
Amount: a.BuyQuote[x].Size})
}
for x := range a.SellQuote {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Price: a.SellQuote[x].Price,
Amount: a.SellQuote[x].Size})
}
orderBook.Pair = p
orderBook.ExchangeName = b.Name
orderBook.AssetType = assetType
err = orderBook.Process()
orderbook.Reverse(book.Asks) // Reverse asks for correct alignment
book.Pair = p
book.ExchangeName = b.Name
book.AssetType = assetType
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(b.Name, p, assetType)
}

26
exchanges/coinbasepro/coinbasepro_wrapper.go
View File

@@ -167,7 +167,7 @@ func (c *CoinbasePro) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: c.GenerateDefaultSubscriptions,
Features: &c.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: true,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
})
if err != nil {
@@ -400,35 +400,33 @@ func (c *CoinbasePro) FetchOrderbook(p currency.Pair, assetType asset.Item) (*or
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (c *CoinbasePro) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: c.Name, Pair: p, AssetType: assetType}
fpair, err := c.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := c.GetOrderbook(fpair.String(), 2)
if err != nil {
return nil, err
return book, err
}
obNew := orderbookNew.(OrderbookL1L2)
orderBook := new(orderbook.Base)
for x := range obNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{Amount: obNew.Bids[x].Amount, Price: obNew.Bids[x].Price})
book.Bids = append(book.Bids, orderbook.Item{
Amount: obNew.Bids[x].Amount,
Price: obNew.Bids[x].Price})
}
for x := range obNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{Amount: obNew.Asks[x].Amount, Price: obNew.Asks[x].Price})
book.Asks = append(book.Asks, orderbook.Item{
Amount: obNew.Asks[x].Amount,
Price: obNew.Asks[x].Price})
}
orderBook.Pair = p
orderBook.ExchangeName = c.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(c.Name, p, assetType)
}

8
exchanges/coinbene/coinbene_types.go
View File

@@ -220,10 +220,10 @@ type WsOrderbookData struct {
Topic string `json:"topic"`
Action string `json:"action"`
Data []struct {
Bids [][]string `json:"bids"`
Asks [][]string `json:"asks"`
Version int64 `json:"version"`
Timestamp int64 `json:"timestamp"`
Bids [][2]string `json:"bids"`
Asks [][2]string `json:"asks"`
Version int64 `json:"version"`
Timestamp int64 `json:"timestamp"`
} `json:"data"`
}

33
exchanges/coinbene/coinbene_websocket.go
View File

@@ -233,6 +233,10 @@ func (c *Coinbene) wsHandleData(respRaw []byte) error {
return err
}
if len(orderBook.Data) != 1 {
return errors.New("incomplete orderbook data has been received")
}
newPair, err = c.getCurrencyFromWsTopic(assetType, orderBook.Topic)
if err != nil {
return err
@@ -254,14 +258,24 @@ func (c *Coinbene) wsHandleData(respRaw []byte) error {
})
}
for j := range orderBook.Data[0].Bids {
amount, err = strconv.ParseFloat(orderBook.Data[0].Bids[j][1], 64)
if err != nil {
return err
}
price, err = strconv.ParseFloat(orderBook.Data[0].Bids[j][0], 64)
if err != nil {
return err
}
if price == 0 {
// Last level is coming back as a float with not enough decimal
// places e.g. ["0.000","1001.95"]],
// This needs to be filtered out as this can skew orderbook
// calculations
continue
}
amount, err = strconv.ParseFloat(orderBook.Data[0].Bids[j][1], 64)
if err != nil {
return err
}
bids = append(bids, orderbook.Item{
Amount: amount,
Price: price,
@@ -427,11 +441,18 @@ func (c *Coinbene) getCurrencyFromWsTopic(assetType asset.Item, channelTopic str
// Subscribe sends a websocket message to receive data from the channel
func (c *Coinbene) Subscribe(channelsToSubscribe []stream.ChannelSubscription) error {
maxSubsPerHour := 240
if len(channelsToSubscribe) > maxSubsPerHour {
return fmt.Errorf("channel subscriptions length %d exceeds coinbene's limit of %d, try reducing enabled pairs",
len(channelsToSubscribe),
maxSubsPerHour)
}
var sub WsSub
sub.Operation = "subscribe"
// enabling all currencies can lead to a message too large being sent
// and no subscriptions being made
chanLimit := 10
chanLimit := 15
for i := range channelsToSubscribe {
if len(sub.Arguments) > chanLimit {
err := c.Websocket.Conn.SendJSONMessage(sub)
@@ -456,7 +477,7 @@ func (c *Coinbene) Unsubscribe(channelToUnsubscribe []stream.ChannelSubscription
unsub.Operation = "unsubscribe"
// enabling all currencies can lead to a message too large being sent
// and no unsubscribes being made
chanLimit := 10
chanLimit := 15
for i := range channelToUnsubscribe {
if len(unsub.Arguments) > chanLimit {
err := c.Websocket.Conn.SendJSONMessage(unsub)

21
exchanges/coinbene/coinbene_wrapper.go
View File

@@ -182,7 +182,7 @@ func (c *Coinbene) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: c.GenerateDefaultSubscriptions,
Features: &c.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: true,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
})
if err != nil {
@@ -405,15 +405,15 @@ func (c *Coinbene) FetchOrderbook(p currency.Pair, assetType asset.Item) (*order
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (c *Coinbene) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
resp := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: c.Name, Pair: p, AssetType: assetType}
if !c.SupportsAsset(assetType) {
return nil,
return book,
fmt.Errorf("%s does not support asset type %s", c.Name, assetType)
}
fpair, err := c.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
var tempResp Orderbook
@@ -428,11 +428,8 @@ func (c *Coinbene) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orde
)
}
if err != nil {
return nil, err
return book, err
}
resp.ExchangeName = c.Name
resp.Pair = p
resp.AssetType = assetType
for x := range tempResp.Asks {
item := orderbook.Item{
Price: tempResp.Asks[x].Price,
@@ -441,7 +438,7 @@ func (c *Coinbene) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orde
if assetType == asset.PerpetualSwap {
item.OrderCount = tempResp.Asks[x].Count
}
resp.Asks = append(resp.Asks, item)
book.Asks = append(book.Asks, item)
}
for x := range tempResp.Bids {
item := orderbook.Item{
@@ -451,11 +448,11 @@ func (c *Coinbene) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orde
if assetType == asset.PerpetualSwap {
item.OrderCount = tempResp.Bids[x].Count
}
resp.Bids = append(resp.Bids, item)
book.Bids = append(book.Bids, item)
}
err = resp.Process()
err = book.Process()
if err != nil {
return nil, err
return book, err
}
return orderbook.Get(c.Name, p, assetType)
}

6
exchanges/coinut/coinut_test.go
View File

@@ -650,9 +650,9 @@ func TestGetNonce(t *testing.T) {
func TestWsOrderbook(t *testing.T) {
pressXToJSON := []byte(`{
"buy":
[ { "count": 7, "price": "750.00000000", "qty": "0.07000000" },
{ "count": 1, "price": "751.00000000", "qty": "0.01000000" },
{ "count": 1, "price": "751.34500000", "qty": "0.01000000" } ],
[ { "count": 1, "price": "751.34500000", "qty": "0.01000000" },
{ "count": 1, "price": "751.00000000", "qty": "0.01000000" },
{ "count": 7, "price": "750.00000000", "qty": "0.07000000" } ],
"sell":
[ { "count": 6, "price": "750.58100000", "qty": "0.06000000" },
{ "count": 1, "price": "750.58200000", "qty": "0.01000000" },

30
exchanges/coinut/coinut_wrapper.go
View File

@@ -150,7 +150,7 @@ func (c *COINUT) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: c.GenerateDefaultSubscriptions,
Features: &c.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: true,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
SortBufferByUpdateIDs: true,
})
@@ -453,44 +453,42 @@ func (c *COINUT) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (c *COINUT) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: c.Name, Pair: p, AssetType: assetType}
err := c.loadInstrumentsIfNotLoaded()
if err != nil {
return orderBook, err
return book, err
}
fpair, err := c.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
instID := c.instrumentMap.LookupID(fpair.String())
if instID == 0 {
return orderBook, errLookupInstrumentID
return book, errLookupInstrumentID
}
orderbookNew, err := c.GetInstrumentOrderbook(instID, 200)
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Buy {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{Amount: orderbookNew.Buy[x].Quantity, Price: orderbookNew.Buy[x].Price})
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Buy[x].Quantity,
Price: orderbookNew.Buy[x].Price})
}
for x := range orderbookNew.Sell {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{Amount: orderbookNew.Sell[x].Quantity, Price: orderbookNew.Sell[x].Price})
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Sell[x].Quantity,
Price: orderbookNew.Sell[x].Price})
}
orderBook.Pair = p
orderBook.ExchangeName = c.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(c.Name, p, assetType)
}

35
exchanges/exmo/exmo_wrapper.go
View File

@@ -235,25 +235,31 @@ func (e *EXMO) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbook
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (e *EXMO) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
callingBook := &orderbook.Base{ExchangeName: e.Name, Pair: p, AssetType: assetType}
enabledPairs, err := e.GetEnabledPairs(assetType)
if err != nil {
return nil, err
return callingBook, err
}
pairsCollated, err := e.FormatExchangeCurrencies(enabledPairs, assetType)
if err != nil {
return nil, err
return callingBook, err
}
result, err := e.GetOrderbook(pairsCollated)
if err != nil {
return nil, err
return callingBook, err
}
for i := range enabledPairs {
book := &orderbook.Base{
ExchangeName: e.Name,
Pair: enabledPairs[i],
AssetType: assetType}
curr, err := e.FormatExchangeCurrency(enabledPairs[i], assetType)
if err != nil {
return nil, err
return callingBook, err
}
data, ok := result[curr.String()]
@@ -261,20 +267,19 @@ func (e *EXMO) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderboo
continue
}
orderBook := new(orderbook.Base)
for y := range data.Ask {
var price, amount float64
price, err = strconv.ParseFloat(data.Ask[y][0], 64)
if err != nil {
return orderBook, err
return book, err
}
amount, err = strconv.ParseFloat(data.Ask[y][1], 64)
if err != nil {
return orderBook, err
return book, err
}
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Price: price,
Amount: amount,
})
@@ -284,27 +289,23 @@ func (e *EXMO) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderboo
var price, amount float64
price, err = strconv.ParseFloat(data.Bid[y][0], 64)
if err != nil {
return orderBook, err
return book, err
}
amount, err = strconv.ParseFloat(data.Bid[y][1], 64)
if err != nil {
return orderBook, err
return book, err
}
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Price: price,
Amount: amount,
})
}
orderBook.Pair = enabledPairs[i]
orderBook.ExchangeName = e.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
}
return orderbook.Get(e.Name, p, assetType)

18
exchanges/ftx/ftx.go
View File

@@ -123,7 +123,13 @@ func (f *FTX) GetOrderbook(marketName string, depth int64) (OrderbookData, error
result := struct {
Data TempOBData `json:"result"`
}{}
strDepth := strconv.FormatInt(depth, 10)
strDepth := "20" // If we send a zero value we get zero asks from the
// endpoint
if depth != 0 {
strDepth = strconv.FormatInt(depth, 10)
}
var resp OrderbookData
err := f.SendHTTPRequest(fmt.Sprintf(ftxAPIURL+getOrderbook, marketName, strDepth), &result)
if err != nil {
@@ -131,13 +137,15 @@ func (f *FTX) GetOrderbook(marketName string, depth int64) (OrderbookData, error
}
resp.MarketName = marketName
for x := range result.Data.Asks {
resp.Asks = append(resp.Asks, OData{Price: result.Data.Asks[x][0],
Size: result.Data.Asks[x][1],
resp.Asks = append(resp.Asks, OData{
Price: result.Data.Asks[x][0],
Size: result.Data.Asks[x][1],
})
}
for y := range result.Data.Bids {
resp.Bids = append(resp.Bids, OData{Price: result.Data.Bids[y][0],
Size: result.Data.Bids[y][1],
resp.Bids = append(resp.Bids, OData{
Price: result.Data.Bids[y][0],
Size: result.Data.Bids[y][1],
})
}
return resp, nil

20
exchanges/ftx/ftx_wrapper.go
View File

@@ -177,6 +177,7 @@ func (f *FTX) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: f.GenerateDefaultSubscriptions,
Features: &f.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
})
if err != nil {
return err
@@ -332,31 +333,28 @@ func (f *FTX) FetchOrderbook(currency currency.Pair, assetType asset.Item) (*ord
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (f *FTX) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: f.Name, Pair: p, AssetType: assetType}
formattedPair, err := f.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
tempResp, err := f.GetOrderbook(formattedPair.String(), 0)
tempResp, err := f.GetOrderbook(formattedPair.String(), 100)
if err != nil {
return orderBook, err
return book, err
}
for x := range tempResp.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: tempResp.Bids[x].Size,
Price: tempResp.Bids[x].Price})
}
for y := range tempResp.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: tempResp.Asks[y].Size,
Price: tempResp.Asks[y].Price})
}
orderBook.Pair = p
orderBook.ExchangeName = f.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(f.Name, p, assetType)
}

7
exchanges/gateio/gateio_types.go
View File

@@ -525,3 +525,10 @@ type TradeHistoryEntry struct {
TradeID string `json:"tradeID"`
Type string `json:"type"`
}
// wsOrderbook defines a websocket orderbook
type wsOrderbook struct {
Asks [][]string `json:"asks"`
Bids [][]string `json:"bids"`
ID int64 `json:"id"`
}

42
exchanges/gateio/gateio_websocket.go
View File

@@ -297,7 +297,8 @@ func (g *Gateio) wsHandleData(respRaw []byte) error {
case strings.Contains(result.Method, "depth"):
var IsSnapshot bool
var c string
var data = make(map[string][][]string)
var data wsOrderbook
err = json.Unmarshal(result.Params[0], &IsSnapshot)
if err != nil {
return err
@@ -314,42 +315,29 @@ func (g *Gateio) wsHandleData(respRaw []byte) error {
}
var asks, bids []orderbook.Item
askData, askOk := data["asks"]
for i := range askData {
var amount, price float64
amount, err = strconv.ParseFloat(askData[i][1], 64)
var amount, price float64
for i := range data.Asks {
amount, err = strconv.ParseFloat(data.Asks[i][1], 64)
if err != nil {
return err
}
price, err = strconv.ParseFloat(askData[i][0], 64)
price, err = strconv.ParseFloat(data.Asks[i][0], 64)
if err != nil {
return err
}
asks = append(asks, orderbook.Item{
Amount: amount,
Price: price,
})
asks = append(asks, orderbook.Item{Amount: amount, Price: price})
}
bidData, bidOk := data["bids"]
for i := range bidData {
var amount, price float64
amount, err = strconv.ParseFloat(bidData[i][1], 64)
for i := range data.Bids {
amount, err = strconv.ParseFloat(data.Bids[i][1], 64)
if err != nil {
return err
}
price, err = strconv.ParseFloat(bidData[i][0], 64)
price, err = strconv.ParseFloat(data.Bids[i][0], 64)
if err != nil {
return err
}
bids = append(bids, orderbook.Item{
Amount: amount,
Price: price,
})
}
if !askOk && !bidOk {
g.Websocket.DataHandler <- errors.New("gatio websocket error - cannot access ask or bid data")
bids = append(bids, orderbook.Item{Amount: amount, Price: price})
}
var p currency.Pair
@@ -359,14 +347,6 @@ func (g *Gateio) wsHandleData(respRaw []byte) error {
}
if IsSnapshot {
if !askOk {
g.Websocket.DataHandler <- errors.New("gatio websocket error - cannot access ask data")
}
if !bidOk {
g.Websocket.DataHandler <- errors.New("gatio websocket error - cannot access bid data")
}
var newOrderBook orderbook.Base
newOrderBook.Asks = asks
newOrderBook.Bids = bids

22
exchanges/gateio/gateio_wrapper.go
View File

@@ -164,7 +164,7 @@ func (g *Gateio) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: g.GenerateDefaultSubscriptions,
Features: &g.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: true,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
})
if err != nil {
return err
@@ -278,40 +278,34 @@ func (g *Gateio) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (g *Gateio) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: g.Name, Pair: p, AssetType: assetType}
curr, err := g.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := g.GetOrderbook(curr.String())
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Amount,
Price: orderbookNew.Bids[x].Price,
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x].Amount,
Price: orderbookNew.Asks[x].Price,
})
}
orderBook.Pair = p
orderBook.ExchangeName = g.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(g.Name, p, assetType)
}

2
exchanges/gemini/gemini_websocket.go
View File

@@ -389,6 +389,8 @@ func (g *Gemini) wsProcessUpdate(result WsMarketUpdateResponse, pair currency.Pa
})
}
}
orderbook.Reverse(bids) // Correct bid alignment
var newOrderBook orderbook.Base
newOrderBook.Asks = asks
newOrderBook.Bids = bids

26
exchanges/gemini/gemini_wrapper.go
View File

@@ -143,7 +143,7 @@ func (g *Gemini) Setup(exch *config.ExchangeConfig) error {
Connector: g.WsConnect,
Features: &g.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: true,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
})
}
@@ -294,34 +294,32 @@ func (g *Gemini) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (g *Gemini) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: g.Name, Pair: p, AssetType: assetType}
fPair, err := g.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
orderbookNew, err := g.GetOrderbook(fPair.String(), url.Values{})
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{Amount: orderbookNew.Bids[x].Amount, Price: orderbookNew.Bids[x].Price})
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Amount,
Price: orderbookNew.Bids[x].Price})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{Amount: orderbookNew.Asks[x].Amount, Price: orderbookNew.Asks[x].Price})
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x].Amount,
Price: orderbookNew.Asks[x].Price})
}
orderBook.Pair = fPair
orderBook.ExchangeName = g.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(g.Name, fPair, assetType)
}

2
exchanges/hitbtc/hitbtc_test.go
View File

@@ -661,7 +661,7 @@ func TestWsOrderbook(t *testing.T) {
},
{
"price": "0.054590",
"size": "0.000"
"size": "1.000"
},
{
"price": "0.054591",

22
exchanges/hitbtc/hitbtc_wrapper.go
View File

@@ -166,7 +166,7 @@ func (h *HitBTC) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: h.GenerateDefaultSubscriptions,
Features: &h.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: true,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
SortBufferByUpdateIDs: true,
})
@@ -366,40 +366,34 @@ func (h *HitBTC) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (h *HitBTC) UpdateOrderbook(c currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: h.Name, Pair: c, AssetType: assetType}
fpair, err := h.FormatExchangeCurrency(c, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := h.GetOrderbook(fpair.String(), 1000)
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Amount,
Price: orderbookNew.Bids[x].Price,
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x].Amount,
Price: orderbookNew.Asks[x].Price,
})
}
orderBook.Pair = c
orderBook.ExchangeName = h.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(h.Name, c, assetType)
}

21
exchanges/huobi/huobi_wrapper.go
View File

@@ -164,6 +164,7 @@ func (h *HUOBI) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: h.GenerateDefaultSubscriptions,
Features: &h.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
})
if err != nil {
return err
@@ -385,42 +386,36 @@ func (h *HUOBI) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderboo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (h *HUOBI) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: h.Name, Pair: p, AssetType: assetType}
fpair, err := h.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := h.GetDepth(OrderBookDataRequestParams{
Symbol: fpair.String(),
Type: OrderBookDataRequestParamsTypeStep0,
})
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x][1],
Price: orderbookNew.Bids[x][0],
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x][1],
Price: orderbookNew.Asks[x][0],
})
}
orderBook.Pair = p
orderBook.ExchangeName = h.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(h.Name, p, assetType)
}

2
exchanges/itbit/itbit_test.go
View File

@@ -61,7 +61,7 @@ func TestGetTicker(t *testing.T) {
func TestGetOrderbook(t *testing.T) {
t.Parallel()
_, err := i.GetOrderbook("XBTSGD")
_, err := i.GetOrderbook("XBTUSD")
if err != nil {
t.Error("GetOrderbook() error", err)
}

26
exchanges/itbit/itbit_wrapper.go
View File

@@ -186,9 +186,10 @@ func (i *ItBit) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderboo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (i *ItBit) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: i.Name, Pair: p, AssetType: assetType, NotAggregated: true}
fpair, err := i.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := i.GetOrderbook(fpair.String())
@@ -196,18 +197,17 @@ func (i *ItBit) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
return nil, err
}
orderBook := new(orderbook.Base)
for x := range orderbookNew.Bids {
var price, amount float64
price, err = strconv.ParseFloat(orderbookNew.Bids[x][0], 64)
if err != nil {
return orderBook, err
return book, err
}
amount, err = strconv.ParseFloat(orderbookNew.Bids[x][1], 64)
if err != nil {
return orderBook, err
return book, err
}
orderBook.Bids = append(orderBook.Bids,
book.Bids = append(book.Bids,
orderbook.Item{
Amount: amount,
Price: price,
@@ -218,28 +218,22 @@ func (i *ItBit) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
var price, amount float64
price, err = strconv.ParseFloat(orderbookNew.Asks[x][0], 64)
if err != nil {
return orderBook, err
return book, err
}
amount, err = strconv.ParseFloat(orderbookNew.Asks[x][1], 64)
if err != nil {
return orderBook, err
return book, err
}
orderBook.Asks = append(orderBook.Asks,
book.Asks = append(book.Asks,
orderbook.Item{
Amount: amount,
Price: price,
})
}
orderBook.Pair = p
orderBook.ExchangeName = i.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(i.Name, p, assetType)
}

2
exchanges/kraken/kraken.go
View File

@@ -57,7 +57,7 @@ const (
krakenRequestRate = 1
// Status consts
StatusOpen = "open"
statusOpen = "open"
)
var (

129
exchanges/kraken/kraken_test.go
View File

@@ -18,6 +18,7 @@ import (
"github.com/thrasher-corp/gocryptotrader/exchanges/asset"
"github.com/thrasher-corp/gocryptotrader/exchanges/kline"
"github.com/thrasher-corp/gocryptotrader/exchanges/order"
"github.com/thrasher-corp/gocryptotrader/exchanges/orderbook"
"github.com/thrasher-corp/gocryptotrader/exchanges/sharedtestvalues"
"github.com/thrasher-corp/gocryptotrader/exchanges/stream"
"github.com/thrasher-corp/gocryptotrader/portfolio/withdraw"
@@ -1084,7 +1085,7 @@ func TestWsOrdrbook(t *testing.T) {
"channelID": 13333337,
"channelName": "book",
"event": "subscriptionStatus",
"pair": "XBT/EUR",
"pair": "XBT/USD",
"status": "subscribed",
"subscription": {
"name": "book"
@@ -1112,7 +1113,42 @@ func TestWsOrdrbook(t *testing.T) {
"5542.70000",
"0.64700000",
"1534614244.654432"
]
],
[
"5544.30000",
"2.50700000",
"1534614248.123678"
],
[
"5545.80000",
"0.33000000",
"1534614098.345543"
],
[
"5546.70000",
"0.64700000",
"1534614244.654432"
],
[
"5547.70000",
"0.64700000",
"1534614244.654432"
],
[
"5548.30000",
"2.50700000",
"1534614248.123678"
],
[
"5549.80000",
"0.33000000",
"1534614098.345543"
],
[
"5550.70000",
"0.64700000",
"1534614244.654432"
]
],
"bs": [
[
@@ -1129,7 +1165,42 @@ func TestWsOrdrbook(t *testing.T) {
"5539.50000",
"5.00000000",
"1534613831.243486"
]
],
[
"5538.20000",
"1.52900000",
"1534614248.765567"
],
[
"5537.90000",
"0.30000000",
"1534614241.769870"
],
[
"5536.50000",
"5.00000000",
"1534613831.243486"
],
[
"5535.20000",
"1.52900000",
"1534614248.765567"
],
[
"5534.90000",
"0.30000000",
"1534614241.769870"
],
[
"5533.50000",
"5.00000000",
"1534613831.243486"
],
[
"5532.50000",
"5.00000000",
"1534613831.243486"
]
]
},
"book-100",
@@ -1153,7 +1224,8 @@ func TestWsOrdrbook(t *testing.T) {
"0.40100000",
"1534614248.456738"
]
]
],
"c": "4187525586"
},
"book-10",
"XBT/USD"
@@ -1171,7 +1243,8 @@ func TestWsOrdrbook(t *testing.T) {
"0.00000000",
"1534614335.345903"
]
]
],
"c": "4187525586"
},
"book-10",
"XBT/USD"
@@ -1529,3 +1602,49 @@ func TestGetHistoricTrades(t *testing.T) {
t.Error(err)
}
}
var testOb = orderbook.Base{
Asks: []orderbook.Item{
{Price: 0.05005, Amount: 0.00000500},
{Price: 0.05010, Amount: 0.00000500},
{Price: 0.05015, Amount: 0.00000500},
{Price: 0.05020, Amount: 0.00000500},
{Price: 0.05025, Amount: 0.00000500},
{Price: 0.05030, Amount: 0.00000500},
{Price: 0.05035, Amount: 0.00000500},
{Price: 0.05040, Amount: 0.00000500},
{Price: 0.05045, Amount: 0.00000500},
{Price: 0.05050, Amount: 0.00000500},
},
Bids: []orderbook.Item{
{Price: 0.05000, Amount: 0.00000500},
{Price: 0.04995, Amount: 0.00000500},
{Price: 0.04990, Amount: 0.00000500},
{Price: 0.04980, Amount: 0.00000500},
{Price: 0.04975, Amount: 0.00000500},
{Price: 0.04970, Amount: 0.00000500},
{Price: 0.04965, Amount: 0.00000500},
{Price: 0.04960, Amount: 0.00000500},
{Price: 0.04955, Amount: 0.00000500},
{Price: 0.04950, Amount: 0.00000500},
},
}
const krakenAPIDocChecksum = 974947235
func TestChecksumCalculation(t *testing.T) {
expected := "5005"
if v := trim("0.05005"); v != expected {
t.Fatalf("expected %s but received %s", expected, v)
}
expected = "500"
if v := trim("0.00000500"); v != expected {
t.Fatalf("expected %s but received %s", expected, v)
}
err := validateCRC32(&testOb, krakenAPIDocChecksum, 5, 8)
if err != nil {
t.Fatal(err)
}
}

4
exchanges/kraken/kraken_types.go
View File

@@ -465,7 +465,7 @@ type WebsocketErrorResponse struct {
type WebsocketChannelData struct {
Subscription string
Pair currency.Pair
ChannelID int64
ChannelID *int64
}
// WsTokenResponse holds the WS auth token
@@ -485,7 +485,7 @@ type wsSystemStatus struct {
}
type wsSubscription struct {
ChannelID int64 `json:"channelID"`
ChannelID *int64 `json:"channelID"`
ChannelName string `json:"channelName"`
ErrorMessage string `json:"errorMessage"`
Event string `json:"event"`

310
exchanges/kraken/kraken_websocket.go
View File

@@ -4,6 +4,7 @@ import (
"encoding/json"
"errors"
"fmt"
"hash/crc32"
"net/http"
"strconv"
"strings"
@@ -52,6 +53,7 @@ const (
krakenWsCancelAllOrderStatus = "cancelAllStatus"
krakenWsRateLimit = 50
krakenWsPingDelay = time.Second * 27
krakenWsOrderbookDepth = 1000
)
// orderbookMutex Ensures if two entries arrive at once, only one can be
@@ -62,7 +64,8 @@ var pingRequest = WebsocketBaseEventRequest{Event: stream.Ping}
// Channels require a topic and a currency
// Format [[ticker,but-t4u],[orderbook,nce-btt]]
var defaultSubscribedChannels = []string{krakenWsTicker,
var defaultSubscribedChannels = []string{
krakenWsTicker,
krakenWsTrade,
krakenWsOrderbook,
krakenWsOHLC,
@@ -346,9 +349,7 @@ func (k *Kraken) wsHandleData(respRaw []byte) error {
}
k.addNewSubscriptionChannelData(&sub)
if sub.RequestID > 0 {
if k.Websocket.Match.IncomingWithData(sub.RequestID, respRaw) {
return nil
}
k.Websocket.Match.IncomingWithData(sub.RequestID, respRaw)
}
default:
k.Websocket.DataHandler <- stream.UnhandledMessageWarning{
@@ -379,7 +380,10 @@ func (k *Kraken) wsPingHandler() error {
func (k *Kraken) wsReadDataResponse(response WebsocketDataResponse) error {
if cID, ok := response[0].(float64); ok {
channelID := int64(cID)
channelData := getSubscriptionChannelData(channelID)
channelData, err := getSubscriptionChannelData(channelID)
if err != nil {
return err
}
switch channelData.Subscription {
case krakenWsTicker:
t, ok := response[1].(map[string]interface{})
@@ -398,6 +402,21 @@ func (k *Kraken) wsReadDataResponse(response WebsocketDataResponse) error {
if !ok {
return errors.New("received invalid orderbook data")
}
if len(response) == 5 {
ob2, okob2 := response[2].(map[string]interface{})
if !okob2 {
return errors.New("received invalid orderbook data")
}
// Squish both maps together to process
for k, v := range ob2 {
if _, ok := ob[k]; ok {
return errors.New("cannot merge maps, conflict is present")
}
ob[k] = v
}
}
return k.wsProcessOrderBook(&channelData, ob)
case krakenWsSpread:
s, ok := response[1].([]interface{})
@@ -412,8 +431,9 @@ func (k *Kraken) wsReadDataResponse(response WebsocketDataResponse) error {
}
return k.wsProcessTrades(&channelData, t)
default:
return fmt.Errorf("%s received unidentified data: %+v",
return fmt.Errorf("%s received unidentified data for subscription %s: %+v",
k.Name,
channelData.Subscription,
response)
}
}
@@ -597,13 +617,17 @@ func (k *Kraken) addNewSubscriptionChannelData(response *wsSubscription) {
}
// getSubscriptionChannelData retrieves WebsocketChannelData based on response ID
func getSubscriptionChannelData(id int64) WebsocketChannelData {
func getSubscriptionChannelData(id int64) (WebsocketChannelData, error) {
for i := range subscriptionChannelPair {
if id == subscriptionChannelPair[i].ChannelID {
return subscriptionChannelPair[i]
if subscriptionChannelPair[i].ChannelID == nil {
continue
}
if id == *subscriptionChannelPair[i].ChannelID {
return subscriptionChannelPair[i], nil
}
}
return WebsocketChannelData{}
return WebsocketChannelData{},
fmt.Errorf("could not get subscription data for id %d", id)
}
// wsProcessTickers converts ticker data and sends it to the datahandler
@@ -733,17 +757,30 @@ func (k *Kraken) wsProcessOrderBook(channelData *WebsocketChannelData, data map[
} else {
askData, asksExist := data["a"].([]interface{})
bidData, bidsExist := data["b"].([]interface{})
checksum, ok := data["c"].(string)
if !ok {
return fmt.Errorf("could not process orderbook update checksum not found")
}
if asksExist || bidsExist {
k.wsRequestMtx.Lock()
defer k.wsRequestMtx.Unlock()
err := k.wsProcessOrderBookUpdate(channelData, askData, bidData)
err := k.wsProcessOrderBookUpdate(channelData, askData, bidData, checksum)
if err != nil {
subscriptionToRemove := &stream.ChannelSubscription{
go func(resub *stream.ChannelSubscription) {
// This was locking the main websocket reader routine and a
// backlog occurred. So put this into it's own go routine.
errResub := k.Websocket.ResubscribeToChannel(resub)
if errResub != nil {
log.Errorf(log.WebsocketMgr,
"resubscription failure for %v: %v",
resub,
errResub)
}
}(&stream.ChannelSubscription{
Channel: krakenWsOrderbook,
Currency: channelData.Pair,
Asset: asset.Spot,
}
k.Websocket.ResubscribeToChannel(subscriptionToRemove)
})
return err
}
}
@@ -814,22 +851,39 @@ func (k *Kraken) wsProcessOrderBookPartial(channelData *WebsocketChannelData, as
}
// wsProcessOrderBookUpdate updates an orderbook entry for a given currency pair
func (k *Kraken) wsProcessOrderBookUpdate(channelData *WebsocketChannelData, askData, bidData []interface{}) error {
func (k *Kraken) wsProcessOrderBookUpdate(channelData *WebsocketChannelData, askData, bidData []interface{}, checksum string) error {
update := buffer.Update{
Asset: asset.Spot,
Pair: channelData.Pair,
Asset: asset.Spot,
Pair: channelData.Pair,
MaxDepth: krakenWsOrderbookDepth,
}
// Calculating checksum requires incoming decimal place checks for both
// price and amount as there is no set standard between currency pairs. This
// is calculated per update as opposed to snapshot because changes to
// decimal amounts could occur at any time.
var priceDP, amtDP int
var highestLastUpdate time.Time
// Ask data is not always sent
for i := range askData {
asks := askData[i].([]interface{})
price, err := strconv.ParseFloat(asks[0].(string), 64)
priceStr, ok := asks[0].(string)
if !ok {
return errors.New("price type assertion failure")
}
price, err := strconv.ParseFloat(priceStr, 64)
if err != nil {
return err
}
amount, err := strconv.ParseFloat(asks[1].(string), 64)
amountStr, ok := asks[1].(string)
if !ok {
return errors.New("amount type assertion failure")
}
amount, err := strconv.ParseFloat(amountStr, 64)
if err != nil {
return err
}
@@ -838,7 +892,13 @@ func (k *Kraken) wsProcessOrderBookUpdate(channelData *WebsocketChannelData, ask
Amount: amount,
Price: price,
})
timeData, err := strconv.ParseFloat(asks[2].(string), 64)
timeStr, ok := asks[2].(string)
if !ok {
return errors.New("time type assertion failure")
}
timeData, err := strconv.ParseFloat(timeStr, 64)
if err != nil {
return err
}
@@ -847,17 +907,43 @@ func (k *Kraken) wsProcessOrderBookUpdate(channelData *WebsocketChannelData, ask
if highestLastUpdate.Before(askUpdatedTime) {
highestLastUpdate = askUpdatedTime
}
if i == len(askData)-1 {
pSplit := strings.Split(priceStr, ".")
if len(pSplit) != 2 {
return errors.New("incorrect decimal data returned for price")
}
priceDP = len(pSplit[1])
aSplit := strings.Split(amountStr, ".")
if len(aSplit) != 2 {
return errors.New("incorrect decimal data returned for amount")
}
amtDP = len(aSplit[1])
}
}
// Bid data is not always sent
for i := range bidData {
bids := bidData[i].([]interface{})
price, err := strconv.ParseFloat(bids[0].(string), 64)
priceStr, ok := bids[0].(string)
if !ok {
return errors.New("price type assertion failure")
}
price, err := strconv.ParseFloat(priceStr, 64)
if err != nil {
return err
}
amount, err := strconv.ParseFloat(bids[1].(string), 64)
amountStr, ok := bids[1].(string)
if !ok {
return errors.New("amount type assertion failure")
}
amount, err := strconv.ParseFloat(amountStr, 64)
if err != nil {
return err
}
@@ -866,7 +952,13 @@ func (k *Kraken) wsProcessOrderBookUpdate(channelData *WebsocketChannelData, ask
Amount: amount,
Price: price,
})
timeData, err := strconv.ParseFloat(bids[2].(string), 64)
timeStr, ok := bids[2].(string)
if !ok {
return errors.New("time type assertion failure")
}
timeData, err := strconv.ParseFloat(timeStr, 64)
if err != nil {
return err
}
@@ -875,9 +967,85 @@ func (k *Kraken) wsProcessOrderBookUpdate(channelData *WebsocketChannelData, ask
if highestLastUpdate.Before(bidUpdatedTime) {
highestLastUpdate = bidUpdatedTime
}
if i == len(bidData)-1 {
pSplit := strings.Split(priceStr, ".")
if len(pSplit) != 2 {
return errors.New("incorrect decimal data returned for price")
}
priceDP = len(pSplit[1])
aSplit := strings.Split(amountStr, ".")
if len(aSplit) != 2 {
return errors.New("incorrect decimal data returned for amount")
}
amtDP = len(aSplit[1])
}
}
update.UpdateTime = highestLastUpdate
return k.Websocket.Orderbook.Update(&update)
err := k.Websocket.Orderbook.Update(&update)
if err != nil {
return err
}
book := k.Websocket.Orderbook.GetOrderbook(channelData.Pair, asset.Spot)
if book == nil {
return fmt.Errorf("cannot calculate websocket checksum: book not found for %s %s",
channelData.Pair,
asset.Spot)
}
token, err := strconv.ParseInt(checksum, 10, 64)
if err != nil {
return err
}
return validateCRC32(book, uint32(token), priceDP, amtDP)
}
func validateCRC32(b *orderbook.Base, token uint32, decPrice, decAmount int) error {
if len(b.Asks) < 10 || len(b.Bids) < 10 {
return fmt.Errorf("%s %s insufficient bid and asks to calculate checksum",
b.Pair,
b.AssetType)
}
if decPrice == 0 || decAmount == 0 {
return fmt.Errorf("%s %s trailing decimal count not calculated", b.Pair,
b.AssetType)
}
var checkStr strings.Builder
for i := 0; i < 10; i++ {
priceStr := trim(strconv.FormatFloat(b.Asks[i].Price, 'f', decPrice, 64))
checkStr.WriteString(priceStr)
amountStr := trim(strconv.FormatFloat(b.Asks[i].Amount, 'f', decAmount, 64))
checkStr.WriteString(amountStr)
}
for i := 0; i < 10; i++ {
priceStr := trim(strconv.FormatFloat(b.Bids[i].Price, 'f', decPrice, 64))
checkStr.WriteString(priceStr)
amountStr := trim(strconv.FormatFloat(b.Bids[i].Amount, 'f', decAmount, 64))
checkStr.WriteString(amountStr)
}
if check := crc32.ChecksumIEEE([]byte(checkStr.String())); check != token {
return fmt.Errorf("%s %s invalid checksum %d, expected %d",
b.Pair,
b.AssetType,
check,
token)
}
return nil
}
// trim removes '.' and prefixed '0' from subsequent string
func trim(s string) string {
s = strings.Replace(s, ".", "", 1)
s = strings.TrimLeft(s, "0")
return s
}
// wsProcessCandles converts candle data and sends it to the data handler
@@ -970,65 +1138,74 @@ func (k *Kraken) GenerateAuthenticatedSubscriptions() ([]stream.ChannelSubscript
// Subscribe sends a websocket message to receive data from the channel
func (k *Kraken) Subscribe(channelsToSubscribe []stream.ChannelSubscription) error {
var subs []WebsocketSubscriptionEventRequest
var subscriptions = make(map[string]*[]WebsocketSubscriptionEventRequest)
channels:
for x := range channelsToSubscribe {
for y := range subs {
if subs[y].Subscription.Name == channelsToSubscribe[x].Channel {
subs[y].Pairs = append(subs[y].Pairs,
channelsToSubscribe[x].Currency.String())
subs[y].Channels = append(subs[y].Channels, channelsToSubscribe[x])
continue channels
for i := range channelsToSubscribe {
s, ok := subscriptions[channelsToSubscribe[i].Channel]
if !ok {
s = &[]WebsocketSubscriptionEventRequest{}
subscriptions[channelsToSubscribe[i].Channel] = s
}
for j := range *s {
if len((*s)[j].Channels) >= 20 {
// Batch outgoing subscriptions as there are limitations on the
// orderbook snapshots
continue
}
(*s)[j].Pairs = append((*s)[j].Pairs, channelsToSubscribe[i].Currency.String())
(*s)[j].Channels = append((*s)[j].Channels, channelsToSubscribe[i])
continue channels
}
var id int64
if common.StringDataContains(authenticatedChannels, channelsToSubscribe[x].Channel) {
id = k.Websocket.AuthConn.GenerateMessageID(false)
} else {
id = k.Websocket.Conn.GenerateMessageID(false)
}
resp := WebsocketSubscriptionEventRequest{
Event: krakenWsSubscribe,
Subscription: WebsocketSubscriptionData{
Name: channelsToSubscribe[x].Channel,
},
id := k.Websocket.Conn.GenerateMessageID(false)
outbound := WebsocketSubscriptionEventRequest{
Event: krakenWsSubscribe,
RequestID: id,
Subscription: WebsocketSubscriptionData{
Name: channelsToSubscribe[i].Channel,
},
}
if channelsToSubscribe[x].Channel == "book" {
// TODO: Add ability to make depth customisable
resp.Subscription.Depth = 1000
if channelsToSubscribe[i].Channel == "book" {
outbound.Subscription.Depth = 1000
}
if !channelsToSubscribe[x].Currency.IsEmpty() {
resp.Pairs = []string{channelsToSubscribe[x].Currency.String()}
if !channelsToSubscribe[i].Currency.IsEmpty() {
outbound.Pairs = []string{channelsToSubscribe[i].Currency.String()}
}
if channelsToSubscribe[x].Params != nil {
resp.Subscription.Token = authToken
if channelsToSubscribe[i].Params != nil {
outbound.Subscription.Token = authToken
}
resp.Channels = append(resp.Channels, channelsToSubscribe[x])
subs = append(subs, resp)
outbound.Channels = append(outbound.Channels, channelsToSubscribe[i])
*s = append(*s, outbound)
}
var errs common.Errors
for i := range subs {
if common.StringDataContains(authenticatedChannels, subs[i].Subscription.Name) {
_, err := k.Websocket.AuthConn.SendMessageReturnResponse(subs[i].RequestID, subs[i])
for subType, subs := range subscriptions {
for i := range *subs {
if common.StringDataContains(authenticatedChannels, (*subs)[i].Subscription.Name) {
_, err := k.Websocket.AuthConn.SendMessageReturnResponse((*subs)[i].RequestID, (*subs)[i])
if err != nil {
errs = append(errs, err)
continue
}
k.Websocket.AddSuccessfulSubscriptions((*subs)[i].Channels...)
continue
}
if subType == "book" {
// There is an undocumented subscription limit that is present
// on websocket orderbooks, to subscribe to the channel while
// actually receiving the snapshots a rudimentary sleep is
// imposed and requests are batched in allotments of 20 items.
time.Sleep(time.Second)
}
_, err := k.Websocket.Conn.SendMessageReturnResponse((*subs)[i].RequestID, (*subs)[i])
if err != nil {
errs = append(errs, err)
continue
}
k.Websocket.AddSuccessfulSubscriptions(subs[i].Channels...)
continue
k.Websocket.AddSuccessfulSubscriptions((*subs)[i].Channels...)
}
_, err := k.Websocket.Conn.SendMessageReturnResponse(subs[i].RequestID, subs[i])
if err != nil {
errs = append(errs, err)
continue
}
k.Websocket.AddSuccessfulSubscriptions(subs[i].Channels...)
}
if errs != nil {
return errs
@@ -1052,8 +1229,7 @@ channels:
}
var depth int64
if channelsToUnsubscribe[x].Channel == "book" {
// TODO: Add ability to make depth customisable
depth = 1000
depth = krakenWsOrderbookDepth
}
var id int64

24
exchanges/kraken/kraken_wrapper.go
View File

@@ -187,7 +187,7 @@ func (k *Kraken) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: k.GenerateDefaultSubscriptions,
Features: &k.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: true,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
})
if err != nil {
@@ -425,40 +425,34 @@ func (k *Kraken) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (k *Kraken) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: k.Name, Pair: p, AssetType: assetType}
fpair, err := k.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := k.GetDepth(fpair.String())
if err != nil {
return nil, err
return book, err
}
var orderBook = new(orderbook.Base)
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Amount,
Price: orderbookNew.Bids[x].Price,
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x].Amount,
Price: orderbookNew.Asks[x].Price,
})
}
orderBook.Pair = p
orderBook.ExchangeName = k.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(k.Name, p, assetType)
}
@@ -757,7 +751,7 @@ func (k *Kraken) GetOrderInfo(orderID string, pair currency.Pair, assetType asse
}
price := orderInfo.Price
if orderInfo.Status == StatusOpen {
if orderInfo.Status == statusOpen {
price = orderInfo.Description.Price
}

25
exchanges/lakebtc/lakebtc_wrapper.go
View File

@@ -138,6 +138,7 @@ func (l *LakeBTC) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: l.GenerateDefaultSubscriptions,
Features: &l.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
})
}
@@ -268,34 +269,32 @@ func (l *LakeBTC) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderb
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (l *LakeBTC) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
book := &orderbook.Base{ExchangeName: l.Name, Pair: p, AssetType: assetType}
fPair, err := l.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderBook := new(orderbook.Base)
orderbookNew, err := l.GetOrderBook(fPair.String())
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{Amount: orderbookNew.Bids[x].Amount, Price: orderbookNew.Bids[x].Price})
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Amount,
Price: orderbookNew.Bids[x].Price})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{Amount: orderbookNew.Asks[x].Amount, Price: orderbookNew.Asks[x].Price})
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x].Amount,
Price: orderbookNew.Asks[x].Price})
}
orderBook.Pair = fPair
orderBook.ExchangeName = l.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(l.Name, fPair, assetType)
}

26
exchanges/lbank/lbank_wrapper.go
View File

@@ -243,26 +243,26 @@ func (l *Lbank) FetchOrderbook(currency currency.Pair, assetType asset.Item) (*o
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (l *Lbank) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: l.Name, Pair: p, AssetType: assetType}
fpair, err := l.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
a, err := l.GetMarketDepths(fpair.String(), "60", "1")
if err != nil {
return orderBook, err
return book, err
}
for i := range a.Asks {
price, convErr := strconv.ParseFloat(a.Asks[i][0], 64)
if convErr != nil {
return orderBook, convErr
return book, convErr
}
amount, convErr := strconv.ParseFloat(a.Asks[i][1], 64)
if convErr != nil {
return orderBook, convErr
return book, convErr
}
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Price: price,
Amount: amount,
})
@@ -270,25 +270,21 @@ func (l *Lbank) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
for i := range a.Bids {
price, convErr := strconv.ParseFloat(a.Bids[i][0], 64)
if convErr != nil {
return orderBook, convErr
return book, convErr
}
amount, convErr := strconv.ParseFloat(a.Bids[i][1], 64)
if convErr != nil {
return orderBook, convErr
return book, convErr
}
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Price: price,
Amount: amount,
})
}
orderBook.Pair = p
orderBook.ExchangeName = l.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(l.Name, p, assetType)
}

35
exchanges/localbitcoins/localbitcoins.go
View File

@@ -128,7 +128,7 @@ func (l *LocalBitcoins) GetAccountInformation(username string, self bool) (Accou
}
} else {
path := fmt.Sprintf("%s/%s/%s/", l.API.Endpoints.URL, localbitcoinsAPIAccountInfo, username)
err := l.SendHTTPRequest(path, &resp)
err := l.SendHTTPRequest(path, &resp, request.Unset)
if err != nil {
return resp.Data, err
}
@@ -335,14 +335,14 @@ func (l *LocalBitcoins) GetTradeInfo(contactID string) (dbi DashBoardInfo, err e
// GetCountryCodes returns a list of valid and recognized countrycodes
func (l *LocalBitcoins) GetCountryCodes() error {
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPICountryCodes, nil)
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPICountryCodes, nil, request.Unset)
}
// GetCurrencies returns a list of valid and recognized fiat currencies. Also
// contains human readable name for every currency and boolean that tells if
// currency is an altcoin.
func (l *LocalBitcoins) GetCurrencies() error {
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPICurrencies, nil)
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPICurrencies, nil, request.Unset)
}
// GetDashboardInfo returns a list of trades on the data key contact_list. This
@@ -470,13 +470,13 @@ func (l *LocalBitcoins) MarkNotifications() error {
// and code for payment methods, and possible limitations in currencies and bank
// name choices.
func (l *LocalBitcoins) GetPaymentMethods() error {
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPIPaymentMethods, nil)
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPIPaymentMethods, nil, request.Unset)
}
// GetPaymentMethodsByCountry returns a list of valid payment methods filtered
// by countrycodes.
func (l *LocalBitcoins) GetPaymentMethodsByCountry(countryCode string) error {
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPIPaymentMethods+countryCode, nil)
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPIPaymentMethods+countryCode, nil, request.Unset)
}
// CheckPincode checks the given PIN code against the token owners currently
@@ -511,7 +511,7 @@ func (l *LocalBitcoins) CheckPincode(pin int) (bool, error) {
// sell listings for each.
// TODO
func (l *LocalBitcoins) GetPlaces() error {
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPIPlaces, nil)
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPIPlaces, nil, request.Unset)
}
// VerifyUsername returns list of real name verifiers for the user. Returns a
@@ -639,20 +639,22 @@ func (l *LocalBitcoins) GetWalletAddress() (string, error) {
// GetBitcoinsWithCashAd returns buy or sell as cash local advertisements.
// TODO
func (l *LocalBitcoins) GetBitcoinsWithCashAd() error {
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPICashBuy, nil)
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPICashBuy, nil, request.Unset)
}
// GetBitcoinsOnlineAd this API returns buy or sell Bitcoin online ads.
// TODO
func (l *LocalBitcoins) GetBitcoinsOnlineAd() error {
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPIOnlineBuy, nil)
return l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPIOnlineBuy, nil, request.Unset)
}
// GetTicker returns list of all completed trades.
func (l *LocalBitcoins) GetTicker() (map[string]Ticker, error) {
result := make(map[string]Ticker)
return result, l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPITicker, &result)
return result,
l.SendHTTPRequest(l.API.Endpoints.URL+localbitcoinsAPITicker,
&result,
tickerLimiter)
}
// GetTradableCurrencies returns a list of tradable fiat currencies
@@ -673,9 +675,11 @@ func (l *LocalBitcoins) GetTradableCurrencies() ([]string, error) {
// GetTrades returns all closed trades in online buy and online sell categories,
// updated every 15 minutes.
func (l *LocalBitcoins) GetTrades(currency string, values url.Values) ([]Trade, error) {
path := common.EncodeURLValues(fmt.Sprintf("%s%s/trades.json", l.API.Endpoints.URL+localbitcoinsAPIBitcoincharts, currency), values)
path := common.EncodeURLValues(fmt.Sprintf("%s%s/trades.json",
l.API.Endpoints.URL+localbitcoinsAPIBitcoincharts, currency),
values)
var result []Trade
return result, l.SendHTTPRequest(path, &result)
return result, l.SendHTTPRequest(path, &result, request.Unset)
}
// GetOrderbook returns buy and sell bitcoin online advertisements. Amount is
@@ -688,9 +692,9 @@ func (l *LocalBitcoins) GetOrderbook(currency string) (Orderbook, error) {
Asks [][]string `json:"asks"`
}
path := fmt.Sprintf("%s/%s/orderbook.json", l.API.Endpoints.URL+localbitcoinsAPIBitcoincharts, currency)
path := l.API.Endpoints.URL + localbitcoinsAPIBitcoincharts + currency + "/orderbook.json"
resp := response{}
err := l.SendHTTPRequest(path, &resp)
err := l.SendHTTPRequest(path, &resp, orderBookLimiter)
if err != nil {
return Orderbook{}, err
@@ -730,7 +734,7 @@ func (l *LocalBitcoins) GetOrderbook(currency string) (Orderbook, error) {
}
// SendHTTPRequest sends an unauthenticated HTTP request
func (l *LocalBitcoins) SendHTTPRequest(path string, result interface{}) error {
func (l *LocalBitcoins) SendHTTPRequest(path string, result interface{}, ep request.EndpointLimit) error {
return l.SendPayload(context.Background(), &request.Item{
Method: http.MethodGet,
Path: path,
@@ -738,6 +742,7 @@ func (l *LocalBitcoins) SendHTTPRequest(path string, result interface{}) error {
Verbose: l.Verbose,
HTTPDebugging: l.HTTPDebugging,
HTTPRecording: l.HTTPRecording,
Endpoint: ep,
})
}

20
exchanges/localbitcoins/localbitcoins_wrapper.go
View File

@@ -92,7 +92,8 @@ func (l *LocalBitcoins) SetDefaults() {
}
l.Requester = request.New(l.Name,
common.NewHTTPClientWithTimeout(exchange.DefaultHTTPTimeout))
common.NewHTTPClientWithTimeout(exchange.DefaultHTTPTimeout),
request.WithLimiter(SetRateLimit()))
l.API.Endpoints.URLDefault = localbitcoinsAPIURL
l.API.Endpoints.URL = l.API.Endpoints.URLDefault
@@ -213,33 +214,30 @@ func (l *LocalBitcoins) FetchOrderbook(p currency.Pair, assetType asset.Item) (*
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (l *LocalBitcoins) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: l.Name, Pair: p, AssetType: assetType}
orderbookNew, err := l.GetOrderbook(p.Quote.String())
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x].Amount / orderbookNew.Bids[x].Price,
Price: orderbookNew.Bids[x].Price,
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x].Amount / orderbookNew.Asks[x].Price,
Price: orderbookNew.Asks[x].Price,
})
}
orderBook.Pair = p
orderBook.ExchangeName = l.Name
orderBook.AssetType = assetType
err = orderBook.Process()
book.NotAggregated = true
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(l.Name, p, assetType)

37
exchanges/localbitcoins/rate_limit.go Normal file
View File

@@ -0,0 +1,37 @@
package localbitcoins
import (
"time"
"github.com/thrasher-corp/gocryptotrader/exchanges/request"
"golang.org/x/time/rate"
)
const orderBookLimiter request.EndpointLimit = 1
const tickerLimiter request.EndpointLimit = 2
// RateLimit define s custom rate limiter scoped for orderbook requests
type RateLimit struct {
Orderbook *rate.Limiter
Ticker *rate.Limiter
}
// Limit executes rate limiting functionality for Binance
func (r *RateLimit) Limit(f request.EndpointLimit) error {
if f == orderBookLimiter {
time.Sleep(r.Orderbook.Reserve().Delay())
} else if f == tickerLimiter {
time.Sleep(r.Ticker.Reserve().Delay())
}
return nil
}
// SetRateLimit returns the rate limit for the exchange
func SetRateLimit() *RateLimit {
return &RateLimit{
// 4 seconds per book fetching is the best time frame to actually
// receive without retying. There is undocumentated rate limit.
Orderbook: request.NewRateLimit(4*time.Second, 1),
Ticker: request.NewRateLimit(time.Second, 1),
}
}

40
exchanges/okgroup/okgroup_wrapper.go
View File

@@ -50,6 +50,7 @@ func (o *OKGroup) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: o.GenerateDefaultSubscriptions,
Features: &o.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
})
if err != nil {
return err
@@ -77,9 +78,14 @@ func (o *OKGroup) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderb
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (o *OKGroup) UpdateOrderbook(p currency.Pair, a asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{
ExchangeName: o.Name,
Pair: p,
AssetType: a,
}
if a == asset.Index {
return orderBook, errors.New("no orderbooks for index")
return book, errors.New("no orderbooks for index")
}
fPair, err := o.FormatExchangeCurrency(p, a)
@@ -91,17 +97,17 @@ func (o *OKGroup) UpdateOrderbook(p currency.Pair, a asset.Item) (*orderbook.Bas
InstrumentID: fPair.String(),
}, a)
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Bids {
amount, convErr := strconv.ParseFloat(orderbookNew.Bids[x][1], 64)
if convErr != nil {
return orderBook, err
return book, err
}
price, convErr := strconv.ParseFloat(orderbookNew.Bids[x][0], 64)
if convErr != nil {
return orderBook, err
return book, err
}
var liquidationOrders, orderCount int64
@@ -109,16 +115,16 @@ func (o *OKGroup) UpdateOrderbook(p currency.Pair, a asset.Item) (*orderbook.Bas
if len(orderbookNew.Bids[x]) == 4 {
liquidationOrders, convErr = strconv.ParseInt(orderbookNew.Bids[x][2], 10, 64)
if convErr != nil {
return orderBook, err
return book, err
}
orderCount, convErr = strconv.ParseInt(orderbookNew.Bids[x][3], 10, 64)
if convErr != nil {
return orderBook, err
return book, err
}
}
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: amount,
Price: price,
LiquidationOrders: liquidationOrders,
@@ -129,11 +135,11 @@ func (o *OKGroup) UpdateOrderbook(p currency.Pair, a asset.Item) (*orderbook.Bas
for x := range orderbookNew.Asks {
amount, convErr := strconv.ParseFloat(orderbookNew.Asks[x][1], 64)
if convErr != nil {
return orderBook, err
return book, err
}
price, convErr := strconv.ParseFloat(orderbookNew.Asks[x][0], 64)
if convErr != nil {
return orderBook, err
return book, err
}
var liquidationOrders, orderCount int64
@@ -141,16 +147,16 @@ func (o *OKGroup) UpdateOrderbook(p currency.Pair, a asset.Item) (*orderbook.Bas
if len(orderbookNew.Asks[x]) == 4 {
liquidationOrders, convErr = strconv.ParseInt(orderbookNew.Asks[x][2], 10, 64)
if convErr != nil {
return orderBook, err
return book, err
}
orderCount, convErr = strconv.ParseInt(orderbookNew.Asks[x][3], 10, 64)
if convErr != nil {
return orderBook, err
return book, err
}
}
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: amount,
Price: price,
LiquidationOrders: liquidationOrders,
@@ -158,13 +164,9 @@ func (o *OKGroup) UpdateOrderbook(p currency.Pair, a asset.Item) (*orderbook.Bas
})
}
orderBook.Pair = p
orderBook.AssetType = a
orderBook.ExchangeName = o.Name
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(o.Name, fPair, a)

2
exchanges/orderbook/calculator_test.go
View File

@@ -25,7 +25,7 @@ func TestWhaleBomb(t *testing.T) {
t.Parallel()
b := testSetup()
// invalid price amout
// invalid price amount
_, err := b.WhaleBomb(-1, true)
if err == nil {
t.Error("unexpected result")

278
exchanges/orderbook/orderbook.go
View File

@@ -1,7 +1,6 @@
package orderbook
import (
"errors"
"fmt"
"sort"
"strings"
@@ -11,25 +10,21 @@ import (
"github.com/thrasher-corp/gocryptotrader/currency"
"github.com/thrasher-corp/gocryptotrader/dispatch"
"github.com/thrasher-corp/gocryptotrader/exchanges/asset"
"github.com/thrasher-corp/gocryptotrader/log"
)
// Get checks and returns the orderbook given an exchange name and currency pair
// if it exists
func Get(exchange string, p currency.Pair, a asset.Item) (*Base, error) {
o, err := service.Retrieve(exchange, p, a)
if err != nil {
return nil, err
}
return o, nil
return service.Retrieve(exchange, p, a)
}
// SubscribeOrderbook subcribes to an orderbook and returns a communication
// channel to stream orderbook data updates
func SubscribeOrderbook(exchange string, p currency.Pair, a asset.Item) (dispatch.Pipe, error) {
exchange = strings.ToLower(exchange)
service.RLock()
defer service.RUnlock()
book, ok := service.Books[exchange][p.Base.Item][p.Quote.Item][a]
service.Lock()
defer service.Unlock()
book, ok := service.Books[exchange][a][p.Base.Item][p.Quote.Item]
if !ok {
return dispatch.Pipe{},
fmt.Errorf("orderbook item not found for %s %s %s",
@@ -42,10 +37,9 @@ func SubscribeOrderbook(exchange string, p currency.Pair, a asset.Item) (dispatc
// SubscribeToExchangeOrderbooks subcribes to all orderbooks on an exchange
func SubscribeToExchangeOrderbooks(exchange string) (dispatch.Pipe, error) {
exchange = strings.ToLower(exchange)
service.RLock()
defer service.RUnlock()
id, ok := service.Exchange[exchange]
service.Lock()
defer service.Unlock()
id, ok := service.Exchange[strings.ToLower(exchange)]
if !ok {
return dispatch.Pipe{}, fmt.Errorf("%s exchange orderbooks not found",
exchange)
@@ -57,43 +51,49 @@ func SubscribeToExchangeOrderbooks(exchange string) (dispatch.Pipe, error) {
func (s *Service) Update(b *Base) error {
name := strings.ToLower(b.ExchangeName)
s.Lock()
book, ok := s.Books[name][b.Pair.Base.Item][b.Pair.Quote.Item][b.AssetType]
if ok {
book.b.Bids = b.Bids
book.b.Asks = b.Asks
book.b.LastUpdated = b.LastUpdated
ids := append(book.Assoc, book.Main)
s.Unlock()
return s.mux.Publish(ids, b)
m1, ok := s.Books[name]
if !ok {
m1 = make(map[asset.Item]map[*currency.Item]map[*currency.Item]*Book)
s.Books[name] = m1
}
switch {
case s.Books[name] == nil:
s.Books[name] = make(map[*currency.Item]map[*currency.Item]map[asset.Item]*Book)
fallthrough
case s.Books[name][b.Pair.Base.Item] == nil:
s.Books[name][b.Pair.Base.Item] = make(map[*currency.Item]map[asset.Item]*Book)
fallthrough
case s.Books[name][b.Pair.Base.Item][b.Pair.Quote.Item] == nil:
s.Books[name][b.Pair.Base.Item][b.Pair.Quote.Item] = make(map[asset.Item]*Book)
m2, ok := m1[b.AssetType]
if !ok {
m2 = make(map[*currency.Item]map[*currency.Item]*Book)
m1[b.AssetType] = m2
}
err := s.SetNewData(b, name)
if err != nil {
m3, ok := m2[b.Pair.Base.Item]
if !ok {
m3 = make(map[*currency.Item]*Book)
m2[b.Pair.Base.Item] = m3
}
book, ok := m3[b.Pair.Quote.Item]
if !ok {
book = new(Book)
m3[b.Pair.Quote.Item] = book
err := s.SetNewData(b, book, name)
s.Unlock()
return err
}
book.b.Bids = append(b.Bids[:0:0], b.Bids...) // nolint:gocritic // Short hand to not use make and copy
book.b.Asks = append(b.Asks[:0:0], b.Asks...) // nolint:gocritic // Short hand to not use make and copy
book.b.LastUpdated = b.LastUpdated
ids := append(book.Assoc, book.Main)
s.Unlock()
return nil
return s.mux.Publish(ids, b)
}
// SetNewData sets new data
func (s *Service) SetNewData(b *Base, fmtName string) error {
ids, err := s.GetAssociations(b, fmtName)
func (s *Service) SetNewData(ob *Base, book *Book, exch string) error {
var err error
book.Assoc, err = s.getAssociations(strings.ToLower(exch))
if err != nil {
return err
}
singleID, err := s.mux.GetID()
book.Main, err = s.mux.GetID()
if err != nil {
return err
}
@@ -101,34 +101,24 @@ func (s *Service) SetNewData(b *Base, fmtName string) error {
// Below instigates orderbook item separation so we can ensure, in the event
// of a simultaneous update via websocket/rest/fix, we don't affect package
// scoped orderbook data which could result in a potential panic
cpyBook := *b
cpyBook.Bids = make([]Item, len(b.Bids))
copy(cpyBook.Bids, b.Bids)
cpyBook.Asks = make([]Item, len(b.Asks))
copy(cpyBook.Asks, b.Asks)
s.Books[fmtName][b.Pair.Base.Item][b.Pair.Quote.Item][b.AssetType] = &Book{
b: &cpyBook,
Main: singleID,
Assoc: ids}
cpy := *ob
cpy.Bids = append(cpy.Bids[:0:0], cpy.Bids...)
cpy.Asks = append(cpy.Asks[:0:0], cpy.Asks...)
book.b = &cpy
return nil
}
// GetAssociations links a singular book with it's dispatch associations
func (s *Service) GetAssociations(b *Base, fmtName string) ([]uuid.UUID, error) {
if b == nil {
return nil, errors.New("orderbook is nil")
}
func (s *Service) getAssociations(exch string) ([]uuid.UUID, error) {
var ids []uuid.UUID
exchangeID, ok := s.Exchange[fmtName]
exchangeID, ok := s.Exchange[exch]
if !ok {
var err error
exchangeID, err = s.mux.GetID()
if err != nil {
return nil, err
}
s.Exchange[fmtName] = exchangeID
s.Exchange[exch] = exchangeID
}
ids = append(ids, exchangeID)
@@ -137,45 +127,34 @@ func (s *Service) GetAssociations(b *Base, fmtName string) ([]uuid.UUID, error)
// Retrieve gets orderbook data from the slice
func (s *Service) Retrieve(exchange string, p currency.Pair, a asset.Item) (*Base, error) {
exchange = strings.ToLower(exchange)
s.RLock()
defer s.RUnlock()
if _, ok := s.Books[exchange]; !ok {
s.Lock()
defer s.Unlock()
m1, ok := s.Books[strings.ToLower(exchange)]
if !ok {
return nil, fmt.Errorf("no orderbooks for %s exchange", exchange)
}
if _, ok := s.Books[exchange][p.Base.Item]; !ok {
return nil, fmt.Errorf("no orderbooks associated with base currency %s",
p.Base)
}
if _, ok := s.Books[exchange][p.Base.Item][p.Quote.Item]; !ok {
return nil, fmt.Errorf("no orderbooks associated with quote currency %s",
p.Quote)
}
var liveOrderBook *Book
var ok bool
if liveOrderBook, ok = s.Books[exchange][p.Base.Item][p.Quote.Item][a]; !ok {
m2, ok := m1[a]
if !ok {
return nil, fmt.Errorf("no orderbooks associated with asset type %s",
a)
}
localCopyOfAsks := make([]Item, len(s.Books[exchange][p.Base.Item][p.Quote.Item][a].b.Asks))
localCopyOfBids := make([]Item, len(s.Books[exchange][p.Base.Item][p.Quote.Item][a].b.Bids))
copy(localCopyOfBids, liveOrderBook.b.Bids)
copy(localCopyOfAsks, liveOrderBook.b.Asks)
ob := Base{
Pair: liveOrderBook.b.Pair,
Bids: localCopyOfBids,
Asks: localCopyOfAsks,
LastUpdated: liveOrderBook.b.LastUpdated,
LastUpdateID: liveOrderBook.b.LastUpdateID,
AssetType: liveOrderBook.b.AssetType,
ExchangeName: liveOrderBook.b.ExchangeName,
m3, ok := m2[p.Base.Item]
if !ok {
return nil, fmt.Errorf("no orderbooks associated with base currency %s",
p.Base)
}
book, ok := m3[p.Quote.Item]
if !ok {
return nil, fmt.Errorf("no orderbooks associated with base currency %s",
p.Quote)
}
ob := *book.b
ob.Bids = append(ob.Bids[:0:0], ob.Bids...)
ob.Asks = append(ob.Asks[:0:0], ob.Asks...)
return &ob, nil
}
@@ -206,62 +185,131 @@ func (b *Base) Update(bids, asks []Item) {
b.LastUpdated = time.Now()
}
// Verify ensures that the orderbook items are correctly sorted
// Verify ensures that the orderbook items are correctly sorted prior to being
// set and will reject any book with incorrect values.
// Bids should always go from a high price to a low price and
// asks should always go from a low price to a higher price
func (b *Base) Verify() {
var lastPrice float64
var sortBids, sortAsks bool
for x := range b.Bids {
if lastPrice != 0 && b.Bids[x].Price >= lastPrice {
sortBids = true
break
// Asks should always go from a low price to a higher price
func (b *Base) Verify() error {
// Checking for both ask and bid lengths being zero has been removed and
// a warning has been put in place some exchanges e.g. LakeBTC return zero
// level books. In the event that there is a massive liquidity change where
// a book dries up, this will still update so we do not traverse potential
// incorrect old data.
if len(b.Asks) == 0 || len(b.Bids) == 0 {
log.Warnf(log.OrderBook,
bookLengthIssue,
b.ExchangeName,
b.Pair,
b.AssetType,
len(b.Bids),
len(b.Asks))
}
for i := range b.Bids {
if b.Bids[i].Price == 0 {
return fmt.Errorf(bidLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errPriceNotSet)
}
lastPrice = b.Bids[x].Price
}
lastPrice = 0
for x := range b.Asks {
if lastPrice != 0 && b.Asks[x].Price <= lastPrice {
sortAsks = true
break
if b.Bids[i].Amount <= 0 {
return fmt.Errorf(bidLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errAmountInvalid)
}
if b.IsFundingRate && b.Bids[i].Period == 0 {
return fmt.Errorf(bidLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errPeriodUnset)
}
if i != 0 {
if b.Bids[i].Price > b.Bids[i-1].Price {
return fmt.Errorf(bidLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errOutOfOrder)
}
if !b.NotAggregated && b.Bids[i].Price == b.Bids[i-1].Price {
return fmt.Errorf(bidLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errDuplication)
}
if b.Bids[i].ID != 0 && b.Bids[i].ID == b.Bids[i-1].ID {
return fmt.Errorf(bidLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errIDDuplication)
}
}
lastPrice = b.Asks[x].Price
}
if sortBids {
sort.Sort(sort.Reverse(byOBPrice(b.Bids)))
}
for i := range b.Asks {
if b.Asks[i].Price == 0 {
return fmt.Errorf(askLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errPriceNotSet)
}
if b.Asks[i].Amount <= 0 {
return fmt.Errorf(askLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errAmountInvalid)
}
if b.IsFundingRate && b.Asks[i].Period == 0 {
return fmt.Errorf(askLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errPeriodUnset)
}
if i != 0 {
if b.Asks[i].Price < b.Asks[i-1].Price {
return fmt.Errorf(askLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errOutOfOrder)
}
if sortAsks {
sort.Sort((byOBPrice(b.Asks)))
if !b.NotAggregated && b.Asks[i].Price == b.Asks[i-1].Price {
return fmt.Errorf(askLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errDuplication)
}
if b.Asks[i].ID != 0 && b.Asks[i].ID == b.Asks[i-1].ID {
return fmt.Errorf(askLoadBookFailure, b.ExchangeName, b.Pair, b.AssetType, errIDDuplication)
}
}
}
return nil
}
// Process processes incoming orderbooks, creating or updating the orderbook
// list
func (b *Base) Process() error {
if b.ExchangeName == "" {
return errors.New(errExchangeNameUnset)
return errExchangeNameUnset
}
if b.Pair.IsEmpty() {
return errors.New(errPairNotSet)
return errPairNotSet
}
if b.AssetType.String() == "" {
return errors.New(errAssetTypeNotSet)
}
if len(b.Asks) == 0 && len(b.Bids) == 0 {
return errors.New(errNoOrderbook)
return errAssetTypeNotSet
}
if b.LastUpdated.IsZero() {
b.LastUpdated = time.Now()
}
b.Verify()
err := b.Verify()
if err != nil {
return err
}
return service.Update(b)
}
// Reverse reverses the order of orderbook items; some bid/asks are
// returned in either ascending or descending order. One bid or ask slice
// depending on whats received can be reversed. This is usually faster than
// using a sort algorithm as the algorithm could be impeded by a worst case time
// complexity when elements are shifted as opposed to just swapping element
// values.
func Reverse(elem []Item) {
eLen := len(elem)
var target int
for i := eLen/2 - 1; i >= 0; i-- {
target = eLen - 1 - i
elem[i], elem[target] = elem[target], elem[i]
}
}
// SortAsks sorts ask items to the correct ascending order if pricing values are
// scattered. If order from exchange is descending consider using the Reverse
// function.
func SortAsks(d []Item) []Item {
sort.Sort(byOBPrice(d))
return d
}
// SortBids sorts bid items to the correct descending order if pricing values
// are scattered. If order from exchange is ascending consider using the Reverse
// function.
func SortBids(d []Item) []Item {
sort.Sort(sort.Reverse(byOBPrice(d)))
return d
}

270
exchanges/orderbook/orderbook_test.go
View File

@@ -1,6 +1,7 @@
package orderbook
import (
"errors"
"log"
"math/rand"
"os"
@@ -46,14 +47,7 @@ func TestSubscribeOrderbook(t *testing.T) {
}
b.ExchangeName = "SubscribeOBTest"
err = b.Process()
if err == nil {
t.Error("error cannot be nil")
}
b.Bids = []Item{{}}
b.Bids = []Item{{Price: 100, Amount: 1}, {Price: 99, Amount: 1}}
err = b.Process()
if err != nil {
t.Error("process error", err)
@@ -61,7 +55,7 @@ func TestSubscribeOrderbook(t *testing.T) {
_, err = SubscribeOrderbook("SubscribeOBTest", p, asset.Spot)
if err != nil {
t.Error("error cannot be nil")
t.Error(err)
}
// process redundant update
@@ -120,7 +114,7 @@ func TestSubscribeToExchangeOrderbooks(t *testing.T) {
Pair: p,
AssetType: asset.Spot,
ExchangeName: "SubscribeToExchangeOrderbooks",
Bids: []Item{{}},
Bids: []Item{{Price: 100, Amount: 1}, {Price: 99, Amount: 1}},
}
err = b.Process()
@@ -138,21 +132,87 @@ func TestVerify(t *testing.T) {
t.Parallel()
b := Base{
ExchangeName: "TestExchange",
AssetType: asset.Spot,
Pair: currency.NewPair(currency.BTC, currency.USD),
Bids: []Item{
{Price: 100}, {Price: 101}, {Price: 99},
},
Asks: []Item{
{Price: 100}, {Price: 99}, {Price: 101},
},
}
b.Verify()
if r := b.Bids[1].Price; r != 100 {
t.Error("unexpected result")
err := b.Verify()
if err != nil {
t.Fatalf("expecting %v error but received %v", nil, err)
}
if r := b.Asks[1].Price; r != 100 {
t.Error("unexpected result")
b.Asks = []Item{{ID: 1337, Price: 99, Amount: 1}, {ID: 1337, Price: 100, Amount: 1}}
err = b.Verify()
if err == nil || !errors.Is(err, errIDDuplication) {
t.Fatalf("expecting %s error but received %v", errIDDuplication, err)
}
b.Asks = []Item{{Price: 100, Amount: 1}, {Price: 100, Amount: 1}}
err = b.Verify()
if err == nil || !errors.Is(err, errDuplication) {
t.Fatalf("expecting %s error but received %v", errDuplication, err)
}
b.Asks = []Item{{Price: 100, Amount: 1}, {Price: 99, Amount: 1}}
b.IsFundingRate = true
err = b.Verify()
if err == nil || !errors.Is(err, errPeriodUnset) {
t.Fatalf("expecting %s error but received %v", errPeriodUnset, err)
}
b.IsFundingRate = false
err = b.Verify()
if err == nil || !errors.Is(err, errOutOfOrder) {
t.Fatalf("expecting %s error but received %v", errOutOfOrder, err)
}
b.Asks = []Item{{Price: 100, Amount: 1}, {Price: 100, Amount: 0}}
err = b.Verify()
if err == nil || !errors.Is(err, errAmountInvalid) {
t.Fatalf("expecting %s error but received %v", errAmountInvalid, err)
}
b.Asks = []Item{{Price: 100, Amount: 1}, {Price: 0, Amount: 100}}
err = b.Verify()
if err == nil || !errors.Is(err, errPriceNotSet) {
t.Fatalf("expecting %s error but received %v", errPriceNotSet, err)
}
b.Bids = []Item{{ID: 1337, Price: 100, Amount: 1}, {ID: 1337, Price: 99, Amount: 1}}
err = b.Verify()
if err == nil || !errors.Is(err, errIDDuplication) {
t.Fatalf("expecting %s error but received %v", errIDDuplication, err)
}
b.Bids = []Item{{Price: 100, Amount: 1}, {Price: 100, Amount: 1}}
err = b.Verify()
if err == nil || !errors.Is(err, errDuplication) {
t.Fatalf("expecting %s error but received %v", errDuplication, err)
}
b.Bids = []Item{{Price: 99, Amount: 1}, {Price: 100, Amount: 1}}
b.IsFundingRate = true
err = b.Verify()
if err == nil || !errors.Is(err, errPeriodUnset) {
t.Fatalf("expecting %s error but received %v", errPeriodUnset, err)
}
b.IsFundingRate = false
err = b.Verify()
if err == nil || !errors.Is(err, errOutOfOrder) {
t.Fatalf("expecting %s error but received %v", errOutOfOrder, err)
}
b.Bids = []Item{{Price: 100, Amount: 1}, {Price: 100, Amount: 0}}
err = b.Verify()
if err == nil || !errors.Is(err, errAmountInvalid) {
t.Fatalf("expecting %s error but received %v", errAmountInvalid, err)
}
b.Bids = []Item{{Price: 100, Amount: 1}, {Price: 0, Amount: 100}}
err = b.Verify()
if err == nil || !errors.Is(err, errPriceNotSet) {
t.Fatalf("expecting %s error but received %v", errPriceNotSet, err)
}
}
@@ -517,16 +577,172 @@ func TestProcessOrderbook(t *testing.T) {
wg.Wait()
}
func TestSetNewData(t *testing.T) {
err := service.SetNewData(nil, "")
func deployUnorderedSlice() []Item {
var items []Item
rand.Seed(time.Now().UnixNano())
for i := 0; i < 1000; i++ {
items = append(items, Item{Amount: 1, Price: rand.Float64(), ID: rand.Int63()}) // nolint:gosec // Not needed in tests
}
return items
}
func TestSorting(t *testing.T) {
var b Base
b.Asks = deployUnorderedSlice()
err := b.Verify()
if err == nil {
t.Error("error cannot be nil")
t.Fatal("error cannot be nil")
}
SortAsks(nil)
SortAsks(b.Asks)
err = b.Verify()
if err != nil {
t.Fatal(err)
}
b.Bids = deployUnorderedSlice()
err = b.Verify()
if err == nil {
t.Fatal("error cannot be nil")
}
SortBids(nil)
SortBids(b.Bids)
err = b.Verify()
if err != nil {
t.Fatal(err)
}
}
func TestGetAssociations(t *testing.T) {
_, err := service.GetAssociations(nil, "")
func deploySliceOrdered() []Item {
rand.Seed(time.Now().UnixNano())
var items []Item
for i := 0; i < 1000; i++ {
items = append(items, Item{Amount: 1, Price: float64(i + 1), ID: rand.Int63()}) // nolint:gosec // Not needed in tests
}
return items
}
func TestReverse(t *testing.T) {
var b Base
length := 1000
b.Bids = deploySliceOrdered()
if len(b.Bids) != length {
t.Fatal("incorrect length")
}
err := b.Verify()
if err == nil {
t.Error("error cannot be nil")
t.Fatal("error cannot be nil")
}
Reverse(nil)
Reverse(b.Bids)
err = b.Verify()
if err != nil {
t.Fatal(err)
}
b.Asks = append(b.Bids[:0:0], b.Bids...) // nolint:gocritic // Short hand
err = b.Verify()
if err == nil {
t.Fatal("error cannot be nil")
}
Reverse(b.Asks)
err = b.Verify()
if err != nil {
t.Fatal(err)
}
}
// 705985 1856 ns/op 0 B/op 0 allocs/op
func BenchmarkReverse(b *testing.B) {
length := 1000
s := deploySliceOrdered()
if len(s) != length {
b.Fatal("incorrect length")
}
for i := 0; i < b.N; i++ {
Reverse(s)
}
}
// 20209 56385 ns/op 49189 B/op 2 allocs/op
func BenchmarkSortAsksDecending(b *testing.B) {
s := deploySliceOrdered()
for i := 0; i < b.N; i++ {
ts := append(s[:0:0], s...)
SortAsks(ts)
}
}
// 14924 79199 ns/op 49206 B/op 3 allocs/op
func BenchmarkSortBidsAscending(b *testing.B) {
s := deploySliceOrdered()
Reverse(s)
for i := 0; i < b.N; i++ {
ts := append(s[:0:0], s...)
SortBids(ts)
}
}
// 9842 133761 ns/op 49194 B/op 2 allocs/op
func BenchmarkSortAsksStandard(b *testing.B) {
s := deployUnorderedSlice()
for i := 0; i < b.N; i++ {
ts := append(s[:0:0], s...)
SortAsks(ts)
}
}
// 7058 155057 ns/op 49214 B/op 3 allocs/op
func BenchmarkSortBidsStandard(b *testing.B) {
s := deployUnorderedSlice()
for i := 0; i < b.N; i++ {
ts := append(s[:0:0], s...)
SortBids(ts)
}
}
// 20565 57001 ns/op 49188 B/op 2 allocs/op
func BenchmarkSortAsksAscending(b *testing.B) {
s := deploySliceOrdered()
for i := 0; i < b.N; i++ {
ts := append(s[:0:0], s...)
SortAsks(ts)
}
}
// 12565 97257 ns/op 49208 B/op 3 allocs/op
func BenchmarkSortBidsDescending(b *testing.B) {
s := deploySliceOrdered()
Reverse(s)
for i := 0; i < b.N; i++ {
ts := append(s[:0:0], s...)
SortBids(ts)
}
}
// 923154 1169 ns/op 4096 B/op 1 allocs/op
func BenchmarkDuplicatingSlice(b *testing.B) {
s := deploySliceOrdered()
for i := 0; i < b.N; i++ {
_ = append(s[:0:0], s...)
}
}
// 705922 1546 ns/op 4096 B/op 1 allocs/op
func BenchmarkCopySlice(b *testing.B) {
s := deploySliceOrdered()
for i := 0; i < b.N; i++ {
cpy := make([]Item, len(s))
copy(cpy, s)
}
}

32
exchanges/orderbook/orderbook_types.go
View File

@@ -1,6 +1,7 @@
package orderbook
import (
"errors"
"sync"
"time"
@@ -12,21 +13,31 @@ import (
// const values for orderbook package
const (
errExchangeNameUnset = "orderbook exchange name not set"
errPairNotSet = "orderbook currency pair not set"
errAssetTypeNotSet = "orderbook asset type not set"
errNoOrderbook = "orderbook bids and asks are empty"
bidLoadBookFailure = "cannot load book for exchange %s pair %s asset %s for Bids: %w"
askLoadBookFailure = "cannot load book for exchange %s pair %s asset %s for Asks: %w"
bookLengthIssue = "Potential book issue for exchange %s pair %s asset %s length Bids %d length Asks %d"
)
// Vars for the orderbook package
var (
service *Service
errExchangeNameUnset = errors.New("orderbook exchange name not set")
errPairNotSet = errors.New("orderbook currency pair not set")
errAssetTypeNotSet = errors.New("orderbook asset type not set")
errNoOrderbook = errors.New("orderbook bids and asks are empty")
errPriceNotSet = errors.New("price cannot be zero")
errAmountInvalid = errors.New("amount cannot be less or equal to zero")
errOutOfOrder = errors.New("pricing out of order")
errDuplication = errors.New("price duplication")
errIDDuplication = errors.New("id duplication")
errPeriodUnset = errors.New("funding rate period is unset")
)
func init() {
service = new(Service)
service.mux = dispatch.GetNewMux()
service.Books = make(map[string]map[*currency.Item]map[*currency.Item]map[asset.Item]*Book)
service.Books = make(map[string]map[asset.Item]map[*currency.Item]map[*currency.Item]*Book)
service.Exchange = make(map[string]uuid.UUID)
}
@@ -39,10 +50,10 @@ type Book struct {
// Service holds orderbook information for each individual exchange
type Service struct {
Books map[string]map[*currency.Item]map[*currency.Item]map[asset.Item]*Book
Books map[string]map[asset.Item]map[*currency.Item]map[*currency.Item]*Book
Exchange map[string]uuid.UUID
mux *dispatch.Mux
sync.RWMutex
sync.Mutex
}
// Item stores the amount and price values
@@ -51,6 +62,9 @@ type Item struct {
Price float64
ID int64
// Funding rate field
Period int64
// Contract variables
LiquidationOrders int64
OrderCount int64
@@ -65,6 +79,10 @@ type Base struct {
LastUpdateID int64 `json:"lastUpdateId"`
AssetType asset.Item `json:"assetType"`
ExchangeName string `json:"exchangeName"`
// NotAggregated defines whether an orderbook can contain duplicate prices
// in a payload
NotAggregated bool `json:"-"`
IsFundingRate bool `json:"fundingRate"`
}
type byOBPrice []Item

55
exchanges/poloniex/poloniex_websocket.go
View File

@@ -463,57 +463,58 @@ func (p *Poloniex) wsHandleTickerData(data []interface{}) error {
// WsProcessOrderbookSnapshot processes a new orderbook snapshot into a local
// of orderbooks
func (p *Poloniex) WsProcessOrderbookSnapshot(ob []interface{}, symbol string) error {
askdata := ob[0].(map[string]interface{})
var asks []orderbook.Item
if len(ob) != 2 {
return errors.New("incorrect orderbook data returned")
}
askdata, ok := ob[0].(map[string]interface{})
if !ok {
return errors.New("assertion failed for ask data")
}
var book orderbook.Base
for price, volume := range askdata {
assetPrice, err := strconv.ParseFloat(price, 64)
p, err := strconv.ParseFloat(price, 64)
if err != nil {
return err
}
assetVolume, err := strconv.ParseFloat(volume.(string), 64)
a, err := strconv.ParseFloat(volume.(string), 64)
if err != nil {
return err
}
asks = append(asks, orderbook.Item{
Price: assetPrice,
Amount: assetVolume,
})
book.Asks = append(book.Asks, orderbook.Item{Price: p, Amount: a})
}
bidData, ok := ob[1].(map[string]interface{})
if !ok {
return errors.New("assertion failed for bid data")
}
bidData := ob[1].(map[string]interface{})
var bids []orderbook.Item
for price, volume := range bidData {
assetPrice, err := strconv.ParseFloat(price, 64)
p, err := strconv.ParseFloat(price, 64)
if err != nil {
return err
}
assetVolume, err := strconv.ParseFloat(volume.(string), 64)
a, err := strconv.ParseFloat(volume.(string), 64)
if err != nil {
return err
}
bids = append(bids, orderbook.Item{
Price: assetPrice,
Amount: assetVolume,
})
book.Bids = append(book.Bids, orderbook.Item{Price: p, Amount: a})
}
var newOrderBook orderbook.Base
newOrderBook.Asks = asks
newOrderBook.Bids = bids
newOrderBook.AssetType = asset.Spot
// Both sides are completely out of order - sort needs to be used
book.Asks = orderbook.SortAsks(book.Asks)
book.Bids = orderbook.SortBids(book.Bids)
book.AssetType = asset.Spot
var err error
newOrderBook.Pair, err = currency.NewPairFromString(symbol)
book.Pair, err = currency.NewPairFromString(symbol)
if err != nil {
return err
}
newOrderBook.ExchangeName = p.Name
book.ExchangeName = p.Name
return p.Websocket.Orderbook.LoadSnapshot(&newOrderBook)
return p.Websocket.Orderbook.LoadSnapshot(&book)
}
// WsProcessOrderbookUpdate processes new orderbook updates

30
exchanges/poloniex/poloniex_wrapper.go
View File

@@ -167,6 +167,7 @@ func (p *Poloniex) Setup(exch *config.ExchangeConfig) error {
GenerateSubscriptions: p.GenerateDefaultSubscriptions,
Features: &p.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
SortBuffer: true,
SortBufferByUpdateIDs: true,
})
@@ -318,50 +319,51 @@ func (p *Poloniex) FetchOrderbook(currencyPair currency.Pair, assetType asset.It
}
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (p *Poloniex) UpdateOrderbook(currencyPair currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
func (p *Poloniex) UpdateOrderbook(c currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
callingBook := &orderbook.Base{ExchangeName: p.Name, Pair: c, AssetType: assetType}
orderbookNew, err := p.GetOrderbook("", poloniexMaxOrderbookDepth)
if err != nil {
return nil, err
return callingBook, err
}
enabledPairs, err := p.GetEnabledPairs(assetType)
if err != nil {
return nil, err
return callingBook, err
}
for i := range enabledPairs {
book := &orderbook.Base{
ExchangeName: p.Name,
Pair: enabledPairs[i],
AssetType: assetType}
fpair, err := p.FormatExchangeCurrency(enabledPairs[i], assetType)
if err != nil {
return nil, err
return book, err
}
data, ok := orderbookNew.Data[fpair.String()]
if !ok {
continue
}
orderBook := new(orderbook.Base)
for y := range data.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: data.Bids[y].Amount,
Price: data.Bids[y].Price,
})
}
for y := range data.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: data.Asks[y].Amount,
Price: data.Asks[y].Price,
})
}
orderBook.Pair = enabledPairs[i]
orderBook.ExchangeName = p.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
}
return orderbook.Get(p.Name, currencyPair, assetType)
return orderbook.Get(p.Name, c, assetType)
}
// UpdateAccountInfo retrieves balances for all enabled currencies for the

462
exchanges/stream/buffer/buffer.go
View File

@@ -10,8 +10,31 @@ import (
"github.com/thrasher-corp/gocryptotrader/exchanges/orderbook"
)
const packageError = "websocket orderbook buffer error: %w"
var (
errUnsetExchangeName = errors.New("exchange name unset")
errUnsetDataHandler = errors.New("datahandler unset")
errIssueBufferEnabledButNoLimit = errors.New("buffer enabled but no limit set")
errUpdateIsNil = errors.New("update is nil")
errUpdateNoTargets = errors.New("update bid/ask targets cannot be nil")
)
// Setup sets private variables
func (w *Orderbook) Setup(obBufferLimit int, bufferEnabled, sortBuffer, sortBufferByUpdateIDs, updateEntriesByID bool, exchangeName string, dataHandler chan interface{}) {
func (w *Orderbook) Setup(obBufferLimit int,
bufferEnabled,
sortBuffer,
sortBufferByUpdateIDs,
updateEntriesByID bool, exchangeName string, dataHandler chan interface{}) error {
if exchangeName == "" {
return fmt.Errorf(packageError, errUnsetExchangeName)
}
if dataHandler == nil {
return fmt.Errorf(packageError, errUnsetDataHandler)
}
if bufferEnabled && obBufferLimit < 1 {
return fmt.Errorf(packageError, errIssueBufferEnabledButNoLimit)
}
w.obBufferLimit = obBufferLimit
w.bufferEnabled = bufferEnabled
w.sortBuffer = sortBuffer
@@ -19,6 +42,19 @@ func (w *Orderbook) Setup(obBufferLimit int, bufferEnabled, sortBuffer, sortBuff
w.updateEntriesByID = updateEntriesByID
w.exchangeName = exchangeName
w.dataHandler = dataHandler
w.ob = make(map[currency.Code]map[currency.Code]map[asset.Item]*orderbookHolder)
return nil
}
// validate validates update against setup values
func (w *Orderbook) validate(u *Update) error {
if u == nil {
return fmt.Errorf(packageError, errUpdateIsNil)
}
if len(u.Bids) == 0 && len(u.Asks) == 0 {
return fmt.Errorf(packageError, errUpdateNoTargets)
}
return nil
}
// Update updates a local buffer using bid targets and ask targets then updates
@@ -27,13 +63,12 @@ func (w *Orderbook) Setup(obBufferLimit int, bufferEnabled, sortBuffer, sortBuff
// Price target not found; append of price target
// Price target found; amend volume of price target
func (w *Orderbook) Update(u *Update) error {
if (u.Bids == nil && u.Asks == nil) || (len(u.Bids) == 0 && len(u.Asks) == 0) {
return fmt.Errorf("%v cannot have bids and ask targets both nil",
w.exchangeName)
if err := w.validate(u); err != nil {
return err
}
w.m.Lock()
defer w.m.Unlock()
obLookup, ok := w.ob[u.Pair][u.Asset]
obLookup, ok := w.ob[u.Pair.Base][u.Pair.Quote][u.Asset]
if !ok {
return fmt.Errorf("ob.Base could not be found for Exchange %s CurrencyPair: %s AssetType: %s",
w.exchangeName,
@@ -42,246 +77,345 @@ func (w *Orderbook) Update(u *Update) error {
}
if w.bufferEnabled {
overBufferLimit := w.processBufferUpdate(obLookup, u)
if !overBufferLimit {
processed, err := w.processBufferUpdate(obLookup, u)
if err != nil {
return err
}
if !processed {
return nil
}
} else {
w.processObUpdate(obLookup, u)
err := w.processObUpdate(obLookup, u)
if err != nil {
return err
}
}
err := obLookup.Process()
err := obLookup.ob.Process()
if err != nil {
return err
}
if w.bufferEnabled {
// Reset the buffer
w.buffer[u.Pair][u.Asset] = nil
}
// Process in data handler
w.dataHandler <- obLookup
select {
case w.dataHandler <- obLookup.ob:
default:
}
return nil
}
func (w *Orderbook) processBufferUpdate(o *orderbook.Base, u *Update) bool {
if w.buffer == nil {
w.buffer = make(map[currency.Pair]map[asset.Item][]*Update)
}
if w.buffer[u.Pair] == nil {
w.buffer[u.Pair] = make(map[asset.Item][]*Update)
}
bufferLookup := w.buffer[u.Pair][u.Asset]
if len(bufferLookup) <= w.obBufferLimit {
bufferLookup = append(bufferLookup, u)
if len(bufferLookup) < w.obBufferLimit {
w.buffer[u.Pair][u.Asset] = bufferLookup
return false
}
// processBufferUpdate stores update into buffer, when buffer at capacity as
// defined by w.obBufferLimit it well then sort and apply updates.
func (w *Orderbook) processBufferUpdate(o *orderbookHolder, u *Update) (bool, error) {
*o.buffer = append(*o.buffer, *u)
if len(*o.buffer) < w.obBufferLimit {
return false, nil
}
if w.sortBuffer {
// sort by last updated to ensure each update is in order
if w.sortBufferByUpdateIDs {
sort.Slice(bufferLookup, func(i, j int) bool {
return bufferLookup[i].UpdateID < bufferLookup[j].UpdateID
sort.Slice(*o.buffer, func(i, j int) bool {
return (*o.buffer)[i].UpdateID < (*o.buffer)[j].UpdateID
})
} else {
sort.Slice(bufferLookup, func(i, j int) bool {
return bufferLookup[i].UpdateTime.Before(bufferLookup[j].UpdateTime)
sort.Slice(*o.buffer, func(i, j int) bool {
return (*o.buffer)[i].UpdateTime.Before((*o.buffer)[j].UpdateTime)
})
}
}
for i := range bufferLookup {
w.processObUpdate(o, bufferLookup[i])
for i := range *o.buffer {
err := w.processObUpdate(o, &(*o.buffer)[i])
if err != nil {
return false, err
}
}
w.buffer[u.Pair][u.Asset] = bufferLookup
return true
// clear buffer of old updates
*o.buffer = nil
return true, nil
}
func (w *Orderbook) processObUpdate(o *orderbook.Base, u *Update) {
o.LastUpdateID = u.UpdateID
// processObUpdate processes updates either by its corresponding id or by
// price level
func (w *Orderbook) processObUpdate(o *orderbookHolder, u *Update) error {
o.ob.LastUpdateID = u.UpdateID
if w.updateEntriesByID {
w.updateByIDAndAction(o, u)
} else {
w.updateAsksByPrice(o, u)
w.updateBidsByPrice(o, u)
return o.updateByIDAndAction(u)
}
return o.updateByPrice(u)
}
func (w *Orderbook) updateAsksByPrice(o *orderbook.Base, u *Update) {
updates:
for j := range u.Asks {
for k := range o.Asks {
if o.Asks[k].Price == u.Asks[j].Price {
if u.Asks[j].Amount <= 0 {
o.Asks = append(o.Asks[:k], o.Asks[k+1:]...)
continue updates
// updateByPrice ammends amount if match occurs by price, deletes if amount is
// zero or less and inserts if not found.
func (o *orderbookHolder) updateByPrice(updts *Update) error {
askUpdates:
for j := range updts.Asks {
for target := range o.ob.Asks {
if o.ob.Asks[target].Price == updts.Asks[j].Price {
if updts.Asks[j].Amount == 0 {
o.ob.Asks = append(o.ob.Asks[:target], o.ob.Asks[target+1:]...)
continue askUpdates
}
o.Asks[k].Amount = u.Asks[j].Amount
continue updates
o.ob.Asks[target].Amount = updts.Asks[j].Amount
continue askUpdates
}
}
if u.Asks[j].Amount == 0 {
if updts.Asks[j].Amount <= 0 {
continue
}
o.Asks = append(o.Asks, u.Asks[j])
insertAsk(updts.Asks[j], &o.ob.Asks)
if updts.MaxDepth != 0 && len(o.ob.Asks) > updts.MaxDepth {
o.ob.Asks = o.ob.Asks[:updts.MaxDepth]
}
}
sort.Slice(o.Asks, func(i, j int) bool {
return o.Asks[i].Price < o.Asks[j].Price
})
}
func (w *Orderbook) updateBidsByPrice(o *orderbook.Base, u *Update) {
updates:
for j := range u.Bids {
for k := range o.Bids {
if o.Bids[k].Price == u.Bids[j].Price {
if u.Bids[j].Amount <= 0 {
o.Bids = append(o.Bids[:k], o.Bids[k+1:]...)
continue updates
bidUpdates:
for j := range updts.Bids {
for target := range o.ob.Bids {
if o.ob.Bids[target].Price == updts.Bids[j].Price {
if updts.Bids[j].Amount == 0 {
o.ob.Bids = append(o.ob.Bids[:target], o.ob.Bids[target+1:]...)
continue bidUpdates
}
o.Bids[k].Amount = u.Bids[j].Amount
continue updates
o.ob.Bids[target].Amount = updts.Bids[j].Amount
continue bidUpdates
}
}
if u.Bids[j].Amount == 0 {
if updts.Bids[j].Amount <= 0 {
continue
}
o.Bids = append(o.Bids, u.Bids[j])
insertBid(updts.Bids[j], &o.ob.Bids)
if updts.MaxDepth != 0 && len(o.ob.Bids) > updts.MaxDepth {
o.ob.Bids = o.ob.Bids[:updts.MaxDepth]
}
}
sort.Slice(o.Bids, func(i, j int) bool {
return o.Bids[i].Price > o.Bids[j].Price
})
return nil
}
// updateByIDAndAction will receive an action to execute against the orderbook
// it will then match by IDs instead of price to perform the action
func (w *Orderbook) updateByIDAndAction(o *orderbook.Base, u *Update) {
switch u.Action {
case "update":
for x := range u.Bids {
for y := range o.Bids {
if o.Bids[y].ID == u.Bids[x].ID {
o.Bids[y].Amount = u.Bids[x].Amount
break
}
}
func (o *orderbookHolder) updateByIDAndAction(updts *Update) (err error) {
switch updts.Action {
case Amend:
err = applyUpdates(updts.Bids, o.ob.Bids)
if err != nil {
return err
}
for x := range u.Asks {
for y := range o.Asks {
if o.Asks[y].ID == u.Asks[x].ID {
o.Asks[y].Amount = u.Asks[x].Amount
break
}
}
err = applyUpdates(updts.Asks, o.ob.Asks)
if err != nil {
return err
}
case "delete":
for x := range u.Bids {
for y := 0; y < len(o.Bids); y++ {
if o.Bids[y].ID == u.Bids[x].ID {
o.Bids = append(o.Bids[:y], o.Bids[y+1:]...)
break
}
}
case Delete:
// edge case for Bitfinex as their streaming endpoint duplicates deletes
bypassErr := o.ob.ExchangeName == "Bitfinex" && o.ob.IsFundingRate
err = deleteUpdates(updts.Bids, &o.ob.Bids, bypassErr)
if err != nil {
return fmt.Errorf("%s %s %v", o.ob.AssetType, o.ob.Pair, err)
}
for x := range u.Asks {
for y := 0; y < len(o.Asks); y++ {
if o.Asks[y].ID == u.Asks[x].ID {
o.Asks = append(o.Asks[:y], o.Asks[y+1:]...)
break
}
}
err = deleteUpdates(updts.Asks, &o.ob.Asks, bypassErr)
if err != nil {
return fmt.Errorf("%s %s %v", o.ob.AssetType, o.ob.Pair, err)
}
case "insert":
o.Bids = append(o.Bids, u.Bids...)
sort.Slice(o.Bids, func(i, j int) bool {
return o.Bids[i].Price > o.Bids[j].Price
})
o.Asks = append(o.Asks, u.Asks...)
sort.Slice(o.Asks, func(i, j int) bool {
return o.Asks[i].Price < o.Asks[j].Price
})
case "update/insert":
case Insert:
insertUpdatesBid(updts.Bids, &o.ob.Bids)
insertUpdatesAsk(updts.Asks, &o.ob.Asks)
case UpdateInsert:
updateBids:
for x := range u.Bids {
for y := range o.Bids {
if o.Bids[y].ID == u.Bids[x].ID {
o.Bids[y].Amount = u.Bids[x].Amount
for x := range updts.Bids {
for target := range o.ob.Bids { // First iteration finds ID matches
if o.ob.Bids[target].ID == updts.Bids[x].ID {
if o.ob.Bids[target].Price != updts.Bids[x].Price {
// Price change occurred so correct bid alignment is
// needed - delete instance and insert into correct
// price level
o.ob.Bids = append(o.ob.Bids[:target], o.ob.Bids[target+1:]...)
break
}
o.ob.Bids[target].Amount = updts.Bids[x].Amount
continue updateBids
}
}
o.Bids = append(o.Bids, u.Bids[x])
insertBid(updts.Bids[x], &o.ob.Bids)
}
updateAsks:
for x := range u.Asks {
for y := range o.Asks {
if o.Asks[y].ID == u.Asks[x].ID {
o.Asks[y].Amount = u.Asks[x].Amount
for x := range updts.Asks {
for target := range o.ob.Asks {
if o.ob.Asks[target].ID == updts.Asks[x].ID {
if o.ob.Asks[target].Price != updts.Asks[x].Price {
// Price change occurred so correct ask alignment is
// needed - delete instance and insert into correct
// price level
o.ob.Asks = append(o.ob.Asks[:target], o.ob.Asks[target+1:]...)
break
}
o.ob.Asks[target].Amount = updts.Asks[x].Amount
continue updateAsks
}
}
o.Asks = append(o.Asks, u.Asks[x])
insertAsk(updts.Asks[x], &o.ob.Asks)
}
default:
return fmt.Errorf("invalid action [%s]", updts.Action)
}
return nil
}
// applyUpdates amends amount by ID and returns an error if not found
func applyUpdates(updts, book []orderbook.Item) error {
updates:
for x := range updts {
for y := range book {
if book[y].ID == updts[x].ID {
book[y].Amount = updts[x].Amount
continue updates
}
}
return fmt.Errorf("update cannot be applied id: %d not found",
updts[x].ID)
}
return nil
}
// deleteUpdates removes updates from orderbook and returns an error if not
// found
func deleteUpdates(updt []orderbook.Item, book *[]orderbook.Item, bypassErr bool) error {
updates:
for x := range updt {
for y := range *book {
if (*book)[y].ID == updt[x].ID {
*book = append((*book)[:y], (*book)[y+1:]...) // nolint:gocritic
continue updates
}
}
// bypassErr is for expected duplication from endpoint.
if !bypassErr {
return fmt.Errorf("update cannot be deleted id: %d not found",
updt[x].ID)
}
}
return nil
}
func insertAsk(updt orderbook.Item, book *[]orderbook.Item) {
for target := range *book {
if updt.Price < (*book)[target].Price {
insertItem(updt, book, target)
return
}
}
*book = append(*book, updt)
}
func insertBid(updt orderbook.Item, book *[]orderbook.Item) {
for target := range *book {
if updt.Price > (*book)[target].Price {
insertItem(updt, book, target)
return
}
}
*book = append(*book, updt)
}
// insertUpdatesBid inserts on **correctly aligned** book at price level
func insertUpdatesBid(updt []orderbook.Item, book *[]orderbook.Item) {
updates:
for x := range updt {
for target := range *book {
if updt[x].Price > (*book)[target].Price {
insertItem(updt[x], book, target)
continue updates
}
}
*book = append(*book, updt[x])
}
}
// LoadSnapshot loads initial snapshot of ob data, overwrite allows full
// ob to be completely rewritten because the exchange is a doing a full
// update not an incremental one
func (w *Orderbook) LoadSnapshot(newOrderbook *orderbook.Base) error {
if len(newOrderbook.Asks) == 0 || len(newOrderbook.Bids) == 0 {
return fmt.Errorf("%v snapshot ask and bids are nil", w.exchangeName)
// insertUpdatesBid inserts on **correctly aligned** book at price level
func insertUpdatesAsk(updt []orderbook.Item, book *[]orderbook.Item) {
updates:
for x := range updt {
for target := range *book {
if updt[x].Price < (*book)[target].Price {
insertItem(updt[x], book, target)
continue updates
}
}
*book = append(*book, updt[x])
}
}
if newOrderbook.Pair.IsEmpty() {
return errors.New("websocket orderbook pair unset")
}
if newOrderbook.AssetType.String() == "" {
return errors.New("websocket orderbook asset type unset")
}
if newOrderbook.ExchangeName == "" {
return errors.New("websocket orderbook exchange name unset")
}
// insertItem inserts item in slice by target element this is an optimization
// to reduce the need for sorting algorithms
func insertItem(update orderbook.Item, book *[]orderbook.Item, target int) {
// TODO: extend slice by incoming update length before this gets hit
*book = append(*book, orderbook.Item{})
copy((*book)[target+1:], (*book)[target:])
(*book)[target] = update
}
// LoadSnapshot loads initial snapshot of ob data from websocket
func (w *Orderbook) LoadSnapshot(book *orderbook.Base) error {
w.m.Lock()
defer w.m.Unlock()
if w.ob == nil {
w.ob = make(map[currency.Pair]map[asset.Item]*orderbook.Base)
}
if w.ob[newOrderbook.Pair] == nil {
w.ob[newOrderbook.Pair] = make(map[asset.Item]*orderbook.Base)
}
w.ob[newOrderbook.Pair][newOrderbook.AssetType] = newOrderbook
err := newOrderbook.Process()
err := book.Process()
if err != nil {
return err
}
w.dataHandler <- newOrderbook
m1, ok := w.ob[book.Pair.Base]
if !ok {
m1 = make(map[currency.Code]map[asset.Item]*orderbookHolder)
w.ob[book.Pair.Base] = m1
}
m2, ok := m1[book.Pair.Quote]
if !ok {
m2 = make(map[asset.Item]*orderbookHolder)
m1[book.Pair.Quote] = m2
}
m3, ok := m2[book.AssetType]
if !ok {
m3 = &orderbookHolder{ob: book, buffer: &[]Update{}}
m2[book.AssetType] = m3
} else {
m3.ob.Bids = book.Bids
m3.ob.Asks = book.Asks
}
w.dataHandler <- book
return nil
}
// GetOrderbook use sparingly. Modifying anything here will ruin hash
// calculation and cause problems
// GetOrderbook returns orderbook stored in current buffer
func (w *Orderbook) GetOrderbook(p currency.Pair, a asset.Item) *orderbook.Base {
w.m.Lock()
ob := w.ob[p][a]
w.m.Unlock()
return ob
defer w.m.Unlock()
ptr, ok := w.ob[p.Base][p.Quote][a]
if !ok {
return nil
}
cpy := *ptr.ob
cpy.Asks = append(cpy.Asks[:0:0], cpy.Asks...)
cpy.Bids = append(cpy.Bids[:0:0], cpy.Bids...)
return &cpy
}
// FlushBuffer flushes w.ob data to be garbage collected and refreshed when a
// connection is lost and reconnected
func (w *Orderbook) FlushBuffer() {
w.m.Lock()
w.ob = nil
w.buffer = nil
w.ob = make(map[currency.Code]map[currency.Code]map[asset.Item]*orderbookHolder)
w.m.Unlock()
}
// FlushOrderbook flushes independent orderbook
func (w *Orderbook) FlushOrderbook(p currency.Pair, a asset.Item) error {
w.m.Lock()
defer w.m.Unlock()
book, ok := w.ob[p.Base][p.Quote][a]
if !ok {
return fmt.Errorf("orderbook not associated with pair: [%s] and asset [%s]", p, a)
}
book.ob.Bids = nil
book.ob.Asks = nil
return nil
}

569
exchanges/stream/buffer/buffer_test.go
View File

@@ -1,7 +1,7 @@
package buffer
import (
"fmt"
"errors"
"math/rand"
"testing"
"time"
@@ -12,12 +12,12 @@ import (
)
var itemArray = [][]orderbook.Item{
{{Price: 1000, Amount: 1, ID: 1}},
{{Price: 2000, Amount: 1, ID: 2}},
{{Price: 3000, Amount: 1, ID: 3}},
{{Price: 3000, Amount: 2, ID: 4}},
{{Price: 4000, Amount: 0, ID: 6}},
{{Price: 5000, Amount: 1, ID: 5}},
{{Price: 1000, Amount: 1, ID: 1000}},
{{Price: 2000, Amount: 1, ID: 2000}},
{{Price: 3000, Amount: 1, ID: 3000}},
{{Price: 3000, Amount: 2, ID: 4000}},
{{Price: 4000, Amount: 0, ID: 6000}},
{{Price: 5000, Amount: 1, ID: 5000}},
}
var cp, _ = currency.NewPairFromString("BTCUSD")
@@ -39,7 +39,12 @@ func createSnapshot() (obl *Orderbook, asks, bids []orderbook.Item, err error) {
snapShot1.Bids = bids
snapShot1.AssetType = asset.Spot
snapShot1.Pair = cp
obl = &Orderbook{exchangeName: exchangeName, dataHandler: make(chan interface{}, 100)}
snapShot1.NotAggregated = true
obl = &Orderbook{
exchangeName: exchangeName,
dataHandler: make(chan interface{}, 100),
ob: make(map[currency.Code]map[currency.Code]map[asset.Item]*orderbookHolder),
}
err = obl.LoadSnapshot(&snapShot1)
return
}
@@ -76,7 +81,8 @@ func BenchmarkUpdateBidsByPrice(b *testing.B) {
UpdateTime: time.Now(),
Asset: asset.Spot,
}
ob.updateBidsByPrice(ob.ob[cp][asset.Spot], update)
holder := ob.ob[cp.Base][cp.Quote][asset.Spot]
holder.updateByPrice(update)
}
}
@@ -95,7 +101,8 @@ func BenchmarkUpdateAsksByPrice(b *testing.B) {
UpdateTime: time.Now(),
Asset: asset.Spot,
}
ob.updateAsksByPrice(ob.ob[cp][asset.Spot], update)
holder := ob.ob[cp.Base][cp.Quote][asset.Spot]
holder.updateByPrice(update)
}
}
@@ -114,7 +121,7 @@ func BenchmarkBufferPerformance(b *testing.B) {
Price: 1337.1337,
ID: 1337,
}
obl.ob[cp][asset.Spot].Bids = append(obl.ob[cp][asset.Spot].Bids, dummyItem)
obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids = append(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids, dummyItem)
update := &Update{
Bids: bids,
Asks: asks,
@@ -149,7 +156,7 @@ func BenchmarkBufferSortingPerformance(b *testing.B) {
Price: 1337.1337,
ID: 1337,
}
obl.ob[cp][asset.Spot].Bids = append(obl.ob[cp][asset.Spot].Bids, dummyItem)
obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids = append(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids, dummyItem)
update := &Update{
Bids: bids,
Asks: asks,
@@ -185,7 +192,7 @@ func BenchmarkBufferSortingByIDPerformance(b *testing.B) {
Price: 1337.1337,
ID: 1337,
}
obl.ob[cp][asset.Spot].Bids = append(obl.ob[cp][asset.Spot].Bids, dummyItem)
obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids = append(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids, dummyItem)
update := &Update{
Bids: bids,
Asks: asks,
@@ -219,7 +226,7 @@ func BenchmarkNoBufferPerformance(b *testing.B) {
Price: 1337.1337,
ID: 1337,
}
obl.ob[cp][asset.Spot].Bids = append(obl.ob[cp][asset.Spot].Bids, dummyItem)
obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids = append(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids, dummyItem)
update := &Update{
Bids: bids,
Asks: asks,
@@ -245,7 +252,8 @@ func TestUpdates(t *testing.T) {
t.Error(err)
}
obl.updateAsksByPrice(obl.ob[cp][asset.Spot], &Update{
holder := obl.ob[cp.Base][cp.Quote][asset.Spot]
holder.updateByPrice(&Update{
Bids: itemArray[5],
Asks: itemArray[5],
Pair: cp,
@@ -256,7 +264,7 @@ func TestUpdates(t *testing.T) {
t.Error(err)
}
obl.updateAsksByPrice(obl.ob[cp][asset.Spot], &Update{
holder.updateByPrice(&Update{
Bids: itemArray[0],
Asks: itemArray[0],
Pair: cp,
@@ -267,7 +275,7 @@ func TestUpdates(t *testing.T) {
t.Error(err)
}
if len(obl.ob[cp][asset.Spot].Asks) != 3 {
if len(holder.ob.Asks) != 3 {
t.Error("Did not update")
}
}
@@ -294,14 +302,13 @@ func TestHittingTheBuffer(t *testing.T) {
t.Fatal(err)
}
}
if len(obl.ob[cp][asset.Spot].Asks) != 3 {
t.Log(obl.ob[cp][asset.Spot])
if len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Asks) != 3 {
t.Errorf("expected 3 entries, received: %v",
len(obl.ob[cp][asset.Spot].Asks))
len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Asks))
}
if len(obl.ob[cp][asset.Spot].Bids) != 3 {
if len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids) != 3 {
t.Errorf("expected 3 entries, received: %v",
len(obl.ob[cp][asset.Spot].Bids))
len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids))
}
}
@@ -316,6 +323,9 @@ func TestInsertWithIDs(t *testing.T) {
obl.obBufferLimit = 5
for i := range itemArray {
asks := itemArray[i]
if asks[0].Amount <= 0 {
continue
}
bids := itemArray[i]
err = obl.Update(&Update{
Bids: bids,
@@ -329,13 +339,13 @@ func TestInsertWithIDs(t *testing.T) {
t.Fatal(err)
}
}
if len(obl.ob[cp][asset.Spot].Asks) != 6 {
if len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Asks) != 6 {
t.Errorf("expected 6 entries, received: %v",
len(obl.ob[cp][asset.Spot].Asks))
len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Asks))
}
if len(obl.ob[cp][asset.Spot].Bids) != 6 {
if len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids) != 6 {
t.Errorf("expected 6 entries, received: %v",
len(obl.ob[cp][asset.Spot].Bids))
len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids))
}
}
@@ -363,93 +373,13 @@ func TestSortIDs(t *testing.T) {
t.Fatal(err)
}
}
if len(obl.ob[cp][asset.Spot].Asks) != 3 {
if len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Asks) != 3 {
t.Errorf("expected 3 entries, received: %v",
len(obl.ob[cp][asset.Spot].Asks))
len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Asks))
}
if len(obl.ob[cp][asset.Spot].Bids) != 3 {
if len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids) != 3 {
t.Errorf("expected 3 entries, received: %v",
len(obl.ob[cp][asset.Spot].Bids))
}
}
// TestDeleteWithIDs logic test
func TestDeleteWithIDs(t *testing.T) {
obl, _, _, err := createSnapshot()
if err != nil {
t.Fatal(err)
}
// This is to ensure we do not send in zero orderbook info to our main book
// in orderbook.go, orderbooks should not be zero even after an update.
dummyItem := orderbook.Item{
Amount: 1333337,
Price: 1337.1337,
ID: 1337,
}
obl.ob[cp][asset.Spot].Bids = append(obl.ob[cp][asset.Spot].Bids, dummyItem)
obl.ob[cp][asset.Spot].Asks = append(obl.ob[cp][asset.Spot].Asks,
itemArray[2][0])
obl.ob[cp][asset.Spot].Asks = append(obl.ob[cp][asset.Spot].Asks,
itemArray[1][0])
obl.updateEntriesByID = true
for i := range itemArray {
asks := itemArray[i]
bids := itemArray[i]
err = obl.Update(&Update{
Bids: bids,
Asks: asks,
Pair: cp,
UpdateTime: time.Now(),
Asset: asset.Spot,
Action: "delete",
})
if err != nil {
t.Fatal(err)
}
}
if len(obl.ob[cp][asset.Spot].Asks) != 0 {
t.Errorf("expected 0 entries, received: %v",
len(obl.ob[cp][asset.Spot].Asks))
}
if len(obl.ob[cp][asset.Spot].Bids) != 1 {
t.Errorf("expected 1 entries, received: %v",
len(obl.ob[cp][asset.Spot].Bids))
}
}
// TestUpdateWithIDs logic test
func TestUpdateWithIDs(t *testing.T) {
obl, _, _, err := createSnapshot()
if err != nil {
t.Fatal(err)
}
obl.updateEntriesByID = true
for i := range itemArray {
asks := itemArray[i]
bids := itemArray[i]
err = obl.Update(&Update{
Bids: bids,
Asks: asks,
Pair: cp,
UpdateTime: time.Now(),
Asset: asset.Spot,
Action: "update",
})
if err != nil {
t.Fatal(err)
}
}
if len(obl.ob[cp][asset.Spot].Asks) != 1 {
t.Log(obl.ob[cp][asset.Spot])
t.Errorf("expected 1 entries, received: %v",
len(obl.ob[cp][asset.Spot].Asks))
}
if len(obl.ob[cp][asset.Spot].Bids) != 1 {
t.Errorf("expected 1 entries, received: %v",
len(obl.ob[cp][asset.Spot].Bids))
len(obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Bids))
}
}
@@ -480,9 +410,9 @@ func TestOutOfOrderIDs(t *testing.T) {
}
}
// Index 1 since index 0 is price 7000
if obl.ob[cp][asset.Spot].Asks[1].Price != 2000 {
if obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Asks[1].Price != 2000 {
t.Errorf("expected sorted price to be 3000, received: %v",
obl.ob[cp][asset.Spot].Asks[1].Price)
obl.ob[cp.Base][cp.Quote][asset.Spot].ob.Asks[1].Price)
}
}
@@ -565,8 +495,7 @@ func TestRunUpdateWithoutAnyUpdates(t *testing.T) {
if err == nil {
t.Fatal("expected an error running update with no snapshot loaded")
}
if err.Error() != fmt.Sprintf("%v cannot have bids and ask targets both nil",
exchangeName) {
if err.Error() != "websocket orderbook buffer error: update bid/ask targets cannot be nil" {
t.Fatal("expected nil asks and bids error")
}
}
@@ -574,18 +503,15 @@ func TestRunUpdateWithoutAnyUpdates(t *testing.T) {
// TestRunSnapshotWithNoData logic test
func TestRunSnapshotWithNoData(t *testing.T) {
var obl Orderbook
obl.ob = make(map[currency.Code]map[currency.Code]map[asset.Item]*orderbookHolder)
obl.dataHandler = make(chan interface{}, 1)
var snapShot1 orderbook.Base
snapShot1.Asks = []orderbook.Item{}
snapShot1.Bids = []orderbook.Item{}
snapShot1.AssetType = asset.Spot
snapShot1.Pair = cp
snapShot1.ExchangeName = "test"
obl.exchangeName = "test"
err := obl.LoadSnapshot(&snapShot1)
if err == nil {
t.Fatal("expected an error loading a snapshot")
}
if err.Error() != "test snapshot ask and bids are nil" {
if err != nil {
t.Fatal(err)
}
}
@@ -594,6 +520,7 @@ func TestRunSnapshotWithNoData(t *testing.T) {
func TestLoadSnapshot(t *testing.T) {
var obl Orderbook
obl.dataHandler = make(chan interface{}, 100)
obl.ob = make(map[currency.Code]map[currency.Code]map[asset.Item]*orderbookHolder)
var snapShot1 orderbook.Base
snapShot1.ExchangeName = "SnapshotWithOverride"
asks := []orderbook.Item{
@@ -618,11 +545,11 @@ func TestFlushbuffer(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if obl.ob[cp][asset.Spot] == nil {
if obl.ob[cp.Base][cp.Quote][asset.Spot] == nil {
t.Error("expected ob to have ask entries")
}
obl.FlushBuffer()
if obl.ob[cp][asset.Spot] != nil {
if obl.ob[cp.Base][cp.Quote][asset.Spot] != nil {
t.Error("expected ob be flushed")
}
}
@@ -631,6 +558,7 @@ func TestFlushbuffer(t *testing.T) {
func TestInsertingSnapShots(t *testing.T) {
var obl Orderbook
obl.dataHandler = make(chan interface{}, 100)
obl.ob = make(map[currency.Code]map[currency.Code]map[asset.Item]*orderbookHolder)
var snapShot1 orderbook.Base
snapShot1.ExchangeName = "WSORDERBOOKTEST1"
asks := []orderbook.Item{
@@ -699,8 +627,8 @@ func TestInsertingSnapShots(t *testing.T) {
{Price: 39, Amount: 7, ID: 22},
}
snapShot2.Asks = asks
snapShot2.Bids = bids
snapShot2.Asks = orderbook.SortAsks(asks)
snapShot2.Bids = orderbook.SortBids(bids)
snapShot2.AssetType = asset.Spot
snapShot2.Pair, err = currency.NewPairFromString("LTCUSD")
if err != nil {
@@ -740,8 +668,8 @@ func TestInsertingSnapShots(t *testing.T) {
{Price: 39, Amount: 7, ID: 22},
}
snapShot3.Asks = asks
snapShot3.Bids = bids
snapShot3.Asks = orderbook.SortAsks(asks)
snapShot3.Bids = orderbook.SortBids(bids)
snapShot3.AssetType = "FUTURES"
snapShot3.Pair, err = currency.NewPairFromString("LTCUSD")
if err != nil {
@@ -751,20 +679,20 @@ func TestInsertingSnapShots(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if obl.ob[snapShot1.Pair][snapShot1.AssetType].Asks[0] != snapShot1.Asks[0] {
if obl.ob[snapShot1.Pair.Base][snapShot1.Pair.Quote][snapShot1.AssetType].ob.Asks[0] != snapShot1.Asks[0] {
t.Errorf("loaded data mismatch. Expected %v, received %v",
snapShot1.Asks[0],
obl.ob[snapShot1.Pair][snapShot1.AssetType].Asks[0])
obl.ob[snapShot1.Pair.Base][snapShot1.Pair.Quote][snapShot1.AssetType].ob.Asks[0])
}
if obl.ob[snapShot2.Pair][snapShot2.AssetType].Asks[0] != snapShot2.Asks[0] {
if obl.ob[snapShot2.Pair.Base][snapShot1.Pair.Quote][snapShot2.AssetType].ob.Asks[0] != snapShot2.Asks[0] {
t.Errorf("loaded data mismatch. Expected %v, received %v",
snapShot2.Asks[0],
obl.ob[snapShot2.Pair][snapShot2.AssetType].Asks[0])
obl.ob[snapShot2.Pair.Base][snapShot1.Pair.Quote][snapShot2.AssetType].ob.Asks[0])
}
if obl.ob[snapShot3.Pair][snapShot3.AssetType].Asks[0] != snapShot3.Asks[0] {
if obl.ob[snapShot3.Pair.Base][snapShot1.Pair.Quote][snapShot3.AssetType].ob.Asks[0] != snapShot3.Asks[0] {
t.Errorf("loaded data mismatch. Expected %v, received %v",
snapShot3.Asks[0],
obl.ob[snapShot3.Pair][snapShot3.AssetType].Asks[0])
obl.ob[snapShot3.Pair.Base][snapShot1.Pair.Quote][snapShot3.AssetType].ob.Asks[0])
}
}
@@ -774,24 +702,66 @@ func TestGetOrderbook(t *testing.T) {
t.Fatal(err)
}
ob := obl.GetOrderbook(cp, asset.Spot)
if obl.ob[cp][asset.Spot] != ob {
t.Error("Failed to get orderbook")
bufferOb := obl.ob[cp.Base][cp.Quote][asset.Spot]
if bufferOb.ob == ob {
t.Error("orderbooks should be separate in pointer value and not linked to orderbook package")
}
if len(bufferOb.ob.Asks) != len(ob.Asks) ||
len(bufferOb.ob.Bids) != len(ob.Bids) ||
bufferOb.ob.AssetType != ob.AssetType ||
bufferOb.ob.ExchangeName != ob.ExchangeName ||
bufferOb.ob.LastUpdateID != ob.LastUpdateID ||
bufferOb.ob.NotAggregated != ob.NotAggregated ||
bufferOb.ob.Pair != ob.Pair {
t.Fatal("data on both books should be the same")
}
}
func TestSetup(t *testing.T) {
w := Orderbook{}
w.Setup(1, true, true, true, true, "hi", make(chan interface{}))
if w.obBufferLimit != 1 ||
err := w.Setup(0, false, false, false, false, "", nil)
if err == nil || !errors.Is(err, errUnsetExchangeName) {
t.Fatalf("expected error %v but received %v", errUnsetExchangeName, err)
}
err = w.Setup(0, false, false, false, false, "test", nil)
if err == nil || !errors.Is(err, errUnsetDataHandler) {
t.Fatalf("expected error %v but received %v", errUnsetDataHandler, err)
}
err = w.Setup(0, true, false, false, false, "test", make(chan interface{}))
if err == nil || !errors.Is(err, errIssueBufferEnabledButNoLimit) {
t.Fatalf("expected error %v but received %v", errIssueBufferEnabledButNoLimit, err)
}
err = w.Setup(1337, true, true, true, true, "test", make(chan interface{}))
if err != nil {
t.Fatal(err)
}
if w.obBufferLimit != 1337 ||
!w.bufferEnabled ||
!w.sortBuffer ||
!w.sortBufferByUpdateIDs ||
!w.updateEntriesByID ||
w.exchangeName != "hi" {
w.exchangeName != "test" {
t.Errorf("Setup incorrectly loaded %s", w.exchangeName)
}
}
func TestValidate(t *testing.T) {
w := Orderbook{}
err := w.validate(nil)
if err == nil || !errors.Is(err, errUpdateIsNil) {
t.Fatalf("expected error %v but received %v", errUpdateIsNil, err)
}
err = w.validate(&Update{})
if err == nil || !errors.Is(err, errUpdateNoTargets) {
t.Fatalf("expected error %v but received %v", errUpdateNoTargets, err)
}
}
func TestEnsureMultipleUpdatesViaPrice(t *testing.T) {
obl, _, _, err := createSnapshot()
if err != nil {
@@ -799,7 +769,8 @@ func TestEnsureMultipleUpdatesViaPrice(t *testing.T) {
}
asks := bidAskGenerator()
obl.updateAsksByPrice(obl.ob[cp][asset.Spot], &Update{
holder := obl.ob[cp.Base][cp.Quote][asset.Spot]
holder.updateByPrice(&Update{
Bids: asks,
Asks: asks,
Pair: cp,
@@ -810,7 +781,327 @@ func TestEnsureMultipleUpdatesViaPrice(t *testing.T) {
t.Error(err)
}
if len(obl.ob[cp][asset.Spot].Asks) <= 3 {
if len(holder.ob.Asks) <= 3 {
t.Errorf("Insufficient updates")
}
}
func TestInsertItem(t *testing.T) {
update := []orderbook.Item{{Price: 4}}
// Correctly aligned
asks := []orderbook.Item{
{
Price: 1,
},
{
Price: 2,
},
{
Price: 3,
},
{
Price: 5,
},
{
Price: 6,
},
{
Price: 7,
},
}
insertUpdatesAsk(update, &asks)
if asks[3].Price != 4 {
t.Fatal("incorrect insertion")
}
bids := []orderbook.Item{
{
Price: 7,
},
{
Price: 6,
},
{
Price: 5,
},
{
Price: 3,
},
{
Price: 2,
},
{
Price: 1,
},
}
insertUpdatesBid(update, &bids)
if asks[3].Price != 4 {
t.Fatal("incorrect insertion")
}
}
func deploySliceOrdered(size int) []orderbook.Item {
rand.Seed(time.Now().UnixNano())
var items []orderbook.Item
for i := 0; i < size; i++ {
items = append(items, orderbook.Item{Amount: 1, Price: rand.Float64() + float64(i), ID: rand.Int63()}) // nolint:gosec // Not needed for tests
}
return items
}
func TestUpdateByIDAndAction(t *testing.T) {
holder := orderbookHolder{}
asks := deploySliceOrdered(100)
bids := append(asks[:0:0], asks...)
orderbook.Reverse(bids)
book := &orderbook.Base{
Bids: append(bids[:0:0], bids...),
Asks: append(asks[:0:0], asks...),
}
err := book.Verify()
if err != nil {
t.Fatal(err)
}
holder.ob = book
err = holder.updateByIDAndAction(&Update{})
if err == nil {
t.Fatal("error cannot be nil")
}
err = holder.updateByIDAndAction(&Update{
Action: Amend,
Bids: []orderbook.Item{
{
Price: 100,
ID: 6969,
},
},
})
if err == nil {
t.Fatal("error cannot be nil")
}
// append to slice
err = holder.updateByIDAndAction(&Update{
Action: UpdateInsert,
Bids: []orderbook.Item{
{
Price: 0,
ID: 1337,
},
},
Asks: []orderbook.Item{
{
Price: 100,
ID: 1337,
},
},
})
if err != nil {
t.Fatal(err)
}
if book.Bids[len(book.Bids)-1].Price != 0 {
t.Fatal("did not append bid item")
}
if book.Asks[len(book.Asks)-1].Price != 100 {
t.Fatal("did not append ask item")
}
// Change amount
err = holder.updateByIDAndAction(&Update{
Action: UpdateInsert,
Bids: []orderbook.Item{
{
Price: 0,
ID: 1337,
Amount: 100,
},
},
Asks: []orderbook.Item{
{
Price: 100,
ID: 1337,
Amount: 100,
},
},
})
if err != nil {
t.Fatal(err)
}
if book.Bids[len(book.Bids)-1].Amount != 100 {
t.Fatal("did not update bid amount")
}
if book.Asks[len(book.Asks)-1].Amount != 100 {
t.Fatal("did not update ask amount")
}
// Change price level
err = holder.updateByIDAndAction(&Update{
Action: UpdateInsert,
Bids: []orderbook.Item{
{
Price: 100,
ID: 1337,
Amount: 99,
},
},
Asks: []orderbook.Item{
{
Price: 0,
ID: 1337,
Amount: 99,
},
},
})
if err != nil {
t.Fatal(err)
}
if book.Bids[0].Amount != 99 && book.Bids[0].Price != 100 {
t.Fatal("did not adjust bid item placement and details")
}
if book.Asks[0].Amount != 99 && book.Asks[0].Amount != 0 {
t.Fatal("did not adjust ask item placement and details")
}
book.Bids = append(bids[:0:0], bids...) // nolint:gocritic
book.Asks = append(asks[:0:0], asks...) // nolint:gocritic
// Delete - not found
err = holder.updateByIDAndAction(&Update{
Action: Delete,
Asks: []orderbook.Item{
{
Price: 0,
ID: 1337,
Amount: 99,
},
},
})
if err == nil {
t.Fatal("error cannot be nil")
}
err = holder.updateByIDAndAction(&Update{
Action: Delete,
Bids: []orderbook.Item{
{
Price: 0,
ID: 1337,
Amount: 99,
},
},
})
if err == nil {
t.Fatal("error cannot be nil")
}
// Delete - found
err = holder.updateByIDAndAction(&Update{
Action: Delete,
Asks: []orderbook.Item{
asks[0],
},
})
if err != nil {
t.Fatal(err)
}
if len(book.Asks) != 99 {
t.Fatal("element not deleted")
}
// Apply update
err = holder.updateByIDAndAction(&Update{
Action: Amend,
Asks: []orderbook.Item{
{ID: 123456},
},
})
if err == nil {
t.Fatal("error cannot be nil")
}
update := book.Asks[0]
update.Amount = 1337
err = holder.updateByIDAndAction(&Update{
Action: Amend,
Asks: []orderbook.Item{
update,
},
})
if err != nil {
t.Fatal(err)
}
if book.Asks[0].Amount != 1337 {
t.Fatal("element not updated")
}
}
func TestFlushOrderbook(t *testing.T) {
w := &Orderbook{}
err := w.Setup(5, false, false, false, false, "test", make(chan interface{}, 2))
if err != nil {
t.Fatal(err)
}
var snapShot1 orderbook.Base
snapShot1.ExchangeName = "Snapshooooot"
asks := []orderbook.Item{
{Price: 4000, Amount: 1, ID: 8},
}
bids := []orderbook.Item{
{Price: 4000, Amount: 1, ID: 9},
}
snapShot1.Asks = asks
snapShot1.Bids = bids
snapShot1.AssetType = asset.Spot
snapShot1.Pair = cp
err = w.FlushOrderbook(cp, asset.Spot)
if err == nil {
t.Fatal("book not loaded error cannot be nil")
}
o := w.GetOrderbook(cp, asset.Spot)
if o != nil {
t.Fatal("book not loaded, this should not happen")
}
err = w.LoadSnapshot(&snapShot1)
if err != nil {
t.Fatal(err)
}
err = w.LoadSnapshot(&snapShot1)
if err != nil {
t.Fatal(err)
}
err = w.FlushOrderbook(cp, asset.Spot)
if err != nil {
t.Fatal(err)
}
o = w.GetOrderbook(cp, asset.Spot)
if o == nil {
t.Fatal("cannot get book")
}
if o.Bids != nil && o.Asks != nil {
t.Fatal("orderbook items not flushed")
}
}

37
exchanges/stream/buffer/buffer_types.go
View File

@@ -12,8 +12,7 @@ import (
// Orderbook defines a local cache of orderbooks for amending, appending
// and deleting changes and updates the main store for a stream
type Orderbook struct {
ob map[currency.Pair]map[asset.Item]*orderbook.Base
buffer map[currency.Pair]map[asset.Item][]*Update
ob map[currency.Code]map[currency.Code]map[asset.Item]*orderbookHolder
obBufferLimit int
bufferEnabled bool
sortBuffer bool
@@ -24,13 +23,39 @@ type Orderbook struct {
m sync.Mutex
}
type orderbookHolder struct {
ob *orderbook.Base
buffer *[]Update
}
// Update stores orderbook updates and dictates what features to use when processing
type Update struct {
UpdateID int64 // Used when no time is provided
UpdateTime time.Time
Asset asset.Item
Action string // Used in conjunction with UpdateEntriesByID
Bids []orderbook.Item
Asks []orderbook.Item
Pair currency.Pair
Action
Bids []orderbook.Item
Asks []orderbook.Item
Pair currency.Pair
// Determines if there is a max depth of orderbooks and after an append we
// should remove any items that are outside of this scope. Kraken is the
// only exchange utilising this field.
MaxDepth int
}
// Action defines a set of differing states required to implement an incoming
// orderbook update used in conjunction with UpdateEntriesByID
type Action string
const (
// Amend applies amount adjustment by ID
Amend Action = "update"
// Delete removes price level from book by ID
Delete Action = "delete"
// Insert adds price level to book
Insert Action = "insert"
// UpdateInsert on conflict applies amount adjustment or appends new amount
// to book
UpdateInsert Action = "update/insert"
)

3
exchanges/stream/websocket.go
View File

@@ -108,14 +108,13 @@ func (w *Websocket) Setup(s *WebsocketSetup) error {
w.Wg = new(sync.WaitGroup)
w.SetCanUseAuthenticatedEndpoints(s.AuthenticatedWebsocketAPISupport)
w.Orderbook.Setup(s.OrderbookBufferLimit,
return w.Orderbook.Setup(s.OrderbookBufferLimit,
s.BufferEnabled,
s.SortBuffer,
s.SortBufferByUpdateIDs,
s.UpdateEntriesByID,
w.exchangeName,
w.DataHandler)
return nil
}
// SetupNewConnection sets up an auth or unauth streaming connection

3
exchanges/stream/websocket_test.go
View File

@@ -92,7 +92,7 @@ func TestSetup(t *testing.T) {
if err == nil {
t.Fatal("error cannot be nil")
}
w = &Websocket{}
w = &Websocket{DataHandler: make(chan interface{})}
err = w.Setup(nil)
if err == nil {
t.Fatal("error cannot be nil")
@@ -1207,6 +1207,7 @@ func TestSetupNewConnection(t *testing.T) {
Init: true,
TrafficAlert: make(chan struct{}),
ReadMessageErrors: make(chan error),
DataHandler: make(chan interface{}),
}
err = web.Setup(defaultSetup)

8
exchanges/ticker/ticker_types.go
View File

@@ -47,6 +47,14 @@ type Price struct {
ExchangeName string `json:"exchangeName"`
AssetType asset.Item `json:"assetType"`
LastUpdated time.Time
// Funding rate field variables
FlashReturnRate float64
BidPeriod float64
BidSize float64
AskPeriod float64
AskSize float64
FlashReturnRateAmount float64
}
// Ticker struct holds the ticker information for a currency pair and type

20
exchanges/yobit/yobit_wrapper.go
View File

@@ -236,18 +236,18 @@ func (y *Yobit) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderboo
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (y *Yobit) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: y.Name, Pair: p, AssetType: assetType}
fpair, err := y.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := y.GetDepth(fpair.String())
if err != nil {
return orderBook, err
return book, err
}
for i := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids,
book.Bids = append(book.Bids,
orderbook.Item{
Price: orderbookNew.Bids[i][0],
Amount: orderbookNew.Bids[i][1],
@@ -255,22 +255,16 @@ func (y *Yobit) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbo
}
for i := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks,
book.Asks = append(book.Asks,
orderbook.Item{
Price: orderbookNew.Asks[i][0],
Amount: orderbookNew.Asks[i][1],
})
}
orderBook.Pair = p
orderBook.ExchangeName = y.Name
orderBook.AssetType = assetType
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(y.Name, p, assetType)
}

2
exchanges/zb/zb.go
View File

@@ -170,7 +170,7 @@ func (z *ZB) GetTicker(symbol string) (TickerResponse, error) {
return res, err
}
// GetTicker returns a ticker for a given symbol
// GetTrades returns trades for a given symbol
func (z *ZB) GetTrades(symbol string) (TradeHistory, error) {
urlPath := fmt.Sprintf("%s/%s/%s/%s?market=%s", z.API.Endpoints.URL, zbData, zbAPIVersion, zbTrades, symbol)
var res TradeHistory

1
exchanges/zb/zb_types.go
View File

@@ -229,6 +229,7 @@ var orderSideMap = map[int64]order.Side{
1: order.Sell,
}
// TradeHistory defines a slice of historic trades
type TradeHistory []struct {
Amount float64 `json:"amount,string"`
Date int64 `json:"date"`

19
exchanges/zb/zb_websocket.go
View File

@@ -122,17 +122,16 @@ func (z *ZB) wsHandleData(respRaw []byte) error {
return err
}
var asks []orderbook.Item
var book orderbook.Base
for i := range depth.Asks {
asks = append(asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: depth.Asks[i][1].(float64),
Price: depth.Asks[i][0].(float64),
})
}
var bids []orderbook.Item
for i := range depth.Bids {
bids = append(bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: depth.Bids[i][1].(float64),
Price: depth.Bids[i][0].(float64),
})
@@ -144,14 +143,12 @@ func (z *ZB) wsHandleData(respRaw []byte) error {
return err
}
var newOrderBook orderbook.Base
newOrderBook.Asks = asks
newOrderBook.Bids = bids
newOrderBook.AssetType = asset.Spot
newOrderBook.Pair = cPair
newOrderBook.ExchangeName = z.Name
orderbook.Reverse(book.Asks) // Reverse asks for correct alignment
book.AssetType = asset.Spot
book.Pair = cPair
book.ExchangeName = z.Name
err = z.Websocket.Orderbook.LoadSnapshot(&newOrderBook)
err = z.Websocket.Orderbook.LoadSnapshot(&book)
if err != nil {
return err
}

21
exchanges/zb/zb_wrapper.go
View File

@@ -167,6 +167,7 @@ func (z *ZB) Setup(exch *config.ExchangeConfig) error {
Subscriber: z.Subscribe,
Features: &z.Features.Supports.WebsocketCapabilities,
OrderbookBufferLimit: exch.WebsocketOrderbookBufferLimit,
BufferEnabled: exch.WebsocketOrderbookBufferEnabled,
})
if err != nil {
return err
@@ -291,40 +292,34 @@ func (z *ZB) FetchOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.B
// UpdateOrderbook updates and returns the orderbook for a currency pair
func (z *ZB) UpdateOrderbook(p currency.Pair, assetType asset.Item) (*orderbook.Base, error) {
orderBook := new(orderbook.Base)
book := &orderbook.Base{ExchangeName: z.Name, Pair: p, AssetType: assetType}
currFormat, err := z.FormatExchangeCurrency(p, assetType)
if err != nil {
return nil, err
return book, err
}
orderbookNew, err := z.GetOrderbook(currFormat.String())
if err != nil {
return orderBook, err
return book, err
}
for x := range orderbookNew.Bids {
orderBook.Bids = append(orderBook.Bids, orderbook.Item{
book.Bids = append(book.Bids, orderbook.Item{
Amount: orderbookNew.Bids[x][1],
Price: orderbookNew.Bids[x][0],
})
}
for x := range orderbookNew.Asks {
orderBook.Asks = append(orderBook.Asks, orderbook.Item{
book.Asks = append(book.Asks, orderbook.Item{
Amount: orderbookNew.Asks[x][1],
Price: orderbookNew.Asks[x][0],
})
}
orderBook.Pair = p
orderBook.AssetType = assetType
orderBook.ExchangeName = z.Name
err = orderBook.Process()
err = book.Process()
if err != nil {
return orderBook, err
return book, err
}
return orderbook.Get(z.Name, p, assetType)
}

3
go.sum
View File

@@ -252,8 +252,7 @@ github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzR
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
github.com/mattn/go-sqlite3 v1.14.5 h1:1IdxlwTNazvbKJQSxoJ5/9ECbEeaTTyeU7sEAZ5KKTQ=
github.com/mattn/go-sqlite3 v1.14.5/go.mod h1:WVKg1VTActs4Qso6iwGbiFih2UIHo0ENGwNd0Lj+XmI=
github.com/mattn/go-sqlite3 v1.14.6 h1:dNPt6NO46WmLVt2DLNpwczCmdV5boIZ6g/tlDrlRUbg=
github.com/mattn/go-sqlite3 v1.14.6/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=

8125
testdata/http_mock/binance/binance.json vendored
View File

File diff suppressed because it is too large Load Diff